I'm subscribed to one mailing list whose messages have a weird long header, like this:
X-Xfinity-VAAS: gggruggvucftvghtrhhoucdtuddrgedujedrvdeggdekhecutefuodetggdotefrodftvfcurfhrohhfihhlvgemucevohhmtggrshhtqdftvghsihdpqfgfvfdppffquffrtefokffrnecuuegrihhlohhuthemuceftddunecusecvtfgvtghiphhivghnthhsucdlqddutddtmdenogetfedtuddqtdduucdludehmdenucfjughrpefhtgggfffukffvofesrgdtmherhhdtjeenucfhrhhomhepffhonhhnrgcuoeguohhnnhgrughinhhgihhllhhoudestghomhgtrghsthdrnhgvtheqnecuggftrfgrthhtvghrnhepfeevhfelieevffejvddtvdevgfdvhefhtdffgfefkefgleevgeefgfevvdeigeetnecuffhomhgrihhnpehinhhsthhithhuthgvtghhrhhishhttghonhhstghiohhushhnvghsshdrohhrghdpiihoohhmrdhushenucfkphepudefjedruddtfedrvddvtddrvdehudenucevlhhushhtvghrufhiiigvpedtnecurfgrrhgrmhephhgvlhhopegludelvddrudeikedruddrgegnpdhinhgvthepudefjedruddtfedrvddvtddrvdehuddpmhgrihhlfhhrohhmpeguohhnnhgrughinhhgihhllhhoudestghomhgtrghsthdrnhgvthdprhgtphhtthhopeguohhnnhgrughinhhgihhllhhoudestghomhgtrghsthdrnhgvthdprhgtphhtthhopehjihhmtghlvghvvghlrghnugesmhgvrdgtohhmpdhrtghpthhtoheprhgvtghorhgushholhhuthhiohhnshesrgholhdrtghomhdprhgtphhtth
hopegtrghthhgrrhhtvghmphhlvgesghhmrghilhdrtghomhdprhgtphhtthhopegsohhnghhlrggtshhonhesihgvvhholhhvvghphhdrtghomhdprhgtphhtthhopegrmhgsvghrghgvnhgvsehssggtghhlohgsrghlrdhnvghtpdhrtghpthhtohepmhgvrhhmrghiughlohhrvgesihgtlhhouhgurdgtohhmpdhrtghpthhtoheprghnnhgrlhihnhehiedvsehgmhgrihhlrdgtohhmpdhrtghpthhtohepjhgrtghquhgvshdrrhgvvhgvrhhsvggruhesghhmrghilhdrtghomhdprhgtphhtthhopegurghnihgvlhgprhhivhejjeeshigrhhhoohdrtghomhdprhgtphhtthhopehqohhinhhsthdvsehgmhgrihhlrdgtohhmpdhrtghpthhtohepghhrvggvshgvleesmhhsnhdrtghomhdprhgtphhtthhopehtlhhuvhhlhihlihhghhhtsehgmhgrihhlrdgtohhmpdhrtghpthhtohepuhgsrdhomhgrhhgrsehgmhgrihhlrdgtohhmpdhrtghpthhtohepjhgrnhgvthgpfhgrlhgsohesmhgrtgdrtghomhdprhgtphhtthhopeguhihlrghnrheshiejmhgrihhlrdgtohhmpdhrtghpthhtohepohhnohhjrghfrhhiuggrhidvtddtgeesghhmrghilhdrtghomhdprhgtphhtthhopehmlhhorhgvlhhlvdelsehgmhgrihhlrdgtohhmpdhrtghpthhtohepsghilhhlhihgrhgrnhhtgiesghhmrghilhdrtghomhdprhgtphhtthhopehsrghrshhurhhsuhhmseihrghhohhordgtohhmpdhrtghpthhtohephhiivghrudestggvnhh
tuhhrhihlihhnkhdrnhgvthdprhgtphhtthhopehhvggrthhhvghr
tggrrhhriedtieesihgtlhhouhgurdgtohhmpdhrtghpthhtoheprghrtghshhhivghlughstghtsehfrhhonhhtihgvrhdrtghomhdprhgtphhtthhopegshihrohhnsehorhhighhinhhprhgvshhsrdgtohhmpdhrtghpthhtoheprhhosghinhhsmhhitghhrggvlhduudduudeshigrhhhoohdrtghomhdprhgtphhtthhopehiuggvrghlnhgvthifohhrkhhprhhojhgvtghtshesghhmrghilhdrtghomhdprhgtphhtthhopegrnhgurhgvrgdrlhhishhsvghnsggvrhhgsehgmhgrihhlrdgtohhmpdhrtghpthhtoheprghnnhhsrghfvgesjhhunhhordgtohhmpdhrtghpthhtohepughrghhnohhrrghtohesvhgvrhhiiihonhdrnhgvthdprhgtphhtthhopehknhhighhhthhguhgrrhguihgrnhdusehgmhgrihhlrdgtohhmpdhrtghpthhtohepughouhgslhgvmhgvrhhkrggsrgeshhhothhmrghilhdrtghomhdprhgtphhtthhopehjrggtvhgrihhllhgrnhgtohhurhhtseihrghhohhordgtrgdprhgtphhtthhopehfthhlvggrshhttghhrghmsggvrheshigrhhhoohdrtghomhdprhgtphhtthhopehgvghrrghlughtfhgrrhhlvgihsehhohhtmhgrihhlrdgtohhmpdhrtghpthhtohepsgguvghruhihthgvrheshigrhhhoohdrtghomhdprhgtphhtthhopehjvghsrghikhgrfhhithiiseihrghhohhordgtohhmpdhrtghpthhtohepnhhuthhrihhvvggurgeshhhothhmrghilhdrtghomhdprhgtphhtthhopehjoh
hhnhhfudehfeeshigrhhhoohdrtghomhdprhgtphhtthhopehlrghhrhgvvggusehhohhtmhgrihhlrdgtohhmpdhrtghpthhtohepshgsihguihgrkhhithgvseihrghhohhordgtohhmpdhrtghpthhtoheprggtkhhuudesohhuthhlohhokhdrtghomhdprhgtphhtthhopehphhhilhhiphhtvggthhhpohhinhhtseihrghhohhordgtohhmpdhrtghpthhtohepohgrkhhlvgihvddvsegsvghllhhsohhuthhhrdhnvghtpdhrtghpthhtoheptghoohhkihgvmhhonhhsthgvrhdtjeejkeesghhmrghilhdrtghomhdprhgtphhtthhopehjohhsvghphhdrvhgrtghkrgihihhlsehgmhgrihhlrdgtohhmpdhrtghpthhtoheplhhovhgvjhgvnhhnrgesghhmrghilhdrtghomhdprhgtphhtthhopegsvggtkhholhgrjeejseihrghhohhordgtohhmpdhrtghpthhtohepvhhmrggtkhhkieelsehgmhgrihhlrdgtohhmpdhrtghpthhtohepvggumhhunhgukhhuvghllhesghhmrghilhdrtghomhdprhgtphhtthhopehmihgthhgvlhhlvghrvghiugdufedufeeshigrhhhoohdrtghomhdprhgtphhtthhopehmsggrrhhkvghrmhhorhhrihhsseihrghhohhordgtohhmpdhrtghpthhtoheprhefmhhilhhlvghrsegtohhmtggrshhtrdhnvghtpdhrtghpthhtoheptggrrhhlrggtlhgrshhonhesghhmrghilhdrtghomhdprhgtphhtthhopeguvggvshhtihgtkhestghomhgtrghsthdrnhgvthdprhgtphhtthhopeifvghrohg
tkhifvghrohhllhesghhmrghilhdrtghomhdprhgtphhtthhopehj
rghnlhgrphhpfeejfeejsehgmhgrihhlrdgtohhmpdhrtghpthhtohepughonhhsmhihthhhgedvsehgmhgrihhlrdgtohhmpdhrtghpthhtohepmhgrrhhivggsihhtvghnghesghhmrghilhdrtghomhdprhgtphhtthhopegsrghrsghoohhoohegsehgmhgrihhlrdgtohhmpdhrtghpthhtoheplhhinhgurghlohhruggrnheshigrhhhoohdrtghomhdprhgtphhtthhopehsthgttgdrtggvlhestggvlhgvhhhnvghrrdgtohhmpdhrtghpthhtoheplhgvohhnrdhvrghnuggvrhhpohhlsehgmhgrihhlrdgtohhmpdhrtghpthhtohepihgrmhhlrghurhgrmhgrhhgvrhesghhmrghilhdrtghomhdprhgtphhtthhopegrthhsvghrvghmrgesshgrrhhsrdhgohhvrdiirgdprhgtphhtthhopegvughlohhvvgdvsehouhhtlhhoohhkrdgtohhmpdhrtghpthhtohepghholhguvghnthhimhgviiejjeesghhmrghilhdrtghomhdprhgtphhtthhopehkrgihuhhmrghnghhgihhorhhgrghnihgtsehgmhgrihhlrdgtohhmpdhrtghpthhtohepmhgrrhihlhhivhhinhhgshhtohhnsehushgrrdhnvghtpdhrtghpthhtoheplhgruhhrihgvieeljeestghomhgtrghsthdrnhgvthdprhgtphhtthhopegslhhuvghmohhonheggeegsehssggtghhlohgsrghlrdhnvghtpdhrtghpthhtohepghhgsehgvghrrghlhihnrdhushdprhgtphhtthhopehkrghgtddvleeksehhohhtmhgrihhlrdgtohhmpdhrtghpthhtohepmhhsrghlth
The problem appears to be that the header is broken into multiple lines, some of which do not start with a space. It seems that this header - along with a shorter X-Xfinity-VMeta header - is added by the sender's MTA, Comcast. The result is that aerc shows this error in the status line at the bottom and proceeds to not render the message list for that mailbox any further. I then use another mail program to move the offending message out of the way, and restart aerc.
I agree that this header is malformed. But it would be nice if this didn't break things. I am able to view the message normally in Sylpheed, Claws Mail, and K-9 Mail. In Claws Mail when I choose "Show All Headers", it appears to try to interpret the subsequent lines as additional headers. But Sylpheed and K-9 Mail show the whole thing as one long header.
Possible solutions:
Non-aerc solutions:
This issue probably comes from the fact that go-message does not accept headers longer than 4000 chars.
An issue has been raised in go-message, and can be seen here: https://github.com/emersion/go-message/issues/110
Hi. Potential new aerc user here, I just tried latest master against my mail archive of 20+ years and ran into this issue. AFAICT, in my case it's not due to the length of the header in question (see below). Here's an excerpt from the aerc log:
2020/10/01 15:29:08 ->(ui) *types.Error:*types.FetchMessageHeaders UbePfg IDLE 2020/10/01 15:29:08 (ui)<= *types.Error(77):*types.FetchMessageHeaders(76) 2020/10/01 15:29:08 could not read header: message: malformed MIME header key: From ocaml+verp-e302c1e62be500c5e8b6d21ad8bd39dd@discoursemail.com Sun Sep 27 22There's nothing outlandishly long about the above header, these are just standard mails from discourse. Just to be sure, I took a look at the linked go-message issue, and patched the version aerc uses to increase
maxLineOctets, it did not help.
~mato show the raw headers of that mail
~labrat Here you go, raw from
Maildir/...:From ocaml+verp-e302c1e62be500c5e8b6d21ad8bd39dd@discoursemail.com Sun Sep 27 22:21:57 2020 Return-Path: <ocaml+verp-e302c1e62be500c5e8b6d21ad8bd39dd@discoursemail.com> X-Original-To: XXXXXX@lucina.net Delivered-To: XXXX@smtp.lucina.net Received-SPF: Pass (sender SPF authorized) identity=mailfrom; client-ip=66.220.12.154; helo=mx-out-01b.sjc2.discourse.cloud; envelope-from=ocaml+verp-e302c1e62be500c5e8b6d21ad8bd39dd@discoursemail.com; receiver=XXXXXX@lucina.net Authentication-Results: smtp.lucina.net; dkim=pass reason="2048-bit key; unprotected key" header.d=discoursemail.com header.i=@discoursemail.com header.b=A76H01pm; dkim-adsp=pass; dkim-atps=neutral Received: from mx-out-01b.sjc2.discourse.cloud (mx-out-01b.sjc2.discourse.cloud [66.220.12.154]) by smtp.lucina.net (Postfix) with ESMTPS id 7E8E4122804 for <XXXXXX@lucina.net>; Sun, 27 Sep 2020 22:21:56 +0200 (CEST) Received: from localhost.localdomain (unknown [IPv6:2001:470:14e:2::2ef:8bb1:35b]) by mx-out-01b.sjc2.discourse.cloud (Postfix) with ESMTP id BE2AF140F65 for <XXXXXX@lucina.net>; Sun, 27 Sep 2020 20:21:53 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=discoursemail.com; s=sjc2; t=1601238113; bh=SsOPhhQHxgYN9sJBiKt7oBkNnCDS9YoPf8MPnyOxHmM=; h=Date:From:Reply-To:To:Subject:List-Unsubscribe; b=A76H01pm65XtixKTwK4HQtBnowHFUG9xzbwbiWKyVrB7ADkTNYbLVlgRKkxX3SaRO DPicT+5+c6EIYKFdanmq9d+LlRwgKExu1Tu96QRmFOfkOt2UHg3UTVMrfCuEhSb+BS znX9jF1izJOUG94c83I9vFqqwt3XUeuFO4ydpy5ec1wxo0nrN7i1mZkjz+IXo+nJL+ v18PSHbHdqnCGX0iAcXR7u0pFFE7MbsKDmCE19f+Y5GOrQxzwVjkv7atWUZn8vdwUg 3gDYvBTwZa518O6XszgaKs50iOKl+wk6LrjyEIcb/VA5mSwCEUZ/JvCSSTWurmjw3F zYL7WrLwRx7tA== Date: Sun, 27 Sep 2020 20:21:53 +0000 From: OCaml <ocaml@discoursemail.com> Reply-To: OCaml <ocaml@discoursemail.com> To: XXXXXX@lucina.net Message-ID: <070b4d50-1b7a-4af9-ad6e-34469b824755@discuss.ocaml.org> Subject: [OCaml] Summary Mime-Version: 1.0 Content-Type: multipart/alternative; boundary="--==_mimepart_5f70f46019d41_3e2b3ff44a8fe95828483eb"; charset=UTF-8 Content-Transfer-Encoding: 7bit List-Unsubscribe: <https://discuss.ocaml.org/email/unsubscribe/REDACTED> X-Auto-Response-Suppress: All Auto-Submitted: auto-generated Feedback-ID: ocaml:digest:discoursemailNow I'm wondering why I have a mbox-style
From xxxxxenvelope sender line in myMaildir/s, but AFAICT it's always been that way.
still broken, it's not a valid header and as such not parsable.
~labrat You're referring to the first line of the message headers, i.e.
From xxxxxx? I agree it's not a valid header. However, something in my Postfix setup is adding it; I suspectmaildropwhich is being used as the LDA. Also, other mail clients (mutt, mailx, roundcube, even Windows Mail) don't complain about it, presumably due to legacy UNIX mbox-handling code?
other clients don't care because they adapted to people who send broken email. As broken mails were not an issue people kept sending broken emails... meaning those clients are part of the problem.
If postfix adds it, you have a misconfigured postfix instance Would have been much better to just say "broken"
bah, scratch the last part... typing on a mobile...
(Edited the headers a bit for privacy)
~labrat Look, I'm just trying to be helpful. I'll see what I can do to fix my Postfix instance, but I obviously need to take care not to lose data or break a setup that has been working fine for years.
well, "working fine" is not how I would put it if you set invalid headers. Don't take this as an offense btw, just stating a fact.
The problem is how to handle it... Sure we could work around it by implementing our own parser, but how broken do you wanna get?
The code currently uses the information in the header to do things like show the proper fields in the msglist, sort them etc. If we can't even assume a basic header set (from,to date) what shall we do?
It's all specified in RFCs how a mail should look. Now, both examples here are invalid mails yours doesn't add a : and OP's has continuation lines without preceding whitespace.
How do we want to parse that? from the point where we encounter the error the structure is invalid. Should we just assume that the thing only has one error and the rest is fine? What if it's not?
~labrat I don't have any good answers, but given that e-mail is an ancient protocol some judicious application of Postel's law [1] in aerc is warranted. I managed to make things work for me by patching go-message to ignore the first header line when parsing, if and only if it starts with
From<SP>; I can post that patch here if you're interested.
nah no need. We don't do such particular workarounds, if we would do something it would need to be more robust than that.
Mixing mbox and maildir certainly isn't a common issue anymore, considering that all common mail sync tools do follow the spec (isync, offlineimap) and even a default dovecot setup uses maildir or is at least not hard to setup with maildir (heck, I could manage it ;) )
There's some discussion going on here: https://github.com/emersion/go-message/pull/101
Well after reading this I'm a little relieved to know the issue and that the fault lies with my school. But, still stinks that this wont be fixed (or broken ;)). I'm under the impression I will need to use another email client for access my school's email, and keep aerc for my personal emails. At the very least I learned something new from all the reading on this topic!
Jordan
~mato I'd be interested in your workaround if it's still valid? Every single internal email sent by the company I'm contracting for has malformed headers :(
~ceuk I'm using this patch against a locally vendored copy of go-message:
--- a/textproto/header.go +++ b/textproto/header.go @@ -485,7 +485,7 @@ func trimAroundNewlines(v []byte) string { const ( maxHeaderLines = 1000 - maxLineOctets = 4000 + maxLineOctets = 8000 ) // ReadHeader reads a MIME header from r. The header is a sequence of possibly @@ -493,6 +493,10 @@ const ( func ReadHeader(r *bufio.Reader) (Header, error) { fs := make([]*headerField, 0, 32) + if buf, err := r.Peek(5); err == nil && bytes.Equal(buf, []byte("From ")) { + _, _ = readLineSlice(r, nil) + } + // The first line cannot start with a leading space. if buf, err := r.Peek(1); err == nil && isSpace(buf[0]) { line, err := readLineSlice(r, nil)