could not read header: message: malformed MIME header line

I'm subscribed to one mailing list whose messages have a weird long header, like this:

X-Xfinity-VAAS: gggruggvucftvghtrhhoucdtuddrgedujedrvdeggdekhecutefuodetggdotefrodftvfcurfhrohhfihhlvgemucevohhmtggrshhtqdftvghsihdpqfgfvfdppffquffrtefokffrnecuuegrihhlohhuthemuceftddunecusecvtfgvtghiphhivghnthhsucdlqddutddtmdenogetfedtuddqtdduucdludehmdenucfjughrpefhtgggfffukffvofesrgdtmherhhdtjeenucfhrhhomhepffhonhhnrgcuoeguohhnnhgrughinhhgihhllhhoudestghomhgtrghsthdrnhgvtheqnecuggftrfgrthhtvghrnhepfeevhfelieevffejvddtvdevgfdvhefhtdffgfefkefgleevgeefgfevvdeigeetnecuffhomhgrihhnpehinhhsthhithhuthgvtghhrhhishhttghonhhstghiohhushhnvghsshdrohhrghdpiihoohhmrdhushenucfkphepudefjedruddtfedrvddvtddrvdehudenucevlhhushhtvghrufhiiigvpedtnecurfgrrhgrmhephhgvlhhopegludelvddrudeikedruddrgegnpdhinhgvthepudefjedruddtfedrvddvtddrvdehuddpmhgrihhlfhhrohhmpeguohhnnhgrughinhhgihhllhhoudestghomhgtrghsthdrnhgvthdprhgtphhtthhopeguohhnnhgrughinhhgihhllhhoudestghomhgtrghsthdrnhgvthdprhgtphhtthhopehjihhmtghlvghvvghlrghnugesmhgvrdgtohhmpdhrtghpthhtoheprhgvtghorhgushholhhuthhiohhnshesrgholhdrtghomhdprhgtphhtth

The problem appears to be that the header is broken into multiple lines, some of which do not start with a space. It seems that this header - along with a shorter X-Xfinity-VMeta header - is added by the sender's MTA, Comcast. The result is that aerc shows this error in the status line at the bottom and proceeds to not render the message list for that mailbox any further. I then use another mail program to move the offending message out of the way, and restart aerc.

I agree that this header is malformed. But it would be nice if this didn't break things. I am able to view the message normally in Sylpheed, Claws Mail, and K-9 Mail. In Claws Mail when I choose "Show All Headers", it appears to try to interpret the subsequent lines as additional headers. But Sylpheed and K-9 Mail show the whole thing as one long header.

Possible solutions:

  • Treat the additional unindented header lines as part of the same header.
  • Treat the additional unindented header lines as separate headers without a value.
  • Discard the additional unindented header lines, maybe showing a warning.
  • Continue to abort rendering this message, but continue processing other messages in the mailbox, and support operations like moving the message to another mailbox, or piping/saving it to a file.

Non-aerc solutions:

  • I configure my postfix MDA to ignore or repair the broken header
  • Comcast MTA - or whatever is doing it - stops adding this broken header
  • Sender stops using Comcast MTA
Assigned to
3 years ago
2 years ago
No labels applied.

~yzzyx 3 years ago

This issue probably comes from the fact that go-message does not accept headers longer than 4000 chars.

An issue has been raised in go-message, and can be seen here: https://github.com/emersion/go-message/issues/110

~mato 3 years ago

Hi. Potential new aerc user here, I just tried latest master against my mail archive of 20+ years and ran into this issue. AFAICT, in my case it's not due to the length of the header in question (see below). Here's an excerpt from the aerc log:

2020/10/01 15:29:08 ->(ui) *types.Error:*types.FetchMessageHeaders
2020/10/01 15:29:08 (ui)<= *types.Error(77):*types.FetchMessageHeaders(76)
2020/10/01 15:29:08 could not read header: message: malformed MIME header key: From ocaml+verp-e302c1e62be500c5e8b6d21ad8bd39dd@discoursemail.com  Sun Sep 27 22

There's nothing outlandishly long about the above header, these are just standard mails from discourse. Just to be sure, I took a look at the linked go-message issue, and patched the version aerc uses to increase maxLineOctets, it did not help.

~labrat 3 years ago

~mato show the raw headers of that mail

~mato 3 years ago*

~labrat Here you go, raw from Maildir/...:

From ocaml+verp-e302c1e62be500c5e8b6d21ad8bd39dd@discoursemail.com  Sun Sep 27 22:21:57 2020
Return-Path: <ocaml+verp-e302c1e62be500c5e8b6d21ad8bd39dd@discoursemail.com>
X-Original-To: XXXXXX@lucina.net
Delivered-To: XXXX@smtp.lucina.net
Received-SPF: Pass (sender SPF authorized) identity=mailfrom; client-ip=; helo=mx-out-01b.sjc2.discourse.cloud; envelope-from=ocaml+verp-e302c1e62be500c5e8b6d21ad8bd39dd@discoursemail.com; receiver=XXXXXX@lucina.net 
Authentication-Results: smtp.lucina.net; dkim=pass
	reason="2048-bit key; unprotected key"
	header.d=discoursemail.com header.i=@discoursemail.com
	header.b=A76H01pm; dkim-adsp=pass; dkim-atps=neutral
Received: from mx-out-01b.sjc2.discourse.cloud (mx-out-01b.sjc2.discourse.cloud [])
	by smtp.lucina.net (Postfix) with ESMTPS id 7E8E4122804
	for <XXXXXX@lucina.net>; Sun, 27 Sep 2020 22:21:56 +0200 (CEST)
Received: from localhost.localdomain (unknown [IPv6:2001:470:14e:2::2ef:8bb1:35b])
	by mx-out-01b.sjc2.discourse.cloud (Postfix) with ESMTP id BE2AF140F65
	for <XXXXXX@lucina.net>; Sun, 27 Sep 2020 20:21:53 +0000 (UTC)
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=discoursemail.com;
	s=sjc2; t=1601238113;
Date: Sun, 27 Sep 2020 20:21:53 +0000
From: OCaml <ocaml@discoursemail.com>
Reply-To: OCaml <ocaml@discoursemail.com>
To: XXXXXX@lucina.net
Message-ID: <070b4d50-1b7a-4af9-ad6e-34469b824755@discuss.ocaml.org>
Subject: [OCaml] Summary
Mime-Version: 1.0
Content-Type: multipart/alternative;
Content-Transfer-Encoding: 7bit
List-Unsubscribe: <https://discuss.ocaml.org/email/unsubscribe/REDACTED>
X-Auto-Response-Suppress: All
Auto-Submitted: auto-generated
Feedback-ID: ocaml:digest:discoursemail

Now I'm wondering why I have a mbox-style From xxxxx envelope sender line in my Maildir/s, but AFAICT it's always been that way.

~labrat 3 years ago

still broken, it's not a valid header and as such not parsable.

~mato 3 years ago

~labrat You're referring to the first line of the message headers, i.e. From xxxxxx? I agree it's not a valid header. However, something in my Postfix setup is adding it; I suspect maildrop which is being used as the LDA. Also, other mail clients (mutt, mailx, roundcube, even Windows Mail) don't complain about it, presumably due to legacy UNIX mbox-handling code?

~labrat 3 years ago

other clients don't care because they adapted to people who send broken email. As broken mails were not an issue people kept sending broken emails... meaning those clients are part of the problem.

If postfix adds it, you have a misconfigured postfix instance Would have been much better to just say "broken"

~labrat 3 years ago

bah, scratch the last part... typing on a mobile...

~mato 3 years ago

(Edited the headers a bit for privacy)

~labrat Look, I'm just trying to be helpful. I'll see what I can do to fix my Postfix instance, but I obviously need to take care not to lose data or break a setup that has been working fine for years.

~labrat 3 years ago

well, "working fine" is not how I would put it if you set invalid headers. Don't take this as an offense btw, just stating a fact.

The problem is how to handle it... Sure we could work around it by implementing our own parser, but how broken do you wanna get?

The code currently uses the information in the header to do things like show the proper fields in the msglist, sort them etc. If we can't even assume a basic header set (from,to date) what shall we do?

It's all specified in RFCs how a mail should look. Now, both examples here are invalid mails yours doesn't add a : and OP's has continuation lines without preceding whitespace.

How do we want to parse that? from the point where we encounter the error the structure is invalid. Should we just assume that the thing only has one error and the rest is fine? What if it's not?

~mato 3 years ago

~labrat I don't have any good answers, but given that e-mail is an ancient protocol some judicious application of Postel's law [1] in aerc is warranted. I managed to make things work for me by patching go-message to ignore the first header line when parsing, if and only if it starts with From<SP>; I can post that patch here if you're interested.

[1] https://en.wikipedia.org/wiki/Robustness_principle

~labrat 3 years ago

nah no need. We don't do such particular workarounds, if we would do something it would need to be more robust than that.

Mixing mbox and maildir certainly isn't a common issue anymore, considering that all common mail sync tools do follow the spec (isync, offlineimap) and even a default dovecot setup uses maildir or is at least not hard to setup with maildir (heck, I could manage it ;) )

There's some discussion going on here: https://github.com/emersion/go-message/pull/101

~jordan31 3 years ago

Well after reading this I'm a little relieved to know the issue and that the fault lies with my school. But, still stinks that this wont be fixed (or broken ;)). I'm under the impression I will need to use another email client for access my school's email, and keep aerc for my personal emails. At the very least I learned something new from all the reading on this topic!


~ceuk 2 years ago

~mato I'd be interested in your workaround if it's still valid? Every single internal email sent by the company I'm contracting for has malformed headers :(

~mato 2 years ago

~ceuk I'm using this patch against a locally vendored copy of go-message:

--- a/textproto/header.go
+++ b/textproto/header.go
@@ -485,7 +485,7 @@ func trimAroundNewlines(v []byte) string {

 const (
        maxHeaderLines = 1000
-       maxLineOctets  = 4000
+       maxLineOctets  = 8000

 // ReadHeader reads a MIME header from r. The header is a sequence of possibly
@@ -493,6 +493,10 @@ const (
 func ReadHeader(r *bufio.Reader) (Header, error) {
        fs := make([]*headerField, 0, 32)

+       if buf, err := r.Peek(5); err == nil && bytes.Equal(buf, []byte("From ")) {
+               _, _ = readLineSlice(r, nil)
+       }
        // The first line cannot start with a leading space.
        if buf, err := r.Peek(1); err == nil && isSpace(buf[0]) {
                line, err := readLineSlice(r, nil)
Register here or Log in to comment, or comment via email.