~rjarry/aerc#9: 
"too many open files" error on maildir

My maildir archive folder has over 9000 messages and most of the times aerc fail to display it with the following error:

open /path/to/maildir/path/to/one/message: too many open files

I suppose that this is due to too many files are being opened in parallel before enough is closed and the number grows larger than the OS's limit.

Status
RESOLVED FIXED
Submitter
~cnx
Assigned to
No-one
Submitted
7 months ago
Updated
6 months ago
Labels
No labels applied.

~richardhuxton 6 months ago

I think the error is in worker/maildir/message.go method NewReader. That method claims to read a message into memory but in fact opens a file, returns an io.Reader based on it and then looks like it leaks the filehandle.

I tested the patch shown below using ulimit -n 40 and then starting aerc from the same shell. Without the change you get the "..." loading indicator that gets stuck. With the change, it seems to load just fine.

I'm not very familiar with Go's standard library, so if the use of ioutil/bytes looks wrong it probably is.

diff --git a/worker/maildir/message.go b/worker/maildir/message.go
index dbc9ade..5752694 100644
--- a/worker/maildir/message.go
+++ b/worker/maildir/message.go
@@ -1,9 +1,10 @@
 package maildir
 
 import (
-       "bufio"
+       "bytes"
        "fmt"
        "io"
+       "io/ioutil"
 
        "github.com/emersion/go-maildir"
        "github.com/emersion/go-message"
@@ -25,7 +26,12 @@ func (m Message) NewReader() (io.Reader, error) {
        if err != nil {
                return nil, err
        }
-       return bufio.NewReader(f), nil
+       defer f.Close()
+       contents, err := ioutil.ReadAll(f)
+       if err != nil {
+               return nil, err
+       }
+       return bytes.NewReader(contents), nil
 }
 
 // Flags fetches the set of flags currently applied to the message.

~sircmpwn 6 months ago

That seems like a red herring, and will just replace "out of files" with "out of RAM" soon enough. A better solution would be to rework this function to return an io.ReadCloser instead of an io.Reader.

~rjarry 6 months ago

I believe this code was modified for this exact reason in this patch: https://git.sr.ht/~rjarry/aerc/commit/8ecf0b73f41f9319f9d89d61f8d4619aa4e69758

I cannot find any examples of io.ReadCloser. I'll trying digging in that direction.

~rjarry REPORTED FIXED 6 months ago

Nguyễn Gia Phong referenced this ticket in commit 904ffac.

~rjarry 6 months ago

Nguyễn Gia Phong referenced this ticket in commit 904ffac.

~rjarry 6 months ago

Nguyễn Gia Phong referenced this ticket in commit 904ffac.

~rjarry 6 months ago

Nguyễn Gia Phong referenced this ticket in commit 904ffac.

Register here or Log in to comment, or comment via email.