~sircmpwn/aerc2#324: maildir: changing to a folder with lots of messages is very slow

My work email contains a lot of emails (GitHub notifications, Jira, etc…) so my archive folder contains something like ~200k messages.

When I use maildir & I try to switch to this folder. Sometimes my CPU reaches 100%, fans go full speed & I can't do anything other than killing the app similar to https://todo.sr.ht/~sircmpwn/aerc2/46 but with maildir

I updated to the latest release the CPU issue doesn't happen anymore, but it's very slow to switch & hangs the UI.

2019/11/22 15:12:13 Starting up aerc
2019/11/22 15:12:13 (ui)=> *types.Configure
2019/11/22 15:12:13 (ui)=> *types.Connect
2019/11/22 15:12:13 <-(ui) *types.Configure(2)
2019/11/22 15:12:13 configured base maildir: /Users/ahmed/.mail/Work
2019/11/22 15:12:13 ->(ui) *types.Done:*types.Configure
2019/11/22 15:12:13 <-(ui) *types.Connect(3)
2019/11/22 15:12:13 ->(ui) *types.Done:*types.Connect
2019/11/22 15:12:13 Starting Unix server
2019/11/22 15:12:13 Failed to start Unix server: listen unix /var/folders/x3/_cx1vn0173b5s54dhcwmrgv80000gn/T/aerc.sock: bind: address already in use (non-fatal)
2019/11/22 15:12:13 (ui)<= *types.Done(4):*types.Configure(2)
2019/11/22 15:12:13 (ui)<= *types.Done(5):*types.Connect(3)
2019/11/22 15:12:13 Listing mailboxes...
2019/11/22 15:12:13 (ui)=> *types.ListDirectories
2019/11/22 15:12:13 <-(ui) *types.ListDirectories(6)
2019/11/22 15:12:13 ->(ui) *types.Directory:*types.ListDirectories
2019/11/22 15:12:13 ->(ui) *types.Directory:*types.ListDirectories
2019/11/22 15:12:13 ->(ui) *types.Directory:*types.ListDirectories
2019/11/22 15:12:13 ->(ui) *types.Directory:*types.ListDirectories
2019/11/22 15:12:13 ->(ui) *types.Directory:*types.ListDirectories
2019/11/22 15:12:13 ->(ui) *types.Directory:*types.ListDirectories
2019/11/22 15:12:13 ->(ui) *types.Directory:*types.ListDirectories
2019/11/22 15:12:13 ->(ui) *types.Directory:*types.ListDirectories
2019/11/22 15:12:13 ->(ui) *types.Directory:*types.ListDirectories
2019/11/22 15:12:13 ->(ui) *types.Directory:*types.ListDirectories
2019/11/22 15:12:13 ->(ui) *types.Directory:*types.ListDirectories
2019/11/22 15:12:13 ->(ui) *types.Directory:*types.ListDirectories
2019/11/22 15:12:13 ->(ui) *types.Directory:*types.ListDirectories
2019/11/22 15:12:13 ->(ui) *types.Done:*types.ListDirectories
2019/11/22 15:12:13 (ui)<= *types.Directory(7):*types.ListDirectories(6)
2019/11/22 15:12:13 (ui)<= *types.Directory(8):*types.ListDirectories(6)
2019/11/22 15:12:13 (ui)<= *types.Directory(9):*types.ListDirectories(6)
2019/11/22 15:12:13 (ui)<= *types.Directory(10):*types.ListDirectories(6)
2019/11/22 15:12:13 (ui)<= *types.Directory(11):*types.ListDirectories(6)
2019/11/22 15:12:13 (ui)<= *types.Directory(12):*types.ListDirectories(6)
2019/11/22 15:12:13 (ui)<= *types.Directory(13):*types.ListDirectories(6)
2019/11/22 15:12:13 (ui)<= *types.Directory(14):*types.ListDirectories(6)
2019/11/22 15:12:13 (ui)<= *types.Directory(15):*types.ListDirectories(6)
2019/11/22 15:12:13 (ui)<= *types.Directory(16):*types.ListDirectories(6)
2019/11/22 15:12:13 (ui)<= *types.Directory(17):*types.ListDirectories(6)
2019/11/22 15:12:13 (ui)<= *types.Directory(18):*types.ListDirectories(6)
2019/11/22 15:12:13 (ui)<= *types.Directory(19):*types.ListDirectories(6)
2019/11/22 15:12:13 (ui)<= *types.Done(20):*types.ListDirectories(6)
2019/11/22 15:12:13 (ui)=> *types.OpenDirectory
2019/11/22 15:12:13 Connected.
2019/11/22 15:12:13 <-(ui) *types.OpenDirectory(21)
2019/11/22 15:12:13 opening Archive
2019/11/22 15:14:05 ->(ui) *types.DirectoryInfo
2019/11/22 15:14:05 ->(ui) *types.DirectoryInfo
2019/11/22 15:14:05 ->(ui) *types.Done:*types.OpenDirectory
2019/11/22 15:14:05 (ui)<= *types.DirectoryInfo(23)
2019/11/22 15:14:05 (ui)<= *types.DirectoryInfo(23)
2019/11/22 15:14:05 (ui)=> *types.FetchDirectoryContents
2019/11/22 15:14:05 (ui)<= *types.Done(24):*types.OpenDirectory(21)
2019/11/22 15:14:05 <-(ui) *types.FetchDirectoryContents(25)
2019/11/22 15:14:05 ->(ui) *types.DirectoryContents:*types.FetchDirectoryContents
2019/11/22 15:14:05 ->(ui) *types.Done:*types.FetchDirectoryContents
2019/11/22 15:14:05 (ui)<= *types.DirectoryContents(26):*types.FetchDirectoryContents(25)
2019/11/22 15:14:05 (ui)<= *types.Done(27):*types.FetchDirectoryContents(25)
2019/11/22 15:14:06 (ui)=> *types.FetchMessageHeaders
2019/11/22 15:14:06 <-(ui) *types.FetchMessageHeaders(28)
2019/11/22 15:14:06 ->(ui) *types.MessageInfo:*types.FetchMessageHeaders
2019/11/22 15:14:06 ->(ui) *types.MessageInfo:*types.FetchMessageHeaders
2019/11/22 15:14:06 ->(ui) *types.MessageInfo:*types.FetchMessageHeaders
2019/11/22 15:14:06 ->(ui) *types.MessageInfo:*types.FetchMessageHeaders
2019/11/22 15:14:06 ->(ui) *types.MessageInfo:*types.FetchMessageHeaders
2019/11/22 15:14:06 ->(ui) *types.MessageInfo:*types.FetchMessageHeaders
2019/11/22 15:14:06 ->(ui) *types.MessageInfo:*types.FetchMessageHeaders
2019/11/22 15:14:06 ->(ui) *types.MessageInfo:*types.FetchMessageHeaders
2019/11/22 15:14:06 ->(ui) *types.MessageInfo:*types.FetchMessageHeaders
2019/11/22 15:14:06 ->(ui) *types.MessageInfo:*types.FetchMessageHeaders
2019/11/22 15:14:06 ->(ui) *types.MessageInfo:*types.FetchMessageHeaders
2019/11/22 15:14:06 ->(ui) *types.MessageInfo:*types.FetchMessageHeaders
2019/11/22 15:14:06 ->(ui) *types.MessageInfo:*types.FetchMessageHeaders
2019/11/22 15:14:06 ->(ui) *types.MessageInfo:*types.FetchMessageHeaders
2019/11/22 15:14:06 ->(ui) *types.MessageInfo:*types.FetchMessageHeaders
2019/11/22 15:14:06 ->(ui) *types.MessageInfo:*types.FetchMessageHeaders
2019/11/22 15:14:06 ->(ui) *types.MessageInfo:*types.FetchMessageHeaders
2019/11/22 15:14:06 ->(ui) *types.MessageInfo:*types.FetchMessageHeaders
2019/11/22 15:14:06 ->(ui) *types.MessageInfo:*types.FetchMessageHeaders
2019/11/22 15:14:06 ->(ui) *types.MessageInfo:*types.FetchMessageHeaders
2019/11/22 15:14:06 ->(ui) *types.MessageInfo:*types.FetchMessageHeaders
2019/11/22 15:14:06 ->(ui) *types.MessageInfo:*types.FetchMessageHeaders
2019/11/22 15:14:06 ->(ui) *types.MessageInfo:*types.FetchMessageHeaders
2019/11/22 15:14:06 ->(ui) *types.MessageInfo:*types.FetchMessageHeaders
2019/11/22 15:14:06 ->(ui) *types.MessageInfo:*types.FetchMessageHeaders
2019/11/22 15:14:06 ->(ui) *types.MessageInfo:*types.FetchMessageHeaders
2019/11/22 15:14:06 ->(ui) *types.MessageInfo:*types.FetchMessageHeaders
2019/11/22 15:14:06 ->(ui) *types.MessageInfo:*types.FetchMessageHeaders
2019/11/22 15:14:06 ->(ui) *types.MessageInfo:*types.FetchMessageHeaders
2019/11/22 15:14:06 ->(ui) *types.MessageInfo:*types.FetchMessageHeaders
2019/11/22 15:14:06 ->(ui) *types.MessageInfo:*types.FetchMessageHeaders
2019/11/22 15:14:06 ->(ui) *types.MessageInfo:*types.FetchMessageHeaders
2019/11/22 15:14:06 (ui)<= *types.MessageInfo(29):*types.FetchMessageHeaders(28)
2019/11/22 15:14:06 (ui)<= *types.MessageInfo(30):*types.FetchMessageHeaders(28)
2019/11/22 15:14:06 (ui)<= *types.MessageInfo(31):*types.FetchMessageHeaders(28)
2019/11/22 15:14:06 (ui)<= *types.MessageInfo(32):*types.FetchMessageHeaders(28)
2019/11/22 15:14:06 (ui)<= *types.MessageInfo(33):*types.FetchMessageHeaders(28)
2019/11/22 15:14:06 (ui)<= *types.MessageInfo(34):*types.FetchMessageHeaders(28)
2019/11/22 15:14:06 (ui)<= *types.MessageInfo(35):*types.FetchMessageHeaders(28)
2019/11/22 15:14:06 (ui)<= *types.MessageInfo(36):*types.FetchMessageHeaders(28)
2019/11/22 15:14:06 (ui)<= *types.MessageInfo(37):*types.FetchMessageHeaders(28)
2019/11/22 15:14:06 (ui)<= *types.MessageInfo(38):*types.FetchMessageHeaders(28)
2019/11/22 15:14:06 (ui)<= *types.MessageInfo(39):*types.FetchMessageHeaders(28)
2019/11/22 15:14:06 (ui)<= *types.MessageInfo(40):*types.FetchMessageHeaders(28)
2019/11/22 15:14:06 (ui)<= *types.MessageInfo(41):*types.FetchMessageHeaders(28)
2019/11/22 15:14:06 (ui)<= *types.MessageInfo(42):*types.FetchMessageHeaders(28)
2019/11/22 15:14:06 (ui)<= *types.MessageInfo(43):*types.FetchMessageHeaders(28)
2019/11/22 15:14:06 (ui)<= *types.MessageInfo(44):*types.FetchMessageHeaders(28)
2019/11/22 15:14:06 (ui)<= *types.MessageInfo(45):*types.FetchMessageHeaders(28)
2019/11/22 15:14:06 (ui)<= *types.MessageInfo(46):*types.FetchMessageHeaders(28)
2019/11/22 15:14:06 (ui)<= *types.MessageInfo(47):*types.FetchMessageHeaders(28)
2019/11/22 15:14:06 (ui)<= *types.MessageInfo(48):*types.FetchMessageHeaders(28)
2019/11/22 15:14:06 (ui)<= *types.MessageInfo(49):*types.FetchMessageHeaders(28)
2019/11/22 15:14:06 (ui)<= *types.MessageInfo(50):*types.FetchMessageHeaders(28)
2019/11/22 15:14:06 (ui)<= *types.MessageInfo(51):*types.FetchMessageHeaders(28)
2019/11/22 15:14:06 (ui)<= *types.MessageInfo(52):*types.FetchMessageHeaders(28)
2019/11/22 15:14:06 (ui)<= *types.MessageInfo(53):*types.FetchMessageHeaders(28)
2019/11/22 15:14:06 (ui)<= *types.MessageInfo(54):*types.FetchMessageHeaders(28)
2019/11/22 15:14:06 (ui)<= *types.MessageInfo(55):*types.FetchMessageHeaders(28)
2019/11/22 15:14:06 (ui)<= *types.MessageInfo(56):*types.FetchMessageHeaders(28)
2019/11/22 15:14:06 (ui)<= *types.MessageInfo(57):*types.FetchMessageHeaders(28)
2019/11/22 15:14:06 (ui)<= *types.MessageInfo(58):*types.FetchMessageHeaders(28)
2019/11/22 15:14:06 (ui)<= *types.MessageInfo(59):*types.FetchMessageHeaders(28)
2019/11/22 15:14:06 (ui)<= *types.MessageInfo(60):*types.FetchMessageHeaders(28)
2019/11/22 15:14:06 ->(ui) *types.MessageInfo:*types.FetchMessageHeaders
2019/11/22 15:14:06 ->(ui) *types.MessageInfo:*types.FetchMessageHeaders
2019/11/22 15:14:06 ->(ui) *types.MessageInfo:*types.FetchMessageHeaders
2019/11/22 15:14:06 ->(ui) *types.MessageInfo:*types.FetchMessageHeaders
2019/11/22 15:14:06 ->(ui) *types.MessageInfo:*types.FetchMessageHeaders
2019/11/22 15:14:06 ->(ui) *types.MessageInfo:*types.FetchMessageHeaders
2019/11/22 15:14:06 ->(ui) *types.MessageInfo:*types.FetchMessageHeaders
2019/11/22 15:14:06 ->(ui) *types.MessageInfo:*types.FetchMessageHeaders
2019/11/22 15:14:06 ->(ui) *types.MessageInfo:*types.FetchMessageHeaders
2019/11/22 15:14:06 ->(ui) *types.MessageInfo:*types.FetchMessageHeaders
2019/11/22 15:14:06 ->(ui) *types.MessageInfo:*types.FetchMessageHeaders
2019/11/22 15:14:06 ->(ui) *types.MessageInfo:*types.FetchMessageHeaders
2019/11/22 15:14:06 ->(ui) *types.MessageInfo:*types.FetchMessageHeaders
2019/11/22 15:14:06 ->(ui) *types.MessageInfo:*types.FetchMessageHeaders
2019/11/22 15:14:06 ->(ui) *types.MessageInfo:*types.FetchMessageHeaders
2019/11/22 15:14:06 ->(ui) *types.MessageInfo:*types.FetchMessageHeaders
2019/11/22 15:14:06 (ui)<= *types.MessageInfo(61):*types.FetchMessageHeaders(28)
2019/11/22 15:14:06 (ui)<= *types.MessageInfo(62):*types.FetchMessageHeaders(28)
2019/11/22 15:14:06 (ui)<= *types.MessageInfo(63):*types.FetchMessageHeaders(28)
2019/11/22 15:14:06 (ui)<= *types.MessageInfo(64):*types.FetchMessageHeaders(28)
2019/11/22 15:14:06 (ui)<= *types.MessageInfo(65):*types.FetchMessageHeaders(28)
2019/11/22 15:14:06 (ui)<= *types.MessageInfo(66):*types.FetchMessageHeaders(28)
2019/11/22 15:14:06 (ui)<= *types.MessageInfo(67):*types.FetchMessageHeaders(28)
2019/11/22 15:14:06 (ui)<= *types.MessageInfo(68):*types.FetchMessageHeaders(28)
2019/11/22 15:14:06 (ui)<= *types.MessageInfo(69):*types.FetchMessageHeaders(28)
2019/11/22 15:14:06 (ui)<= *types.MessageInfo(70):*types.FetchMessageHeaders(28)
2019/11/22 15:14:06 (ui)<= *types.MessageInfo(71):*types.FetchMessageHeaders(28)
2019/11/22 15:14:06 (ui)<= *types.MessageInfo(72):*types.FetchMessageHeaders(28)
2019/11/22 15:14:06 (ui)<= *types.MessageInfo(73):*types.FetchMessageHeaders(28)
2019/11/22 15:14:06 (ui)<= *types.MessageInfo(74):*types.FetchMessageHeaders(28)
2019/11/22 15:14:06 (ui)<= *types.MessageInfo(75):*types.FetchMessageHeaders(28)
2019/11/22 15:14:06 (ui)<= *types.MessageInfo(76):*types.FetchMessageHeaders(28)
2019/11/22 15:14:06 ->(ui) *types.MessageInfo:*types.FetchMessageHeaders
2019/11/22 15:14:06 ->(ui) *types.MessageInfo:*types.FetchMessageHeaders
2019/11/22 15:14:06 ->(ui) *types.MessageInfo:*types.FetchMessageHeaders
2019/11/22 15:14:06 ->(ui) *types.MessageInfo:*types.FetchMessageHeaders
2019/11/22 15:14:06 ->(ui) *types.MessageInfo:*types.FetchMessageHeaders
2019/11/22 15:14:06 ->(ui) *types.MessageInfo:*types.FetchMessageHeaders
2019/11/22 15:14:06 ->(ui) *types.MessageInfo:*types.FetchMessageHeaders
2019/11/22 15:14:06 ->(ui) *types.MessageInfo:*types.FetchMessageHeaders
2019/11/22 15:14:06 ->(ui) *types.MessageInfo:*types.FetchMessageHeaders
2019/11/22 15:14:06 ->(ui) *types.MessageInfo:*types.FetchMessageHeaders
2019/11/22 15:14:06 ->(ui) *types.MessageInfo:*types.FetchMessageHeaders
2019/11/22 15:14:06 ->(ui) *types.MessageInfo:*types.FetchMessageHeaders
2019/11/22 15:14:06 ->(ui) *types.MessageInfo:*types.FetchMessageHeaders
2019/11/22 15:14:06 ->(ui) *types.MessageInfo:*types.FetchMessageHeaders
2019/11/22 15:14:06 ->(ui) *types.MessageInfo:*types.FetchMessageHeaders
2019/11/22 15:14:06 ->(ui) *types.MessageInfo:*types.FetchMessageHeaders
2019/11/22 15:14:06 ->(ui) *types.MessageInfo:*types.FetchMessageHeaders
2019/11/22 15:14:06 (ui)<= *types.MessageInfo(77):*types.FetchMessageHeaders(28)
2019/11/22 15:14:06 (ui)<= *types.MessageInfo(78):*types.FetchMessageHeaders(28)
2019/11/22 15:14:06 (ui)<= *types.MessageInfo(79):*types.FetchMessageHeaders(28)
2019/11/22 15:14:06 (ui)<= *types.MessageInfo(80):*types.FetchMessageHeaders(28)
2019/11/22 15:14:06 (ui)<= *types.MessageInfo(81):*types.FetchMessageHeaders(28)
2019/11/22 15:14:06 (ui)<= *types.MessageInfo(82):*types.FetchMessageHeaders(28)
2019/11/22 15:14:06 (ui)<= *types.MessageInfo(83):*types.FetchMessageHeaders(28)
2019/11/22 15:14:06 (ui)<= *types.MessageInfo(84):*types.FetchMessageHeaders(28)
2019/11/22 15:14:06 (ui)<= *types.MessageInfo(85):*types.FetchMessageHeaders(28)
2019/11/22 15:14:06 (ui)<= *types.MessageInfo(86):*types.FetchMessageHeaders(28)
2019/11/22 15:14:06 (ui)<= *types.MessageInfo(87):*types.FetchMessageHeaders(28)
2019/11/22 15:14:06 (ui)<= *types.MessageInfo(88):*types.FetchMessageHeaders(28)
2019/11/22 15:14:06 (ui)<= *types.MessageInfo(89):*types.FetchMessageHeaders(28)
2019/11/22 15:14:06 (ui)<= *types.MessageInfo(90):*types.FetchMessageHeaders(28)
2019/11/22 15:14:06 (ui)<= *types.MessageInfo(91):*types.FetchMessageHeaders(28)
2019/11/22 15:14:06 ->(ui) *types.MessageInfo:*types.FetchMessageHeaders
2019/11/22 15:14:06 (ui)<= *types.MessageInfo(92):*types.FetchMessageHeaders(28)
2019/11/22 15:14:06 (ui)<= *types.MessageInfo(93):*types.FetchMessageHeaders(28)
2019/11/22 15:14:06 (ui)<= *types.MessageInfo(94):*types.FetchMessageHeaders(28)
2019/11/22 15:14:06 ->(ui) *types.MessageInfo:*types.FetchMessageHeaders
2019/11/22 15:14:06 ->(ui) *types.MessageInfo:*types.FetchMessageHeaders
2019/11/22 15:14:06 ->(ui) *types.Done:*types.FetchMessageHeaders
2019/11/22 15:14:06 (ui)<= *types.MessageInfo(95):*types.FetchMessageHeaders(28)
2019/11/22 15:14:06 (ui)<= *types.MessageInfo(96):*types.FetchMessageHeaders(28)
2019/11/22 15:14:06 (ui)<= *types.Done(97):*types.FetchMessageHeaders(28)

Status: REPORTED
Submitter: ~ahmedelgabri
Assigned to: No-one
Submitted: 3 years ago
Updated: 2 years ago
Labels: maildir

~ahmedelgabri added maildir 3 years ago

~labrat 3 years ago

You really should set up notmuch and limit by time.... there's no way you'd ever want to look at that many messages at the same time

~ahmedelgabri 3 years ago

~labrat how can I do that? where to read about it?

~labrat 3 years ago

To be fair, adapting a notmuch based workflow means you'll have to change your habits quite a bit.

But as far as aerc is concerned man aerc-notmuch The rest, google notmuch

Your issue is that in order for things like sorting to work aerc parses all mails in the folder and this, well, takes a while for 200k mails.

You should see a similar effect in say neomutt

~ahmedelgabri 3 years ago

Ok, thanks will have a look

~mmf 3 years ago

Regarding this, I have to ideas that might help: add an option to hide some folders and/or add an option to choose the sort order of the folders. I'd really like to see custom sorting and could probably implement it myself, but first I need to have some free time to look into how workers are implemented and what are the conventions between them.

~labrat 3 years ago

~mmf you can already hide folders, although currently it's only a whitelist. We should extend this in a way that also has negation (say "!archive").

The dirlist is worker agnostic so there's no need to go deeply into that.

~mmf 3 years ago

Just in case anyone is interested, I've submitted a patch for custom sorting of folders: https://lists.sr.ht/~sircmpwn/aerc/patches/9229

~ninjin 3 years ago*

To add a word in favour of notmuch over plain maildir in a situation like this, along with some numbers. My sent directory contains ~35,000 e-mails, it loads in ~1s with Mutt, ~60s with aerc-maildir, and instantly with aerc-notmuch. Most likely Mutt accomplishes this with extensive caching, but even on a high-end SSD it will become very unpleasant when you approach a million e-mails and notmuch (or a similar) integration becomes essential. For example, opening my archive with ~500,000 e-mails takes a good few minutes in Mutt, so the only way to interface with it is really using notmuch in some shape or form.

Edit: Spelling.

~kim 2 years ago

I am using aerc with notmuch, and am seeing similar behaviour when using :cf tag:mytag (ie. do a custom notmuch search as a "folder"). The query returns almost immediately when using notmuch directly, but it takes almost a minute until the result shows up in aerc.

I am not sure how to instrument aerc in order to be able to submit more useful info, any pointers would be appreciated.