~sircmpwn/aerc2#324: 
maildir: changing to a folder with lots of messages is very slow

My work email contains a lot of emails (GitHub notifications, Jira, etc…) so my archive folder contains something like ~200k messages.

When I use maildir & I try to switch to this folder. Sometimes my CPU reaches 100%, fans go full speed & I can't do anything other than killing the app similar to https://todo.sr.ht/~sircmpwn/aerc2/46 but with maildir

I updated to the latest release the CPU issue doesn't happen anymore, but it's very slow to switch & hangs the UI.

2019/11/22 15:12:13 Starting up aerc
2019/11/22 15:12:13 (ui)=> *types.Configure
2019/11/22 15:12:13 (ui)=> *types.Connect
2019/11/22 15:12:13 <-(ui) *types.Configure(2)
2019/11/22 15:12:13 configured base maildir: /Users/ahmed/.mail/Work
2019/11/22 15:12:13 ->(ui) *types.Done:*types.Configure
2019/11/22 15:12:13 <-(ui) *types.Connect(3)
2019/11/22 15:12:13 ->(ui) *types.Done:*types.Connect
2019/11/22 15:12:13 Starting Unix server
2019/11/22 15:12:13 Failed to start Unix server: listen unix /var/folders/x3/_cx1vn0173b5s54dhcwmrgv80000gn/T/aerc.sock: bind: address already in use (non-fatal)
2019/11/22 15:12:13 (ui)<= *types.Done(4):*types.Configure(2)
2019/11/22 15:12:13 (ui)<= *types.Done(5):*types.Connect(3)
2019/11/22 15:12:13 Listing mailboxes...
2019/11/22 15:12:13 (ui)=> *types.ListDirectories
2019/11/22 15:12:13 <-(ui) *types.ListDirectories(6)
2019/11/22 15:12:13 ->(ui) *types.Directory:*types.ListDirectories
2019/11/22 15:12:13 ->(ui) *types.Directory:*types.ListDirectories
2019/11/22 15:12:13 ->(ui) *types.Directory:*types.ListDirectories
2019/11/22 15:12:13 ->(ui) *types.Directory:*types.ListDirectories
2019/11/22 15:12:13 ->(ui) *types.Directory:*types.ListDirectories
2019/11/22 15:12:13 ->(ui) *types.Directory:*types.ListDirectories
2019/11/22 15:12:13 ->(ui) *types.Directory:*types.ListDirectories
2019/11/22 15:12:13 ->(ui) *types.Directory:*types.ListDirectories
2019/11/22 15:12:13 ->(ui) *types.Directory:*types.ListDirectories
2019/11/22 15:12:13 ->(ui) *types.Directory:*types.ListDirectories
2019/11/22 15:12:13 ->(ui) *types.Directory:*types.ListDirectories
2019/11/22 15:12:13 ->(ui) *types.Directory:*types.ListDirectories
2019/11/22 15:12:13 ->(ui) *types.Directory:*types.ListDirectories
2019/11/22 15:12:13 ->(ui) *types.Done:*types.ListDirectories
2019/11/22 15:12:13 (ui)<= *types.Directory(7):*types.ListDirectories(6)
2019/11/22 15:12:13 (ui)<= *types.Directory(8):*types.ListDirectories(6)
2019/11/22 15:12:13 (ui)<= *types.Directory(9):*types.ListDirectories(6)
2019/11/22 15:12:13 (ui)<= *types.Directory(10):*types.ListDirectories(6)
2019/11/22 15:12:13 (ui)<= *types.Directory(11):*types.ListDirectories(6)
2019/11/22 15:12:13 (ui)<= *types.Directory(12):*types.ListDirectories(6)
2019/11/22 15:12:13 (ui)<= *types.Directory(13):*types.ListDirectories(6)
2019/11/22 15:12:13 (ui)<= *types.Directory(14):*types.ListDirectories(6)
2019/11/22 15:12:13 (ui)<= *types.Directory(15):*types.ListDirectories(6)
2019/11/22 15:12:13 (ui)<= *types.Directory(16):*types.ListDirectories(6)
2019/11/22 15:12:13 (ui)<= *types.Directory(17):*types.ListDirectories(6)
2019/11/22 15:12:13 (ui)<= *types.Directory(18):*types.ListDirectories(6)
2019/11/22 15:12:13 (ui)<= *types.Directory(19):*types.ListDirectories(6)
2019/11/22 15:12:13 (ui)<= *types.Done(20):*types.ListDirectories(6)
2019/11/22 15:12:13 (ui)=> *types.OpenDirectory
2019/11/22 15:12:13 Connected.
2019/11/22 15:12:13 <-(ui) *types.OpenDirectory(21)
2019/11/22 15:12:13 opening Archive
2019/11/22 15:14:05 ->(ui) *types.DirectoryInfo
2019/11/22 15:14:05 ->(ui) *types.DirectoryInfo
2019/11/22 15:14:05 ->(ui) *types.Done:*types.OpenDirectory
2019/11/22 15:14:05 (ui)<= *types.DirectoryInfo(23)
2019/11/22 15:14:05 (ui)<= *types.DirectoryInfo(23)
2019/11/22 15:14:05 (ui)=> *types.FetchDirectoryContents
2019/11/22 15:14:05 (ui)<= *types.Done(24):*types.OpenDirectory(21)
2019/11/22 15:14:05 <-(ui) *types.FetchDirectoryContents(25)
2019/11/22 15:14:05 ->(ui) *types.DirectoryContents:*types.FetchDirectoryContents
2019/11/22 15:14:05 ->(ui) *types.Done:*types.FetchDirectoryContents
2019/11/22 15:14:05 (ui)<= *types.DirectoryContents(26):*types.FetchDirectoryContents(25)
2019/11/22 15:14:05 (ui)<= *types.Done(27):*types.FetchDirectoryContents(25)
2019/11/22 15:14:06 (ui)=> *types.FetchMessageHeaders
2019/11/22 15:14:06 <-(ui) *types.FetchMessageHeaders(28)
2019/11/22 15:14:06 ->(ui) *types.MessageInfo:*types.FetchMessageHeaders
2019/11/22 15:14:06 ->(ui) *types.MessageInfo:*types.FetchMessageHeaders
2019/11/22 15:14:06 ->(ui) *types.MessageInfo:*types.FetchMessageHeaders
2019/11/22 15:14:06 ->(ui) *types.MessageInfo:*types.FetchMessageHeaders
2019/11/22 15:14:06 ->(ui) *types.MessageInfo:*types.FetchMessageHeaders
2019/11/22 15:14:06 ->(ui) *types.MessageInfo:*types.FetchMessageHeaders
2019/11/22 15:14:06 ->(ui) *types.MessageInfo:*types.FetchMessageHeaders
2019/11/22 15:14:06 ->(ui) *types.MessageInfo:*types.FetchMessageHeaders
2019/11/22 15:14:06 ->(ui) *types.MessageInfo:*types.FetchMessageHeaders
2019/11/22 15:14:06 ->(ui) *types.MessageInfo:*types.FetchMessageHeaders
2019/11/22 15:14:06 ->(ui) *types.MessageInfo:*types.FetchMessageHeaders
2019/11/22 15:14:06 ->(ui) *types.MessageInfo:*types.FetchMessageHeaders
2019/11/22 15:14:06 ->(ui) *types.MessageInfo:*types.FetchMessageHeaders
2019/11/22 15:14:06 ->(ui) *types.MessageInfo:*types.FetchMessageHeaders
2019/11/22 15:14:06 ->(ui) *types.MessageInfo:*types.FetchMessageHeaders
2019/11/22 15:14:06 ->(ui) *types.MessageInfo:*types.FetchMessageHeaders
2019/11/22 15:14:06 ->(ui) *types.MessageInfo:*types.FetchMessageHeaders
2019/11/22 15:14:06 ->(ui) *types.MessageInfo:*types.FetchMessageHeaders
2019/11/22 15:14:06 ->(ui) *types.MessageInfo:*types.FetchMessageHeaders
2019/11/22 15:14:06 ->(ui) *types.MessageInfo:*types.FetchMessageHeaders
2019/11/22 15:14:06 ->(ui) *types.MessageInfo:*types.FetchMessageHeaders
2019/11/22 15:14:06 ->(ui) *types.MessageInfo:*types.FetchMessageHeaders
2019/11/22 15:14:06 ->(ui) *types.MessageInfo:*types.FetchMessageHeaders
2019/11/22 15:14:06 ->(ui) *types.MessageInfo:*types.FetchMessageHeaders
2019/11/22 15:14:06 ->(ui) *types.MessageInfo:*types.FetchMessageHeaders
2019/11/22 15:14:06 ->(ui) *types.MessageInfo:*types.FetchMessageHeaders
2019/11/22 15:14:06 ->(ui) *types.MessageInfo:*types.FetchMessageHeaders
2019/11/22 15:14:06 ->(ui) *types.MessageInfo:*types.FetchMessageHeaders
2019/11/22 15:14:06 ->(ui) *types.MessageInfo:*types.FetchMessageHeaders
2019/11/22 15:14:06 ->(ui) *types.MessageInfo:*types.FetchMessageHeaders
2019/11/22 15:14:06 ->(ui) *types.MessageInfo:*types.FetchMessageHeaders
2019/11/22 15:14:06 ->(ui) *types.MessageInfo:*types.FetchMessageHeaders
2019/11/22 15:14:06 (ui)<= *types.MessageInfo(29):*types.FetchMessageHeaders(28)
2019/11/22 15:14:06 (ui)<= *types.MessageInfo(30):*types.FetchMessageHeaders(28)
2019/11/22 15:14:06 (ui)<= *types.MessageInfo(31):*types.FetchMessageHeaders(28)
2019/11/22 15:14:06 (ui)<= *types.MessageInfo(32):*types.FetchMessageHeaders(28)
2019/11/22 15:14:06 (ui)<= *types.MessageInfo(33):*types.FetchMessageHeaders(28)
2019/11/22 15:14:06 (ui)<= *types.MessageInfo(34):*types.FetchMessageHeaders(28)
2019/11/22 15:14:06 (ui)<= *types.MessageInfo(35):*types.FetchMessageHeaders(28)
2019/11/22 15:14:06 (ui)<= *types.MessageInfo(36):*types.FetchMessageHeaders(28)
2019/11/22 15:14:06 (ui)<= *types.MessageInfo(37):*types.FetchMessageHeaders(28)
2019/11/22 15:14:06 (ui)<= *types.MessageInfo(38):*types.FetchMessageHeaders(28)
2019/11/22 15:14:06 (ui)<= *types.MessageInfo(39):*types.FetchMessageHeaders(28)
2019/11/22 15:14:06 (ui)<= *types.MessageInfo(40):*types.FetchMessageHeaders(28)
2019/11/22 15:14:06 (ui)<= *types.MessageInfo(41):*types.FetchMessageHeaders(28)
2019/11/22 15:14:06 (ui)<= *types.MessageInfo(42):*types.FetchMessageHeaders(28)
2019/11/22 15:14:06 (ui)<= *types.MessageInfo(43):*types.FetchMessageHeaders(28)
2019/11/22 15:14:06 (ui)<= *types.MessageInfo(44):*types.FetchMessageHeaders(28)
2019/11/22 15:14:06 (ui)<= *types.MessageInfo(45):*types.FetchMessageHeaders(28)
2019/11/22 15:14:06 (ui)<= *types.MessageInfo(46):*types.FetchMessageHeaders(28)
2019/11/22 15:14:06 (ui)<= *types.MessageInfo(47):*types.FetchMessageHeaders(28)
2019/11/22 15:14:06 (ui)<= *types.MessageInfo(48):*types.FetchMessageHeaders(28)
2019/11/22 15:14:06 (ui)<= *types.MessageInfo(49):*types.FetchMessageHeaders(28)
2019/11/22 15:14:06 (ui)<= *types.MessageInfo(50):*types.FetchMessageHeaders(28)
2019/11/22 15:14:06 (ui)<= *types.MessageInfo(51):*types.FetchMessageHeaders(28)
2019/11/22 15:14:06 (ui)<= *types.MessageInfo(52):*types.FetchMessageHeaders(28)
2019/11/22 15:14:06 (ui)<= *types.MessageInfo(53):*types.FetchMessageHeaders(28)
2019/11/22 15:14:06 (ui)<= *types.MessageInfo(54):*types.FetchMessageHeaders(28)
2019/11/22 15:14:06 (ui)<= *types.MessageInfo(55):*types.FetchMessageHeaders(28)
2019/11/22 15:14:06 (ui)<= *types.MessageInfo(56):*types.FetchMessageHeaders(28)
2019/11/22 15:14:06 (ui)<= *types.MessageInfo(57):*types.FetchMessageHeaders(28)
2019/11/22 15:14:06 (ui)<= *types.MessageInfo(58):*types.FetchMessageHeaders(28)
2019/11/22 15:14:06 (ui)<= *types.MessageInfo(59):*types.FetchMessageHeaders(28)
2019/11/22 15:14:06 (ui)<= *types.MessageInfo(60):*types.FetchMessageHeaders(28)
2019/11/22 15:14:06 ->(ui) *types.MessageInfo:*types.FetchMessageHeaders
2019/11/22 15:14:06 ->(ui) *types.MessageInfo:*types.FetchMessageHeaders
2019/11/22 15:14:06 ->(ui) *types.MessageInfo:*types.FetchMessageHeaders
2019/11/22 15:14:06 ->(ui) *types.MessageInfo:*types.FetchMessageHeaders
2019/11/22 15:14:06 ->(ui) *types.MessageInfo:*types.FetchMessageHeaders
2019/11/22 15:14:06 ->(ui) *types.MessageInfo:*types.FetchMessageHeaders
2019/11/22 15:14:06 ->(ui) *types.MessageInfo:*types.FetchMessageHeaders
2019/11/22 15:14:06 ->(ui) *types.MessageInfo:*types.FetchMessageHeaders
2019/11/22 15:14:06 ->(ui) *types.MessageInfo:*types.FetchMessageHeaders
2019/11/22 15:14:06 ->(ui) *types.MessageInfo:*types.FetchMessageHeaders
2019/11/22 15:14:06 ->(ui) *types.MessageInfo:*types.FetchMessageHeaders
2019/11/22 15:14:06 ->(ui) *types.MessageInfo:*types.FetchMessageHeaders
2019/11/22 15:14:06 ->(ui) *types.MessageInfo:*types.FetchMessageHeaders
2019/11/22 15:14:06 ->(ui) *types.MessageInfo:*types.FetchMessageHeaders
2019/11/22 15:14:06 ->(ui) *types.MessageInfo:*types.FetchMessageHeaders
2019/11/22 15:14:06 ->(ui) *types.MessageInfo:*types.FetchMessageHeaders
2019/11/22 15:14:06 (ui)<= *types.MessageInfo(61):*types.FetchMessageHeaders(28)
2019/11/22 15:14:06 (ui)<= *types.MessageInfo(62):*types.FetchMessageHeaders(28)
2019/11/22 15:14:06 (ui)<= *types.MessageInfo(63):*types.FetchMessageHeaders(28)
2019/11/22 15:14:06 (ui)<= *types.MessageInfo(64):*types.FetchMessageHeaders(28)
2019/11/22 15:14:06 (ui)<= *types.MessageInfo(65):*types.FetchMessageHeaders(28)
2019/11/22 15:14:06 (ui)<= *types.MessageInfo(66):*types.FetchMessageHeaders(28)
2019/11/22 15:14:06 (ui)<= *types.MessageInfo(67):*types.FetchMessageHeaders(28)
2019/11/22 15:14:06 (ui)<= *types.MessageInfo(68):*types.FetchMessageHeaders(28)
2019/11/22 15:14:06 (ui)<= *types.MessageInfo(69):*types.FetchMessageHeaders(28)
2019/11/22 15:14:06 (ui)<= *types.MessageInfo(70):*types.FetchMessageHeaders(28)
2019/11/22 15:14:06 (ui)<= *types.MessageInfo(71):*types.FetchMessageHeaders(28)
2019/11/22 15:14:06 (ui)<= *types.MessageInfo(72):*types.FetchMessageHeaders(28)
2019/11/22 15:14:06 (ui)<= *types.MessageInfo(73):*types.FetchMessageHeaders(28)
2019/11/22 15:14:06 (ui)<= *types.MessageInfo(74):*types.FetchMessageHeaders(28)
2019/11/22 15:14:06 (ui)<= *types.MessageInfo(75):*types.FetchMessageHeaders(28)
2019/11/22 15:14:06 (ui)<= *types.MessageInfo(76):*types.FetchMessageHeaders(28)
2019/11/22 15:14:06 ->(ui) *types.MessageInfo:*types.FetchMessageHeaders
2019/11/22 15:14:06 ->(ui) *types.MessageInfo:*types.FetchMessageHeaders
2019/11/22 15:14:06 ->(ui) *types.MessageInfo:*types.FetchMessageHeaders
2019/11/22 15:14:06 ->(ui) *types.MessageInfo:*types.FetchMessageHeaders
2019/11/22 15:14:06 ->(ui) *types.MessageInfo:*types.FetchMessageHeaders
2019/11/22 15:14:06 ->(ui) *types.MessageInfo:*types.FetchMessageHeaders
2019/11/22 15:14:06 ->(ui) *types.MessageInfo:*types.FetchMessageHeaders
2019/11/22 15:14:06 ->(ui) *types.MessageInfo:*types.FetchMessageHeaders
2019/11/22 15:14:06 ->(ui) *types.MessageInfo:*types.FetchMessageHeaders
2019/11/22 15:14:06 ->(ui) *types.MessageInfo:*types.FetchMessageHeaders
2019/11/22 15:14:06 ->(ui) *types.MessageInfo:*types.FetchMessageHeaders
2019/11/22 15:14:06 ->(ui) *types.MessageInfo:*types.FetchMessageHeaders
2019/11/22 15:14:06 ->(ui) *types.MessageInfo:*types.FetchMessageHeaders
2019/11/22 15:14:06 ->(ui) *types.MessageInfo:*types.FetchMessageHeaders
2019/11/22 15:14:06 ->(ui) *types.MessageInfo:*types.FetchMessageHeaders
2019/11/22 15:14:06 ->(ui) *types.MessageInfo:*types.FetchMessageHeaders
2019/11/22 15:14:06 ->(ui) *types.MessageInfo:*types.FetchMessageHeaders
2019/11/22 15:14:06 (ui)<= *types.MessageInfo(77):*types.FetchMessageHeaders(28)
2019/11/22 15:14:06 (ui)<= *types.MessageInfo(78):*types.FetchMessageHeaders(28)
2019/11/22 15:14:06 (ui)<= *types.MessageInfo(79):*types.FetchMessageHeaders(28)
2019/11/22 15:14:06 (ui)<= *types.MessageInfo(80):*types.FetchMessageHeaders(28)
2019/11/22 15:14:06 (ui)<= *types.MessageInfo(81):*types.FetchMessageHeaders(28)
2019/11/22 15:14:06 (ui)<= *types.MessageInfo(82):*types.FetchMessageHeaders(28)
2019/11/22 15:14:06 (ui)<= *types.MessageInfo(83):*types.FetchMessageHeaders(28)
2019/11/22 15:14:06 (ui)<= *types.MessageInfo(84):*types.FetchMessageHeaders(28)
2019/11/22 15:14:06 (ui)<= *types.MessageInfo(85):*types.FetchMessageHeaders(28)
2019/11/22 15:14:06 (ui)<= *types.MessageInfo(86):*types.FetchMessageHeaders(28)
2019/11/22 15:14:06 (ui)<= *types.MessageInfo(87):*types.FetchMessageHeaders(28)
2019/11/22 15:14:06 (ui)<= *types.MessageInfo(88):*types.FetchMessageHeaders(28)
2019/11/22 15:14:06 (ui)<= *types.MessageInfo(89):*types.FetchMessageHeaders(28)
2019/11/22 15:14:06 (ui)<= *types.MessageInfo(90):*types.FetchMessageHeaders(28)
2019/11/22 15:14:06 (ui)<= *types.MessageInfo(91):*types.FetchMessageHeaders(28)
2019/11/22 15:14:06 ->(ui) *types.MessageInfo:*types.FetchMessageHeaders
2019/11/22 15:14:06 (ui)<= *types.MessageInfo(92):*types.FetchMessageHeaders(28)
2019/11/22 15:14:06 (ui)<= *types.MessageInfo(93):*types.FetchMessageHeaders(28)
2019/11/22 15:14:06 (ui)<= *types.MessageInfo(94):*types.FetchMessageHeaders(28)
2019/11/22 15:14:06 ->(ui) *types.MessageInfo:*types.FetchMessageHeaders
2019/11/22 15:14:06 ->(ui) *types.MessageInfo:*types.FetchMessageHeaders
2019/11/22 15:14:06 ->(ui) *types.Done:*types.FetchMessageHeaders
2019/11/22 15:14:06 (ui)<= *types.MessageInfo(95):*types.FetchMessageHeaders(28)
2019/11/22 15:14:06 (ui)<= *types.MessageInfo(96):*types.FetchMessageHeaders(28)
2019/11/22 15:14:06 (ui)<= *types.Done(97):*types.FetchMessageHeaders(28)
Status
REPORTED
Submitter
~ahmedelgabri
Assigned to
No-one
Submitted
1 year, 10 months ago
Updated
a month ago
Labels
maildir

~labrat 1 year, 10 months ago

You really should set up notmuch and limit by time.... there's no way you'd ever want to look at that many messages at the same time

~ahmedelgabri 1 year, 10 months ago

~labrat how can I do that? where to read about it?

~labrat 1 year, 10 months ago

To be fair, adapting a notmuch based workflow means you'll have to change your habits quite a bit.

But as far as aerc is concerned man aerc-notmuch The rest, google notmuch

Your issue is that in order for things like sorting to work aerc parses all mails in the folder and this, well, takes a while for 200k mails.

You should see a similar effect in say neomutt

~ahmedelgabri 1 year, 10 months ago

Ok, thanks will have a look

~mmf 1 year, 10 months ago

Regarding this, I have to ideas that might help: add an option to hide some folders and/or add an option to choose the sort order of the folders. I'd really like to see custom sorting and could probably implement it myself, but first I need to have some free time to look into how workers are implemented and what are the conventions between them.

~labrat 1 year, 10 months ago

~mmf you can already hide folders, although currently it's only a whitelist. We should extend this in a way that also has negation (say "!archive").

The dirlist is worker agnostic so there's no need to go deeply into that.

~mmf 1 year, 9 months ago

Just in case anyone is interested, I've submitted a patch for custom sorting of folders: https://lists.sr.ht/~sircmpwn/aerc/patches/9229

~ninjin 1 year, 14 days ago*

To add a word in favour of notmuch over plain maildir in a situation like this, along with some numbers. My sent directory contains ~35,000 e-mails, it loads in ~1s with Mutt, ~60s with aerc-maildir, and instantly with aerc-notmuch. Most likely Mutt accomplishes this with extensive caching, but even on a high-end SSD it will become very unpleasant when you approach a million e-mails and notmuch (or a similar) integration becomes essential. For example, opening my archive with ~500,000 e-mails takes a good few minutes in Mutt, so the only way to interface with it is really using notmuch in some shape or form.

Edit: Spelling.

~kim a month ago

I am using aerc with notmuch, and am seeing similar behaviour when using :cf tag:mytag (ie. do a custom notmuch search as a "folder"). The query returns almost immediately when using notmuch directly, but it takes almost a minute until the result shows up in aerc.

I am not sure how to instrument aerc in order to be able to submit more useful info, any pointers would be appreciated.

Register here or Log in to comment, or comment via email.