~soywod/pimalaya#88: 
Accidental INBOX folder deletion

#Description

If the Inbox folder is not present (or present under another name (ussually INBOX), running himalaya account sync will try to create the folder Inbox.

The naive fix deletes the Inbox folder:

Changing the config so that under folder aliases inbox = "INBOX" and running himalaya account sync deletes both Inbox and INBOX locally and remotely (if using an IMAP server).

#Issue

I would argue that this is most likely never what the user wants, and when running any command that is not delete or folder delete mails and folders should never be deleted without prompting the user.

#Proposed solution

Prompt the user before deleting folders (like when running folder delete) and hide the option to delete remote folders under a config option/flag that defaults to off (that way any local mistakes don't actually delete data).

Maybe also be a bit smarter when updating aliases and see if the folder already exists and keep it without deleting it.

#Steps to reproduce

  1. Setup a mail account with an Inbox folder called INBOX
  2. Do not set a folder alias of inbox = "INBOX"
  3. Run himalaya account sync
  4. Set a folder alias of inbox = "INBOX"
  5. Run himalaya account sync again
  6. Watch as the folders Inbox and INBOX are deleted locally and from the remote server

PS: The first time this happened to me I managed to recover my emails, but forgot to submit the bug report, this time I'm not sure I can recover everything, so if I worded something badly that is not my intention, I knew this could happen.

Status
REPORTED
Submitter
Jalil David Salamé Messina
Assigned to
No-one
Submitted
3 months ago
Updated
a month ago
Labels
0:bug 1:email-lib 2:himalaya

~soywod 3 months ago

This is quite a serious issue. Thanks for reporting it.

If the Inbox folder is not present (or present under another name (ussually INBOX), running himalaya account sync will try to create the folder Inbox.

The default inbox folder name (if no folder alias is given) is "INBOX", not "Inbox", so how come the sync tries to create an "Inbox" folder? How did you install Himalaya?

Changing the config so that under folder aliases inbox = "INBOX" and running himalaya account sync deletes both Inbox and INBOX locally and remotely (if using an IMAP server).

[…]

#Steps to reproduce

  1. Setup a mail account with an Inbox folder called INBOX
  2. Do not set a folder alias of inbox = "INBOX"
  3. Run himalaya account sync
  4. Set a folder alias of inbox = "INBOX"
  5. Run himalaya account sync again
  6. Watch as the folders Inbox and INBOX are deleted locally and from the remote server

I tried to reproduce sth similar in an integration test 1 and it worked as expected. I must miss sth in your flow. Could you please:

  • List your IMAP folders before sync

  • Sync

  • List you IMAP folders and you Maildir folders (should be at ~/.local/share/himalaya/)

  • Add the folder alias

  • Sync with the --dry-run option to see what the sync plans to do

PS: The first time this happened to me I managed to recover my emails, but forgot to submit the bug report, this time I'm not sure I can recover everything, so if I worded something badly that is not my intention, I knew this could happen.

I am sorry about this, I really hope you will be able to recover your inbox folder. The tool is still young, hence the disclaimers I put a bit everywhere:

Disclaimer: this feature is still in beta, we recommand you to backup your emails before using it. Do not use in production before the v1.0.0.

Although I am really surprised about your issue, I don't understand how both folders could be deleted remotely and locally. I designed (and tested) the sync feature in such a way to reduce as much as possible data losses. Let's find the reason behind and fix it!

Jalil David Salamé Messina 3 months ago · edit

Thankfully I managed to recover all the emails (they were on ~/.local/himalaya//.Inbox)!

I am attaching a markdown document with all the steps taken/configuration and a log of the dry run sync, I tried to remove as much personal information as possible.

Hopefully this helps to reproduce the issue.

I knew I was using beta software, so don't worry, I am happy to help improve this as much as possible (even if I get some scares in the process).

On Sun, Jun 4, 2023 at 3:22 PM ~soywod outgoing@sr.ht wrote:

This is quite a serious issue. Thanks for reporting it.

If the Inbox folder is not present (or present under another name (ussually INBOX), running himalaya account sync will try to create the folder Inbox.

The default inbox folder name (if no folder alias is given) is "INBOX", not "Inbox", so how come the sync tries to create an "Inbox" folder? How did you install Himalaya?

Changing the config so that under folder aliases inbox = "INBOX" and running himalaya account sync deletes both Inbox and INBOX locally and remotely (if using an IMAP server).

[…]

#Steps to reproduce

  1. Setup a mail account with an Inbox folder called INBOX
  2. Do not set a folder alias of inbox = "INBOX"
  3. Run himalaya account sync
  4. Set a folder alias of inbox = "INBOX"
  5. Run himalaya account sync again
  6. Watch as the folders Inbox and INBOX are deleted locally and from the remote server

I tried to reproduce sth similar in an integration test 1 and it worked as expected. I must miss sth in your flow. Could you please:

  • List your IMAP folders before sync

  • Sync

  • List you IMAP folders and you Maildir folders (should be at ~/.local/share/himalaya/)

  • Add the folder alias

  • Sync with the --dry-run option to see what the sync plans to do

PS: The first time this happened to me I managed to recover my emails, but forgot to submit the bug report, this time I'm not sure I can recover everything, so if I worded something badly that is not my intention, I knew this could happen.

I am sorry about this, I really hope you will be able to recover your inbox folder. The tool is still young, hence the disclaimers I put a bit everywhere:

Disclaimer: this feature is still in beta, we recommand you to backup your emails before using it. Do not use in production before the v1.0.0.

Although I am really surprised about your issue, I don't understand how both folders could be deleted remotely and locally. I designed (and tested) the sync feature in such a way to reduce as much as possible data losses. Let's find the reason behind and fix it!

-- View on the web: https://todo.sr.ht/~soywod/pimalaya/88#event-241038

Jalil David Salamé Messina referenced this from #88 3 months ago

~soywod 3 months ago

Thankfully I managed to recover all the emails (they were on ~/.local/himalaya//.Inbox)!

Good news! Happy you recovered your emails. Also happy to known that Himalaya does not trash folders, data are still somewhere :D

I am attaching a markdown document with all the steps taken/configuration and a log of the dry run sync, I tried to remove as much personal information as possible.

Did you attach the document or you will in another email? Because I see nothing in this one.

I knew I was using beta software, so don't worry, I am happy to help improve this as much as possible (even if I get some scares in the process).

Thank you for you help! I also lost emails during the process :D

Jalil David Salamé Messina 3 months ago · edit

It seems the log file is too big for sr.ht, the markdown file is embedded below:

#System state

$ himalaya -V
himalaya 0.7.3

nixpkgs is not yet up to date with himalaya

#Himalaya Configuration

# ~/.config/himalaya/config.toml
display-name = "Jalil David Salamé Messina"

["jalil.salame@gmail.com"]
backend = "imap"
default = true
display-name = "Jalil David Salamé Messina"
email = "jalil.salame@gmail.com"
imap-host = "imap.gmail.com"
imap-login = "jalil.salame@gmail.com"
imap-passwd-cmd = "[[REDACTED]]"
imap-port = 993
imap-ssl = true
imap-starttls = false
sender = "smtp"
smtp-host = "smtp.gmail.com"
smtp-login = "jalil.salame@gmail.com"
smtp-passwd-cmd = "[[REDACTED]]"
smtp-port = 465
smtp-ssl = true
smtp-starttls = false
sync = true

["jalil.salame@gmail.com".folder-aliases]
drafts = "[Gmail]/Drafts"
# inbox = "INBOX" # This line will be uncommented in the second sync
sent = "[Gmail]/Sent Mail"
trash = "[Gmail]/Trash"

#Commands

#Folders Before

$ himalaya folders list -o json
[{"delim":"/","name":"INBOX","desc":"INBOX"}]
$ himalaya folders list -o json --disable-cache
[{"delim":"/","name":"INBOX","desc":"Extension(\"\\\\HasNoChildren\")"},{"delim":"/","name":"[Gmail]/All Mail","desc":"All, Extension(\"\\\\HasNoChildren\")"},{"delim":"/","name":"[Gmail]/Drafts","desc":"Drafts, Extension(\"\\\\HasNoChildren\")"},{"delim":"/","name":"[Gmail]/Important","desc":"Extension(\"\\\\HasNoChildren\"), Extension(\"\\\\Important\")"},{"delim":"/","name":"[Gmail]/Sent Mail","desc":"Extension(\"\\\\HasNoChildren\"), Sent"},{"delim":"/","name":"[Gmail]/Spam","desc":"Extension(\"\\\\HasNoChildren\"), Junk"},{"delim":"/","name":"[Gmail]/Starred","desc":"Flagged, Extension(\"\\\\HasNoChildren\")"},{"delim":"/","name":"[Gmail]/Trash","desc":"Extension(\"\\\\HasNoChildren\"), Trash"}]

#Do sync

$ himalaya account sync
[ 1/8] [Gmail]/Spam
[ 2/8] [Gmail]/Important
[ 3/8] [Gmail]/Drafts
[ 4/8] [Gmail]/All Mail
[ 5/8] INBOX
[ 6/8] [Gmail]/Sent Mail
[ 7/8] [Gmail]/Starred
[ 8/8] [Gmail]/Trash

#Folders After

$ himalaya folders list -o json
[{"delim":"/","name":"INBOX","desc":"INBOX"},{"delim":"/","name":"[Gmail]/Starred","desc":"%5BGmail%5D%2FStarred"},{"delim":"/","name":"[Gmail]/Drafts","desc":"%5BGmail%5D%2FDrafts"},{"delim":"/","name":"[Gmail]/Important","desc":"%5BGmail%5D%2FImportant"},{"delim":"/","name":"[Gmail]/Trash","desc":"%5BGmail%5D%2FTrash"},{"delim":"/","name":"[Gmail]/Spam","desc":"%5BGmail%5D%2FSpam"},{"delim":"/","name":"[Gmail]/All Mail","desc":"%5BGmail%5D%2FAll%20Mail"},{"delim":"/","name":"[Gmail]/Sent Mail","desc":"%5BGmail%5D%2FSent%20Mail"}]
$ himalaya folders list -o json --disable-cache
[{"delim":"/","name":"INBOX","desc":"Extension(\"\\\\HasNoChildren\")"},{"delim":"/","name":"[Gmail]/All Mail","desc":"All, Extension(\"\\\\HasNoChildren\")"},{"delim":"/","name":"[Gmail]/Drafts","desc":"Drafts, Extension(\"\\\\HasNoChildren\")"},{"delim":"/","name":"[Gmail]/Important","desc":"Extension(\"\\\\HasNoChildren\"), Extension(\"\\\\Important\")"},{"delim":"/","name":"[Gmail]/Sent Mail","desc":"Extension(\"\\\\HasNoChildren\"), Sent"},{"delim":"/","name":"[Gmail]/Spam","desc":"Extension(\"\\\\HasNoChildren\"), Junk"},{"delim":"/","name":"[Gmail]/Starred","desc":"Flagged, Extension(\"\\\\HasNoChildren\")"},{"delim":"/","name":"[Gmail]/Trash","desc":"Extension(\"\\\\HasNoChildren\"), Trash"}]

$ ls -a1 ~/.local/share/himalaya/jalil.salame@gmail.com/
.%5BGmail%5D%2FAll%20Mail
.%5BGmail%5D%2FDrafts
.%5BGmail%5D%2FImportant
.%5BGmail%5D%2FSent%20Mail
.%5BGmail%5D%2FSpam
.%5BGmail%5D%2FStarred
.%5BGmail%5D%2FTrash
.id-mapper.sqlite
.sync.sqlite
cur
new
tmp

#New Sync

Remove commented line in configuration

$ himalaya folders list -o json
[{"delim":"/","name":"INBOX","desc":"INBOX"},{"delim":"/","name":"[Gmail]/Starred","desc":"%5BGmail%5D%2FStarred"},{"delim":"/","name":"[Gmail]/Drafts","desc":"%5BGmail%5D%2FDrafts"},{"delim":"/","name":"[Gmail]/Important","desc":"%5BGmail%5D%2FImportant"},{"delim":"/","name":"[Gmail]/Trash","desc":"%5BGmail%5D%2FTrash"},{"delim":"/","name":"[Gmail]/Spam","desc":"%5BGmail%5D%2FSpam"},{"delim":"/","name":"[Gmail]/All Mail","desc":"%5BGmail%5D%2FAll%20Mail"},{"delim":"/","name":"[Gmail]/Sent Mail","desc":"%5BGmail%5D%2FSent%20Mail"}]
$ himalaya folders list -o json --disable-cache
[{"delim":"/","name":"INBOX","desc":"Extension(\"\\\\HasNoChildren\")"},{"delim":"/","name":"[Gmail]/All Mail","desc":"All, Extension(\"\\\\HasNoChildren\")"},{"delim":"/","name":"[Gmail]/Drafts","desc":"Drafts, Extension(\"\\\\HasNoChildren\")"},{"delim":"/","name":"[Gmail]/Important","desc":"Extension(\"\\\\HasNoChildren\"), Extension(\"\\\\Important\")"},{"delim":"/","name":"[Gmail]/Sent Mail","desc":"Extension(\"\\\\HasNoChildren\"), Sent"},{"delim":"/","name":"[Gmail]/Spam","desc":"Extension(\"\\\\HasNoChildren\"), Junk"},{"delim":"/","name":"[Gmail]/Starred","desc":"Flagged, Extension(\"\\\\HasNoChildren\")"},{"delim":"/","name":"[Gmail]/Trash","desc":"Extension(\"\\\\HasNoChildren\"), Trash"}]

$ RUST_LOG=info himalaya accounts sync --dry-run 2> himalaya_sync.log
Envelopes patch:
 - Removing envelope 1685954412.M173895356P24647V42I218917.gemini,S=36842 from local backend folder [Gmail]/Important
 - Copying remote envelope 422 to local folder [Gmail]/Important
 - Removing envelope 1685954394.M8240699P24647V42I218645.gemini,S=11593 from local backend folder [Gmail]/Important
 - Copying remote envelope 216 to local folder [Gmail]/Important
 - Removing envelope 1685954431.M753909868P24647V42I219220.gemini,S=13753 from local backend folder [Gmail]/Important
 - Copying remote envelope 199 to local folder [Gmail]/Important
 - Removing envelope 1685954397.M735317537P24647V42I218702.gemini,S=7157 from local backend folder [Gmail]/Important
 - Copying remote envelope 396 to local folder [Gmail]/Important
 - Removing envelope 1685954405.M505189656P24647V42I218816.gemini,S=42027 from local backend folder [Gmail]/Important
 - Copying remote envelope 299 to local folder [Gmail]/Important
 - Removing envelope 1685954406.M296509738P24647V42I218829.gemini,S=42201 from local backend folder [Gmail]/Important
 - Copying remote envelope 251 to local folder [Gmail]/Important
 - Removing envelope 1685954386.M682154938P24647V42I218531.gemini,S=38598 from local backend folder [Gmail]/Important
 - Copying remote envelope 113 to local folder [Gmail]/Important
 - Removing envelope 1685954447.M546356278P24647V42I219473.gemini,S=35811 from local backend folder [Gmail]/Important
 - Copying remote envelope 99 to local folder [Gmail]/Important
 - Removing envelope 1685954430.M162943518P24647V42I219211.gemini,S=7539 from local backend folder [Gmail]/Important
 - Copying remote envelope 398 to local folder [Gmail]/Important
 - Removing envelope 1685954421.M777360301P24647V42I219072.gemini,S=36829 from local backend folder [Gmail]/Important
 - Copying remote envelope 507 to local folder [Gmail]/Important
 - Removing envelope 1685954378.M718188938P24647V42I218402.gemini,S=30853 from local backend folder [Gmail]/Important
 - Copying remote envelope 133 to local folder [Gmail]/Important
 - Removing envelope 1685954446.M771313721P24647V42I219462.gemini,S=51577 from local backend folder [Gmail]/Important
 - Copying remote envelope 219 to local folder [Gmail]/Important
 - Removing envelope 1685954390.M340369365P24647V42I218590.gemini,S=32058 from local backend folder [Gmail]/Important
 - Copying remote envelope 443 to local folder [Gmail]/Important
 - Removing envelope 1685954442.M159300398P24647V42I219393.gemini,S=22314 from local backend folder [Gmail]/Important
 - Copying remote envelope 204 to local folder [Gmail]/Important
 - Removing envelope 1685954456.M339786862P24647V42I219622.gemini,S=42029 from local backend folder [Gmail]/Important
 - Copying remote envelope 314 to local folder [Gmail]/Important
 - Removing envelope 1685954406.M900005691P24647V42I218839.gemini,S=37944 from local backend folder [Gmail]/Important
 - Copying remote envelope 70 to local folder [Gmail]/Important
 - Removing envelope 1685954416.M348796778P24647V42I218982.gemini,S=6186 from local backend folder [Gmail]/Important
 - Copying remote envelope 717 to local folder [Gmail]/Important
 - Removing envelope 1685954397.M662393292P24647V42I218699.gemini,S=11993 from local backend folder [Gmail]/Important
 - Copying remote envelope 719 to local folder [Gmail]/Important
 - Removing envelope 1685954548.M372479635P24647V42I221076.gemini,S=27482 from local backend folder [Gmail]/All Mail
 - Copying remote envelope 14 to local folder [Gmail]/All Mail
 - Removing envelope 1685954546.M703805313P24647V42I221052.gemini,S=6920 from local backend folder [Gmail]/All Mail
 - Copying remote envelope 165 to local folder [Gmail]/All Mail
 - Removing envelope 1685954504.M749738288P24647V42I220404.gemini,S=30235 from local backend folder [Gmail]/All Mail
 - Copying remote envelope 664 to local folder [Gmail]/All Mail
 - Removing envelope 1685954556.M296316640P24647V42I221190.gemini,S=36749 from local backend folder [Gmail]/All Mail
 - Copying remote envelope 550 to local folder [Gmail]/All Mail
 - Removing envelope 1685954487.M429560833P24647V42I220126.gemini,S=7320 from local backend folder [Gmail]/All Mail
 - Copying remote envelope 936 to local folder [Gmail]/All Mail
 - Removing envelope 1685954509.M623632234P24647V42I220484.gemini,S=146168 from local backend folder [Gmail]/All Mail
 - Copying remote envelope 915 to local folder [Gmail]/All Mail
 - Removing envelope 1685954482.M35507576P24647V42I220030.gemini,S=36798 from local backend folder [Gmail]/All Mail
 - Copying remote envelope 619 to local folder [Gmail]/All Mail
 - Removing envelope 1685954511.M468280954P24647V42I220516.gemini,S=25330 from local backend folder [Gmail]/All Mail
 - Copying remote envelope 251 to local folder [Gmail]/All Mail
 - Removing envelope 1685954521.M922851520P24647V42I220677.gemini,S=74045 from local backend folder [Gmail]/All Mail
 - Copying remote envelope 667 to local folder [Gmail]/All Mail
 - Removing envelope 1685954553.M787260324P24647V42I221153.gemini,S=7505 from local backend folder [Gmail]/All Mail
 - Copying remote envelope 822 to local folder [Gmail]/All Mail
 - Removing envelope 1685954481.M55430994P24647V42I220014.gemini,S=12717 from local backend folder [Gmail]/All Mail
 - Copying remote envelope 911 to local folder [Gmail]/All Mail
 - Removing envelope 1685954541.M435878842P24647V42I220966.gemini,S=36833 from local backend folder [Gmail]/All Mail
 - Copying remote envelope 645 to local folder [Gmail]/All Mail
 - Removing envelope 1685954496.M491201903P24647V42I220270.gemini,S=7762 from local backend folder [Gmail]/All Mail
 - Copying remote envelope 832 to local folder [Gmail]/All Mail
 - Removing envelope 1685954519.M819798400P24647V42I220644.gemini,S=32030 from local backend folder [Gmail]/All Mail
 - Copying remote envelope 566 to local folder [Gmail]/All Mail
 - Removing envelope 1685954521.M921840072P24647V42I220676.gemini,S=40048 from local backend folder [Gmail]/All Mail
 - Copying remote envelope 773 to local folder [Gmail]/All Mail
 - Removing envelope 1685954485.M631693127P24647V42I220093.gemini,S=36822 from local backend folder [Gmail]/All Mail
 - Copying remote envelope 536 to local folder [Gmail]/All Mail
 - Removing envelope 1685954563.M570730396P24647V42I221309.gemini,S=57111 from local backend folder [Gmail]/All Mail
 - Copying remote envelope 67 to local folder [Gmail]/All Mail
 - Removing envelope 1685954470.M565715688P24647V42I219831.gemini,S=36772 from local backend folder [Gmail]/All Mail
 - Copying remote envelope 526 to local folder [Gmail]/All Mail
 - Removing envelope 1685954563.M569282790P24647V42I221308.gemini,S=39887 from local backend folder [Gmail]/All Mail
 - Copying remote envelope 179 to local folder [Gmail]/All Mail
 - Removing envelope 1685954487.M548643571P24647V42I220129.gemini,S=29751 from local backend folder [Gmail]/All Mail
 - Copying remote envelope 280 to local folder [Gmail]/All Mail
 - Removing envelope 1685954560.M919950487P24647V42I221267.gemini,S=30332 from local backend folder [Gmail]/All Mail
 - Copying remote envelope 292 to local folder [Gmail]/All Mail
 - Removing envelope 1685954585.M659359754P24647V42I221649.gemini,S=42184 from local backend folder INBOX
 - Copying remote envelope 1264 to local folder INBOX
 - Removing envelope 1685954654.M998834509P24647V42I222804.gemini,S=6908 from local backend folder INBOX
 - Copying remote envelope 1277 to local folder INBOX
 - Removing envelope 1685954615.M340185437P24647V42I222133.gemini,S=42151 from local backend folder INBOX
 - Copying remote envelope 1211 to local folder INBOX
 - Removing envelope 1685954656.M401594645P24647V42I222830.gemini,S=38227 from local backend folder INBOX
 - Copying remote envelope 1464 to local folder INBOX
 - Removing envelope 1685954633.M48159360P24647V42I222432.gemini,S=69251 from local backend folder INBOX
 - Copying remote envelope 1207 to local folder INBOX
 - Removing envelope 1685954580.M893174040P24647V42I221568.gemini,S=42151 from local backend folder INBOX
 - Copying remote envelope 1169 to local folder INBOX
 - Removing envelope 1685954638.M837461877P24647V42I222535.gemini,S=40997 from local backend folder INBOX
 - Copying remote envelope 1024 to local folder INBOX
 - Removing envelope 1685954635.M548213222P24647V42I222477.gemini,S=4258 from local backend folder INBOX
 - Copying remote envelope 839 to local folder INBOX
 - Removing envelope 1685954609.M346183548P24647V42I222032.gemini,S=36797 from local backend folder INBOX
 - Copying remote envelope 1109 to local folder INBOX
 - Removing envelope 1685954578.M381220111P24647V42I221524.gemini,S=112977 from local backend folder INBOX
 - Copying remote envelope 1554 to local folder INBOX
 - Removing envelope 1685954589.M822106318P24647V42I221712.gemini,S=36855 from local backend folder INBOX
 - Copying remote envelope 1085 to local folder INBOX
 - Removing envelope 1685954633.M221338237P24647V42I222436.gemini,S=8220 from local backend folder INBOX
 - Copying remote envelope 903 to local folder INBOX

Estimated patch length for account jalil.salame@gmail.com to be synchronized: 102

Thankfully I managed to recover all the emails (they were on ~/.local/himalaya//.Inbox)!

Good news! Happy you recovered your emails. Also happy to known that Himalaya does not trash folders, data are still somewhere :D

I am attaching a markdown document with all the steps taken/configuration and a log of the dry run sync, I tried to remove as much personal information as possible.

Did you attach the document or you will in another email? Because I see nothing in this one.

I knew I was using beta software, so don't worry, I am happy to help improve this as much as possible (even if I get some scares in the process).

Thank you for you help! I also lost emails during the process :D

-- View on the web: https://todo.sr.ht/~soywod/pimalaya/88#event-241219

Jalil David Salamé Messina referenced this from #88 3 months ago

~soywod 3 months ago

It seems the log file is too big for sr.ht, the markdown file is embedded below:

I don't see any remote deletion in the logs. Plus I don't see any "Inbox" but only "INBOX" (and I think gmail inbox is also named INBOX). Plus the integration test I did does the same and all works as expected. So I am a bit confused.

My intuition: the problem could come from the gmail's folder "[Gmail]/All Mail", because it is not really a folder. It is more like an alias. Removing an email from the "INBOX" folder also removes it from the "[Gmail]/All Mail", which could lead to some strange sync state.

Could you try once again your flow with this option:

sync-folders-strategy.exclude = ["[Gmail]/Sent Mail"]

~soywod added 2:himalaya 3 months ago

~soywod added 1:email-lib 3 months ago

~soywod added 0:bug 3 months ago

Jalil David Salamé Messina 3 months ago · edit

I don't think it's related to GMail, the first time it happened it was on the gmail account I tested it with (thi one), but the thing that made me report the bug was that it happened on my University email account which is using the Exchange mailbox (Microsoft based I think).

I don't want to test it there because it doesn't have the "All Mail" folder, and I am scared of actually loosing the mails there. I have lost a (thankfully unimportant) email already, just a single one, because it came in in between me trying to recover my emails so I hadn't backed it up.

This is the data from that account (almost the same configuration, just diferent imap/smtp hosts).

--- Folders --- [{"delim":"/","name":"Calendar","desc":"Extension("\\HasChildren")"},{"delim":"/","name":"Calendar/Birthdays","desc":"Extension("\\HasNoChildren")"},{"delim":"/","name":"Contacts","desc":"Extension("\\HasChildren")"},{"delim":"/","name":"Deleted Items","desc":"Extension("\\HasNoChildren"), Trash"},{"delim":"/","name":"Drafts","desc":"Extension("\\HasNoChildren"), Drafts"},{"delim":"/","name":"INBOX","desc":"Marked, Extension("\\HasNoChildren")"},{"delim":"/","name":"Journal","desc":"Extension("\\HasNoChildren")"},{"delim":"/","name":"Junk Email","desc":"Extension("\\HasNoChildren"), Junk"},{"delim":"/","name":"Notes","desc":"Extension("\\HasNoChildren")"},{"delim":"/","name":"Outbox","desc":"Extension("\\HasNoChildren")"},{"delim":"/","name":"Sent","desc":"Extension("\\HasNoChildren")"},{"delim":"/","name":"Sent Items","desc":"Extension("\\HasNoChildren"), Sent"},{"delim":"/","name":"Tasks","desc":"Extension("\\HasNoChildren")"},{"delim":"/","name":"Trash","desc":"Extension("\\HasNoChildren")"}] --- Folders END ---

--- Local Folders --- drwxr-xr-x - jalil 4 Jun 12:43  .Calendar drwxr-xr-x - jalil 4 Jun 12:43  .Calendar%2FBirthdays drwxr-xr-x - jalil 4 Jun 12:43  .Contacts drwxr-xr-x - jalil 4 Jun 12:43  .Deleted%20Items drwxr-xr-x - jalil 4 Jun 12:43  .Drafts .rw-r--r-- 1.2M jalil 5 Jun 12:16  .id-mapper.sqlite drwxr-xr-x - jalil 5 Jun 09:55  .Inbox drwxr-xr-x - jalil 4 Jun 12:43  .Journal drwxr-xr-x - jalil 4 Jun 12:43  .Junk%20Email drwxr-xr-x - jalil 4 Jun 12:43  .Notes drwxr-xr-x - jalil 4 Jun 12:43  .Outbox drwxr-xr-x - jalil 4 Jun 12:43  .Sent drwxr-xr-x - jalil 4 Jun 12:43  .Sent%20Items .rw-r--r-- 2.2M jalil 5 Jun 14:47  .sync.sqlite drwxr-xr-x - jalil 4 Jun 12:43  .Tasks drwxr-xr-x - jalil 4 Jun 12:43  .Trash drwxr-xr-x - jalil 5 Jun 14:47  cur drwxr-xr-x - jalil 4 Jun 13:25  new drwxr-xr-x - jalil 5 Jun 14:47  tmp --- Local Folders END ---

--- Dry Run --- Folders patch:

  • Adding folder Inbox to local cache
  • Adding folder Inbox to remote cache
  • Adding folder Inbox to remote backend
  • Removing folder INBOX from local cache
  • Removing folder INBOX from remote cache
  • Removing folder INBOX from remote backend

Envelopes patch:

  • Removing envelope 1685959980.M598937483P35553V42I228798.gemini,S=9426 from local cache folder INBOX
  • Removing envelope 1685875847.M838537102P128943V42I166074.gemini,S=9426 from local backend folder INBOX
  • Removing envelope 7849 from remote cache folder INBOX [[SKIPPED]] ~300KiB of the same messages
  • Removing envelope 1685875776.M261748851P128943V42I164970.gemini,S=7126 from local cache folder Inbox
  • Removing envelope 1685875776.M261748851P128943V42I164970.gemini,S=7126 from local backend folder Inbox
  • Removing envelope 8040 from remote cache folder Inbox

Estimated patch length for account [[REDACTED]] to be synchronized: 4201 --- Dry Run END ---

It seems the log file is too big for sr.ht, the markdown file is embedded below:

I don't see any remote deletion in the logs. Plus I don't see any "Inbox" but only "INBOX" (and I think gmail inbox is also named INBOX). Plus the integration test I did does the same and all works as expected. So I am a bit confused.

My intuition: the problem could come from the gmail's folder "[Gmail]/All Mail", because it is not really a folder. It is more like an alias. Removing an email from the "INBOX" folder also removes it from the "[Gmail]/All Mail", which could lead to some strange sync state.

Could you try once again your flow with this option:

sync-folders-strategy.exclude = ["[Gmail]/Sent Mail"]

-- View on the web: https://todo.sr.ht/~soywod/pimalaya/88#event-241244

Jalil David Salamé Messina referenced this from #88 3 months ago

~soywod 3 months ago

Sorry for the delay. Now that the new sync is on master I can focus on your issue.

I don't think it's related to GMail, the first time it happened it was on the gmail account I tested it with (thi one), but the thing that made me report the bug was that it happened on my University email account which is using the Exchange mailbox (Microsoft based I think).

I still cannot reproduce this behaviour in a test suite. What was the original name of your remote IMAP inbox name?

Meanwhile I continue my tests and let you know.

~soywod 3 months ago

The best I could get is a sync error with this procedure:

  • init an IMAP server with only one folder "INBOX"

  • sync

  • the "INBOX" folder is present both remotely and locally

  • add a folder alias inbox = "MyCustomInbox"

  • sync again leads to an error:

    thread 'bug_88' panicked at 'called `Result::unwrap()` on an `Err` value:
SyncEnvelopesError(BackendError(MaildirBackendError(ReadDirError("/tmp/.tmpHLYgVK/sync-dir/.MyCustomInbox"))))',
email/tests/sync.rs:501:25

I am thinking that maybe folders alias should not be taken into consideration during synchronization. It is better to stay as close as possible from original folders name. Your issue seems to be related to the one I just discover. I will push a fix and let you know, I hope it will fix your issue as well!

~soywod 3 months ago

I am thinking that maybe folders alias should not be taken into consideration during synchronization. It is better to stay as close as possible from original folders name. Your issue seems to be related to the one I just discover. I will push a fix and let you know, I hope it will fix your issue as well!

Good news, folders aliases are not used during sync, the problem comes only from the INBOX folder. The synchronization put cached emails in a Maildir++ folder, which means that inbox mailbox is at the root level and other mailboxes are in folders starting by a dot. I am afraid that Maildir++ is not suitable for keeping folders hierarchy flat.

Jalil David Salamé Messina 3 months ago · edit

I don't know if this changes something but the initial configuration was using home-manager, which does set a default alias of inbox to "Inbox", which I changed to "INBOX", maybe that is the issue?

I will test it when I have time... It may take me a week or two though...

Sorry for the delay, I've been (and will continue to be) busy, and the GMail app has no (easy) way to send plaintext emails.

On Tue, Jun 13, 2023 at 5:52 PM ~soywod outgoing@sr.ht wrote:

I am thinking that maybe folders alias should not be taken into consideration during synchronization. It is better to stay as close as possible from original folders name. Your issue seems to be related to the one I just discover. I will push a fix and let you know, I hope it will fix your issue as well!

Good news, folders aliases are not used during sync, the problem comes only from the INBOX folder. The synchronization put cached emails in a Maildir++ folder, which means that inbox mailbox is at the root level and other mailboxes are in folders starting by a dot. I am afraid that Maildir++ is not suitable for keeping folders hierarchy flat.

-- View on the web: https://todo.sr.ht/~soywod/pimalaya/88#event-242483

Jalil David Salamé Messina referenced this from #88 3 months ago

~soywod 3 months ago

I don't know if this changes something but the initial configuration was using home-manager, which does set a default alias of inbox to "Inbox", which I changed to "INBOX", maybe that is the issue?

Yes indeed, home manager mail module sets the default inbox to "Inbox", but then it should not even work. I also use home manager and when I first used it I got an error, I needed to add "INBOX" in my folders alias. Strange that I cannot reproduce the same behaviour as yours. Let me know whenever you have news!

Register here or Log in to comment, or comment via email.