~soywod/pimalaya#192: 
Crashes with "unexpected end of file" when manipulating large IMAP folder

I had an IMAP folder with 1.3M mail in it.  Naively trying to list all the headers by giving a large page size failed as below. I bisected to find it crashes when s > 16389:

$ himalaya list -s16390 Error: cannot fetch imap emails within uid range 1303885,1303884,1303883,1303882,1303881,1303880,1303879,1303878,1303877,1303876,1303875,1303874,1303873,1303872,1303871,1303870,1303869,1303868,1303867,1303866,1303865,1303864,1303863,1303862,1303861,1303860,1303859,1303858,1303857,1303856,1303855,1303854,1303853,1303852,1303851,1303850,1303849,1303848,1303847,1303846,1303845,1303844,1303843,1303842,1303841,1303840,1303839,1303838,1303837,1303836,1303835,1303834,1303833,1303832,1303831,1303830,1303829,1303828,1303827,1303826,1303825,1303824,1303823,1303822,1303821,1303820,1303819,1303818,1303817,1303816,1303815,1303814,1303813,1303812,1303811,1303810,1303809,1303808,1303807,1303806,1303805,1303804,1303803,1303802,...,1287499,1287498,1287497,1287496

Caused by: unexpected end of file

The workaround is apparently -s0, which completed succesfully.

Similarly, trying to move large blocks of mails failed too.  Even when splitting into batches of 1000, some of the batches failed in a similar way:

$ himalaya move Folder cat uids

Error: cannot move email(s) 430964,430960,430958,430956,430953,430931,430930,430920,430915,430896,430889,430882,430867,430851,430787,430566,430544,430536,430485,430429,430428,430413,430386,430378,430371,430357,430355,430333,430286,430237,430203,430187,430160,430110,430096,430090,430069,430059,429990,429980,429976,429975,429973,429965,429963,429956,429955,429954,429951,429923,429916,429901,429891,429889,429868,429846,429700,429684,429660,429608,429565,429472,429430,429403,429400,429390,429364,429356,429353,429343,429323,429231,429219,429217,429208,429206,429181,429059,429020,428996,428991,428973,428972,428961,428949,428923,428893,428890,428883,428875,428863,428848,428842,428810,428750,428732,428715,428693,428566,428509,428490,428488,428486,428476,428439,428433,428430,428427,428418,428410,428384,428380,428379,428369,428355,428330,428329,428323,428321,428316,428299,428298,428297,428295,428294,428290,428287,428286,428262,428254,428248,428247,428240,428231,428225,428217,428196,428183,428181,428174,428138,428094,427975,427947,427945,427920,427885,427867,427863,427859,427855,427852,427851,427847,427846,427819,427817,427811,427807,427798,427796,427767,427763,427746,427618,427537,427399,427398,427388,427383,427379,427377,427374,427373,427371,427370,427346,427337,427300,427257,427196,427161,426991,426990,426986,426985,426984,426973,426971,426965,426955,426948,426905,426901,426722,426718,426684,426604,426589,426587,426583,426527,426515,426506,426345,426183,426168,426151,426131,426086,426071,425976,425933,425904,425887,425882,425852,425846,425841,425815,425800,425799,425726,425647,425642,425614,425537,425530,425506,425492,425487,425486,425451,425437,425432,425231,425209,425173,425168,425154,425091,425086,425076,425043,424913,424877,424869,424804,424770,424767,424761,424694,424643,424614,424588,424583,424532,424481,424409,424404,424351,424347,423952,423951,423668,423667,423666,422531,422125,421854,421553,421130,420973,420873,420048,419114,418880,418749,418606,417930,417893,417793,417760,417724,417621,417510,417118,417110,417092,417088,417082,417047,417023,416956,416938,416764,416741,416733,416705,416704,416685,416288,416174,416164,416146,416145,416137,416125,415424,415239,415090,415003,414955,414902,414894,414893,414512,414360,414352,414351,414259,414258,414245,414227,414188,413988,413919,413786,413612,413607,413479,413447,413428,413419,413410,413361,413203,413127,412959,412934,412637,412598,412071,412020,412015,412013,412010,411997,411991,411985,411982,411981,411961,411922,411899,411895,411866,411865,411820,411797,411795,411688,411513,411471,411456,411455,411446,411419,411350,411322,411099,411093,411068,411053,411005,410694,410682,410565,410454,410390,410377,410365,410364,410277,410197,410187,410134,410115,410018,409770,409720,409716,409713,409712,409709,409704,409478,409185,409055,408911,408841,408759,408491,408061,407995,407929,407912,407664,407556,407533,407522,407521,407480,407441,407434,407433,407432,407417,407413,407364,407344,407334,407281,407280,407197,407139,406984,406734,406729,406589,406526,406403,406276,406147,405793,405503,405435,405427,405402,405240,405103,405025,404829,404743,404735,404435,404369,404311,404308,404302,403825,403686,403427,403392,403316,403269,403160,402827,402618,402381,401719,401273,401231,401137,401038,401019,400917,400795,400731,400663,400433,400413,389076,385324,381095,379751,376966,376740,376545,374764,374452,365571,365088,364613,362319,357057,356628,356027,355458,351044,350003,341188,340679,340330,339810,339356,338927,336422,334799,333441,331513,331017,330414,330270,330221,329453,326964,324734,274790,267703,204823,204793,204781,204644,204570,204508,121906,114060,109451,78499,78495,71058,63769,63683,63682,62750,62695,62350,60765,59880,58687,54980,40689,40688,28728,24734,24571,18755,18438,6828,6827,6826,6062,6009,6008,6006,6005,6004,5963,5846,5795,5666,5276,5019,5018,5017,5016,4944,4837,4836,4748,4682,4493,4369,4284,4281,4278 from INBOX to Folder

Caused by: unexpected end of file

Which makes it feel like it's crashing on a particular UID somehow.

This is against an old cyrus imapd (2.5.10), using himalaya 0.9.0 from Cargo. I can probably repro if needed.

-- Matthew Hodgson Matrix.org

Status
REPORTED
Submitter
Matthew Hodgson
Assigned to
Submitted
4 months ago
Updated
4 months ago
Labels
0:todo

~soywod 4 months ago

I had an IMAP folder with 1.3M mail in it. Naively trying to list all the headers by giving a large page size failed as below. I bisected to find it crashes when s > 16389:

$ himalaya list -s16390 Error: cannot fetch imap emails within uid range

I discussed about your issue with people from another Matrix room dedicated to emails, they all came to the same conclusion: this is an error from your IMAP server. According to RFCs, if a server cannot handle a request, it should return peacefully (not with unexpected end of file). Also, RFCs mention nothing about the maximum length of a sequence range. So it is clients' reponsability to avoid big sequences.

I propose the following: if a sequence range exceed a specific length (100? 200? 500? 1000?), we split it and we do multiple calls the the IMAP server. What do you think?

Register here or Log in to comment, or comment via email.