~xerool/fennel-ls#36: Emacs 29.4 eglot seems to die instantly after the initial exchange

So after fixing issues described in #34 and #35, I finally got to the state where eglot and fennel-ls managed to perform the initial exchange. However, at that point the next problem appears, and I'm totally not sure about this one this time.

It goes like this:

#1. fennel-ls receives from eglot, apparently in 3 different messages, as indicated in #35:

Content-Length: 2382

{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"processId":31636,"rootPath":"d:/work/repos/Fennel/","rootUri":"file:///d%3A/work/repos/Fennel","initializationOptions":{},"capabilities":{"workspace":{"applyEdit":true,"executeCommand":{"dynamicRegistration":false},"workspaceEdit":{"documentChanges":true},"didChangeWatchedFiles":{"dynamicRegistration":true},"symbol":{"dynamicRegistration":false},"configuration":true,"workspaceFolders":true},"textDocument":{"synchronization":{"dynamicRegistration":false,"willSave":true,"willSaveWaitUntil":true,"didSave":true},"completion":{"dynamicRegistration":false,"completionItem":{"snippetSupport":false,"deprecatedSupport":true,"resolveSupport":{"properties":["documentation","details","additionalTextEdits"]},"tagSupport":{"valueSet":[1]}},"contextSupport":true},"hover":{"dynamicRegistration":false,"contentFormat":["plaintext"]},"signatureHelp":{"dynamicRegistration":false,"signatureInformation":{"parameterInformation":{"labelOffsetSupport":true},"activeParameterSupport":true}},"references":{"dynamicRegistration":false},"definition":{"dynamicRegistration":false,"linkSupport":true},"declaration":{"dynamicRegistration":false,"linkSupport":true},"implementation":{"dynamicRegistration":false,"linkSupport":true},"typeDefinition":{"dynamicRegistration":false,"linkSupport":true},"documentSymbol":{"dynamicRegistration":false,"hierarchicalDocumentSymbolSupport":true,"symbolKind":{"valueSet":[1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26]}},"documentHighlight":{"dynamicRegistration":false},"codeAction":{"dynamicRegistration":false,"codeActionLiteralSupport":{"codeActionKind":{"valueSet":["quickfix","refactor","refactor.extract","refactor.inline","refactor.rewrite","source","source.organizeImports"]}},"isPreferredSupport":true},"formatting":{"dynamicRegistration":false},"rangeFormatting":{"dynamicRegistration":false},"rename":{"dynamicRegistration":false},"inlayHint":{"dynamicRegistration":false},"publishDiagnostics":{"relatedInformation":false,"codeDescriptionSupport":false,"tagSupport":{"valueSet":[1,2]}}},"window":{"workDoneProgress":true},"general":{"positionEncodings":["utf-32","utf-8","utf-16"]},"experimental":{}},"workspaceFolders":[{"uri":"file:///d%3A/work/repos/Fennel","name":"d:/work/repos/Fennel/"}]}}

#2. fennel-ls answers to eglot:

{ ["id"] = 1,["jsonrpc"] = 2.0,["result"] = { ["capabilities"] = { ["completionProvider"] = { ["workDoneProgress"] = false,["resolveProvider"] = false,["triggerCharacters"] = { [1] = (,[2] = [,[3] = {,[4] = .,[5] = :,[6] = ",} ,["completionItem"] = { ["labelDetailsSupport"] = false,} ,} ,["positionEncoding"] = utf-8,["referencesProvider"] = { ["workDoneProgress"] = false,} ,["codeActionProvider"] = { ["workDoneProgress"] = false,} ,["definitionProvider"] = { ["workDoneProgress"] = false,} ,["hoverProvider"] = { ["workDoneProgress"] = false,} ,["renameProvider"] = { ["workDoneProgress"] = false,} ,["textDocumentSync"] = { ["openClose"] = true,["change"] = 2,} ,} ,["serverInfo"] = { ["version"] = 0.1.0,["name"] = fennel-ls,} ,} ,}

#3. What follows then in the eglot events window:

[internal] Sat Sep 28 15:50:38 2024:
(:message "Connection state changed" :change "killed\n")
----------b---y---e---b---y---e----------

#4. Then there's a 30 seconds pause, nothing is going on. But asking Emacs about eglot servers with `M-x eglot-server-menu` says "no servers", so it's indeed already dead.

#5. fennel-ls main loop executes exactly 30 times, in quick succession, as if reading 30 `nil`s. I'm able to track this because I've added `io.stderr:write()` with an incremental counter around the read and write calls in the loop. Writing to `stderr` because writing to `stdout` or just `print()`-ing doesn't produce output in the eglot window.

#6. On exactly the 30th time the read of fennel-ls starts, but this time it doesn't proceed beyond it, and this is what's printed:

[stderr] 
[stderr] nil
[stderr] nil
[stderr] Process EGLOT (Fennel/(fennel-mode)) stderr finished

I'm totally at a loss here. The eglot server seems to immediately terminate after the handhake, when the "byebye" is written. But why?

Status: REPORTED
Submitter: ~noncom
Assigned to: No-one
Submitted: 2 months ago
Updated: 2 months ago
Labels: No labels applied.

~noncom 2 months ago*

After a lot of debugging this, I tend to think that there's probably some fundamental issue in how Emacs or Eglot on Windows communicate with programs via stdio. From debugging the output of eglot it seems that every \n that fennel-ls sends is automatically prepended by a \r. Therefore when fennel-ls writes \r\n, the output in the debug is \r\r\n, and whenI change it to \n only, the output is \r\n.

The official specification of the LSP defines that the headers should be separated by just \r\n, and the headers from the content should be separated by \r\n (thus last header is separated from the body by \r\n\r\n). I tried to replicate this, testing various combinations, and even when the output looked exactly as the described in the specification, it still was not working. Eglot seems to never understand that fennel-ls responded to it, and keeps waiting, timing out after 30 seconds. This is happening even though the debug output that I'm reading is the output taken from the Emacs/Eglot streams themselevs, so it does receive the information, just something isn't working.

I also tried including various combinatins of \r\n at the end of the answers from fennel-ls, just in case, because I read somewhere that some LSP implementations were known to require that, even though it wasn't in the spec. This didn't have any effect.

Communicating the other way, as described in #34, apparently, leaves \r out completely from the messages received by fennel-ls, so that's another strangeness. It is fixed by making fennel-ls to expect \n, and not \r\n. Sadly, I couldn't yet find the way to fix the other way around.

An interesting fact is that when I tried launching fennel-ls from the CLI by executing lua fennel-ls.lua in cmd, and manually entered the same header and content as Eglot sends to it, pressing the return button where each \r\n should be, I was able to communicate with fennel-ls reliably. There was no timeout, and no problem, the messages were flowig both directions, we chatted a bit, without any issues.

Also, I tried using fennel-ls in Emacs on Ubuntu, and it worked immediately without any issues. The configuration of Emacs is the same, and is absolutely minimal.

So at this point I'm not sure what else can I do, or if anyone can do anything at all. The issue is probably in some cross-platform specifics of some communication facility in Emacs. Might revisit this later.