~tsileo/microblog.pub#96: 
Ingestion is not restarted after failure

I noticed that my stream stopped updating, but new posts were still showing up on uvicorn.log

Looking at the incoming.log, it seems to have stopped with:

2022-12-29 16:06:02.861 | INFO     | app.utils.workers:run_forever:56 - Waiting for tasks to finish done={<Task finished name='Task-2' coro=<Worker._main_loop() done, defined at /app/app/utils/workers.py:28> exception=MemoryError()>}/pending={<Task pending name='Task-3' coro=<Worker._until_stopped() running at /app/app/utils/workers.py:38> wait_for=>} 2022-12-29 16:06:02.889 | INFO     | app.utils.workers:run_forever:58 - Cancelling 1 tasks Exception ignored in thread started by: <object repr() failed> MemoryError:

Restarting the docker container seems to have been enough to get the missing inbox entities.

Maybe there should be some automatic restart logic when the ingestion fails with a fatal memory error (or at least kill the docker container, so that the supervisor can restart it)?

Status
REPORTED
Submitter
João Costa
Assigned to
No-one
Submitted
a month ago
Updated
28 days ago
Labels
No labels applied.

~tsileo 28 days ago

Hey, I will take a look at this, thanks for reporting!

Register here or Log in to comment, or comment via email.