~tsileo/microblog.pub#162: Excessive memory use for inv processes (2G virtual)

Hello:

I am running 9c8693ea555d6 in a docker container. The inv processes uses a lot of swap. After 24 hours I have seen 1G+ per inv process. When the container is started:

USER PID PR NI VIRT RES S %CPU %MEM TIME+ P COMMAND me 2792 20 0 204.5m 105.5m S 0.3 2.8 0:14.55 1 inv me 2793 20 0 200.1m 102.0m S 0.3 2.7 0:07.06 1 inv

memory use is reasonable. The RES stays roughly the same but VIRT balloons. I have seen it as high as 2G. Given that my system only has 4G memory and 4G swap things slow down and the system seems to hang/crash shortly after if I can't restart the container.

I have rebooted my system to enable cgroup swap management. I have also changed the docker compose file to max out memory use at 1G. But this is a band-aid at best. Any idea why inv is usuing so much memory?

Thanks.

#-- -- rouilj John Rouillard

My employers don't acknowledge my existence much less my opinions.

Status: REPORTED
Submitter: John P. Rouillard
Assigned to: No-one
Submitted: 4 months ago
Updated: 4 months ago
Labels: No labels applied.

~nigini 4 months ago

Hey, I have just experienced a similar problem. Although, in my case -- a tiny machine -- my vCPU was eaten by these INV processes. When I looked into the logs, it was covered in failed (timed-out) requests for federated user keys. My working hypothesis is that it happened after I got a couple of new follow(er)s... Do you remember having changes in that regard as well, John?

I think this is related to some Federation protocol that is running havoc. That would not be the first time I heard of these cases. I will keep monitoring and investigating...

John P. Rouillard 4 months ago · edit

Hello Nigini:

In message D2ZZTOXCUBU7.2990X12CKLM20@meta.sr.ht, "~nigini" writes:

Hey, I have just experienced a similar problem.[...] When I looked into the logs, it was covered in failed (timed-out) requests for federated user keys. My working hypothesis is that it happened after I got a couple of new follow(er)s... Do you remember having changes in that regard as well, John?

I have 0 followers shown in microblog. So I don't think gaining followers is showing anything.

docker logs microblogpub doesn't show any errors.

So I think it's a different issue.

I think this is related to some Federation protocol that is running havoc. That would not be the first time I heard of these cases. I will keep monitoring and investigating...

Even if it's not my issue, tracking down a spamming federation protocol would be useful.

#-- -- rouilj John Rouillard

My employers don't acknowledge my existence much less my opinions.