~cloutier/bird.makeup#11: 
So many 429 errors disk fills up

The changes from the past week or so generate a very high number of 429 errors, which pretty quickly fills up my disk. To resolve I have to stop and remove the container. Error examples below:

fail: BirdsiteLive.Pipeline.Processors.SubTasks.RetrieveTweetsProcessor[0]
      Object reference not set to an instance of an object.
fail: BirdsiteLive.Twitter.TwitterTweetsService[0]
      Error retrieving timeline of [redacted]; refreshing client
      System.Net.Http.HttpRequestException: Response status code does not indicate success: 429 (Too Many Requests).
         at System.Net.Http.HttpResponseMessage.EnsureSuccessStatusCode()
         at BirdsiteLive.Twitter.TwitterTweetsService.GetTimelineAsync(String username, Int64 fromTweetId) in /home/build/bird.makeup/src/BirdsiteLive.Twitter/TwitterTweetsService.cs:line 111
fail: BirdsiteLive.Pipeline.Processors.SubTasks.RetrieveTweetsProcessor[0]
      Object reference not set to an instance of an object.
fail: BirdsiteLive.Twitter.TwitterTweetsService[0]
      Error retrieving timeline of [redacted]; refreshing client
      System.Net.Http.HttpRequestException: Response status code does not indicate success: 429 (Too Many Requests).
         at System.Net.Http.HttpResponseMessage.EnsureSuccessStatusCode()
         at BirdsiteLive.Twitter.TwitterTweetsService.GetTimelineAsync(String username, Int64 fromTweetId) in /home/build/bird.makeup/src/BirdsiteLive.Twitter/TwitterTweetsService.cs:line 111
fail: BirdsiteLive.Pipeline.Processors.SubTasks.RetrieveTweetsProcessor[0]
      Object reference not set to an instance of an object.
fail: BirdsiteLive.Twitter.TwitterTweetsService[0]
      Error retrieving timeline of [redacted]; refreshing client
      System.Net.Http.HttpRequestException: Response status code does not indicate success: 429 (Too Many Requests).
         at System.Net.Http.HttpResponseMessage.EnsureSuccessStatusCode()
         at BirdsiteLive.Twitter.TwitterTweetsService.GetTimelineAsync(String username, Int64 fromTweetId) in /home/build/bird.makeup/src/BirdsiteLive.Twitter/TwitterTweetsService.cs:line 111
fail: BirdsiteLive.Pipeline.Processors.SubTasks.RetrieveTweetsProcessor[0]
      Object reference not set to an instance of an object.
fail: BirdsiteLive.Twitter.TwitterTweetsService[0]
      Error retrieving timeline of [redacted]; refreshing client
      System.Net.Http.HttpRequestException: Response status code does not indicate success: 429 (Too Many Requests).
         at System.Net.Http.HttpResponseMessage.EnsureSuccessStatusCode()
         at BirdsiteLive.Twitter.TwitterTweetsService.GetTimelineAsync(String username, Int64 fromTweetId) in /home/build/bird.makeup/src/BirdsiteLive.Twitter/TwitterTweetsService.cs:line 111
fail: BirdsiteLive.Pipeline.Processors.SubTasks.RetrieveTweetsProcessor[0]
      Object reference not set to an instance of an object.
fail: BirdsiteLive.Twitter.TwitterTweetsService[0]
      Error retrieving timeline of [redacted]; refreshing client
      System.Net.Http.HttpRequestException: Response status code does not indicate success: 429 (Too Many Requests).
         at System.Net.Http.HttpResponseMessage.EnsureSuccessStatusCode()
         at BirdsiteLive.Twitter.TwitterTweetsService.GetTimelineAsync(String username, Int64 fromTweetId) in /home/build/bird.makeup/src/BirdsiteLive.Twitter/TwitterTweetsService.cs:line 111
fail: BirdsiteLive.Pipeline.Processors.SubTasks.RetrieveTweetsProcessor[0]
      Object reference not set to an instance of an object.
fail: BirdsiteLive.Twitter.TwitterTweetsService[0]
      Error retrieving timeline of [redacted]; refreshing client
      System.Net.Http.HttpRequestException: Response status code does not indicate success: 429 (Too Many Requests).
         at System.Net.Http.HttpResponseMessage.EnsureSuccessStatusCode()
         at BirdsiteLive.Twitter.TwitterTweetsService.GetTimelineAsync(String username, Int64 fromTweetId) in /home/build/bird.makeup/src/BirdsiteLive.Twitter/TwitterTweetsService.cs:line 111
fail: BirdsiteLive.Twitter.TwitterTweetsService[0]
      Error retrieving timeline of [redacted]; refreshing client
      System.Net.Http.HttpRequestException: Response status code does not indicate success: 429 (Too Many Requests).
         at System.Net.Http.HttpResponseMessage.EnsureSuccessStatusCode()
         at BirdsiteLive.Twitter.TwitterTweetsService.GetTimelineAsync(String username, Int64 fromTweetId) in /home/build/bird.makeup/src/BirdsiteLive.Twitter/TwitterTweetsService.cs:line 111
fail: BirdsiteLive.Pipeline.Processors.SubTasks.RetrieveTweetsProcessor[0]
      Object reference not set to an instance of an object.
fail: BirdsiteLive.Pipeline.Processors.SubTasks.RetrieveTweetsProcessor[0]
      Object reference not set to an instance of an object.
fail: BirdsiteLive.Twitter.TwitterTweetsService[0]
      Error retrieving timeline of [redacted]; refreshing client
      System.Net.Http.HttpRequestException: Response status code does not indicate success: 429 (Too Many Requests).
         at System.Net.Http.HttpResponseMessage.EnsureSuccessStatusCode()
         at BirdsiteLive.Twitter.TwitterTweetsService.GetTimelineAsync(String username, Int64 fromTweetId) in /home/build/bird.makeup/src/BirdsiteLive.Twitter/TwitterTweetsService.cs:line 111
fail: BirdsiteLive.Pipeline.Processors.SubTasks.RetrieveTweetsProcessor[0]
      Object reference not set to an instance of an object.
fail: BirdsiteLive.Twitter.TwitterTweetsService[0]
      Error retrieving timeline of [redacted]; refreshing client
      System.Net.Http.HttpRequestException: Response status code does not indicate success: 429 (Too Many Requests).
         at System.Net.Http.HttpResponseMessage.EnsureSuccessStatusCode()
         at BirdsiteLive.Twitter.TwitterTweetsService.GetTimelineAsync(String username, Int64 fromTweetId) in /home/build/bird.makeup/src/BirdsiteLive.Twitter/TwitterTweetsService.cs:line 111
fail: BirdsiteLive.Pipeline.Processors.SubTasks.RetrieveTweetsProcessor[0]
      Object reference not set to an instance of an object.
fail: BirdsiteLive.Twitter.TwitterTweetsService[0]
      Error retrieving timeline of [redacted]; refreshing client
      System.Net.Http.HttpRequestException: Response status code does not indicate success: 429 (Too Many Requests).
         at System.Net.Http.HttpResponseMessage.EnsureSuccessStatusCode()
         at BirdsiteLive.Twitter.TwitterTweetsService.GetTimelineAsync(String username, Int64 fromTweetId) in /home/build/bird.makeup/src/BirdsiteLive.Twitter/TwitterTweetsService.cs:line 111
fail: BirdsiteLive.Twitter.TwitterTweetsService[0]
      Error retrieving timeline of [redacted]; refreshing client
      System.Net.Http.HttpRequestException: Response status code does not indicate success: 429 (Too Many Requests).
         at System.Net.Http.HttpResponseMessage.EnsureSuccessStatusCode()
         at BirdsiteLive.Twitter.TwitterTweetsService.GetTimelineAsync(String username, Int64 fromTweetId) in /home/build/bird.makeup/src/BirdsiteLive.Twitter/TwitterTweetsService.cs:line 111
fail: BirdsiteLive.Twitter.TwitterTweetsService[0]
      Error retrieving timeline of [redacted]; refreshing client
      System.Net.Http.HttpRequestException: Response status code does not indicate success: 429 (Too Many Requests).
         at System.Net.Http.HttpResponseMessage.EnsureSuccessStatusCode()
         at BirdsiteLive.Twitter.TwitterTweetsService.GetTimelineAsync(String username, Int64 fromTweetId) in /home/build/bird.makeup/src/BirdsiteLive.Twitter/TwitterTweetsService.cs:line 111
fail: BirdsiteLive.Pipeline.Processors.SubTasks.RetrieveTweetsProcessor[0]
      Object reference not set to an instance of an object.
fail: BirdsiteLive.Pipeline.Processors.SubTasks.RetrieveTweetsProcessor[0]
      Object reference not set to an instance of an object.
fail: BirdsiteLive.Pipeline.Processors.SubTasks.RetrieveTweetsProcessor[0]
      Object reference not set to an instance of an object.
fail: BirdsiteLive.Twitter.TwitterTweetsService[0]
      Error retrieving timeline of [redacted]; refreshing client
      System.Net.Http.HttpRequestException: Response status code does not indicate success: 429 (Too Many Requests).
         at System.Net.Http.HttpResponseMessage.EnsureSuccessStatusCode()
         at BirdsiteLive.Twitter.TwitterTweetsService.GetTimelineAsync(String username, Int64 fromTweetId) in /home/build/bird.makeup/src/BirdsiteLive.Twitter/TwitterTweetsService.cs:line 111
fail: BirdsiteLive.Pipeline.Processors.SubTasks.RetrieveTweetsProcessor[0]
      Object reference not set to an instance of an object.
fail: BirdsiteLive.Twitter.TwitterTweetsService[0]
      Error retrieving timeline of [redacted]; refreshing client
      System.Net.Http.HttpRequestException: Response status code does not indicate success: 429 (Too Many Requests).
         at System.Net.Http.HttpResponseMessage.EnsureSuccessStatusCode()
         at BirdsiteLive.Twitter.TwitterTweetsService.GetTimelineAsync(String username, Int64 fromTweetId) in /home/build/bird.makeup/src/BirdsiteLive.Twitter/TwitterTweetsService.cs:line 111
fail: BirdsiteLive.Pipeline.Processors.SubTasks.RetrieveTweetsProcessor[0]
      Object reference not set to an instance of an object.
fail: BirdsiteLive.Twitter.TwitterTweetsService[0]
      Error retrieving timeline of [redacted]; refreshing client
      System.Net.Http.HttpRequestException: Response status code does not indicate success: 429 (Too Many Requests).
         at System.Net.Http.HttpResponseMessage.EnsureSuccessStatusCode()
         at BirdsiteLive.Twitter.TwitterTweetsService.GetTimelineAsync(String username, Int64 fromTweetId) in /home/build/bird.makeup/src/BirdsiteLive.Twitter/TwitterTweetsService.cs:line 111
fail: BirdsiteLive.Pipeline.Processors.SubTasks.RetrieveTweetsProcessor[0]
      Object reference not set to an instance of an object.
fail: BirdsiteLive.Twitter.TwitterTweetsService[0]
      Error retrieving timeline of [redacted]; refreshing client
      System.Net.Http.HttpRequestException: Response status code does not indicate success: 429 (Too Many Requests).
         at System.Net.Http.HttpResponseMessage.EnsureSuccessStatusCode()
         at BirdsiteLive.Twitter.TwitterTweetsService.GetTimelineAsync(String username, Int64 fromTweetId) in /home/build/bird.makeup/src/BirdsiteLive.Twitter/TwitterTweetsService.cs:line 111
fail: BirdsiteLive.Pipeline.Processors.SubTasks.RetrieveTweetsProcessor[0]
      Object reference not set to an instance of an object.
fail: BirdsiteLive.Twitter.TwitterTweetsService[0]
      Error retrieving timeline of [redacted]; refreshing client
      System.Net.Http.HttpRequestException: Response status code does not indicate success: 429 (Too Many Requests).
         at System.Net.Http.HttpResponseMessage.EnsureSuccessStatusCode()
         at BirdsiteLive.Twitter.TwitterTweetsService.GetTimelineAsync(String username, Int64 fromTweetId) in /home/build/bird.makeup/src/BirdsiteLive.Twitter/TwitterTweetsService.cs:line 111
fail: BirdsiteLive.Pipeline.Processors.SubTasks.RetrieveTweetsProcessor[0]
      Object reference not set to an instance of an object.
fail: BirdsiteLive.Twitter.TwitterTweetsService[0]
      Error retrieving timeline of [redacted]; refreshing client
      System.Net.Http.HttpRequestException: Response status code does not indicate success: 429 (Too Many Requests).
         at System.Net.Http.HttpResponseMessage.EnsureSuccessStatusCode()
         at BirdsiteLive.Twitter.TwitterTweetsService.GetTimelineAsync(String username, Int64 fromTweetId) in /home/build/bird.makeup/src/BirdsiteLive.Twitter/TwitterTweetsService.cs:line 111
fail: BirdsiteLive.Pipeline.Processors.SubTasks.RetrieveTweetsProcessor[0]
      Object reference not set to an instance of an object.
fail: BirdsiteLive.Twitter.TwitterTweetsService[0]
      Error retrieving timeline of [redacted]; refreshing client
      System.Net.Http.HttpRequestException: Response status code does not indicate success: 429 (Too Many Requests).
         at System.Net.Http.HttpResponseMessage.EnsureSuccessStatusCode()
         at BirdsiteLive.Twitter.TwitterTweetsService.GetTimelineAsync(String username, Int64 fromTweetId) in /home/build/bird.makeup/src/BirdsiteLive.Twitter/TwitterTweetsService.cs:line 111
fail: BirdsiteLive.Pipeline.Processors.SubTasks.RetrieveTweetsProcessor[0]
      Object reference not set to an instance of an object.
fail: BirdsiteLive.Twitter.TwitterTweetsService[0]
      Error retrieving timeline of [redacted]; refreshing client
      System.Net.Http.HttpRequestException: Response status code does not indicate success: 429 (Too Many Requests).
         at System.Net.Http.HttpResponseMessage.EnsureSuccessStatusCode()
         at BirdsiteLive.Twitter.TwitterTweetsService.GetTimelineAsync(String username, Int64 fromTweetId) in /home/build/bird.makeup/src/BirdsiteLive.Twitter/TwitterTweetsService.cs:line 111
fail: BirdsiteLive.Pipeline.Processors.SubTasks.RetrieveTweetsProcessor[0]
      Object reference not set to an instance of an object.
Status
REPORTED
Submitter
~eric-pierce
Assigned to
No-one
Submitted
1 year, 10 months ago
Updated
1 year, 8 months ago
Labels
No labels applied.

~eric-pierce 1 year, 10 months ago*

For now I've disabled logging for the BMU container by adding the following to my docker-compose:

    logging:
      driver: "none"

~felixmueller 1 year, 10 months ago

Hi ~eric-pierce, this happens to me too every now and then. I wrote a cronjob that runs every minute and checks for certain error messages in the log. If it detects for example ‘Response status code does not indicate success‘, it just restarts the container and the errors go away. It seems to work fine for the last 2 days now. Hope it helps.

~eric-pierce 1 year, 10 months ago

Thanks ~felixmueller - it seems to happen pretty much immediately for me, though it's tough to tell if the container "recovers" and still is able to consistently pull tweets or not. I do get some tweets sent to my mastodon instance but not sure if I'm missing any.

It looks like this is some rate limiting being done by twitter - assuming that's a change that'll stick I wonder if we just need some rate limiting controls beyond the two "parallel" parameters available today.

Would you be up to share your cron script/job?

~eric-pierce 1 year, 10 months ago

Noting that https://git.sr.ht/~cloutier/bird.makeup/commit/4d3eb30feab1f73ab59a6ec51e49e702cf4584c7 seems to have resolved most of the 429 errors, and by limiting the max logs the container can store this looks to be mostly resolved

~davinkevin 1 year, 8 months ago

I have the same problem, with a lot of 429.

I think I already had the modification you've pointed out ~eric-pierce, but because I can't be sure… I've deployed the last version today. Let see if it's better.

I've coded something to automatically restart the pod in k8s every 24h… which is the usual time before getting a lot of errors (extracted from grafana monitoring dashboards).

However, I found one behaviour of the app a bit suspicious… it is contently updating again and again everything… like if it was blocked in an infinite loop, and a lot of time for the same account in the same minute… It could explain why we hit some kind of rate-limiting on twitter side too. I only have ~500 accounts on this instance FYI.

~felixmueller 1 year, 8 months ago

~davinkevin Have you tried setting the config variable Instance:ParallelTwitterRequests to a lower value than the default of 20? I am using the value 1 and I have no issues with 429 on my small personal instance.

~davinkevin 1 year, 8 months ago

~felixmueller, yes and it's better, less 429… but a lot of Error retrieving timeline of XYZ; refreshing client now 😅

~davinkevin 1 year, 8 months ago

After 2 or 3 days, I got in the same situation again, impossible to see tweets in the UI and a lot of error fetching tweets from a lot of accounts.

To solve that, I've automated the restart of the app every 8 hours, it should be enough to keep the log quiet and to make the app working.

~rursache 1 year, 8 months ago

maybe it would help if we could pick another/a specific nitter mirror?

~davinkevin 1 year, 8 months ago

~rursache from the code, I don't think this app calls a nitter instance. It seems to use the graphql endpoint on twitter itself directly.

See TwitterUserService.cs. I didn't find anything about nitter in the code.

Register here or Log in to comment, or comment via email.