~cloutier/bird.makeup#33: 
Bird Makeup using a ton of download bandwidth, despite not pulling new tweets for most accounts

Been experimenting with birdmakeup for personal use. I'm following 22 accounts, including for example:

https://twitter.com/hosico_cat

But as you can see at

https://fuwafuwa.town/@hosico_cat@bm.fuwafuwa.town

the most recent tweets it's pulled are over a year old. There's been 11 posts since the environment was created on Jan 26 and today but none of them have made it to my instance.

There are a couple of accounts which do seem to pull new tweets in but even then not as many as I'd expect.

I've also observed that the birdmakeup container is pulling abour 320 gigabytes of data from Twitter daily. My Unifi dash is showing that 93% of my daily traffic is from Twitter and the graph shows dips when the birdmakeup container reboots hourly (plus a dip at the end of the graph after I stopped the container to test) - I have a screenshot I can provide.

I'm wondering if it's maybe trying to do a full pull of all tweets for all 22 users, but it keeps getting interrupted and restarts when I restart my container? What should my next steps be here?

Status
REPORTED
Submitter
~rosey
Assigned to
No-one
Submitted
5 months ago
Updated
4 months ago
Labels
No labels applied.

~rosey 5 months ago*

Another thing to add. It's pulling each of those 22 users constantly - for example it's pulling hosico_cat every two seconds. It's checking every account in sequence which I assume is where the high bandwidth is coming from. Is that normal?

2024-02-13T05:18:04.242873453Z info: BirdsiteLive.Pipeline.Processors.SubTasks.RetrieveTweetsProcessor[0] 2024-02-13T05:18:04.242929709Z 21/21 Got 0 tweets from user hosico_cat

2024-02-13T05:18:06.348198821Z info: BirdsiteLive.Pipeline.Processors.SubTasks.RetrieveTweetsProcessor[0] 2024-02-13T05:18:06.348254009Z 21/21 Got 0 tweets from user hosico_cat

2024-02-13T05:18:08.682007468Z info: BirdsiteLive.Pipeline.Processors.SubTasks.RetrieveTweetsProcessor[0] 2024-02-13T05:18:08.682030969Z 21/21 Got 0 tweets from user hosico_cat

2024-02-13T05:18:10.692963142Z info: BirdsiteLive.Pipeline.Processors.SubTasks.RetrieveTweetsProcessor[0] 2024-02-13T05:18:10.692977701Z 21/21 Got 0 tweets from user hosico_cat

2024-02-13T05:18:12.907403247Z info: BirdsiteLive.Pipeline.Processors.SubTasks.RetrieveTweetsProcessor[0] 2024-02-13T05:18:12.907423159Z 21/21 Got 0 tweets from user hosico_cat

~rosey 5 months ago

I've done some more digging and it seems like this might not actually be a bird makeup issue. bm has pulled 91 tweets for hosico_cat, most recently from April 2023. If I open hosico_cat in a private browser window (i.e. not logged into twitter) and scroll back as far as it'll let me, I get 90 tweets AND the most recent tweet I see there is the most recent tweet bm has managed to pull.

An account I observed pulling tweets on bird makeup is also showing recent tweets when I open it in a private window. So I'm guessing the only tweets bm can see are the tweets I can see when I'm logged out.

I know Twitter are making it difficult to view tweets outside the service now (RIP nitter), is what I'm seeing here 'expected' behaviour in the current context? Is there a workaround, for example passing login credentials? I intend for this to be for solely private use so I'm not concerned about rate limiting myself.

~davinkevin 4 months ago

I have the same issue with all accounts being pulled constantly in an infinite loop.

When I work on a local instance, with only one followed account, it's checked 4 times a second in this loop.

in bird.makeup context, with thousands of accounts, it potentially makes senseā€¦ but for a single-user consumer (-ish) mode, a pause could be good for CPU/IO and rate limiting.

wdyt?

Register here or Log in to comment, or comment via email.