Hello,
Attempting to setup a fresh new instance of this using Docker-Compose.
The site loads, but whenever I submit a Twitter handle I get the following error:
2023-07-20 12:57:28 fail: Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware[1] 2023-07-20 12:57:28 An unhandled exception has occurred while executing the request. 2023-07-20 12:57:28 System.NullReferenceException: Object reference not set to an instance of an object. 2023-07-20 12:57:28 at BirdsiteLive.Controllers.UsersController.Index(String id) in /home/build/bird.makeup/src/BirdsiteLive/Controllers/UsersController.cs:line 133 2023-07-20 12:57:28 at Microsoft.AspNetCore.Mvc.Infrastructure.ActionMethodExecutor.TaskOfIActionResultExecutor.Execute(ActionContext actionContext, IActionResultTypeMapper mapper, ObjectMethodExecutor executor, Object controller, Object[] arguments) 2023-07-20 12:57:28 at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.g__Awaited|12_0(ControllerActionInvoker invoker, ValueTask`1 actionResultValueTask) 2023-07-20 12:57:28 at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.g__Awaited|10_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted) 2023-07-20 12:57:28 at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Rethrow(ActionExecutedContextSealed context) 2023-07-20 12:57:28 at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted) 2023-07-20 12:57:28 at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.g__Awaited|13_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted) 2023-07-20 12:57:28 at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.g__Awaited|25_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted) 2023-07-20 12:57:28 at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Rethrow(ResourceExecutedContextSealed context) 2023-07-20 12:57:28 at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted) 2023-07-20 12:57:28 at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.g__Awaited|20_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted) 2023-07-20 12:57:28 at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.g__Awaited|17_0(ResourceInvoker invoker, Task task, IDisposable scope) 2023-07-20 12:57:28 at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.g__Awaited|17_0(ResourceInvoker invoker, Task task, IDisposable scope) 2023-07-20 12:57:28 at Microsoft.AspNetCore.Routing.EndpointMiddleware.g__AwaitRequestTask|6_0(Endpoint endpoint, Task requestTask, ILogger logger) 2023-07-20 12:57:28 at Microsoft.AspNetCore.Authorization.AuthorizationMiddleware.Invoke(HttpContext context) 2023-07-20 12:57:28 at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddlewareImpl.g__Awaited|8_0(ExceptionHandlerMiddlewareImpl middleware, HttpContext context, Task task)
~peanutlasko would you paste the birdmakeup part of your docker-compose.yml here as well? Also, has anything populated into the databases you set up?
server:
image: cloutier/bird.makeup:latest restart: always hostname: birdmakeup.local container_name: ${COMPOSE_PROJECT_NAME}_birdmakeup environment: - Instance:Domain=birdmakeup.$DOMAINNAME2 - Instance:Name=birdmakeup.$DOMAINNAME2 - Instance:AdminEmail=$EMAIL - Instance:ResolveMentionsInProfiles=true - Instance:ParallelTwitterRequests=20 - Instance:ParallelFediverseRequests=20 #- Instance:PublishReplies=true - Db:Type=postgres - Db:Host=birdmakeupdb.local - Db:Name=birdmakeupdb - Db:User=$DBUSERNAME - Db:Password=$DBPASSWORD #- Twitter:ConsumerKey=$APIKEY #- Twitter:ConsumerSecret=$APIKEYSECRET - Moderation:FollowersWhiteListing=${WHITELIST} #- Logging:insights - PUID=$PUID - PGID=$PGID - TZ=$TZ networks: - macdocker_proxy1 - birdsitelivenetwork # volumes: # - type: bind # source: $HOME_DIR/$COMPOSE_PROJECT_NAME/birdmakeup/key.json # target: /app/key.json # #ports: # - "5000:80" depends_on: - db labels: - 'com.centurylinklabs.watchtower.enable=true' - 'traefik.docker.network=macdocker_proxy1' #enabled - 'traefik.enable=true' ## TCP Routers - 'traefik.http.routers.birdsitelive-rtr.entrypoints=https' - 'traefik.http.routers.birdsitelive-rtr.rule=Host(`birdmakeup.$DOMAINNAME2`)' - 'traefik.http.routers.birdsitelive-rtr.tls=true' #middleware - 'traefik.http.routers.birdsitelive-rtr.middlewares=chain-no-auth@file' #- 'traefik.http.routers.birdsitelive-rtr.middlewares=chain-basic-auth@file' #- 'traefik.http.routers.birdsitelive-rtr.middlewares=chain-oauth@file' #defines router and port - 'traefik.http.routers.birdsitelive-rtr.service=birdsitelive-svc' - 'traefik.http.services.birdsitelive-svc.loadbalancer.server.port=${BIRDSITE_PORT}'
db:
image: postgres:15 restart: always hostname: birdmakeupdb.local container_name: ${COMPOSE_PROJECT_NAME}_birdmakeupdb environment: - POSTGRES_USER=$DBUSERNAME - POSTGRES_PASSWORD=$DBPASSWORD - POSTGRES_DB=birdmakeupdb networks: - birdsitelivenetwork volumes: - postgresbirdmakeup:/var/lib/postgresql/data labels: - 'com.centurylinklabs.watchtower.enable=true' - 'com.centurylinklabs.watchtower.monitor-only=true'
edit: nvm got it
~peanutlasko it looks like the error is being thrown because birdmakeup is trying to reference a user that doesn't exist:
2023-07-20 12:57:28 at BirdsiteLive.Controllers.UsersController.Index(String id) in /home/build/bird.makeup/src/BirdsiteLive/Controllers/UsersController.cs:line 133
Do you have data populated in the twitter_users table in your postgres database? Here's an example of what that data should look like: https://todo.sr.ht/~cloutier/bird.makeup/4#event-234718. If that table is empty, populating it with data for a single user to follow may resolve this.
Do you have data populated in the twitter_users table in your postgres database? Here's an example of what that data should look like: https://todo.sr.ht/~cloutier/bird.makeup/4#event-234718. If that table is empty, populating it with data for a single user to follow may resolve this.
It looks like ~cloutier pushed some updates to address the missing data in tables, but I haven't heard any news as to how that works. if you're missing data in those tables the fix may not have resolved it.
One other thing to try is to change the image you're using from the one on dockerhub, to the one ~rursache hosted on github. The github image uses version 7 of aspnet and the dotnet SDK instead of version 6 used in the dockerhub version. You'd just need to update the image piece of your docker-compose to be:
image: ghcr.io/rursache/bird.makeup:latest
instead of
image: cloutier/bird.makeup:latest
Note: don't update to the github hosted image if you run on an arm64 device like a raspberry pi as that image only supports x86 architecture. I did open a patchset for the changes in ~rursache's Dockerfile but it doesn't look like ~cloutier has looked or merged it.
~peanutlasko instructions of how to populate the empty table are also available here: https://github.com/rursache/bird.makeup/issues/3
if you solved your issue leave explicit details on how you did it so others will benefit too.
Thanks ~rusache, currently stuck on how to get pgadmin working in a docker container. Frustrating to say the least
docker run -d \ --name pgadmin \ --restart unless-stopped \ -p 2210:80 \ -e 'PGADMIN_DEFAULT_EMAIL=YOUR@EMAIL.TLD' \ -e 'PGADMIN_DEFAULT_PASSWORD=UNIQUEPASS' \ -v /home/USER/.pgadmin:/var/lib/pgadmin \ dpage/pgadmin4:latest
Eric, I tried using the github version you posted on a clean DB and doesn't appear to work either.
~peanutlasko you can also query from the command line without pgadmin https://stackoverflow.com/questions/71104319/postgresql-select-from-table-inside-docker-container-bash if getting that image up and running is tough. No UI so you have to use SQL directly but may be faster/easier.
You'd run the following after replacing the parameters with the values you're using in your .env file
sudo docker exec -it ${COMPOSE_PROJECT_NAME}_birdmakeupdb psql -U $DBUSERNAME birdmakeupdb
Then when that gives you a SQL prompt just type in
select * from twitter_users;
and see what it returns. If nothing comes back and you need to insert a row you'd have to use a SQL insert statement, which I can help with if it comes to that.
So I got the pg4admin running, I've ran the queries and both my twitter_users and followers tables are empty. I've entered in the data, but as soon as i boot up my birdmakeup container, the tables get wiped and im back at 0 again with the same error as before. This is beyond frustrating.
Also noted my followers table doesnt have a column for followingssyncstatus nor does my twitter_users have a asttweetsynchronizedforallfollowersid column
~peanutlasko yes fully feel the frustration, it seems like the setup may be broken. I'm going to try setting up a fresh install and see if I can re-create what you're seeing, and hopefully solve.
~peanutlasko Ok I was able to re-create the issues you're seeing on a fresh install, and got everything working. The problem is resolved by populating the data below in the two tables (followers, twitter_users).
May be worth doing a fresh install rather than trying to resolve your current setup, so stop and remove the container, delete the database and birdmakeup user from postgres using pgadmin.
After you've stopped and remove the bmu container, and removed the db/user from postgres, do the following:
- Create BMU user in postgres
- Give superuser privileges
- Create the birdmakeup database, and set it to be owned by your birdmakeup user
Start up the container, you should get a working UI at your birdmakeup domain
In pgadmin, paste the following values into the two relevant tables:
followers:
id followings acct host inboxroute sharedinboxroute actorid postingerrorcount 1 {1} peanutlasko mas.to /users/peanutlasko/inbox /inbox https://mas.to/users/peanutlasko 0
twitter_users - NOTE that the lastsync timestamp has spaces in it (2023-07-21 14:54:07.16) and the following 0 is for the fetchingerrorcount. Leave fediverseaccount NULL
id acct lasttweetpostedid lastsync fetchingerrorcount twitteruserid fediverseaccount 1 npr 1646138106595139584 2023-07-21 14:54:07.16 0 5392522
Then restart the bmu container and it should be working
better now! Thanks a lot for all this help Eric. So it appears to be working somewhat - I can log into my Mastodon instance and follow users from twitter. However, if I visit the actual birdmakeup..com that I self host, then I enter in any Twitter handle - @MKBHD for example, I still get the error:
2023-07-21 15:40:02 fail: Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware[1] 2023-07-21 15:40:02 An unhandled exception has occurred while executing the request. 2023-07-21 15:40:02 System.NullReferenceException: Object reference not set to an instance of an object. 2023-07-21 15:40:02 at BirdsiteLive.Controllers.UsersController.Index(String id) in /src/BirdsiteLive/Controllers/UsersController.cs:line 133 2023-07-21 15:40:02 at Microsoft.AspNetCore.Mvc.Infrastructure.ActionMethodExecutor.TaskOfIActionResultExecutor.Execute(ActionContext actionContext, IActionResultTypeMapper mapper, ObjectMethodExecutor executor, Object controller, Object[] arguments) 2023-07-21 15:40:02 at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.g__Awaited|12_0(ControllerActionInvoker invoker, ValueTask`1 actionResultValueTask) 2023-07-21 15:40:02 at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.g__Awaited|10_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted) 2023-07-21 15:40:02 at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Rethrow(ActionExecutedContextSealed context) 2023-07-21 15:40:02 at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted) 2023-07-21 15:40:02 at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.g__Awaited|13_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted) 2023-07-21 15:40:02 at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.g__Awaited|25_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted) 2023-07-21 15:40:02 at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Rethrow(ResourceExecutedContextSealed context) 2023-07-21 15:40:02 at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted) 2023-07-21 15:40:02 at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.g__Awaited|20_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted) 2023-07-21 15:40:02 at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.g__Awaited|17_0(ResourceInvoker invoker, Task task, IDisposable scope) 2023-07-21 15:40:02 at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.g__Awaited|17_0(ResourceInvoker invoker, Task task, IDisposable scope) 2023-07-21 15:40:02 at Microsoft.AspNetCore.Routing.EndpointMiddleware.g__AwaitRequestTask|6_0(Endpoint endpoint, Task requestTask, ILogger logger) 2023-07-21 15:40:02 at Microsoft.AspNetCore.Authorization.AuthorizationMiddleware.Invoke(HttpContext context) 2023-07-21 15:40:02 at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddlewareImpl.g__Awaited|8_0(ExceptionHandlerMiddlewareImpl middleware, HttpContext context, Task task)
~peanutlasko weird I was able to look up the account I seeded in the DB using the bmu interface without that error - but frankly there isn't much value in using the interface. You can just look up users in the mastodon UI and follow them from there, so long as that's working you should be in good shape.
Right, I can find NPR just fine too. I was under the impression though you should be able to search ANY twitter handle (even if it's not in the database yet), correct?
In Birdsitelive that was the case. Then you could get the Mastodon handle, copy/paste it into Mastodon.
Wait, I think i understand now - it's trying to find someone who follows this handle already (@MKBHD@myselfhosteddomain.com) and fails (because nobody does)
If I add it via Mastodon FIRST, then search on Birdmakeup, it's working. So the error only happens whenever I'm searching and haven't added them prior in Mastodon. Makes sense actually.
~peanutlasko gotcha, glad it sounds like it's working as intended now - clearly the setup issue raised wasn't addressed by ~cloutier for new installs, so unfortunately the manual insert of data is needed, and the table structures have changed since I put together the sample values a couple of months ago
Now my next question is, lol - how can I mass add all my twitter followers into this. I assume with a DB insert ?
~peanutlasko as far as I know the only way is to follow each account from your mastodon instance, as both BMU and your mastodon account have to recognize and register the follow
what i did to batch follow around 250 accounts was to create a txt file with each twitter username on a separate line like:
@username@birdmakeup.instance.tld @username2@birdmakeup.instance.tld ….
and then go to mastodon account settings and use the “import following accounts” option + selecting the file.
took around 20mins but it all went great.
~eric-pierce thanks for providing updated initial db imports, i've also updated my "guide" so new users will struggle less: https://github.com/rursache/bird.makeup/issues/3
Hi, I'm having the same issue, I've imported the two lines into the database but even so, it doesn't work. Even the web interface finds the Twitter user, Mastodon doesn't.
birdmakeup=# SELECT * FROM followers; SELECT * FROM twitter_users; id | followings | acct | host | inboxroute | sharedinboxroute | actorid | postingerrorcount ----+------------+----------+----------+-----------------------+------------------+---------------------------------+------------------- 3 | {1} | retiolus | mamot.fr | /users/retiolus/inbox | /inbox | https://mamot.fr/users/retiolus | 0 (1 row) id | acct | lasttweetpostedid | lastsync | fetchingerrorcount | twitteruserid | fediverseaccount | statusescount | extradata ----+------+---------------------+------------------------+--------------------+---------------+------------------+---------------+----------- 3 | npr | 1646138106595139584 | 2023-07-21 14:54:07.16 | 0 | 5392522 | | | (1 row)
~retiolus the version hosted here on sr.ht hasn't worked for me for awhile, I've been using the version hosted on github here: https://github.com/rursache/bird.makeup which has some modifications and is slower to roll out changes, but seemingly is more stable. Thanks to ~rursache for working on this
One note that the version on github uses database version 3.1 not 3.2 as the sr.ht version does. To get the github version working you only need to edit the entry in the db_version table to be 3, 1 instead of 3, 2, and then drop the "extradata" columns from the twitterusers and instagramusers tables.