~ols

sheffield, england

https://ols.wtf

Trackers

~ols/veri

Last active 5 days ago

~ols/ols.wtf

Last active 1 year, 3 months ago

~ols/docs

Last active 2 years ago

#5 Custom user agent 6 days ago

Comment by ~ols on ~ols/veri

Currently the User-Agent is blank. This should be changed so that it contains the details of the project including an abuse and/or contact email address. This should be configurable easily so that not all self-hosted instances reference the parent project: e.g.

veri-[web|gemini|gopher]-scraper operated by <domain> (<email>)

#4 Respect robots.txt 6 days ago

Comment by ~ols on ~ols/veri

The colly library we are using to grab URLs from pages respects robots.txt, but this is passed to go-readability which has no concept of robots.txt. We either need to implement our own checking of the site's robots.txt before running go-readability, or else give the list of links another pass through colly to filter out ones that don't match the site's robots.txt

#2 Write output to database 6 days ago

Comment by ~ols on ~ols/veri

Database technology as yet undecided

#1 Add concurrency 6 days ago

Comment by ~ols on ~ols/veri

I'm looking at this at the moment

#5 Custom user agent 6 days ago

Ticket created by ~ols on ~ols/veri

#4 Respect robots.txt 6 days ago

Ticket created by ~ols on ~ols/veri

#3 Add tier (1st, 2nd, etc) to database for ranking 6 days ago

Ticket created by ~ols on ~ols/veri

#2 Write output to database 6 days ago

Ticket created by ~ols on ~ols/veri

#1 Add concurrency 6 days ago

Ticket created by ~ols on ~ols/veri

#60 OpenBSD 6.6 "failed to open database" sujoctl 6 months ago

Comment by ~ols on ~emersion/soju

Thanks, I just compiled on OpenBSD instead

REPORTED RESOLVED BY_DESIGN