Personal projects go here
CTO at https://scie.nz , engineer at Netflix
At this point the only missing thing is support for
applies_to. The config parser currently treats it as a no-op.
Cases to support:
All set: The
/twitchendpoint is automatically disabled if
TWITCH_CLIENT_*are not provided. If only one of the two is provided, it's treated as an error. Meanwhile the twitch credentials are treated as required for
force-rss twitch <account>.
Done, ended up just sticking with manual since clap doesn't seem to have great support for what I was trying to do with positional args
The http scraper in particular risks an amplification attack if the content being fetched isn't cached. Thinking that a small HTTP GET to
force-rssresults in potentially large fetch from
force-rssagainst the target URL.
The twitch side of things should already have caching so hopefully we can just reuse that logic for html
There's some prior art in miniflux for example. But it's a bit different because it's meant for scraping a single page, whereas in our case we want to scrape the list of articles.
If the user doesn't specify
TWITCH_CLIENT_*then disable twitch client mode and only support proxying http sites.
The current arg syntax is just
force-rss [twitch-username]. It'd make sense to allow more options than that - so lets switch to some proper arg handling via
clapor similar. See sample
Thinking the end result would support three modes:
- http proxy:
- twitch fetch:
force-rss twitch <username>(modified from current username mode)
- http fetch:
force-rss http <https://example.com/listpage>(new)