~nickbp/kapiti#9: 
Run filter update/parse as a background task

  • (DONE) Run initial updates in background, don't block DNS service if a filter is slow to update
  • Schedule periodic updates to run in the background while the service runs:
    • Local files could update every few minutes, but poll rather than inotify to avoid issues with NFS or similar
    • Remote files could be checked every few hours or more.
Status
RESOLVED FIXED
Submitter
~nickbp
Assigned to
No-one
Submitted
5 months ago
Updated
5 months ago
Labels
Blocker Next

~nickbp 5 months ago

This ties into an idea of restructuring the internals as independent actors/threads, separated from each other by queues. Once that's been done, it should be straightforward to treat filter management/updates as its own actor. Now that #10 is done and we have an idea of baseline performance, we can know if that restructuring leads to any performance issues.

~nickbp 5 months ago

The actor-like structure is already there. For example the cache is currently running as a separate event loop fed off of a request queue. It may be straightforward to approach updating filters and checking filters the same way.

As an aside, we should probably also serve a health/readiness check endpoint on a private HTTP port. For now it would mainly expose when filters have all been successfully applied after initial startup, but it could be expanded into other things like upstream client statuses later.

~nickbp referenced this from #27 5 months ago

~nickbp REPORTED FIXED 5 months ago

This is pretty much there now. At the moment it's just refreshing all files at the same time rather than having a separate timer for local files, but to be honest local file support is probably only going to mean /etc/hosts anyway so I don't think there's much reason to complicate the refresh logic further.

I also implemented tidy teardown of the process upon sigkill/sigint(ctrl+c), so if we're in the middle of a filter update it will wait until the update has concluded before actually exiting.

So at this point there really isn't much left to do here. Additional work may be needed if the list of filters becomes dynamic at runtime, but we can deal with that if/when it becomes an issue.

Register here or Log in to comment, or comment via email.