We should implement some kind of ring buffer of build caches on the build runners. If the runner has a cache, it uses it; if not, oh well.
This is going to be a great feature. I am currently building a Go project and my dependencies get downloaded for each task. Temporarily there could be some sort of
before_taskskey in the manifest, that is run at the same level as the distro package installation. I could then run
go geton all my dependencies. Thoughts?
Referring to this build: https://builds.sr.ht/~tristan957/job/11249.
My dependencies are fetched once per task. Am I doing something wrong?
That entire page represents a "job" and each log represents a "task"in that job. So far as I can tell the only time your (go) dependencies are fetched is in the build task, right?
I'll pull a different build. One sec. That on efailed due to docker for some reason
Ah, docker hangs on builds.sr.ht for unknown reasons, I'm afraid.
I don't really know what to do about this.
Yeah that is interesting to say the least. And I thought Docker within Docker was funky. I'll follow the issue for fun though
Docker layer caching would be great to have... not sure if that'd be within-scope for this ticket, or if I ought to submit a new one though!
to be clear, this is about those who build docker images on your nodes, not about running builds within docker containers instead of on hosts. you could cache the layers on a project level so that they subsequently use less of the platform's resources on average and result in faster build times for the end user.
I think a generic cache capability, like "cache node_modules", could cache docker layers (somewhere under
/var/lib/docker) just as well. It's nothing like https://circleci.com/docs/docker-layer-caching, where the cached layers are directly network-mounted onto your node, but I also don't think the service should specialize for docker.
But just saying, I would be really interested in caching capabilities. Many of my pipelines pull assets from S3 because for obvious reasons I don't want 1-2 gigabytes of images in my repo. We can save a lot of bandwidth and time with caching. That said, I would be interested to help implement this should there be a need.
I can probably make something based on build artifact. A task upload "build cache" as build artifact, and later tasks query previous build jobs and fetch the "build cache".
I use Rust in my pipeline, and rebuilding from scratch waste CPU time.
I made a very flaky prototype.
notice the "build 0 -> 1" and "1 -> 2" in final task's log
- it only looks for previous build artifact on first page of result
- it uses tag to find which build job is from the same repo. Therefore, it doesn't trigger on resubmitted job. A more robust solution might be to parse the build manifest (YAML).
Unrelated question: how does 'hut' get access token in CI? I don't know how, so I added my own personal token as secret for the above demo.
~iacore: you should revoke your personal access token, it's no longer a secret. See https://man.sr.ht/builds.sr.ht/#keeping-your-secrets-a-secret