We should implement some kind of ring buffer of build caches on the build runners. If the runner has a cache, it uses it; if not, oh well.
This is going to be a great feature. I am currently building a Go project and my dependencies get downloaded for each task. Temporarily there could be some sort of
before_taskskey in the manifest, that is run at the same level as the distro package installation. I could then rungo geton all my dependencies. Thoughts?
What advantage would that provide over just running
go getin a normal task?
Referring to this build: https://builds.sr.ht/~tristan957/job/11249.
My dependencies are fetched once per task. Am I doing something wrong?
That entire page represents a "job" and each log represents a "task"in that job. So far as I can tell the only time your (go) dependencies are fetched is in the build task, right?
and the test task
I'll pull a different build. One sec. That on efailed due to docker for some reason
Wait no. I am stupid. Never mind my comments :)
Ah, docker hangs on builds.sr.ht for unknown reasons, I'm afraid.
https://github.com/moby/moby/issues/37575
I don't really know what to do about this.
Yeah that is interesting to say the least. And I thought Docker within Docker was funky. I'll follow the issue for fun though
What's the sr.ht way to say "I am also interested in this feature" ?
There isn't, by design. You could mention it in IRC, though.
Docker layer caching would be great to have... not sure if that'd be within-scope for this ticket, or if I ought to submit a new one though!
Neither. We don't use Docker and don't intend to start.
to be clear, this is about those who build docker images on your nodes, not about running builds within docker containers instead of on hosts. you could cache the layers on a project level so that they subsequently use less of the platform's resources on average and result in faster build times for the end user.
I think a generic cache capability, like "cache node_modules", could cache docker layers (somewhere under
/var/lib/docker) just as well. It's nothing like https://circleci.com/docs/docker-layer-caching, where the cached layers are directly network-mounted onto your node, but I also don't think the service should specialize for docker.But just saying, I would be really interested in caching capabilities. Many of my pipelines pull assets from S3 because for obvious reasons I don't want 1-2 gigabytes of images in my repo. We can save a lot of bandwidth and time with caching. That said, I would be interested to help implement this should there be a need.
I can probably make something based on build artifact. A task upload "build cache" as build artifact, and later tasks query previous build jobs and fetch the "build cache".
I use Rust in my pipeline, and rebuilding from scratch waste CPU time.
I made a very flaky prototype.
two builds: https://builds.sr.ht/~iacore/job/826339 https://builds.sr.ht/~iacore/job/826341
notice the "build 0 -> 1" and "1 -> 2" in final task's log
caveats:
- it only looks for previous build artifact on first page of result
- it uses tag to find which build job is from the same repo. Therefore, it doesn't trigger on resubmitted job. A more robust solution might be to parse the build manifest (YAML).
Unrelated question: how does 'hut' get access token in CI? I don't know how, so I added my own personal token as secret for the above demo.
~iacore: you should revoke your personal access token, it's no longer a secret. See https://man.sr.ht/builds.sr.ht/#keeping-your-secrets-a-secret
~adnano Thanks. I revoked it yesterday. It only needs JOBS:R anyway.