Implement build caches

We should implement some kind of ring buffer of build caches on the build runners. If the runner has a cache, it uses it; if not, oh well.

Assigned to
4 years ago
16 days ago
No labels applied.

~tristan957 3 years ago

This is going to be a great feature. I am currently building a Go project and my dependencies get downloaded for each task. Temporarily there could be some sort of before_tasks key in the manifest, that is run at the same level as the distro package installation. I could then run go get on all my dependencies. Thoughts?

~sircmpwn 3 years ago

What advantage would that provide over just running go get in a normal task?

~tristan957 3 years ago

Referring to this build: https://builds.sr.ht/~tristan957/job/11249.

My dependencies are fetched once per task. Am I doing something wrong?

~sircmpwn 3 years ago

That entire page represents a "job" and each log represents a "task"in that job. So far as I can tell the only time your (go) dependencies are fetched is in the build task, right?

~tristan957 3 years ago

and the test task

~tristan957 3 years ago

I'll pull a different build. One sec. That on efailed due to docker for some reason

~tristan957 3 years ago

Wait no. I am stupid. Never mind my comments :)

~sircmpwn 3 years ago

Ah, docker hangs on builds.sr.ht for unknown reasons, I'm afraid.


I don't really know what to do about this.

~tristan957 3 years ago

Yeah that is interesting to say the least. And I thought Docker within Docker was funky. I'll follow the issue for fun though

~andrewrk 3 years ago

What's the sr.ht way to say "I am also interested in this feature" ?

~sircmpwn 3 years ago

There isn't, by design. You could mention it in IRC, though.

~mrzv referenced this from #240 2 years ago

~erin 1 year, 2 months ago

Docker layer caching would be great to have... not sure if that'd be within-scope for this ticket, or if I ought to submit a new one though!

~sircmpwn 1 year, 2 months ago

Neither. We don't use Docker and don't intend to start.

~erin 1 year, 2 months ago*

to be clear, this is about those who build docker images on your nodes, not about running builds within docker containers instead of on hosts. you could cache the layers on a project level so that they subsequently use less of the platform's resources on average and result in faster build times for the end user.

~haowenl 16 days ago

I think a generic cache capability, like "cache node_modules", could cache docker layers (somewhere under /var/lib/docker) just as well. It's nothing like https://circleci.com/docs/docker-layer-caching, where the cached layers are directly network-mounted onto your node, but I also don't think the service should specialize for docker.

But just saying, I would be really interested in caching capabilities. Many of my pipelines pull assets from S3 because for obvious reasons I don't want 1-2 gigabytes of images in my repo. We can save a lot of bandwidth and time with caching. That said, I would be interested to help implement this should there be a need.

Register here or Log in to comment, or comment via email.