Build Caching Explained: Speeding Up CI Without Breaking Reproducibility

Build Caching Explained is the set of rules that let CI reuse work and still ship the same bits. When the cache key is wrong, you either wait too long or you ship stale code. Therefore you should treat the cache as a speed tool with a correctness budget.

What Build Caching Is and Why It Fails in Production

A build cache stores a result and a key. The next build with the same key reuses the result. If the key ignores an input that changed, you reuse the wrong result. That is a silent bug, not a speed win.

In my experience, caches fail in production for a few plain reasons. First, the key is only the branch name, so Tuesday’s deps serve Wednesday’s code. Then, a local cache on one runner never helps the next runner, so times swing at random. Also, a poisoned entry from a flaky compile gets reused until someone deletes the world.

A common mistake I have seen is a green main build that baked an old file. The source changed, but the cache key did not include that path. Because the tests also used the cached binary, they passed on old behavior.

After the deploy, users saw last week’s bug under today’s commit. The pipeline had cached away the change.

Another failure is non determinism. Two builds with the same inputs produce different bytes because of a timestamp or a random temp path. When the cache stores one of them, later builds flip between hit and miss.

Still, both are called reproducible. So remove clocks and absolute paths from the outputs you cache.

You also fail when the cache holds secrets. A layer or an object file can embed a token from the build env. The cache then shares that token with every job that can read the key.

Scope cache access like you scope artifact access. Do not use a world readable bucket for a private build.

Architecture and Implementation

Separate three caches. First, the dependency download cache. Next, the compile or test cache. Then, the image layer cache.

Finally, decide which of these may be remote and shared. The Docker build cache docs cover layers. The same key idea applies to language builds.

Keys that match inputs

A key should name every input that can change the output. That includes the lockfile, the toolchain version, the OS, and the relevant source. If any of those change, you want a miss, not a hit.

Put the volatile part last if your tool uses prefix restore. You can restore a dep cache from an older key and then rebuild the app. You should not restore an app binary from a key that omitted the source.

The GitHub Actions cache docs show key and restore key. Use restore keys for deps. Use exact keys for compiled outputs.

Your dependency lockfiles belong in the key. A cache keyed only on the manifest will mix resolved versions. Then two builds of the same commit can still differ if the resolve was not frozen. Pin the lockfile and hash it.

Local, remote, and shared caches

A local cache is fast and private to one machine. It does nothing for a fresh CI runner. A remote cache lets every runner reuse safe results.

The Bazel remote caching docs describe a strict form of this. You can get a weaker form from CI cache actions.

Shared caches need access control and a size cap. One team should not read another team’s objects if the outputs can hold source. Also set a time to live so a bad entry dies. Infinite retention turns a one hour bug into a permanent one.

What you must not cache

Do not cache a step that talks to prod or that mints a secret. Do not cache a test that depends on the clock unless you froze the clock. Do not cache the final prod image under a key that ignores the git commit. Your CI/CD pipelines should build the release artifact from inputs you can name, then store it as an artifact, not as a fuzzy cache hit.

Cache the expensive middle. Keep the ends honest. Download deps from a cache.

Compile with a cache if the tool is deterministic. Tag and push the image by digest every time so the registry, not the cache, is the source of what you deploy.

How to Roll a Cache Out

Start with the dependency cache. It is the usual win and the smallest correctness risk. Measure cold and warm times before you add more. If you cannot explain a hit, do not share that cache across branches yet.

Add a kill switch. A single variable should disable cache read so you can prove a bug is cache related. When an incident might be stale bits, rebuild with the cache off.

Then fix the key. Do not only delete one entry and hope.

1. Hash the lockfile, toolchain, and OS into the dep key.
2. Restore deps on a prefix. Require an exact key for binaries.
3. Keep secrets and prod calls out of cached steps.
4. Cap size and set a time to live.
5. Provide a one switch rebuild that ignores the cache.

Trade-offs and Comparison

A miss is safe and slow. A bad hit is fast and wrong. You want hits on work that is pure, and misses when any input changed. If you cannot make the step pure, do not cache it.

Remote caches cost storage and network. They pay off when runners are ephemeral and cold starts dominate. A sticky runner with a warm disk can be cheaper for a small team. It will not scale when you add concurrent builds.

Choose a dep cache first for almost every CI system. Add a compile cache when you trust the key and the tool. If a cache bug would ship the wrong binary, keep that output out until you can prove hits are exact.

Pitfalls and Failure Modes

Key collisions are the severe bug. Two different inputs hash to one key, or the key never included the input. You deploy stale code with a green build.

Include the full lockfile hash and the source hash for any cached binary. Do not key only on a short prefix of a commit.

Fallback restore keys can over reach. A restore prefix of deps-linux- will load last month’s tree and then fail in a strange way, or worse, succeed with mixed versions. Use a restore key that still includes the lockfile family. If the lockfile changed, take the miss.

Layer order in Docker images is a cache bug factory. COPY of the whole repo before a dep install busts the useful layer, or a wildcard COPY skips a file the key thought it hashed. Follow a stable order. Invalidate when the base digest changes, not only when the tag string changes.

Poisoned entries spread. A runner with a dirty workspace writes a bad object under a good key. Every later hit inherits it.

Isolate the workspace. Prefer content addressed storage so the bytes define the id. If you must overwrite keys, scope them to a branch and a day, not to all of main forever.

Caches hide flaky networks until the day they miss. The build works only when the cache is warm. A cold build cannot download a package that was yanked.

Keep a lockfile and a registry you control. Your cache is not a backup of the internet.

Permissions drift when the cache stores root owned files and the next job runs as a user. The build fails in a way that looks like a source bug. Normalize file owners inside the cached archive. Or do not cache tools that bake absolute users and paths.

A Practical Cache Key

The sketch below keys a dependency cache on OS, toolchain, and the lockfile. It uses a narrower restore key so a lockfile change does not reuse old deps. When you cache a binary, add the source hash and do not use a wide restore.

deps-key: deps-linux-go122-${LOCKFILE_SHA}
deps-restore: deps-linux-go122-
binary-key: bin-linux-go122-${LOCKFILE_SHA}-${SOURCE_SHA}
binary-restore: none

Note what this sketch does not do. It does not key on the branch name alone. It does not restore a binary from a partial match.

If SOURCE_SHA changes, the binary misses on purpose. Also log hit or miss on every job so a sudden miss rate is visible before people blame the compiler.

Performance, Scale, and Cost

The win is minutes of wait and minutes of CI bill. A dep cache often cuts a large share of a cold build, an illustrative production range, when downloads used to dominate. Measure before and after. A cache that saves ten seconds and risks stale binaries is a bad trade.

Storage cost grows with unique keys. If you put a timestamp in the key, you will never hit and you will fill the bucket. If you forget to include the lockfile, you will hit too often.

Watch hit rate and wrong output reports together. A high hit rate can still be a bug.

At scale, the cache service becomes a dependency of every build. When it is slow, CI is slow. When it is down, builds should still succeed on a miss.

Do not fail the job only because the cache is unreachable. Fall back to a full build. Your infrastructure as code should set quotas so one busy repo cannot evict the keys everyone else needs.

We once hit a bottleneck when every branch wrote a full image cache and the eviction policy deleted main’s hot layers. Main builds went cold every afternoon. The fix was to prefer main keys and to cap branch caches. After that, the hot path stayed warm and feature branches took the misses.

Network distance matters. A remote cache across a region can be slower than a rebuild for small artifacts. Put the cache near the runners.

For large artifacts, the cache still wins. Measure bytes and time, not only hit counts.

Plan a flush procedure. You will need to drop a bad key during an incident. Document who can delete, how fast it propagates, and how you rebuild. A cache you cannot flush is a cache you cannot trust under pressure.

Key Takeaways

• Cache results only when the key names every input that matters.
• Use wide restore keys for deps and exact keys for binaries.
• Keep secrets, clocks, and prod calls out of cached steps.
• Make a cold build work when the cache is down or flushed.
• Log hits and misses so a bad key is visible.
• Cap size, set a time to live, and protect the main branch keys.
• Ship from a digest or an artifact, not from a fuzzy cache hit.

FAQ

Should feature branches share the main cache?

They can read a dep cache if the key includes the lockfile and the toolchain. They should not overwrite main’s binary cache with branch outputs. Give branches their own write scope. Let main keep the hot, trusted keys.

How do I know the cache is wrong?

Rebuild the same commit with the cache disabled. If the bytes or the test results change, the key is incomplete or the step is not pure. Fix that before you tune hit rate. A faster wrong build is not a win.

Is a Docker layer cache enough?

It helps image builds and it does not replace a language dep cache inside the build. You often want both. The layer cache must still invalidate when the base digest or the lockfile changes. A tag only key is not enough.

What should I do during an incident if I suspect the cache?

Turn reads off and rebuild from the lockfile and the commit you mean to ship. Compare the digest to the one that is live. If they differ, the cache or a moving tag lied. Flush the bad key after you have a good artifact, not before you can build without it.

Build Caching Explained comes down to a key that matches the inputs and a path that still builds when the cache is cold. Pick one pipeline. Next, key the dep cache on the lockfile and the toolchain, and turn off binary restore. Then time a cold build and a warm build so you know the win before you share the cache widely.

Last updated on 13 September 2026.