By Imran M ·
· 12 min read
Caching in System Design: Patterns, Eviction, and Hit Ratio
Cache-aside vs read-through, write-through vs write-back, LRU eviction, and the hit-ratio math that decides whether a cache pays for itself, part 1 of the Caching in System Design series.
On most request paths, the slowest thing the user waits for is a database: disk, a network round trip, contention with other queries. The cruel part is that the same rows are fetched over and over; traffic is skewed, and a small hot set serves most requests. The fix is not a faster database. It is refusing to repeat work that was already done. That refusal is caching, and it is the highest-leverage move in most performance work, as long as its price, stale data, is paid deliberately.
Caching is the practice of storing the result of an expensive computation or data fetch in a faster layer of storage, close to the reader, so repeated requests are served from the stored copy instead of repeating the work. Every cache answers two questions for each request: do I have this, and can it still be trusted?
Why caching exists
- Latency. Memory access is orders of magnitude faster than a disk-backed, contended database read. A hit serves the request from memory; a miss pays the full price. The budget a cache spends its hits on is described in latency vs throughput.
- Load. The database’s queue is everyone’s queue. Reads absorbed by a cache never join it, which is why a cache is as much a database-defender as a latency tool.
- Cost. The copy is cheap to serve; the original is not. Serving the hot set from a cache moves traffic off the expensive tier.
Caching has a cousin at the network edge: a CDN caches HTTP responses in locations close to users, and that story (TTLs, edges, purge) belongs to it. This series is about the caches inside your infrastructure, between the application and the database.
Where the cache sits
Three positions cover most application caches:
- In-process. A local cache inside the application, a map in memory. Fastest possible, private to one process, lost on restart, invisible to other instances.
- A dedicated cache server. Redis or Memcached, shared by a fleet: one network hop, visible to every instance, surviving any single process. The workhorse position, and the one this series mostly assumes.
- The database’s own buffer pool. The database caches too; data already in memory there is cheap to serve again. A dedicated cache in front still wins on network distance and on shielding the database from query work.
This article is about one cache in front of one database:
Client ↓ Application ← checks the cache first ↓ Cache ← hit: return the copy ↓ miss Database ← the expensive fetch ↓ Fill the cache, return the result
When one cache grows into a fleet of its own (tiers, hot keys spread across nodes, invalidation crossing process boundaries) that becomes a distributed caching problem, and it gets its own treatment in distributed caching, part 2 of this series. Everything here is the single-cache foundation it builds on.
Cache hit ratio
The hit ratio is the number that runs the whole show: hits divided by all requests, hits / (hits + misses). What makes it the number that matters is the arithmetic it feeds:
effective latency = hit_ratio x cache_latency + (1 - hit_ratio) x source_latency
Say, for illustration, that a cache hit costs one time unit and a database read costs one hundred. At a 90% hit ratio the average read costs 10.9 units: ten times better than no cache, still ten times worse than a hit. At 99% it costs 1.99. The lesson is in the curve: the expensive misses dominate the average, and the difference between a good cache and a great one lives in the last few points of ratio.
Four things move the ratio:
- Working set vs memory. If the hot set fits in the cache, hits stabilize; if it does not, eviction churns the cache against itself and the ratio sags. Sizing the cache to the working set (not to a round number) is the first decision.
- TTL length. Long TTLs trust copies longer, raising hits and staleness together. Short TTLs refresh faster and miss more.
- Request skew. Caches run on popularity; uniformly random access hits for nobody. Real traffic is skewed, which is why caching works at all.
- Key granularity. Cache the entity, and every reader of that entity shares one copy; cache per-user views, and the same data is stored once per user with a miss in each. Coarser keys, higher hits, until they serve one user another user’s shared view.
Measure the ratio traffic-weighted: bytes or expensive-query count, not just request count. Ninety percent of requests hitting while all the heavy queries miss is a cache flattering itself.
Read patterns: cache-aside vs read-through
Every read is a decision about who fetches on a miss. Two patterns cover the field.
Cache-aside (lazy loading) makes the application the choreographer: check the cache, on a miss read the database, write the copy into the cache, continue. The cache sits beside the application, unaware of the database; populated only by what is actually requested, and never holding data nobody asked for.
Read-through makes the cache the path: the application asks the cache, and the cache itself fetches from the source on a miss before answering. The fill logic lives in one place, and every consumer gets it for free.
| Dimension | Cache-aside | Read-through |
|---|---|---|
| Who fetches on a miss | The application | The cache layer |
| What occupies memory | Only what was requested | Whatever the layer loads |
| When the cache fails | Application falls through to the database | Reads fail unless a fallback exists |
| Where the logic lives | In every caller, or a shared library | Once, in the cache tier |
| First read of a key | Full price, paid per caller | Full price, paid once |
The trade is dependency versus duplication: cache-aside keeps the cache disposable (the system degrades to direct reads when it fails) while read-through concentrates logic and makes the cache a load-bearing component. Many systems start cache-aside and grow into read-through as the number of services makes duplicated fill logic intolerable.
Write patterns: write-through vs write-back
Reads decide how copies are filled; writes decide how they stay honest. Three patterns, two of which share the keyphrase.
Write-through writes the cache and the database together, synchronously: the write pays for both, and the moment it completes, the copy and the source agree. What was just written is immediately a cache hit.
Write-around writes to the database and skips the cache; deliberate for write-once, read-rarely data, which would otherwise evict the hot set to sit unloved in memory.
Write-back (write-behind) acknowledges the write once the cache holds it, and flushes to the database asynchronously: batched, on a timer, when convenient. Writes run at cache speed and the database sees a smoothed, batched flow. The price is the window: until the flush lands, the write exists only in the cache, and a crash inside the window loses an operation the client was told succeeded. That is a durability trade, in the precise sense defined in availability vs reliability vs durability, not a performance quirk.
| Dimension | Write-through | Write-back |
|---|---|---|
| Write latency | Database speed; the write waits for the source | Cache speed; the write returns on the cache fill |
| After the write returns | Copy and source agree | Copy leads; the source follows later |
| If the cache crashes | The write is already durable | Unflushed writes are lost |
| Database load | Every write, immediately | Batched, smoothed, deferred |
| Best fit | Writes that must not be lost | High-volume writes tolerating a loss window |
The pattern choice is really one question: can this data afford to exist in a single place for a few seconds? Order records usually cannot; view counters and analytics events usually can.
Eviction: LRU and friends
A cache forgets for two different reasons, and keeping them apart prevents half the confusion: TTL is about time (a copy’s permission to be served expires) while eviction is about space; memory is full and something must go. Both run at once; a copy can expire with room to spare, or be evicted while perfectly fresh.
LRU eviction, least recently used; bets that the recent past predicts the near future: evict whatever has gone longest without being read. The bet is usually sound because access is temporally local, and the implementation is famously cheap: a hash map pointing into a doubly linked list; every read moves the entry to the front, eviction removes the tail, both in constant time. That constant-time combination is why LRU is the default eviction policy almost everywhere.
The alternatives earn their places at the margins:
- LFU; least frequently used: evicts by count, not recency, and survives one-time sweeps that would wash an LRU cache’s entire hot set out the tail.
- FIFO: cheapest to reason about, blind to recency and frequency alike; fine where the working set turns over wholesale anyway.
- Random; an old result that keeps earning its place: with no ordering to poison, a sweep of cold data evicts hot entries only by chance, and most of the hot set survives.
Real cache servers make these policies configurable (Redis, for instance, chooses among allkeys-lru, volatile-lru, LFU, and others via its maxmemory policy) and how such a server implements eviction, persistence, and its data structures is part 5’s subject, Redis architecture.
The price: consistency
Every cache is a copy, and copies drift. How long a reader can see stale data is not an accident of implementation; it is a dial set by the pattern choices: cache-aside with long TTLs can serve yesterday’s state, while write-through keeps the window near zero. When freshness matters more than the pattern delivers, the dial turns toward explicit invalidation (purges, versioned keys, and the rest of that taxonomy) which is part 6, cache invalidation strategies. The vocabulary for what a system promises about its copies (the consistency models) is laid out in the CAP theorem and consistency models.
And when caches fail, they fail at scale: a popular key expiring under load becomes a stampede of simultaneous misses; a dead cache can dump a fleet’s traffic straight onto the database. The failure taxonomy (stampede, avalanche, penetration) and its mitigations are part 3, cache stampede and failure modes.
When not to cache
- Write-heavy data. A cache accelerates reads; for data written more than read, write-through doubles the write cost and write-back prices it in durability risk. The right tool for write-heavy work is usually a queue, not a cache.
- Strictly fresh reads. Where a read must see the just-written state (balances, inventories, authorization decisions) the stale window a cache opens is a correctness bug, not a trade.
- Small datasets already in memory. A ten-megabyte table lives in the database’s own buffer pool; a dedicated cache in front adds a component and a staleness decision to change nothing.
- One-pass traffic. Exports, scans, unique-key requests: hit ratio near zero, pure overhead. A cache pays for itself on repetition alone.
- Before fixing the query. Caching an unindexed scan hides the problem and hands it back on the first miss. The database-level fix is an index (how database indexes work) and it composes with caching instead of competing with it.
The common thread: a cache is a copy, plus a component, plus a failure domain, plus a staleness decision. It has to earn each one.
Common mistakes
- No TTL at all. Entries that never expire turn the cache into a second database nobody administers: permanently stale, and evicting the hot set to stay that way.
- No hit-ratio telemetry. A cache without metrics is a bet, not an optimization. The ratio and its trend decide whether the cache is working and what to fix when it is not.
- Caching everything. Junk keys evict the hot set; the ratio falls, and the cache costs latency without saving any.
- Wrong key granularity. Per-user keys over shared data store the same row once per user and hit for none of them.
- Treating the cache as the source of truth. The database is the source; the cache is a disposable copy. A cache you cannot flush and rebuild from the source without incident is designed wrong.
- One TTL for all data. A product description and a stock level age at different speeds; one TTL is right for neither.
FAQ
What is caching in system design?
Storing the result of an expensive computation or data fetch in a faster layer of storage, close to the reader, so repeated requests are served from the copy instead of repeating the work. It trades a small amount of staleness for a large amount of latency and load, governed by three decisions: the read pattern, the write pattern, and the eviction and TTL policy.
What is the difference between cache-aside and read-through?
Who fetches on a miss. In cache-aside, the application reads the database and fills the cache itself, which keeps the cache disposable; the system degrades to direct reads if it fails. In read-through, the cache layer fetches from the source, centralizing the fill logic and making the cache a load-bearing component every reader depends on.
When should you use write-through vs write-back?
Write-through completes both writes (cache and database) before acknowledging, so it costs write latency and guarantees the copy and the source agree. Write-back acknowledges on the cache fill and flushes to the database later, so writes are fast but a crash inside the flush window loses acknowledged writes, a durability trade. Anything that cannot tolerate that window: orders, payments, permissions, belongs in write-through.
What is LRU eviction?
Least recently used: when the cache is full, evict whatever has gone longest without being read, on the bet that recent access predicts future access. It is implemented with a hash map into a doubly linked list so both a hit and an eviction cost constant time, which is why it is the default policy in most cache servers. Its weakness is sweeps: one pass over cold data can evict the entire hot set.
What is a good cache hit ratio?
There is no universal number; the ratio depends on the working set, the memory, the TTL, and the skew of the traffic. What to watch is the traffic-weighted trend: the share of expensive work served from the cache, and whether it moves when the knobs move. A ratio that looks high while the heavy queries miss is a cache flattering itself.
Related articles
- Next read: distributed caching; part 2: what changes when one cache becomes a fleet: tiers, hot keys, and cross-node coordination.
- cache stampede and failure modes; part 3: what happens when caches fail, and how to survive it.
- redis caching, part 4: implementing these patterns with the standard tool.
- redis architecture, part 5: the data structures, persistence, and Sentinel vs Cluster underneath.
- cache invalidation strategies; part 6: purges, versioned keys, and keeping copies honest.
- how a CDN works; caching at the network edge, before traffic reaches any of this.
- latency vs throughput; the budget a cache spends its hits on.
Last updated on 7 September 2026
