Memory Leaks in Backend Services: Diagnosis, Tools, and Production Fixes

Memory Leaks in Backend Services matter because a slow rise in RAM ends in a hard kill. When the host kills the process, in-flight work dies with it. You then pay a restart cost on every pod that hits the same bug. Users see errors while the new process warms up.

A leak is memory you keep after the job that needed it is done. The runtime cannot free it, because some root still points at it. The graph of live objects grows even when the request rate is flat. That shape is the first clue you should trust.

In my experience, teams blame the collector first. Still, a collector cannot drop a map you never clear. If you tune pauses while a cache has no bound, the process still dies. You should prove what is retained before you change GC flags.

This guide shows how to tell a leak from normal heap use. It also shows where leaks hide in real services. Then it covers the trade-offs of each fix, the traps that waste a day, and what the leak costs at scale.

What a leak is and why it fails in production

You can split leaks into two kinds. A managed leak lives on the GC heap. A native leak lives in buffers, arenas, and C code that the GC does not scan. Both show up as a climb in RSS or in the cgroup working set.

Managed leaks are the common case in Java, Go, dotnet, and Python. A global map, a static list, or a session store keeps objects alive. When you take two heap snapshots an hour apart, the same stacks grow. That diff is your bug.

Native leaks are harder. The GC heap looks flat, but RSS still climbs. For example, a driver may allocate buffers and never free them. You then need a native profiler, not only a heap dump.

Production fails in a few loud ways. First, the cgroup limit hits and the kernel sends a kill signal. Health checks can still pass until that moment, because they only probe a port. After the kill, the pod restarts and the leak clock starts again.

A second failure mode is a false scale-out. If you scale on CPU, memory pressure does not add pods. Old pods sit near the limit. A traffic spike then kills them together.

As a result, you get a thundering restart, not more capacity. New pods start clean while old pods die in a bunch. If retries are uncapped, the survivors climb even faster.

How to confirm the shape

Read the graph before you dump

Start with three lines on one chart. Plot request rate, GC heap, and RSS or container working set. When rate is flat and both memory lines climb, you have a leak. When rate climbs and memory follows, you may just need a larger heap or a tighter cache.

Also plot goroutine count, thread count, and open file counts. A leak of goroutines looks like a memory leak, because each stack holds frames and buffers. If those counts climb with RSS, start there. You will fix more than the heap.

Compare heap and RSS. If the heap is flat and RSS climbs, suspect native memory or a metric that includes page cache. Container working set usually drops file cache. Process RSS on a VM may not.

Capture two snapshots

Take one profile when the process is calm. Take a second profile after it has grown. Then diff them. A single snapshot shows what is large.

In Go, use the heap profile from the diagnostics endpoint. The Go diagnostics docs show how to turn on pprof. Sample in-use space, not only alloc space.

In Python, Python tracemalloc can snapshot traces of blocks. Compare two snapshots and sort by size diff. Keep the trace depth high enough to see your frame, not only the allocator.

In Java and dotnet, dump the heap or use a live view of allocations. Do this on one pod. Set a memory limit so a dump cannot fill the disk.

Where leaks hide in the design

Caches and maps

The most common leak I have seen is a cache with no cap and no TTL. A map from user id to session grows with every new user. After a week, the map is the heap.

Bound the cache by count and by bytes. Evict on TTL and on size. If you need fresh data, do not cache the write path without a plan to drop stale keys. A smaller cache that misses is cheaper than a pod that dies.

Clients, pools, and context

HTTP calls leak when you do not close the body. The connection stays in the pool, and the buffer stays alive. Close the body on every path, including errors. A defer or a finally block is the usual fix.

Context listeners leak when you register them and never remove them. A request callback stored on a global bus will pin the request. After the handler returns, that tree should be free. If it is not, find the register call.

Metrics can leak too. A label with a user id or a raw URL creates a new series forever. Cap label values. When the exporter is down, bound the queue or you will leak the outage into RAM.

Trade-offs of each fix

You have four practical moves. You can bound the data structure. You can fix a missing close or cancel. You can move state out of the process.

Bounding a cache adds misses and can add tail delay. Moving state to a remote store adds a network hop and a new failure domain. A timed restart hides the bug and still drops in-flight work. Still, a restart policy is a fair bridge while you ship the real fix.

Do not raise the memory limit as your only fix. A larger pod dies later, and it costs more every hour. Raise the limit only to stop a loop of kills while the patch rolls out. Then put the limit back.

Pitfalls and failure modes

Most failed hunts share the same traps. You can waste a day on a clean heap while native memory climbs. You can also blame a leak when the process is only busy. Read this list before you page the whole team.

• Calling GC pressure a leak when the heap drops after a full collection.
• Reading RSS on a VM and treating page cache as a leak.
• Taking a full dump on every pod at once.
• Diffing a profile from before a deploy with one from after.
• Ignoring native memory because the GC heap looks fine.
• Setting a memory limit with no headroom for a dump spike.

A common mistake I have seen is to compare pods of different ages. A new pod has a small heap. An old pod has a large one.

1. Chart rate, heap, and working set for one pod.
2. If the heap grows, capture two profiles and diff them.
3. If only RSS grows, profile native allocs.
4. Patch the retain path and ship it to one canary.
5. Watch that canary for a full day before you close the incident.

A capture you can run

The snippet below diffs two Go heap profiles. Take the first when RSS is calm. Take the second after it grows. Then look at cumulative bytes, because the parent frame often holds the map.

# Compare two heap profiles from the same pod.
# Take heap-1 when RSS is calm. Take heap-2 an hour later.
go tool pprof -diff_base=heap-1.pb.gz heap-2.pb.gz

# In the pprof shell, show retained bytes by call stack.
top -cum
list cache.Set

Read the top frames with your package name, not only runtime frames. If the growth sits in a cache set method, open that function. If it sits in a client body, find the missing close. After the patch, repeat the same diff on the canary.

If you run under Kubernetes, set requests and limits with the Kubernetes memory limits guidance. Leave headroom above the steady heap. A limit that matches the steady RSS will kill you during a spike or a dump.

Performance, scale, and cost

Memory is a capacity bill. Each extra GB on every replica is money you pay while the leak runs. In an illustrative production range, a few hundred MB per pod is noise.

A climb of several GB per day is an incident. You should page on the slope, not only on the limit. A flat high line can be a healthy cache. A rising line with flat traffic is a bug.

The user-facing cost shows up as tail latency. When a pod is killed, its in-flight calls fail or retry. Retries hit the pods that remain. Those pods climb faster.

Autoscaling does not fix a leak. It can delay the pain if you scale on memory and you have quota. New pods start clean. Old pods still die.

GC work rises as the live set grows. The collector scans more pointers to free less trash. That is why a leak also looks like a CPU bug.

If short-lived trash is the real issue, look at garbage collection tuning instead. A true leak stays after a full collection. If a full collection drops the heap, you do not have a leak.

Shared maps that grow also raise lock time. If many requests update one cache, you can harm thread safety while you chase RAM. A concurrent map with no bound is still a leak. Make the bound part of the type, so a caller cannot skip it.

Set an alert on the slope of working set per pod, grouped by build. Page when the slope stays high for an hour and the request rate does not explain it. Also alert when restarts have an out-of-memory reason. Those two signals catch most leaks before the whole fleet is old.

Key Takeaways

• A leak is retained memory after the work is done, and the slope matters more than one high reading.
• Diff two profiles from the same pod before you change GC flags or raise the limit.
• If the heap is flat and RSS climbs, look at native allocs, file cache, and cgroup metrics.
• Bound caches, close bodies, and drop listeners. Those three paths cover most service leaks.
• Do not dump every pod at once. Capture on a canary, then watch the slope for a full day.
• Page on working-set slope and on out-of-memory restarts, not only on a static memory cap.
• Scale-out and timed restarts buy time. They do not remove the retain path.

FAQ

Is a slow heap climb always a leak?

No. A climb that tracks request rate, cache warm-up, or a larger live set can be normal. When the rate is flat and the heap still grows for hours, treat it as a leak. Confirm with a diff of two profiles from the same process.

Should you raise the memory limit first?

Raise it only to stop a kill loop while a patch rolls out. A bigger pod costs more and dies later. If you leave the higher limit in place, the next leak hides until the new cap is hit. Put the limit back after the slope is flat.

Which profile should you trust in Go?

Trust in-use space when you hunt retained objects. Alloc space is useful when you hunt churn and GC load. Take both if you are unsure. Then diff in-use space across time, because that view shows what you failed to drop.

Can a restart policy replace a code fix?

It cannot. A timer restart drops work and resets the clock. The bug returns as soon as the pod is old. Use a restart only as a bridge.

Ship a bound, a close, or a cancel, and prove the slope is flat on a canary. If the canary still climbs, the retain path is still there. Do not close the incident on a restart alone.

Pick one pod that is climbing and chart rate, heap, and working set today. If the heap grows, diff two profiles and patch the retain path. If only RSS grows, profile native allocs before you touch GC flags. Then watch the canary slope for a full day, and page on that slope so the next leak is obvious.

Last updated on 07 September 2026.