DNS for Backend Engineers: Resolution, Caching, Failover, and Debugging

DNS for Backend Engineers matters because a bad name lookup can stall every call in your path. When a cache holds a dead address, clients keep dialing a host that is gone. You then see timeouts, even though the app code is fine. Still, teams treat DNS as a solved detail until the first long outage.

A name is a pointer, and the pointer can lie for as long as the TTL allows. If you cut over a service and forget the caches, the new box stays idle. Because of that, failover plans that ignore DNS fail in a quiet way. Also, the error shows up as a connect timeout, so the first dashboard you open is often the wrong one.

What DNS is and why it fails in production

DNS maps a name to data, most often an address. RFC 1034 defines the tree of names, and RFC 1035 defines the records on the wire. When your process calls getaddrinfo, it does not talk to the owner of the zone first. Instead, it asks a stub path that may answer from memory.

Production fails when that memory is wrong, slow, or split. A common mistake I have seen is a health check that flips a record while thousands of tasks still hold the old answer. Then the load moves on paper, and the packets do not. After the TTL, traffic finally shifts, and people call the delay a mystery.

Another failure is the negative cache. If a name is missing at boot, the resolver stores the miss. RFC 2308 covers how long that miss may live.

When the name appears later, clients still get NXDOMAIN until the negative TTL ends. So a brand new service looks down, even though the zone is correct.

Search suffixes add a third trap. If ndots is high, a short name grows extra labels before it is tried as a full name. Then a lookup for a public host can leak to an internal zone, or the other way around. Also, each extra try burns the timeout budget before the real query starts.

How resolution and caching are built

Think in three roles. First, the stub lives in the host or in the language runtime. Next, the recursive resolver walks the tree when the stub has no answer.

Finally, the source server for the zone holds the records you edit. When any role caches, you have more than one copy of the truth.

The stub may cache inside the process. Java, Go, and some HTTP clients do this even when the OS does not. Also, nscd or systemd-resolved may cache for every process on the box.

Because those layers do not share one clock, a flush in one place leaves the others stale. In my experience, people restart the app and still see the old address.

What a single lookup does

The stub sends a query, usually over UDP, to the recursive resolver. If the packet is too big, the reply sets the truncate bit and the stub retries on TCP. That is why TCP and UDP both belong in a DNS debug session. When the recursive resolver has no cache hit, it asks the root, then the TLD, then your zone.

EDNS lets the reply grow past the old 512 byte limit. If a firewall drops those larger UDP packets, you get timeouts that look random. Then the client falls back to TCP, if it still has time. Also, a resolver that cannot reach port 53 on TCP will fail only on the big answers, such as a wide TXT record.

Records you actually use

An A or AAAA record is the common case. A CNAME is an alias, and the client must follow it to a final address. When a CNAME points at another CNAME, the chain adds delay and another cache entry. Also, a CNAME at the zone apex is a common source of pain for mail and for the zone itself.

SRV records carry host and port, which is useful for internal RPC. If your client ignores SRV and hard codes the port, the record cannot move the service. Meanwhile, TXT records hold checks for mail and for domain ownership. Still, a huge TXT set can push you into TCP and into the timeout path above.

Trade-offs for TTL, failover, and scope

You pick a TTL for a reason, not as a default you never touch. A short TTL makes cutovers faster, and it also raises query load. A long TTL is cheap and stable, and it also makes a bad push last a long time. Therefore you match the TTL to how often the address changes, and to how fast you must shed a dead target.

Failover that only changes a DNS record is slow by design. While the TTL runs down, both the old and the new target get traffic. If you need a fast shift, put health checks on a load balancer and keep the name stable. Read about load balancer failover before you bet an incident on a one minute TTL alone.

Split horizon is powerful and easy to misuse. When the same name answers differently inside the VPC and on the public internet, a test on your laptop will lie. Also, a CI runner outside the network will fail a check that works in prod. Because of that, log the resolver address next to the answer when you debug.

Pitfalls and failure modes

Caches hide the bug until a roll, a scale event, or a region loss. However, the symptoms look like app bugs. You see connect errors, TLS handshake errors, or a spike in p99. Before you rewrite a client, check whether the name still points at the target you think.

A single recursive resolver is a quiet SPOF. If that fleet fails, every service that does not cache forever will stall together. Also, a resolver that rate limits you will drop queries under a deploy storm. Then many tasks miss cache at once, and the thundering herd hits the same resolver.

I have also seen zones that depend on one another across a broken link. That can look like network partitions even when your app network is fine. If the resolver cannot reach the source server, it may serve a stale answer or return SERVFAIL. Still, some clients treat both as a hard fail and do not try a second address.

1. First, note the exact name, the search list, and the resolver address.
2. Next, query that resolver and compare the answer with the zone file.
3. Then flush only one cache layer and test again, so you know which layer was stale.
4. After that, check negative caches and CNAME chains before you blame the app.
5. Finally, confirm TCP port 53 works, because a truncated UDP reply needs it.

TLS failures often sit one step past DNS. The name can resolve to a new host that does not hold the cert you expect. When that happens, the handshake fails and the DNS TTL is no longer the story. Pair this debug path with TLS checks so you do not stop at the A record.

A resolver setup you can copy

The snippet below is a starting point for a service host, not a full recursive design. It caps the time a stub will wait, and it tries more than one resolver. Also, rotate spreads queries so one resolver does not sit idle. If you copy it, change the addresses and the search list for your network.

# stub resolver for a service host
nameserver 10.0.0.2
nameserver 10.0.0.3
options timeout:1 attempts:2 rotate ndots:1
search svc.internal

Set the app cache to a short bound as well. A five minute process cache on top of a five minute TTL can stretch a cutover to ten minutes. Because the two clocks start at different times, the worst case is the sum. Therefore, pick one layer as the source of delay and keep the other layer tiny.

For the zone, publish at least two addresses when the client knows how to walk them. Also, keep the SOA minimum honest, since it feeds negative caching. When you add a name, query it from a fresh resolver before you send traffic. Then you avoid filling the world with a cached miss.

Performance, scale, and cost

DNS is cheap until your design makes it hot. A TTL of a few seconds on a name that every request looks up will melt a small resolver. In an illustrative production range, a busy service can emit tens of thousands of lookups per second during a cold start. If you cache inside the process for even a short time, that rate drops by a large factor.

Latency stacks in a way people forget. A cold walk from root to zone can cost tens of milliseconds, and a timeout retry can cost a full second. When this sits on the request path, your p99 becomes the DNS timeout, not the handler time. So look up names at start, or on a refresh loop, instead of on the first byte of each call.

Cost shows up as resolver fleet size, log volume, and failed requests. Anycast resolvers help with scale, and they also make debugging harder because two queries may hit two sites. Meanwhile, DNS based global failover looks simple on a slide and slow in an incident. If you need both scale and a fast cut, keep DNS stable and shift traffic closer to the connection.

Do not open a new lookup for every retry. While a call fails, a tight retry loop can multiply query load and extend the outage. Also, honor a deadline that is shorter than the user request, so DNS cannot eat the whole budget. After the incident, record which cache still held the bad answer.

Key Takeaways

• Also treat every cache as a copy of the record that can be stale.
• When you fail over, plan for the TTL plus every client cache, not the TTL alone.
• Because negative caching hides new names, test a fresh resolver before you send load.
• If UDP replies are truncated, you still need TCP to the resolver or lookups will stall.
• Still log the resolver address and the raw answer next to connect errors.
• Therefore keep hot names in a short process cache so a cold start cannot stampede.

FAQ

Why did the cutover finish in DNS but not in traffic?

Because caches still hold the old address until each TTL ends. Also, some runtimes cache on their own clock, so a zone change is not the whole story. When you need a faster shift, move traffic at the load balancer and keep the name still. Then use a short TTL only for names that must move.

Should every service use a one second TTL?

No, because a one second TTL turns DNS into a hot dependency on the request path. If the name rarely changes, a longer TTL is cheaper and more stable. However, you should not pick a day long TTL for a target you may need to drain in an incident. Instead, match the TTL to the real change rate.

What is the first cache to check?

First, check the process cache, since a restart clears it and a flush of the host may not. Next, check the host resolver, such as systemd-resolved or nscd. Then query the recursive resolver directly and compare it with the source zone. After that, you will know which layer still lies.

Does DNS failover replace health checks?

It does not, since DNS is slow and caches are many. While health checks can pull a bad target out of rotation in seconds, DNS waits on TTL. Also, some clients will keep a dead address until they open a new process. Therefore use DNS for discovery, and use health checks for fast removal.

Start with one name that has hurt you before. Write down each cache, the TTL, and the resolver addresses, then run the five debug steps on a quiet host. After that, fix the layer that was stale, and only then shorten a TTL. If you do this once, the next incident will point at the right cache instead of the wrong service.

Last updated on 16 September 2026.