Service Discovery: Registry, Patterns, and Consul vs etcd

This article is the architecture cluster’s fourth stop, and it sits directly under an assumption the last one made. The event driven architecture stop (and the microservices architecture hub before it) described services finding and calling each other without ever saying how the finding works. This is the how. A fleet cannot hardcode addresses, because in a distributed system the members come and go (deployments replace instances, autoscaling adds them, crashes remove them) and a hand-maintained map is stale before it finishes loading. The lineage runs deeper than the cluster, too: the coordination services that keep registries trustworthy are the same ones that decide who the leader is (the leader election article’s subject) and this article builds on that machinery instead of re-deriving it.

The problem sounds like plumbing because it is plumbing, and that is exactly why it deserves care: discovery failures never read as discovery failures. A stale registry surfaces as mystery latency, as calls dialed to instances that no longer exist, as load beached on one unlucky survivor, symptoms that get blamed on the network long before anyone audits the phone book. What follows defines the mechanism, then spends its length on the three questions that decide whether a registry tells the truth: what it stores and how its entries die, which side of the call does the resolving, and which tool the registry runs on.

What is service discovery

Service discovery is the mechanism by which the instances of a distributed system find each other’s live addresses automatically: instances register with a shared registry, keep their registrations warm, and callers resolve names to endpoints at request time instead of at deploy time.

The cast has three roles. The service instance registers its address on startup and renews the registration on a schedule; a heartbeat, or a lease with a TTL, where the entry expires unless the instance keeps proving it is alive. The service registry is the shared source of truth: a live database of who is up, at what address, in what version. The resolver is whoever needs to make a call (a client, a gateway, a balancer) and the discovery question is always which side of the call does the resolving, a question this article answers in full below.

The lifecycle is where the honesty lives. An instance registers, serves, heartbeats, deregisters, and the interesting parts are the transitions. A graceful shutdown deregisters itself; a crash produces no farewell, so the registry learns from silence: the heartbeat stops, the lease expires, the entry dies on a timer. That timer is this article’s central tension, the failure-detection window. Expire fast, and a slow-but-alive instance gets evicted: flapping, a registry oscillating between confident and wrong. Expire slow, and callers keep dialing a corpse for the whole window. Every registry decision (TTL length, heartbeat interval, eviction policy) is a bet on how fast death can be told from slowness, and the honest answer is that they cannot always be told apart. Health checking is the refinement that keeps the registry from guessing alone, and the rest of this article treats that window as the thing being managed.

Service registry

A service registry is the shared, live database of who is up: service names mapped to live endpoints, each entry carrying an address, a version tag, optional metadata, and a lease the owner must keep renewing or lose the entry.

What an entry holds decides what discovery can do. The minimum is a name and an address; the useful additions are the port, a version tag, zone or rack hints for routing locality, and health status. Who writes the entry is the first real design decision. In self-registration, the instance registers itself on startup and keeps its lease warm; one less component, at the cost of two failure modes the instance cannot see from inside: a crash leaves no farewell, which the lease’s TTL cleans up, and a boot that registers before the instance is actually ready serves traffic it cannot yet honor. In third-party registration, a supervisor watches the fleet and writes the registry on each instance’s behalf; deregistration becomes someone else’s clean job, at the cost of a new component whose own failure now hides inside the discovery path.

Keeping the registry honest is a race between truth and silence. Heartbeats on a schedule say “still here”; leases with a TTL say the same thing by construction; the entry dies unless re-proven, so a crashed instance needs no witness, only a timer. Active health checking goes further: the registry, or a checker it trusts, polls each instance and evicts on evidence rather than on silence, which separates a dead instance from a paused one better than any heartbeat can. The balancer’s half of this story (the health checks that decide who deserves traffic in the first place) is the load balancing article’s to own; the registry’s half is only the admission question: is this entry still true?

The registry itself is infrastructure, and it is the fleet’s most concentrated piece: every caller depends on it, so a single registry node is a single point of failure by construction. An honest registry is therefore not one node but a small cluster that keeps one shared truth through failures and partitions, which is the consensus problem, solved in practice by algorithms like the raft consensus algorithm, the same machinery this cluster’s coordination lineage runs on. Registry availability is bought the way all availability is bought: redundancy plus failover, sized by how many nodes the fleet is willing to lose at once.

Client-side vs server-side discovery

Client-side vs server-side discovery is the placement question: who turns a service name into a live address: the caller itself, or a fixed piece of infrastructure in between? The registry is the same in both; only the resolver moves.

In client-side discovery, the caller queries the registry, receives the live instance list, picks one (with whatever balance policy it likes) and calls it directly. The strengths are structural: no middle hop, the full list in the caller’s hands, zone-aware choices and smart retries available on the spot. The costs are structural too: every client now speaks the registry’s protocol and carries balancing logic, in every language the fleet contains, and a change in that logic is not a server-side rollout but a firmware update to every caller in the system. Client-side discovery trades operational simplicity for client sophistication, and the bill arrives the day the fleet’s language count grows.

In server-side discovery, the caller dials one fixed name (a balancer or a gateway per service) and lets infrastructure do the resolving: the router consults the registry, picks a live instance, and forwards. The client stays dumb and language-agnostic, routing fixes land in one place, and the pattern is what makes front doors work; an API gateway resolving internal services is server-side discovery wearing its most familiar suit. The costs are a new hop, a new tier to run and make redundant, and a subtle shift: the staleness this article has been chasing now lives in the router’s cache of the registry, so the failure-detection window is paid at the routing layer instead of in the client.

The honest answer is that the choice is per-boundary, not per-system. North-south traffic (requests entering from outside) resolves at the edge, where the gateway already stands. East-west traffic (service to service) is where the choice is real: client-side stays lean where the fleet is homogeneous and the registry has a mature client library; server-side wins where languages multiply or where routing policy wants one throat to choke. Service meshes are this spectrum industrialized (a sidecar next to every instance doing client-side discovery and balancing on the service’s behalf, with the registry underneath unchanged) which is why a mesh is best understood as discovery’s most automated placement, not a replacement for it.

Consul vs etcd

Consul vs etcd is the tooling question, and it is smaller than it looks: both are strongly consistent, Raft-backed coordination stores, so the real difference is not whether they keep the truth but how much of the discovery pattern they ship along with it.

etcd is a distributed, consistent key-value store, deliberately primitive. It offers the substrate (a small, highly available store with watch semantics and leases that expire) and none of the service-awareness: no health-check model beyond what you build, no DNS face, no notion that one key is an address and one expiry is a dead instance. Its most famous consumer is Kubernetes, which builds an entire discovery layer (services, endpoints, DNS) on top of it, and that is the honest summary of what etcd is for: bring your own discovery semantics, keep the consistency core small and battle-tested.

Consul ships the whole pattern. Each node runs an agent; agents form the cluster, run health checks on the instances beside them, expose the catalog over both an HTTP API and DNS (resolving service names to records whose staleness the TTL controls) and can grow into a service mesh, sidecars and all. Registration, admission, failure detection, and resolution arrive as one integrated product, at the price of committing to that product’s model of the fleet. The elder sibling in this space is ZooKeeper; the coordination kernel whose elections and leases preceded both, whose lineage the leader election article covers, and which many fleets still run for discovery exactly because it was there first.

The choice, then, is a build-versus-buy question wearing tool names. If the platform team wants discovery semantics of their own design on a small consistent core (the Kubernetes move) etcd is the primitive and the fleet writes the pattern. If discovery should be a product the fleet adopts rather than builds (agents, checks, DNS included) Consul is the turnkey answer, and the mesh is already half-installed. What the choice is not: a consensus-algorithm comparison. The Raft both tools run is the same machinery, and the algorithm trade-offs are consensus algorithms‘ to compare, not this article’s shopping trip.

Common mistakes

• Registering on boot, not on ready. An instance that registers at process start spends its warm-up window receiving traffic it cannot honor: cold caches, unfinished migrations, unopened pools. Register at readiness, when the instance can actually serve, and treat the registration as the last step of startup, not the first. The mirror mistake is just as common: deregistering on shutdown before draining, so live callers receive an address that has already stopped listening.
• Evicting on one witness. A single missed heartbeat, one failed health check, and the entry is gone, until the instance, slow but alive, re-registers, and the registry flaps between confident and wrong while callers ride the oscillation. Failure detection wants a majority over a window: several checks, a real threshold, a deliberate eviction policy, and hysteresis that keeps a borderline instance out until it has proven itself, not until it has pinged once.
• Client caches that never expire. The client resolves the fleet once, pins the list in memory, and calls it forever, which is a hardcoded address map wearing a discovery costume, with all of the staleness and none of the auditability. Cached resolutions need the registry’s own TTL or a re-resolve cadence, because the moment callers cache without expiry, every deploy becomes a stale-dial lottery.
• Running the registry on one node. A single registry instance is the fleet’s most efficient single point of failure: every caller’s truth flows through it, and its outage is a fleet-wide outage by construction. The honest registry is a small cluster (three or five nodes, sized like any redundant tier by the high availability math) and “the registry is up” must mean “a quorum is up,” not “the container restarted.”
• Flying blind on registry health. Registration churn, eviction rate, stale-hit rate, the failure-detection window in practice; none of these appear in request logs, which is why stale-registry failures get blamed on the network for months. Discovery is infrastructure with its own golden signals, and the telemetry that surfaces them is monitoring and observability territory: wired from day one, not after the first mystery-latency incident.

FAQ

What is the difference between service discovery and DNS?
DNS is one implementation of the registry (names to addresses, queried at resolve time) and a perfectly good one until churn meets caching. Plain DNS records say what an operator wrote, not who is alive, and resolver caches turn every record change into a waiting game on the TTL. DNS-based discovery exists and works well (Consul’s DNS interface, Kubernetes service DNS) because it puts live, health-checked truth behind the familiar interface and disciplines the TTL. The pattern and the protocol are not rivals; the honesty is what matters.

Do you need service discovery if you run Kubernetes?
You have it already; Kubernetes is service discovery with a management plane on top. Services, endpoints, and cluster DNS are the registry, the admission logic, and the resolver; the etcd-backed API server is the consistent core; kube-proxy and the mesh sidecars are the placement choices. The question is not whether to add discovery to Kubernetes but whether the fleet understands what it configured; the failure-detection window and the cache TTLs are still yours to reason about.

What is the difference between a service mesh and service discovery?
Discovery answers “which live instance should I call”; the mesh answers “what happens to the call once the instance is chosen”; retries, mutual TLS, routing policy, traffic splitting, telemetry, all handled by a sidecar next to every instance. A mesh contains discovery; the sidecar does client-side resolution, but it needs the registry underneath like every other placement, so it is a superset, not a substitute. Fleets adopt meshes for the traffic features; they still run the registry the mesh resolves against.

How does service discovery relate to load balancing?
Discovery supplies the live list; balancing picks from it. The two compose rather than compete: client-side discovery makes the caller do both (resolve, then balance locally) while server-side discovery moves both jobs to the router. The balancer’s own health checks, the ones that decide which instances deserve traffic at all, are the load balancing article’s half; the registry’s admission checks are this article’s half; honest fleets wire both to the same truth.

What happens if the registry itself goes down?
Resolution of new calls stops or goes stale, while established traffic keeps flowing; leases do not evaporate because the registry hiccuped, and instances keep serving what they were already serving. That asymmetry is why the registry is clustered in the first place: three or five consensus-backed nodes, sized so that losing one is a non-event and losing a quorum is the actual emergency. A one-node registry outage is fleet-wide silence; a quorate registry losing a member is a log line.

Related articles

• Next read: circuit breaker pattern; the reading spine’s next stop: the failure-detection window applied to the call itself, when a live-but-failing dependency stops deserving traffic, and the mechanism that fails fast instead of piling up hopeless attempts.
• microservices architecture; the cluster hub: the trade-off ledger that explains why a fleet of services needs a shared truth about its own members at all.
• leader election; the coordination lineage underneath the honest registry: how a small cluster keeps one shared truth, and what happens when it briefly cannot.
• load balancing; the traffic half of the pair: the algorithms and health checks that decide which live instance actually receives the call discovery just resolved.
• high availability; the redundancy math the registry cluster is sized by: single points of failure, n+1 redundancy, and failover that does not become its own incident.

Last updated on 6 September 2026.