What Is a Distributed System? Partial Failure, Clocks, and the Fallacies

Few teams set out to build a distributed system. They set out to survive more traffic than one machine can take, to keep serving users when a machine fails, to put data closer to the people using it, and distribution is what those goals cost. The moment a second machine joins and the two must cooperate, the system inherits problems that no amount of hardware fixes, because they are not hardware problems. They are problems of physics and coordination: latency that never reaches zero, failures that arrive in pieces, and clocks that do not agree.

A distributed system is a group of machines connected by a network that cooperate by passing messages to achieve a single logical result: components running concurrently on separate nodes, sharing no memory, and communicating only through the network. Client-server architecture is the smallest example; a globally replicated database is the full-sized one.

Distributed system characteristics

A distributed system can be recognized by four characteristics, and each is a liability as much as a capability:

• Multiple nodes. The system is several machines (application servers, cache nodes, databases) each with its own memory and its own view of the world.
• Message passing. Nodes coordinate only by sending messages across a network. Every message can be delayed, reordered, duplicated, or lost, which turns coordination into a protocol problem instead of a function call.
• Concurrency. Events happen on many machines at once, and no node observes them in one agreed order. Correctness must survive interleavings that no single thread of execution ever chose.
• No global state. No machine holds the true current state of the whole system. Every node holds a partial, slightly stale copy, and the system’s state is whatever the union of those copies says.

These four characteristics are why a distributed system is harder to build than one big machine with the same total capacity. The rest of this article is about the two that cause the most damage (partial failure and clocks) followed by the classic list of assumptions, the fallacies, that engineers make when they design for the machine they wish they had.

Why systems become distributed

No one distributes for fun. Three forces do the pushing, and it is worth knowing which one is pushing you, because they fail differently:

• Scale. A machine’s capacity (CPU, memory, network) is a box, and traffic eventually leaves the box. Vertical scaling buys time at a price that grows faster than the capacity; horizontal scaling spreads the load across many ordinary machines.
• Availability. One machine is a single point of failure: its reboot is everyone’s outage. Several machines, arranged so no single failure takes them all out, are what availability is built from at the systems level.
• Geography. Light in fiber travels roughly 200 kilometers per millisecond. Users on other continents cannot be served fast from one data center, so data has to live closer to the readers.

Notice what the third item quietly admits: a distributed system is sometimes not a choice at all. If the users are far apart and the latency budget is tight, distribution is the requirement, and the engineering task is surviving what it costs.

Partial failure: the defining property

On one machine, failure is total: the power supply dies and everything stops together. In a distributed system, failure arrives in pieces. One node crashes while nine keep serving. The network partitions and each half believes the other is down. A NIC silently drops frames while the process above it reports success. This is partial failure (the defining property of distributed systems) and it is more dangerous than total failure precisely because the surviving parts keep running, and keep making decisions on incomplete information.

Partial failure is why distributed correctness is harder than single-machine correctness. When a remote call does not answer, the caller cannot distinguish between “the request died on the way,” “the request was processed and the reply died on the way back,” and “the node is down and processing nothing.” Three different pasts, indistinguishable in the present, each possibly demanding a different response. A standard taxonomy of faults helps here:

• Crash faults. The node stops and never comes back on its own: the common case, and the least harmful.
• Omission faults. The node keeps running but drops messages: dropped requests, dropped replies. It looks alive from a distance and behaves dead up close.
• Byzantine faults. The node keeps answering, wrongly: corrupting data, misreporting state. Rare in one company’s hardware, routine when the “node” is an untrusted participant, and the only kind that requires distrusting the answerer himself.

Every protocol in the rest of this pillar (leader election, consensus, distributed locks, saga coordination) exists to answer one question: how do correct nodes keep working when some participants crash, stall, or misbehave within that taxonomy? The protocols differ because their answers cost different amounts of consistency and availability. The CAP theorem and consistency models covers the trade space, and the mechanics of surviving faults live in fault-tolerant systems.

Clocks: no two machines agree on now

The second defining property is time. Every machine has a clock, and every engineer’s first instinct is to use it: timestamp events, compare them, order them. The instinct fails quietly: no two clocks tick in agreement. NTP keeps machines within roughly tens of milliseconds of each other on a good day, and that error is large compared with the events a system wants to order. If node A writes a value at 10:00:00.005 by its clock and node B writes another at 10:00:00.003 by its, the timestamps say B happened first, and nothing in the system can confirm or deny it.

The classic resolution, from Lamport’s 1978 paper on time in distributed systems, is to stop asking what happened “when” and ask what happened “before what.” The happens-before relation defines order without a clock: if event A causes event B, every observer agrees A came first; if two events are unrelated, their order is genuinely arbitrary and can stay that way. Logical clocks (Lamport clocks, vector clocks) capture this ordering instead of time-of-day. The practical consequence for a designer is a rule: timestamps are for forensics, not correctness. A system whose correctness depends on two machines agreeing on the time of day is a system waiting for a subtle, unreproducible failure.

The fallacies of distributed computing

A list of eight assumptions about the network (the fallacies of distributed computing, compiled at Sun Microsystems in the 1990s) works as a checklist of the mistakes that follow from designing for the machine you wish you had. Each fallacy is an assumption that is false, and each has a standard consequence:

• The network is reliable. It is not. Retries, timeouts, and duplicate handling are mandatory design elements, not optional hardening.
• Latency is zero. A call across a network costs time even when everything works. Interfaces that assume instant remote access produce chatty protocols.
• Bandwidth is infinite. Bulk transfers and fan-out patterns have limits. A saturated link is an outage with a different name.
• The network is secure. Traffic crosses infrastructure you do not own. Encryption and authentication are baseline, not extras.
• Topology doesn’t change. Nodes are replaced, IPs are reassigned, deploys move components. Hardcoded topology rots quickly.
• There is one administrator. Many teams own different parts. Cross-component changes need protocols and documentation, not one operator’s intuition.
• Transport cost is zero. Marshalling, serialization, and protocol overhead are real work with real CPU cost.
• The network is homogeneous. Mixed hardware, versions, and settings are the norm. Protocols must tolerate participants that behave differently.

Read as a list of facts, the fallacies are unremarkable. Read as design review questions (where does this design assume the network is reliable, where does it assume latency is zero) they catch most of the mistakes this article has described from another angle.

What distribution buys, and what it costs

The buys are the three forces from the top of the article, achieved: scale beyond one machine, availability through redundancy, proximity to users. The costs are less often itemized, and honest system design itemizes them:

• Consistency. Copies on multiple nodes drift apart unless protocols force them to agree, and the protocols cost latency and availability. This is the territory of the CAP theorem; a trade, not a problem to be solved.
• Latency. Coordination messages replace function calls. Every agreement the nodes reach costs round trips, in the currency of latency vs throughput.
• Operational complexity. Deployment, configuration, monitoring, and debugging all multiply. “Which node held the old value?” is a question a single machine never has to answer.
• Partial-failure engineering. Every design must state what happens when each component dies alone; the property that makes distributed systems work, and the work that makes them correct.

Where this cluster goes

This article is the entry point of the Distributed Systems pillar, and the rest of the cluster builds on it directly. Each earns its own visit when the need arrives:

• Consistent hashing, how a fleet decides which node holds which key, without a directory.
• Distributed consensus, how nodes agree on one value despite partial failure.
• Consensus algorithms: Raft, Paxos, and the family, compared.
• The Raft consensus algorithm: the modern consensus algorithm, in depth.
• The Paxos algorithm: the classic, made readable at last.
• Leader election: choosing the coordinator, and the split-brain risk that follows.
• Distributed transactions; two-phase commit and why it blocks.
• The saga pattern, long transactions without holding locks across services.
• Distributed locks; mutual exclusion across machines, and how it fails.
• Fault-tolerant systems: redundancy, failover, and failure detection.

Two neighbors in other pillars belong on the same shelf: microservices architecture is the decomposition style that distributes application logic, and monitoring and observability is what makes distributed systems debuggable at all; a fleet you cannot observe is a fleet you cannot operate.

Common mistakes

• Assuming the local machine’s rules. Shared memory, atomic operations, one clock, total failure, none of them survive the network. Design for messages, partial failure, and disagreement first.
• Trusting timestamps for order. Clock skew guarantees that some timestamps lie. Order by causality where correctness matters; treat wall-clock ordering as a convenience.
• Designing for the happy path. In a distributed system the happy path is the minority case. The design question is what happens when the network stalls, a node dies mid-operation, and the retry arrives twice.
• Mistaking a load-balanced web tier for the hard kind. Stateless request distribution is the easy case; the state lives in the database. The hard kind puts state itself on many nodes, and every consistency problem this article described follows from it.

FAQ

What is a distributed system?

A group of machines, connected by a network, that cooperate by passing messages to achieve a single logical result. The defining properties: components run concurrently, share no memory, communicate only through messages, and no single node holds the complete current state. Almost every production web system qualifies once more than one machine is doing the work.

What is partial failure?

Failure that arrives in pieces: some nodes crash while others keep running, or the network splits the fleet into groups that cannot see each other. It is the defining property of distributed systems because it forces every design to state what the surviving parts do, and it is harder than total failure because the survivors keep making decisions on incomplete information.

What are the fallacies of distributed computing?

Eight false network assumptions, compiled at Sun Microsystems in the 1990s: the network is reliable; latency is zero; bandwidth is infinite; the network is secure; topology doesn’t change; there is one administrator; transport cost is zero; the network is homogeneous. They function as a design-review checklist: any design that silently assumes one of them will break in production at the exact point where the assumption fails.

Why can’t distributed systems agree on time?

Because no two physical clocks tick at the same rate, and synchronization (NTP) only bounds the disagreement, typically to milliseconds or tens of milliseconds, which is coarse compared with the ordering questions a system asks. The standard resolution is causal ordering: Lamport’s happens-before relation and logical clocks, which capture what caused what without pretending any single wall clock is true.

Is a load-balanced web application a distributed system?

Yes, and the easy kind. Multiple stateless servers behind a balancer leave the hard part, the state, with the database. The genuinely hard distributed problems: consensus, replication, coordination; begin when state itself is spread across nodes that must agree despite partial failure.

Related articles

• Next read: consistent hashing; the first concrete problem this pillar solves: which node holds which key.
• the CAP theorem and consistency models, the trade space behind every consistency decision in this article.
• latency vs throughput; the currency coordination pays in.
• availability vs reliability vs durability; precise words for what distribution is bought for.
• client-server architecture: the smallest distributed arrangement, and the one everything else extends.