Monitoring and Observability: Knowing What Your System Is Doing

Observability is the property of a system whose internal state can be inferred from the outputs it exposes: a term imported from control theory, applied to software as the ability to answer questions about production, including questions nobody thought to ask in advance, from the telemetry the system already emits.

The boundaries stay where the fleet drew them. Distributed systems owns the phenomenon being observed; partial failure, and why a fleet’s internals cannot be inspected the way a single process’s can. Availability and reliability own the definitions this article measures, and high availability owns the architecture that produces the outcomes. Fault-tolerant systems owns detection in protocol time (health checks, heartbeats, failure detectors) and this page owns what humans do with the verdicts: the telemetry pipeline, the shared vocabulary, and the overview of the pillars. The depth belongs to the parts: logging, metrics, and distributed tracing own their own queries, and this page carries the one comparison table that says which pillar answers which question.

Observability vs monitoring

The honest version of the distinction is about what you know in advance. Monitoring is the practice of watching a system for the failure modes you predicted: the dashboard of CPU and memory, the alert when error rate crosses a threshold, the check that the certificate expires soon. Every one of those is a question written down before the incident: a known unknown, registered in advance and watched forever. It is necessary work, and no observability program replaces it.

Observability covers the other half: the questions you could not have registered, because nobody imagined them. “Why is checkout slow for one payment provider, in one region, on mobile, since Tuesday?” is not a dashboard anyone pre-built. Answering it requires evidence collected before the question existed: the raw events, the per-request context, the dimensions of provider, region, and device retained somewhere queryable. A system is observable to the degree that such questions can be answered interactively, from what the system already emitted, without shipping new code to catch the failure the next time it happens.

The word comes from control theory, where a system is observable if its internal state can be reconstructed from its outputs: a property of the system, not of the tools pointed at it. Software engineering borrowed the term for exactly that reason: a distributed fleet’s internal state, where a request is, what a cache holds, which instance is slow; cannot be inspected directly the way a single process can be stepped through in a debugger. The outputs are all there is, and the property is earned in what the fleet emits.

That is why observability is not monitoring rebranded, and also why the word alone buys nothing. The property comes from instrumentation breadth (every request emits, not only the errors), retained fidelity (the raw event with its identifying dimensions, not just pre-aggregated counts), and the ability to query all of it without pre-building the answer. Vendors sell observability suites, but the suite is the consumption layer; the property lives in the fleet. The test is brutal and cheap: after the next incident, ask a brand-new question about what happened. If it can be answered from what was already collected, the system was observable for that question. If the honest answer is “add a log line and wait for it to happen again,” that question was monitored, not observed, and the gap is the work.

The gap announces itself in the same scene every time. The alert fires, the dashboard is green, and the on-call engineer spends an hour asking five teams what their service saw, because the one question that would settle it, sliced by the dimensions that matter, was never registered anywhere. A fleet in that position has monitoring. What it does not have is the evidence to ask a follow-up, and the follow-up is where the incident actually gets solved.

What is observability in practice

Translated into engineering, the property decomposes into four requirements, and each is a decision. Breadth: instrumentation that emits on every request, not only on failure; the slow-but-successful path is where the interesting incidents live, and an instrumentation scheme that only fires on errors cannot see a fleet that is degrading politely. Context: identifiers that join records across process and network boundaries; a request ID that survives the hop from gateway to service to queue, so one journey can be followed at all; event-driven fleets know the failure mode by name, because “an event flow without end-to-end correlation is a ghost” and the archaeology it forces is the tax. Fidelity: the event kept raw, with its high-cardinality fields (user ID, request ID, instance, feature flag) rather than pre-collapsed into averages that cannot be un-averaged later. Retention: the evidence kept long enough for the question to arrive, because the question always arrives after the incident, usually in a postmortem, sometimes in an audit.

Cardinality is the honest limit running through all four. A field like user ID has as many distinct values as users; a field like region has a handful. Traditional monitoring infrastructure was built for the second kind: cheap to store, cheap to aggregate, impossible to slice by the first kind after the fact. Observable systems keep the first kind queryable somewhere, which is precisely the engineering trade the three pillars make differently: metrics bound cardinality to stay cheap, logs keep everything and pay in storage, and traces keep it per-request by construction. The trade is the subject of the comparison table below.

There is a budget on breadth, and pretending there is not is how telemetry programs die. Emitting on every request costs CPU on the critical path, network on the shipping path, and storage on the retention path; the honest design instruments everything and samples deliberately (keeping all of the anomalous and a chosen fraction of the ordinary) rather than instrumenting only what a committee could name in advance. The observable fleet is not the one that emits the most; it is the one whose emitted evidence can answer questions, which makes sampling policy, retention windows, and per-shape storage the actual design surface of this discipline.

What the property buys is speed in the moments that define operations: the difference between an answer in minutes and an archaeology dig across a fleet where correlation identifiers were never threaded through the steps. The fleet already knows what happened, every hop saw it. Observability is the discipline of making sure that knowledge survives the request, in a form the on-call engineer can query at 3 a.m.

The three pillars of observability

Telemetry data is the umbrella term for what an instrumented fleet emits, and it arrives in three shapes. Logs are discrete, timestamped records of events: the richest pillar, carrying arbitrary fields, full context, and human-readable messages; they are also the most expensive to store and the slowest to search at fleet scale. Metrics are numbers aggregated over time windows (error counts, request rates, latency distributions) cheap to store, cheap to query, and the natural home of alerting; they trade individual detail for fleet-wide trend. Traces are the causal path of one request as a tree of spans across services; the only pillar that can answer, for a distributed call chain, “where did the time go?”

One failing request shows all three at work. The metric registers the event: error rate ticks up on one endpoint, a latency percentile crosses a line. The log records the instance’s own account: the exception, the stack trace, the order ID, the tenant. The trace shows what neither can: the request spent most of its budget waiting on one downstream call, a fact about the journey visible only to the pillar that followed it. The API gateway, “often the richest observability point in the system,” sees every call cross one tier and can emit all three (the access log, the latency metric, and the tracing context) which is why this vocabulary pays off first at the fleet’s front door.

All three shapes share one pipeline, the same four stages regardless of pillar:

1. Instrument. Application and infrastructure code emits records; structured and uniform, carrying the context (request ID, service, version) the later stages will need to join.
2. Collect. A shipping layer (per-host agents, library-side batching, or both) moves records off the critical path, applying the sampling policy where the plan says to.
3. Store. Each shape lands where it queries best: a log store built for field search, a time series database for metrics, a trace backend for span trees, with retention set per shape, because costs differ by an order of magnitude.
4. Consume. Dashboards, alerts, and ad-hoc queries turn stored telemetry into answers, the layer where monitoring’s pre-registered questions and observability’s unregistered ones finally diverge.

The parts of this series each take one shape to full depth; logging owns the record and its search, metrics owns the aggregates and the alerting discipline, distributed tracing owns the journey, and this page owns the choice between them.

Which to use when: the comparison table

One decision table, on the axes that actually separate the pillars:

The reading of the table is that the pillars are not competitors but a division of labor with different cost structures. Metrics watch the fleet and ring the bell; the canary’s ramp runs on them, its abort criteria “written against” golden signals and time series, and the availability a fleet reports is a metric before it is anything else. Logs answer “what exactly happened?”; the record that turns an alert into a diagnosis, and the reason the dead letter queue is a safety net only if someone reads it. Traces answer the question neither can: where time and failures travel inside a distributed call chain. The honest sequencing is not a menu: metrics from day one, logs from the first debugging session, traces from the moment call chains cross service boundaries.

Dashboards, alerts, and the operating discipline

The telemetry is the supply side; the demand side is how a team consumes it, and that discipline decides whether the pipeline produces answers or noise. Alerts belong on symptoms, not causes. An alert on “checkout errors exceed budget” pages a human for a customer-visible fire; an alert on “queue depth above a threshold” pages a human for a number that may mean nothing. Cause-based alerts multiply until the rotation learns to ignore them, and a fleet that ignores its alerts has monitoring that is worse than none, it manufactures confidence. The circuit breaker states the general rule from its own narrow case: every state transition is “worth a metric and, for opens that persist, an alert,” and nothing else about the breaker is worth waking anyone for.

Budgets make alerts meaningful. A threshold means nothing without the level it protects: the SLA/SLO/SLI vocabulary turns “latency is high” into “this read class is violating its declared staleness budget”; the form replication-lag monitoring already uses, alerting against per-class budgets instead of generic thresholds. The same budget thinking is how a fleet attacks detection time, the slowest step in the RTO: the machines notice in seconds, the humans declare in minutes, and the gap is trained with drills that watch the clock. An alert that names the violated budget and the customer impact is a runbook’s first line; an alert that names a naked number is a habit the fleet has to unlearn.

Dashboards are for the knowns; exploration is for the unknowns. A dashboard is a snapshot of pre-registered questions; useful in an incident’s first minutes, useless in its second hour, when the question becomes one no panel anticipated. That is where the observability property cashes out: the engineer who can pivot from “the dashboard says checkout errors are up” to “show me the slow requests for this one provider since the deploy” is working against retained, queryable evidence: “the difference between watching a fleet and knowing what it is doing” that the deployment series named as its standing dependency. The honest measure of the whole program is the postmortem: every incident closes with “what could we not answer, and what would have let us?” A fleet whose action items say “instrument the thing we could not see” is compounding the property; a fleet whose action items say “add another alert” is growing the noise it will one day sleep through.

FAQ

Is observability just a rebrand of monitoring?
The word has been used to sell software, but the distinction it points at is real. Monitoring covers questions registered in advance; observability is the property that lets you answer unregistered ones from evidence already collected. Monitoring is an activity an observable fleet performs (the first consumer of the same telemetry) not a rival category. If a suite cannot answer a question nobody pre-built, it is monitoring with better branding.

Do we need all three pillars on day one?
No, and pretending otherwise stalls the program. Metrics come first, the fleet’s pulse and alerting surface. Logs follow immediately, because the first debugging session needs the record. Traces earn their cost when call chains cross service boundaries; before that, correlation IDs in logs are a workable substitute. What should exist from day one is the pipeline shape (instrument, collect, store, consume) so the pillars plug into it instead of being retrofitted.

How much telemetry should we keep, and for how long?
By shape and by value. Metrics are cheap enough to keep for years. Raw logs are the expensive pillar: sample the healthy traffic, keep errors and security events long, and let retention fall with age. Traces sample by default: a fraction of healthy traffic, all of the slow and failing. The budget that matters is being able to answer “what happened last Tuesday” for the requests that mattered, not for all of them.

Where do OpenTelemetry and vendor tools fit?
OpenTelemetry is the instrumentation standard (the code-side API, SDK, and wire formats) while vendors compete on storage, query, and dashboards. That division of labor is the strategic point: own the instrumentation layer and the storage layer stays replaceable. The tracing article covers the standard in depth, including where it fits before any vendor is chosen.

Related articles

• Next read: logging, series part 2: the record itself; structured logging, log levels, centralized logging, and the aggregation pipeline that makes a fleet’s logs searchable.
• metrics, series part 3: the fleet’s pulse; golden signals, RED and USE, time series data, and the alerting discipline that keeps pages honest.
• distributed tracing, series part 4: the journey: spans, context propagation, OpenTelemetry, and sampling for one request across many services.
• what is a distributed system; the phenomenon being observed: partial failure, and why a fleet’s state can only be reached through its outputs.
• fault-tolerant systems, the other half of the story: failure detection in protocol time, which this series builds on rather than repeats.
• blue-green vs canary deployment, the canary’s judge: the signals and telemetry that make a ramp decidable, and the abort criteria they carry.
• high availability: the architecture whose observable outcome this series measures, from the hunt for SPOFs to the failover budget.
• message queues; the dead letter queue, and why an unwatched one is an outage with a delay.

Last updated on 15 September 2026.