Disaster Recovery: RTO, RPO, and Getting Back Online

Disaster recovery is the practice of restoring service and data after a failure too large for the availability architecture to absorb: a lost region, a corrupted dataset, an account nobody can log into. It is planned around two numbers (how long until the system serves again, and how much recent data the gap is allowed to cost) and it is proven only by the drill that runs it.

The boundary with high availability is the one that article drew: its architecture absorbs component failures (redundant nodes, failover, budgets) so that a dead machine is a capacity event. This page owns the scenarios that defeat that architecture, where the component is not the unit of failure but the whole site, the whole dataset, or the whole account, and where durability, not availability, is the property on trial. Availability keeps the lights on; disaster recovery assumes the lights are out and plans the relight.

What is disaster recovery

The events this page plans for are the ones that defeat component-level thinking. The region is unreachable (power, flood, fiber, or the provider’s own bad day) and every redundant node in it is redundantly dead together. The data is gone or poisoned (a deletion bug with an id in the wrong place, a migration that corrupted what it touched, an encryption event with someone else’s public key) and here is the trap that separates DR from plain failover: replication replicates mistakes as faithfully as it replicates data, so the redundant copy is exactly as destroyed as the primary, just later. And the quietest category, the self-inflicted: the deploy, the script, the configuration change that no hardware failure resembles. A disaster is any event that removes the system’s ability to serve faster than the availability architecture can promote a survivor.

That makes DR a different discipline from high availability, not a bigger version of it. High availability optimizes the mean time to repair for components inside the architecture’s coverage: a node dies, a survivor is promoted, minutes pass. Disaster recovery plans for the coverage itself failing, and its honest starting point is that the plan is insurance against events that will probably never happen to your specific system, while the events that actually happen most (the data-destroying ones, the self-inflicted ones) are exactly the ones teams underprepare for, because they feel like embarrassments rather than disasters. The discipline treats them identically: an event that puts the lights out, a relight that has been rehearsed.

Everything in the field organizes around two questions, and they are business questions before they are engineering ones. How long until we are serving again?: priced in revenue, contracts, and trust. How much recent data may the gap cost?; priced in what a lost hour of orders, or a lost day, is actually worth. Those two answers (the recovery time objective and the recovery point objective) size the entire plan: the strategy tier to run, the infrastructure to pay for, and the drills to run. The next section defines them precisely, because every dollar and every hour in the rest of this article is spent against them.

RTO and RPO

The two numbers are defined precisely because the whole plan is spent against them. The recovery time objective is the maximum acceptable time between the disaster and restored service: the answer to “how long until we are serving again,” agreed in advance, priced by the business. The recovery point objective is the maximum acceptable data loss, measured backward from the moment of the disaster: the answer to “how much data did the gap cost.” An RPO of fifteen minutes means fifteen minutes of writes may vanish and the business survives the sentence; an RTO of one hour means the relight (including the decision to relight) must be over inside sixty minutes. They are objectives: design targets the architecture is built to meet and the drill is run to verify, not properties that exist because a document asserts them.

RPO is set, in practice, by the cadence of the copies. Asynchronous cross-region replication gives you an RPO floor equal to the replication lag at the moment of failure; seconds to minutes on a good day, and lag is worst exactly when the system is struggling. Log-based point-in-time recovery, continuous backups from the write-ahead log; shrinks it the rest of the way and, crucially, restores to a point before the mistake, which is the only answer to the poisoned-data disaster that replication alone cannot give. Scheduled snapshot backups set it to the snapshot interval: an RPO of “yesterday’s backup” is a sentence a business signs only under duress. And an RPO of zero across a region requires synchronous cross-region commits; a trade with a price in write latency and a failure mode in availability that the replication article prices honestly: the tighter the copy, the further away the commit must reach.

RTO is set mostly by things that are not computers. Detection: how long before the event is even known to be a disaster; the detection machinery is the observability stack’s job, and monitoring and observability own its mechanics. Decision: who declares the disaster and on what evidence; the slowest, hardest step, because declaring is expensive and embarrassing if wrong, and the runbook that names the decider and the criteria in advance is worth more than any hardware in the plan. Provisioning and restore: bringing the alternate up from whatever state the strategy tier keeps it in. Validation: proving the recovered system is truthful before pointing customers at it. The computers in that list are the fast parts; the humans and the process are where the RTO actually goes, which is why the last section of this article is a drill and not a diagram.

Disaster recovery strategies

The strategies form a ladder, and each rung is the same trade: more money standing by, for less time and less loss when the day arrives. The base rung is backup and restore: copies of the data stored away from the system (ideally immutable and offline, on the three-copies-two-media-one-offsite pattern) and a restore procedure that provisions a new home and rebuilds. It is the cheapest rung and the slowest: RTO in hours to days (provision, restore, validate, in that order, each one slower than the diagram suggests) and RPO equal to the backup interval. Its one unforgivable failure mode is not cost but assumption: a backup that has never been restored is a hope, and the restore test is what converts it into a strategy.

One rung up, pilot light: the data is already living in the second region, continuously replicated, with a minimal core of infrastructure kept warm (the data stores and just enough plumbing to reach them) while everything else is provisioned on demand. The relight is hours instead of days, because the slowest part (the data) is already there and the rest is automation. The cost step is real but modest: storage and replication, a little compute, and no idle fleet. Pilot light is the honest middle for systems whose RTO is measured in hours and whose RPO is measured in minutes of replication lag.

The next rung is warm standby: a complete but scaled-down copy of the system, continuously replicated and ready; scale it up, point the traffic at it, and it serves. The relight is tens of minutes to an hour, and the copy’s size is the cost: a second fleet running at reduced capacity, burning money in daylight for speed you hope never to need. This is the tier where the RTO objective stops being a hope and starts being a design parameter: the scale-up time is measured, not estimated.

The top rung is active-active multi-region: both regions serve traffic all the time, a region failure costs capacity instead of service, and the “recovery” is a routing decision measured in minutes. It buys the best numbers on the board (RTO near the detection-and-decision time, RPO at the replication lag) and it is priced accordingly: roughly double the fleet, the full complexity of cross-region data placement, and a consistency story that must be true, because an active-active system that cannot explain its own writes is not a DR strategy but a distributed bug. The mechanics of that top rung (and of region failover at any tier) are the next section.

Multi-region failover

Region failover begins with the same move as every other failover (send the traffic to a survivor) but at planetary scale the routing itself is the design. The steering layer is global: DNS with health-based policies, or a global balancing tier that holds the map of which region is serving, and the switch it makes is load balancing with the whole planet as the back end. The honest trap is that the switch is advisory for as long as caches believe the old answer: DNS time-to-live values decide how long the internet keeps knocking on the dead region’s door, and the TTL is a standing trade between failover speed and query load that must be chosen before the disaster, not during it. Clients in the switching window see errors that look like outages, and the well-behaved ones (pacing their retries instead of stampeding) are what keeps the recovering region from being greeted by a wall.

The data is the harder half. Cross-region replication is asynchronous by default, and the lag is the RPO floor; the writes that were in flight when the region died are the writes the recovery loses, and nothing downstream can reduce a loss already taken. Shrinking it means either synchronous commits across the distance (the write now pays the round trip between regions, an availability-and-latency trade the replication article prices) or a data placement that keeps each user’s writes pinned to their home region and replicates for read, which converts the loss window into a boundary instead of a hope. And the failing region must be assumed to still be running: the region that comes back (power restored, network healed, processes alive) re-enters with its unreplicated writes and its stale beliefs, and the defense is the same fencing discipline consensus systems use to depose a leader: the returning region must be ignorable until it has been reconciled, or the disaster’s second act is split-brain at fleet scale.

The trigger decision is where multi-region plans are actually won or lost. The criteria are written in advance: multi-source health evidence (the region is failing from the outside, from the inside, and from its own services’ point of view, because any single view can be lied to) and a named human with the authority to declare, on evidence that would be embarrassing to be wrong about, that the region is gone. The asymmetry that makes it hard: failing over when the region was merely slow is an own-goal (the entire fleet’s worth of the “failover that becomes its own incident,” now with a region’s worth of in-flight work as the casualty) so the evidence bar is deliberately high and the decision is deliberately slow. And the path back (failback, the return of traffic to a healed region) is itself a migration, complete with its own risks, which is why the runbook drills it in both directions.

The runbook and the drill

A recovery plan is an executable document, not prose. The runbook is the recovery as an ordered list: what to check first, who declares, which steps run automatically and which need a human, where the backups live and how to reach them when the primary console cannot be reached: every step with an owner, a command, and a verification. Automation is the difference between an RTO of sixty minutes and six hours, because the steps that run without a human are the steps that do not wait on a war room; the steps that still need one are the ones the runbook names people for, in advance, with deputies. And the runbook decays as the system changes (every new dependency is a new paragraph that is not there) so it is versioned with the system and reviewed on the same schedule as the architecture.

Then the drill, which is the only part of disaster recovery that proves anything. The ladder of rehearsal: the table-top, where the team walks the runbook against a scenario and finds the steps that do not exist; the partial restore, where a backup is restored into a clean environment and the data is validated; and the full failover, run on schedule, in daylight, with the clock running; the region is declared gone, traffic is moved, and the recovery is measured the way it will be judged. The gaps the drill finds are always humbling and never hypothetical: the credential that expired, the config that lives only in the dead region, the step that takes four hours because it was automated by someone who left. Every one of them is a discount on the disaster’s bill: paid in daylight, instead of the disaster charging full price.

The metric that ends the article is the honest one: publish the measured drill numbers, not the document’s claims. The gap between the RTO in the plan and the RTO in the last drill is the system’s true recovery capability, and closing that gap is the whole discipline; the strategy tier buys infrastructure, but only the drill buys recovery. Disaster recovery is insurance against a day that will probably never come, priced in infrastructure that idles and rehearsals that cost a working day, and the premium is the point: the event you did not prepare for costs the business; the preparation you never test costs only the money, until the day it is the business.

FAQ

What is the difference between RTO and RPO?
RTO is time; RPO is data. The recovery time objective is the maximum acceptable gap between the disaster and restored service, how long the business tolerates being dark. The recovery point objective is the maximum acceptable data loss, measured backward from the disaster, how many minutes or hours of recent writes the business tolerates losing. The two are independent: a system can be back in an hour but missing a day of orders, or current to the second but down for two, and every strategy and every dollar in the plan is spent against one axis or the other.

What is the difference between disaster recovery and high availability?
High availability absorbs component failures inside the architecture’s coverage: a node dies, a survivor is promoted, the budget spends minutes. Disaster recovery plans for the failures that defeat that coverage; the lost region, the destroyed dataset, the event where there is no survivor to promote. HA is the discipline of staying up; DR is the discipline of coming back. The boundary is the one high availability drew: it ends where the question stops being “which node survives?” and becomes “how long until we serve again, and at what cost in data?”

What are the main disaster recovery strategies?
The ladder, in cost and speed order: backup and restore: copies stored away, rebuilt on demand, cheapest and slowest; pilot light: the data replicated and a minimal core warm in the second region, relight in hours; warm standby: a scaled-down full copy, scaled up in tens of minutes; and active-active multi-region, both regions serving, where a region failure costs capacity instead of service. Each rung buys a faster RTO and a tighter RPO at a higher standing price, and the correct rung is whichever first meets the two objectives the business actually signed.

How is RPO achieved in practice?
By the cadence of the copies, nothing else. Asynchronous replication sets the floor at the replication lag at the moment of failure; log-based point-in-time recovery shrinks it to seconds and adds the ability to restore to before a mistake; scheduled snapshots set it to the snapshot interval and should be reserved for data the business can afford to lose wholesale. An RPO of zero across a region exists only as synchronous cross-region commits: a real purchase, priced in write latency. The RPO is not a wish: it is exactly as good as the slowest copy in the chain.

How often should a disaster recovery plan be tested?
On a schedule, with the cadence scaled to the tier: table-top exercises frequently and cheaply, partial restores (a backup validated in a clean environment) quarterly or so, and a full failover drill at whatever frequency the business can buy, with the clock running and the numbers published. The plan’s claimed RTO is a hypothesis; the last drill’s measured RTO is the capability. A plan that has never been tested is not a plan but a document, and the gap between the two is measured only once, during the event.

Related articles

• Next read: blue-green vs canary deployment; the last resilience question in the series: shipping changes without betting the fleet; the release strategies that stop deploys from being the self-inflicted disaster this article keeps finding.
• high availability; the pillar this page builds on: the component-level architecture that makes single points of failure rare, before the region-level question ever arrives.
• availability, reliability, and durability (the definitions anchor, with durability) the property actually on trial when the data, not the machines, is the casualty, precisely separated from its siblings.
• database replication, the mechanism under every RPO: replication topologies, synchronous versus asynchronous, and the lag that sets the loss floor.
• monitoring and observability; the detection half of the RTO: knowing the region is failing, from multiple points of view, before the failover decision can even be made.
• load balancing, the routing half of failover: health checks and traffic steering, at fleet scale and at planet scale.
• fault-tolerant systems; the component-level coverage this page plans for losing: redundancy, failover, and failure detection as mechanisms, so DR can plan for their defeat.