By Imran M ·
· 10 min read
Backup Strategies for Backend Systems: RPO, RTO, and Restore Testing
Backup Strategies for Backend Systems protect data you cannot recreate. Learn how to set RPO and RTO, then prove restores before a real incident hits.
Backup Strategies for Backend Systems exist so you can rebuild data you cannot recreate. When a disk dies, a bug deletes rows, or an attacker encrypts a volume, the backup is the product. If you have never restored it, you do not have a backup. You have a hope and a bill.
What It Is and Why It Fails
A backup is a copy you can restore later. A replica is a live copy that can fail with you. When people mix those two, they skip backups because the standby looks healthy.
Then a bad delete replicates, and both sides are wrong. Therefore keep a backup that lags on purpose, or that cannot be overwritten by the same credentials.
Two numbers set the design. Recovery point objective is how much data you can lose, measured in time. Recovery time objective is how long you can take to serve again.
When the point objective is a few minutes, you need continuous log shipping. When the time objective is many hours, a nightly snapshot may be enough. Also write both numbers down with product, because they are promises, not storage settings.
Backups fail in quiet ways. The job is green, and the files are empty. The job is green, and the files are in the same account as the attacker.
The job is green, and nobody has restored one in a year. A common mistake I have seen is a success metric that only checks that the dump process exited zero. It never opened the archive.
PostgreSQL backup and restore separates logical dumps from physical base backups. Logical dumps are portable and slow to restore when the data is large. Physical copies plus continuous archiving let you recover to a point in time. When you pick one, match it to the time objective, not to the tool you already installed.
What You Must Be Able to Lose
Not all data deserves the same copy. When a cache can be rebuilt, do not spend a tight point objective on it. When a ledger cannot be rebuilt, spend the money.
Also split secrets, object storage, and the primary database, because one policy will not fit all three. Then list the systems that have no backup and decide if that is acceptable.
Application consistency matters when a write spans a database and a queue. If you snapshot one and not the other, the restore will replay a ghost message or lose one. When you can, use a transaction boundary or a quiesce step.
If you cannot, document the repair you will run after restore. Still test that repair, because a note in a wiki is not a procedure.
Retention and Immutability
Retention is how long you can go back. When a bad deploy is noticed in an hour, a short window is enough. When a silent corruption hides for weeks, you need that age of history.
Also keep at least one copy that the production role cannot delete. If the same key can drop tables and drop backups, ransomware wins twice.
Store that locked copy in another account or project. When you use multi-region deployments, put a copy in another region too. A region loss should not take the only backup. Although cross-region copy costs egress, it is cheaper than rebuilding a ledger by hand.
Architecture and Implementation
A practical design has four parts. First, a scheduled full copy. Second, a stream of changes so you can roll forward.
Third, a catalog that says which copy is restorable. Fourth, a restore test that runs without a human hero. If any part is manual, it will be skipped on the week you need it.
Take the full copy from a standby when you can, so the primary does not stall. Then ship logs from the primary so you do not lose the last minutes. If the standby is delayed, your point objective is that delay plus the log gap.
Measure it. Do not assume the vendor default matches the promise.
Encrypt backups with a key the database host does not hold alone. When the host is compromised, the thief should not also get a readable archive. Also restrict who can restore, because a restore is a production write. A shared password in a chat log is not an access model.
Point-in-time recovery needs a tested target time. When support says the bad delete happened around a clock hour, you need logs that cover that hour. If log shipping stalled last night, the full copy is your only point.
Alert on log lag. A green full backup does not prove the stream is alive.
Restore as the Real Feature
- Pick a non-production account that can receive a restore.
- Automate the restore on a schedule, at least once a week for critical stores.
- Apply logs to a specific time, then run a read query the app would run.
- Check a row count or a checksum against a published expectation.
- Record how long the restore took, and compare it to the time objective.
- Page when the test fails or when it exceeds the time limit.
Trade-offs You Should Name
Frequent copies lower the point objective and raise cost. Fast restores lower the time objective and need more parallel hardware. Logical dumps are flexible when you must move versions.
Physical copies are faster when the data set is large. Also, immutable storage protects you from deletion and slows a normal cleanup. Choose with the promise, not with the storage price alone.
| Method. | Best when. | Main risk. | Restore shape. |
|---|---|---|---|
| Logical dump. | Use it when portability matters. | Slow if the data set is huge. | Rebuilds by replay. |
| Snapshot. | Use it when disks are large. | Crash consistency if writes were in flight. | Fast volume attach. |
| Log shipping. | Use it when loss must stay small. | A gap if shipping stalls. | Roll forward to a time. |
| Immutable copy. | Use it when deletion is the threat. | Cost grows with retention. | Also needs a normal restore path. |
Hot standbys feel like backups and are not. They cut the time objective for a host failure. They do not save you from a bad migration.
Therefore keep both. Failover architecture should promote a replica for a crash. Backups should cover delete, corruption, and account loss. If you only fund one, write down which disaster you are choosing to lose.
Pitfalls and Failure Modes
The worst failure is a backup that restores into a broken app. Schema and code must match the data. When you restore last week’s data onto today’s binary, the app may refuse to start.
Also keep the deploy artifact that matches the backup, or practice a forward migration as part of the test. Otherwise the time objective dies in a compatibility surprise.
- Trusting a job status that never reads the archive back.
- Keeping backups in the same account and the same keys as production.
- Letting log shipping lag past the point objective with no page.
- Restoring only the database and forgetting object storage or queues.
- Skipping a permissions test, so the restore role fails on the real day.
- Measuring backup duration and never measuring restore duration.
Partial restores cause split brain in the data. If you restore the user table and not the orders table, the product lies. When tables are related, restore a consistent set.
Then run a check that joins them. A single table drill is useful only if you know the others are untouched.
Clocks and names fail too. A backup labeled latest may be the latest failed attempt. When the catalog is wrong, you will restore empty data and call it success.
Therefore name copies by start time and by the last log they contain. Also alert when a new copy does not appear inside the schedule.
We once hit a bottleneck when the restore test shared a small disk with production backups. The test filled the disk, the backup job failed, and the alert was blamed on the test. Give the test its own quota.
If the test can break the backup path, it is not a test. It is an incident generator.
A Policy You Can Start From
The policy below is illustrative for a primary database. It takes a daily base backup, ships logs, keeps two weeks, and restores every week in another account. Change the minutes to your own point and time objectives.
If you cannot meet them in the test, change the promise or the design. Do not leave a number you have never hit.
store: orders-db
method: basebackup_plus_wal
full_every: 24h
wal_shipping: on
retain_days: 14
immutable_copy: second_account
copy_region: remote
restore_test:
cadence: weekly
target: isolated_account
checks:
- latest_wal_applied
- sample_read_query
- restore_duration
rpo_target_minutes: 5
rto_target_minutes: 60
alert_when: test_fails or wal_lag_exceeds_target
Hook the alert to the same on-call path as user incidents. A failed backup is a future incident, so it should not sit in a mailbox nobody reads. When the test exceeds the time objective, treat it as a design bug. Add hardware, parallelize, or loosen the promise in public.
Performance, Scale, and Cost
Backup windows steal disk and network. When a full copy runs during peak, users feel it. Run it from a standby, or during a known quiet hour, and still watch lag.
If the copy pushes the standby too far behind, failover will be stale. That is a reliability cost, not only a storage cost.
Retention is the main bill. A daily full copy for months, plus logs, can cost more than the primary. Also tier old copies to cheaper storage if the time to fetch them still fits a loose objective.
Keep the newest copy on fast storage so the time objective stays real. Cold storage for yesterday’s copy is a false economy when you must be back in an hour.
At large size, restore time dominates. A dump that takes a day misses almost any time objective. Therefore use physical copies, parallel restore, and a rehearsal that measures the clock.
If the data set doubles, rerun the test. Last year’s duration is not this year’s duration.
Track cost per restorable day, and track the last successful restore age. Those two numbers tell you if the strategy is alive. SLOs for backend engineers can include a backup freshness objective for critical stores.
A user SLO will not notice a missing backup until the day you need it. The freshness objective will.
Finally, delete on purpose. Infinite retention raises cost and raises legal risk. When the policy says 14 days, enforce 14 days on the immutable copy’s lock too.
An orphan disk in an old account is not a strategy. It is an unknown.
Key Takeaways
- Set a point objective and a time objective with product, then design to them.
- Keep a copy the production keys cannot delete.
- Ship change logs if you cannot lose the whole day.
- Restore on a schedule and time the restore, not only the backup job.
- Match application versions to the data you bring back.
- Alert when log lag or a failed test breaks the promise.
FAQ
Is a replica a backup?
No. A replica follows deletes and bad writes. It is for failover, not for history.
Also keep a backup that can return to a time before the mistake. When you have both, you can survive a crash and a bad migration.
How often should you test a restore?
Test critical stores at least weekly, and after any change to the backup job. Also test when the data size has grown a lot. If the test is only a yearly drill, the first real restore will be the drill. You do not want that.
What should the restore test check?
Check that logs applied, that a real read works, and that the clock time met the objective. A file that exists is not enough. When the app cannot query the restored data, the user is still down.
Should backups include queues and files?
Yes, when those stores hold state you cannot rebuild. A database restore beside an empty bucket is a partial product. Also document the order. If the queue must drain after the database is up, put that step in the test.
Write the point objective and the time objective for your primary store. Then schedule a restore in an isolated account and record the duration.
If the test misses the time objective, change the design or the promise before the next launch. When the test passes, alert on log lag so the next quiet failure does not wait for an incident.
Last updated on 20 September 2026.
