Schema Versioning Strategies: Backward Compatibility, Expand-Contract, and API Evolution

Schema Versioning Strategies matter because writers and readers do not upgrade in one instant. A new field, a renamed column, or a tighter type can break a consumer that is still on the old shape. You need rules for what can change while both sides stay up.

What Schema Versioning Is

Schema versioning is the set of rules for how a data shape changes over time. The shape might be a table, a JSON body, a queue payload, or a protobuf message. The rule answers one question. Can the old code still read what the new code writes, and the other way around?

Backward compatible means new writers still make sense to old readers. Old readers ignore fields they do not know, and they still find the fields they need. Forward compatible means old writers still make sense to new readers. New readers tolerate a missing field.

Most rollouts need both, for a while. During a deploy, new pods and old pods run together. A queue also stores old messages. If you only handle one direction, the other direction fails in the middle of the release.

A version number in the payload does not replace those rules. A number tells you which reader to pick. It does not make an unsafe change safe.

Use the number when you must break compatibility. Do not bump it for every new optional field.

Why It Fails in Production

The usual break is a rename disguised as a cleanup. You change status to order_status and deploy the writer. Old readers look up status, get nothing, and drop the message or store a null.

The writer logs success. The data is wrong downstream.

I have seen a team change a field from a string to an object. New code wrote the object. An old worker parsed the string and threw it away.

The retry queue then filled with poison messages. The fix was to write both shapes until every worker was new.

Database and API schemas drift apart when teams version only one of them. The column moves, and the JSON key moves in the same release. Mobile clients still send the old key.

The server rejects the body. API versioning strategies should move slower than an internal column rename.

Consumers you do not own make this worse. A partner, a warehouse job, or a BI tool can read a table you thought was private. Contract tests against your own services will stay green.

The partner fails on Monday. Publish the compatibility rules, and give unknown consumers a date before you remove a field.

How Expand and Contract Works

Expand adds the new shape beside the old one. Add a column, a JSON key, or a protobuf field with a new number. Keep the old field.

Writers that you have shipped fill both. Readers still on the old build keep working.

Then backfill, and switch readers to the new field. Do this only after every writer fills the new field. Otherwise a new reader sees gaps and invents a default that is wrong.

Defaults are part of the contract. Pick them on purpose.

Contract removes the old field last. Wait until metrics show no reader needs it. Wait until a rollback of the app would still be safe, or accept that rollback cannot cross this step. Database migrations at scale use the same three beats on tables.

Protobuf makes the field rule concrete. You add fields. You do not reuse a number.

You reserve numbers and names when you remove a field, so an old payload cannot be read as a new meaning. The Protocol Buffers language guide states those update rules.

Compatible and breaking changes

Adding an optional field is usually safe. Removing a field, changing a type, or tightening a range is not. Making a required field out of an optional one is a break for every old writer. Google’s AIP on backwards compatibility lists changes that are safe for public APIs and changes that are not.

Enums need care. A new enum value is a break for a reader that rejects unknown values. Prefer readers that keep the raw value and treat unknown as a named unknown. Then you can add values without a flag day.

HTTP caching and conditional requests have their own version story. ETags and validators are defined in RFC 9110. If you change a representation, change the validator. Otherwise a cache can serve the old body to a client that asked for the new shape.

Trade-offs You Should Weigh

Compatible evolution is slower to design and safer to ship. You carry two fields and a dual write. Storage grows a little.

Code stays branchy until contract. For a busy API, that cost is smaller than a client outage.

A hard version bump is clearer when the change cannot be bridged. You publish v2, run both, and move clients on a schedule. You pay for two stacks.

You must still define when v1 dies. A bump with no sunset is how you collect versions forever.

A schema registry can reject an incompatible publish. That is a strong gate for events. It is only as good as the compatibility mode you set.

Full transitive checks catch more breaks and also block some safe cleanups. Pick the mode with the consumers you actually have.

During rolling deployments, half the fleet may still be old. Any strategy that needs a flag day will fail in that window. If you need a flag day, drain the queue, stop the old binary, and then switch. Do not do that on a path that must stay up.

Pitfalls and Failure Modes

First, dual write without a single order will diverge. One path writes the new field and fails before the old field. Readers then disagree.

Write both in one database transaction when they share a row. If they do not, pick a source of truth and derive the other.

Second, unknown-field behavior must be a test, not a hope. Some JSON libraries drop unknown keys on write-back. A reader that loads, tweaks one field, and saves can erase the new key.

That is a silent contract break. Use a patch or an explicit field list.

Third, reusing a field number or a column meaning is worse than a new name. Old data now means something new. You cannot tell a real new value from residue.

Reserve retired names. Add a new field instead of recycling.

Fourth, defaults can hide missing backfills. A new reader treats null as zero and places a real order. The zero was “not migrated,” not “free.” Use a distinct unknown state until the backfill finishes. Do not overload a valid business value as the default.

Fifth, rollback after contract is a data loss event. Rollback strategies for backend systems should say which schema steps are reversible. If the old binary cannot boot on the new schema, you cannot roll the binary back. Delay contract until the previous binary is out of the rollback window.

A Practical Dual-Write Example

The update below fills the old status and the new status in one statement. Both stay in sync for this writer. An old reader can still select the old column.

A new reader can select the new one. Remove the old column only in a later change.

UPDATE orders
SET status_text = 'paid',
    status_v2 = 'paid'
WHERE order_key = 42
  AND status_text = 'pending';

If this update changes zero rows, someone else already moved the order. Do not write the new column in a second statement that ignores the predicate. You would race and store a status that lost. A common mistake I have seen is a dual write split across two requests with no shared check.

Release sequence

Ship the reader that understands both shapes before you ship the writer that sends the new shape. Then you can roll forward or back one side at a time. Reverse that order and the old reader meets a payload it rejects.

1. Add the new field without removing the old one.
2. Deploy readers that accept both shapes.
3. Deploy writers that fill both fields.
4. Backfill stored rows and queued messages.
5. Remove the old field only after usage hits zero.

Performance, Scale, and Cost

Dual writes add a small amount of CPU and log traffic. On a hot row, the extra column is cheap next to the lock or the index you already pay for. The expensive part is a backfill of history. Throttle it, or replica lag becomes the outage.

Payload size grows when you keep old and new fields. For events, that can push you over a message cap. In an illustrative production range, a few extra short fields are noise, while embedding a second full copy of a large document is not. Drop the duplicate as soon as consumers move.

A registry check on every publish adds latency and a hard dependency. Cache the schema in the producer. Fail open or fail closed on purpose.

Fail closed stops bad data and can stop the pipeline when the registry blips. Fail open protects availability and lets a break through. Choose with the cost of bad data in mind.

Versioned APIs cost more in tests and docs than in CPU. Each live version needs a contract test and an owner. Three live majors usually mean two of them are unfunded.

Sunset aggressively. The storage you save is minor next to the engineer time you get back.

Scale of consumers is the real limiter. One extra consumer that pins an old field can block contract for months. Track field usage in logs or traces.

A field with no reads is a candidate. A field you cannot measure is not safe to drop.

Key Takeaways

• Assume old and new code run at the same time.
• Add fields and dual write before you remove anything.
• Do not reuse field numbers, names, or meanings.
• Treat unknown enum values as data, not as errors.
• Test that a read-modify-write does not drop new keys.
• Delay contract until rollback no longer needs the old shape.
• Bump a major version only when you cannot bridge the change.

FAQ

Do you need a new version for every field?

No. An optional field that old readers ignore does not need a new major. You need a new major when you remove, rename, or retarget a field that existing clients require.

Extra version numbers teach clients to ignore your process. Keep the number stable while the change stays compatible.

Which direction matters during a deploy?

Both. New writers meet old readers, and old writers meet new readers, while pods overlap. Queues add time, so old payloads arrive after the deploy looks done. Stay compatible in both directions until the queue is drained and the old binary is gone.

Should internal tables follow the same rules?

Yes when more than one service or job reads them. A table with a single writer and a single reader can change faster, and only if you deploy them together. The moment a report or a second service appears, you are back to expand and contract. Assume that moment will come.

How long should dual write last?

Long enough to deploy every reader, finish the backfill, and cover your rollback window. That is often more than one release. It should not be permanent.

Put a date on the contract step and assign an owner. A dual write with no end becomes the next surprise break.

Pick one schema you plan to change and mark each change as compatible or breaking. For the compatible path, add the field, ship tolerant readers, then dual write, and only later remove the old field. For a breaking path, ship a new major and a sunset date. Do not start the writer until the readers that must survive are already live.

Last updated on 20 September 2026.