By Imran M · 08/08/2018 · 10 min read

Terraform in Production: Modules, State, Workspaces, and Drift Management

Terraform in Production covers modules, state, workspaces, and drift. Learn how state locks fail, how to split stacks, and how to plan changes safely.

Executive Summary: When Terraform state is sloppy, a completely normal-looking apply can destroy the wrong object, which is why state has to be treated as production data with the same care as a database, not a local file someone can regenerate. This guide covers structuring modules and splitting stacks so a bad plan has a small blast radius, the state-locking failures that corrupt a team’s shared infrastructure, and reading a plan closely enough to catch a destructive change before you type yes.

Terraform in Production is the practice of planning, applying, and locking real infra from reviewed code. When state is sloppy, a normal apply can destroy the wrong object. Therefore you should treat state and modules as production data, not as local files.

What It Is and Why It Fails in Production

Terraform records what it created in a state file. The next plan diffs that memory against the code and against the live API. If the memory is wrong, the plan is wrong, even when the code looks fine.

In my experience, Terraform fails in production for a few plain reasons. First, state lives on one laptop and that laptop is the only copy. Then, two applies run at once and corrupt the file. Also, a module update forces a replace that nobody saw because the plan was too long to read.

A common mistake I have seen is a workspace used as an environment. Prod and staging share one code path and one mistake away from the wrong credentials. Because the workspace name is easy to miss, a plan for staging was applied to prod. After that day, we split accounts and split state so the wrong place was hard to reach.

Another failure is a provider upgrade in the same change as a resource edit. The plan mixes noise and danger. When something breaks, you cannot tell which move did it.

Still, the apply was green. So split tool upgrades from infra edits.

You also fail when drift is ignored for weeks. A manual rule sits in the cloud. The next apply removes it during an unrelated fix.

The security hole returns, or a needed exception vanishes. Detect drift on a schedule, and resolve it before you stack more changes on top.

Architecture and Implementation

Use a remote backend, a lock, and one state per stack. First, bootstrap the backend with a tiny trusted config. Next, point every stack at it. Then, run plan and apply from CI.

Finally, alert on drift. The Terraform state docs explain what the file holds. Read them before you copy state to a chat paste.

State, locks, and backends

Remote state should be encrypted, versioned, and locked. A versioned bucket lets you recover a bad write. A lock stops two runners from applying together.

If the lock table is missing, you do not have a lock. You have a hope.

Restrict who can read state. It often contains secrets and resource IDs. Give CI a role that can lock and write.

Give humans read in a break glass role. Do not hand the whole team the admin key because apply is annoying.

Never commit state to git. It will leak, and it will go stale. If you already did, rotate anything inside it, then purge the file from history. Your infrastructure as code layout should make the remote backend the default so a new stack cannot forget.

Modules that stay reviewable

A good module has a small input set, a version, and outputs that other stacks need. The Terraform modules docs show the syntax. The design is the hard part. If callers must read the module source to use it safely, the API is wrong.

Pin versions with a range you mean. A pin of main is not a pin. When you publish a breaking module change, bump the major version. Callers should see a failed plan in a branch, not a surprise in prod.

Do not put provider credentials inside a module. Pass providers from the root. If a module opens its own provider with a copied key, you will leak that key and you will hide which account you target.

Workspaces and accounts

Workspaces store many states for one config. The Terraform workspaces docs describe the feature. They fit short lived copies, such as a preview stack with the same shape. They are a weak way to split prod and staging.

Prefer separate state keys, separate accounts, and separate credentials for prod. Then a staging apply cannot reach prod even if the code is the same. If you use workspaces, print the workspace name in the CI log in a way a reviewer cannot miss.

How to Run Plan and Apply

Plan on the pull request. Post the plan summary where the reviewer sits. Apply only after merge, from CI/CD pipelines, with the same code that was planned.

If the branch moved, plan again. A stale plan is how you apply a change you did not review.

Fail the pipeline when the plan replaces a database, a bucket with data, or a cluster. Allow an override only with a named approver. Your rollback strategies should say that some applies cannot be undone without a restore.

Open a pull request with a small stack change.
Read the plan, not only the git diff.
Block applies that replace stateful resources.
Apply from CI with a lock after merge.
Re-plan if drift appears before the apply starts.

Trade-offs and Comparison

Terraform is strong when you want one language across clouds and a clear plan. It is awkward when a vendor feature is too new for the provider. You can use a narrow native tool at that edge. Do not fork the whole stack to chase one flag.

A monolith state is simple until it is not. Many small states are more files to wire up. The trade is lock time and blast radius. Small states win once more than one team applies changes.

Choice.	When to use it.	What you give up.
Local state.	A toy you will delete today.	No lock, no backup, no team use.
One remote state.	A single owner and a small graph.	A global lock and a large blast radius.
State per stack.	Several teams and real prod.	More backends to wire and to audit.
Workspaces.	Short lived copies of one shape.	Easy to apply to the wrong copy.

Choose state per stack for prod. If you are tempted to use one workspace per customer, check the plan time and the blast radius first. A bug in a shared module can still touch every customer. Isolation has to be real, not only a name.

Pitfalls and Failure Modes

A stuck lock after a killed CI job blocks everyone. The apply may still be running. Before you force unlock, check the runner.

If you unlock during a live apply, you can corrupt state. Then the recovery is a careful state surgery, not a rerun.

State rm and import are sharp tools. They fix adoption and they cause orphans. Use them in a change that plans to no difference afterward.

If you remove a resource from state and also delete it from code, the next apply will not destroy the live object. You now have a leak and a bill.

Ignore_changes hides drift forever. It is valid for a field that a controller rewrites, such as a scaling target. It is not valid as a way to avoid a hard plan.

Review every ignore. Put a comment that names the writer. Otherwise the next owner will not know why the code does not match live.

Provider bugs and default changes ship inside a version bump. Pin the provider. Read the upgrade notes.

Apply the bump alone. If you combine it with a new resource, you will not know which part failed.

Kubernetes resources in Terraform fight with controllers that also write those objects. The plan shows a perpetual diff. People then ignore all diffs.

Manage the cluster and the node pools here if you must. Leave app workloads to the app deploy. Your Kubernetes scheduling specs change faster than the network, and they do not belong in the same state.

Partial apply leaves a mix of old and new. Terraform marks the error, but some objects already changed. Read the error, fix forward or revert the code, and plan again. Do not assume a failed apply means nothing changed.

A Practical Backend and Module Pin

The sketch below locks state in a versioned bucket and pins a module. It does not target prod and staging with one workspace flag. When you bump the module, do it in its own pull request so the plan is only that bump.

terraform {
  required_version = "1.9.0"
  backend "s3" {
    bucket         = "example-tf-state"
    key            = "prod/network/terraform.tfstate"
    region         = "us-east-1"
    dynamodb_table = "example-tf-locks"
    encrypt        = true
  }
}

module "network" {
  source  = "git::https://example.com/modules/network.git?ref=v1.4.2"
  cidr    = "10.4.0.0/16"
  name    = "prod"
}

Note what this sketch does not do. It does not use a workspace to mean prod. It does not float the module at main.

If the plan shows a replace, stop. Also keep required_version tight so CI and laptops do not produce different plans from different language behavior.

Performance, Scale, and Cost

Plan time is an API walk. Large graphs and slow data sources dominate. A stack that takes more than about 10 minutes to plan, an illustrative production range, will stall reviews.

Split it. Avoid data sources that list the whole account on every plan.

State file size affects refresh and the blast of a corrupt write. Huge state is also a juicy secret store. Smaller files are faster to lock and safer to restore from version history. Pay the wiring cost up front.

At scale, the lock is a social tool as much as a technical one. If applies queue for an hour, teams will apply from laptops and you will lose the audit. Split state until the queue is short. Keep a fast lane only if it still plans the same way.

We once hit a bottleneck when every service module refreshed the same shared data source. Plans that should have been seconds took many minutes, and CI rate limited the cloud API. The fix was to pass the ID in as an input from the stack that owns it. After that, app plans stopped listing the world.

Drift detection should be cheaper than a full human review each day. Run a scheduled plan and alert only on non empty diffs for sensitive stacks. A noisy drift channel gets muted. Then the real public bucket hides in the noise.

Count the cost of a bad apply, not only the CI minutes. One replaced database dwarfs a year of runner time. Spend review time where the plan shows destroy or replace. Skim the pure adds with less fear, but still read them.

Key Takeaways

Remote, locked, versioned state is the minimum for prod.
One state per stack beats one state for the whole company.
Do not use workspaces as your only wall between prod and staging.
Pin modules and providers, and upgrade them alone.
Fail plans that replace stateful resources unless a named person overrides.
Treat drift as a defect. Fix the code or revert the click.
Apply from CI so a laptop cannot race the lock.

FAQ

Should I store prod and staging in one state?

No. Separate state and separate credentials. A shared state means a shared blast radius and an easy mix up.

You can still share module source. The memory of what exists should not be shared.

When are workspaces a good fit?

Use them for many short lived copies of the same small config, such as preview stacks, if the credentials cannot touch prod. Do not use them as the only split between prod and staging. If a typo can apply to prod, the design is wrong.

How do I recover a bad state write?

Stop applies. Restore the previous object version from the backend if you have versioning. Then plan and see what the live world still holds.

If objects were created or destroyed, fix those by import or by a careful follow up. Do not keep applying on top of a confused file.

What belongs in ignore_changes?

Only fields that another system must own, such as a desired count that an autoscaler sets. Document the owner. If you ignore a field to silence a diff you do not understand, you will miss a real change later. Fix the understanding first.

Terraform in Production stays safe when state is locked, stacks are small, and plans are read. Pick one prod stack. Next, move it to a versioned remote backend and pin its modules. Then turn on a drift plan that fails loudly when the live world disagrees with the code.

Last updated on 12 September 2026.