Encryption at Rest: Key Management, Envelope Encryption, and Compliance

Encryption at Rest matters because a stolen disk, snapshot, or backup should not be a full copy of your customer data. When the key sits beside the ciphertext, the copy is still readable. In my experience, the volume was encrypted and the backup bucket was not. Therefore, you should encrypt every copy, split data keys from master keys, and control who may unwrap them.

What encryption at rest is and why it fails

Encryption at rest means stored bytes are ciphertext when the process is not using them. Disks, database files, object buckets, and backups all count. Also, a column of national id numbers can be encrypted even when the rest of the row is not. If you only encrypt the laptop disk, the warehouse export is still plain.

The math is rarely the failure. AES is a standard block cipher, and NIST FIPS 197 specifies it. A common mistake I have seen is a data key written into the same bucket as the objects it protects.

Because anyone who can read the bucket can read the key, the wrap was cosmetic. Still, the audit checkbox said encrypted.

Another failure is a shared key for every tenant. One unwrap then opens every customer. If a debug tool dumps that key into logs, the blast radius is the whole table. Consequently, bind keys to a tenant or a data class, and treat the key policy as part of the design.

Replicas and snapshots forget the setting. You turn on encryption in the primary region, then a cross-region copy lands in a bucket with default settings. When a contractor pulls that copy, the primary control does not apply. Furthermore, test restores belong in the threat model, because a restore host often has weaker access rules.

Architecture and how you implement it

Use envelope encryption. A data encryption key, the DEK, encrypts the payload. A key encryption key, the KEK, wraps the DEK.

The KMS holds the KEK and never needs to see the payload. Google Cloud envelope encryption documents this split, and the AWS KMS concepts guide uses the same idea with customer master keys.

First, generate a random DEK. Next, encrypt the plaintext with an authenticated mode such as AES-GCM. Then, ask the KMS to wrap the DEK.

Finally, store the ciphertext, the wrapped DEK, and the key id together. On read, unwrap the DEK, check the tag, and only then use the plaintext. Drop the DEK from memory when the request ends, or cache it briefly under a tight scope.

Who may call unwrap is an IAM problem. Pair the key policy with IAM roles for backend services so the app role can decrypt and a human cannot. Also, an admin who can disable the key should not be the same role that serves traffic. Since the KMS audit log is the record of unwraps, keep it even when you trim other logs.

Where to apply the cipher

Disk encryption stops a raw disk theft. It does not stop a process that is allowed to mount the disk. Database transparent encryption is similar: the engine decrypts for anyone who can query.

Specifically, use column or application encryption when the database admin should not read the field. Although that breaks some queries, that is the point for secrets and national ids.

Application envelope encryption gives you the tenant boundary. You can put tenant id into the additional authenticated data so a swapped ciphertext fails the tag check. As a result, a row copied into another tenant does not decrypt cleanly. Meanwhile, the app must not log the plaintext “just for support.”

Signing keys and JWT private keys are data at rest too. Store them with secrets management for backend systems, and rotate them with JWT rotation and revocation. Before you encrypt a field, decide whether you will ever need to search it. After you encrypt, equality search needs a blind index or a separate design, not a hope.

Rotation without a big-bang rewrite

You can rotate the KEK by re-wrapping DEKs. The payload stays put, and you only rewrite the small wrapped key. If you rotate the DEK, you must read and write the payload again. Therefore, KEK rotation can be frequent, and DEK rotation can follow a slower data-class schedule.

Keep the old KEK until every wrapped DEK has moved. If you disable it early, reads fail. In an illustrative production range, a re-wrap job can finish in hours for key blobs and in days when you also re-encrypt large objects. Plan the overlap before you schedule the job.

Trade-offs among encryption layers

Each layer stops a different thief. Overall, stack disk encryption with a tighter layer for the fields that matter. Do not pretend one checkbox covers backups, admins, and tenant isolation.

Disk encryption is cheap, and it is not tenant isolation. TDE is a good baseline, and DBAs can still select the column. Field encryption hides the column, and it complicates search. Envelope encryption puts the real control in the KMS policy, and it needs careful caching.

Do not build a custom cipher. First, turn on volume encryption and encrypted backups. Next, move KEK operations to a KMS.

Then, envelope-encrypt the columns that audits name. Finally, review who can unwrap, and remove human decrypt from the app role.

Pitfalls and failure modes

Authenticated encryption matters. If you use a mode with no tag, an attacker can flip bits and you may not notice. While GCM is a fine default, nonce reuse in GCM breaks the key. If you generate nonces from a clock on many hosts, two writers can collide.

1. Use AES-GCM or another authenticated mode.
2. Use a unique nonce for every encryption under the same DEK.
3. Bind tenant or row id into the additional data.
4. Store the wrapped DEK apart from broad human read access.
5. Encrypt backups, snapshots, and replicas, not only the primary disk.
6. Alert on decrypt errors and on unusual unwrap volume.

Key disable is a production outage on purpose. Therefore, practice it in staging and know the recovery role. A break-glass decrypt path should be loud and short. Also, a cache of unwrapped DEKs must die when you disable the KEK, or the disable does nothing until the cache expires.

Compliance language often says “encrypted at rest” and stops there. Auditors still ask who can decrypt and whether backups are in scope. Consequently, write the key policy and the backup path into the control. We once hit a bottleneck when every row unwrap called the KMS and the quota stalled checkout traffic.

Transit encryption is a different control. A perfectly sealed disk does not protect the query on the wire. Read encryption in transit for service calls and TLS for backend engineers so the plaintext does not leak after you decrypt it. Specifically, decrypt as late as you can and avoid putting plaintext in queue payloads.

A practical envelope path

The steps below keep the DEK out of the database in the clear. The store holds ciphertext and a wrapped key. When the tenant id does not match the additional data, the tag check fails. Thus, a copied blob cannot be decrypted under a different tenant.

dek = random_bytes(32)
nonce = random_bytes(12)
aad = "tenant:" + tenant_id
ciphertext, tag = aes_gcm_encrypt(dek, nonce, plaintext, aad)
wrapped = kms_wrap(kek_id, dek)
save(row_id, kek_id, nonce, ciphertext, tag, wrapped)
wipe(dek)

# read path
dek = kms_unwrap(kek_id, wrapped)
plain = aes_gcm_decrypt(dek, nonce, ciphertext, tag, aad)
wipe(dek)

Cache the unwrapped DEK only inside the process, keyed by tenant, with a short TTL. Do not write it to a local disk “for speed.” If the KMS is down, reads that need a cold unwrap should fail, not fall back to a stale world-readable key file. After an incident, you want the audit log to show which role unwrapped which kek id.

Also, version the kek id on the row so rotation can re-wrap in the background. Since two versions may be live, the reader must honor the id stored with the row. Before you delete an old KEK, confirm no row still points at it.

Performance, scale, and cost

AES on modern CPUs is cheap next to a database round trip. The KMS call is the expensive step. Therefore, wrap once per object or per tenant DEK, not once per row if many rows share a DEK.

In an illustrative production range, caching a tenant DEK for a few minutes cuts KMS traffic by a large factor. Measure your quota before a sale event.

Re-encrypt jobs compete with live traffic. If you rewrite a hot table in one thread pool, you can lock it or fill the write budget. Consequently, throttle the job and watch replica lag. Meanwhile, KEK re-wrap is much smaller and can run first.

Cost is KMS requests, key storage, and the human time to explain the design to auditors. A managed HSM raises the bill and helps when the contract demands it. Specifically, do not buy an HSM to hide a missing backup encryption setting. As a result, fix the data path before you upgrade the key hardware.

Multi-region keys remove a hard dependency on one KMS region, and they add a replication choice. A stolen wrapped DEK in the second region is still useless without unwrap rights. However, those rights must be as tight as the primary. After you add a region, copy the deny rules too.

Key Takeaways

• Encrypt primary data, backups, snapshots, and replicas.
• Use envelope encryption so the KMS wraps keys, not whole files, on each read.
• Keep unwrap rights on the app role, not on every human.
• Use authenticated encryption and unique nonces.
• Bind tenant identity into the additional data.
• Rotate KEKs by re-wrapping, and rotate DEKs when the data class requires it.
• Cache DEKs briefly in memory so the KMS is not on the hot path.

FAQ

Is disk encryption enough for compliance?

It satisfies a narrow control and fails a careful review of backups and admins. When the standard asks who can read customer data, disk encryption has no answer. Also, show the key policy and the backup encryption setting.

Should the database store the plaintext DEK?

No. Store only the wrapped DEK. If the database holds the raw DEK, anyone who can dump the table can decrypt offline. Therefore, unwrap through the KMS at read time, then wipe the DEK.

Can we search encrypted columns?

Not with a normal index on the ciphertext. You can keep a separate blind index for equality, or you can leave a non-sensitive key in the clear. If you need rich search, encrypt at a coarser layer and restrict query access instead. However, do not weaken the cipher so LIKE still works.

What should we do if the KMS region fails?

Serve from in-memory DEKs until they expire, and fail new unwraps that cannot reach a replica key. A local key file as a silent fallback becomes the real key store. Since that choice is a security decision, write it down and page when the fallback is used.

List every store that holds customer data, including backups and test restores. Then turn on volume encryption, move wraps to a KMS, and envelope-encrypt the columns auditors will ask about. Next, remove human decrypt from daily roles and alert on unwrap spikes. Finally, run a restore drill and confirm the restored copy is still ciphertext without the app role.

Last updated on 13 September 2026.