Encryption in Transit: mTLS, TLS Termination, and Service-to-Service Security

Encryption in Transit matters because a token or a query is exposed the moment it leaves the process. When TLS stops at the load balancer, every later hop is a clear-text copy. In my experience, a debug sidecar on the node captured bearer tokens the edge had already decrypted. Therefore, you should decide where decryption happens, prove both sides when you can, and rotate workload certs before they expire.

What transit encryption is and why it fails

Encryption in Transit means the bytes on the network are ciphertext, and the peer is the one you meant to call. RFC 9846 is the TLS version you should prefer for that hop. Also, server auth alone proves the server name.

It does not prove which client opened the socket. If any pod on the network can call a sensitive port, server auth is only half the story.

The usual failure is a gap in the path. The browser uses HTTPS, the load balancer terminates it, and the app speaks HTTP on the private network. Because that interior path carries the same session cookie or token, a tapped interface reads the secret. Still, the architecture slide shows a single green lock.

A common mistake I have seen is hostname verification turned off “until the cert names are fixed.” The fix never lands. When a wrong host presents any trusted cert, the client accepts it.

Consequently, a private CA that issues broadly becomes a skeleton key. Furthermore, a client that trusts that CA and skips the name check will talk to an impostor.

Identity and encryption are different. A valid client cert means the caller holds a key the CA signed. It does not mean the caller may refund an order.

If you skip the authorization check, mTLS becomes a very expensive any-allow. Pair the cert identity with IAM roles for backend services or with your own policy.

Architecture and how you implement it

Draw every hop: browser to edge, edge to ingress, ingress to app, app to database, app to queue, and job to job. First, mark which hops are already TLS. Next, mark who terminates and who can read clear text after that point.

Then, choose server auth or mTLS for each remaining hop. Finally, name the CA and the rotation owner. A hop with no owner will expire in prod.

Termination at the edge is convenient for routing and for WAF features. Re-encryption to the app protects the interior. Passthrough sends the original handshake to the app, which sees the client cert, and the edge cannot route on HTTP paths as easily. Since each mode hides different fields, pick it per route, not as a cluster-wide slogan.

mTLS means the client also presents a certificate. The server checks the client chain and a name or SPIFFE-style id. The client still checks the server name.

Kubernetes TLS certificates show how a cluster can issue and mount certs for workloads. Also, mesh sidecars can do this outside the app. If you choose sidecars, the app may see plain HTTP on localhost, so localhost is now part of the trust boundary.

Certificates as workload identity

Short-lived workload certs beat a static key in an image. A platform CA issues a cert for the service account, valid for hours, and rotates it. Store the private key with the same care as secrets management for backend systems. Specifically, the key file mode should be owner-read, and a shared volume should not expose it to a debug container.

Put a stable id in the cert subject or a URI SAN. Your policy should allow “payments may call ledger,” not “any cert from this CA.” Although the CA is private, a leaked issuer key can mint any service.

As a result, protect the CA key offline or in a KMS, and limit who can request names. Meanwhile, publish a revocation or a short lifetime so a bad cert dies fast.

Tokens and certs can work together. Use mTLS so the caller is a known service, and use OAuth and OIDC for backend services when a user is involved. Then the service identity and the user identity are both present. If you put the user token on a hop without transit encryption, the cert work was wasted.

What the TLS article already covers

Handshake details, cipher floors, and trust-store drift live in TLS for backend engineers. Use that piece when a client fails the chain or the name. Use this piece when you are choosing termination, mTLS, and service policy. The Mozilla Server Side TLS guide is still the right baseline for versions and ciphers on each proxy.

Data at rest is the other half. After you decrypt a call, the plaintext hits a disk or a queue. Read encryption at rest so a snapshot does not undo the socket protection. Before you call the path done, check both states: on the wire and on disk.

Trade-offs among path designs

The design changes who sees plaintext and how hard rotation is. Overall, encrypt every hop that carries secrets, and add mTLS where the caller must be a named service.

Edge-only is easy, and it leaves east-west traffic open. Re-encryption closes that gap, and it still allows any client that can route. Sidecar mTLS gives identity without app changes, and localhost becomes sensitive. In-app mTLS avoids the sidecar, and every language must get the trust store right.

Do not start with a full mesh on three services. First, close plain interior hops that carry tokens. Next, turn on hostname checks.

Then, issue workload certs for the sensitive calls. Finally, enforce a policy of which service may call which service.

Pitfalls and failure modes

Rotation is the outage source. A cert expires, a CA rotates without overlap, or a pod keeps an old bundle in memory. While half the fleet trusts the new CA, calls fail in one direction only. If health checks use a different name than the app, they stay green while users fail.

1. Automate issuance and reload, and alert before expiry.
2. Overlap old and new CAs during a trust rotation.
3. Verify server name and client identity on every sensitive hop.
4. Do not treat a valid cert as an authorization decision.
5. Include localhost and sidecars in the threat boundary.
6. Redact tokens in proxy logs and traces.

Broken authorization shows up as a confused deputy. A gateway holds a powerful client cert and forwards user requests without a user check. Therefore, the gateway must pass a proven user identity, not only its own cert.

In an illustrative production range, workload certs of a few hours limit how long a stolen pod key works. Shorter is safer if your issuer can handle the renewals.

HTTP and TLS timeouts interact badly. A idle connection may die at a firewall while the pool still thinks it is healthy. Consequently, set idle limits and retries that are safe for the method.

We once hit a bottleneck when every retry opened a full handshake because the pool size was one. Raising the pool and enabling resumption removed the CPU spike.

Clear text on a queue is still transit. A message between two encrypted services can sit unread on a broker in plaintext. Specifically, turn on broker TLS and then decide whether the body also needs encryption at rest. After you do, a broker admin should not become the new bypass.

A practical mTLS pair

The server below requires a client cert and names the CA that may issue it. The client checks the server name and presents its own cert. When either file is missing, the process should fail at start. Thus, you do not serve “degraded” without mTLS by accident.

server:
  bind: 0.0.0.0:8443
  cert_file: /certs/ledger.crt
  key_file: /certs/ledger.key
  client_ca: /certs/workload-ca.pem
  client_auth: require_and_verify
  min_version: TLS1.2

client:
  target_name: ledger.internal
  cert_file: /certs/billing.crt
  key_file: /certs/billing.key
  root_ca: /certs/workload-ca.pem
  # Policy still checks that the client id may call this method.

TLS1.2 is the floor in that snippet. Prefer the newer version when both sides support it. Also, map the client cert id to an allow list per method.

Since a cert for billing should not call the admin refund route, encode that in policy. Before you ship, test three failures: expired client cert, wrong server name, and a cert from a different CA.

Reload certs on rotation if the server supports it. If it does not, roll pods after the secret updates. After the roll, confirm old pods are gone so they cannot present a retired identity.

Performance, scale, and cost

A handshake per request will dominate CPU and latency. Therefore, pool connections and reuse them. HTTP/2 or similar multiplexing lets one handshake carry many calls.

In an illustrative production range, pools of tens of connections per destination are enough until you measure otherwise. Resumption makes the next handshake cheaper.

mTLS adds a client cert check, which is small next to a cold connection. The expensive part is issuing and distributing certs at fleet scale. Consequently, let the platform rotate them, and keep app code on a stable file path or socket.

Meanwhile, a stampede of renewals at the same second can overload a CA. Jitter the renewals.

Cost is CA operations, mesh CPU, and incident time from expiry. A public cert on an internal name is often the wrong tool, because you may not want those names in public logs. Specifically, use a private CA for east-west names and a public CA for public edges. As a result, interior name policy stays under your control.

Scale policy checks beside the TLS check. A central call for every authorization can erase the benefit of a local handshake. Cache policy for a short time, and fail closed on sensitive methods when the cache is cold and the policy service is down. However, document that choice so on-call does not “temporarily” allow all.

Key Takeaways

• Encrypt every hop that carries tokens, cookies, or customer data.
• Know exactly where TLS terminates and who can read plaintext there.
• Use mTLS when the caller must be a named workload, not just any client.
• Treat a client cert as identity, then still authorize the action.
• Rotate workload certs automatically and overlap CA trust changes.
• Pool connections so handshakes are not on the per-request path.
• Include queues, sidecars, and localhost in the same review.

FAQ

Is a private network a substitute for transit encryption?

No. Shared clusters, tapped ports, and compromised nodes sit on that network. When the payload is sensitive, encrypt the hop. Also, private routing is still useful as a second control.

Should every internal call use mTLS?

Use mTLS for calls where the peer identity matters. Server auth plus a network policy may be enough for a low-risk read. If you cannot rotate certs yet, do not pretend mTLS is on. Therefore, start with the hops that move money or personal data.

Who should terminate TLS?

Terminate where you need to inspect or route, and re-encrypt after that if the next hop is shared. The app should terminate itself when it must see the client cert. Since the terminator sees plaintext, keep its logs free of tokens and bodies.

What happens when a workload cert expires?

New handshakes fail, and pooled connections may linger until they idle out. Clients should reload the new cert before the old one expires. If you miss the window, restart or reload both sides after issuance. After recovery, add an alert earlier in the lifetime so you do not repeat it.

Map every hop that carries a secret and mark where plaintext still exists. Then turn on TLS for those gaps, with hostname checks, and add mTLS for service identity. Next, automate cert rotation and authorize actions after the handshake. Finally, run a test that presents the wrong client cert and confirm the call is denied.

Last updated on 06 September 2026.