Kubernetes for Backend Engineers: Architecture, Scheduling, Autoscaling, and Failure Modes

Kubernetes for Backend Engineers is the control loop that places your processes, restarts them, and scales them. When you treat the cluster as a magic host, you get surprise evictions and noisy bills. Therefore you should learn what the scheduler, the kubelet, and the autoscaler each decide.

What Kubernetes Is and Why It Fails in Production

Kubernetes keeps a desired count of pods and tries to make the cluster match that desire. A pod is one or more containers that share a network and volumes. If a node dies, the control plane should start replacements elsewhere.

In my experience, clusters fail in production for a few plain reasons. First, requests and limits are guesses, so the scheduler packs too tight or leaves the node idle. Then, a probe kills healthy pods during a slow dependency. Also, a deploy rolls out faster than the new pods can become ready, so you drop capacity on purpose.

A common mistake I have seen is a missing request. The pod schedules anywhere, then a noisy neighbor squeezes it. Because there is no request, the scheduler had no floor to protect.

After the latency spike, the dashboard blames the app. The real fault was placement.

Another failure is identity. Two replicas run a job that must be single. A leader lock is missing, or the lock is in memory.

When a second pod starts during a rollout, you double charge a customer. Still, both pods look healthy. So design the work so two copies are safe, or take a real lock outside the process.

You also fail when the cluster and the app disagree about shutdown. The kubelet sends SIGTERM, waits, then kills. If the app ignores the signal, in flight requests die.

Drain and preStop hooks exist so you can leave the pool before you exit. Use them.

Architecture and Implementation

Four loops matter to an app owner. First, the API server stores desired state. Next, the scheduler binds pods to nodes. Then, the kubelet starts containers and runs probes.

Finally, controllers such as the Deployment and the autoscaler change the desired count. The Kubernetes components docs name each piece. You do not need to run them all by hand to use them well.

Scheduling and requests

The scheduler scores nodes that can fit the pod requests. It does not look at live usage unless you add a custom policy. If you set requests far above real use, you waste nodes. If you set them too low, you overpack and then throttle.

Set requests from a week of real CPU and memory, then add headroom for a spike. Limits cap a runaway. A memory limit that is too close to the request will OOM kill you on a normal GC.

The Kubernetes scheduler docs describe the default filters. Read them before you write a complex affinity rule.

Spread pods across zones when a zone loss should not take the service down. Pin a pod to a pool only when it needs a special device or a noisy isolation boundary. Too much affinity makes the scheduler fail even when spare CPU exists elsewhere.

Deployments and rollouts

A Deployment owns ReplicaSets. A rollout creates a new ReplicaSet and scales the old one down. Your rolling deployments should set max unavailable and max surge so capacity never falls below a floor you chose.

Pin the image by digest. Your Docker images are the bytes. A tag that moves during a rollout can mix two builds in one ReplicaSet history. Then a rollback to the tag does not return to the old bytes.

Package the manifests with Helm releases or with a similar tool so the review shows the values that change. Hand edited live objects drift. The next apply will wipe a hotfix you made in the cluster and forgot.

Autoscaling

The horizontal pod autoscaler changes replica count from a metric, often CPU. It cannot help if the process is blocked on a lock and CPU looks idle. Scale on a signal that tracks the bottleneck, such as queue depth or requests in flight.

The cluster autoscaler adds nodes when pods are pending. It is slow compared with pod scale. If you need capacity in seconds, keep a buffer of warm nodes.

The Horizontal Pod Autoscaler docs show the pod half. Plan the node half with the same care.

Scale down is where incidents start. A new node count can evict pods that hold local cache or long jobs. Protect those pods with a budget, or move the work to a queue that can resume. Do not assume scale down is free.

How to Shape a Service

One Deployment per process type. Do not put the API and the worker in one pod unless they must share a disk and a fate. When you scale the API, you would also scale the worker, and the worker may not be safe to multiply.

Keep config in the cluster as data, not as a new image for every flag flip. Use feature flags when you need a fast off switch that does not wait for a rollout. If the flag service is down, the app needs a default that is safe.

1. Set CPU and memory requests from real usage.
2. Set memory limits with room for spikes and GC.
3. Add readiness and a shutdown hook.
4. Choose surge so a rollout keeps a capacity floor.
5. Scale on the real bottleneck, then test scale down.

Trade-offs and Comparison

Kubernetes is a strong default when you run many services and you want one way to place them. It is a weak default for a single small app on one VM. The control plane is a product you operate, or you pay someone to operate it.

Managed control planes remove a class of etcd pain. You still own node pools, quotas, and the app specs. Self managed clusters give you deep control. They also give you upgrade nights.

Choose managed Kubernetes if your team wants to ship services, not etcd. If you cannot name an owner for node upgrades, do not run the control plane yourself. More control is not a goal. A calm on call is the goal.

Pitfalls and Failure Modes

Probe mistakes restart the fleet. A ready check that calls a slow downstream will mark every pod unready when that downstream blips. Then the Service has no endpoints, and you cause a total outage from a partial one. Keep ready checks local and cheap.

OOM kills look like random restarts. The limit is below the live heap. After you raise the limit without a request change, the node can overcommit memory and the kubelet will evict someone else. Raise both with a plan.

Pod disruption budgets that are too strict block node drains forever. You cannot patch the node. The budget that was meant to protect uptime now blocks the fix. Set a budget that allows one disruption when you have three or more replicas.

Config maps are not magic reloads. Some apps read env at start only. A config change does nothing until you roll the pods.

If you expect a hot reload, test it. Do not discover the gap during an incident.

DNS and conntrack limits show up as random timeouts under load. The app looks fine in tests. When the node has many services, DNS or the connection table saturates.

Watch node level errors, not only app logs. A retry storm will make this worse.

Default limits in a namespace are a footgun. A team copies a sample, gets a tiny CPU limit, and then spends a week in throttle. Make defaults obvious. Your infrastructure as code should set namespace quotas in review, not as a surprise at runtime.

A Practical Pod Spec

The sketch below sets requests, limits, a ready probe, and a short grace period. It pins an image tag only as a stand in. In prod, pin a digest. When the process gets SIGTERM, it should stop taking work and then exit before the grace period ends.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: api
spec:
  replicas: 3
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 0
      maxSurge: 1
  template:
    spec:
      terminationGracePeriodSeconds: 30
      containers:
        - name: api
          image: registry.example.com/api:abc123
          ports:
            - containerPort: 8080
          readinessProbe:
            httpGet:
              path: /ready
              port: 8080
          resources:
            requests:
              cpu: "250m"
              memory: "256Mi"
            limits:
              cpu: "1"
              memory: "512Mi"

Note what this sketch does not do. It does not set a liveness probe that hits a remote database. It does not allow zero capacity during the rollout.

If your traffic is bursty, raise the memory limit before you raise the replica count. More pods with the same OOM will just crash in parallel.

Performance, Scale, and Cost

Cluster cost is mostly nodes, plus a smaller control plane fee. Empty requests waste whole nodes. Tight requests cause throttle and retries, which can cost more in user pain than the node you saved. Measure both.

Bin packing improves when requests match use. A node with 4 CPU and pods that each request 3 CPU will run one pod and waste the rest. Pick request sizes that divide the node. Also keep a little slack so a burst pod can land without a scale up wait.

At scale, etcd and the API server feel chatty controllers. A thousand teams that poll the API from laptops and from CI can slow the cluster. Cache list calls. Do not write a sidecar that watches all pods in the cluster if it only needs one namespace.

We once hit a bottleneck when the autoscaler added pods faster than the image could pull. Pending pods piled up, the cluster autoscaler added nodes, and the bill spiked while error rates stayed high. The fix was a smaller image and a lower scale rate. After that, new pods became ready before the next scale step.

Plan for a zone loss. If you spread across three zones, you should still serve when one zone is gone. That means you pay for spare capacity on purpose. If you pack to 100 percent, a zone loss becomes an outage even when the scheduler is healthy.

Upgrade nodes on a schedule. Old nodes miss patches and they drift from the new ones. A slow drain with a disruption budget is safer than a heroic weekend.

If drain stalls, fix the budget or the pod that will not exit. Do not force delete as a habit.

Key Takeaways

• Requests place the pod. Limits cap a runaway. Set both from real use.
• Roll out with a capacity floor, and pin the image digest.
• Keep readiness local so one bad dependency does not empty the Service.
• Scale on the real bottleneck, and test scale down.
• Shutdown on SIGTERM before the grace period ends.
• Leave spare capacity so one zone can fail open.
• Own node upgrades, or the cluster will own your incident.

FAQ

Do I need Kubernetes for one service?

Often no. A single VM or a small platform you already run can be simpler. Move when you have many services, a need to reschedule, and an owner for the cluster. If nobody wants the on call for nodes, stay on a smaller platform.

Should I set CPU limits?

Memory limits should be set, because a memory leak takes the node down. CPU limits are a trade. They stop a loop from stealing the node, but they also throttle you.

If you set them, watch throttle time. If throttle tracks your latency, raise the limit or the request.

How many replicas should I run?

At least two for anything that must stay up during a node loss, and three if you spread zones and roll one at a time. A single replica will go down on every node drain. If the work cannot run twice, keep one replica and block disruption with care, and accept the downtime on node loss.

What should I alert on first?

Alert on user pain, then on causes you can act on. Error rate, latency, and restart count beat a noisy CPU chart. Also alert when pods stay pending, because that means the scheduler cannot place them. A pending pod is a capacity or a constraint bug, not an app log line.

Kubernetes for Backend Engineers pays off when you set requests, rollouts, and shutdown with intent. Pick one service you own. Next, add requests, a local ready check, and a grace period that matches the process. Then drain one node in staging and watch the service stay up.

Last updated on 13 September 2026.