Cron Jobs in Distributed Systems: Scheduling, Idempotency, and Missed Runs

Cron Jobs in Distributed Systems break when more than one machine thinks it is the only clock. You feel that bug when a nightly bill runs twice or a cleanup never runs at all. If you treat a single host crontab as a plan, the next deploy will surprise you. This guide covers locks, missed runs, and how to keep a job safe when hosts come and go.

What they are and why they fail

A cron job is work that should start on a schedule. On one server, the daemon reads a table and starts a process. In a cluster, many servers can read the same table. Then two processes start unless you add a lock.

The failure is usually quiet. Users see a double email or a missing report. Logs show two starts a few seconds apart.

Because each host had a correct clock, nobody thinks the schedule is wrong. The missing piece is a single owner for that tick.

In my experience, teams move a crontab into a container and replicate the container for availability. Both copies fire. The job was safe on one box and unsafe the moment you scaled it. Therefore, write the lock into the design before you add a second replica.

Cron Jobs in Distributed Systems also fail when a run lasts longer than the gap between ticks. The next start overlaps the first. They fight over the same rows. If the job is not safe to overlap, you must skip, queue, or wait.

How to schedule with one owner

You have three common shapes. A cluster scheduler can start one pod. A database lease can let one worker win.

An external scheduler can push one message per tick. Pick one owner path and do not stack three by accident.

A scheduler that starts one pod

The Kubernetes CronJob docs describe a controller that creates a job on a schedule. Set the concurrency policy so a second run does not start while the first is active. Also set a deadline so a stuck run does not block the calendar forever.

This shape fits when the cluster is already your control plane. It fails when the controller itself is down during the tick. You must decide if a missed tick should run late or wait for the next slot. Write that choice in the job spec, not in a chat thread.

A lease in the database

A lease is a row that one worker updates with its id and an expiry. Others see the row and exit. The PostgreSQL explicit locking docs cover locks you can use, including advisory locks for short critical sections. Use a lease when the job may run longer than a transaction should stay open.

Renew the lease on a timer if the job is long. If the worker dies, the lease expires and another worker may take the next tick. If you renew too slowly, a second worker starts while the first is still writing. Size the lease from the high percentile of run time, not the average.

A scheduler that enqueues one job

An external clock can publish one message per tick. Workers then compete to run it. The Amazon EventBridge cron expressions page shows how a managed schedule fires a target. This shape keeps hosts stateless.

It still needs an idempotent handler. The scheduler can retry the push. Your consumer can see the same tick twice. Store the schedule key, such as the job name plus the planned time, in a unique column.

That key is the same idea as timeouts and idempotent retries on a request path. A schedule is just a request that a clock sends. If you skip the unique key, retries become double work.

Missed runs and overlap

Clocks drift, deploys restart pods, and a tick can land while you roll. You need a rule for missed runs. Catch up means run the missed tick as soon as you can. Skip means wait for the next planned time and do nothing for the gap.

Catch up is right when each tick has unique work, like a daily export for a date. Skip is right when the job always scans the current state, like a cache refresh. If you catch up a refresh, you do the same scan many times and add load for no new result.

Overlap needs its own rule. Forbid means the new tick exits when the old run holds the lease. Queue means you store the tick and run it after. Replace means you cancel the old run, which is rare and easy to get wrong.

Although forbid is the safe default, it can hide a stuck job. If the lease never expires, every later tick exits and the backlog of real work grows. Alert when a run exceeds its budget. Then a human can kill it or let the lease lapse on purpose.

Trade-offs you should weigh

Use the table to pick a starting shape. Then test a deploy during a scheduled minute. If two logs show a start, the lock is not real yet.

A managed clock costs a small fee per rule and per event. A database lease costs you code and a hot row. However, the lease stays inside your failure domain if you do not want a new vendor.

Do not pick a shape only because it is already installed. Pick it because you can explain a missed hour.

Jitter matters when many jobs share a minute. If every job starts at second zero, the database sees a spike. Spread start times with a hash of the job name.

For example, a job can wait up to thirty seconds before it takes the lease. Users rarely care, and the database does.

Pitfalls and failure modes

A common mistake I have seen is a lock that lives only in process memory. It works in tests with one process. After you scale out, each process has its own memory and each one runs. Put the lock in a system both sides can see.

Another trap is a clock that jumps. A host with a bad NTP step can fire early or late. Compare the planned time to your own store, not only to the local clock. If the skew is larger than your window, skip and alert.

Deploys cause a third trap. You roll the fleet at the top of the hour. The old pod dies mid job, and the new pod starts the same tick.

Without a lease, both do part of the work. With a lease, the new pod must wait until the old lease expires or is released on shutdown.

Release the lease on a clean exit. Do not release it if you are unsure the work finished. A half done job plus a released lease invites a second runner to redo the same rows. Finish, commit, then release.

Side effects outside your database need extra care. Sending mail or charging a card is not rolled back when a transaction aborts. Store the schedule key before the call. If the process dies, the next run sees the key and skips or resumes from a saved step.

Failed ticks should not vanish. Park the error where you can replay it, in the same spirit as dead letter queues. A cron log that rotates away is not a queue. If the export for Tuesday failed, you still need Tuesday.

Large jobs should fan out. One process that scans a huge table will not finish before the next tick. Publish work in pages onto pub/sub and message queues.

The cron tick only enqueues. Workers compete to finish pages, and the tick itself stays short.

Gate a risky job with a feature flag so you can stop it without a deploy. A bad backfill that runs every minute will do damage until the next release. A flag stops the next tick at once.

Follow this order when a schedule looks wrong. Check ownership before you rerun anything by hand.

1. First, confirm how many runners started for that planned time.
2. Then, check whether the lease or the unique key blocked the second start.
3. Next, see if the run overlapped the following tick.
4. Finally, rerun one missed period with the same schedule key.

A lease sketch you can adapt

The sketch below is a config, not a library. It shows one job, a lease, and a rule for overlap. Tune the seconds after you measure a real run. If the lease is shorter than the job, you will get two writers.

job: nightly-invoice-export
schedule: 15 2 * * *
concurrency: forbid
missed: catch_up_once
lease_seconds: 3600
renew_every_seconds: 60
jitter_seconds: 30
idempotency_key: job_name + planned_time
on_failure: record_and_alert
flag: invoice_export_enabled

The planned time is part of the key so Tuesday and Wednesday do not collide. Catch up once means you run a single missed day, not every missed day in a tight loop. Since the flag wraps the start, an operator can halt the job before the next lease renew.

Renew every minute so a crash is noticed within about a minute after the lease time. Do not renew if you cannot reach the database. A blind local timer would keep a dead leader in your head while another host already owns the row.

Performance, scale, and cost

The schedule tick should be cheap. The heavy work should be spread out. If one query locks a hot table at the same minute as your peak traffic, users will feel the job. Move that scan off peak, or slice it into small pages.

As an illustrative production range, a fleet of a few hundred jobs that all start at midnight can add a sharp CPU and connection spike for one or two minutes. Jitter and page size cut that spike. A managed scheduler fee is usually small next to the database cost of a bad query.

Watch three numbers. Start count per planned time should be one. Run age should stay under the lease.

Missed tick count should match the policy you chose. Still, a green CPU graph does not prove the job ran once.

Key Takeaways

• Also, assume many hosts can start the same job unless a shared lock stops them.
• However, choose catch up or skip on purpose, based on whether the tick has unique work.
• Therefore, store a unique key of job name plus planned time before any side effect.
• Still, forbid overlap by default and alert when a run outlives its lease budget.
• Because deploys kill pods mid job, release the lease only after a successful commit.
• After you scale the data, let the tick enqueue pages instead of scanning everything itself.
• Finally, add jitter so many jobs do not hit the database at the same second.

FAQ

When should a missed cron run catch up?

Catch up when each planned time names a distinct slice of work, such as a date in a file name. Skip when the job only refreshes current state. If you were down for many periods, cap the catch up at one period. Then decide if older slices still matter.

What happens if two workers take the lease?

You get two side effects unless the work itself is unique. A correct lease allows one owner. Check that the write of the lease is conditional on the previous expiry. If both updates can succeed, the lock is only a hint and you will see double runs.

How should you stop a bad job quickly?

When the next tick is minutes away, a deploy is too slow. Put a flag in front of the start path and turn it off. Also refuse a new lease while the flag is off. Let the current run finish or kill it if it is still doing harm.

Can you rely on Cron Jobs in Distributed Systems for exact timing?

No schedule on a busy cluster hits the exact second every day. Use the schedule as a target, and allow skew and jitter. If a user action needs a precise instant, do not wait for cron. Trigger that work from the request path and use cron only for sweep and repair.

Next, list every scheduled job and write its owner, overlap rule, and missed tick rule. Add a unique key and a lease or a controller policy before you run a second replica. Then trigger a deploy during a test tick and confirm only one start is logged. If two starts appear, fix the lock before you trust the result.

Last updated on 11 September 2026.