A batch job starts at midnight with one million records to process. It reads the file in order, writes results to a database, and finishes around 2 AM on a good night.
At 1:07 AM the process dies. Five hundred thousand records are already done. The scheduler restarts the job. The job opens the file from the top and starts again. Some records are written twice. Some totals are counted twice. A few downstream systems receive duplicate events and treat them as new work.
The next version gets checkpoints. Every 100,000 records, the job writes its current position somewhere durable. If it crashes, it resumes from the latest checkpoint. That helps until the checkpoint says record 500,000 but the database only committed up to 492,317. Now the operator has to decide whether to replay, skip, delete partial output, or patch the checkpoint by hand.
Another version is built differently. Each record has a stable ID. Each write uses that ID. Reprocessing a record produces the same final row. The database records which IDs completed. When the job restarts, it asks the database what remains and continues from the next unprocessed record.
That job can die at 1:07 AM, 1:08 AM, and 1:09 AM. The restart is boring each time. It checks the durable state and keeps moving.
What Restartability Actually Means
A restartable system can be killed in the middle of useful work and come back without a human reconstructing the scene.
The kill can happen anywhere. The current item may be half processed. The transaction may be open. The worker may be holding a lease. The machine disappears, the container is evicted, the deploy rolls through.
On restart, the system finds its own place. It can tell what completed, what partially happened, and what still needs doing. It does not need an operator reading logs at 2 AM and guessing where the safe line is.
Designing for restartability requires several properties:
- Idempotency. Operations can be repeated safely. Running the same operation twice produces the same result as running it once.
- External state. Progress is tracked outside the process, in a database, queue, file, object store, or another durable system.
- Deterministic ordering. Work is processed in a stable order, so restart logic does not depend on whatever happened to be in memory.
- No side effects from incomplete operations. An interrupted operation does not leave the system half-changed in a way the next run cannot understand.
Without those properties, recovery becomes choreography. Someone has to preserve the right state, resume from the right point, and hope the world outside the process matches the checkpoint inside it.
Why Checkpoint-Based Systems Are Fragile
Checkpointing usually arrives after the first painful failure. The job was built as one long line of execution. Then it crashed late enough to hurt. Now the team adds “save current state” every few minutes and calls recovery handled.
The fragile part is not the checkpoint file itself. The fragile part is everything the checkpoint claims to represent.
State consistency. The system’s internal state has to be captured completely. Variables, buffers, counters, temporary files, partial database writes, and in-flight work all matter. Missing one piece turns the checkpoint into a confident lie.
Version compatibility. A deploy can land between crash and restart. The old checkpoint format now has to be read by new code. Sometimes that works. Sometimes recovery fails before the job even reaches the data.
Checkpoint corruption. A checkpoint can be written while another operation is only partly committed. The checkpoint says the job reached a position the surrounding state does not actually support.
Atomicity of checkpoint. Saving the checkpoint is itself an operation that can fail. A partial checkpoint creates a new recovery problem while trying to solve the old one.
Rollback complexity. If recovery goes wrong, the operator has to understand both the checkpoint and the side effects created after it. There may be no clean way to invalidate the checkpoint and start over.
After a crash, the checkpoint may be true, stale, partial, or compatible only with yesterday’s code. Recovery starts with a question the system cannot answer alone: “Do we trust this?”
Idempotent Design Is Restartable By Nature
Idempotent systems have a much calmer recovery path.
A record arrives with ID invoice_18492. The worker writes the result using that ID as the key. If the worker runs the same record again, the database lands in the same final state. The second attempt might be a no-op. It might update the same row with the same values. Either way, it does not create a second invoice.
Now the restart path is simple. Start the worker. Query durable state. Find records without completed results. Process those. If the worker repeats something, the write is safe.
The progress does not live in memory. It lives where operators can inspect it and other workers can agree on it: a database, queue, object store, ledger, or task table.
A data processing job marks each record as processed. A deployment tool applies configuration changes by desired state, then reads the current state and applies the missing pieces. A message consumer records completed message IDs before acknowledging them. Each system can be restarted because the outside world contains enough truth to continue.
The Relationship Between Restartability and Simplicity
A non-restartable job teaches operators to be careful. They watch it during deploys. They avoid killing it. They wait for a “safe” window. If it fails, someone has to inspect partial output, compare logs with database rows, and decide how much to replay.
A restartable job changes the operating posture. The deploy can restart it. The node can disappear. The worker can crash and come back. If the first restart fails, another restart is allowed. The system keeps converging on completed work instead of depending on one uninterrupted run.
That is where the simplicity shows up. Not in fewer lines of code, necessarily, but in fewer special instructions for the humans around it.
Designing for Restartability From the Start
Restartability works best when it appears in the first design sketch, before the first long-running process exists.
Use external state for progress tracking. Keep progress somewhere durable and queryable. After restart, the system should be able to ask what has completed and what remains.
Design operations to be idempotent. Give every unit of work a stable identity. Check whether that identity has already completed. Make the second attempt land in the same final state as the first.
Order work deterministically. Process items by ID, timestamp, priority, or another stable ordering. After restart, the next unit of work should not depend on whatever happened to be in memory.
Keep state simple. The less state the process carries, the less state a restart can lose. Prefer small, queryable records over complex in-memory structures.
Separate progress from processing. One part records what is pending, running, completed, and failed. Another does the work. A crashed worker can be replaced because the work state survived it.
Test restartability. Kill the job mid-record, mid-batch, and mid-write. Restart it. Verify final output, not just process exit.
Examples of Restartable Designs
Restartability tends to show up through a few plain patterns:
Transactions with external state. A payment system receives a transaction ID, checks the database for that ID, and records pending, processed, or failed state durably. A retry with the same ID does not charge twice.
Event sourcing. The system appends immutable events to a log. After a crash, it rebuilds current state from the log. Duplicate events still need stable IDs or deduplication, but the recovery source is durable.
Message queue consumers. A consumer pulls a message, processes it, records completion, and acknowledges it. If it crashes before acknowledgment, the message returns. If it crashes after completion, the idempotency key prevents duplicate effects.
File-based batch processing. A batch job reads records in a stable order and writes results by record ID. Restarting the job rechecks output state and skips completed records.
Schedulers with task state. A scheduler stores task state as pending, running, completed, or failed. On restart, old running tasks can be expired, retried, or inspected by policy instead of disappearing with the process.
When to Accept Non-Restartable Design
Some systems can live without explicit restartability.
Short-lived processes that rarely fail. A process that always completes in seconds has little mid-execution surface area. Restartability may cost more than it returns.
Stateless services with external resource management. A stateless API service has no progress to reconstruct. If it crashes, a new instance starts and serves the next request.
Real-time systems with tight latency requirements. A microsecond-sensitive system may not be able to query durable state during the hot path. Replication and failover may carry more of the reliability burden.
Interactive systems where state is held by the user. A multi-step form may keep important state in the browser or session. If the server restarts, the user interaction owns the recovery path.
The tradeoff is usually acceptable when the work is short, stateless, replicated, or owned by an external session. Long-running jobs, infrastructure changes, data migrations, billing flows, and queue consumers rarely get that luxury.
The Cost of Not Having Restartability
The cost shows up later, under pressure:
Operational complexity. When the system fails, operators have to recover manually. They need to understand internal state, partial output, retries, and cleanup rules.
Unavoidable downtime. A process that cannot resume has to complete normally or be replayed. Long-running operations turn small failures into expensive delays.
Cascading failures. A bad recovery can make the original failure worse. Duplicate processing, skipped work, and manual cleanup can spread damage into downstream systems.
Low confidence in operations. Operators become nervous about restarts. They delay deploys, avoid maintenance, and leave unhealthy processes running because stopping them might create a recovery problem.
Difficult to scale. A job that restarts from the beginning gets worse as the dataset grows. A failure 23.95 hours into a 24-hour run becomes a full-day replay.
Restartability as Infrastructure Feature
Some platforms supply part of the restartability story.
Kubernetes and container orchestration. If a container crashes, the orchestrator starts another one. The application still has to tolerate starting from clean process memory.
Workflow engines. Tools like Apache Airflow and Temporal track task state and retry incomplete work. The tasks still need safe retry behavior.
Distributed databases. Databases recover committed state after crashes. They do not automatically make application side effects idempotent.
Message brokers. Kafka, RabbitMQ, and similar systems can redeliver unacknowledged messages. The consumer still has to handle duplicates safely.
These platforms restart containers, resume workflows, replay logs, and redeliver messages. They do not make unsafe work safe. The application still has to use idempotent operations, durable progress, and simple recovery state.
Restartability and Reliability
A job that fails every week and resumes cleanly is annoying. A job that fails once a year and needs three people to reconstruct partial state is dangerous.
Reliability still matters. Fewer failures are better than more failures. But production systems eventually meet deploys, node failures, process crashes, dependency outages, and operator mistakes. Restartability decides whether those events become routine recovery or bespoke incident work.
The best systems do both: they avoid failures where they can, and they recover cleanly when avoidance loses.
The Mindset Shift
The mindset shift is visible in the questions engineers ask.
The fragile version asks: how do we keep this from stopping?
The production version asks: what happens when this stops here, with this write half done, this message unacknowledged, this lock held, and this deploy rolling through?
Those questions produce different designs. One depends on careful choreography. The other expects interruption and stores enough state to recover.
Choose Restartability
A long-running system should be able to stop without becoming a puzzle.
Give work stable IDs. Make retries safe. Store progress outside the process. Test crashes in the middle, not just success at the end.
Then restart becomes an ordinary operation instead of a recovery ritual.
