Long-running software eventually gets interrupted.
A deployment replaces the process. A container is rescheduled. A machine reboots, a dependency fails, a worker crashes, or an operator kills a job that appears stuck. The longer a piece of work runs, the less reasonable it becomes to assume that one process will survive from beginning to end.
That gives restartable systems a simple but demanding test:
Can I kill this process at an arbitrary point and restart it without a human repairing state?
If the answer is yes, restartability is part of the system’s design. If the answer is no, recovery depends on knowing exactly where the process died, which side effects already happened, which records are safe to repeat, and what someone needs to repair before execution can continue.
Restartability is therefore not a workaround for unreliable infrastructure. It is a property that makes interruption ordinary.
Long-running work
│
▼
Interruption at any point
│
▼
Start another process
│
▼
Recover durable state
│
▼
Determine remaining work
│
▼
Safely retry or replay
│
▼
Continue toward completionThe difficult part is not starting the executable again. The difficult part is ensuring that the new process can reconstruct enough truth to continue safely.
Assume the Process Will Disappear
Short-lived programs can sometimes get away with keeping important progress in memory. If execution lasts a fraction of a second, the probability of interruption during that window may be small enough that starting again from the beginning is harmless.
Long-running work changes the calculation.
A migration may run for hours. A media-processing job may handle thousands of files, while a workflow may wait days for external events. Batch imports, background workers, reconciliation processes, data pipelines, and distributed workflows all spend enough time running that interruption should be considered part of normal operation.
The fragile design looks like this:
Start
│
▼
Load work
│
▼
Process thousands of items
│
X crash at item 73,421If everything important existed only inside that process, the replacement has little reliable information. It may know that the job failed, but not which work completed successfully, which operation was in flight, or whether an external side effect occurred immediately before the crash.
A restartable design treats the process as disposable. The durable truth about the work exists somewhere that survives it.
That changes the architecture:
Durable state
▲
│
Process A ───┼──► progress
X │
│
Process B ───┼──► recover and continueThe process performs work, but it does not own the only copy of the information needed to understand that work, the same ownership boundary that matters when timeouts do not cancel downstream work.
This is the foundation of restartability: execution can disappear without taking the system’s understanding of progress with it.
Durable State Needs Stable Work Identity
Externalizing state is necessary, but simply writing “progress” to a database is not enough. The replacement process also needs to know what logical work that state belongs to.
That requires stable identity.
Suppose a job is processing an import containing 100,000 customer records. If each attempt invents a new identity, a restart cannot easily distinguish a continuation from a completely new import.
A stable job identity gives both executions a common reference:
job_id = import-2026-09-03-1842
Process A
│
├── customer 1 complete
├── customer 2 complete
└── customer 3 interrupted
Process B
│
└── same job_id
│
▼
recover stateThe same idea applies at smaller units of work. Individual messages, files, orders, workflow steps, or records may need stable identities so that repeated execution can be recognized as another attempt at the same logical operation.
This matters because restarts naturally create repetition.
A process can fail after doing something but before recording that it did it. From the replacement process’s perspective, that operation may still appear incomplete.
The system therefore needs to tolerate uncertainty around the interruption boundary.
Stable identity gives it something durable to reason about. Instead of asking whether “some previous process” performed an operation, it can ask whether a specific logical piece of work has already reached a known state.
Idempotency Makes Repetition Safe
The most dangerous restart point is between a side effect and the record that says the side effect happened.
Imagine a worker that charges an account and then records completion:
Charge account
│
▼
External payment succeeds
│
X process dies
│
▼
Record "payment complete"The restart sees no completion record.
If it simply repeats the operation, the customer could be charged twice. If it assumes the operation succeeded, it could skip a payment that actually failed before reaching the provider.
This ambiguity is fundamental in distributed systems. A process cannot make interruption occur only at convenient boundaries.
Restartable systems therefore depend heavily on idempotent operations.
If the payment request carries a stable idempotency key, retrying the same logical operation can return the previous result instead of creating another payment:
payment operation
key = order-4821-payment
│
▼
provider succeeds
│
X
crash
│
▼
restart
│
▼
retry same key
│
▼
existing result returnedNot every operation is naturally idempotent, but the same principle can often be implemented through unique constraints, operation records, deduplication keys, conditional updates, or APIs designed to recognize repeated requests.
The important property is that replay does not blindly multiply effects.
This is what allows a restart strategy to be conservative. When the system cannot prove that an operation completed, it can retry safely rather than requiring an operator to inspect the world and decide what happened.
Recovery Needs Deterministic Progress
Restartability also becomes harder when the order and selection of work change unpredictably between attempts.
Suppose a job repeatedly runs a query equivalent to “give me some unprocessed records” without stable ordering. The first process may receive one set, while the restarted process sees a different arrangement because new records arrived or the database chose another query plan.
That does not necessarily make recovery impossible, but it makes “where was I?” much less meaningful.
A restartable process should instead be able to derive its next work from durable facts.
For example, individual records might have explicit states:
record A complete
record B complete
record C pending
record D pendingThe replacement process does not need the previous process’s memory of what it was doing. It can inspect durable state and determine which work remains.
Deterministic ordering can strengthen that model when order matters. Stable IDs, sequence numbers, timestamps with suitable tie-breakers, or explicit workflow transitions can ensure that repeated execution sees the same logical progression.
The deeper principle is deterministic progress, not merely sorting.
Given the same durable state, the system should be able to make a predictable decision about what has completed and what can happen next. If recovery depends on ephemeral memory, random ordering, or reconstructing the intentions of a dead process, restartability remains fragile.
Incomplete Side Effects Are the Real Recovery Problem
A process rarely fails neatly between two complete operations. It can disappear while interacting with a database, object store, external API, message broker, filesystem, or another service.
That means a restart strategy has to consider incomplete work explicitly.
Some operations are easy to repeat. Writing a file to a deterministic location or setting a database value to a desired state can often be made naturally idempotent.
Other operations require additional protection. Sending an email, publishing a message, charging a payment method, or invoking an external system may create an externally visible effect that cannot simply be erased.
The recovery design should therefore ask what happens if the process dies immediately before and immediately after every important side effect, a question that also sits at the center of agent transaction boundaries.
interruption
↓
───────────────X────────────────
external operation
Did it happen?
Did it partly happen?
Can it be detected?
Can it be repeated?
Can it be compensated?Those questions are much more useful than assuming the failure window is too small to matter.
When an operation can safely be retried, recovery is straightforward. When it cannot, the system may need a durable operation record, an idempotency mechanism provided by the downstream system, reconciliation against external state, or a compensating action.
The goal is not necessarily exactly-once execution of every instruction. The practical goal is that uncertain execution does not require manual repair to determine a safe next action.
Once that property exists, retry and replay become ordinary recovery mechanisms rather than dangerous last resorts.
Checkpoints Record Position, but Position Is Not Necessarily Truth
Checkpointing is a common restart technique.
A long-running process periodically records where it has reached:
items 1–1000 complete
checkpoint = 1000
items 1001–2000 complete
checkpoint = 2000
items 2001–...
X crashAfter restarting, it loads the checkpoint and resumes from there.
This can be extremely useful, particularly when replaying all previous work would be expensive. The danger is treating the checkpoint as stronger evidence than it really is.
A checkpoint often records position, not the complete truth about every side effect before and after that position.
Consider:
process item 2001
│
▼
external side effect succeeds
│
X crash
│
▼
checkpoint still says 2000Resuming from 2001 repeats work whose side effect may already exist.
The reverse ordering has its own danger:
checkpoint says 2001 complete
│
X crash
│
▼
external side effect never happensNow recovery may skip work that was never actually completed.
Checkpointing therefore works best when combined with the other restartability properties rather than used as a substitute for them. Stable work identity, idempotent operations, durable state, and deterministic progress make replay around a checkpoint safe.
The checkpoint then becomes an optimization:
Durable truth
│
├── what work exists
├── what completed
└── what can safely repeat
Checkpoint
│
└── where recovery can efficiently resumeThat is an important distinction. A checkpoint can tell you where to start looking; it does not automatically prove what happened.
The Best Restartability Test Is to Kill the Process
Restartability is difficult to establish purely through code review because the dangerous cases occur at interruption boundaries.
The most direct test is therefore the one implied by the design requirement: kill the process while it is working.
Do it at inconvenient moments.
Terminate it while a batch is partially complete, during retries, around external operations, after writing some durable state, and while work remains queued. Then restart it and observe whether the system can recover without an operator editing database rows, deleting partial files, resetting flags, or guessing whether an external action happened.
The test is simple:
Start work
│
▼
Kill process at arbitrary point
│
▼
Restart
│
▼
Does it recover safely?
/ \
yes no
│ │
▼ ▼
good hidden recovery dependencyRepeated kill/restart testing exposes assumptions that normal success-path testing rarely finds. An in-memory counter suddenly matters, unstable ordering becomes visible, duplicate side effects appear, or a checkpoint turns out not to represent what developers thought it represented, much like configuration drift only becomes visible when environments are forced to prove what state they actually hold.
The strongest version of this testing is intentionally adversarial. A system should not only recover when terminated immediately after a carefully chosen checkpoint; it should tolerate interruption around the boundaries where state and side effects can disagree.
If killing the process requires a runbook explaining which tables an operator must repair before restarting it, the process is not meaningfully restartable yet.
Restartability Buys Operational Simplicity
Designing for restartability requires more thought up front. Work needs identity, progress must survive the process, operations need safe retry semantics, and side effects have to be considered around failure boundaries.
The payoff is simpler operations.
Deployments become less frightening because replacing a worker does not imply abandoning its current work. Containers can be rescheduled, unhealthy processes can be terminated, machines can reboot, and jobs can resume after transient infrastructure failures without turning every interruption into a recovery exercise, especially when Kubernetes termination is treated as a normal lifecycle event.
The same property also reduces pressure to make processes immortal. Operators do not need increasingly complicated mechanisms to preserve one particular worker because the system is designed to tolerate that worker disappearing.
Non-restartable system
process failure
│
▼
inspect partial state
│
▼
determine what happened
│
▼
manually repair
│
▼
carefully resume
Restartable system
process failure
│
▼
restart
│
▼
recover durable state
│
▼
retry / replay safely
│
▼
continueThat is why restartability is better understood as a feature than a recovery hack. It changes interruption from an exceptional event requiring forensic work into an expected transition the software already knows how to handle.
The standard for long-running work can therefore remain deliberately severe: can the process be killed at an arbitrary point and restarted without a human repairing its state?
Meeting that standard requires durable external state, stable work identity, idempotent operations, deterministic progress, and deliberate handling of incomplete side effects. Checkpoints can make recovery faster, but they are useful only when the system understands that recorded position and actual truth are not always the same thing.
When those properties are built in, retries and replay stop being frightening. Processes become disposable, recovery becomes routine, and the system becomes easier to deploy and operate precisely because it no longer depends on uninterrupted execution.
