A nightly import stops after applying thousands of records. Starting the executable again is easy. Knowing whether to reapply the last record, skip it, or reconcile an external update is the real recovery problem.
Restartability means that another execution can recover enough durable evidence to continue under the job’s defined guarantees. Some jobs can safely repeat everything. Others need checkpoints, per-item state, or intervention for uncertain effects. The appropriate design depends on what the work changes and what repetition would mean.
A useful recovery contract states the interruptions it handles, the outcomes it preserves, and the conditions requiring escalation. Claiming that any process can always be killed without repair is too broad, especially where external systems or physical actions cannot provide the necessary evidence.
Define the Unit of Work and Its Inputs
A restart needs stable work identity. An import identifier can distinguish another attempt at the same import from a new run with different data. Per-record identities may also be necessary where individual effects are retried or reconciled.
Bind that identity to an understood input. A filename reused every day is not necessarily the same dataset, and a live query can return different rows after a restart. Use an immutable source version, a manifest, a snapshot, or another documented selection rule appropriate to the job.
Record the transformation and policy versions that matter. Resuming halfway through with new code can produce a mixed result if the interpretation of a field changed. Decide whether recovery continues the original transformation, migrates progress, or starts a separately identified operation.
This does not require identical low-level execution on every attempt. Parallel scheduling and retry timing may differ. The requirement is that the intended business outcome remains within the contract despite those differences, with unresolved cases represented explicitly.
Persist Local Results With Progress Where Possible
For a database-only transformation, it may be possible to apply one bounded unit and record its completion in the same transaction. If the transaction fails, neither change commits; if it succeeds, the replacement process can see both the result and progress.
That transaction has a scope. It does not automatically include a file upload, remote API call, or message sent through another system. Identify which effects share the local commit and which remain separate before calling the whole job atomic.
A transactional outbox can record publication intent alongside local changes. A separate publisher delivers it. The pattern supports recoverable intent, while receivers still need to handle repeated delivery or uncertain outcomes according to their operation contract.
Choose transaction size with recovery and resource use in mind. One transaction for an entire large import can hold locks or require repeating substantial work. Smaller committed units can ease recovery, but their partial visibility must be acceptable to downstream readers.
Reconcile Effects That Cross the Transaction Boundary
The dangerous interruption can occur after an external system acts but before the worker stores confirmation. The replacement execution sees no confirmed result, but that absence does not prove the effect never happened.
A stable idempotency key or operation lookup can help when the service supports it. AWS’s discussion of idempotent APIs explains why explicit request identity and intended semantics matter. Repeating a request with a familiar identifier is safe only under the provider’s actual contract.
Check retention and parameter rules. If the provider forgets keys before the job’s possible retry window ends, replay may create a new operation. If the same key is reused with changed inputs, the provider may reject it or apply rules the caller did not expect.
An atomic local claim can prevent two workers from starting the same local unit simultaneously, but it does not make the remote effect and local completion one commit. Preserve an uncertain state and enough references to reconcile. When safe reconciliation is unavailable, escalation can be the correct recovery action.
Compensation may also be necessary. Releasing a reservation or issuing a refund is another operation with its own failures and audit needs, rather than deletion of the original effect. The orchestrator pattern examines that workflow-level responsibility.
Make Checkpoints Match the Progress They Represent
A checkpoint can make recovery faster by identifying work already accounted for. Its meaning must match the source’s ordering and the processing completion model. The maximum identifier seen is not automatically a safe position to resume after.
For example, two concurrent transactions can allocate identifiers in one order and commit in another. PostgreSQL’s sequence documentation also explains that sequence values are not reclaimed when transactions abort. A sequence is therefore neither a gapless ledger nor a general commit-order contract.
Parallel processing creates another distinction. A worker may finish item 102 while item 101 remains in progress. Recording 102 as the completed prefix can skip 101 after a crash unless the checkpoint model accounts for outstanding work separately.
Use a source position with documented semantics, a completed contiguous prefix, or durable per-unit outcomes as appropriate. The article on sequence numbers explains why each ordering claim needs a defined scope.
| Durable information | What recovery uses it for | Assumption to avoid |
|---|---|---|
| Input version or manifest | Recover the same intended source | A reused filename means identical data |
| Unit identity | Recognise a repeated attempt | Every run is necessarily new work |
| Confirmed outcome | Avoid repeating completed effects | Missing confirmation means no effect |
| Checkpoint and outstanding units | Select remaining work | Highest observed position is fully complete |
| Policy or code version | Interpret the intended transformation | Latest code is always compatible with progress |
Control Overlapping Executions
A restart can overlap a worker that is slow rather than dead. A lease or ownership record can reduce that risk, but a paused worker may resume after its lease expires. The design needs a way to reject stale writes or otherwise prevent conflicting progress.
Conditional updates and version checks can protect local state. Where a fencing mechanism is used, the receiving resource needs to enforce the token’s ordering or ownership rule. Merely generating a new token does not stop an old worker from changing an external system that ignores it.
Define shutdown behaviour too. A graceful stop can cease accepting new work, complete or checkpoint an active unit, and release resources. It is useful operationally but cannot replace crash recovery, because a process may disappear without running cleanup.
Keep ownership distinct from outcome. A lease saying worker B is responsible does not establish whether worker A already completed an external effect. That question still requires operation evidence and reconciliation.
Test Interruption at Meaningful Boundaries
Use a controlled environment to interrupt work before an effect, after the effect, before progress persistence, and after progress persistence. Include duplicate delivery, delayed responses, and an old worker resuming. These cases challenge the actual recovery mechanism rather than only checking that a process starts twice.
Verify the business result after recovery. A job that reaches complete while duplicating a charge or skipping an input has not met its contract. Check totals, identities, unresolved units, and the effects that matter for that workflow.
Also test the intended recovery tools and permissions. An operator needs to distinguish safe retry, reconciliation, correction, and a new run. A generic restart button can be misleading when those operations have different consequences.
Record what the tests did not cover. Loss of the durable store, unavailable provider history, or incompatible old input may require another recovery plan. Restartability can be strong without pretending it covers every possible loss of evidence.
Represent Work That Cannot Be Applied
A malformed record can fail on every restart without any infrastructure problem. Decide whether the job stops, quarantines that unit for review, or continues under an explicit partial-completion policy. Silently skipping it makes the progress indicator easier to advance while losing the original obligation.
Keep a rejected or unresolved unit identifiable and explain why it was not applied. Reconciliation should account for accepted, rejected, pending, and uncertain work according to the job’s contract. An overall complete status may legitimately mean that every unit has an accounted-for outcome; it should not imply that every input was successfully applied unless that is what the records establish.
Keep Recovery State Understandable Over Time
Observability should expose remaining work, oldest pending age, retry effort, uncertain outcomes, and confirmed progress. Execution logs help explain a failure, while the durable state should support the decision needed to continue.
Reconciliation and retention need to outlive the ordinary run when necessary. Deleting operation records immediately after success can make a delayed duplicate or later repair difficult to interpret. Define the retained evidence according to delivery windows and the business process.
A restartable job is easier to operate when another worker can establish what it is authorised to do from the records that survived. That is the useful feature: interruption no longer forces the team to reconstruct every decision from memory, while the remaining uncertainty is visible enough to handle deliberately.
