An order workflow reserves inventory, authorises payment, and asks a fulfilment service to create a shipment. The payment provider succeeds, but the coordinator loses the response. Restarting the coordinator does not answer whether another authorisation would repeat the effect.
The orchestrator pattern gives a workflow an explicit place to track progress and decide what should happen next. It can coordinate dependencies, wait for events, apply workflow policy, and drive recovery. The participating services still need to enforce their own domain rules and permissions.
A coordinator is therefore more than a sequence of remote calls, but it is not a universal transaction manager by default. Its reliability depends on durable intent, supported operation identities, clear state transitions, and evidence sufficient to resolve partial or uncertain outcomes.
Define What the Workflow Owns
An orchestrator can own the rule that shipment creation follows confirmed payment and stock reservation. That ordering is itself business policy. The useful boundary is not that the coordinator contains no business logic, but that it does not duplicate every participating service’s domain invariants.
Inventory should decide whether stock can be reserved, and the payment service should enforce its authorisation rules. The coordinator consumes those outcomes and decides the next workflow transition. If it independently reproduces every stock or credit calculation, the services can disagree about which rule applies.
Use a defined workflow instance reference and identify the operation each command requests. Keep a distinction between the overall order, the payment operation, and a particular delivery attempt. Those references let recovery ask which obligation remains and what has already been attempted.
Model states according to the decisions they support. Payment requested, payment confirmed, and payment outcome uncertain are more informative than one generic failed flag. A state model should make it possible for another authorised worker to determine what action is valid next.
Persist Progress and Dispatch Intent Together Where Needed
A process-local call stack cannot preserve a long-running workflow through replacement. Store the relevant progress in a durable application state machine or use an appropriate workflow runtime. Temporal’s workflow-execution documentation provides one example of durable execution managed by a platform.
Persisting a next-step field and sending a command remain two operations unless the design connects them. A coordinator can crash after recording the step but before sending, or after sending but before recording its result. Recovery must account for both windows.
A transactional outbox can persist local workflow changes and command-publication intent in one suitable local transaction. A publisher then sends the pending command. This prevents one kind of lost intent but does not make the external service part of that transaction.
Publication can still repeat if acknowledgement or local progress is lost. Retain stable command identities and design the receiver’s processing accordingly. A durable workflow records the obligation; the delivery and business-operation contracts determine how repeated messages are handled.
Resolve Uncertainty Before Repeating Effects
A timeout is not a confirmed rejection. The provider may still be working, may have completed successfully, or may never have received the request. The coordinator needs a supported way to distinguish those cases where the consequence of repetition matters.
A payment service may offer an idempotency key and an operation lookup. Use the documented scope and retention period rather than assuming any repeated identifier is safe forever. If the workflow can retry after the provider forgets the key, recovery needs another policy.
Checking a processed flag and then calling the provider is insufficient by itself. Two workers can both observe the flag as absent, and either can crash after the effect but before saving completion. An atomic claim controls local competition but does not remove the external-effect window.
When reconciliation cannot establish the outcome safely, retain an explicit uncertain state and escalate according to policy. Marking the attempt failed to make progress can cause a duplicate action. The discussion of timeouts and continuing work explains why the caller’s observation is limited.
| Workflow condition | Appropriate question | Potential next step |
|---|---|---|
| Confirmed rejection | Is the failure recoverable under policy? | Retry later or end the workflow |
| Confirmed success | Is the next dependency now satisfied? | Persist progress and continue |
| Unknown outcome | Can the service identify the operation? | Reconcile or request intervention |
| Failed compensation | What remains changed? | Retain repair work and escalate |
| Duplicate delivery | Is this the same intended command? | Return or recover the recorded outcome |
Treat Compensation as Another Business Operation
AWS’s saga-orchestration guidance describes coordinating local transactions and compensating actions. Compensation is useful when a later failure requires an earlier effect to be addressed. It is not equivalent to rolling every service back to a shared earlier instant.
Releasing reserved stock, voiding an authorisation, refunding a captured payment, and cancelling a shipment have different meanings and availability. A carrier may already have collected the parcel. The policy needs to say what can be compensated and what requires another resolution.
Compensation can fail, time out, or be delivered more than once. Give it stable identities, durable progress, and a repair process just like the forward steps. A workflow is not resolved merely because it attempted an undo command.
Sometimes forward recovery is preferable: wait for stock, ask the customer to choose an alternative, or complete an outstanding update. The coordinator needs accountable business decisions about those options, rather than an automatic assumption that every failure should reverse all earlier work.
Protect the State Machine From Concurrent Transitions
A logical coordinator need not be one physical process. Workers can scale and recover while the workflow’s state is protected by the runtime or an appropriate concurrency mechanism. The design must prevent two workers from advancing the same instance inconsistently.
Use conditional state transitions, version checks, or the platform’s supported execution guarantees. A late response should be interpreted against the current state and operation identity. A duplicated completion event must not silently start a second shipment.
Timeouts and external messages can race. The workflow may schedule a timeout recovery while a success response arrives. Define which transitions remain valid and how a later message is reconciled. Merely ordering timestamps from separate machines may not establish the correct decision.
Audit records should identify the transition, its cause, the previous relevant version, and the operation references. They help explain why the workflow advanced, while durable state supports the next action. Correlation and tracing can connect execution evidence without replacing that state.
Handle New Requests While the Workflow Waits
A customer may cancel an order while a payment or stock reservation is still pending. Model that request as another event with a defined policy, rather than deleting the workflow instance. Deletion can remove the context needed to interpret a late success and release the resulting obligation.
Decide what cancellation means at each stage. Before dispatch it may prevent an operation; after confirmation it may require compensation; during an uncertain result it may need reconciliation first. These are different transitions even when the interface offers one cancel button.
Record the accepted intent and the actual completion separately. Telling the customer that a cancellation request was received is not the same promise as confirming that shipment and payment effects have been resolved.
Plan for Workflow Versions and Operational Limits
Long-running instances may outlive a deployment. A new workflow definition can interpret an old state differently or introduce an unsupported step. Establish a compatibility and migration policy rather than assuming every instance restarts under the latest code safely.
Some durable runtimes reconstruct execution using history. Temporal’s event-history documentation illustrates that model. Its determinism and versioning requirements are part of the runtime contract, not properties every custom coordinator acquires automatically.
Bound retries, queue growth, and waiting time according to the business requirement. An indefinitely pending workflow can retain resources and conceal obligations long after a customer expects an answer. Define expiry and escalation without discarding the evidence needed for repair.
Give operators controlled actions for reconciliation, retry, compensation, and cancellation. These are consequential operations and need permissions, validation, and an audit trail. Editing a state field directly to make the dashboard green can leave the participating systems inconsistent.
Choose Coordination Where It Clarifies the Whole Operation
Orchestration makes a workflow’s sequence and recovery policy visible in one logical place. Choreography distributes reactions among participants through events. Both still need state, identity, compatibility, and ownership; neither removes distributed failure.
A mixed design can be appropriate. Independent analytics consumers may react to an order event, while payment and fulfilment follow an explicitly coordinated process. Choose according to which decisions need a shared workflow view and which work can proceed independently.
Test failure windows at meaningful boundaries: before dispatch, after an external effect, before progress persistence, during a concurrent completion, and during compensation. Verify that another worker can resume using durable evidence. The valuable outcome is a workflow whose next decision remains explainable when execution is interrupted, rather than a coordinator that only works while every call returns normally.
