A distributed workflow often begins simply. One service receives a request, calls another service, waits for a result, calls a third service, and eventually completes the operation.
The difficulty appears when the workflow becomes longer and failures become normal rather than exceptional. A payment succeeds but inventory reservation fails, a downstream service times out after completing its work, or the process crashes halfway through an operation that has already changed several systems.
The orchestrator pattern addresses that problem by giving one component responsibility for coordinating the workflow. The orchestrator owns the sequence, workflow state, and recovery process, while participating services continue to own the business operations they perform.
ORCHESTRATOR
sequence + state + recovery
│
┌─────────────┼─────────────┐
▼ ▼ ▼
Payment Inventory Shipping
│ │ │
▼ ▼ ▼
payment rules stock rules shipping rules
SERVICES OWN BUSINESS OPERATIONSThat boundary is crucial. An orchestrator should know that payment must happen before shipping, but it should not become the place where payment rules, inventory logic, and shipping decisions are implemented. Once it starts absorbing those responsibilities, central coordination can quietly turn into a distributed monolith.
The Orchestrator Owns the Sequence and Workflow State
Consider an order workflow that requires payment authorization, inventory reservation, and shipment creation. Those operations may belong to independent services, but the overall process still has an order.
Order created
│
▼
Authorize payment
│
▼
Reserve inventory
│
▼
Create shipment
│
▼
Order completeSomeone has to know what has already happened and what should happen next. In an orchestrated workflow, that responsibility belongs to the coordinator.
The orchestrator might send an AuthorizePayment command and wait for the result. If authorization succeeds, it records that progress and moves to inventory; if inventory succeeds, it proceeds to shipping.
That sounds similar to ordinary application code calling several functions, but distributed execution introduces an important difference: the workflow cannot safely exist only on the current call stack.
A process can crash after payment succeeds but before inventory is requested. A network timeout can leave the coordinator uncertain whether a command failed or merely whether its response was lost. A workflow may also pause for minutes, hours, or longer while waiting for another system.
The coordinator therefore needs durable workflow state, the same foundation that makes restartability possible after a process disappears.
Conceptually, it might preserve something like:
workflow_id: order-4821
state: inventory_pending
completed:
- payment_authorized
next:
- reserve_inventoryThe exact representation varies, but the principle is more important than the storage format. The workflow’s progress must survive the process currently executing it, which is why workflow engines such as Temporal treat workflow state as durable execution history rather than stack memory.
Durability changes the model from “run these calls in order” to “advance a persistent state machine.” After every meaningful transition, the orchestrator knows where the workflow stands and what transition is valid next.
Retries Require Idempotent Business Operations
Distributed systems fail in ambiguous ways.
Suppose the orchestrator asks the payment service to authorize a £100 payment. The payment service completes the authorization, but the response disappears because the connection fails.
From the orchestrator’s perspective, the operation is unresolved:
Orchestrator Payment
Authorize £100 ──────────────►
authorization succeeds
X ◄─────────────
response lostRetrying is a reasonable recovery strategy, but blindly repeating the operation could authorize the payment twice, especially when timeouts do not cancel work and the first attempt may still complete.
This is why retries and idempotency belong together.
The orchestrator can attach a stable operation or idempotency key to the command. If it retries because the outcome was uncertain, the payment service recognizes that the logical operation has already been handled and returns the existing result instead of performing another payment.
Authorize payment
key = order-4821-payment
│
▼
timeout
│
▼
Retry same operation
key = order-4821-payment
│
▼
Payment service recognizes duplicate
│
▼
Return existing resultThe ownership boundary still matters here. The orchestrator decides whether and when the workflow should retry payment, but the payment service decides how payment authorization behaves idempotently.
The coordinator should not duplicate the payment service’s internal rules in an attempt to make retries safe. Services remain responsible for making their own business operations safe under the delivery semantics the workflow expects.
Retries also need limits. A transient timeout may justify exponential backoff and another attempt, while a declined payment is a business result rather than a temporary infrastructure failure.
The orchestrator can own the recovery policy without pretending that every unsuccessful response means the same thing.
Failure Handling Sometimes Requires Compensation
Retries solve failures where trying the same operation again may succeed. They do not solve a workflow in which earlier steps succeeded and a later step cannot complete.
Suppose payment has been authorized and inventory reserved, but shipment creation fails permanently:
Authorize payment ✓
Reserve inventory ✓
Create shipment ✗There is no database transaction spanning those independent services that can simply roll everything back. The system may instead need compensating operations that deliberately undo or offset work already completed.
Create shipment fails
│
▼
Release inventory
│
▼
Void payment authorization
│
▼
Workflow compensatedCompensation is not the same as database rollback. A refund does not erase the fact that a payment happened, cancelling a reservation may have its own business consequences, and some real-world actions cannot be perfectly reversed at all.
The orchestrator’s role is to know which compensating steps belong to the workflow and when they should be invoked. The participating service still owns the actual business meaning of that compensation.
For example, the coordinator may request ReleaseInventory, but the inventory service determines how a reservation is released. Likewise, the orchestrator can request VoidAuthorization, while the payment service owns the rules governing whether an authorization can still be voided or must instead be refunded.
That separation keeps workflow recovery centralized without centralizing every business rule.
Durable State Makes Restart and Resume Possible
A workflow engine that handles retries and compensation still has another failure to survive: its own.
Imagine the orchestrator successfully reserves inventory and then crashes before requesting shipment creation. If progress existed only in memory, restarting the process leaves it unable to distinguish a new workflow from one that has already completed several steps.
Durable state provides the answer:
Before crash
payment_authorized ✓
inventory_reserved ✓
shipment_created -
CRASH
After restart
load workflow state
│
▼
inventory_reserved
│
▼
resume with shipment creationRestartability is one of the major advantages of explicit orchestration. The coordinator can reconstruct the workflow from durable state instead of relying on a long-lived process surviving from beginning to end, and it is easier to trust when stable operation identity lets attempts refer to the same logical work.
That becomes especially important for workflows that naturally span time. A process might wait for a human approval, an external callback, a delayed retry, a scheduled event, or a message from another system before it can continue.
The workflow is still logically active even though no application thread should remain blocked waiting for it.
A durable orchestrator can record that it is waiting, stop executing, and resume when the relevant event arrives. The workflow becomes a persistent state machine rather than a fragile chain of synchronous calls.
This also makes operational investigation easier. Instead of knowing only that an order is “stuck,” engineers can inspect the workflow state and determine whether it is waiting for payment, retrying inventory, awaiting approval, compensating a previous action, or permanently failed, especially when distributed tracing connects the workflow to the service calls it made.
Orchestration and Choreography Put Control in Different Places
Central orchestration is not the only way to coordinate distributed work.
In a choreographed system, services react to events without one component owning the entire sequence. An order service might publish OrderCreated, causing payment to react; payment publishes PaymentAuthorized, inventory reacts to that event, and the workflow emerges from those interactions, a style often associated with event-driven architecture.
Choreography
Order
│ OrderCreated
▼
Payment
│ PaymentAuthorized
▼
Inventory
│ InventoryReserved
▼
ShippingThis removes a central coordinator and can allow services to evolve around event-driven interactions. It works particularly well when events represent facts that several independent consumers may legitimately react to.
The trade-off is that the workflow becomes distributed across those reactions. Understanding the complete sequence may require inspecting several services, while retry behaviour, failure handling, and compensation can become harder to see because no single component necessarily owns the end-to-end process, making observability signals more important.
Orchestration makes that control explicit:
Orchestration
Orchestrator
/ | \
▼ ▼ ▼
Payment Inventory ShippingNeither approach is inherently superior. The useful question is where workflow ownership belongs.
If a process has a clear sequence, complex recovery rules, durable waits, compensating actions, or a need to expose its current state, central orchestration can make those concerns much easier to reason about. If services are mostly reacting independently to business events and no single sequence needs central ownership, choreography may preserve looser coupling.
The two approaches can also coexist. An orchestrated order workflow may publish domain events that unrelated services consume choreographically, without forcing every reaction into the central workflow.
Central Control Helps Only While Its Boundary Stays Narrow
The orchestrator pattern becomes dangerous when “owns the workflow” gradually turns into “owns the business.”
Imagine an order orchestrator that begins with straightforward coordination:
1. Request payment authorization
2. Reserve inventory
3. Create shipmentOver time, developers may find it convenient to put more decisions there. The orchestrator starts calculating payment eligibility, deciding which warehouse should fulfill an order, applying inventory rules, calculating shipping prices, and determining refund policy.
The system still has separate payment, inventory, and shipping services, but those services have become increasingly passive. The real business logic has accumulated in the coordinator.
That creates a distributed monolith:
ORCHESTRATOR
┌─────────────────────────────────┐
│ payment rules │
│ inventory rules │
│ shipping rules │
│ workflow sequence │
│ retry policy │
│ compensation logic │
│ recovery state │
└─────────────────────────────────┘
│ │ │
▼ ▼ ▼
Payment Inventory Shipping
"do it" "do it" "do it"The services may be deployed independently, but meaningful changes still depend on the central component. The architecture has gained network boundaries without gaining useful ownership boundaries.
A healthier division is narrower:
ORCHESTRATOR OWNS
────────────────────────
sequence
workflow state
recovery
retries
compensation flow
SERVICES OWN
────────────────────────
business rules
domain validation
local state
business operations
operation semanticsThis does not mean the orchestrator contains no decisions. It necessarily decides what transition follows a particular workflow outcome and how the overall process should recover from failure.
The distinction is about which kind of decision belongs there. “If payment succeeds, attempt inventory reservation” is workflow logic; “is this customer eligible for this payment method?” is payment-domain logic and belongs with the service that owns payment, a boundary that can drift when configuration drift and ad hoc overrides start changing runtime behavior.
That boundary determines whether orchestration clarifies a distributed workflow or merely creates a central brain surrounded by remote functions.
Central coordination also has a cost even when designed well. The orchestrator becomes important infrastructure, its workflow definitions need careful evolution, and excessive centralization can couple services to one giant process model.
For small interactions, introducing durable workflow machinery may be unnecessary. For loosely related event consumers, choreography may better reflect the domain. The orchestrator earns its complexity when the workflow itself has meaningful state, ordering, recovery, and failure semantics that otherwise become scattered across the system.
The orchestrator pattern is therefore less about having one service call several others and more about giving distributed workflow state an explicit owner.
That owner coordinates sequence, persists progress, decides when to retry, drives compensation, and knows how execution should resume after interruption. The participating services retain ownership of the business operations that make those steps meaningful.
Orchestrator owns sequence, state, and recovery. Services own their business operations. Keeping that line clear gives a distributed workflow central control where central control helps, without allowing the coordinator to become the business logic for the entire system.
