A function receives an order event, calls a payment provider, and times out before saving the result. The next invocation starts on a different execution environment. It can read the order from a database, but it does not yet know whether the payment succeeded.
Moving the order outside the function solved a durability problem. It did not solve the uncertainty between the external payment and the local record. Reliable recovery needs enough durable information, and an appropriate provider operation, to determine what happened without charging the customer again.
This is the useful distinction behind serverless state. The compute can be replaceable while the application remains deeply stateful. The design has to establish where each kind of state lives, what guarantees that store provides, and how work continues when an invocation stops partway through.
Separate the Function From Its Execution Environment
A function is code invoked to handle work. The platform runs that code in an execution environment whose lifetime and reuse depend on the service. A later invocation may reuse an environment, but the application should not require that reuse to preserve a customer’s business state.
AWS Lambda’s best-practice guidance recommends reusing resources such as SDK clients and connections across invocations where possible. This can reduce repeated setup. The guidance also cautions against retaining user data in the execution environment in ways that could expose it across invocations.
That distinction gives local state a legitimate role. A parsed schema or reusable client can be recreated if the environment disappears. A shopping cart stored only in that environment cannot be recovered reliably by another worker. Local files may similarly be useful scratch space without being a durable application record.
A warm environment can also retain stale information. Cached configuration, expired credentials, or objects left over from an earlier request need appropriate refresh and isolation rules. Reuse is an optimisation with a lifecycle, rather than evidence that a process will remain available indefinitely.
Choose Storage by the Guarantee Required
External is not a durability guarantee by itself. A remote cache may evict entries, a queue may expire old messages, and a database’s recovery properties depend on its configuration and operation. Select the service and settings according to what losing or delaying the state would mean.
An order record generally needs durable storage and a defined update model. An uploaded file can live in object storage with an object identifier and version recorded alongside the workflow. A derived lookup result may be cached if it can be reconstructed and its permitted staleness is understood.
Queues hold work awaiting processing, but retention and acknowledgement rules define how long that work remains recoverable. Amazon SQS standard queues document at-least-once delivery, which means duplicate handling belongs in the consumer’s design. A queue should not silently become the only permanent record of a business obligation.
| State | Possible location | Guarantee to establish |
|---|---|---|
| Order and payment attempt | Transactional database | Durable history and controlled concurrent updates |
| Uploaded document | Object storage | Retention, access, and identifiable versions |
| Pending task | Queue plus business record where needed | Delivery, retention, and recovery policy |
| Workflow progress | Durable orchestration service or application store | Resume behaviour and effect handling |
| Reusable client or parsed schema | Local memory | Safe reconstruction after replacement |
| Derived query result | Local or shared cache | Acceptable loss and staleness |
The distinction is not that everything outside the worker is safe. It is that important state needs an explicitly supported lifetime and recovery model independent of any particular worker.
Shared State Still Needs Concurrency Control
Two function invocations can both read that one unit of stock remains and both attempt to sell it. Saving inventory in a shared database makes the record reachable by both workers, but a separate read followed by an unconditional write does not prevent the race.
Use an operation whose concurrency guarantee matches the decision. A conditional update can require sufficient remaining stock or an expected version before applying the change. DynamoDB’s condition expressions provide one concrete example of conditional writes; transactional databases offer other mechanisms with their own scope and isolation rules.
The losing invocation needs a defined outcome. It might report that stock is unavailable, reload the latest state, or retry a decision that is still valid. Repeating the original write without reconsidering its assumptions simply repeats the conflict.
Version checks also need meaning. An entity version can guard against an update based on an older revision, while a sequence allocation does not necessarily prove which transaction committed first. The article on sequence-number semantics examines that distinction in detail.
Retries Need More Than a Processed Flag
Retry behaviour depends on the invocation mechanism. AWS documents different responsibilities for direct invocation and integrations. Do not assume every serverless call is automatically retried, or that every trigger shares the same retry count, ordering, and failure handling.
For work that can be repeated, choose a stable operation identity. Two deliveries of the same payment request should refer to the same intended operation. Merely checking a processed flag and then performing the effect is insufficient: concurrent workers can both observe that the flag is absent, and a crash can occur after the effect but before the flag is saved.
An atomic claim or conditional transition can control which worker proceeds, but the external-effect boundary remains. A payment provider’s supported idempotency key or lookup operation may allow the application to reconcile an uncertain attempt. Its retention period and semantics need to cover the application’s possible retries.
If the provider cannot support safe reconciliation, the workflow needs a policy for uncertain outcomes, potentially including manual review. Recording the uncertainty is more accurate than marking the payment failed simply because the invocation timed out. A timeout describes the caller’s observation, not necessarily the provider’s final result.
Persist Intent When Publishing Work
Another failure window appears when a function updates a database and then publishes a message. The database commit may succeed while message publication fails. Reversing the order creates the opposite possibility: a message is visible even though the intended database change never commits.
A transactional outbox addresses this boundary by writing the business change and an outbound-message record in the same local database transaction. A separate publisher sends outstanding messages and records progress. This requires an appropriate transactional store; two unrelated writes do not become atomic merely because the application calls them an outbox.
The publisher can still send a message more than once if it crashes after sending but before recording completion. Consumers therefore need an appropriate deduplication or idempotent-processing strategy. The outbox preserves the intent to publish; it does not create a universal transaction across every downstream service.
Apply the same reasoning to acknowledgements. A worker should acknowledge according to the trigger’s documented contract and its own durable completion policy. If work is acknowledged too early, a crash can lose the obligation. If acknowledgement is delayed or lost, redelivery must be safe.
Durable Workflows Preserve Progress, Not Universal Atomicity
A long-running workflow can persist its progress and schedule individual activities without keeping one function invocation alive throughout. Azure’s durable orchestration documentation describes checkpointing and history-based recovery. This lets the programming model expose durable progress while the runtime manages its persistence.
Such services qualify the claim that serverless code must always manage every state transition explicitly. A framework can manage substantial workflow state on the application’s behalf. Its guarantees still need to be understood, including replay constraints and the handling of activities that interact with external systems.
A completed payment, reserved stock item, and shipping request do not automatically form one atomic transaction because an orchestrator coordinates them. Failed workflows may need compensating operations, reconciliation, or intervention. Compensation itself is another business operation, such as issuing a refund, rather than erasing the original payment from history.
Keep the durable business record aligned with the workflow’s intended outcome. A workflow instance marked complete is useful evidence, but users may also need an order status, provider reference, and explanation of any repair. Those records make the result understandable outside the orchestration engine.
Test Replacement and Recovery as Normal Operation
Scaling increases the number of workers that can compete for database connections, provider quotas, and shared records. Bound concurrency according to downstream capacity rather than treating the platform’s ability to start more invocations as proof that the complete system can handle them.
Observe the durable operation identifier, attempt, entity version, and confirmed outcome. Avoid relying on logs from one invocation as the only record of business completion. Logs help reconstruct execution, while the application’s durable state should support the decisions needed for recovery.
Test failures at meaningful boundaries: after claiming work, after the external effect, before saving its result, and before acknowledgement. Also test a replacement environment with no local cache. These cases reveal whether the application can continue from its records or depends on information that disappeared with the worker.
The resulting design can use local memory efficiently and delegate persistence to managed services without confusing either choice with statelessness. The important outcome is that another authorised worker can determine what remains to be done, what has already happened, and which uncertainty must be resolved before acting again.
