Serverless platforms are often described as stateless.
Strictly speaking, they aren’t.
A serverless function doesn’t own long-lived application state in the way a traditional server does, but the execution environment often survives between invocations. AWS documents this Lambda execution environment reuse: global objects, connections, temporary files, and background work can persist when an environment is reused. Global variables remain in memory, /tmp files persist, database connections stay open, and configuration loaded during startup may continue to be reused.
The result is a confusing programming model: state may exist, disappear, or be replaced entirely depending on how the platform manages the underlying container.
That’s why serverless functions are better described as having uncertain state lifetimes than being truly stateless.
Cold Start │ Initialisation │ Invocation 1 │ Invocation 2 │ Invocation 3 │ Container Recycled │ Cold Start Again
Warm Containers Remember
A cold start provisions an execution environment, loads the runtime, runs module initialization, and invokes the handler. A warm invocation reuses that environment. The handler runs again, but initialization does not.
That reuse is the reason warm invocations are faster. It is also the reason old state is still there.
request_count = 0
def handler(event, context):
global request_count
request_count += 1
return {"count": request_count}In a clean-process model, the counter always returns 1. In a reused container, it increments until the container disappears. It is neither a reliable counter nor a clean scratch variable.
The same pattern breaks initialization flags. If setup fails but a global initialized flag is set, warm invocations can skip setup forever. If setup is expensive and the flag resets on cold start, latency jumps unpredictably.
/tmp Is Small, Persistent, and Easy to Forget
Serverless functions can write to temporary storage. Temporary does not mean per-invocation.
const fs = require('fs');
const path = '/tmp/data.json';
exports.handler = async (event) => {
if (fs.existsSync(path)) {
const data = JSON.parse(fs.readFileSync(path));
data.count += 1;
fs.writeFileSync(path, JSON.stringify(data));
} else {
fs.writeFileSync(path, JSON.stringify({ count: 1 }));
}
return { statusCode: 200 };
};The file survives warm invocations. It disappears on a new container. A function that expects a clean directory may read stale files. A function that writes small files and never deletes them may eventually fill the available space.
AWS Lambda’s /tmp has a size limit. The failure may arrive after hundreds of individually harmless invocations. No single request wrote too much. The container remembered too much.
Connections and Config Go Stale
Opening a database connection outside the handler is common because connection setup is slow.
import psycopg2
conn = psycopg2.connect(database="prod", user="app")
def handler(event, context):
cursor = conn.cursor()
cursor.execute("SELECT * FROM users WHERE id = %s", (event['id'],))
return cursor.fetchone()The connection persists across invocations. That is efficient until the database closes it, the network changes, credentials rotate, or the pool leaks a connection from a previous invocation. The next request inherits the broken state.
Module-level configuration has the same problem.
const config = fetchConfigFromS3();
exports.handler = async (event) => {
console.log(config.apiKey);
};fetchConfigFromS3() runs once per container. If the config changes, warm containers keep the old value. New containers get the new one. For a while, both configurations are live.
Environment variable updates can behave similarly. Some containers may see old values until replaced. If the variable controls a feature flag or API key, requests split across versions of configuration without an obvious deployment boundary, creating the same kind of configuration drift seen in longer-lived services.
Memory Leaks Survive the Handler
A serverless process may handle thousands of invocations before it is recycled.
let cache = [];
exports.handler = async (event) => {
cache.push(event.data);
return cache.length;
};Each invocation completes. The array remains. Memory rises until the container fails or gets replaced. The error looks intermittent because cold containers start clean and warm containers carry history.
Developers who would never leave an unbounded global cache in a long-running service sometimes do it in serverless code because the function feels short-lived. The invocation is short-lived. The process may not be.
Background work creates another version of the same issue, especially when timeouts do not actually cancel work. A module starts a metrics thread during initialization. The handler finishes in 100ms, but the thread keeps running between invocations. Per-invocation timeout limits do not bound per-container background activity.
Scale Creates External State Pressure
Concurrent traffic starts more containers. Each container has its own globals, files, connections, and caches. They share nothing in memory, but they all hit the same external systems.
One container opens one database connection during initialization. One hundred simultaneous requests can create one hundred containers and one hundred connections. If the database limit is fifty, half the containers fail or block.
The development environment with one warm container looks fine. Production scale-up turns hidden per-container state into external contention: database connections, API rate limits, file handles, queue writes, log volume, and downstream capacity.
Serverless removes server management. It does not remove shared-resource math.
Deployments Leave Old State Behind
A new function version does not instantly erase old containers. The platform brings up new environments while existing warm environments may continue serving requests.
If the new version writes a different database shape, old and new shapes coexist. If it sends a different queue message format, consumers see both. If it changes a cache key or environment variable contract, warm containers can keep the previous behavior for some period.
The deployment is a transition, not a cut line, which is the same operational shape behind model version drift. Schema changes, message formats, and external side effects need compatibility across old and new containers.
Concurrency Can Share a Container
Many people learn “one invocation per container” and stop there. Some platforms and configurations allow concurrent invocations inside the same environment. Async runtimes can also interleave work inside one event loop.
Logs begin to mix:
Processing user 123
Processing user 456
Completed user 123
Completed user 456Shared globals become shared between invocations, not just across invocations. Code that was safe under single-invocation execution can corrupt state when concurrency settings change.
The isolation boundary is provider-specific and configuration-specific. Treating it as universal is how hidden state becomes a race condition.
Cold Starts Reset the Wrong Things
Warm state helps performance until a cold start removes it.
A global cache makes warm requests fast and cold requests slow. A deduplication set stored in memory catches retries until the container is replaced. After a cold start, the same retry looks new. A configuration value cached at startup works until credentials rotate and the warm container keeps using the old key.
The cache hit rate reflects platform reuse behavior as much as application behavior. The system may look fast, slow, correct, or broken depending on whether the request landed on a warm environment.
Stateless Functions Still Create Stateful Systems
Even if internal state vanished perfectly, the function still writes state elsewhere.
A function creates DynamoDB rows, S3 objects, SQS messages, CloudWatch logs, metrics, temporary export files, and audit records. Without lifecycle management, those external artifacts accumulate. Costs rise. Queries slow down. Queues fill. Old objects linger beyond their usefulness.
Serverless encourages small handlers. It does not automatically provide cleanup policy, retention design, idempotency, or external state ownership.
Observability Often Measures the Wrong Unit
Provider metrics are usually per invocation: duration, memory, error rate. Hidden state lives across invocations and per container.
Those metrics may miss global variable growth, /tmp usage, stale connection pools, background threads, leaked memory, config skew, and old containers after deploys. A handler can report 100ms duration while the container carries a leaking cache. Memory can look fine on cold invocations and fail only on warm ones.
Useful monitoring needs container-lifetime signals too, plus the structured logs and trace identifiers needed to connect one invocation to the next: initialization count, warm invocation count, /tmp usage, connection reuse failures, config version seen by each invocation, cache size, background worker health, and cold-start rate.
Stateless Does Not Mean Idempotent
Another common misconception is that stateless functions are automatically safe to retry.
They are not.
A function may keep no useful state between invocations and still perform an operation that cannot safely happen twice.
Consider a payment handler.
The function receives an order, charges a customer’s card, and records the transaction.
If the platform retries the invocation because of a timeout after the payment succeeded but before the response reached the caller, the function may charge the card a second time. The function itself remained stateless. The external system did not.
Idempotency protects that external state.
An idempotent operation produces the same result no matter how many times it is repeated with the same request. Instead of charging the payment every time, the function first checks an idempotency key or transaction identifier. If the request has already been processed, it returns the previous result instead of performing the operation again.
This becomes even more important in serverless platforms because retries are part of normal operation.
Functions may be retried after timeouts, transient network failures, throttling, asynchronous event processing, or partial infrastructure failures. Most retries are automatic. Some occur long after the original invocation has already completed successfully.
The platform cannot know whether your business operation already happened.
Only your application can.
Designing a function to be stateless makes it easier to scale and replace execution environments. Designing it to be idempotent makes retries safe. The two concepts solve different problems, and confusing them often leads to duplicate payments, repeated notifications, multiple database writes, or other unintended side effects.
Serverless functions should assume they may execute more than once for the same event, especially when retry policies amplify failures instead of containing them. Correctness comes not from hoping retries never happen, but from making repeated execution produce the same business outcome.
Design for Uncertain State Lifetime
Serverless state has a strange contract: it may persist, and it may disappear.
Use global state for safe caches, not correctness. Validate connections before reuse. Clean /tmp. Bound memory. Make external writes idempotent. Design deployments so old and new containers can overlap. Store durable workflow state outside the function. Give external artifacts retention and cleanup rules.
A serverless function is stateless only if the system design makes hidden state irrelevant. Otherwise the platform remembers just long enough to make the bug intermittent.
