Serverless functions are marketed as stateless. Each invocation is independent. No shared memory. No persistent processes. Functions start clean, execute, and terminate.
This is not true.
Serverless functions maintain state across invocations. Global variables persist. File system changes accumulate. Network connections stay open. The execution environment is reused.
The state is hidden, undocumented, and unpredictable. It creates failure modes that developers do not anticipate because they trust the stateless abstraction.
The Container Persists Between Invocations
A serverless function does not start from scratch on every invocation. The runtime reuses containers.
When a function is first invoked, the provider provisions a container, loads the runtime, executes initialization code, and runs the handler. This is a cold start.
When the function is invoked again within minutes, the same container is reused. Initialization code does not run again. The handler runs in an already-initialized environment. This is a warm invocation.
The warm invocation is faster because initialization is skipped. It is also stateful because the container has not been reset.
Global variables retain their values. Imported modules are already loaded. Connections opened during initialization remain open. The file system contains files written by previous invocations.
Developers write code assuming each invocation is independent. The code runs in an environment where state bleeds across invocations.
Global Variables Are Not Cleared
request_count = 0
def handler(event, context):
global request_count
request_count += 1
return {"count": request_count}This function increments a global counter. In a stateless model, the counter would always be 1. In serverless, the counter increments across invocations as long as the container is warm.
Invocation 1 returns {"count": 1}. Invocation 2 returns {"count": 2}. Invocation 3 returns {"count": 3}.
Then the container is recycled. Invocation 4 returns {"count": 1} again.
The counter is not a reliable counter. It resets unpredictably whenever the container is replaced. But it persists long enough to cause bugs.
If the function uses global state to track whether initialization has happened, duplicate initialization is skipped on warm invocations. If initialization fails on cold start but the variable is set anyway, the function believes it is initialized when it is not.
Global variables are scoped to the container, not the invocation. They accumulate state until the container dies.
File System Writes Persist
Serverless functions have access to /tmp storage. Writes to /tmp persist across invocations within the same container.
const fs = require('fs');
const path = '/tmp/data.json';
exports.handler = async (event) => {
if (fs.existsSync(path)) {
const data = JSON.parse(fs.readFileSync(path));
data.count += 1;
fs.writeFileSync(path, JSON.stringify(data));
} else {
fs.writeFileSync(path, JSON.stringify({ count: 1 }));
}
return { statusCode: 200 };
};This function writes to /tmp. On the first invocation, the file does not exist. On subsequent invocations, it does. The count increments.
The file persists until the container is recycled. The function cannot predict when recycling happens. The file may persist for seconds, minutes, or hours.
If the function expects /tmp to be clean, it is wrong. Previous invocations have littered it with files. If the function writes large files and does not clean up, /tmp fills up and the function fails with disk full errors.
Worse, /tmp has a size limit (512 MB in AWS Lambda). Accumulated writes across many invocations can exhaust the limit. The function stops working not because any single invocation wrote too much, but because many invocations wrote a little and the total accumulated.
The file system is shared state. Treating it as ephemeral creates resource leaks.
Connections Are Not Closed
import psycopg2
conn = psycopg2.connect(database="prod", user="app")
def handler(event, context):
cursor = conn.cursor()
cursor.execute("SELECT * FROM users WHERE id = %s", (event['id'],))
return cursor.fetchone()This function opens a database connection outside the handler. The connection is established once during container initialization and reused across invocations.
This is efficient. Opening a connection per invocation is slow. Reusing the connection amortizes the cost.
It is also stateful. The connection persists. If the connection is closed by the database (idle timeout, restart, network partition), the function does not reopen it. The next invocation tries to use a dead connection and fails.
If the function does not handle connection failures, every invocation fails until the container is recycled and a new connection is established.
Connection pooling libraries help but do not eliminate the problem. The pool persists across invocations. If the pool is exhausted by previous invocations and connections are not returned, the next invocation blocks waiting for an available connection.
Connections are shared state. They are not reset between invocations.
Module-Level Code Runs Once Per Container
const config = fetchConfigFromS3();
exports.handler = async (event) => {
console.log(config.apiKey);
};The fetchConfigFromS3() call runs during module initialization, not during handler execution. It runs once when the container is created, not on every invocation.
If the configuration changes in S3, the function does not see the change. The old configuration is cached in the module-level variable for the lifetime of the container.
This is a feature (fast lookups) and a bug (stale data). The function cannot control when the container is recycled and fresh configuration is loaded.
If the configuration contains credentials that are rotated, the function uses old credentials until the container is replaced. If credentials are rotated hourly and containers live for hours, the function eventually uses expired credentials and fails.
If module-level code has side effects (writes to a database, sends metrics, registers with a service), those side effects happen once per container, not once per invocation. The function has no visibility into how many containers are running or when they start.
Module-level initialization is shared state. It executes unpredictably.
Memory Leaks Accumulate
let cache = [];
exports.handler = async (event) => {
cache.push(event.data);
return cache.length;
};This function appends to a global array. Each invocation adds data. The array grows without bound.
Eventually, the array exhausts container memory. The function fails with out-of-memory errors. The failure is not triggered by any single invocation. It is the cumulative result of hundreds of invocations.
Memory leaks in long-running processes are bad. Memory leaks in serverless functions are worse because the developer assumes the process is short-lived and does not persist.
The process does persist. A container may handle thousands of invocations before being recycled. Small leaks compound into large failures.
If the function allocates memory proportional to input size, a sequence of large inputs can exhaust memory even if each input individually fits. The memory from previous invocations was never freed.
Memory is shared state. It accumulates across invocations.
Environment Variables Change Unexpectedly
Serverless platforms allow updating environment variables without redeploying code. Operators change a configuration value. The platform propagates it to running containers.
Or does it?
Some platforms propagate changes to new containers but not existing containers. Some platforms propagate changes immediately. Some platforms propagate changes on the next invocation.
A function reads an environment variable during initialization. The value is cached in a global variable. An operator updates the environment variable. Half the containers see the new value. Half see the old value.
The function behaves inconsistently across invocations. Some requests use the old configuration. Some use the new. Which requests see which configuration is non-deterministic.
If the configuration controls feature flags, some users see the new feature and some do not. If the configuration controls API keys, some requests use the old key and some use the new.
Environment variables are shared state. Changes propagate unpredictably.
Concurrent Invocations Share Nothing
A serverless function scales by running multiple containers. Each container handles one invocation at a time.
If 100 requests arrive simultaneously, the platform starts 100 containers. Each container is independent. They share no state.
If each container opens a database connection during initialization, 100 connections are opened. If the database connection limit is 50, half the containers fail to initialize.
The failure is not visible until scale-up. During development with one container, everything works. In production with 100 containers, initialization fails.
The statelessness model implies containers are independent and can scale infinitely. The reality is that containers compete for shared external resources: database connections, API rate limits, file handles, memory.
The lack of shared state between containers creates contention on external state.
Function Updates Do Not Stop Old Containers
You deploy a new version of a function. The platform provisions new containers with the updated code. Existing containers continue running.
Some requests route to old containers. Some route to new containers. Both versions are live simultaneously.
If the new version changes the schema of data written to a database, old containers write old schema and new containers write new schema. The database contains a mix.
If the new version changes the format of messages sent to a queue, old containers send old format and new containers send new format. Consumers must handle both.
The deployment is not atomic. There is a transition period where state from old and new versions coexists.
The statelessness model implies function updates are clean cutoffs. The reality is gradual migration with overlapping state.
Execution Limits Are Per-Invocation, Not Per-Container
A serverless function has a maximum execution time (15 minutes in AWS Lambda). This limit is per invocation, not per container.
A function runs for 14 minutes and completes. The container is still alive. The next invocation runs for 14 minutes. The container has now been running for 28 minutes.
Background processes started during initialization continue running between invocations. A function starts a metrics reporting thread during initialization. The thread runs continuously, even when the handler is idle.
If the thread leaks memory, crashes, or consumes CPU, it affects all subsequent invocations. The handler completes in 100ms, but the container’s CPU usage is high because the background thread is running.
Per-invocation limits do not constrain per-container resource usage. Long-lived containers accumulate background processes that are not terminated between invocations.
Logs Interleave Across Invocations
A container handles invocation A. Invocation A logs “Processing user 123”. Before invocation A completes, the same container handles invocation B. Invocation B logs “Processing user 456”.
The logs interleave:
Processing user 123
Processing user 456
Completed user 123
Completed user 456If the function is not thread-safe, concurrent invocations within the same container corrupt shared state.
Wait, serverless functions handle one invocation at a time per container, right?
Not always. Some platforms (Azure Functions, Google Cloud Functions with concurrency settings) allow concurrent invocations within the same container.
Even on platforms that enforce one invocation per container, asynchronous code can interleave. If the handler uses async/await and yields during I/O, the platform may start a new invocation in the same event loop.
Logs from multiple invocations interleave. State machines from multiple invocations interleave. The container is not as isolated as the stateless model implies.
Cold Starts Reset State Unpredictably
State persists across warm invocations. It resets on cold starts. Cold starts happen unpredictably.
A function caches API responses in a global variable. For warm invocations, the cache is hot. For cold invocations, the cache is empty.
If cold starts happen randomly, cache hit rate is unpredictable. Latency varies by 10x depending on whether the invocation is warm or cold.
If the function detects and reports cache hit rate, the metric is meaningless. It reflects container reuse patterns, not actual cache efficiency.
If the function uses the cache to deduplicate requests (tracking seen request IDs), duplicate detection fails across cold starts. A request processed before a cold start is not recognized as a duplicate after the cold start.
State persistence is conditional on container reuse. Container reuse is unpredictable. State persistence is unpredictable.
External State Accumulates Without Cleanup
A serverless function writes to a DynamoDB table. Each invocation writes one item. The function does not clean up old items.
Over time, the table grows. Query performance degrades. Costs increase. Eventually, the table hits storage limits or becomes too slow to query.
The function assumed it is stateless and does not need cleanup logic. The external state accumulated anyway.
The same happens with:
- SQS messages that are written but never consumed
- S3 objects that are created but never deleted
- CloudWatch log streams that grow without retention policies
- Metrics that are emitted but never aggregated
Serverless functions do not persist state internally, so they persist state externally. The external state accumulates without lifecycle management.
The statelessness of the function does not imply statelessness of the system.
Observability Assumes Statelessness
Monitoring tools track per-invocation metrics: duration, memory usage, error rate. These metrics assume invocations are independent.
They do not capture:
- State accumulation in global variables
- File system usage in
/tmp - Connection pool exhaustion
- Memory leaks across invocations
- Background processes started during initialization
A function reports 128 MB memory usage per invocation. The container actually uses 512 MB because memory leaked from previous invocations. The metric is misleading.
A function reports 100ms duration per invocation. The container has 10 background threads consuming CPU. The actual resource usage is higher than reported.
Observability assumes the stateless model. It does not surface the hidden state that actually exists.
Statelessness Is a Leaky Abstraction
Serverless functions are not stateless. They are:
- Stateful within a container lifetime
- Stateless across container lifetimes
- Unpredictably stateful depending on container reuse
The abstraction is that state does not persist. The reality is that state persists until the platform decides to recycle the container, which happens based on undocumented heuristics.
Developers cannot rely on state persisting. Developers cannot rely on state not persisting. The behavior is non-deterministic.
This creates bugs:
- Functions fail when warm because they assume cold start initialization
- Functions fail when cold because they assume warm state persistence
- Functions fail intermittently because container reuse is unpredictable
The stateless model is a lie. Serverless functions are stateful with unpredictable state lifecycle.
Understanding where state hides determines whether the function works reliably or fails unpredictably in production.