Real-time data processing for AI applications is marketed as millisecond-latency inference on streaming data. The vendor demo shows sub-100ms p99 latency on clean, consistent event streams. Production shows tail latencies in seconds, stale predictions from delayed data, and silent failures when event ordering breaks.
The failure mode is not that real-time systems are slow. It is that “real-time” as a guarantee does not survive contact with distributed systems, inconsistent data sources, network partitions, and the CAP theorem realities vendors do not discuss during sales calls.
Systems work when data arrives on schedule with consistent schemas and predictable volumes. They break when upstream systems stall, events arrive out of order, schema changes propagate mid-stream, and load spikes overwhelm processing capacity. The gap between demo performance and production behavior is where real-time promises collapse.
Real-Time Is Not a Latency Number, It Is a Consistency Model
Teams building real-time AI systems focus on latency. The model must respond within 100ms. The pipeline must process events within 500ms. Latency is measurable and vendors optimize for it in benchmarks.
Real-time systems fail on consistency, not latency. An event stream arrives at high velocity. Processing lags behind ingestion. The system maintains low latency by skipping events or processing data out of order. Latency stays low. Results are wrong.
A fraud detection model sees transaction events in real-time. To maintain latency guarantees, it processes each transaction independently without waiting for related events. A user makes two transactions milliseconds apart. The model sees them out of order due to network jitter or processing queue reordering. The first transaction is flagged as anomalous because context from the second transaction has not arrived yet. By the time ordering is restored, the alert is already sent.
Or the system processes events in strict order to maintain consistency. This requires coordination and buffering. Latency increases. The system maintains consistency guarantees but no longer meets latency requirements. You choose between consistent results and fast results. Marketing claims you get both.
Vendors selling real-time AI gloss over this trade-off. They show latency metrics from systems where consistency was relaxed. Or they show consistency from systems where latency was sacrificed. Both are real-time in different senses. Neither is what the customer expects when promised “real-time AI.”
Stream Processing Hides Failures Until They Cascade
Stream processing frameworks promise fault tolerance and exactly-once semantics. Events are processed reliably even when machines fail. State is checkpointed and recovered. The system handles failures transparently.
This works when failures are isolated and transient. It breaks when failures cascade or persist. An upstream data source slows down. The stream processor buffers incoming events to maintain throughput. Buffers fill. Backpressure propagates to upstream systems. The entire pipeline stalls.
The failure is silent from the perspective of monitoring latency metrics. Events are still being processed at normal latency when they reach the model. The queue is just growing. Metrics show p99 latency is fine. Reality is events are delayed by minutes in the buffer before they reach processing.
By the time the queue is noticed, it is too late. The model is making predictions on data that is minutes stale. Real-time fraud detection is useless if transactions are scored after the funds transferred. Real-time recommendation is pointless if the user already left the site.
Stream frameworks checkpoint state for recovery. Checkpointing introduces overhead. Under heavy load, checkpointing slows processing. Slower processing increases queue depth. Increased queue depth means more state to checkpoint. The system enters a failure mode where checkpointing overhead makes the queue depth problem worse.
Recovering from checkpoints after a crash restores state but does not restore processing capacity. The system recovers and immediately falls behind again because the load that caused the initial failure is still present. Recovery oscillates between running and crashed states.
Event Ordering Guarantees Do Not Survive Network Partitions
Stream processing systems offer ordering guarantees. Events from the same partition are processed in order. Causally related events maintain their ordering. The system preserves dependencies.
These guarantees depend on network reliability and clock synchronization that do not hold in distributed deployments. Events from different sources arrive through different network paths with different latencies. Timestamps are assigned by source systems with unsynchronized clocks. Ordering guarantees break.
An AI model predicts user behavior based on session events. Events must be processed in order to maintain session state. Events arrive from web servers distributed across regions. Network latency varies between regions. Events from one region consistently arrive before events from another region even when they were generated later.
The stream processor orders events by timestamp. Timestamps are assigned by web servers using system clocks. Clocks drift. One region’s clocks run fast. Events from that region are always ordered first even when they should be last. Session state is corrupted. Predictions are wrong.
Or the system uses event sequence numbers instead of timestamps. Sequence numbers are assigned at ingestion. An upstream system restarts and resets its sequence counter. Events appear to be from the beginning of time. The stream processor detects sequence number regression and drops events as duplicates. New data is ignored because the restart broke sequence numbering.
Fixing this requires logical clocks, vector clocks, or distributed consensus. Each solution adds latency and complexity. Real-time systems that actually maintain ordering under partition conditions are no longer real-time in terms of latency.
Feature Stores Cache Stale State
Real-time AI models depend on features computed from streaming data. Feature computation is expensive. Features are cached to reduce latency. The model reads features from cache instead of recomputing on every prediction.
Caching introduces staleness. The cache holds features computed from data that is seconds or minutes old. The model makes predictions using stale features. Predictions lag reality.
A recommendation model predicts which products a user will buy based on browsing behavior. Features capture recent browsing history. Features are cached with 60-second TTL to maintain low latency. The user views a product, adds it to cart, and checks out within 30 seconds. The model still sees cached features from before the product was added to cart. It recommends the product the user just purchased.
Reducing cache TTL reduces staleness but increases load on feature computation. More requests bypass cache and recompute features. Feature computation slows down. Latency increases. You trade staleness for latency.
Feature stores promise real-time features with low latency. The implementation is either cached features with staleness or computed features with latency. Real-time with low latency requires scaling feature computation to handle uncached load. That scaling is expensive and often not provisioned.
Schema Evolution Breaks In-Flight Pipelines
Data schemas change. New fields are added. Old fields are deprecated. Types change. Stream processing systems must handle schema evolution while maintaining uptime.
Schema changes propagate asynchronously through pipelines. Upstream producers deploy new schema. Downstream consumers still expect old schema. Events arrive with new fields that consumers do not recognize or are missing fields that consumers expect. Processing breaks.
Schema registries promise to solve this. Producers and consumers register schemas. The registry enforces compatibility. Evolution is controlled. This works when all components use the registry and deploy changes in coordinated order. In practice, components bypass the registry or deploy changes out of order.
An upstream service adds a new required field to events. The change is deployed to production. Events with the new schema arrive at the stream processor. The processor’s schema is out of date. It rejects events with the new field as malformed. The pipeline drops valid events.
Or the service makes the new field optional and provides a default. The stream processor accepts the events. The AI model downstream expects the new field to exist with meaningful values. It receives the default value for all events. Predictions degrade because the model is not getting the signal it expects.
Fixing this requires coordinated deployment across the entire pipeline. Deploy schema changes to all consumers before deploying to producers. This coordination is fragile and fails when teams deploy independently or when rollbacks happen out of order.
Backpressure Mechanisms Are Not Consistently Implemented
Stream processing systems implement backpressure to prevent fast producers from overwhelming slow consumers. When the consumer cannot keep up, backpressure signals the producer to slow down. This prevents queue overflow and memory exhaustion.
Backpressure works when producers and consumers are both designed to support it. It breaks when producers do not respect backpressure signals or when backpressure propagates across system boundaries that do not support it.
A Kafka producer sends events to a stream processor. The processor applies backpressure by slowing consumer reads. The producer does not see backpressure because Kafka buffers events. The producer continues writing at full speed. Kafka’s disk fills. Events are dropped. The producer and consumer both think the system is healthy.
Or backpressure propagates from the stream processor to the upstream application. The application is not designed to handle backpressure. It retries failed writes aggressively. Retries increase load on the already-overloaded stream processor. Backpressure makes the problem worse instead of better.
HTTP-based event ingestion implements backpressure by returning 429 or 503 status codes. Clients are supposed to back off and retry with exponential backoff. Some clients retry immediately. Some clients give up and drop events. Some clients buffer events locally and replay them later, causing delayed spikes. Backpressure behavior is inconsistent.
Window Boundaries Create Arbitrary Discontinuities
Stream processing aggregates data over time windows. Count events per minute. Calculate average latency per hour. Windows make infinite streams tractable by chunking data into finite pieces.
Window boundaries are arbitrary. Whether an event lands in one window or another depends on timestamps that are not guaranteed to be precise. Events near window boundaries behave unpredictably.
A monitoring system counts error events per minute. An error spike happens at 14:23:59. Half the errors land in the 14:23 window. Half land in the 14:24 window. Neither window shows a clear spike. The alert threshold is not crossed. The spike is invisible because it straddled a boundary.
Or the system uses sliding windows instead of tumbling windows to smooth over boundary effects. Sliding windows overlap. Events are counted multiple times. Aggregates are no longer independent. Correlations appear in the data due to window overlap, not underlying patterns. Models trained on sliding window features learn window artifacts instead of real signals.
Windows also assume events arrive in time-order. Events that arrive late are either dropped or retroactively added to closed windows. Dropping late events loses data. Updating closed windows means aggregates change retroactively. Downstream systems see aggregate values that change in the past. This breaks assumptions about immutability.
Monitoring Shows Averages, Production Hits Tail Latencies
Real-time systems are monitored with metrics like average latency, median latency, and p99 latency. Dashboards show these metrics are healthy. Production users experience latencies far worse than metrics indicate.
P99 latency is 50ms. That means 99% of requests complete within 50ms. It also means 1% of requests take longer. If you process a million requests per hour, 10,000 requests exceed p99. For those requests, latency could be 100ms, 1 second, or 30 seconds. P99 tells you nothing about the worst cases.
Models serving real-time predictions have a timeout. If the model does not respond within 200ms, the request fails and a fallback is used. Most requests complete quickly. 1% hit the timeout. The monitoring shows p99 of 190ms, which looks fine. Users experience 1% failure rate, which is terrible.
Averages hide bimodal distributions. Most requests complete in 20ms. Some requests complete in 5 seconds due to garbage collection pauses, network retries, or cache misses. The average is 50ms. The metric looks acceptable. Half the users experience unacceptable latency.
Monitoring aggregates metrics over time periods. 1-minute averages smooth over spikes. A 10-second latency spike is averaged with 50 seconds of normal performance. The 1-minute average looks fine. The 10-second spike broke user experience.
AI Inference Is Not Constant Time
Real-time AI systems assume model inference is fast and predictable. Models are optimized to run in milliseconds. Latency is tested with synthetic data. Production shows variance that breaks latency guarantees.
Inference time depends on input size and complexity. A language model processes short text quickly and long text slowly. Variable-length inputs create variable latency. Systems designed for constant-time processing assume fixed input size that does not reflect production traffic.
Models use caching and memoization to improve performance. Cache hit rates in test environments are high because test data is repetitive. Production data has more variety. Cache hit rate drops. Inference slows down. The system no longer meets latency targets.
Batch inference is faster than single-item inference due to amortized overhead and hardware parallelism. Real-time systems process one item at a time for low latency. They give up batch efficiency. Latency is good but throughput is poor. The system cannot handle production load.
Dynamic batching tries to get both latency and throughput by buffering requests and batching them when possible. This works when request rate is high and predictable. It breaks when request rate is bursty. During low traffic, requests wait in the buffer for batching. Latency increases. During high traffic, batches are too large to process within latency budgets. Latency increases.
Code That Looks Real-Time but Is Not
Engineers building real-time AI pipelines write code that looks like it processes events immediately. The code is actually buffering, caching, and batching in ways that introduce delay.
def process_event(event):
features = extract_features(event) # Reads from feature store (cached)
prediction = model.predict(features) # Batches internally
store_result(prediction) # Writes to database (buffered)
return predictionThis code appears to process events synchronously. In reality:
extract_features reads from a feature store with cached values up to 60 seconds old.
model.predict queues the request for batch inference every 100ms.
store_result writes to a buffer that flushes every second.
The function returns immediately with a prediction. That prediction is based on minute-old features, delayed by batching, and the result write is asynchronous. Nothing about this is real-time except the function call returns quickly.
Latency metrics measure the function’s return time, not end-to-end data freshness or result persistence latency. Monitoring shows low latency. Actual data flow has multi-second delays.
Engineers do not deliberately hide latency. The libraries and frameworks they use introduce buffering and caching as default behaviors to optimize throughput. Real-time becomes an API contract, not a performance guarantee.
When Real-Time Actually Matters vs. When It Is Marketing
Not all AI applications require real-time processing. Many benefit from batch processing with lower complexity and cost. Real-time becomes a requirement because it sounds advanced, not because the use case demands it.
Fraud detection is the canonical real-time use case. Fraudulent transactions must be blocked before they complete. This requires sub-second inference. Real-time is necessary.
Product recommendations are usually not real-time. Showing a user products based on their last 10 clicks does not require processing those clicks in real-time. Recommendations computed from 5-minute-old data work fine. Real-time adds complexity without user-visible benefit.
Anomaly detection in server logs is claimed as real-time but is often batch. Processing logs every minute is fast enough to catch issues before they cascade. Sub-second processing adds no value. The real-time requirement is not from operational need but from marketing positioning.
Customer churn prediction does not need real-time. Churn happens over days or weeks. Daily batch predictions are sufficient. Real-time churn scoring is solving a problem that does not exist.
The decision to build real-time pipelines should be grounded in operational requirements. What happens if data is 1 second old? 1 minute old? 1 hour old? If the answer is “nothing significant,” real-time is over-engineering.
Where Batch Processing Is Simpler and Works
Batch processing is deterministic, debuggable, and cheaper than real-time streaming. Results are reproducible. State is explicit. Failures can be retried without worrying about ordering, exactly-once semantics, or distributed state.
A recommendation model recomputes user embeddings daily based on the last 30 days of interaction data. This is batch processing. It works because recommendations do not need to reflect the user’s last click. Day-old data is sufficient.
The batch job runs on a schedule, processes all data, and writes results to a database. If the job fails, it is rerun. The job’s logic is simple because it does not handle streaming windows, backpressure, or late events.
Processing is cheaper because batch jobs use spot instances and scale horizontally without worrying about partitioning or state. Stream processing requires reserved capacity to handle peak load. Batch processing provisions for average load.
Results are auditable because the entire dataset and model version are versioned together. Given the same input data and model, the batch job produces the same output. Stream processing has non-deterministic ordering and timing that makes reproduction difficult.
Batch processing is not real-time. It is also not fragile, expensive, or non-deterministic. For use cases where latency is measured in minutes or hours, batch is strictly better.
What It Actually Takes to Build Real-Time AI That Works
Real-time AI that works in production requires accepting trade-offs vendors do not mention and operational complexity they do not support.
Accept eventual consistency. Real-time systems cannot guarantee strong consistency without sacrificing latency. Design for eventual consistency and build compensating logic for when consistency is violated.
Provision for tail latencies, not averages. If p99 latency is acceptable, peak load is 10x p99 load. Capacity planning based on average or median latencies leads to systems that fall over under load.
Implement dead-letter queues for events that cannot be processed. Do not assume all events will process successfully. Build mechanisms to inspect, replay, or discard failed events without blocking the pipeline.
Monitor data freshness, not just system latency. Measure how old data is when it reaches the model, not how fast the system processes events after they arrive. Staleness is the actual metric users experience.
Use circuit breakers and fallbacks for when the real-time system fails. If the model is unavailable, return a cached prediction, a default value, or a degraded response. Do not fail the entire request because real-time inference is down.
Version schemas and coordinate deployments. Schema evolution is not automatic. Plan schema changes with backward and forward compatibility. Deploy changes in coordinated order across producers and consumers.
Accept that some problems do not benefit from real-time. Build batch systems where they suffice. Reserve real-time complexity for use cases where it actually delivers value.
Real-time data processing for AI is feasible when requirements are clear, trade-offs are accepted, and operational complexity is resourced. Most teams building real-time systems underestimate complexity, ignore trade-offs, and lack operational capacity to run distributed stateful systems reliably.
The result is systems that are real-time in marketing materials and eventually consistent with unpredictable latency in production. Vendors sell the former. Engineers maintain the latter.