A fraud model returns a score in twenty milliseconds. The request is fast, but one feature was last refreshed several minutes ago because its update stream has a backlog. Another feature is current but missing a delayed transaction. The system needs to know whether the resulting decision is still appropriate.
Inference latency, data age, and completeness describe different properties. A fast response can use old inputs, a fresh event can arrive without related information, and a technically complete calculation can use the wrong feature definition. Real-time AI needs requirements for the decision as a whole.
Those requirements depend on the task. Monitoring a trend, selecting a recommendation, and controlling a physical process can tolerate different delays and mistakes. A percentile latency objective is also different from a hard real-time guarantee that every required deadline is met under defined conditions.
Give the Decision an End-to-End Budget
Define when the clock starts and when the result becomes useful. An incoming transaction may need a decision before authorisation proceeds. A maintenance alert may need action before a worsening condition crosses a limit. Model execution is one stage within those paths.
An illustrative 120-millisecond working budget might allocate 30 milliseconds to transport and queueing, 40 to feature retrieval and preparation, 20 to inference, and 30 to applying or returning the decision. These are assumed allocations, not a benchmark or a universal real-time target.
The allocations need to fit the actual requirement and leave suitable allowance for variability. Measure the complete path rather than adding component-level percentiles: a p99 duration from each independent dashboard does not automatically produce the end-to-end p99.
Define the response when the budget cannot be met. The system might use an approved simpler model, return a pending status, route for review, or decline to make an automatic decision. Which option is acceptable belongs to the business and safety requirements, not to a generic preference for always returning a score.
Separate Event Time From Arrival and Processing Time
Apache Flink’s event-time documentation distinguishes the time associated with the event from when processing occurs. A record can reach the processor long after the underlying activity, and records can arrive out of event-time order.
Record the timestamps needed for the decision’s interpretation: source event time where trustworthy, ingestion or availability time, processing time, and the time the result is used. Cross-system clocks and source timestamp quality need consideration before subtracting them as though they were exact measurements.
Data age at decision time can be expressed as the decision timestamp minus the relevant source timestamp. That is a useful measurement with a defined clock basis. It is not a complete freshness score for every input in a model containing several features.
The latest event timestamp is especially easy to misread. A quiet account may legitimately have no recent event, while a partition with one newly arrived event can still be missing older ones. Measure coverage and progress according to the source contract instead of assuming a recent maximum timestamp proves completeness.
Define how the system recognises a quiet source. A heartbeat or source progress marker can establish that ingestion is functioning even when no business events occurred. Its absence can indicate a different problem from an old but still valid feature value. The marker must have an understood meaning; a healthy network connection alone does not prove the producer has processed all expected work.
Use that distinction in alerts. Page an operator for a violated delivery or decision requirement, with the affected source and population identified. Otherwise harmless periods of inactivity can generate repeated freshness alarms while a genuine gap in an active source remains hidden by one recent record.
Treat Completeness as a Policy With Consequences
When related events can arrive late, waiting longer can improve the available view while delaying the decision. Acting earlier can meet a deadline with incomplete information. Define the permitted lateness and the correction behaviour for the particular operation.
Apache Beam’s treatment of watermarks and late data explains how stream processors manage event-time progress and incomplete arrivals. A watermark is a mechanism for reasoning about progress under the chosen source and runtime rules; it is not an absolute guarantee that no older event can ever appear.
Decide whether late data is discarded, updates an aggregate, produces a correction, or triggers review. Each has different consequences. Updating a recommendation is different from retracting a customer notification or repairing a financial decision already acted upon.
Preserve event identity and an appropriate version or ordering contract. Arrival order alone may not represent source order. The article on sequence-number meaning explains why ordering claims need a scope and why identifiers cannot supply every guarantee at once.
Establish What Each Feature Was Able to Know
A model can be fast while its online feature store is behind. Define acceptable age and completeness for each important feature, including how missing or stale values are represented. A default of zero should not silently combine genuinely zero activity with unavailable data.
Keep feature definitions and versions aligned between training and serving. An aggregate computed over one window during training and another online is a semantic mismatch, even if both values were produced moments ago. Freshness cannot repair the wrong calculation.
Feast’s point-in-time join documentation describes retrieving historical feature values relative to event timestamps. This helps avoid using values from the future when building training examples. Historical availability needs attention too: an event timestamp alone may not show whether a late-arriving record was actually available at the original decision time.
For evaluation and reconstruction, retain suitable feature snapshots or references, versions, and availability metadata. The required detail depends on the consequence and retention policy. A trace showing a successful feature read does not establish which business facts the value represented.
| Property | Example question | Evidence to retain |
|---|---|---|
| Response time | Did the result arrive before the deadline? | End-to-end timing and outcome |
| Input age | How old was the relevant observation? | Source time and clock assumptions |
| Completeness | Which expected events were represented? | Progress, coverage, and late-data policy |
| Feature semantics | Was the intended calculation used? | Feature definition and version |
| Reproducibility | What could the model know then? | Input reference and availability context |
Protect Capacity Before Backlogs Defeat the Deadline
Backpressure can prevent a downstream stage from being overwhelmed, but it does not remove excess demand. Work may wait upstream instead, making it older by the time it reaches the model. Monitor the waiting and the eventual decision, not only the protected component’s execution time.
Consider an illustrative pipeline receiving 1,200 events per second while it can process 1,000. Over five minutes, the backlog grows by 60,000 events. If arrivals later fall to 800 per second and processing stays at 1,000, draining that backlog takes another five minutes under these simplified assumptions.
Scaling must address the actual bottleneck. Adding workers may not help a hot key, a single partition, an external quota, or a state store under contention. Autoscaling also takes time, so capacity planning needs headroom and a policy for excess work before scale-out is effective.
Decide which events must all be processed and which state updates can be coalesced. Dropping intermediate pointer positions can suit an interface; dropping individual transactions can corrupt a financial aggregate. The event’s meaning determines whether load shedding is acceptable.
Make Recovery Part of the Data Contract
A restart can replay events that already contributed to a feature or triggered an action. Use stable identities and supported processing guarantees to prevent inappropriate duplication. Checkpoints help recover progress within the runtime’s documented scope.
Kafka’s delivery-semantics documentation distinguishes guarantees and explains transactional processing within Kafka. Exactly-once processing within a supported boundary does not automatically make an external payment, email, or database write occur exactly once. Those destinations need appropriate cooperation and effect handling.
Recovery can also change which model or feature version processes an old event. Decide whether replay reconstructs the original result, recomputes under current policy, or performs a repair. These are different operations and should not share an ambiguous command called retry.
Control replay speed so recovering work does not starve current decisions. Monitor backlog age and recovery progress separately. A service that resumes producing low-latency responses may still have unresolved historical gaps affecting its features.
Observe the Decision Population, Not Only Successful Requests
Record deadline misses, stale-feature fallbacks, unavailable inputs, corrections, and abandoned or rejected work. A latency chart containing only successful inference calls can improve while more customer decisions fail before reaching the model.
Use distributed tracing to investigate selected paths, with aggregate measures for the relevant population and durable records for consequential outcomes. Align their definitions so one fast stage cannot stand in for the complete workflow.
Test skewed partitions, delayed sources, stale caches, schema changes, and replay alongside ordinary load. The useful target is a decision process that meets its defined timing and data requirements, including an acceptable response when they cannot be met. Some applications can satisfy that target with periodic processing; others need streaming. The requirement should determine the architecture.
