A checkout service depends on identity, pricing, inventory, and a payment provider. Each supplier publishes an availability commitment that looks compatible with the product’s target. Customers still encounter failures because checkout needs several of those systems to work together within a deadline.
Dependency reliability matters, but multiplying advertised percentages does not establish the service’s achievable availability. Supplier commitments can use different definitions, failures can share causes, and the application may tolerate some dependency failures while making others critical. The customer operation is the right starting point for the analysis.
A useful reliability model connects that operation to its required dependencies, states its assumptions, and is checked against production evidence. It helps explain what the team can improve and what a proposed service commitment actually requires.
Separate measurement, objectives, and agreements
A service level indicator measures a property of service behaviour. An objective sets a target for that indicator over an agreed window. A service level agreement establishes commitments and consequences under specified terms. Those concepts are related, but a contractual percentage is not automatically an observed probability of success for your application’s requests.
Google’s SRE guidance on implementing SLOs recommends indicators and objectives grounded in useful service behaviour. For checkout, that might mean eligible purchase attempts reaching a confirmed outcome within a stated duration. Define the population, success condition, and measurement point so teams know which failures count.
A supplier agreement may measure something different. Amazon S3’s SLA, for example, specifies service commitments, credit conditions, and exclusions. Read the applicable terms rather than treating a headline number as a technical prediction. A credit can address a contractual consequence without restoring the customer’s interrupted operation.
Avoid deriving a legal conclusion from an engineering estimate. A model showing reliability risk is a reason to examine the architecture and proposed terms together. Whether an actual event breaches an agreement depends on that agreement’s definitions and evidence, not solely on a calculation in a design document.
Model the dependencies required for one outcome
List the services and local components that must succeed for the chosen operation. Include the application’s own code, state, routing, and execution capacity. Distinguish dependencies needed for immediate success from work that can be deferred without misrepresenting the outcome to the customer.
Suppose a simplified request requires five dependencies, each with an actual success probability of 99.9% under the same relevant conditions. If their successes are independent, and all five must succeed, the model gives:
0.999 ** 5 = 0.995009990004999
Combined success probability: about 99.501%This is arithmetic for a stated model. It is not a universal maximum derived from five supplier SLAs. Calling the dependencies concurrently rather than sequentially does not remove the requirement that all succeed, although it can change latency and the way failures consume resources.
The independence assumption is consequential. Two services can share a region, identity provider, network route, or configuration change. Their failures may overlap. Identical failure periods would produce a different intersection of successful periods from independent failures with the same individual percentages.
| Situation | Useful question |
|---|---|
| All dependencies required | Which joint successes constitute the user outcome? |
| Some results optional | What degraded result is still genuinely useful? |
| Two providers offered as alternatives | Can either complete the operation independently? |
| Shared infrastructure | Which failure can disable several paths together? |
| Different supplier measurements | Are the percentages comparable to this operation? |
Model a few representative operations separately. Browsing a catalogue, submitting an order, and viewing a receipt may have different critical paths. One product-wide average can hide an unreliable operation behind a much larger volume of successful low-consequence traffic.
Include the deadline in the success condition
A dependency that eventually responds may still be unusable for a request with a shorter deadline. Measure the joint condition the customer experiences: the correct result arrived within the permitted time. Keep that definition distinct from a supplier’s broader availability measure.
Percentiles alone do not reveal every timeout rate. If the reported p95 latency is 200 milliseconds and p99 is 800 milliseconds, those points do not establish exactly what fraction exceeds a 500-millisecond timeout. Examine the distribution or count requests exceeding the chosen threshold.
Do not multiply an uptime figure by a latency percentage unless the definitions and relationship justify it. Timeouts may already be included in the measured error rate. Multiplication can double-count the same failures or introduce an unsupported independence assumption.
Allocate a request’s available time across connection setup, dependency work, possible recovery, and response handling. Pass deadlines where supported and examine what happens after a caller stops waiting. Timeouts do not necessarily cancel work, so a failed response path can leave a business operation in progress.
Degradation changes which dependencies are critical
A recommendation panel can sometimes be unavailable while a customer completes a purchase. A payment authorisation generally cannot be omitted while the interface reports payment confirmed. Degradation needs an explicit product meaning rather than an empty value substituted for every failure.
For notification delivery, a durable queue can separate confirmation from sending an email. That improves the immediate path only if the order is safely recorded and the queued work has suitable capacity, retry, and monitoring. The queue and its recovery process become part of the delayed outcome the organisation still owes the customer.
For a dashboard, show unavailable sections as unavailable instead of replacing them with data that implies a meaningful zero. Define which information is necessary to act safely. An interface that remains visible while presenting misleading state is not automatically a successful degraded service.
Mock servers can exercise these scenarios deliberately: missing recommendations, delayed notifications, or an uncertain payment result. Check the user-visible state and outgoing requests as well as the returned status code.
Recovery mechanisms affect the whole system
Retries can recover transient failures, but they also add load and consume time. If each of three nested layers makes three attempts, one originating request can generate up to 27 attempts at the deepest dependency. That is a worst-case attempt count under the stated policy, not a prediction for every request.
AWS’s guidance on timeouts, retries, backoff, and jitter explains how recovery traffic can aggravate overload. Choose retry ownership, limits, backoff, and an overall deadline. For state-changing operations, establish whether an earlier attempt may have succeeded and how duplicate effects are prevented.
Circuit breakers can reduce repeated calls to a failing dependency and protect shared resources. They can thereby improve the success of other operations and help recovery. They do not make the unavailable dependency itself healthy; the useful effect depends on isolation, fallback behaviour, and how the breaker decides to resume calls.
The same applies to capacity limits. Protecting a checkout path from optional background work may improve customer outcomes during stress. Measure those outcomes rather than assuming that adding a named resilience mechanism produces a fixed availability gain.
Test alternative paths before counting their benefit
Two providers with independent 99% success probabilities would have a 99.99% probability that at least one succeeds in a simplified model. That assumes either can complete the operation and that selection, routing, and the application work correctly. Shared causes and failover delays change the result.
Payment failover introduces an additional question: did the first provider accept the charge before its response disappeared? Sending the same purchase to another provider without resolving that uncertainty can create duplicate charges. A provider-independent operation record and reconciliation process may matter more than fast switching in that case.
Exercise failure detection, traffic switching, recovery, and data consistency under the conditions the design claims to tolerate. Contract checks can examine represented interactions, while integration testing is needed for selected real connections. Keep the distinction clear when reporting what a failover exercise established.
Compare the model with the error budget
An objective supplies a practical way to discuss how much failure the service can tolerate. In a request-based example with one million eligible requests, a 99.9% success objective permits 1,000 unsuccessful requests over that measurement window. If 600 have already failed, 400 remain within the budget, assuming the same population and success definition. This is a request count, not automatically a number of downtime minutes.
Google’s service level objectives chapter discusses the distinction between indicators, objectives, and agreements. Preserve those definitions when reviewing dependency incidents. A supplier’s outage duration and the customer’s failed-request count can describe the same event without being interchangeable measures.
Attribute failures to the customer outcome before adding dependency counts together. A single failed purchase might encounter an identity error and a later retry error; it should not become two failed purchases merely because two systems recorded it. Keep attempt-level diagnostics for investigation while measuring the operation at the level defined by the objective.
Use the model to identify assumptions worth measuring: correlated failures, deadline overruns, exhausted queues, or dependencies that unexpectedly block a degraded path. Revisit it after incidents and important architecture changes. The goal is a service commitment supported by an understood customer outcome, tested recovery behaviour, and measurements that expose where the remaining risk sits.
