Buffering Core Systems from Probabilistic LLMs

Large Language Models (LLMs) are exceptional at extracting information, generating content, and reasoning across unstructured data. What they are not is deterministic.

A deterministic application expects well-defined contracts, explicit failure modes, and predictable behavior. An LLM generates the next most probable token. Even when instructed to return JSON, its output remains the product of a probabilistic process rather than a guarantee of correctness.

This difference matters whenever AI is allowed to influence business operations. Whether you’re extracting invoice data, recommending refunds, classifying support requests, or generating workflow commands, allowing an LLM to write directly into a core business system introduces unnecessary risk. A syntactically valid response may still contain invalid identifiers, unsupported values, contradictory fields, or recommendations that violate business rules.

The goal, therefore, should never be to make the model “trusted.” Instead, the objective is to build an architectural boundary that treats every AI response as untrusted input until it has been proven safe.

This article presents a practical blueprint for doing exactly that using Azure API Management (APIM), Azure OpenAI, and Semantic Kernel.

Rather than allowing AI-generated content to flow directly into a deterministic application, we’ll introduce a validation buffer that progressively transforms an untrusted response into a trusted domain command.

The Architectural Mismatch

Traditional enterprise software is built around contracts.

An API defines expected request formats. Services validate inputs. Domain models enforce business rules. Databases maintain referential integrity. Every layer assumes that invalid data should be rejected before it reaches the next stage.

LLMs operate differently. They don’t execute business logic. They don’t understand your database. They don’t know whether a customer exists, whether an order has already shipped, or whether a payment exceeds the original transaction.

They generate statistically probable responses based on the context they’ve been given.

This distinction is easy to overlook because modern models are remarkably good at producing output that looks correct.

A JSON document may deserialize successfully. Every required property may exist. Every field may match the expected type. Yet the response can still be completely unsuitable for execution.

Consider an AI-powered refund assistant. An LLM might produce the following response:

{
  "customerId": "CUST-1843",
  "orderId": "ORD-9821",
  "refundAmount": 950,
  "currency": "USD",
  "reason": "Duplicate purchase",
  "confidence": 0.94
}

Nothing appears wrong. The JSON is valid. Every field matches the expected schema. The object deserializes without error.

Yet multiple questions remain unanswered.

• Does the customer exist?
• Does the order belong to that customer?
• Was the original payment only $320?
• Has the order already been refunded?
• Is the current user authorized to approve refunds?
• Does a refund above the approval threshold require human review?

None of these questions can be answered by the language model. Nor can they be answered by JSON Schema validation. They belong to the deterministic core of the application.

This is where many AI integrations become dangerous. Developers often confuse well-formed output with trusted output. They’re not the same thing.

Introducing the Validation Buffer

Instead of allowing AI responses to enter the core system directly, introduce a validation buffer between the probabilistic world of the model and the deterministic world of the application.

UNTRUSTED                                          TRUSTED

┌──────────────┐
│ Azure OpenAI │
└──────┬───────┘
       │
       │ Structured Response
       │
       ▼
┌──────────────────────┐
│ Azure API Management │
│ Schema Validation     │
│ Policy Enforcement     │
│ Authentication         │
└──────────┬───────────┘
           │
           ▼
┌──────────────────────┐
│ Semantic Kernel        │
│ Deserialize             │
│ Validate                 │
│ Retry / Repair           │
└──────────┬───────────┘
           │
           ▼
┌──────────────────────┐
│ Domain Validation       │
│ Business Rules            │
│ Authorization               │
└──────────┬───────────┘
           │
           ▼
   Trusted Domain Command

The important idea is that no single layer establishes trust. Instead, confidence is built progressively. Each layer removes another class of failure before the response moves closer to the business domain.

Only after every validation step succeeds should an AI response become a domain command.

Layer 1 – Constrain the Model

The first line of defense is reducing ambiguity.

Modern Azure OpenAI GPT-4.1 and GPT-4o deployments support Structured Outputs through supported API versions. Rather than asking the model to “return JSON,” you define the shape of the response you expect.

Structured outputs substantially reduce formatting failures compared to free-form generation.

However, structured outputs solve only one problem. They improve structural correctness. They do not verify semantic correctness.

A response can perfectly satisfy its schema while still recommending an invalid refund amount or referencing a customer that doesn’t exist.

Structured generation improves reliability. It does not eliminate validation.

Layer 2 – Validate at the API Boundary

Azure API Management provides an ideal location for enforcing transport-level contracts. At this stage the focus is on validating the payload itself rather than its business meaning.

Examples include:

• Required properties
• Unexpected fields
• Payload size
• Content type
• Basic data formats
• JSON Schema validation

Anything that violates the agreed contract should be rejected before it reaches application code.

This keeps malformed responses from propagating through the system while providing a consistent error contract for downstream services.

Importantly, APIM should remain responsible for policy enforcement not business decisions. It knows what a valid request looks like. It does not know whether issuing a refund is appropriate.

Layer 3 – Deserialize into a Transport Model

Once the payload has passed schema validation, Semantic Kernel can deserialize it into a strongly typed response model.

Notice that this model is not your domain model. It represents what the AI proposed. Not what the business has accepted.

public sealed record ProposedRefund(
    string CustomerId,
    string OrderId,
    decimal RefundAmount,
    string Currency,
    string Reason,
    double Confidence);

This distinction is subtle but important. If AI responses are deserialized directly into business entities, the boundary between trusted and untrusted data disappears. Transport models preserve that separation.

Layer 4 – Promote to a Domain Command

This is where deterministic software resumes control.

Business validation begins. Typical checks include:

• Customer existence
• Order ownership
• Duplicate transactions
• Refund limits
• Currency support
• Approval thresholds
• Authorization
• Fraud detection
• Workflow state

Only after every validation succeeds should the proposal become something the business can execute.

IssueRefundCommand

In command-oriented designs, that object represents the explicit business action to execute, not the model’s raw suggestion. Fowler’s Decorated Command discussion is a useful reference for wrapping that action with transactions, authorization, retries, and other execution concerns.

That promotion represents the moment trust is established. Everything before that point is simply evidence. Nothing generated by the model should be treated as an instruction.

Why Semantic Kernel Fits Naturally

Semantic Kernel is often viewed primarily as a prompt orchestration library. In practice, it also serves as an excellent boundary between AI services and deterministic application logic.

It provides a natural location for:

• requesting structured outputs
• deserializing responses
• validating proposals
• performing bounded retries
• logging failures
• invoking plugins
• collecting telemetry

Rather than scattering AI concerns throughout the application, Semantic Kernel centralizes them behind a predictable orchestration layer.

Ideally, the deterministic core consumes validated commands without depending on how they were generated. From its perspective, it simply receives either:

• a valid command,
• a rejected proposal, or
• a request requiring human review.

That separation significantly improves maintainability, testing, and long-term reliability.

End-to-End Implementation

The following implementation uses a refund proposal as the running example.

The flow is deliberately divided into two contracts:

1. The AI response contract describes what the model may propose.
2. The domain command contract describes what the core system is permitted to execute.

These types should remain separate, even when they contain similar properties.

Client
  |
  v
Azure API Management
  | Validate the HTTP request
  v
Refund Orchestration API
  |
  v
Semantic Kernel
  | Request schema-constrained output
  v
Azure OpenAI
  |
  v
ProposedRefund DTO
  | Deserialize and validate
  v
Domain validation
  |
  +--> Rejected
  |
  +--> Human review
  |
  +--> IssueRefundCommand
          |
          v
   Deterministic core system

Azure OpenAI structured outputs make the model follow a supplied JSON Schema. This is stronger than JSON mode, which guarantees valid JSON but not adherence to a particular schema. Semantic Kernel exposes structured outputs through the ResponseFormat execution setting.

Define the AI Response Contract

Start with a type that represents the model’s proposal. This is a transport type, not a domain command.

using System.ComponentModel;
using System.Text.Json.Serialization;

public sealed record ProposedRefund
{
    [JsonPropertyName("customerId")]
    [Description("The customer identifier supplied in the source material.")]
    public required string CustomerId { get; init; }

    [JsonPropertyName("orderId")]
    [Description("The order identifier supplied in the source material.")]
    public required string OrderId { get; init; }

    [JsonPropertyName("refundAmount")]
    [Description("The proposed refund amount as a positive decimal value.")]
    public required decimal RefundAmount { get; init; }

    [JsonPropertyName("currency")]
    [Description("The three-letter ISO 4217 currency code.")]
    public required string Currency { get; init; }

    [JsonPropertyName("reason")]
    [Description("A concise explanation for the proposed refund.")]
    public required string Reason { get; init; }

    [JsonPropertyName("confidence")]
    [Description("Model confidence between 0 and 1.")]
    public required double Confidence { get; init; }
}

The confidence field is a self-assessment generated by the model because it was requested in the response schema. It is not an internal confidence value exposed by Azure OpenAI and should be treated as advisory information rather than proof that the proposal is correct.

Avoid placing behavior or side effects on this type. Its only purpose is to capture untrusted model output in a known shape.

Define the JSON Schema

The equivalent JSON Schema can be versioned alongside the application:

{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "$id": "https://schemas.example.com/ai/proposed-refund-v1.json",
  "title": "ProposedRefund",
  "type": "object",
  "additionalProperties": false,
  "required": [
    "customerId",
    "orderId",
    "refundAmount",
    "currency",
    "reason",
    "confidence"
  ],
  "properties": {
    "customerId": {
      "type": "string",
      "minLength": 1,
      "maxLength": 50,
      "pattern": "^CUST-[A-Za-z0-9-]+$"
    },
    "orderId": {
      "type": "string",
      "minLength": 1,
      "maxLength": 50,
      "pattern": "^ORD-[A-Za-z0-9-]+$"
    },
    "refundAmount": {
      "type": "number",
      "exclusiveMinimum": 0
    },
    "currency": {
      "type": "string",
      "pattern": "^[A-Z]{3}$"
    },
    "reason": {
      "type": "string",
      "minLength": 1,
      "maxLength": 500
    },
    "confidence": {
      "type": "number",
      "minimum": 0,
      "maximum": 1
    }
  }
}

additionalProperties: false is important. Without it, the response may contain fields that the application did not request or anticipate.

The schema deliberately validates only structural properties. It cannot establish whether the customer exists, whether the order belongs to that customer, or whether the proposed refund exceeds the original payment.

Those checks remain the responsibility of application code.

Configure Semantic Kernel

Install the Semantic Kernel Azure OpenAI connector:

dotnet add package Microsoft.SemanticKernel
dotnet add package Microsoft.SemanticKernel.Connectors.AzureOpenAI

Register Semantic Kernel in the application:

using Microsoft.SemanticKernel;

var builder = WebApplication.CreateBuilder(args);

builder.Services.AddSingleton(sp =>
{
    var configuration = sp.GetRequiredService<IConfiguration>();

    var kernelBuilder = Kernel.CreateBuilder();

    kernelBuilder.AddAzureOpenAIChatCompletion(
        deploymentName: configuration["AzureOpenAI:DeploymentName"]
            ?? throw new InvalidOperationException(
                "AzureOpenAI:DeploymentName is missing."),
        endpoint: configuration["AzureOpenAI:Endpoint"]
            ?? throw new InvalidOperationException(
                "AzureOpenAI:Endpoint is missing."),
        apiKey: configuration["AzureOpenAI:ApiKey"]
            ?? throw new InvalidOperationException(
                "AzureOpenAI:ApiKey is missing."));

    return kernelBuilder.Build();
});

For production deployments, prefer managed identity over storing an API key in application configuration. The key-based setup is used here only to keep the example focused on the validation flow.

A corresponding development configuration might look like this:

{
  "AzureOpenAI": {
    "DeploymentName": "refund-extraction-model",
    "Endpoint": "https://your-resource.openai.azure.com/",
    "ApiKey": "use-user-secrets-or-a-secret-store"
  }
}

Do not commit the API key to source control.

Request a Structured Response

Create a service that asks the model to produce a ProposedRefund.

using System.Text.Json;
using Microsoft.SemanticKernel;
using Microsoft.SemanticKernel.ChatCompletion;
using Microsoft.SemanticKernel.Connectors.OpenAI;

public sealed class RefundProposalGenerator
{
    private static readonly JsonSerializerOptions JsonOptions =
        new(JsonSerializerDefaults.Web)
        {
            PropertyNameCaseInsensitive = false
        };

    private readonly Kernel _kernel;
    private readonly IChatCompletionService _chatCompletion;

    public RefundProposalGenerator(Kernel kernel)
    {
        _kernel = kernel;
        _chatCompletion =
            kernel.GetRequiredService<IChatCompletionService>();
    }

    public async Task<ProposedRefund> GenerateAsync(
        string sourceText,
        CancellationToken cancellationToken)
    {
        var history = new ChatHistory();

        history.AddSystemMessage(
            """
            You extract refund proposals from support material.
            Return only information supported by the supplied text.
            Do not invent customer IDs, order IDs, amounts, currencies,
            or reasons.
            The result is a proposal only. Do not claim that a refund
            has been approved or issued.
            """);

        history.AddUserMessage(
            $"""
            Extract a refund proposal from the following support material:

            <support_material>
            {sourceText}
            </support_material>
            """);

        var settings = new OpenAIPromptExecutionSettings
        {
            ResponseFormat = typeof(ProposedRefund),
            Temperature = 0
        };

        var response = await _chatCompletion.GetChatMessageContentAsync(
            history,
            executionSettings: settings,
            kernel: _kernel,
            cancellationToken: cancellationToken);

        var json = response.Content;

        if (string.IsNullOrWhiteSpace(json))
        {
            throw new InvalidOperationException(
                "The model returned an empty response.");
        }

        try
        {
            return JsonSerializer.Deserialize<ProposedRefund>(
                json,
                JsonOptions)
                ?? throw new InvalidOperationException(
                    "The model response deserialized to null.");
        }
        catch (JsonException exception)
        {
            throw new InvalidAiResponseException(
                "The model response could not be deserialized.",
                exception);
        }
    }
}

The important line is:

ResponseFormat = typeof(ProposedRefund)

Semantic Kernel uses the supplied .NET type as the structured-output response format. The model is instructed to return data matching that structure rather than arbitrary prose.

Register the service:

builder.Services.AddScoped<RefundProposalGenerator>();

Structured output reduces formatting failures, but deserialization should still be enclosed in an explicit failure path. Model incompatibility, configuration changes, interrupted responses, or SDK changes can still cause an unusable result.

A dedicated exception prevents AI integration failures from leaking into unrelated application behavior:

public sealed class InvalidAiResponseException : Exception
{
    public InvalidAiResponseException(
        string message,
        Exception? innerException = null)
        : base(message, innerException)
    {
    }
}

Apply Deterministic Domain Validation

The next step validates the proposal against authoritative application state.

First, define the trusted command:

public sealed record IssueRefundCommand(
    string CustomerId,
    string OrderId,
    decimal Amount,
    string Currency,
    string Reason,
    string IdempotencyKey);

Then define the possible validation outcomes:

public abstract record RefundProposalResult
{
    public sealed record Accepted(
        IssueRefundCommand Command)
        : RefundProposalResult;

    public sealed record Rejected(
        IReadOnlyList<string> Errors)
        : RefundProposalResult;

    public sealed record RequiresReview(
        ProposedRefund Proposal,
        string Reason)
        : RefundProposalResult;
}

The validator depends on deterministic repositories and authorization services rather than asking the model to assess its own output:

public interface ICustomerRepository
{
    Task<bool> ExistsAsync(
        string customerId,
        CancellationToken cancellationToken);
}

public sealed record OrderSnapshot(
    string OrderId,
    string CustomerId,
    decimal AmountPaid,
    string Currency,
    bool HasBeenRefunded);

public interface IOrderRepository
{
    Task<OrderSnapshot?> FindAsync(
        string orderId,
        CancellationToken cancellationToken);
}

public interface IRefundAuthorizationService
{
    Task<bool> CanIssueRefundAsync(
        string customerId,
        decimal amount,
        CancellationToken cancellationToken);
}

The validation service can now make a deterministic decision:

public sealed class RefundProposalValidator
{
    private const decimal AutomaticApprovalLimit = 500m;
    private const double MinimumAutomaticConfidence = 0.85;

    private readonly ICustomerRepository _customers;
    private readonly IOrderRepository _orders;
    private readonly IRefundAuthorizationService _authorization;

    public RefundProposalValidator(
        ICustomerRepository customers,
        IOrderRepository orders,
        IRefundAuthorizationService authorization)
    {
        _customers = customers;
        _orders = orders;
        _authorization = authorization;
    }

    public async Task<RefundProposalResult> ValidateAsync(
        ProposedRefund proposal,
        CancellationToken cancellationToken)
    {
        var errors = new List<string>();

        if (string.IsNullOrWhiteSpace(proposal.CustomerId))
        {
            errors.Add("Customer ID is required.");
        }

        if (string.IsNullOrWhiteSpace(proposal.OrderId))
        {
            errors.Add("Order ID is required.");
        }

        if (proposal.RefundAmount <= 0)
        {
            errors.Add("Refund amount must be greater than zero.");
        }

        if (proposal.Confidence is < 0 or > 1)
        {
            errors.Add("Confidence must be between 0 and 1.");
        }

        if (errors.Count > 0)
        {
            return new RefundProposalResult.Rejected(errors);
        }

        var customerExists = await _customers.ExistsAsync(
            proposal.CustomerId,
            cancellationToken);

        if (!customerExists)
        {
            errors.Add("The proposed customer does not exist.");
        }

        var order = await _orders.FindAsync(
            proposal.OrderId,
            cancellationToken);

        if (order is null)
        {
            errors.Add("The proposed order does not exist.");
        }
        else
        {
            if (!string.Equals(
                order.CustomerId,
                proposal.CustomerId,
                StringComparison.Ordinal))
            {
                errors.Add(
                    "The proposed order does not belong to the customer.");
            }

            if (order.HasBeenRefunded)
            {
                errors.Add("The order has already been refunded.");
            }

            if (!string.Equals(
                order.Currency,
                proposal.Currency,
                StringComparison.OrdinalIgnoreCase))
            {
                errors.Add(
                    "The proposed currency does not match the order.");
            }

            if (proposal.RefundAmount > order.AmountPaid)
            {
                errors.Add(
                    "The proposed refund exceeds the amount paid.");
            }
        }

        if (errors.Count > 0)
        {
            return new RefundProposalResult.Rejected(errors);
        }

        var isAuthorized =
            await _authorization.CanIssueRefundAsync(
                proposal.CustomerId,
                proposal.RefundAmount,
                cancellationToken);

        if (!isAuthorized)
        {
            return new RefundProposalResult.Rejected(
                ["The current workflow is not authorized to issue this refund."]);
        }

        if (proposal.RefundAmount > AutomaticApprovalLimit)
        {
            return new RefundProposalResult.RequiresReview(
                proposal,
                $"Refunds above ${AutomaticApprovalLimit:0.00} require human approval.");
        }

        if (proposal.Confidence < MinimumAutomaticConfidence)
        {
            return new RefundProposalResult.RequiresReview(
                proposal,
                "The proposal confidence is below the automatic approval threshold.");
        }

        var command = new IssueRefundCommand(
            CustomerId: proposal.CustomerId,
            OrderId: proposal.OrderId,
            Amount: proposal.RefundAmount,
            Currency: proposal.Currency.ToUpperInvariant(),
            Reason: proposal.Reason.Trim(),
            IdempotencyKey:
                $"refund:{proposal.OrderId}:{proposal.RefundAmount:0.00}");

        return new RefundProposalResult.Accepted(command);
    }
}

This service contains the actual trust boundary.

The proposal becomes an IssueRefundCommand only after structural checks, database checks, authorization checks, financial checks, and review thresholds have passed.

Expose a Stable Orchestration Endpoint

The API endpoint should return a finite set of outcomes rather than leaking model text or provider exceptions.

public sealed record CreateRefundProposalRequest(
    string SupportMaterial);

public sealed record RefundProposalResponse(
    string Status,
    string CorrelationId,
    ProposedRefund? Proposal = null,
    IReadOnlyList<string>? Errors = null,
    string? ReviewReason = null);

The endpoint connects generation and validation:

var app = builder.Build();

app.MapPost(
    "/v1/refund-proposals",
    async (
        CreateRefundProposalRequest request,
        HttpContext httpContext,
        RefundProposalGenerator generator,
        RefundProposalValidator validator,
        CancellationToken cancellationToken) =>
    {
        var correlationId =
            httpContext.TraceIdentifier;

        if (string.IsNullOrWhiteSpace(request.SupportMaterial))
        {
            return Results.BadRequest(
                new RefundProposalResponse(
                    Status: "rejected",
                    CorrelationId: correlationId,
                    Errors: ["Support material is required."]));
        }

        try
        {
            var proposal = await generator.GenerateAsync(
                request.SupportMaterial,
                cancellationToken);

            var result = await validator.ValidateAsync(
                proposal,
                cancellationToken);

            return result switch
            {
                RefundProposalResult.Accepted accepted =>
                    Results.Ok(
                        new
                        {
                            status = "accepted",
                            correlationId,
                            command = accepted.Command
                        }),

                RefundProposalResult.RequiresReview review =>
                    Results.Ok(
                        new RefundProposalResponse(
                            Status: "requires_review",
                            CorrelationId: correlationId,
                            Proposal: review.Proposal,
                            ReviewReason: review.Reason)),

                RefundProposalResult.Rejected rejected =>
                    Results.UnprocessableEntity(
                        new RefundProposalResponse(
                            Status: "rejected",
                            CorrelationId: correlationId,
                            Proposal: proposal,
                            Errors: rejected.Errors)),

                _ => Results.StatusCode(
                    StatusCodes.Status500InternalServerError)
            };
        }
        catch (InvalidAiResponseException)
        {
            return Results.Json(
                new RefundProposalResponse(
                    Status: "rejected",
                    CorrelationId: correlationId,
                    Errors: [
                        "The AI response did not satisfy the required contract."
                    ]),
                statusCode: StatusCodes.Status502BadGateway);
        }
    });

app.Run();

In a production system, the accepted command would normally be sent to a command handler or message broker rather than returned to the caller.

For example:

await commandBus.SendAsync(
    accepted.Command,
    cancellationToken);

The command handler should also enforce idempotency before producing a financial side effect. Validation reduces risk, but it does not replace transaction safety.

Register the Schema in Azure API Management

Azure API Management can validate JSON request and response bodies against reusable schemas registered in the APIM instance. The validate-content policy can reject invalid content, prevent undocumented properties, or record validation errors for later policy handling.

In APIM, add the schema under:

APIs → Schemas → Add

Give the schema a stable identifier, such as:

refund-proposal-response-v1

Schema identifiers should be versioned. Changing a schema in place can break existing clients or cause previously accepted responses to fail validation without a corresponding API-version change.

Validate the Incoming Request in APIM

The public request can be validated against the API’s OpenAPI definition or a separately registered schema.

The following policy rejects an invalid request before it reaches the orchestration service:

<policies>
    <inbound>
        <base />
        <set-header name="X-Correlation-ID" exists-action="skip">
            <value>@(Guid.NewGuid().ToString())</value>
        </set-header>
        <validate-content
            unspecified-content-type-action="prevent"
            max-size="32768"
            size-exceeded-action="prevent"
            errors-variable-name="requestValidationErrors">
            <content
                type="application/json"
                validate-as="json"
                action="prevent"
                allow-additional-properties="false" />
        </validate-content>
    </inbound>
    <backend>
        <base />
    </backend>
    <outbound>
        <base />
    </outbound>
    <on-error>
        <base />
        <return-response>
            <set-status code="400" reason="Invalid request" />
            <set-header name="Content-Type" exists-action="override">
                <value>application/problem+json</value>
            </set-header>
            <set-header name="X-Correlation-ID" exists-action="override">
                <value>@(
                    context.Request.Headers.GetValueOrDefault(
                        "X-Correlation-ID",
                        context.RequestId.ToString())
                )</value>
            </set-header>
            <set-body>@{
                var correlationId =
                    context.Request.Headers.GetValueOrDefault(
                        "X-Correlation-ID",
                        context.RequestId.ToString());

                return new JObject(
                    new JProperty(
                        "type",
                        "https://errors.example.com/invalid-request"),
                    new JProperty(
                        "title",
                        "The request did not satisfy the API contract."),
                    new JProperty("status", 400),
                    new JProperty("correlationId", correlationId)
                ).ToString();
            }</set-body>
        </return-response>
    </on-error>
</policies>

The public error response intentionally omits raw validation details. Detailed policy failures may expose implementation information and should instead be recorded in protected telemetry.

Validate the Outbound Response in APIM

APIM should also prevent an invalid orchestration response from reaching a caller.

The following outbound policy validates a successful JSON response against the registered schema:

<policies>
    <inbound>
        <base />
        <set-header name="X-Correlation-ID" exists-action="skip">
            <value>@(Guid.NewGuid().ToString())</value>
        </set-header>
    </inbound>
    <backend>
        <base />
    </backend>
    <outbound>
        <base />
        <choose>
            <when condition="@(
                context.Response.StatusCode >= 200 &&
                context.Response.StatusCode < 300
            )">
                <validate-content
                    unspecified-content-type-action="prevent"
                    max-size="65536"
                    size-exceeded-action="prevent"
                    errors-variable-name="responseValidationErrors">
                    <content
                        type="application/json"
                        validate-as="json"
                        action="prevent"
                        schema-id="refund-proposal-response-v1"
                        allow-additional-properties="false" />
                </validate-content>
            </when>
        </choose>
        <set-header name="X-Correlation-ID" exists-action="override">
            <value>@(
                context.Request.Headers.GetValueOrDefault(
                    "X-Correlation-ID",
                    context.RequestId.ToString())
            )</value>
        </set-header>
    </outbound>
    <on-error>
        <base />
        <return-response>
            <set-status code="502" reason="Invalid upstream response" />
            <set-header name="Content-Type" exists-action="override">
                <value>application/problem+json</value>
            </set-header>
            <set-header name="X-Correlation-ID" exists-action="override">
                <value>@(
                    context.Request.Headers.GetValueOrDefault(
                        "X-Correlation-ID",
                        context.RequestId.ToString())
                )</value>
            </set-header>
            <set-body>@{
                var correlationId =
                    context.Request.Headers.GetValueOrDefault(
                        "X-Correlation-ID",
                        context.RequestId.ToString());

                return new JObject(
                    new JProperty(
                        "type",
                        "https://errors.example.com/invalid-upstream-response"),
                    new JProperty(
                        "title",
                        "The upstream service returned an invalid response."),
                    new JProperty("status", 502),
                    new JProperty("correlationId", correlationId)
                ).ToString();
            }</set-body>
        </return-response>
    </on-error>
</policies>

A response-contract failure is returned as 502 Bad Gateway rather than 400 Bad Request because the caller did not cause the orchestration service to produce an invalid response.

In practice, the request and response policies would normally be combined into one API-level policy document.

Do Not Ask APIM to Perform Domain Validation

APIM can enforce:

• content types
• maximum payload sizes
• required properties
• JSON Schema constraints
• documented response shapes
• authentication and authorization policies
• quotas and rate limits
• stable error envelopes

It should not decide:

• whether an order belongs to a customer
• whether an account is active
• whether a refund exceeds the original payment
• whether an operation is a duplicate
• whether a human must approve the action

Those decisions require authoritative business state and belong in deterministic application services.

This distinction prevents business rules from being duplicated across gateway policies and application code.

Add a Bounded Retry Policy

Retries should be limited to failures that another model invocation could reasonably correct. For example:

• an empty response
• an interrupted response
• a transient provider error
• a response that cannot be deserialized

Do not regenerate merely because the proposal violates a business rule. If the model proposes a $950 refund for an order with a $320 payment, the application already has enough information to reject that proposal. Repeatedly asking the model for a different amount risks turning validation into negotiation.

A bounded wrapper could look like this:

public sealed class RetryingRefundProposalGenerator
{
    private const int MaximumAttempts = 2;

    private readonly RefundProposalGenerator _inner;
    private readonly ILogger<RetryingRefundProposalGenerator> _logger;

    public RetryingRefundProposalGenerator(
        RefundProposalGenerator inner,
        ILogger<RetryingRefundProposalGenerator> logger)
    {
        _inner = inner;
        _logger = logger;
    }

    public async Task<ProposedRefund> GenerateAsync(
        string sourceText,
        CancellationToken cancellationToken)
    {
        Exception? lastException = null;

        for (var attempt = 1;
             attempt <= MaximumAttempts;
             attempt++)
        {
            try
            {
                return await _inner.GenerateAsync(
                    sourceText,
                    cancellationToken);
            }
            catch (InvalidAiResponseException exception)
            {
                lastException = exception;

                _logger.LogWarning(
                    exception,
                    "AI response validation failed on attempt {Attempt}.",
                    attempt);
            }
        }

        throw new InvalidAiResponseException(
            $"The model failed to produce a valid response after " +
            $"{MaximumAttempts} attempts.",
            lastException);
    }
}

The retry count should be observable and small. Every attempt has cost, latency, and the possibility of producing a materially different answer.

Test the Complete Boundary

The most valuable tests are not tests that demonstrate the happy path. They are tests that confirm untrusted output cannot cross the promotion point.

At minimum, test these cases:

Scenario	Expected Outcome
Valid schema + valid business state	Accepted command
Valid schema + unknown customer	Rejected
Valid schema + order belongs to another customer	Rejected
Valid schema + amount exceeds amount paid	Rejected
Valid schema + duplicate refund	Rejected
Valid schema + amount above approval threshold	Human review
Valid schema + low confidence	Human review
Malformed or empty model response	Bounded retry, then 502
Undocumented API response property	Blocked by APIM
Repeated accepted request	Prevented by idempotency handling

A representative validator test might look like this:

[Fact]
public async Task Rejects_refund_that_exceeds_amount_paid()
{
    var customers = Substitute.For<ICustomerRepository>();
    var orders = Substitute.For<IOrderRepository>();
    var authorization =
        Substitute.For<IRefundAuthorizationService>();

    customers
        .ExistsAsync("CUST-1843", Arg.Any<CancellationToken>())
        .Returns(true);

    orders
        .FindAsync("ORD-9821", Arg.Any<CancellationToken>())
        .Returns(
            new OrderSnapshot(
                OrderId: "ORD-9821",
                CustomerId: "CUST-1843",
                AmountPaid: 320m,
                Currency: "USD",
                HasBeenRefunded: false));

    authorization
        .CanIssueRefundAsync(
            "CUST-1843",
            950m,
            Arg.Any<CancellationToken>())
        .Returns(true);

    var validator = new RefundProposalValidator(
        customers,
        orders,
        authorization);

    var proposal = new ProposedRefund
    {
        CustomerId = "CUST-1843",
        OrderId = "ORD-9821",
        RefundAmount = 950m,
        Currency = "USD",
        Reason = "Duplicate purchase",
        Confidence = 0.94
    };

    var result = await validator.ValidateAsync(
        proposal,
        CancellationToken.None);

    var rejected =
        Assert.IsType<RefundProposalResult.Rejected>(result);

    Assert.Contains(
        "The proposed refund exceeds the amount paid.",
        rejected.Errors);
}

This test captures the central point of the architecture.

The model can return flawless JSON with a high confidence score and still be wrong. The deterministic core not the language model, the prompt, or the JSON parser retains final authority.

Prompt injection is another reason the AI response should remain an untrusted proposal. Structured outputs prevent many formatting failures but do not stop a model from producing a structurally valid response influenced by malicious instructions contained within source material. Business validation remains responsible for determining whether the proposed action is permissible.

The Promotion Point

The complete flow can now be summarized as follows:

1. APIM authenticates the caller and validates the HTTP request.
2. Semantic Kernel sends the source material to Azure OpenAI with a structured-output response format.
3. Azure OpenAI returns a schema-constrained proposal.
4. The application deserializes the response into ProposedRefund.
5. Deterministic services verify customer, order, payment, authorization, workflow state, and approval thresholds.
6. Invalid proposals are rejected.
7. Ambiguous or high-impact proposals enter human review.
8. Only a fully validated proposal is promoted into IssueRefundCommand.
9. The core system executes the command using normal transaction and idempotency controls.
10. APIM validates the public response before returning it to the caller.

The key architectural boundary occurs between steps five and eight.

Before that boundary, the data is an AI-generated proposal. After that boundary, it is a trusted domain command. The similarity of their fields does not make them interchangeable.

Designing for Failure

One of the biggest architectural mistakes in AI systems is assuming the model succeeds most of the time.

Instead, begin by assuming that every response may fail for different reasons.

Some responses will violate the schema. Others will satisfy the schema while violating business rules. Occasionally, the model may produce conflicting or incomplete information.

Rather than treating these as exceptional situations, make them part of the design.

A robust integration typically supports four outcomes:

Outcome	Action
Valid response	Promote to a domain command
Structural failure	Reject immediately
Semantic failure	Reject or request regeneration
Ambiguous or high-risk response	Escalate for human review

By making failure explicit, the application remains predictable even when the model is not.

Conclusion

The safest AI architectures do not attempt to make probabilistic systems deterministic. Instead, they acknowledge the strengths and limitations of each.

Language models excel at interpreting ambiguity, extracting meaning, and generating candidate solutions. Deterministic software excels at enforcing rules, validating state, and protecting business integrity.

Azure API Management and Semantic Kernel provide complementary layers that help bridge these two worlds.

APIM enforces the external contract. Semantic Kernel orchestrates AI interactions and validates structured responses. The core application remains responsible for business rules and final authority.

The result is not a system that trusts AI. It is a system that benefits from AI while never depending on it blindly.

The most important architectural principle is therefore not “use structured outputs.” It is this:

An AI response should never become a business command simply because it deserialized successfully.

References

The architecture above builds on several existing platform capabilities and design concepts:

• Microsoft documents APIM body validation in the Azure API Management validate-content policy, including request and response validation against JSON schemas.
• Azure OpenAI documents structured outputs, where model responses are constrained to a supplied JSON Schema rather than merely being valid JSON.
• Semantic Kernel exposes structured-output configuration through OpenAIPromptExecutionSettings.ResponseFormat, which can be used to request JSON Schema-shaped responses from supported OpenAI-compatible models.
• Microsoft recommends managed identities for Azure resources to avoid embedding credentials in application code when Azure services call one another.
• Martin Fowler’s Domain Model pattern explains why business behavior belongs in the domain layer rather than in transport objects or generated payloads.
• Fowler’s discussion of Decorated Command is a useful companion for command-oriented designs where execution, transactions, retries, and authorization are wrapped around an explicit business action.
• OWASP’s Top 10 for Large Language Model Applications frames LLM-specific risks such as prompt injection, insecure output handling, excessive agency, and overreliance.
• OWASP’s LLM01:2025 Prompt Injection guidance reinforces the need to treat model inputs and outputs as untrusted, even when the output is well formed.
• The JSON Schema Draft 2020-12 specification defines the schema vocabulary used to express structural contracts for JSON data.