Shipping AI Features That Survive Production

Shipping AI features to production: what breaks after the demo

A demo runs one prompt, on one input the author chose, once, with the author watching. Production runs that prompt on inputs nobody anticipated, thousands of times a day, against a service you do not control, cannot roll back, and cannot debug from the inside. Nearly every failure in shipping AI features to production traces back to that difference. Almost none of them trace back to the model being insufficiently clever.

The work that closes the gap is unglamorous. It is retry policy, schema validation at the parse boundary, timeout budgets, per-tenant quotas, and deciding what the interface shows when the provider returns an overloaded error at 09:00 on a Monday. None of it improves the output that impressed everyone in the demo. All of it determines whether the feature is still switched on in three months.

What follows is the checklist we work through before an LLM-backed feature meets real users. It assumes a hosted model called over HTTP, which is the shape of most of these builds.

How do you test a feature whose output changes every run?

Conventional tests assert equality. Setting temperature to zero narrows the distribution but does not guarantee identical bytes: batching on the provider’s side, floating-point non-associativity across hardware, expert routing in mixture-of-experts models, and silent serving changes all move the output. A test suite that asserts on exact prose is a flaky-test factory, and a flaky suite is an ignored suite.

Assert properties instead of strings. The useful assertions are the ones that describe the contract between the model and the code that consumes it, and they are all deterministic even though the thing they inspect is not.

• Schema conformance: the response parses, every required field is present, every enum value is inside the allowed set.
• Groundedness: every entity, figure or document ID that appears in the output also appears in the source text you supplied.
• Bounds: the classification is one of your six labels, the extracted date falls in a plausible range, the generated SQL touches only tables the requesting user is permitted to read.
• Refusal behaviour: a known adversarial input still takes the refusal path rather than producing a compliant answer.
• Envelope: the call stays inside the token and latency budget the feature was sized around.

Those run in CI alongside everything else. Anything genuinely about quality — is this summary good, is this reply the right tone — is scored rather than passed, and belongs in an evaluation suite, not the unit test job.

What happens when the provider is down, slow, or rate-limiting you?

You have taken a hard dependency on an external service with published quotas and its own incident history. Rate limits are usually enforced on at least two axes simultaneously — requests per minute and tokens per minute — so a handful of long-context calls can exhaust the token budget while your request count still looks comfortable. HTTP 429 is not an exception in this design, it is expected control flow, and code that treats it as a crash will crash regularly.

Retry with exponential backoff and full jitter, honour any Retry-After header, and cap the total retry wall clock below the caller’s timeout so you never spend the user’s patience on attempts nobody will see. Equally important is refusing to retry what cannot succeed: a malformed request, an authentication failure, a content-filter refusal and a context-length overflow all fail identically the second time, and retrying the last is an expensive route to the same error.

Put your own token bucket in front of the provider rather than discovering their limit under load, and make it per-tenant. Without that, one customer’s bulk import consumes the whole minute’s quota and every interactive user in the product sees a spinner that never resolves — an availability incident caused entirely by fairness, not capacity.

Then decide the fallback ladder before launch, not during the incident. A smaller and faster model from the same provider, a second provider behind an adapter interface, a cached or precomputed answer, a deterministic non-AI path, and finally an honest error state that tells the user what to do next. You do not need all five. You need to have chosen, and the switch between them should be configuration that an on-call engineer can flip without a deploy.

Timeouts, streaming, and requests that never return

Generation latency scales with output tokens, so the distribution has a long tail rather than a tight cluster. A short classification returns quickly and predictably; a multi-page summary with a large context window can occupy tens of seconds. Default HTTP client timeouts are usually too aggressive for the second case, and many managed hosting platforms cap a synchronous request well below the time a long generation needs, which produces a gateway timeout in front of a model call that was going to succeed.

Streaming fixes the perception, not the duration. Server-sent events put the first token in front of the user quickly, but they change the failure model: you have already returned HTTP 200 and half an answer before anything goes wrong, so errors must travel in-band as a frame the client understands, and the client must handle a stream that simply stops. Streaming also fights structured output, since a JSON payload cannot be validated against a schema until the object is complete.

For anything genuinely long, stop pretending it is a request. Persist the job, return an identifier, run the generation on a worker, and deliver the result by poll or webhook. Attach an idempotency key so that a retried submission does not produce two charges, two emails or two rows, and write the request record before the provider call rather than after, so a crash mid-call leaves evidence rather than silence.

When user input reaches the prompt: injection, PII, and what you can actually do

Any text a user controls that ends up concatenated into a prompt is an instruction channel. The model has no reliable way to distinguish the instructions you wrote from instructions embedded in the data you supplied, and the list of things that count as user-controlled is longer than most teams assume: retrieved documents, email bodies, PDF contents, scraped web pages, file names, calendar invitations, and another user’s profile text all qualify.

There is no filter that reliably detects prompt injection, and any vendor claiming otherwise is selling a heuristic. The defence is architectural — constrain what the model is able to do rather than what it is able to read.

• Give tools the narrowest possible scope: a parameterised read against a specific view, not a general query interface with database credentials behind it.
• Authorise every tool call server-side against the end user’s own permissions at the moment of the call. A rule stated in the system prompt is a suggestion, not an access control.
• Require explicit human confirmation for anything irreversible — sending the email, issuing the refund, deleting the record, transferring the file.
• Escape model output before it reaches HTML. A model instructed by a poisoned document to emit an image tag pointing at an attacker’s host will happily emit it, and that request carries whatever is in the URL.
• Delimit untrusted content clearly and label it as data in the prompt, while accepting that delimiters reduce the success rate of an attack rather than eliminating it.

The privacy question gets decided earlier than teams expect. Everything in the prompt leaves your boundary, so enumerate the fields going out and redact what the task does not need — a support classifier rarely needs a customer’s full address to pick a queue. Confirm retention terms and training exclusions in the contract rather than the marketing page. Where health, payment or children’s data is involved, whether a data processing agreement and the residency of inference cover your obligations is a legal review, not a configuration flag.

Validating structured output before it reaches your domain logic

Constrained decoding, JSON schema modes and tool-call schemas have made malformed output rare rather than impossible. A generation that hits the max-token limit truncates mid-object, and a content filter can return prose where you expected an object. Syntactically perfect JSON can still be semantically wrong: an empty string where a name was required, an enum value that reads plausibly but is not one of yours, a date of 0000-00-00.

Treat the model boundary the way you would treat an untrusted HTTP body. Parse once, validate against a schema, and hand the rest of the system a typed value. A raw dictionary from a completion should never reach business logic, because the exception it eventually throws will surface three layers away from the cause.

const Extraction = z.object({
  invoiceNumber: z.string().min(1),
  currency: z.enum(["GBP", "USD", "EUR"]),
  totalMinor: z.number().int().nonnegative(),
  issuedOn: z.string().regex(/^\d{4}-\d{2}-\d{2}$/),
});

type Extraction = z.infer<typeof Extraction>;

async function extract(doc: string): Promise<Extraction | null> {
  const first = await callModel(doc);
  const parsed = Extraction.safeParse(tryJson(first.text));
  if (parsed.success) return parsed.data;

  // One repair attempt, with the validation error fed back in.
  const repaired = await callModel(doc, {
    priorAttempt: first.text,
    validationError: parsed.error.message,
  });
  const second = Extraction.safeParse(tryJson(repaired.text));
  if (second.success) return second.data;

  logParseFailure({ promptVersion: first.promptVersion, raw: repaired.text });
  return null; // caller routes to manual review, not to a crash
}

One repair attempt, not a loop. A model that fails the same schema twice will usually fail it five times, and the loop only multiplies latency and spend. The null return matters more than the retry: the caller needs a defined path for “no valid answer”, normally a queue for human handling rather than an error page.

Logging enough to debug without logging what you must not keep

When a customer reports a bad answer three days later, you need to reconstruct what happened. That means the prompt version, the rendered prompt or its inputs, the model identifier including the dated snapshot, the sampling parameters, the raw completion, token counts, latency, and the trace ID tying it to the originating request. Missing any one of those turns a fifteen-minute investigation into a shrug.

What must not land in that log is equally specific: API keys, whole uploaded documents, payment details, and personal data that has no business being retained. Redact at the logging boundary rather than trusting every call site, and keep prompt logs in their own store with their own short retention, because a prompt log is a copy of user content and inherits every deletion obligation attached to the original. An erasure request has to be able to reach it.

Cost belongs on the same dashboard as latency and error rate, because it fails in the same way — suddenly. Alert on spend per hour and per tenant rather than per month, since a monthly threshold notices a runaway retry loop several thousand pounds after the fact. Set an explicit max-token ceiling on every call, enforce a per-user quota server-side, and record tokens per request as a metric so a prompt change that quietly doubles context length shows up as a graph rather than an invoice.

Prompts are deployable artefacts, not configuration

A prompt determines the behaviour of a feature as completely as the code around it, so it deserves the same handling: in version control, reviewed, tied to a release, reversible. Editing a prompt in a vendor dashboard at midnight is production write access with no audit trail and no rollback.

Pin the model to a dated snapshot rather than a floating alias. Aliases move by design, and when one moves your carefully tuned prompt has been reassigned to a different model without a deploy, a changelog entry or a test run. Upgrading should be a deliberate act: run the new snapshot against your evaluation set, compare it with the incumbent, then change the pinned identifier in a reviewed pull request like any other dependency bump.

Stamp the prompt version onto every completion you log. It costs a string and it is what makes the difference between “the summaries got worse sometime last month” and “the summaries got worse at 14:20 when we shipped v11”.

A pre-flight checklist for shipping AI features to production

Every row below is a failure we have either fixed or watched someone else fix under pressure. The right time to work through it is the week before launch, when the mitigation is a day of engineering rather than an incident review.

None of this is exotic — it is ordinary reliability engineering applied to an unusually unpredictable dependency. The mistake is assuming the AI is the hard part; the model is the easiest component to replace and the least likely reason the feature fails.

If you have a prototype that works and you are now weighing what it takes to put it in front of paying customers, that transition is a large part of what we do at ApexStack. We are happy to walk through the checklist against your specific architecture and tell you which rows actually apply.