What It Costs to Run an LLM Feature in Production

Why input and output tokens are priced differently

Every major provider bills language model usage per token, and every one of them charges more for output than for input — typically several times more. This is not a pricing quirk; it reflects how a transformer actually runs, and understanding the mechanism tells you which lever to pull.

Processing input is the prefill phase: the whole sequence goes through the model in one pass and the GPU computes across all positions in parallel. It is compute-bound and efficient per token. Generating output is the decode phase, and it is fundamentally serial — each token needs its own forward pass through every layer, because it depends on the token before it. Decode is bound by memory bandwidth rather than raw compute; the hardware spends most of its time moving weights, not multiplying them. A thousand input tokens and a thousand output tokens are not the same amount of work, and the price list says so.

The practical consequence is that verbose output is the most expensive thing your feature can do, and a long prompt is the most persistent. A wordy answer costs you once per call at the higher rate. A bloated system prompt costs you on every call for the life of the feature.

One thing to check before modelling anything: token counts are model-specific. Tokenisers differ between vendors and change between generations within a vendor, so a model refresh can shift the count for identical text by a meaningful margin. Use the provider's own token counting endpoint against the exact model you intend to ship on.

Why context length, not request count, drives the bill

Teams estimate LLM API cost in production by counting requests, because that is how every other API they have bought was priced. It is the wrong unit. Two features with identical request volumes can differ in cost by orders of magnitude, entirely because of what sits in the context window.

The reason it compounds is that these APIs are stateless. The model has no memory of the last turn; your client re-sends the whole conversation every time. So a chat feature does not cost n × per-turn — it grows with the square of turn count. If your system prompt is S tokens and each turn adds roughly t tokens of user message plus response, the total input you are billed for across an n-turn conversation is:

total_input_tokens ≈ (n × S) + t × (n × (n + 1) / 2)

# S = 1,500 token system prompt, t = 400 tokens per turn
#  5 turns  →  7,500 +  6,000 =  13,500 input tokens
# 20 turns  → 30,000 + 84,000 = 114,000 input tokens
#
# 4× the turns, 8.4× the input cost.

Nothing in your per-request logs makes this visible. Each call looks reasonable; it is the twentieth in a session that costs fourteen times the first. The same arithmetic applies to agent loops, where each tool call is a fresh request carrying the whole history plus every tool schema, and to RAG, where retrieved chunks are re-sent on every question.

The multipliers nobody puts in the estimate

The naive estimate is tokens × rate × expected requests. It is almost always wrong, and wrong in one direction. Three categories of spend sit outside the happy path and rarely reach the spreadsheet.

Retries come first. A timeout, a schema validation failure, a rate limit, a refusal — each one re-bills the entire input. Retry logic usually lives inside a client SDK with a default of two or three attempts, configured by nobody and costed by nobody. If a schema mismatch pushes retries to a quarter of traffic, you have a serious problem that surfaces as a bill rather than as an error rate.

Evaluation runs come second, and are frequently the largest single line in a pre-launch bill. A five-hundred-case regression suite run on every merge against the highest-quality model is five hundred full-price requests per commit — charged to the production API key unless somebody deliberately separated them, which means your production cost dashboard is measuring your CI pipeline.

Third, the tokens you cannot see. Reasoning tokens are billed as output on every provider that offers them, and depending on the setting the response body may not show the text at all — you will only find them in the usage object. Guardrail classifiers, query rewriting, reranking and summarisation are all extra model calls that a user-facing ‘one request’ quietly fans out into.

Cost drivers, where they hide, and what to do about each

The levers, in the order worth pulling

These are ordered by return on effort. The first two typically account for most of the achievable reduction on a well-built feature.

1. Prompt caching

This is the single largest lever for anything with a stable prefix — a long system prompt, a fixed tool list, a shared few-shot block, a document being asked repeated questions. On Anthropic's Messages API, cached reads are priced at roughly a tenth of the base input rate, while writing to the cache costs 1.25× for the five-minute TTL or 2× for the one-hour TTL. That arithmetic gives you a hard break-even: with the short TTL you are ahead from the second request onward; with the long TTL you need at least three. Other providers apply their own discount and their own hit rules, so read the specific page rather than assuming.

Caching is a prefix match, and this is where implementations fail silently. Any byte that changes anywhere in the prefix invalidates everything after it. A timestamp in the system prompt header, a UUID, a JSON blob serialised without sorted keys, or a per-user tool list will each drop your hit rate to zero while the code looks perfectly correct. Order the prompt by stability — never-changing content first, per-request content last — then verify against the API's reported cache-read token count rather than trusting the design.

2. Model routing by task difficulty

Most features send every request to the strongest model because that is what the prototype did. Traffic is not uniform: classification, routing, extraction and formatting all run acceptably on the smallest tier, where published per-token rates are commonly several times lower than the flagship. Route on a cheap upfront signal — intent class, input length, whether tools are needed — and reserve the expensive model for what needs it. Measure quality per route, or a regression on a minority of traffic hides inside a healthy average.

3. Output caps, context truncation and batching

• Cap max_tokens per route rather than globally. It bounds the most expensive token class and turns a runaway generation into a handled error instead of an invoice.
• Truncate retrieved context after reranking. Recall and prompt length are separate decisions — fetch fifty candidates, rerank, pass five.
• Use the batch endpoint for anything not latency-sensitive. Both Anthropic and OpenAI price batch processing at half the standard rate with a 24-hour window; evaluation suites, backfills and nightly summarisation all qualify.
• Cache deterministic results in your own store. If the same input maps to the same output, a hash lookup costs nothing and the model call costs full price.
• Fire one request before a fan-out. A cache entry only becomes readable once the first response starts streaming, so N parallel calls sharing a prefix all pay full price.

How to build a cost ceiling per user before you launch

Do this before the feature ships; the alternative is discovering your unit economics from an invoice. The goal is not a precise forecast but a defensible worst case you can compare against what the feature earns.

# Per-call cost, from measured token counts and current published rates
call_cost =
    (cached_input_tokens × input_rate × cache_read_discount)
  + (fresh_input_tokens  × input_rate)
  + (output_tokens       × output_rate)

# Per-user monthly ceiling — use the 95th percentile, not the mean
user_ceiling =
    call_cost
  × calls_per_session_p95
  × sessions_per_month_p95
  × retry_multiplier          # e.g. 1.05
  × fanout_multiplier         # guardrails, rerank, agent turns

# Then compare against revenue per user, and set the hard limits:
#   - max_tokens per route
#   - max agent loop iterations
#   - monthly token quota per account, enforced in your own gateway

Three rules make this useful rather than decorative. Use percentiles, not averages — usage distributions have long tails, and the tail is where the loss sits. Populate the token counts from closed-beta traffic rather than your own test prompts, which are always shorter and better behaved than what users send. And leave the rates as variables you look up when you compute, because published pricing changes in both directions.

Then enforce the ceiling in code. A per-account token quota checked in your own gateway before the request goes out is the only thing that reliably stops a single pathological user, a runaway agent loop or a leaked key from producing a five-figure month. Provider-side spend limits are a backstop for the organisation; they cannot protect a per-user margin.

What to instrument on day one

Cost visibility has to be built with the feature, not retrofitted after the first bad month. Every response carries a usage object; logging it is a few lines of work that pays for itself immediately.

• Log input, output, cached-read and cache-write token counts on every call, tagged by route, model and tenant. On providers that separate these, the plain input count is only the uncached remainder — true prompt size is the sum of all three.
• Track cache hit rate as a first-class metric. A prefix change during a routine deploy can silently take it to zero, and the only symptom is a cost step-change a week later.
• Attribute cost per tenant and per feature, not per API key, so you can tell a growth story from a regression.
• Alert on cost per session rather than total spend. Total spend rising with user growth is the plan working; cost per session rising is a bug.
• Keep evaluation and development traffic on separate keys so your production figure means something.

The pattern across most of the LLM cost reviews we run is the same: the model choice was rarely the problem, and the prompt architecture usually was. If you are scoping a feature and want a second opinion on the numbers before they are committed to a roadmap, that is a conversation we are happy to have.