When an LLM bill climbs faster than usage, the first instinct is to swap to a cheaper model. That is usually the wrong first move: quality regressions are easy to introduce and hard to detect. Below is the order worth running on a production system before considering a model swap, with an evaluation set in place so you can show quality held.
0. Measure before you cut
You cannot optimise what you cannot attribute. Log input, cached-input and output tokens for every request, tagged by feature, prompt version and model, and join them to a success signal. Two questions tell you where to start: which features account for most of the spend, and whether that spend is mostly input or output tokens. Freeze an evaluation set so every change below can be checked against it.
1. Turn on prompt caching properly
Many teams enable prompt caching but cache the wrong segment. Caching works on an exact prefix match, so put the largest stable content first — system prompt, tool definitions, reference documents — and the volatile parts (the user's question, timestamps, request IDs) last. A single timestamp near the top of a prompt can quietly stop the cache from ever hitting.
Pricing varies by provider. On Anthropic's API, for example, cache reads are billed at about a tenth of the normal input rate, while writing to the cache costs slightly more than a normal input token. For chatty workloads with long, stable prefixes, this is often the largest single saving. Check it is working: providers report cached tokens in each response's usage data, and a cached count that stays at zero across repeated requests means something in your prefix keeps changing.
2. Stop sending dead context
Audit your prompts. Production prompts accumulate tokens that no longer matter: debug examples left over from development, redundant instructions, retrieved chunks that didn't help. Add a "would we miss this if removed?" review to every prompt change. For retrieval, tune how many chunks you send; more is not always better, and irrelevant chunks can lower answer quality as well as raising cost. For long conversations, summarise or trim older turns instead of resending everything. Token diets are unglamorous and effective.
3. Right-size the output
Output tokens cost several times more than input tokens on most APIs; Anthropic, for instance, prices output at five times the input rate for its current models. Constrain output ruthlessly. If you need a yes/no, ask for "yes" or "no", not a paragraph that ends in yes. If you need JSON, specify the schema with minimal fields, and set a sensible maximum output length per feature. If you use a reasoning model, set its effort or thinking budget deliberately, because reasoning tokens are billed as output. Streaming summaries are nice for chat; they are a cost smell for batch.
4. Route by complexity
Not every request needs your strongest model. A thin classifier — sometimes a regex, sometimes a small LLM — routes simple cases to a cheaper model and reserves the strong model for the genuinely hard ones. Validate the routed-down traffic on your evaluation set separately; an average score can hide one category where the cheaper model fails badly.
5. Batch where latency allows
For non-interactive workloads such as overnight summarisation, document ingestion and evaluation runs, batch APIs from providers including Anthropic and OpenAI charge half the standard price in exchange for asynchronous processing that can take up to 24 hours. For everything that is not user-facing and real-time, batch.
6. Only now, consider the model swap
After the steps above, evaluate whether a cheaper or distilled model holds quality on your evaluation set. If it does, switch. If quality dips, you have still cut the bill before swapping. Teams that swap first often lock in a quality regression and still leave the easier savings on the table.
A worked example
Take a hypothetical support assistant handling 100,000 requests a month. Each request sends 6,000 input tokens, 4,000 of which are a stable system prompt and tool definitions, and receives a 500-token answer. Assume a model priced at $3 per million input tokens and $15 per million output tokens, with cache reads at a tenth of the input price.
| Line item | Before | After |
|---|---|---|
| Stable prefix (4,000 tokens per request) | $1,200 | $120, served from cache |
| Variable prompt (2,000 tokens per request) | $600 | $600 |
| Output (500 tokens, capped to 350) | $750 | $525 |
| Monthly total | $2,550 | about $1,245 |
That is roughly half the bill with the same model, ignoring the small cost of cache writes and assuming steady traffic keeps the cache warm. Each change is checked against the same evaluation set before it ships.
The metric that matters
Track cost per successful task, not cost per token or cost per call. A request that costs $0.002 but fails and is retried three times ($0.008 in total) is more expensive than a $0.005 request that succeeds first time. Build the success metric into your telemetry before you start optimising.
How we approach this at Velura Labs
Our Custom LLM Applications engagements include a cost-optimisation pass after the first month of production. Pair it with AI Strategy & Roadmap if you're sizing the budget envelope up front. Read our evaluation playbook for the quality side of these trade-offs and our buy-vs-build framework for the bigger procurement question. Talk to us if your LLM bill is outpacing your usage growth.
Velura Labs delivers this for teams across the United States — Seattle (Washington), San Francisco and Los Angeles (California), Austin and Dallas (Texas), and New York — as well as Europe (Paris, Milan, Rome and the wider EU), the Middle East (Dubai, Abu Dhabi and Riyadh) and India. Talk to us wherever you operate.