The first time someone shows me an AI bill, it is usually a single number with a model name next to it. That invoice is accurate and nearly useless: it tells you what left the bank account, not which part of your product spent it or whether the spending was reasonable.
Closing that gap is the first real piece of work. It is not hard, it does not need a vendor, and until it is done every optimization is guesswork.
Start with the shape, not the total
Before touching anything, get four numbers for the last full month:
- Total tokens in, total tokens out. Not dollars. Tokens.
- The input:output ratio. This tells you whether you have a prefill problem or a decode problem, which determines everything you do next. See the output premium.
- Call count. Combined with the token totals, this gives average tokens per call — and outliers in that average are where the money hides.
- Spend by model. If you use more than one, the split is frequently not what the team assumes.
Most providers expose enough usage data to get you here without any instrumentation at all. Start there. It costs an hour and it is often enough to make the biggest problem obvious.
Then attribute it
Provider dashboards stop at the API key. They cannot tell you that the summarization step in your onboarding flow is two-thirds of the bill, because they do not know your product exists. You have to add that yourself.
The minimum useful instrumentation, logged on every model call:
- input tokens, output tokens, model, timestamp
- a feature tag — which product surface this call belongs to
- a step tag — which stage within that feature
- the request or trace ID, so a call joins to the rest of your telemetry
That is a handful of fields. Every major SDK returns the token counts in the response; you are writing them down, not computing them. Once this exists you can answer "what does one document review cost us" instead of "what did we spend in July," and that shift in denominator is the whole point.
If you only do one thing from this chapter, do this one.
What to look for
With a week of tagged data, a small number of findings recur across almost every unoptimized account:
A prompt that grew. System prompts accumulate. Every incident adds an instruction, nothing is ever removed, and two years later you are shipping several thousand tokens of accumulated caution on every call. Print your longest system prompt and read it end to end. People are routinely surprised.
Retries nobody counted. A retry on timeout is a second full charge. If the timeout is set below the p99 latency for long generations, you are paying twice for a meaningful fraction of traffic and the success metrics look fine.
Retrieval that over-fetches. Pulling the top 20 chunks when the answer reliably sits in the top 3 is a 6x input bill on that step. Measure how far down the retrieved context the answer actually comes from.
Development traffic on the production key. Test suites, local development, and demos billing against the same key as customers. Not necessarily large, but it distorts every per-unit number you compute, which makes everything else harder to read.
A model chosen once and never revisited. The model picked at the prototype stage frequently stays because nothing forced the question. Prices and capabilities both move; the price index is one way to check whether the original reasoning still holds.
Output nobody reads. Generated text that gets truncated in the UI, summaries produced for a screen users skip, verbose fields discarded at parse time. You paid decode rates for all of it.
What is realistic
On an account that has never been examined, identifying somewhere in the high teens to around 30 percent is a fair expectation, and most of it usually comes from two or three findings rather than a long list of small ones. Accounts that have already been through a serious pass yield less, which is exactly as it should be.
Be skeptical of anyone promising more before they have seen your data — including me. The number is a function of how much slack is in the current architecture, and nobody can know that from the outside.
Also worth separating in your own head: identified savings and realized savings are different things. A finding becomes money when someone ships the change, and some findings are not worth the engineering time they would take. That judgment is part of the work, not a footnote to it.
The order that works
- Get the shape. Tokens, ratio, call count, model split.
- Tag calls by feature and step.
- Find the top three call sites by token volume.
- Fix the largest one. Measure it. Confirm quality held.
- Repeat.
Nothing here requires a platform, a contract, or a rebuild. It requires deciding that the bill is a thing you look at on purpose rather than a thing that arrives.