6 of 50 chapters written · about 24 minutes of reading so far · last revised 2026-08-04
Part 1 — The application layer — APIs and frontier models4/15
You buy tokens from someone else's endpoint. Your levers are what you send, what you ask for back, and how often. Most teams live here and never leave, and that is often correct.
What you are actually paying for4 min
A model bill is a token bill. Understanding the unit is most of the work, and almost nobody does it before they start optimizing.
The output premium4 min
Why an output token costs several times what an input token costs, and why that one fact should reshape your architecture.
Reading your first AI bill4 min
A provider invoice tells you what you spent, not where it went. Here is how to close that gap in an afternoon.
The cost levers, ranked4 min
Every AI cost lever there is, ordered by how much it typically moves the bill against how much it typically costs you to pull.
Tokenizer economicsplanned
Why non-English text and code cost more per unit of meaning.
Prompt caching, preciselyplanned
Prefix matching, TTLs, published discounts, and the ordering bug that silently voids it.
Context windows and what long context really costsplanned
Attention scaling, the price of stuffing, and when retrieval beats a bigger window.
Structured output and schema costplanned
You pay for field names on every call. Constrained decoding and its overhead.
Reasoning models — when thinking earns its priceplanned
Reasoning tokens bill as output. Deciding per call site, not per app.
Embeddings and vector searchplanned
Index build cost, dimensionality, re-embedding, and the storage tail.
Fine-tuning versus promptingplanned
The volume at which a shorter prompt pays for a model you now maintain.
Images, audio, and videoplanned
How non-text modalities are metered, and why they break token intuitions.
Agents — why loops multiply billsplanned
Turn count, context regrowth, tool-call overhead, and runaway guards.
Routing and cascadesplanned
Cheap-first-pass architectures, and building the eval before the router.
The cost of evaluationplanned
Evals are inference too. Budgeting the thing that keeps you honest.
Part 2 — The hosting layer — hyperscaler, neocloud, or your own1/9
You rent accelerators by the hour instead of buying tokens. Cost per token becomes a function of how busy you keep the hardware. Relevant once volume is steady and large enough to fill a GPU.
What an H100-hour actually costs5 min
The same GPU rents for $6.88 or $1.49 an hour depending on where you stand. The spread, the break-even maths, and what the cheap number is not telling you.
Hyperscaler versus neocloudplanned
What the cheaper provider is not selling you, priced honestly.
Spot, preemptible, and the economics of interruptionplanned
Checkpointing cost against discount, and which workloads qualify.
Reservations, savings plans, and commitment mathsplanned
Utilisation forecasting, and hardware generations that turn over faster than terms.
Serverless GPU versus dedicatedplanned
Cold starts, per-second billing, and the idle-tolerance crossover.
Managed inference endpointsplanned
Bedrock, Vertex, Together, Fireworks — the layer between API and metal.
Egress, storage, and the bill around the GPUplanned
The lines that are not the hourly rate and are not small.
Regions, residency, and sovereigntyplanned
What compliance geography does to the price comparison.
The total cost of self-hostingplanned
Engineering time, on-call, and the fixed-for-variable trade nobody prices.
Part 3 — The silicon layer — what the hardware actually costs0/8
Below the hourly rate is a physical machine with a fixed memory bandwidth and a fixed amount of VRAM. Those two numbers set the ceiling on everything above. Relevant when you are choosing accelerators or explaining why the cheap one was not cheaper.
Memory bandwidth is the productplanned
Why decode is bandwidth-bound and what that means for chip selection.
The accelerator ladderplanned
A6000 to B300 — a 14× price span, and how to find the rung you need.
VRAM sizingplanned
Weights, KV cache, activations. What fits, and the cost when it does not.
FLOPs, MFU, and the utilisation trapplanned
Why a busy GPU is usually a badly used one, and how to measure it.
Quantizationplanned
FP16 to FP8 to INT4 — throughput gained against quality surrendered.
TPU, Trainium, and AMDplanned
The non-NVIDIA options, their real price, and their real porting cost.
Interconnect and topologyplanned
NVLink domains, InfiniBand, and when the network becomes the bill.
Owning the hardwareplanned
Depreciation, power, colo, and the utilisation floor that justifies a purchase.
Part 4 — Scaling — serving economics under load0/9
The same GPU serves five requests a second or fifty depending entirely on how you schedule work onto it. This is where cost per token is actually decided, and where the largest wins live for anyone running their own inference.
Continuous batchingplanned
The single largest serving lever, and why static batching wastes most of a GPU.
KV cache and paged attentionplanned
The memory that limits concurrency, and how paging changes the arithmetic.
Prefill/decode disaggregationplanned
Running the two phases on different hardware, because they want different machines.
Speculative decodingplanned
Buying decode throughput with a draft model, and when the trade pays.
Tensor, pipeline, and expert parallelismplanned
Splitting a model across devices, and what each split costs in efficiency.
Autoscaling acceleratorsplanned
Cold starts, model load times, and the idle capacity you cannot avoid.
Tail latency versus costplanned
Queueing, load shaping, and pricing the SLO you actually promised.
Multi-tenancy and co-locationplanned
Packing several models onto one accelerator without a noisy-neighbour incident.
Benchmarking tokens per second per dollarplanned
Measuring serving throughput in a way that survives contact with production.
Part 5 — Operations — keeping it down in production1/9
Savings decay. This part is the practice that holds them: attribution, unit economics, guardrails, and the operating cadence. Applies at every layer above.
Making it stick3 min
Most AI savings decay within two quarters. The fix is not vigilance — it is putting cost where engineers already look.
Unit economics for AI featuresplanned
Choosing the denominator the business actually thinks in.
Attribution at scaleplanned
Tagging, trace joins, and cost per request across a real service graph.
Showback, chargeback, and ownershipplanned
Making cost land on the team that can change it.
Cost in CIplanned
Regression checks in the pull request, budgets, and gates that do not annoy.
Capacity planning and forecastingplanned
Projecting demand for a workload whose unit cost keeps moving.
Incident economicsplanned
Runaway loops, retry storms, blast radius, and hard ceilings.
Vendor management and negotiationplanned
What is negotiable, at what volume, and what to trade.
The operating cadenceplanned
What to review weekly, monthly, and quarterly — and what to ignore.
Why the unwritten chapters are listed.
Because the map is worth more than the territory covered so far. Publishing the whole outline says what this intends to be, lets you tell me which chapter you actually need next, and fixes each chapter’s address from the day it is planned — so a link written today still resolves when the chapter lands. Every infrastructure figure is priced on AWS, GCP, and Azure; a chapter citing one cloud is a draft, not a chapter. Tell me what is missing and it moves up the list.