The economics of LLM inference

In conventional web architectures, doubling traffic increases cloud compute costs roughly linearly. In LLM inference serving, unit costs are dominated by an unforgiving trade-off between concurrency batching and Time To First Token (TTFT).

An 8x H100 GPU node costs roughly $24 per hour on reservation contracts. If your cluster operates at batch size 1 to deliver 15ms per-token streaming latency to interactive human users, your effective cost per million tokens can be 30 times higher than an offline batch pipeline running at batch size 128.

The arithmetic of continuous batching

Static batching wastes massive GPU cycles because generation lengths vary across requests: the entire batch must wait for the longest sequence to finish generating.

Continuous (iteration-level) batching, popularised by systems like vLLM and TensorRT-LLM, schedules new incoming requests at every forward pass token iteration.

# Cost model calculation: Tokens per dollar
def calculate_cost_per_million_tokens(
    gpu_hourly_rate: float, # e.g. $2.50 per H100
    gpus_per_replica: int,  # e.g. 4 for 70B FP16
    throughput_tok_sec: float
) -> float:
    total_hourly_cost = gpu_hourly_rate * gpus_per_replica
    tokens_per_hour = throughput_tok_sec * 3600.0
    cost_per_token = total_hourly_cost / tokens_per_hour
    return cost_per_token * 1_000_000

# Single user interactive (batch=1): ~45 tok/s -> ~$61.70 / MTok
# High concurrency serving (batch=64): ~1800 tok/s -> ~$1.54 / MTok

Hardware utilisation is a latency game. High throughput requires deep batches; low latency requires empty queues.

Speculative decoding: trading FLOPs for memory bandwidth

Because memory bandwidth is the primary bottleneck during autoregressive decoding, modern inference engines employ speculative decoding. A small, fast draft model (e.g., a 1B parameter model) generates \(K\) candidate tokens in rapid sequence. The large target model (e.g., 70B) then verifies all \(K\) tokens in a single parallel forward pass.

• Acceptance rate: On natural code or prose, draft acceptance rates typically hover between 65% and 80%.
• Latency speedup: Delivers a 2.2× to 2.8× speedup in wall-clock latency with mathematical parity to the base model's sampling distribution.

Prefix caching and multi-tenant architectures

For agentic workflows and long system prompts, re-computing the KV cache for identical prefixes on every user turn wastes millions of FLOPs. Paged prefix caching retains the KV blocks of common system headers in VRAM, turning expensive quadratic prefill phases into instantaneous O(1) hash lookups.

Optimising inference costs or designing high-throughput serving clusters? Get in touch.