AI Inference is Rewriting the GPU Buying Playbook

Training lets GPU buyers pretend the problem is clean. Buy enough capacity to finish a large run, then let the benchmark judge the machine.

Inference breaks that simplicity.

A production endpoint must hold weights, manage KV cache, absorb uneven traffic, and control queues. The buying unit is the entire serving configuration: model, precision, framework, GPU, platform, traffic, reliability, and pricing.

A training leader can waste money in low-concurrency serving. An eight-GPU H100 node may be wrong for small bursts, while a cheaper card can fail under long contexts. Only the real workload gives an honest comparison.

Training benchmarks answer the wrong procurement question

Training rewards sustained utilization. Production inference moves between idle periods, bursts, uneven requests, queue spikes, and cache pressure.

Even “fast” becomes ambiguous. TTFT measures the initial delay. TPOT measures streaming pace. Throughput measures total work, while end-to-end latency captures the full experience.

MLPerf and vLLM’s production metrics separate these outcomes because they can conflict. Aggressive batching may increase throughput while making users wait longer. Buying the best isolated metric can leave capacity idle or break service levels.

A request is really two workloads

Prefill processes the prompt before generation. Long inputs make this heavier, which users experience as TTFT.

Decode generates tokens sequentially. A single tokens-per-second number hides whether the bottleneck sits in prefill, decode, scheduling, memory, or the network.

The engine schedules concurrent requests and manages memory. Sweep concurrency while recording TTFT, TPOT, end-to-end latency, queue depth, and throughput. This benchmarks the serving system, not just the model.

The serving configuration is the product

VRAM determines how much model state, context, and KV cache fit. Quantisation affects memory, speed, or quality. Frameworks control scheduling and cache behavior. Pricing determines whether the result is economical.

Start with the workload envelope: model, precision, sequence lengths, concurrency, latency, reliability, memory, framework, quantisation, and cost. Training leaderboards can shorten the shortlist, never finish it.

Latency and memory need envelopes

Define TTFT, p95 or p99 TPOT, end-to-end latency, and queue depth. Client-side timestamps keep routing and network delay visible.

Weight fit is only the first gate. KV cache, workspaces, CUDA graphs, and batching need more memory. Context and concurrency increase pressure. Sizing for maximum context wastes capacity; planning around weights alone invites queueing.

Prefill tends to be compute-intensive, while decode often moves model and cache data. Test the actual model, precision, context shape, and stack before paying for compute that does not improve the bottleneck.

Reliability is part of the configuration

Published uptime does not eliminate quotas, regional limits, slow replacement, or listing variation. Marketplace, reserved, and dedicated products offer different reliability.

Software, automation, observability, storage, replacement, network charges, and egress can erase an hourly-price advantage. Production designs must cover loading, warming, and replacement capacity.

Quantisation and frameworks change the hardware answer

Eight-bit quantisation may roughly halve weight memory versus 16-bit storage, enabling a cheaper GPU or more KV cache. It does not halve runtime, cache, workspace, or batching memory.

Framework and kernel support differs by ecosystem, so choosing between AMD and NVIDIA GPUs requires more than model compatibility. The vLLM, TensorRT-LLM, or TGI path must support the format and optimized execution.

Continuous batching can raise utilization while damaging latency. PagedAttention reduces fragmentation; its original paper reported 2 to 4x higher vLLM throughput than evaluated systems at comparable latency. That shows software leverage, not a universal multiplier.

Record the framework, quantisation, kernels, model, GPU, scheduler, cache use, preemptions, queue time, and quality. Otherwise, results are difficult to reproduce or transfer.

Hourly GPU prices can manufacture false equivalence

Hold the GPU class constant. Even an H100 shortlist remains imperfect because variants, node sizes, host resources, interconnects, and access differ. Here’s a quick comparison of several providers (with per GPU per hour pricing):

Google’s $11.68 per-GPU equivalent is arithmetic normalization, not a standalone SKU. Starting prices do not guarantee equal regions, capacity, deployment, or hosts. Fluence availability and reliability vary by provider.

L40S and H200 are alternatives, not equivalent rows. Fluence lists its L40S at $0.72 per hour for medium models and its 141 GB H200 at $2.96 for long-context or cache-heavy workloads. The CSV price unit requires confirmation.

Hourly price should only create the shortlist. Compare cost per million accepted output tokens at a stated latency, context, concurrency, and quality threshold, including storage, orchestration, idle time, and labor.

Different workloads need different platforms

Interactive endpoints should prioritize TTFT, TPOT, proximity, and predictable capacity. High-concurrency APIs need throughput within latency limits, backed by memory, batching, cache management, and stable capacity.

Long-context workloads should size for high-percentile context and concurrent KV-cache occupancy. Bursty inference may favor serverless; stable demand may favor reserved or dedicated capacity once utilization is measured.

The GPU buying playbook now starts with the workload

Production inference turns GPU procurement into a serving-system decision. Model fit, precision, context length, concurrency, framework support, latency, reliability, and total cost all influence which configuration performs best.

Teams should eliminate configurations that fail hard requirements, then benchmark the remaining options with identical workloads and deployment conditions. Measure TTFT, TPOT, tail latency, queue depth, throughput, utilization, errors, and cost per accepted output token.

The right platform is the one that consistently meets the application’s quality, performance, memory, availability, and cost requirements with manageable operational risk. That answer can only come from testing the complete serving configuration under realistic production traffic.