Chapter 1110 min readReviewed 31 August 2026

Formulas, glossary, and source ledger

The reusable math, vocabulary, caveats, and dated primary sources behind this playbook.

On this page

Formula sheet
Glossary
Source ledger
Maintenance policy

Reusable math

Formula sheet

Use these for first-pass planning, then replace estimates with engine and workload measurements.

weight bytes ≈ parameters × bits per weight ÷ 8

Weight-only storage; scales/metadata and higher-precision layers add overhead.

KV bytes = 2 × layers × KV heads × head dimension × bytes × Σ live tokens

Standard attention; use KV heads, not query heads.

E2E ≈ TTFT + (output tokens − 1) × TPOT

Simple response approximation.

mean in-flight requests ≈ request rate × mean service time

Little’s Law for mean concurrency; tails are higher under bursts.

replicas ≈ ceil(target demand ÷ goodput/replica × (1 + reserve fraction))

Capacity that passes the service contract; reserve fraction is a decimal (20% = 0.20).

$ / 1M output tokens = deployment $/h × 1,000,000 ÷ successful output tokens/h

All loaded cost; only SLO-passing output in the denominator.

break-even utilization = committed effective rate ÷ on-demand rate

First-pass commitment test.

scaling efficiency = throughput at N ÷ (N × throughput at 1)

Same workload and quality target.

Shared vocabulary

Glossary

Accelerator: A device optimized for parallel compute; GPUs are one accelerator family.
HBM: High Bandwidth Memory packaged close to the accelerator. Capacity and bandwidth are distinct.
GDDR: Graphics memory used by many PCIe accelerators; generally a different capacity/bandwidth/power profile from HBM.
TTFT: Time from request submission to the first output token, with the measurement boundary stated.
ITL / TPOT: Actual intervals between tokens / average time per generated token after the first.
Goodput: Useful work completed while meeting the chosen latency, quality, error, or deadline contract.
KV cache: Attention keys and values retained for live tokens so previous context is not recomputed at each decode step.
Continuous batching: Rebuilding the active batch at generation steps so finished work leaves and new work enters.
Tensor parallelism: Sharding operations inside model layers across devices, requiring frequent collectives.
Data parallelism: Replicating a model or training state so different requests/batches run on different workers.
MIG: NVIDIA hardware partitioning on supported GPUs with defined compute/memory slices and isolation.
Time-slicing: Scheduling multiple processes on one GPU without hardware memory/fault isolation.
SXM / OAM: High-power accelerator module form factors used on platform baseboards; not ordinary PCIe cards.
NVLink / Infinity Fabric: Vendor scale-up links used for faster device-to-device communication within supported systems.
RoCE: RDMA over Converged Ethernet, used for low-latency/high-throughput network communication.
Quantization: Representing some weights, activations, or KV state at lower precision with hardware, kernel, and quality implications.
Prefix cache: Reuse of KV blocks for token-identical prefixes across requests.
SLO: A measurable service-level objective such as p99 TTFT, error rate, quality, or job deadline.

Evidence model

Primary-source ledger

All sources below were checked 31 August 2026. Live documentation is marked because compatibility, specifications, and commercial terms can change.

Vendor specifications

NVIDIANVIDIA L4 Tensor Core GPURe-check before use
NVIDIANVIDIA L40S GPURe-check before use
NVIDIANVIDIA A100 data sheetRe-check before use
NVIDIANVIDIA H100 Tensor Core GPURe-check before use
NVIDIANVIDIA H200 Tensor Core GPURe-check before use
NVIDIANVIDIA DGX B200 specificationsRe-check before use
NVIDIANVIDIA GB200 NVL72Re-check before use
AMDAMD Instinct MI300XRe-check before use
AMDAMD Instinct MI325XRe-check before use
AMDAMD Instinct MI350XRe-check before use
AMDAMD Instinct MI355XRe-check before use
IntelGaudi architecture overviewRe-check before use

Official documentation

NVIDIAMIG user guideRe-check before use
NVIDIAGPU sharing with the GPU OperatorRe-check before use
NVIDIANVIDIA GPU OperatorRe-check before use
NVIDIADCGM Exporter metricsRe-check before use
AMDMI300X GPU partitioningRe-check before use
AMDROCm compatibility matrixRe-check before use
IntelGaudi platform support matrixRe-check before use
vLLMvLLM documentationRe-check before use
vLLMvLLM metrics designRe-check before use
vLLMvLLM optimization and tuningRe-check before use
vLLMvLLM parallelism and scalingRe-check before use
vLLMvLLM quantization supportRe-check before use
vLLMAutomatic prefix cachingRe-check before use
SGLangSGLang documentationRe-check before use
NVIDIATensorRT-LLM overviewRe-check before use
NVIDIATensorRT-LLM benchmark methodologyRe-check before use
RayRay Serve LLMRe-check before use
RayRay Serve LLM observabilityRe-check before use
KubernetesSchedule GPUs in KubernetesRe-check before use
Kubernetes SIG SchedulingKueue overviewRe-check before use
Kubernetes SIG SchedulingKueue topology-aware schedulingRe-check before use
VolcanoVolcano scheduler overviewRe-check before use
Hugging FaceText Generation Inference documentationRe-check before use
Hugging FaceKV cache strategiesRe-check before use

Primary papers

Cost and procurement

FinOps FoundationUnit Economics capabilityRe-check before use
AWSAmazon EC2 purchasing optionsRe-check before use
AWSEC2 Spot best practicesRe-check before use
Google CloudSpot VMsRe-check before use
Microsoft AzureAzure Spot Virtual MachinesRe-check before use
AWSAWS Price List APIRe-check before use
Google CloudCloud Billing Pricing APIRe-check before use
Microsoft AzureAzure Retail Prices APIRe-check before use

Keep the playbook honest

Maintenance and claim policy

Claim class	Policy
Architecture concept / formula	Link primary paper or official technical documentation; revise when the model assumptions change
Hardware specification	Name exact variant and scope; link the live vendor page/data sheet; store last-reviewed date
Software compatibility	Link the current support matrix instead of copying long version tables
Price / quota / availability	Never present as evergreen; store provider, region, shape, term, currency, source and verifiedAt
Benchmark	Preserve complete configuration, workload distribution, SLO, quality, raw result and date
Vendor performance or TCO claim	Label as vendor-reported; retain conditions; do not generalize into a universal multiplier
Preliminary product	Mark preliminary and date the statement; do not use roadmap availability as committed supply