aipat.dev
Method

Training vs inference stacks: from 3D parallelism to paged KV—how industry buyers should layer selection and acceptance

Inference Runtime · 2026.07.31

Training bottlenecks (compute/interconnect) differ from inference (KV/throughput)—split stacks and acceptance KPIs; mixed metrics are why PoCs rarely close.

Industry buyers often conflate two different engineering problems when asking whether "compute is enough": training must fit parameters and optimizer state onto a cluster and raise effective FLOPs; inference must fit a dynamically growing KV cache and concurrent requests into limited GPU memory and raise tok/s. Mixing stacks, metrics, and acceptance criteria is a common reason pilots never become signable PoCs. Multipliers below are from public papers/vendor docs under stated conditions—not customer SLAs. Scan the six structure figures first, then read the matching sections.

  1. 1) Problem definition — fit vs move

Lifecycle: pre-training → post-training (SFT / preference / RL) → serving. Different bottlenecks and stacks. Write the scenario before naming a framework.

Fig.1 · Fork first: lifecycle / fit-vs-move, then pick a stack
Fig.1 · Fork first: lifecycle / fit-vs-move, then pick a stack

Kwon et al. (SOSP 2023): ~13B on A100-40GB ≈ 65% weights / 30% KV—KV management sets feasible batch and throughput. FlashAttention (Dao et al., 2022) is IO-aware Attention; PagedAttention solves dynamic serving KV layout—different problems.

  1. 2) Two chains

Training chain: parallelism → ops/comms → data/checkpoint → accept with tokens/GPU/s, MFU, loss.

Inference chain: gateway → scheduling → Runtime (kernels, quant, KV accounting) → accept with throughput–latency, KV peaks, soak.

Fig.2 · Training vs inference chains — two ledgers (FLOPs vs KV/throughput)
Fig.2 · Training vs inference chains — two ledgers (FLOPs vs KV/throughput)

On-site check (training): freeze global_batch, parallelism, precision; report tokens/GPU/s, MFU, checkpoint recovery under injected failure—loss alone does not pass.

On-site check (inference): same model/quant throughput–latency Pareto; peak KV and prefix hit rate if enabled; split decode/prefill KPIs; ≥24h soak without OOM or tail-latency drift beyond threshold.

  1. 3) Critical path

FlashAttention (operator IO) and PagedAttention (serving KV paging) complement—modern engines often run Flash-style kernels over paged KV. Not a brand war.

Fig.3 · FlashAttention complements PagedAttention — stack, do not pick-one
Fig.3 · FlashAttention complements PagedAttention — stack, do not pick-one
Fig.4 · Prefill vs Decode — split KPIs; KV constrains throughput
Fig.4 · Prefill vs Decode — split KPIs; KV constrains throughput

Prefill is more compute-heavy; decode is often bandwidth-heavy. Ban a single "overall speedup" in the acceptance annex.

  1. 4) Four scenarios × four acceptance languages
Fig.5 · Scenario × acceptance KPI matrix
Fig.5 · Scenario × acceptance KPI matrix

Cloud API: curves, P50/P99, KV, prefix hit, soak. Engineering bar: acceptance annex must specify concurrency protocol and soak retest steps; spinning up a demo without a retestable protocol does not pass. Token factory: same-GPU same-quant A/B, decode KPI, cutover cost, on-site retest. Edge/Jetson: SKU, co-resident processes, unified-memory peak. Post-training/RL: rollout engine, training engine, sync protocol.

  1. 5) PoC annex — frozen fields (any edit is scope change)
Fig.6 · PoC freeze fields — public multipliers only in Background
Fig.6 · PoC freeze fields — public multipliers only in Background

[PoC annex · freeze list]

  • KPI: decode_tok_s@P50/P99; ttft_ms@P99; max_concurrency; soak_h≥24; prefill/decode ratio. Ban one overall speedup.
  • Assets: model_id@tag; Dense/MoE; quant + version; reference impl version.
  • Hardware: GPU SKU + driver + CUDA; unified memory yes/no.
  • Baseline: engine==… @commit/version; TRT-LLM build flags for paged KV / reuse.
  • Alignment: token-match / logprob thresholds; fail → no cutover.
  • Citation rule: public multipliers only in Background—never Commitment. Commitment = on-site retest protocol + thresholds.
  • Delivery boundary: contracting party ≠ cited open-source; illustrative ≠ SLA.
  1. 6) Sources & boundaries

Method note only. Figures 2–4×, ~15%, ~3×, A100 profile are author-reported conditions—retest after hardware/model/concurrency change. Primary: arXiv:2309.06180; 2205.14135; 1910.02054; 2104.04473; NVIDIA TensorRT-LLM docs; vLLM / llama.cpp as method references. Contact: contact@aipat.dev

← Back to updatesGet in touch →