Mid-size token/talk factories often look like this: dozens of consumer GPUs, domestic open weights, revenue priced per token. Where does the next dollar go—another card, or higher tokens-per-second (tok/s) on cards you already own?
One judgment: on purchased hardware, first verify whether swapping the inference Runtime on the same weights clears agreed gates; only then scale out. Adding cards raises capacity; it does not automatically fix per-card decode (token-by-token generation) efficiency.
A rough ledger: revenue tends to move with tok/s × cards × price. Once cards are bought and price is market-set, the near-term lever is often same-card throughput.
- 1) Fork first: swap Runtime, or buy another card
Path A — swap Runtime: same weights (often GGUF) + same GPU, A/B versus the current baseline (e.g. a generic stack) on tok/s, latency slices, and quality gates. No retraining. If serving is OpenAI-compatible, cutover can be a base-URL change—after the customer-card measurement wins, not after reading a public dashboard.
Path B — buy another card: raise capacity. If the decode path is still fragmented, new cards multiply inefficiency along with Capex and ops surface.
Getting the order wrong looks familiar: more cards, still-unclear per-card margin; the story becomes “we added GPUs,” not “same-weights Runtime cleared which gate.”
- 2) Three orthogonal layers
- 1) Platform shell — pull models and expose APIs. Convenient; rarely the tok/s battleground.
- 2) Runtime — the engine that produces tokens (load, kernels, KV cache, scheduling, quant path). Phase-1 lives here.
- 3) Model version — name + weights + quant tag. Changing the tag means re-measure; never extrapolate public uplifts.
Memory aid: swapping Runtime ≠ swapping models. Jetson edge, token-factory racks, and full-size multi-GPU serving are different storylines—do not mix illustrative numbers.
- 3) Decode is often the battlefield
Prefill affects time-to-first-token; decode binds “sell tokens” feel. MoE amplifies expert switching and kernel fragmentation. Public engineering techniques (routing control, batch=1-friendly decode ops, scheduling, Attention/KV bandwidth paths) are methodology—not unverified sales multipliers. Any tok/s or speedup is premise-bound; other SKUs need on-site A/B. Quality parity matters as much as throughput.
- 4) Five questions when the model changes
- 1. Format ready for same-weights compare (e.g. GGUF)?
- 2. Structure: MoE, dense, or full-size cluster?
- 3. Quant match to public bench conditions?
- 4. GPU SKU match? Stock cards require measurement.
- 5. Load: context, concurrency, prefill vs decode?
PoC checklist (indicative): same GPU/weights; named baseline; separate decode/prefill KPIs; quality gates; promote staging traffic only if uplift clears. Honest close: public numbers are reproducible under stated conditions; retest on the customer model and card.
- 5) Red lines, talent gap, and entries
Illustration ≠ SLA; open-source reference ≠ contracting party; do not mix scenario packs; unshipped features are not inventory; no implied ROI.
Teams often lack LLM/GPU Infra talent who can write a one-page acceptance annex—not another résumé that says “can deploy.” Diagnose the bench gap first, then match.
- Tech / trial: https://aipat.dev/#/mrf · https://aipat.dev/#/exclusive
- Team diagnosis / Experts: https://aipat.dev/
- contact@aipat.dev
Industry method note only—not investment, employment, or placement offers; indicative figures ≠ customer SLA; scope follows written SOW and customer-environment retests.