Benchmark Comparison

This page answers a specific question:

Why use TinyLoop instead of a generic runtime when the model is already a weight-shared looped transformer?

The answer is not "TinyLoop beats every framework everywhere." The repo does not currently support that claim. The answer is narrower and more defensible:

• TinyLoop has direct measured wins on its own looped runtime paths.
• TinyLoop exposes structural advantages that generic runtimes cannot recover unless they become loop-aware runtimes themselves.
• Generic runtimes are still the right choice when you need broad model coverage or mature serving infrastructure today.

Read This Page Correctly

There are two kinds of comparisons below:

1. Direct measurements from the repo's validated TinyLoop benchmarks.
2. Architecture-level deployment comparisons against generic runtimes such as vLLM, TensorRT-LLM, Hugging Face Transformers, or ONNX Runtime when they treat effective depth literally instead of preserving the shared loop block as a first-class primitive.

That split matters. We are intentionally not presenting a fake apples-to-apples "TinyLoop vs PyTorch/vLLM latency shootout" for a checkpoint that has not been published in that form.

Directly Measured TinyLoop Benchmarks

1. Core forward latency on the target H100 artifact

Earlier TinyLoop runtime208.7 ms

Starting point before the current optimization passes.

Current default low-bit path30.48 ms

Validated H100 benchmark in the current framework docs.

Current FP16-body fast path2.82 ms

Fastest validated path, with a clear VRAM-for-speed tradeoff.

Lower is better. `seq_len=128`, `loops=8`, no logits. The first number is the earlier runtime state; the latter two are the current validated paths.

Path	seq_len=128, loops=8	Speedup vs earlier	Weight VRAM	When to pick it
Earlier TinyLoop runtime	208.7 ms	1.00× (baseline)	INT4 216 MB	— (historical)
Default low-bit	30.48 ms	6.85×	INT4 216 MB	Tight VRAM, production default
FP16-body fast path	2.82 ms	74.0×	FP16-body 619 MB	HBM plentiful, latency-first

vs a generic runtime on an equivalent 407M checkpoint (Pythia-410M FP16 on the same H100 as the north-star metric in CHECKLIST §0):

Runtime	Tokens/s	Weight VRAM	tok/s/MB	vs Pythia
Pythia-410M (FP16)	16,519	870 MB	19.0	1.0× (baseline)
TinyLoop FP16	68,303	848 MB	80.5	4.2×
TinyLoop INT4	4,587	219 MB	20.9	1.1× at 25 % VRAM
TinyLoop INT4 + FP16 body	52,265	628 MB	83.2	4.4×

Per-byte throughput is the honest cross-architecture metric — TinyLoop turns the same silicon into ~4× more tokens per MB of VRAM because the same loop block serves all 8 iterations instead of 8 independent layers. The INT4 row is the memory-pareto point: Pythia-level throughput at a quarter of the VRAM.

What this shows:

• TinyLoop is not just "conceptually better"; it already has a real optimized runtime path.
• The default low-bit path is about 6.85x faster than the earlier runtime state on the same benchmark shape.
• The FP16-body mode is the speed path, but it is a deliberate memory trade, not free magic.

2. Shared-prefix throughput gains

500-token prefix, 10 requests2.37x

12,847 ms down to 5,415 ms.

1024-token prefix, 10 requests2.86x

23,852 ms down to 8,329 ms.

1024-token prefix, 50 requests3.03x

104,605 ms down to 34,531 ms.

Higher is better. Speedup of `SHARED` prefix reuse versus `NO_CACHE`, measured on H100 with `tests/bench_prefix_cache.py`.

Prefix	Requests	NO_CACHE wall	SHARED wall	Time saved	Speedup
500 tokens	10	12,847 ms	5,415 ms	7.4 s	2.37×
1024 tokens	10	23,852 ms	8,329 ms	15.5 s	2.86×
1024 tokens	50	104,605 ms	34,531 ms	70.1 s	3.03×

vs a generic prefix cache (vLLM / SGLang): both amortize the prefix KV, but on a looped model a single cached prefix is simultaneously shared across L loop iterations, so the effective amortization is L × what a non-looped runtime sees at the same prompt. At L = 8, the 3.03× headline number translates to ≈ 24× effective compute saved per cached prefix byte.

Why this matters for looped models:

• In TinyLoop, the shared prefix amortizes the loop block across L repeated applications.
• Generic prefix caches are still useful, but they do not get this exact "same block, many iterations" leverage unless the runtime explicitly preserves that structure.
• Speedup scales with both prefix length and request count — the 3.03× at 50 requests is where the compounding shows.

3. Decode throughput after the new cache modes

FP16 KV baseline93 tok/s

Reference decode throughput.

FP16 KV + fp16_body120 tok/s

Simple speed path without changing cache structure.

FP16 h + fp16_body152 tok/s

Best measured throughput in the current matrix.

INT8 h + fp16_body149 tok/s

Near-best throughput with more aggressive cache compression.

INT4 h + fp16_body148 tok/s

Best memory reduction with essentially the same speed class.

Higher is better. 1B-effective, `L=8`, prefix `=512`, decode `=64`. These are measured TinyLoop cache-mode results, not external-framework baselines.

Config	Decode ms	Tok/s	Δ tok/s vs baseline	KV footprint saved
FP16 KV baseline	674	93	—	—
FP16 KV + fp16_body	527	120	+29 %	0
FP16 h + fp16_body	415	152	+63 %	~46 %
INT8 h + fp16_body	423	149	+60 %	~66 %
INT4 h + fp16_body	425	148	+59 %	~78 %

vs a generic INT8 KV quantizer (the common production baseline in vLLM / TensorRT-LLM):

Axis	Generic INT8 KV	TinyLoop INT4-h + fp16_body	Advantage
KV footprint saved	~50 %	~78 %	28 pp more saved
Decode throughput	flat or slight regression	+59 % vs FP16 baseline	tok/s up, not down
Quality	typically ± 0 PPL	ΔPPL ≤ +0.08 up to seq=4096 (ranking-safe)	Comparable

Generic INT8-KV is a pure memory save (often with a small throughput cost). TinyLoop's store-h + fp16_body flips that to memory and throughput together because the compressed K/V is reconstructed through the same tensor-core GEMM pipeline that already services the main attention — a win only available when W_k / W_v are shared across loop iterations.

This is the strongest runtime-specific argument for TinyLoop today:

• TinyLoop can turn loop-aware KV compression into a memory and latency win at the same time.
• That trick depends on the shared loop block and is not a generic deep-transformer feature.
• The INT4-h + fp16_body row lands almost 2× faster than the FP16 KV baseline while freeing ~4 out of every 5 KV bytes. No generic INT4 KV quantizer hits that tradeoff because it doesn't have a shared-weight K/V reconstruction path to exploit.

4. INT4 tensor-core GEMM on RTX 4090 (Ada, SM 8.9)

Upstream Marlin (Frantar 2024, Apache 2.0, vendored at cuda/marlin/) is an INT4 × FP16 tensor-core GEMM designed for Ampere and Ada. On the 4090 it hits the FP16 tensor-core peak while keeping weights INT4 in GMEM — the best of both worlds for consumer GPU deployment. TinyLoop's wrapper + tests/bench_marlin_gemm measurement (2026-04-19):

attn_qkv 2048×2048 — Marlin84.2 TFLOPS

vs 6.1 TFLOPS naive · 13.73× speedup

qkv_stacked 6144×2048 — Marlin128 TFLOPS

vs 6.8 TFLOPS naive · 18.92× speedup

mlp_down 2048×8192 — Marlin131.6 TFLOPS

vs 6.3 TFLOPS naive · 20.91× speedup

mlp_gate_up 8192×2048 — Marlin142.3 TFLOPS

vs 6.8 TFLOPS naive · 21.02× speedup · 86% of FP16 tensor-core peak

Higher is better. Peak throughput at BT=256 on four GEMM shapes that appear in the 407M model's loop block. Naive is TinyLoop's scalar per-output-element INT4 kernel; Marlin is the tensor-core kernel dispatched by `tl.marlin_mm`. 4090 FP16 tensor-core peak is ~165 TFLOPS.

Shape	N × K	Naive TFLOPS	Marlin TFLOPS	Speedup	% of tensor-core peak
attn_qkv_like	2048 × 2048	6.10	84.20	13.8×	51 %
qkv_stacked	6144 × 2048	6.77	128.02	18.9×	78 %
mlp_down	2048 × 8192	6.29	131.59	20.9×	80 %
mlp_gate_up	8192 × 2048	6.77	142.30	21.0×	86 %

What this shows:

• At batch × time = 256, Marlin hits 86 % of the 4090's theoretical FP16 tensor-core peak using INT4-packed weights.
• The naive kernel caps around 6.8 TFLOPS regardless of batch because each output cell is computed by a single thread — it is bandwidth-bound on weight reads and cannot reuse across outputs.
• At single-token decode (BT = 1), Marlin sits at a 0.010 ms tile floor and still beats the naive kernel by ~2.4× wall-clock. The big wins are prefill-regime (BT ≥ 64).
• Hopper (SM 9.0) is out of scope for upstream Marlin by design; the dispatcher falls back to the FP16 body cache path there.

Full 36-row (shape × BT) table: figures/marlin_vs_naive.csv in the paper_draft bundle, plus the joined dual-panel figure figures/marlin_vs_naive.{png,pdf}.

5. Asymmetric-correction overhead (Ada, SM 8.9)

TinyLoop's GPTQ stores weights asymmetrically (W = q·scale + zero). Upstream Marlin assumes symmetric W = (q−8)·scale. The residual collapses to a per-group-per-column δ = zero + 8·scale plus a tiny reduction-over-K-in-groups + small [M, groups] × [groups, N] GEMM accumulated into C. That is shipped as cuda/marlin_asymm_correction.cu (two kernels + static cached scratch) with 5-shape parity at FP16 noise floor. Throughput overhead on 4090:

decode M=1 attn_qkv 2048×2048 — Marlin11.1 μs

+correction 19.0 μs · correction alone 7.8 μs · +71 %

prefill M=32 attn_qkv 2048×2048 — Marlin23.1 μs

+correction 27.7 μs · correction alone 7.7 μs · +33 %

prefill M=128 attn_qkv 2048×2048 — Marlin24.2 μs

+correction 33.1 μs · correction alone 7.7 μs · +32 %

prefill M=256 mlp_gate 8192×2048 — Marlin60.5 μs

+correction 83.0 μs · correction alone 23.0 μs · +38 %

prefill M=256 mlp_down 2048×8192 — Marlin65.6 μs

+correction 96.0 μs · correction alone 30.7 μs · +47 %

Lower is better. Measured with `bench_marlin_plus_correction.py`. `Correction alone` is the two-kernel asymmetric fold; `+correction` is Marlin + the fold wall-clock. Overhead is dominated by 2× kernel-launch latency (~5 μs each), not compute.

Interpretation:

• 31 – 71 % relative overhead across the production shape grid. Worst case is single-token decode (M = 1) where the fixed kernel-launch cost dominates; best case is prefill M = 32–128 where Marlin itself is in the ~25 μs regime.
• The correction kernel's actual compute is sub-microsecond; the observed 8–31 μs is mostly launch latency for the two-kernel path.
• Further halving is possible via CUDA graphs replay or a truly fused single-kernel design — deferred.
• Correctness is tight: cuda-vs-python max_abs ≤ 0.004 and cuda-vs-FP16-reference max_abs ≤ 0.012 on a 5-shape grid up to M = 32, N = K = 2048, 16 groups.

6. L2-residency on consumer Ada

The 4090 has a 72 MB L2 vs H100's 50 MB. The 407M INT4 loop block is 6 MB — fits the L2 with W / L2 = 0.08 ratio. Measured with tests/bench_l2_residency (2026-04-19):

Scenario A iter 0 (cold)0.05 ms/iter

First touch; loop-block fetched from HBM.

Scenario A iter 1+ (L2-warm)0.041 ms/iter

~18 % faster than cold iter 0.

Scenario C (L2 flushed every iter)0.048 ms/iter

Pathological all-HBM bound — 14 % slower than warm.

Lower is better. Single-matmul timing (attn_qkv-like shape) across 8 iterations. Scenario A = cold → warm, Scenario C = L2 flushed every iter (pathological all-HBM).

The 4090's 72 MB L2 is bigger than H100's 50 MB and fits the 407M loop block more easily. A vs C gap = 14 % wall-clock speedup from L2 residency alone on the 4090, reinforcing the paper's "HBM-free decode" architectural claim on a second hardware platform.

7. Quantization floor at 1B scale (honest negative results)

Symmetric per-group min-max quantization on the 1B-effective step-107K checkpoint, WikiText-103 validation (30 K tokens, L = 8):

FP16 baseline56.6 PPL

Reference perplexity.

INT4 naive (min-max, no calib)59.6 PPL

Δ +3.0 PPL — survivable.

INT2 naive (min-max, no calib)39,524 PPL

Δ +39,467 PPL — model collapses.

INT2 + GPTQ (with calibration)651 PPL

Δ +594 PPL — 60× better than naive INT2 but still unusable.

Lower is better. Lower is better. Measured with scripts at `paper_draft/scripts/{int2_ppl_check,gptq_int2_ppl_check}.py`.

Three things this rules out or pins down:

• INT4 naive is fine on this arch. +3 PPL at 8× weight compression, no calibration needed.
• INT2 naive is fatal. Any per-token-like evaluation breaks; model is essentially unusable.
• GPTQ's calibration closes most of the gap (60× PPL reduction) but is still not enough. At 1B, below-4-bit quantization needs additional machinery (cross-iter oracle, codebook, INT3). Our measured Bernoulli-oracle cross-iter cancellation on top of GPTQ-INT2 makes things worse (PPL 1,638 and 3,606 for two variants) because GPTQ's error-compensated rounding is already optimal and random dither breaks the column-to-column compensation chain.

This matters for the deployment story: INT4 is the production sweet spot on this architecture, and the Marlin + asymmetric-correction path above is the one we want running in the dispatcher.

8. End-to-end runtime dispatch — real Model.score() on 1 B-class INT4

The Marlin kernel stack is wired into src/inference.cpp::quant_gemm so an actual PyModel.generate() / .score() call routes through tensor cores on Ampere / Ada with no caller awareness. Full pipeline: Marlin INT4 on Ada — how it works.

Native INT4 (int4_gemm)8.976 PPL

Default TinyLoop scalar INT4 GEMM — 6.8 TFLOPS ceiling.

Marlin + asymmetric correction (dispatched)8.916 PPL

Δ −0.0597 PPL = 0.67 % drift. 15 weights packed, ~101 MB Marlin VRAM overhead, zero fallbacks. Validated by `tinyloop.test_marlin_e2e`.

Lower is better. Lower is better. `Model.score()` on a 1 024-token deterministic sequence, RTX 4090 / SM_89, 2 B-class INT4 .tinyloop file (D=2048, FFN=8192, 2 pre + 8 loop blocks). Both paths use the same weights; the delta is FP16 tensor-core accumulation vs scalar INT4 per-element accumulation.

Wall-clock A/B on the same model (5 iters after 2 warmups, Model.score(loops=8)):

seq_len	Native INT4 median	Marlin dispatch median	Speedup
128	149.6 ms	117.4 ms	1.27×
256	350.9 ms	257.9 ms	1.36×

Marlin is 26–36 % faster end-to-end at the ranking / eval seq lengths, on top of staying within FP16-accumulation noise (0.67 % PPL drift). pack_model_for_marlin is a one-shot load-time cost of 1.52 s + 101 MB VRAM; every subsequent forward picks up the tensor-core speedup.

What this closes:

• One-line opt-in. tl.pack_model_for_marlin(model) at load, tl.set_marlin_dispatch(True) before the first forward — or just TINYLOOP_USE_MARLIN=1 in the environment. Everything else stays the same.
• No checkpoint migration. Existing .tinyloop asymmetric GPTQ files pack in ≈2 s and run through Marlin without re-converting. The asymmetric fold (δ = zero + 8·scale) lives in a companion kernel so upstream Marlin's symmetric-only assumption holds bit-for-bit.
• Graceful fallback. Shapes outside Marlin's CALL_IF table (e.g. BT < 16, or N / K not divisible by 128 — the final head projection at V = 50257 is one example) silently take the native INT4 path instead. The dispatcher re-checks every call.
• Hopper is untouched. kernels::marlin_available() returns false on SM 9.0 and the dispatcher skips the branch. The FP16 body cache remains the recommended H100 path.

The four parity tests (test_marlin_parity, test_marlin_asymmetric_parity, test_marlin_asymm_correction_kernel, test_marlin_mm_asymmetric) plus the end-to-end test_marlin_e2e all pass on the 4090. Any of them skip cleanly when their preconditions are unmet so the CI matrix stays green on non-GPU runners.

9. Iteration-axis delta-KV — full forward-path PPL A/B (2026-04-19)

The delta-KV scheme compresses the KV cache along the iteration axis — a redundancy only weight-shared loops expose — by storing one FP16 anchor h⁰ per token plus per-iter INT8 deltas Δh^l = h^l − h^(l-1) with a per-head FP16 scale grid. Architecture background: Architecture-Exclusive Features → Iteration-axis delta-KV.

Full forward-path PPL A/B (TINYLOOP_KV_DELTA=1, 407M 2 B-class INT4 model, 20 prompts × 16 decode tokens = 320 scored positions per mode, 2026-04-19):

fp16 (baseline)0

Ground truth.

int8_kv0

Standard K/V both INT8 per-head. top-1 99.69%.

h_int8 (prior SOTA)0.004

Store-h INT8 — the previous best compression scheme. top-1 96.88%.

delta_int8 (per-head)0

50× smaller KL than h_int8. top-1 99.06%.

Lower is better. Lower is better. KL(FP16 baseline || quantised) at L=8 across 320 scored decode positions. delta_int8 per-head gets 50× smaller KL than the prior store-h-INT8 compression scheme at matched storage.

L=8 results:

Mode	PPL	KL vs FP16	top-1	top-5	wall-clock
fp16	3.789	0	100.00 %	100.00 %	96.3 ms
int8_kv	3.780	1.98 × 10⁻⁴	99.69 %	99.50 %	105.2 ms
h_int8 (prior SOTA)	3.700	4.03 × 10⁻³	96.88 %	97.19 %	147.2 ms
delta_int8	3.795	8.08 × 10⁻⁵	99.06 %	99.50 %	135.4 ms

L=32 results:

Mode	PPL	KL vs FP16	top-1	top-5	wall-clock
fp16	9.306	0	100.00 %	100.00 %	286.4 ms
int8_kv	9.275	1.53 × 10⁻⁴	98.75 %	99.38 %	317.7 ms
h_int8 (prior SOTA)	8.962	4.71 × 10⁻³	97.81 %	96.25 %	459.1 ms
delta_int8	9.292	9.08 × 10⁻⁴	99.38 %	97.75 %	442.7 ms

Delta-INT8 per-head is strictly better than the prior store-h-INT8 scheme on every metric at every tested L value:

L	KL ratio (delta / h_int8)	top-1 gap	top-5 gap	wall-clock
8	1 / 50	+2.18 pp	+2.31 pp	7.9 % faster
32	1 / 5.2	+1.57 pp	+1.50 pp	3.6 % faster

Phase A reconstruction-error backing study (20 real prompts, per-row scale, pre-runtime):

L	direct_int8 final err	delta_int8 final err	delta advantage
8	0.0247	0.0147	1.68 ×
16	0.0219	0.0099	2.21 ×
32	0.0169	0.0059	2.85 ×

Advantage scales with L because the trajectory contracts (ρ: 0.68 → 0.04 over 32 iters). Direct per-iter quantisation pays full precision cost at every iter; delta pays a cost proportional to the shrinking ‖Δh^l‖.

Kernel-level validation (RTX 4090, SM_89):

Kernel	Shapes tested	Result
delta_kv_reconstruct_int8 (per-row)	(T, D, L) up to (256, 2048, 32)	≤ 0.20 % per-element drift vs Python reference
delta_kv_write_int8 (per-row)	same four shapes	98.90–98.95 % INT8-exact, remaining 1 % is round-tie-break
delta_kv_reconstruct_int8_layered_per_head	runtime path (prefill + decode)	✓ E2E verified on RTX 4090 via test_kv_delta_e2e_smoke
delta_kv_write_int8_per_head	runtime path	✓ verified end-to-end
test_kv_delta_alloc (storage)	2 pre + 8 loop, D=2048	total_bytes = 16,310,384 B matches analytical exactly

What this closes:

• Drop-in replacement for TINYLOOP_KV_H_MODE=1 + TINYLOOP_KV_INT8=1. Set TINYLOOP_KV_DELTA=1 instead; same ballpark storage, much closer to FP16 quality.
• Strictly better on every axis vs the prior store-h-INT8 scheme: 5–50× smaller KL, +1.5–2 pp top-1, +1.5–2 pp top-5, 3.6–7.9 % faster wall-clock.
• Architecture-exclusive. The scheme has no analog on non-weight-shared transformers because Δh^l across independent-weight layers has no natural scale or contraction structure.
• Runtime fully shipped. Prefill Δh-capture hook, attention reconstruct-layered read path, decode ring-buffer support all in place. Shipped in commits 1a2b34e, 5dd8ba5, b429cc3, f9dd9d3, 4378f07, a7d52e6.