Performance and Memory

TinyLoop has multiple runtime modes, and their numbers are not interchangeable. This page exists to keep benchmark claims disciplined.

Benchmark Shape Matters

The current published benchmark numbers refer to:

• seq_len=128
• loops=8
• no logits materialization
• H100 validation environment

Do not generalize those numbers to:

• arbitrary sequence lengths
• tokenizer-backed service workloads
• batched decode throughput

Current Benchmark States

Path	Result	Interpretation
Default low-bit benchmark	30.48 ms	Main low-bit runtime path
FP16-body benchmark	2.82 ms	Much faster, but much higher VRAM
Full prefill attention	0.143 ms -> 0.087 ms	Safer tiled prefill kernel vs reference

FP16 body cache — 30-65× prefill on 407M INT4

Measured 2026-04-17 on H100 / 407M INT4 / L=8 with TINYLOOP_EXPERIMENTAL_FP16_BODY=1. See the CLI reference for the env flag:

phase	INT4 scalar (default)	FP16 body + cuBLAS TC	speedup
prefill seq=128	183 ms	2.8 ms	65×
prefill seq=256	357 ms	5.2 ms	69×
prefill seq=512	712 ms	13.8 ms	51×
prefill seq=1024	1431 ms	47.5 ms	30×
128-token decode	1241 ms	514 ms	2.4×

Weight VRAM grows ~2.9× (216 → 619 MB on 407M). Output diverges after a few tokens from the INT4 scalar path — FP32 vs FP16 tensor-core accumulator rounding, not a correctness bug.

KV cache INT8 — 2× context at matched VRAM

Measured 2026-04-17 on H100 / 407M INT4 / L=8 with TINYLOOP_KV_INT8=1. Per-token per-head symmetric INT8 quant with one FP16 scale per (token, head):

metric	FP16 KV	INT8 KV	delta
KV memory (max_seq=2048, 10 layers)	167.8 MB	85.2 MB	−49.2%
Effective context at matched VRAM	2048 tok	4032 tok	+97%
Decode time, seq=256	2652 ms	2748 ms	+3.6%
Decode time, seq=1024	15262 ms	16653 ms	+9.1%
Decode time, seq=2048	42668 ms	48462 ms	+13.6%

Greedy output is byte-identical for the first ~40 tokens, then diverges as rounding accumulates — both continuations are valid samples from the quantized model. Composes cleanly with TINYLOOP_EXPERIMENTAL_FP16_BODY=1. Remaining decode regression is the int8→float conversion per element; closing it further would need INT8 tensor cores on Hopper (weeks of work, not currently scoped).

Cross-iteration stochastic rounding — beats naïve INT4 at L ≥ 16

The looped transformer applies the same weight L times per forward, which is the only regime where per-iteration stochastic rounding can cancel quantization bias. §16.9 ships three operating points:

variant	storage	slope	407M PPL vs naïve INT4
naïve INT4 (default)	4 bits/weight	+1.00	± 0 (baseline)
INT4 + 1-bit sign-of-frac	+25 %	+0.80	+2.18 at L=16 (gap shrinks 46 %)
INT4 + 4-bit signed frac	+100 % on the loop block (≈ +15 % model-wide)	+0.50	−0.32 at L=16, −0.67 at L=32

The cheapest variant (1 bit / weight) shrinks the PPL gap with L but does not beat naïve INT4. The 4-bit-frac variant does: at L=16 and L=32 it beats naïve INT4 in measured WikiText-103 PPL, reproducing the sign of the paper §O.2 Bernoulli-oracle simulation.

Enable at runtime with TINYLOOP_CROSS_ITER_ROUND=1 after upgrading a .tinyloop file via tools/tinyloop_add_sign_bit.py:

python3 tools/tinyloop_add_sign_bit.py \
    model_int4.tinyloop model_int4_v5.tinyloop \
    --from-pt model.pt --loop-only --frac-bits 4

--loop-only scopes the sign-bit / frac channel to the looped block only (pre-blocks, embed, head stay plain INT4 — per-iter noise there would hurt because they only run once per forward).

Prefix cache — 3.03× throughput on 10-user same-prefix workload

Measured 2026-04-17 on H100 / 407M INT4 / L=8 via tests/bench_prefix_cache.py:

backend	prefix	N	NO_CACHE	SHARED	speedup
INT4 scalar (default)	500 tok	10	12 847 ms	5 415 ms	2.37×
INT4 scalar (default)	1024 tok	10	23 852 ms	8 329 ms	2.86×
INT4 scalar (default)	1024 tok	50	104 605 ms	34 531 ms	3.03×
FP16 body + TC	1024 tok	10	5 123 ms	4 732 ms	1.08×

The paper-level claim ("prefix cache gets multiplied by L") is conditional on prefill being compute-bound. On the INT4-scalar body we see the full 2.4-3.0× win; on the tensor-core path prefill is already fast enough that prefix sharing only contributes 5-8%.

Key Runtime Switches

TINYLOOP_EXPERIMENTAL_FP16_BODY=1

What it does:

• dequantizes transformer-body weights to persistent FP16 caches at load time
• routes body GEMMs through FP16/cuBLAS

Tradeoff:

• much faster
• much more VRAM

TINYLOOP_DISABLE_FLASH2_PREFILL=1

What it does:

• disables the newer safer tiled prefill attention path
• forces the reference prefill kernel

Use when:

• checking parity against the older path
• isolating the effect of prefill attention changes

TINYLOOP_DISABLE_KV_CACHE=1

What it does:

• disables cached generation
• forces the uncached reference generation path

Use when:

• running parity checks
• debugging decode behavior

TINYLOOP_CUDA_PROFILE=1

What it does:

• prints phase timings during benchmark

Use when:

• locating dominant phases
• checking whether a speedup is in prefill, MLP, attention, or logits

Memory Components

A TinyLoop runtime footprint is roughly:

1. weights
2. persistent buffers
3. optional caches such as FP16 body caches
4. KV cache as sequence length grows

Important current memory levers:

• max_seq_len
• cache_window
• default low-bit path vs FP16 body cache mode

Sliding-Window Cache

cache_window controls how much decode history stays resident in the KV cache.

Semantics:

• 0: full cache
• >0: keep only the last W cached tokens

Use this when:

• long context retention is not required
• bounded KV memory matters more than full history

Prefix Cache Reuse

Prefix caches are a memory and latency strategy for repeated shared prompts.

They help when:

• many requests share a common prefix
• you want to amortize prefill cost

They do not replace batching or paged attention.

How To Publish Numbers Responsibly

When you publish a TinyLoop number, include:

• GPU model
• loop count
• sequence length
• whether logits are materialized
• whether KV cache is involved
• whether TINYLOOP_EXPERIMENTAL_FP16_BODY=1 was enabled
• whether the newer tiled prefill attention path was enabled

Without that context, the number is too easy to misread.