Benchmark Comparison

This page answers a specific question:

Why use TinyLoop instead of a generic runtime when the model is already a weight-shared looped transformer?

The answer is not "TinyLoop beats every framework everywhere." The repo does not currently support that claim. The answer is narrower and more defensible:

• TinyLoop has direct measured wins on its own looped runtime paths.
• TinyLoop exposes structural advantages that generic runtimes cannot recover unless they become loop-aware runtimes themselves.
• Generic runtimes are still the right choice when you need broad model coverage or mature serving infrastructure today.

Where Generic Runtimes Lose On Looped Checkpoints

Generic runtimes are strong at general serving. The problem is architectural mismatch.

If the checkpoint is fundamentally:

embed -> pre blocks -> shared loop block x L -> head

then a generic runtime usually wants to see:

embed -> block 1 -> block 2 -> ... -> block N -> head

That mismatch is exactly where TinyLoop wins. The savings are multi-dimensional; each subsection drills one axis.

Axis	Generic deep-stack baseline	TinyLoop	Saving / gain	Details
Weight VRAM per instance	2,700 MB	204 MB	13.2× smaller	§"Per-instance weight memory"
Instances per 80 GB GPU	29	392	13.5× more	§"Model density on 80 GB GPU"
Tokens/s per MB of VRAM	19.0 (Pythia-410M FP16)	83.2	4.4×	§1 above
KV cache saving vs INT8 KV	~50 %	~78 % (INT4-h)	+28 pp	§3 above
Prefix-cache amortization	1× per cached token	L× per token	8× at L = 8	§2 above
Loop block fits in GPU L2	No (24-layer stack > 50 MB)	Yes (204 MB → fits with 72 MB L2 on 4090)	—	§6 above
INT4 GEMM peak throughput	capped by generic INT4	86 % of FP16 tensor-core peak	—	§4 above

Individual subsections below drill each row.

1. Per-instance weight memory

Generic deep-stack runtime2,700 MB

About 2.7 GB per model instance for the reference comparison in `next.md`.

TinyLoop shared-block runtime204 MB

About 204 MB per instance for the same effective deployment story.

Lower is better. Architecture-level deployment comparison from the project roadmap. This is not a published public latency bakeoff; it is the memory cost of carrying effective depth literally versus preserving the shared block.

Interpretation:

• TinyLoop is compelling when memory is the bottleneck.
• On this comparison, the model footprint is about 13x smaller.
• That advantage compounds with serving density, edge fit, and multi-model deployment.

2. Model density on an 80 GB GPU

Generic deep-stack runtime29 instances

floor(80 GB / 2.7 GB)

TinyLoop shared-block runtime392 instances

floor(80 GB / 204 MB)

Higher is better. Simple back-of-the-envelope density using the same per-instance numbers above.

That is the practical sales story:

• more tenants per GPU
• more checkpoints resident at once
• smaller edge deployment envelope