Claims & Data Sources
Why TinyLoop Is Different From vLLM, TensorRT-LLM, And Similar Stacks
The point is not that those frameworks are weak. It is that they optimize a different default assumption.
| Question | TinyLoop | Generic runtime |
|---|---|---|
| Treats the shared loop block as a first-class runtime primitive | Yes | Usually no |
| Runtime-variable depth from one checkpoint | Yes | Usually not exposed cleanly |
Single-checkpoint self-speculative decode (draft_loops vs verify_loops) | Yes | No, usually needs a second draft model or extra training |
h-mode KV reconstruction from shared W_k / W_v | Yes | No, standard deep stacks do not have the same shared-weight property |
| Best choice for broad arbitrary-model serving today | No | Yes |
| Best choice when the checkpoint family is already looped and weight-shared | Yes | Often not |
Safe Claim vs Unsafe Claim
Safe claim:
- TinyLoop is the better runtime when you want to preserve the actual structure of a looped transformer and monetize that structure in memory, cache design, and single-checkpoint decoding tricks.
Unsafe claim:
- TinyLoop is universally faster than PyTorch, vLLM, TensorRT-LLM, or every other runtime on every workload.
This repo does not support the unsafe claim yet, and the page should not pretend otherwise.
Why Someone Should Choose TinyLoop
Choose TinyLoop when:
- your model family is already weight-shared or looped
- VRAM density is the first-order deployment constraint
- you want depth to stay a runtime knob
- you want loop-specific features such as
h-mode KV storage or single-checkpoint self-speculative decode
Choose a generic runtime when:
- you need wide checkpoint compatibility
- you need mature batching and serving infrastructure immediately
- the model is an ordinary deep stack with no loop-aware structure to preserve
Data Sources In This Repo
The numbers above are assembled from:
docs/framework/current-status.mddocs/framework/python-api/prefix-cache.mddocs/framework/python-api/generation.mdnext.md- New (2026-04-19):
tests/bench_marlin_gemm.cu,tests/test_marlin_parity.py,tests/test_marlin_asymmetric_parity.py,tests/test_marlin_asymm_correction_kernel.py,tests/bench_l2_residency.cu, plus the 4090 / 1B measurement bundle atpaper_draft/4_19.md(§"Marlin INT4 end-to-end on RTX 4090" and §"Afternoon results — three big wins") and the raw logs underpaper_draft/data/.