TinyLoop Framework

TinyLoop is a C++/CUDA inference framework for weight-shared looped transformers.

It is designed for a specific architectural family:

1. embed tokens
2. run a small number of pre-blocks
3. reuse one shared loop block L times
4. normalize and project to the output head

That specialization is the entire point. TinyLoop is not trying to become a universal transformer engine. It is trying to be a serious runtime for a model family that mainstream stacks do not treat as first-class.

Who This Is For

TinyLoop is a fit for:

• research infra teams working on recurrent or looped transformer variants
• systems engineers optimizing low-bit inference for a controlled checkpoint family
• teams that want a reusable runtime surface instead of one-off benchmark scripts

It is not a fit if you need:

• a generic model zoo
• a ready-made hosted serving platform
• broad compatibility across arbitrary transformer checkpoints

What Ships Today

TinyLoop currently provides:

• a reusable C++ library surface
• a CLI for inspect, benchmark, generate, and speculate
• an optional Python binding via pybind11
• a custom .tinyloop model format
• CUDA kernels for quantized GEMM, attention, fused ops, and KV-cache helpers
• default-on cached decode
• regression coverage for cached-vs-uncached behavior

What Makes It Different

TinyLoop is built around properties that are unusual in mainstream runtimes:

• Loop count is a runtime knob. Quality and compute can move without changing the weight set.
• Weight memory is tied to unique blocks, not effective depth.
• Draft and verify can reuse the same weights. That makes self-speculative decode natural.
• Low-bit execution is not an afterthought. The runtime is built around INT2/INT4 and optional FP16 body caches.

Current State

Today TinyLoop is best described as:

• a real specialized runtime
• a framework with a reusable library, CLI, Python binding, and test surface
• not yet a finished serving platform

Current validated H100 numbers for the target artifact:

Path	Result	Notes
Default low-bit benchmark	30.48 ms	seq_len=128, loops=8, no logits
FP16-body fast mode	2.82 ms	TINYLOOP_EXPERIMENTAL_FP16_BODY=1
Full prefill attention	0.143 ms -> 0.087 ms	Safer tiled prefill path vs reference

Current quality statement:

• no regression has been detected in the validated paths
• that evidence comes from CUDA tests, decode parity, CLI regressions, tokenizer-aware regressions, and eval-slice coverage
• this is not a universal proof across all prompts or artifacts

Recommended Reading Order

If you are new to the repo:

1. Installation
2. Quickstart
3. Benchmark Comparison
4. CLI Reference
5. Current Status

If you are integrating TinyLoop:

1. Config Reference
2. C++ API
3. Python API
4. Model Format

If you are working on the runtime itself:

1. Runtime Architecture
2. Performance and Memory
3. Testing and CI
4. Production Roadmap