Current Status
This page is the blunt state-of-the-project document.
Capability Matrix
Core inference
| Area | State | Notes |
|---|---|---|
| Inspect | Working | Metadata-first inspect path with memory estimates |
| Benchmark | Working | Forward-only latency/throughput path |
| Full scoring | Working | Full logits materialization |
| Last-token scoring | Working | Lower-memory decode-style path |
| Generation | Working | Cached decode is the default path |
| Uncached generation | Working | Explicit opt-out via TINYLOOP_DISABLE_KV_CACHE=1 |
| INT3 GEMM | Working | 8 levels, 3 bits/weight. 10 values per 32-bit word. --bits 3 in converter. |
| LoRA amplification | Working (demo) | Rank-r LoRA on loop block is L× more parameter-efficient. |
| JSON-RPC server | Working | Dependency-free TCP RPC for microservice deployment. |
| Benchmark suite | Working | L-sweep tool with decode/prefill latency + logit-lens convergence. |
| Dynamic-L inference | Working | L=2..64 from one model. Per-request, per-token, SLO-aware depth. Architecture-exclusive. |
| Dynamic-L evaluation | Working | tools/dynamic_l_eval.py — per-L agreement, Poisson-L sampling, logit stability. |
| FP16 residual stream | Working (opt-in) | TINYLOOP_FP16_RESIDUAL=1. 52 call sites wired via 10 dispatch helpers. All paths: decode, prefill, chunked, logit-lens, speculative, safety check. H100 verified: FP32 4/4 regression pass, FP16 score+generate+speculative pass. |
KV cache modes — 7 modes from FP16 to INT4-h
| Area | State | Notes |
|---|---|---|
| Prefix caching | Working | Public C++ and Python surfaces |
| Sliding-window cache | Working | cache_window bounds KV storage |
| INT8 KV cache | Working | TINYLOOP_KV_INT8=1 halves KV VRAM |
| Store-h KV (FP16) | Working | Bit-exact, 43–50% cache saving |
| Store-h KV (INT8) | Production-ready | 66–75% saving, greedy-preserving |
| Store-h KV (INT4) | Eval/ranking-safe | 78% saving, use INT8-h for generation |
| Tensor-core reconstruction | Recommended | fp16_body: −37% decode + same cache saving |
| Paged attention | Working | TINYLOOP_PAGED_KV=1 — 16-token pages, 5.6% overhead |
Speculative & adaptive decode
| Area | State | Notes |
|---|---|---|
| Self-speculative decode | Working | L=2 draft, L=8 verify, same weights. 2.77× throughput, zero extra params. Greedy output bit-exact to generate(loops=8). |
| Tree speculative decode | Working (greedy) | One mask-gated forward replaces K sequential verify passes. Bit-exact, zero auxiliary heads. Chain and multi-branch modes. |
| Warm-start mid-loop | Working | Resume from cached iter-k state. −32.5% wall-clock. Break-even at N=1 follow-up. |
| Beam search | Working | Per-beam KV cache, length-normalized ranking |
| Constrained decoding | Working | Regex NFA grammar, JSON mode |
| L scheduling | Working | First N tokens at shallow depth, ~3.3× faster warmup. Architecture-exclusive. |
| Rewind speculation | Working (telemetry) | 88.9% mid-depth hit rate. Architecture-exclusive. |
| Adaptive speculation depth | Working | Draft-ahead K self-tunes in both cached and uncached paths. K=4→6 cached, K=4→8 uncached. |
| CUDA memory pool | Working | TINYLOOP_CUDA_POOL=1 — uses cudaDeviceGetDefaultMemPool + cudaMallocFromPoolAsync for reduced per-allocation overhead and fragmentation. Falls back to cudaMalloc on failure. |
| SLO-aware L downgrade | Working | When predicted wait exceeds SLO, scheduler computes max_l = floor(budget / per_iter_ms) and serves at reduced depth instead of returning HTTP 503. Architecture-exclusive: quality graceful degradation via depth, not separate models. |
| Chrome trace export | Working | TINYLOOP_CUDA_PROFILE=1 tinyloop benchmark | python tools/trace_to_chrome.py > trace.json — converts per-phase CUDA event timings to Chrome Trace Event Format for chrome://tracing visualization. |
| Token trajectory visualizer | Working | python tools/trace_tokens.py --model m.tinyloop --prompt "text" --L 8 — shows predicted argmax at each loop depth via score_logit_lens, highlights iteration flips and stabilization point. Also available as CLI: tinyloop trace. |
| Performance regression tracker | Working | python tools/perf_regression.py --model m.tinyloop --save baseline.json records benchmark/decode latency. --baseline baseline.json compares against saved results with configurable threshold (default 10%). Tested on H100: all PASS. |
| One-command conversion | Working | python tools/convert.py model.pt --bits 4 --rope --n-kv-heads 4 — auto-detects fused QKV vs separate Q/K/V projections, fuses on the fly for GQA models. Writes .tinyloop binary + .json config sidecar. Delegates to gptq_convert.py with --gptq flag. Verified end-to-end: 204.4 MB output loads and generates on H100. |
Model architecture support
| Area | State | Notes |
|---|---|---|
| RoPE | Working | All forward paths, configurable theta |
| GQA | Working | Fewer KV heads, MQA also works |
| GeLU activation | Working (opt-in) | For GPT-2/BERT-style models |
| Post-norm | Working (opt-in) | For post-norm model variants |
| Chunked prefill | Working | Buffer VRAM −62% at chunk=512 |
Architecture-exclusive features
| Area | State | Notes |
|---|---|---|
| Deterministic error cancellation | Working (v5 models) | Even iterations use standard rounding, odd apply sign correction. O(1) total error vs O(L) naive. TINYLOOP_CANCEL_ROUND=1. Needs v5 .tinyloop with sign_bit channel. Novel — no prior work. |
| Per-iteration safety check | Working | After each loop iteration, compute residual L2 norm. If exceeds threshold, abort immediately. L checkpoints per token vs 1 in standard transformers. Cost: 2000× cheaper than the attention it guards. Novel. |
| L scheduling | Working | First N tokens at warmup_L (e.g. L=2), rest at full n_loops (e.g. L=8). Warmup tokens ~3.3× faster. CLI: --l-warmup-tokens 16 --l-warmup-L 2. |
| Rewind speculation | Working (telemetry) | On draft rejection, check if intermediate depth (L=4) agrees with verify (L=8). 88.9% hit rate — most rejections don't need full verify. --rewind-L 4. |
| Dual-bit weight swap | Working | INT2 copy (~17 MB) + INT4 copy (~34 MB). Early iters use INT2, late use INT4. 3.2% faster decode. TINYLOOP_DUAL_BIT=1. Novel. |
| Triple-bit weight swap | Working | Adds INT8 copy (~68 MB) for iter 0 on top of dual-bit. TINYLOOP_TRIPLE_BIT=1. Highest quality for first iteration. |
| L-aware batching | Working (verified) | per_lane_loops=[4,16,32] — each lane at its own depth in one generate_batch call. Bit-exact match with standalone at each L. Free/pro/enterprise from one model. Novel. |
Infrastructure & serving
| Area | State | Notes |
|---|---|---|
| Marlin INT4 tensor-core | Working (Ampere/Ada) | 21× over naive INT4, 0.67% PPL drift |
| Static batching | Working | generate_batch, 1.87× throughput at N=4 |
| HTTP serving | Working | FastAPI with OpenAI-compatible + native endpoints |
| Memory-mapped loading | Working (default) | mmap + direct DMA to GPU |
| Memory arena allocator | Working | Zero-fragmentation multi-model serving |
| Fused GEMM+residual | Working (FP16 body) | cuBLAS beta=1.0, eliminates intermediate write |
| NVTX profiling | Working | Per-phase markers for NSight Systems |
| One-command conversion | Working | python tools/convert.py model.pt --bits 4. Auto-detects fused/separate QKV, supports GQA, config sidecar JSON. |
| Python binding | Working | pybind11 module tinyloop_py |
| Priority tiers | Working | TINYLOOP_TIERS — map tier names to loop depths |
| Trace CLI | Working | tinyloop trace — argmax at each loop depth |
| Unified memory fallback | Working | TINYLOOP_UNIFIED_MEMORY=1 — spill to system RAM |
| Adaptive L from traffic | Working | AdaptiveLController — off-peak L=32, peak L=4. Architecture-exclusive. |
| Model editing amplification | Working (demo) | Loop block edits are L× more effective than single-layer edits. |
| Activation steering | Working (demo) | Depth as gradient-free steering knob with KL measurement. |
| Iteration-dependent scales | Working | Per-iter activation magnitude drops 50% (iter 1→8). |
| Cost-based request priority | Working | Scheduler auto-calculates priority from prompt_len × loops. Short cheap requests dispatched before long expensive ones. Architecture-exclusive: L is a meaningful cost factor only because depth is variable. |
| SentencePiece tokenizer | Working | TINYLOOP_TOKENIZER_MODEL=path/tokenizer.model for Llama-family models. Provides encode/decode/token_to_bytes for grammar integration. |
| Tiktoken tokenizer | Working | TINYLOOP_TIKTOKEN_ENCODING=cl100k_base for GPT-4/Claude-family. |
| Auto tokenizer loading | Working | load_tokenizer_auto() tries SentencePiece → Tiktoken → GPT-2 BPE in priority order. |
| INT8 weight GEMM | Working | cuda/int8_gemm.cu — 1 byte/weight, 256 levels. Per-row fast path. Used by triple-bit path. |
| Async Python API | Working | AsyncModel in python/tinyloop_tools/async_model.py. asyncio.Lock + ThreadPoolExecutor. Supports generate, generate_stream, score*, generate_batch. |
| Torch-native wrapper | Working | TorchModel returns torch.Tensor via zero-copy from_numpy(). |
| HuggingFace integration | Working | TinyLoopForCausalLM.from_pretrained() with standard HF generate() interface. |
| GGUF reader | Working | tools/gguf_reader.py — reads llama.cpp GGUF v2/v3 headers + tensor info. |
| Model sharding | Working | tools/model_shard.py — split .tinyloop into N shards with JSON manifest. |
| β-mixed precision | Working | tools/beta_mixed_precision.py — greedy knapsack bit allocation from measured β. |
| Continuous batching | Not built | Requires scheduler work |
What Is Actually Validated
Validation coverage — 12 test categories
- CUDA kernel/unit tests (INT2, INT4, ops, attention, RoPE, GQA, fused LN+QKV)
- cache parity for decode behavior
- raw-byte CLI generation regressions
- tokenizer-aware Python generation regressions
- small eval-slice parity coverage
- prefix-cache regression coverage
- speculative greedy and sampling regressions
- variable-L parity (bit-exact logit-lens at L=16)
- memory leak detection (5-cycle load/unload, 0 MB growth)
- fuzzing (50 iterations, random seq_len/L/tokens)
- KV-cache regression suite (27-point cached-vs-uncached parity sweep)
- constrained decoding validation (regex, choice, JSON-shape)
Current Performance Statements
H100 benchmark numbers
| Path | Workload | Result |
|---|---|---|
| Default low-bit benchmark | seq_len=128, loops=8, no logits | 30.48 ms |
| FP16-body benchmark | seq_len=128, loops=8, no logits | 2.82 ms |
| Full prefill attention | T=128, D=2048, heads=16 | 0.143 ms -> 0.087 ms |
| Cached decode attention | K=128, D=2048, heads=16 | 0.092 ms |
KV cache mode decode latency — 1B-effective, L=8
| Config | Decode ms | vs baseline |
|---|---|---|
| FP16 KV baseline | 674 | — |
| FP16 KV + fp16_body | 527 | −22% |
| FP16 h + fp16_body | 415 | −38% ⚡ |
| INT8 h + fp16_body | 423 | −37% ⚡ |
| INT4 h + fp16_body | 425 | −37% ⚡ |
Warm-start wall-clock
| Config | Wall time | vs fresh |
|---|---|---|
fresh generate(loops=8) | 329 ms | — |
resume_generate(new_L=8) | 222 ms | −32.5% |
Break-even at N=1 follow-up.
L-sweep decode latency — 2B INT4, H100, 32-token prompt, 16 decode tokens
| L | Wall time (ms) | Throughput (tok/s) |
|---|---|---|
| 2 | 111.8 | 301 |
| 4 | 146.8 | 226 |
| 8 | 161.3 | 198 |
| 16 | 305.5 | 108 |
| 32 | 506.5 | 64 |
Architecture-exclusive: same weight file at all depths. Quality is a compute knob.
Multi-bit weight swap — 2B INT4, H100, L=8
| Config | Decode ms | Throughput | Notes |
|---|---|---|---|
| Baseline (INT4) | 86.9 | 184 tok/s | — |
| Dual-bit (INT2+INT4) | 86.2 | 186 tok/s | ~1% faster |
| Triple-bit (INT8+INT2+INT4) | 91.2 | 175 tok/s | ~5% slower, highest iter-0 quality |
Honest VRAM Comparison (nvidia-smi, H100)
Our 2B single-block model — 2 pre + 1 loop × L=8
| Runtime | VRAM | vs unrolled |
|---|---|---|
| PyTorch FP16 (unrolled 10 layers) | 2225 MB | baseline |
| PyTorch FP16 (weight-shared 2+1) | 798 MB | 2.8× less |
| TinyLoop INT4 | 737 MB | 3.0× less |
| TinyLoop INT4 + FP16 body | 1187 MB | 1.9× less (fastest) |
The 2.8× saving is from weight sharing (architecture property), not the runtime. TinyLoop's INT4 gives a modest additional ~1.1×. The FP16 body cache trades memory for speed (177 tok/s).
Ouro 1.4B — 24 unique layers × 4 UT steps
| Runtime | VRAM | vs PyTorch |
|---|---|---|
| PyTorch FP16 | 3457 MB | baseline |
| TinyLoop FP16 | 3343 MB | ~same |
| TinyLoop INT8 | 1983 MB | 1.7× less |
| TinyLoop INT4 | 1263 MB | 2.7× less |
Ouro has 24 unique layer weights — no weight sharing savings. The VRAM difference comes purely from quantization, same as any other inference engine would give.
What's Honest
- Weight sharing saves VRAM — but that's an architecture property, not a TinyLoop advantage. PyTorch with the same shared weights uses similar memory.
- INT4 quantization saves VRAM — but any quantization engine (llama.cpp, vLLM) gives comparable savings.
- TinyLoop's real advantages are: L as a runtime knob, self-speculative decode, per-request depth, architecture-exclusive features (error cancellation, safety check, dual-bit). These are things no other runtime can do.
- Speed: 177–301 tok/s on H100 at INT4, competitive for the model size.
Claims That Are Safe
- TinyLoop is a real looped-transformer runtime with 100% CTest pass rate
- Cached decode is implemented and enabled by default
- Regression coverage exists for all validated paths
- L is a true runtime parameter: same model at L=2..64
- Multi-block loop support: Ouro-1.4B loads and runs (output quality in progress)
Claims That Need Caution
- Production-ready hosted serving (no continuous batching)
- Broad quality equivalence with PyTorch (Ouro output quality not yet matched)
- "X× less VRAM than PyTorch" without specifying that weight sharing is an architecture property
- Performance claims without noting CUDA context overhead (~500 MB on H100)