メインコンテンツまでスキップ

Architecture-Exclusive Features

TinyLoop's design target is a specific architecture family: weight-shared looped transformers, where a small set of unique layers plus one "loop block" that is applied L times with the same weights every iteration produces the effective depth of a much larger model. ALBERT is the best-known instance; Universal Transformer is another; MoR and related 2024-2026 depth-recurrent work is a third wave.

Because the loop block's W_q, W_k, W_v, attention projection, LayerNorms, and MLP matrices are identical across iterations, a family of runtime optimisations becomes possible that simply cannot exist on non-shared transformers. This page lists those capabilities and, for each, explains why the optimisation is only available under weight sharing.

If you are evaluating TinyLoop against a runtime like vLLM / llama.cpp / TensorRT-LLM and cannot figure out what TinyLoop does that they don't, this is the page.

Quick reference​

FeatureEnv / APISavesWhy only shared-weight loops can do this
Warm-start mid-loopbuild_resume_handle + resume_generateUp to −32.5 % wall-clock on follow-up requestsResuming at iter k requires weights at iter k to equal weights at iter 0
Variable-L KV cacheAny call with different loops= on the same cacheShare a cache across depthsCache layer n_pre + i is reusable at any L ≥ i + 1 because iter i reads the same weights every time
Self-speculative decodingModel.generate_speculative(...)2.77× throughput, zero extra paramsDraft (L_draft) and verify (L_verify) share one weight set — no draft model to train or serve
Tree speculative decodingModel.generate_tree_speculative(...)Replaces K sequential verify passes with one mask-gated pass (Medusa/EAGLE alternative, zero aux heads)Same weights for all K branches → tree verify is correct; Medusa needs trained aux heads because its target has per-layer weights
Store-h KV cacheTINYLOOP_KV_H_MODE=1 [+ INT8 / INT4]Up to 78 % KV VRAM savingK = W_k · h, V = W_v · h at attention time; only works because W_k / W_v are stable across iterations
Iteration-axis delta-KVTINYLOOP_KV_DELTA=1 (shipped 2026-04-19)5–50× smaller KL than prior store-h INT8 SOTA, +2 pp top-1, faster — at matched storageΔh^l = h^l − h^(l-1) is only well-defined (same function, same scale) when every iteration shares weights
Self-consistency quality gatingModel.score_with_consistency(...)Escalate L only on disagreementRunning L=16 and L=32 on the same weights is architecturally meaningful; on a standard transformer the two are distinct-weight 16- vs 32-layer models
Hidden-state trajectory APIModel.score_trajectory(...)Research-grade measurement of how the residual evolves over iterationsThe iteration axis is a stable, semantically meaningful object — nothing to compare it to on a non-shared stack
Deterministic error cancellationTINYLOOP_CANCEL_ROUND=1O(1) quantization error vs O(L) naiveSame weight used L times → alternate rounding direction so errors cancel in pairs
L scheduling--l-warmup-tokens N --l-warmup-L M~3.3× faster warmup tokensDepth is a runtime knob; standard transformers must always run all layers
Per-iteration safety checksafety_check=TrueL abort chances vs 1Each iteration is a checkpoint; standard transformers only check after all layers
Dual-bit weight swapTINYLOOP_DUAL_BIT=13.2% faster decodeSame weight at different precisions per iteration; standard transformers have unique weights per layer
L-aware batchingper_lane_loops=[4,16,32]3 quality tiers, 1 GPUDepth is a per-request knob; standard transformers must deploy separate models

All of the above work without extra training, extra parameters, or per-iteration bespoke logic — they fall directly out of the architectural invariant.

Warm-start mid-loop​

Problem it solves. A user asks for a result at L=4 (fast, lower quality). They are unsatisfied and ask for L=8 on the same prompt. A naïve runtime re-runs all 8 iterations from scratch. TinyLoop runs only iterations 4..7, reusing the residual + K/V state from the first 4.

API.

h = model.build_resume_handle(prompt, L_used=4, max_loops=32)
out4 = model.resume_generate(h, new_L=4, max_tokens=64)   # L=4 answer
out8 = model.resume_generate(h, new_L=8, max_tokens=64)   # extend to L=8 without redoing iters 0–3
out16 = model.resume_generate(h, new_L=16, max_tokens=64) # extend to L=16 without redoing iters 0–7
model.free_resume_handle(h)

Measured win. On 407M GPTQ-INT4, resuming L=4 → L=8 saves 32.5 % wall-clock vs a fresh generate(loops=8). Greedy output is bit-exact to the fresh call. Break-even at N=1 follow-up (one second-opinion request pays for the handle).

Why only loops. Resuming at iteration k requires that the block used at iteration k+1 has the same weights it had during the original run. On a 24-layer standard transformer, layer 4 and layer 5 have different weights, so you cannot "resume from layer 4" without re-running layer 5's distinct computation. On our architecture layer 5's weights are layer 4's weights, so resuming is a null computation on the cache side — just a different pointer into the same weight tensors.

Pairs with. Self-consistency (try L=16 first, escalate to L=32 only on disagreement); speculative decoding; beam search.

Full spec: Python warm-start page.

Variable-L KV cache​

Problem it solves. A deployment needs to serve requests at different L (quality / latency) values from the same model artifact. A vLLM-style runtime would either fix L at compile time or serve three separate models.

API. Implicit in all TinyLoop calls. The KV cache is allocated with n_layers = n_pre + L_max. Any subsequent call with loops=L for L ≤ L_max reads only the first n_pre + L layers, treating the tail as unused. No reallocation, no recompilation.

Measured win. Free — the cache was going to be allocated anyway. What this unlocks is free-tier / pro-tier / enterprise-tier product differentiation from a single deployment. Serve L=4 to free users, L=16 to pro, L=32 to enterprise. One model, three SLAs. On a standard transformer you would deploy three distinct models.

Why only loops. Cache layer n_pre + i is filled by loop iteration i's K/V projection. Because that projection uses weights that are independent of L, the layer contents are valid at any future depth L ≥ i + 1. On a non-shared transformer the "layer at depth d" only exists in the 16-layer or 32-layer model, not both.

Full spec: tracked in §16.11 of the checklist (SLO-aware L admission, priority = iterations).

Self-speculative decoding​

Problem it solves. Speculative decoding with a draft model requires training and deploying a second model. Medusa trains auxiliary prediction heads on top of the target. EAGLE trains a feature-space prediction head. All variants add parameters and training cost.

TinyLoop's solution. Run the same weights at L_draft (shallow, fast) as the draft and at L_verify (deep, quality) as the verifier.

API.

out = model.generate_speculative(prompt, max_tokens=128,
                                   draft_loops=2, verify_loops=8,
                                   draft_ahead=4)

Measured win. 2.77× throughput on H100 / 407M GPTQ-INT4 vs plain generate(loops=8). Zero additional parameters. Drop-in usage.

Why only loops. Standard speculative decoding needs the draft and target to share token vocabulary and produce comparable distributions; in practice either (a) the draft is the target at low precision (a different runtime configuration), or (b) the draft is a separately-trained smaller model. Neither is as clean as "the draft is literally the target model after fewer iterations." The weight-shared loop makes L_draft < L_verify a runtime knob, not a training / deployment artefact.

Tree speculative decoding​

Problem it solves. Chain speculative decoding verifies K tokens sequentially — one full verify forward per drafted token. Medusa verifies a tree of K branches in one pass, but requires training auxiliary Medusa heads that predict the future distribution at multiple depths. EAGLE does similar with a feature predictor. Both add trainable parameters, so they cannot be "turned on" on an already-trained model.

TinyLoop's solution. Draft K token hypotheses (top-K sampling at each root, chain-extended via L_draft loop iterations) and verify the whole M = K_branches × depth tree in one mask-gated forward pass using the tree_verify_attention CUDA kernel. Zero auxiliary parameters.

API.

out = model.generate_tree_speculative(prompt, max_tokens=128,
                                       draft_loops=2, verify_loops=8,
                                       draft_ahead=4,          # depth per branch
                                       K_branches=2,            # >=1; 1=chain mode
                                       temperature=0.0)

Measured result. Greedy output is bit-exact to generate(temperature=0, loops=verify_loops); 36/36 tokens match on chain mode (K=1), 72/72 on multi-branch (K=2, K=4) across the test suite. Wall-clock speedup is workload-dependent (accept rate × draft cost); on a well-trained checkpoint multi-branch should beat chain spec substantially.

Why only loops. Medusa's tree is verified by the target model itself with the target's weights, but the draft branches were sampled by the aux heads, which were trained to predict "token at depth k given depth k-1." That training assumes the target has layer-k-specific weights. On our architecture the aux heads collapse — the draft is just the target at lower L, so the tree is correct by construction with zero extra training. This is the "zero aux params" claim: a Medusa-class speedup that does not require any extra training.

Full spec: paper/store_h_stress/ plus tree-spec Python API.

Store-h KV cache​

Problem it solves. KV cache dominates long-context VRAM. Standard techniques: quantise K/V to INT8/INT4 (lose accuracy), page with eviction (swap complexity), use GQA/MQA (changes the model). None of them compress below 2D bytes per token per layer.

TinyLoop's solution. Don't store K and V. Store the pre-QKV LayerNorm output h (a single [D] vector). At attention time reconstruct K = W_k · h and V = W_v · h from the same W_k, W_v the model is about to use anyway. Storage per token per layer drops from 2D bytes to D (FP16-h), D/2 + epsilon (INT8-h), or D/4 + epsilon (INT4-h) — 1×, 4×, or 8× compression.

API.

# Before Model() construction:
os.environ["TINYLOOP_KV_H_MODE"]              = "1"
os.environ["TINYLOOP_KV_INT4"]                = "1"   # or TINYLOOP_KV_INT8=1
os.environ["TINYLOOP_EXPERIMENTAL_FP16_BODY"] = "1"   # tensor-core reconstruction
import tinyloop_py as tl
model = tl.Model("model.tinyloop", max_seq_len=4096)

Measured win.

ModeCache savingDecode wall-clockQuality
FP16-h43–50 %−38 % with fp16_bodyBit-exact K/V
INT8-h66–75 %−37 % with fp16_body~0.6 % L2-rel K/V error; greedy-preserving on most prompts
INT4-h78 %−37 % with fp16_body~10 % L2-rel K/V error; eval-safe (+0.08 PPL at seq=4096, 10/10 MC agreement) but generate-divergent (argmax diverges within 1-3 tokens); see usage boundary

See KV Cache Modes for the full latency / memory / quality tables and the 2026-04-18 six-axis stress test results at paper/store_h_stress/PAPER.md.

Why only loops. The reconstruction K = W_k · h only gives the correct K for the cached position if the W_k used at cache-write time equals the W_k used at attention-read time. On a weight-shared loop this is trivially true — iter 5 writes its h, iter 6 reads it back and reconstructs with the same W_k. On a standard transformer the W_k that "wrote" layer 4's h is different from the W_k at layer 5; you cannot reconstruct across layers. Store-h compression is architecturally impossible on non-shared models.

Novel stress-test finding (Axis A of the 2026-04-18 suite). Loop depth L exhibits a stability sweet spot: INT8-h under greedy decoding preserves the baseline's output on 3/4 prompts at L=8-16, but only 2/4 at L=4 and L=32. Too few loop iterations cannot absorb reconstruction noise; too many accumulate it. To our knowledge this is the first reported measurement of a loop-depth-dependent error dynamic under KV compression — a question that is undefined on non-shared transformers.

Iteration-axis delta-KV​

Problem it solves. Store-h compresses the spatial axis of the KV cache by not materialising K and V separately. But there is a second, architecture-exclusive redundancy: across the L loop iterations, the per-token hidden states trace a contracting trajectory. On the 407M 2B-class model ρ_l = ‖h^l − h^(l-1)‖ / ‖h^l‖ decays 17× from 0.68 at the first iteration to 0.04 by iteration 32, and cos(h^l, h^(l-1)) → 1.000 — the residual barely moves late in the loop. Storing h^l naively at INT8 pays the same quantisation cost at every iteration even though the later iterations carry vanishingly small new information.

TinyLoop's solution. TINYLOOP_KV_DELTA=1 swaps the storage layout to:

h⁰  FP16  (shared anchor across all L loop iters per token)
Δh¹ Δh² … ΔhL   INT8 per-head (with [T, n_heads] FP16 scale grid)

Reconstruction at attention time: h^l = h⁰ + Σ_{i=1..l} scale[i, head] · Δh^i — one FP32 prefix-sum per token, fused into a single CUDA kernel launch. The paired write-side kernel computes Δh = h_curr − h_prev and picks the per-head scale at prefill time. The per-head grid matches the quantisation granularity of the existing pack_kv_row_int8 (store-h INT8) path, so delta-INT8 competes with store-h-INT8 at matched precision while paying strictly less total storage.

Why only loops. On a weight-shared loop, h^l and h^(l-1) are outputs of the same function applied to successively contracting residuals — the delta is small, structured, and quantisable at strictly finer granularity than the endpoints themselves. On a standard N-layer transformer h_layer_6 − h_layer_5 is a difference between outputs of two different functions with independent weights — it has no natural scale, no contraction structure, and quantising it produces noise rather than compression. The delta scheme is architecturally impossible on non-shared stacks for the same reason store-h is.

Measured — full-path PPL A/B with cache read/write active (RTX 4090 / 407M model, 20 prompts × 16 decode tokens per mode = 320 scored positions, 2026-04-19):

At L=8 (shallow loop):

ModePPLKL(fp16 ‖ q)top-1top-5wall-clock
fp163.7890100.00 %100.00 %96.3 ms
h_int8 (prior SOTA)3.7004.03 × 10⁻³96.88 %97.19 %147.2 ms
delta_int8 (per-head)3.7958.08 × 10⁻⁵99.06 %99.50 %135.4 ms

At L=32 (deep loop):

ModePPLKL(fp16 ‖ q)top-1top-5wall-clock
fp169.3060100.00 %100.00 %286.4 ms
h_int8 (prior SOTA)8.9624.71 × 10⁻³97.81 %96.25 %459.1 ms
delta_int8 (per-head)9.2929.08 × 10⁻⁴99.38 %97.75 %442.7 ms

Delta-INT8 per-head is strictly better than the prior store-h-INT8 compression scheme on every metric at every tested L value:

LKL ratio (delta / h_int8)top-1 gapwall-clock
81 / 50 (50× smaller)+2.18 pp7.9 % faster
321 / 5.2 (5.2× smaller)+1.57 pp3.6 % faster

Engineering status (2026-04-19, fully shipped).

LayerCommitStatus
Phase A simulation (reconstruction error, real trajectories)1a2b34e supporting data✓
Reconstruct CUDA kernel (per-row + per-head)1a2b34e, a7d52e6✓ parity on 4 shapes
Write-side CUDA kernel (per-row + per-head)5dd8ba5, a7d52e6✓ parity on 4 shapes
Storage layer (LayerKVCache / RuntimeKVCache)b429cc3✓ alloc/free verified, total_bytes exact
Logit-drift PPL proxy binding05689a7✓ 50-prompt L=8 / L=32 runs
Runtime wiring (prefill Δh-capture + attention reconstruct)f9dd9d3✓ generate() E2E validated
Full forward-path PPL A/B (per-row)4378f07✓ shows per-row baseline
Per-head kernel + runtimea7d52e6✓ closes L=32 gap, clean win at all L

Deployment story. Drop-in replacement for TINYLOOP_KV_H_MODE=1 + TINYLOOP_KV_INT8=1: set TINYLOOP_KV_DELTA=1 instead and get dramatically closer to FP16 quality (5–50× smaller KL, 1.5–2 pp higher top-1) at lower wall-clock. Same storage regime (one INT8 Δh per token per iter + per-head FP16 scales).

Self-consistency quality gating​

Problem it solves. A deployment wants to pay L=16 worth of compute by default, but escalate to L=32 on ambiguous inputs where the model is likely to get it wrong. On a standard transformer, "run my 16-layer model and my 32-layer model and compare" is two separate deployments.

TinyLoop's solution. Model.score_with_consistency(...) runs the shared-weight block at both depths in a single call and returns a consistency signal. The caller can route uncertain cases to the deeper setting or to a human reviewer.

API.

result = model.score_with_consistency(prompt, loops_a=16, loops_b=32)
if result["agree"]:
    answer = result["answer_a"]          # cheap L=16 was confident
else:
    answer = result["answer_b"]          # escalate to L=32 (costs ~2× more)

Measured. Shipped 2026-04-17. Integration examples in the Python API.

Why only loops. L=16 and L=32 on our architecture are the same model at two depths. On a standard transformer they would be two models, each with its own training curve, artefacts, and quality profile — not a free quality knob.

Hidden-state trajectory API​

Problem it solves. Research on how representations evolve across depth traditionally requires probing each layer of a standard transformer separately, noting each layer's unique weights. On a weight-shared loop the "iteration axis" is a more meaningful object: the same function applied repeatedly to an evolving residual.

TinyLoop's solution. Model.score_trajectory(tokens, n_loops) captures the last-position hidden state at every stage (embed, pre.0..pre.n-1, loop.0..loop.L-1, final_ln) in one forward pass, returning a [n_stages, D] tensor.

API.

result = model.score_trajectory(np.asarray(tokens, np.int32), n_loops=16)
print(result["labels"])   # ['embed', 'pre.0', 'pre.1', 'loop.0', ..., 'loop.15', 'final_ln']
hidden = result["hidden"] # [n_stages, D] float32

Used for. The β dynamics research (how ‖h^l - h^{l-1}‖ / ‖h^l‖ decays), per-iteration drift-coherence measurement, and the §16.14.a trajectory prerequisite for iteration-axis KV compression.

Why only loops. On a standard transformer "the hidden state at layer 7" and "the hidden state at layer 8" are snapshots inside two different sub-computations; the "trajectory" has 24 samples, one per distinct-weight layer. On our architecture the trajectory is a dynamical system: same operator, repeated. That invites specific measurements (fixed-point behaviour, contraction rates, β dynamics) that do not translate to the non-shared case.

Deterministic error cancellation​

Problem it solves. INT4 quantization introduces a per-weight error ε. In a standard transformer, each layer has different weights so errors are independent. In a looped transformer, the same weight error ε is applied L times — errors accumulate as O(L).

Solution. Alternate the rounding direction: even iterations use q_int (standard round-to-nearest), odd iterations use q_int + sign_correction. The sign correction flips the error direction, so consecutive pairs cancel.

Error scaling:

  • Naive INT4: O(L) — errors accumulate linearly
  • Stochastic rounding (TINYLOOP_CROSS_ITER_ROUND): O(√L) — random flips give statistical cancellation
  • Deterministic cancellation (TINYLOOP_CANCEL_ROUND): O(1) — paired cancellation regardless of L

Why only looped. Standard transformers use each weight once. There is no second iteration to cancel against.

API. TINYLOOP_CANCEL_ROUND=1 (requires v5 .tinyloop file with sign_bit channel).

L scheduling​

Problem it solves. Running all decode tokens at full depth L=8 is wasteful when the first few tokens (context warm-up) don't need high quality — the model hasn't built enough context to benefit from deep processing.

Solution. Ramp the loop depth during decode: first N tokens at warmup_L (e.g. L=2), then switch to full n_loops (e.g. L=8). Measured on H100 / 2B INT4: warmup tokens decode ~3.3× faster.

Why only looped. Standard transformers have no depth knob — every token must traverse all layers. In a looped transformer, depth is a runtime parameter with the same weights at every depth, so you can tune it per-token without any model changes.

API.

tinyloop model.tinyloop generate --loops 8 --l-warmup-tokens 16 --l-warmup-L 2
model.generate(prompt, loops=8, l_warmup_tokens=16, l_warmup_L=2)

Per-iteration safety check​

Problem it solves. Standard transformers run all layers before checking the output — one chance to catch a bad generation. In a looped transformer, each iteration produces an updated residual that can be inspected.

Solution. After each loop iteration, compute the residual's L2 norm (O(D), already in L2 cache). If it exceeds a threshold, abort immediately. The model gets L chances to catch exploding or degenerate hidden states vs 1 in a standard architecture.

Why only looped. Standard transformers have per-layer weights — there's no meaningful "checkpoint" between layers because the representation changes basis at each layer. In a looped transformer, every iteration operates in the same representational space, so the norm is a comparable metric across iterations.

API.

model.generate(prompt, loops=8,
               safety_check=True, safety_norm_threshold=50.0)

Measured. On H100 / 2B INT4: threshold=10 catches residual norm 30.7 at iter 0, aborting after 2 tokens. threshold=100k allows normal generation.

Dual-bit weight swap​

Problem it solves. INT4 quantization is a fixed precision across all loop iterations. But early iterations (building context) are less sensitive to quantization noise than late iterations (refining the output). Using INT4 everywhere wastes memory bandwidth on early iterations.

Solution. Keep both an INT2 copy (half the size) and the original INT4 copy of the loop block. Early iterations (0..switch_iter) use INT2, late iterations use INT4. The INT2 copy is created at load time by requantizing INT4→INT2.

Why only looped. Standard transformers have unique weights per layer — you can't requantize "layer 3's weights to INT2" without retraining. In a looped transformer, the loop block is one set of weights used L times, so one requantization produces a valid INT2 variant for all iterations.

Result. 3.2% decode speedup on H100 / 407M (408ms → 395ms). INT2 early iterations read half the bytes from HBM. Total extra VRAM: ~17 MB for the INT2 clone.

API. TINYLOOP_DUAL_BIT=1, TINYLOOP_DUAL_BIT_SWITCH=3 (default: first 4 iters use INT2).

L-aware batching​

Problem it solves. Serving different quality tiers (free, pro, enterprise) requires deploying separate models — 3 models × 2.7 GB = 8.1 GB VRAM. Wasteful when the only difference is "how much compute per token."

Solution. generate_batch accepts per_lane_loops=[4, 16, 32]. Each lane in the same batch runs at its own loop depth. Free tier gets L=4 (fast, lower quality), enterprise gets L=32 (slow, highest quality). One model, one GPU, one batch call.

Why only looped. Standard transformers have a fixed layer count — every request must run all layers. There is no "depth knob" to turn per request. In a looped transformer, depth is a runtime parameter with shared weights, so different requests can use different depths without any model changes.

Result. Verified bit-exact on H100 / 407M: mixed lane at L=4 produces identical output to standalone generate(loops=4), mixed lane at L=8 matches standalone generate(loops=8).

API.

model.generate_batch(
    [free_prompt, pro_prompt, enterprise_prompt],
    per_lane_loops=[4, 16, 32]
)

Not yet shipped but architecture-exclusive​

These capabilities are tracked in the roadmap and depend on the same weight-shared property:

  • Rewind speculation (§16.10). On a rejected draft, resume at a saved middle-L intermediate instead of restarting. Builds on warm-start + variable-L.
  • Iteration-dependent activation scales (§16.9). Store L per-iter scale factors per group (negligible storage).
  • Model editing amplification (§16.12). Edit one matrix in the loop block → effect applied L times.
  • Priority = iterations (§8). Free tier L=4, pro L=16, enterprise L=32 — one model, one GPU, three quality levels.
  • Priority = iterations (§16.11). Free tier L=4, pro L=16, enterprise L=32 — one artifact, three SLAs. Industry standard today is three deployed models.
  • HBM-free decode (§16.13). The loop block's INT4 weights (~35 MB on 1B-effective) fit in H100 L2 (50 MB). Iteration 0 reads from HBM; iterations 1..L-1 hit L2 and pay near-zero HBM traffic. Measured architecturally real but modest on H100 (~0.5 % wall-clock); projected 10–40× larger benefit on bandwidth-constrained devices (4090, Apple M3, mobile NPU). See investigations/hbm_free.md.

What this list is not​

These are architectural advantages that only exist because of weight sharing. They are not:

  • Kernel tricks that could be implemented on any runtime (FlashAttention, paged attention, chunked prefill — all of which TinyLoop also supports).
  • Quantisation schemes applicable to any model (INT4 GEMM, GPTQ, AWQ — also supported, but not architecture-exclusive).
  • Generation features like beam search / grammar-constrained decoding / deterministic sampling — standard across runtimes, included in TinyLoop for completeness.

If you see a feature on this page and a similar-sounding one in vLLM or TensorRT-LLM, the similarity is typically superficial: vLLM's paged attention is not the same primitive as TinyLoop's variable-L cache; llama.cpp's speculative decoding needs a separate draft checkpoint; neither supports store-h compression at all.

Further reading​