Dynamic-L Inference
The Core Idea
In a weight-shared looped transformer, L is not baked into the model. The same weights work at L=2, L=8, L=32, or L=64. This means:
- Quality is a compute knob, not a parameter-memory knob
- Different requests can run at different depths
- The same request can change depth mid-generation
No other production inference framework can do this.
Measured Performance Spectrum
On H100 / 2B INT4, 32-token prompt, 32 decode tokens:
| L | Decode (ms) | Throughput | Use case |
|---|---|---|---|
| 2 | 111.8 | 301 tok/s | Autocomplete, suggestions |
| 4 | 146.8 | 226 tok/s | Chat, fast responses |
| 8 | 161.3 | 198 tok/s | Production default |
| 16 | 305.5 | 108 tok/s | High-quality generation |
| 32 | 506.5 | 64 tok/s | Maximum quality, research |
All from the same 204 MB model file. No reloading, no separate models.
Features Built on Dynamic-L
L Scheduling (per-token depth ramp)
First N tokens at shallow depth, rest at full depth. Warmup tokens decode ~3.3× faster.
tinyloop model.tinyloop generate --loops 8 \
--l-warmup-tokens 16 --l-warmup-L 2
L-Aware Batching (per-request depth)
Different sequences at different depths in the same batch call.
results = model.generate_batch(
[free_prompt, pro_prompt, enterprise_prompt],
per_lane_loops=[4, 16, 32]
)
Self-Speculative Decoding (draft at low L, verify at high L)
L=2 drafts, L=8 verifies. Same weights, zero extra parameters.
SLO-Aware L Downgrade
When predicted wait exceeds SLO, scheduler reduces L instead of returning HTTP 503.
Adaptive L from Traffic
Off-peak: L=32 (maximum quality). Peak: L=4 (fast).
Dynamic-L Evaluation
tools/dynamic_l_eval.py tests model robustness across depths:
python tools/dynamic_l_eval.py --model model.tinyloop \
--L-values 2,4,8,16,32 --mean-L 8
Measures:
- Per-L agreement: how much does each L agree with the maximum-L reference?
- Poisson-L sampling: test with random depths per generation call
- Logit stability: at which iteration does the argmax lock in?
Measured on H100 / 2B INT4:
- L=4 agrees 100% with L=16 reference (converges early)
- Logit lens stabilizes at iteration 5 on average
Why This Is Architecture-Exclusive
In a standard transformer, depth is fixed at architecture time. Varying quality requires separate models at different sizes. TinyLoop makes depth a runtime parameter — the single most powerful consequence of weight sharing.