L Scheduling

What It Does

During a single decode call, vary the loop depth per token. First N tokens run at a shallow depth (cheap warmup), remaining tokens run at full depth (quality output).

How It Works

The implementation is in the decode loop of generate_stream. Before each decode step, the effective loop count is set based on the step index:

step_loops = (step < warmup_tokens) ? warmup_L : n_loops

The KV cache handles variable L naturally — positions written at L=2 only populate cache layers 0-3 (2 pre + 2 loop), while positions at L=8 populate layers 0-9.

Measured

On H100 / 2B INT4, 64-token decode:

Config	Time per warmup token	Time per full token
L=2	~1.7 ms	—
L=8	—	~6.2 ms
L scheduling (16 warmup at L=2, rest at L=8)	~1.7 ms	~6.2 ms

Warmup tokens are ~3.3× faster than full-depth tokens.

Usage

tinyloop model.tinyloop generate --loops 8 \
    --l-warmup-tokens 16 --l-warmup-L 2 \
    --prompt "..." --max-tokens 64

model.generate(prompt, loops=8,
               l_warmup_tokens=16, l_warmup_L=2)

Why Only Looped

Standard transformers have a fixed number of layers — every token must run all of them. In a looped transformer, depth is a runtime parameter with the same weights at every depth, so you can tune compute per-token without model changes.