Warm-Start Mid-Loop
What It Does
If you generated at L=4 and want to try L=8, TinyLoop resumes from the cached iter-4 state — only running iterations 5-8 instead of all 8 from scratch.
Cost: 8 iterations
Cost: 4 iterations (50% saved)
How It Works
build_resume_handle(prompt, L_used=4)— runs prefill at L=4, snapshots the residualresume_generate(handle, new_L=8)— restores the snapshot, runs only iters 5-8- KV cache layers 0-5 (2 pre + 4 loop) are reused; layers 6-9 are freshly computed
Measured
On H100 / 407M INT4, prefix=128, decode=16:
| Operation | Time |
|---|---|
Fresh generate(loops=8) | 329 ms |
build_resume_handle (one-time) | 6.4 ms |
resume_generate(new_L=8) | 222 ms |
| Saving | −32.5% |
Break-even at N=1 follow-up — the 6.4 ms build cost is paid back by the first resume.
Usage
handle = model.build_resume_handle(prompt, L_used=4, max_loops=32)
out_4 = model.resume_generate(handle, new_L=4, max_tokens=64)
# User wants better quality:
out_8 = model.resume_generate(handle, new_L=8, max_tokens=64)
# Even better:
out_16 = model.resume_generate(handle, new_L=16, max_tokens=64)
Why Only Looped
Resuming at iteration k requires the weights at iteration k to be identical to the weights at iteration 0. In a standard transformer, layer 5 has completely different weights than layer 1 — there's no meaningful "resume from layer 5 and continue with layers 5-8" because layers 5-8 in the standard model are a different set of weights than in the looped model.