Rewind Speculation
The Problem
Self-speculative decoding uses L=2 to draft and L=8 to verify. When a draft is rejected, the standard approach is to discard the draft and accept only the verify result. But the verify at L=8 ran all 8 iterations — what if L=4 would have been sufficient?
The Insight
When L=2 is rejected by L=8, we can cheaply check whether an intermediate depth (L=4) would have agreed with L=8. If it does, the rejection point was between L=2 and L=4, and future verifies could potentially run at L=4 instead of L=8.
Measured Results
On H100 / 2B INT4 model:
| Metric | Value |
|---|---|
| Rewind hit rate | 88.9% |
| Rewind miss rate | 11.1% |
| Draft acceptance rate | 59.5% |
This means: of the 40.5% of drafts that are rejected, 88.9% of those rejections happened between L=2 and L=4. The computation at L=5,6,7,8 added nothing.
How It Works
Implementation
- During verify at L=8, the logit-lens is already computed at each iteration
- On rejection, extract the argmax at
--rewind-L(e.g. L=4) - Compare L=4 argmax with L=8 argmax
- Log the hit/miss rate for telemetry
The check is essentially free — the hidden state at L=4 is already computed as part of running L=8.
Usage
# Enable rewind telemetry during speculative decode
tinyloop model.tinyloop speculate \
--draft-loops 2 --verify-loops 8 \
--draft-ahead 4 --rewind-L 4
Output includes:
rewind_hit_rate: 0.889
rewind_checks: 9
rewind_hits: 8
Depth Spectrum
Why This Is Architecture-Exclusive
In standard speculative decoding (e.g., Medusa, EAGLE), the draft and verify models are separate neural networks with different weights. There is no concept of an "intermediate depth" — you either run the draft model or the verify model.
In a looped transformer, L=2, L=4, and L=8 are all the same model at different depths. The rewind check at L=4 doesn't require a separate model, separate memory, or separate training. It's a natural consequence of the architecture.
| System | Draft | Verify | Intermediate | Extra memory |
|---|---|---|---|---|
| Medusa | Medusa heads | Full model | None | +heads |
| EAGLE | EAGLE head | Full model | None | +head |
| SpecInfer | Small model | Large model | None | +model |
| TinyLoop rewind | L=2 | L=8 | L=4 | 0 bytes |
Future: Adaptive Verify Depth
The rewind hit rate tells us the optimal verify depth. If 89% of rejections are caught by L=4, we could:
- Start verify at L=4 instead of L=8
- Only escalate to L=8 when L=4 disagrees with L=2
- This would save ~50% of verify compute on the 89% of rejections
This is the path to fully adaptive self-speculative decoding: draft at L=2, verify at L=4, escalate to L=8 only when needed.