Rewind Speculation

The Problem

Self-speculative decoding uses L=2 to draft and L=8 to verify. When a draft is rejected, the standard approach is to discard the draft and accept only the verify result. But the verify at L=8 ran all 8 iterations — what if L=4 would have been sufficient?

The Insight

When L=2 is rejected by L=8, we can cheaply check whether an intermediate depth (L=4) would have agreed with L=8. If it does, the rejection point was between L=2 and L=4, and future verifies could potentially run at L=4 instead of L=8.

Measured Results

On H100 / 2B INT4 model:

Metric	Value
Rewind hit rate	88.9%
Rewind miss rate	11.1%
Draft acceptance rate	59.5%

This means: of the 40.5% of drafts that are rejected, 88.9% of those rejections happened between L=2 and L=4. The computation at L=5,6,7,8 added nothing.

How It Works

Implementation

1. During verify at L=8, the logit-lens is already computed at each iteration
2. On rejection, extract the argmax at --rewind-L (e.g. L=4)
3. Compare L=4 argmax with L=8 argmax
4. Log the hit/miss rate for telemetry

The check is essentially free — the hidden state at L=4 is already computed as part of running L=8.

Usage

# Enable rewind telemetry during speculative decode
tinyloop model.tinyloop speculate \
    --draft-loops 2 --verify-loops 8 \
    --draft-ahead 4 --rewind-L 4

Output includes:

rewind_hit_rate: 0.889
rewind_checks: 9
rewind_hits: 8

Depth Spectrum

Why This Is Architecture-Exclusive

In standard speculative decoding (e.g., Medusa, EAGLE), the draft and verify models are separate neural networks with different weights. There is no concept of an "intermediate depth" — you either run the draft model or the verify model.

In a looped transformer, L=2, L=4, and L=8 are all the same model at different depths. The rewind check at L=4 doesn't require a separate model, separate memory, or separate training. It's a natural consequence of the architecture.

System	Draft	Verify	Intermediate	Extra memory
Medusa	Medusa heads	Full model	None	+heads
EAGLE	EAGLE head	Full model	None	+head
SpecInfer	Small model	Large model	None	+model
TinyLoop rewind	L=2	L=8	L=4	0 bytes

Future: Adaptive Verify Depth

The rewind hit rate tells us the optimal verify depth. If 89% of rejections are caught by L=4, we could:

1. Start verify at L=4 instead of L=8
2. Only escalate to L=8 when L=4 disagrees with L=2
3. This would save ~50% of verify compute on the 89% of rejections

This is the path to fully adaptive self-speculative decoding: draft at L=2, verify at L=4, escalate to L=8 only when needed.