본문으로 건너뛰기

Self-Speculative Decoding

What It Does​

Uses the same model at two different loop depths — L=2 for fast drafting, L=8 for quality verification. No separate draft model needed.

How It Works​

  1. Draft phase: Run the loop block 2 times per token (fast). Generate K tokens greedily.
  2. Verify phase: Run the loop block 8 times on the full draft sequence. Compare argmax at each position.
  3. Accept: Matching tokens are kept. First mismatch is replaced with the verify token.
  4. Bonus: If all K tokens match, sample one bonus token from the verify logits.

Measured​

On H100 / 407M, L_draft=2, L_verify=8:

  • 2.77× throughput improvement
  • Zero extra parameters — same weights for both depths
  • Greedy output is bit-exact to generate(loops=8)

Usage​

tinyloop model.tinyloop speculate \
    --draft-loops 2 --verify-loops 8 --draft-ahead 4
model.generate_speculative(prompt,
    draft_loops=2, verify_loops=8, draft_ahead=4)

Why Only Looped​

Standard speculative decoding (Leviathan et al., 2023) needs a separate smaller draft model — trained independently, deployed alongside the main model, consuming extra VRAM. TinyLoop uses the same weights at a lower depth. The "draft model" is just the main model run fewer times.