Skip to main content

Iteration-Dependent Activation Scales

What It Does​

In a looped transformer, the same weights are applied L times — but the activations change at each iteration. Early iterations have large activation magnitudes (building context), later iterations have smaller magnitudes (refining). Measuring these per-iteration scales enables better quantization decisions.

The activation magnitude drops ~50% from iteration 1 to iteration 4, then stabilizes.

Why This Matters​

Fixed quantization (same scale/zero for all iterations) is suboptimal because:

  • Early iterations: large activations → quantization error is a small fraction of the signal
  • Late iterations: small activations → same absolute quantization error is a larger fraction

Per-iteration scales let you adjust the effective dynamic range without re-quantizing weights.

How To Measure​

python tools/measure_iter_scales.py \
    --model model.tinyloop --L 8 --n-calib 20

Output:

Per-iteration activation scales (relative to iter 0):
  iter  1: 1.000 ████████████████████
  iter  2: 0.649 ████████████
  iter  3: 0.498 █████████
  iter  4: 0.449 ████████
  iter  5: 0.459 █████████
  iter  6: 0.486 █████████
  iter  7: 0.502 ██████████
  iter  8: 0.509 ██████████

Saves a JSON file with per-iteration scale factors for runtime use.

How It Connects To Other Features​

  • Dual-bit swap: the scale data shows early iterations can tolerate INT2 (activations are large enough to mask quantization noise)
  • Error cancellation: scale-aware rounding could weight the correction by activation magnitude
  • L scheduling: early iterations are not just cheaper (fewer FLOPs) but also less precision-sensitive

Why Only Looped​

Standard transformers have different weights AND different activations at each layer — you can't separate the two effects. In a looped transformer, weights are constant across iterations, so any change in activation magnitude is purely from the iterative refinement process. This makes per-iteration scaling a clean, interpretable intervention.