Skip to main content

Dual-Bit & Triple-Bit Weight Swap

The Insight​

In a looped transformer, early iterations refine a coarse representation — they don't need full precision. Late iterations fine-tune the final output — they benefit from higher precision. By dispatching different bit widths per iteration, we trade bandwidth for quality where it matters.

Early iterations have large activations that mask quantization noise. Late iterations have smaller activations where quantization noise is more visible.

Dual-Bit: INT2 + INT4​

At model load, the INT4 loop block is cloned and requantized to INT2:

Total extra VRAM: ~17 MB for the INT2 clone. LayerNorm weights are shared, not duplicated.

Triple-Bit: INT8 + INT2 + INT4​

Triple-bit adds an INT8 copy for iteration 0 — the highest-sensitivity iteration where the initial representation is formed.

Measured Results​

On H100 / 2B INT4, L=8, 16 decode tokens:

ConfigDecode (ms)Throughputvs baseline
Baseline (INT4 only)86.9184 tok/s—
Dual-bit (INT2+INT4)86.2186 tok/s−0.8% faster
Triple-bit (INT8+INT2+INT4)91.2175 tok/s+5% slower

The tradeoff:

  • Dual-bit is slightly faster because INT2 iterations read half the bytes
  • Triple-bit is slightly slower but gives highest quality at iter 0 where the initial hidden state is formed

Usage​

# Dual-bit: INT2 for iters 0-3, INT4 for iters 4-7
TINYLOOP_DUAL_BIT=1 TINYLOOP_DUAL_BIT_SWITCH=3 \
    tinyloop model.tinyloop generate --loops 8

# Triple-bit: INT8 for iter 0, INT2 for 1-3, INT4 for 4-7
TINYLOOP_DUAL_BIT=1 TINYLOOP_TRIPLE_BIT=1 \
    tinyloop model.tinyloop generate --loops 8
import os
os.environ["TINYLOOP_DUAL_BIT"] = "1"
os.environ["TINYLOOP_TRIPLE_BIT"] = "1"

model = tl.Model("model.tinyloop")
tokens = model.generate(prompt, loops=8)

Connection to β-Aware Mixed Precision​

The tools/beta_mixed_precision.py tool measures per-weight β sensitivity and recommends optimal dual/triple-bit settings:

# Measure β values
tinyloop-measure-beta --model weights.pt --output betas.json

# Get recommendations
python tools/beta_mixed_precision.py --betas betas.json --target-bpp 3.0

# Output:
#   TINYLOOP_DUAL_BIT=1 TINYLOOP_DUAL_BIT_SWITCH=3
#   → iters 0..3: INT2, iters 4+: INT4

Why This Is Architecture-Exclusive​

In a standard transformer, varying precision per layer requires separate quantized weight tensors for each layer at each bit width — 24 layers × 3 bit widths = 72 weight sets. In a looped transformer, you only need 2-3 copies of the single loop block, regardless of how many iterations you run.