Deterministic Error Cancellation

The Problem

When you quantize a weight W to INT4, each value picks up a rounding error ε. In a standard 24-layer transformer, each layer has its own weight — errors are independent, and the total impact is bounded by the sum.

In a looped transformer, the same quantized weight is applied L times. The same rounding error ε is applied L times. Total error grows linearly with L:

total_error ∝ L × ε

At L=8 this is manageable. At L=32 or L=64, it destroys output quality.

The Solution

Alternate the rounding direction across iterations. Even iterations use standard round-to-nearest. Odd iterations apply a sign correction that flips the rounding direction.

Each pair of iterations cancels: (+ε) + (−ε) = 0. Total error is O(1) regardless of L.

How It Works

The sign_bit channel

For each weight value, the quantizer records which direction it rounded:

q_int = round(w_float / scale + zero)
sign = +1 if q_int was rounded DOWN from the true value
sign = -1 if q_int was rounded UP

This 1-bit-per-weight channel is stored in the v5 .tinyloop format as sign_bit — packed 8 bits per byte, adding 12.5% storage overhead.

Per-iteration dispatch

The CUDA kernel int4_gemm_cancel_round reads both the packed INT4 data and the sign_bit channel. On odd iterations, it adds the sign correction (+1 or -1) to each quantized value before dequantization.

Error scaling comparison

Method	L=8	L=32	L=64	Growth
Naive INT4	8ε	32ε	64ε	O(L)
Stochastic rounding	2.8ε	5.7ε	8ε	O(√L)
Deterministic cancellation	0	0	0	O(1)

Usage

# Enable cancellation (requires v5 model with sign_bit)
TINYLOOP_CANCEL_ROUND=1 tinyloop model_v5.tinyloop generate \
    --prompt "Hello" --loops 32

# Add sign_bit channel to an existing model
python tools/tinyloop_add_sign_bit.py model.tinyloop

import tinyloop_py as tl
import os
os.environ["TINYLOOP_CANCEL_ROUND"] = "1"

model = tl.Model("model_v5.tinyloop")
# Now all loop iterations use cancellation rounding
tokens = model.generate(prompt, loops=32)

Requirements

Requirement	Details
Model format	v5 .tinyloop with sign_bit channel
Weight type	INT4 (per-row or per-group)
Architecture	Weight-shared looped transformer only
Runtime flag	TINYLOOP_CANCEL_ROUND=1

Why This Is Architecture-Exclusive

Standard transformers use each weight exactly once — there's no second application where a correction could cancel the first error. This technique is impossible on non-weight-shared architectures.

Relationship to Other Methods

Method	Storage	Error	Needs training?	Needs weight sharing?
AdaRound	0% overhead	Per-layer optimal	Calibration data	No
GPTQ	0% overhead	Per-layer optimal	Calibration data	No
Stochastic rounding	12.5% (sign_bit)	O(√L)	No	Yes
Deterministic cancel	12.5% (sign_bit)	O(1)	No	Yes

Note: GPTQ and cancellation are incompatible — GPTQ optimizes rounding decisions assuming each weight is used once. Per-iteration perturbation breaks GPTQ's column-to-column compensation chain.

Kernel Implementation

The int4_gemm_cancel_round kernel in cuda/int4_gemm_stoch.cu:

1. Reads the iteration index from g_loop_iter_index (thread-local)
2. On even iterations: standard INT4 dequant (q_eff = q_int)
3. On odd iterations: reads sign_bit[n, k/8], extracts bit for column k, applies correction (q_eff = q_int + sign)
4. Dequantizes and accumulates: w = (q_eff - zero) * scale

Cost: one extra byte read per 8 weights on odd iterations. Negligible vs the GEMM compute.