Model Conversion

TinyLoop consumes .tinyloop artifacts rather than raw PyTorch checkpoints.

The conversion and evaluation helpers currently live under tinyloop/tools/.

Main Tools

Tool	Purpose
tools/convert_pytorch.py	Convert a baseline checkpoint into .tinyloop
tools/gptq_convert.py	Experimental GPTQ-oriented conversion path
tools/eval_hellaswag.py	Quick evaluation helper for task-level sanity checks

Baseline Conversion Flow

python tinyloop/tools/convert_pytorch.py \
  --checkpoint model.pt \
  --output model.tinyloop \
  --bits 4 \
  --group-size 0 \
  --tie-weights

Important flags:

Flag	Meaning	Default
--checkpoint	Source checkpoint	required
--output	Output .tinyloop file	required
--factor-dim	SVD factor dim for embed/head. 0 (default) keeps the full rank, which is a lossless SVD round-trip. Aggressive values like 64 compress embed+head by ~32× but destroy vocab identity and produce incoherent generation — use only for engine smoke tests.	0 (full rank)
--bits	Quantization bit-width, currently 2 or 4	4
--group-size	Quantization group size, 0 means per-row	0
--tie-weights	Reuse compatible embedding/head weights	off

:::note File size vs. quality on a 407M checkpoint Measured 2026-04-17: at --factor-dim 64 the artifact is 114 MB (7.1× vs. FP16) but produces garbage on real prompts because the 50257-row embedding is crushed to rank 64. At --factor-dim 0 (default, full rank) the artifact is 529 MB (1.5× vs. FP16) and produces coherent English. Full-rank is the right default for quality; factor down only when you have a quality budget to trade. :::

Recommended Validation Workflow

After conversion:

1. run inspect
2. run a small benchmark
3. compare logits or outputs against a reference path
4. run a task-level sanity check such as eval_hellaswag.py

GPTQ Status

As of 2026-04-17, tools/gptq_convert.py has two capabilities beyond the original reference implementation:

Standard GPTQ — Cholesky stability fix

The old converter took cholesky(inv(H), upper=True), which NaN'd on ill-conditioned calibration Hessians and then silently retried with a hardcoded +0.1 I fallback. The current converter uses a dedicated _upper_chol_hinv() helper that:

• Computes the upper-triangular Cholesky of H⁻¹ via the stable cholesky(H) → cholesky_inverse → cholesky(upper=True) chain, avoiding the unstable intermediate torch.linalg.inv(H) call.
• Escalates damping by 10× per attempt up to 4 attempts before giving up.
• Raises a clear RuntimeError with damping, attempt count, and matrix shape on genuine failure, instead of silently retrying with a hardcoded constant.

This path is covered by tests/test_gptq.py — including a deliberately near-singular Hessian (rank-4 on a 64-dimensional space) that the damping escalation recovers from.

β-weighted GPTQ — paper-level novel contribution

The gptq_quantize() signature now accepts an optional beta= argument that implements the paper's β-weighted quantization idea. The argument has three valid forms:

beta value	Effect
None (default)	Exactly standard GPTQ — bit-identical to omitting the argument.
scalar float	Inflates percdamp by β². Larger β → more conservative rounding. Uniform H scaling alone is GPTQ-invariant (the Cholesky and the update both re-scale), so the damping hook is how scalar β actually moves the output.
1-D tensor of length K	Per-column sensitivities. Scales the effective Hessian H̃ = diag(β) H diag(β) before the Cholesky. High-β columns receive more careful rounding via standard GPTQ machinery.

The per-column form is the paper-level "β-aware quantization" path: you measure a β per weight matrix (or per column within a matrix), then feed it into gptq_quantize() so the converter spends its rounding budget where it matters most. The scalar form is a lower-effort hyperparameter knob for within-matrix conservatism.

Both forms are covered by tests/test_gptq.py: test_beta_none_matches_no_beta, test_beta_scalar_modulates_damping, test_beta_per_column_protects_high_beta_columns, and test_beta_bad_shape_raises. All tests run on CPU in under 100 ms.

Calibration-L matching (--calib-L)

The GPTQ converter now accepts --calib-L N, which controls the loop depth used during Hessian collection. Previously hardcoded to 8.

Why this matters: GPTQ's Hessian H = X^T X is collected from calibration activations at a specific loop depth. Deploying at a different loop depth means the quantizer sees mismatched activation statistics and misplaces its rounding budget. For a looped transformer trained with Poisson-L around 8, this effect is strongly asymmetric: below training L it dominates the quantization error budget; above training L the model itself drifts and calibration at L_train is preferred.

Measured on the 407M checkpoint at WikiText-103 val (30k tokens, group_size=128):

Deploy L	Naive INT4 Δ vs FP16	GPTQ calib=8 Δ	GPTQ calib=L (matched) Δ
1	+19.57	+38.84	+17.51
2	+15.19	+11.38	+6.95
4	+5.45	+2.96	+2.75
8	+3.91	+2.12	+2.12
16	+4.12	+2.47	+2.78 (slightly worse)
32	+6.10	+4.21	+4.36 (slightly worse)

Practical guidance:

• If you deploy at L ≤ L_train, set --calib-L equal to your deployment L.
• If you deploy at L > L_train, leave --calib-L at L_train — calibrating on the out-of-distribution activations of the drifting model makes things worse.

# Example: you plan to deploy at L=2 on a Poisson(mean=8)-trained model.
python tinyloop/tools/gptq_convert.py \
    --checkpoint model.safetensors \
    --output model_gptq_L2.tinyloop \
    --calib-L 2 \
    --n-calib-samples 64 \
    --seq-len 128

β measurement CLI

The β values consumed by β-weighted GPTQ are produced by the tinyloop-measure-beta console script (declared in pyproject.toml, backed by python/tinyloop_tools/measure_beta.py). It accepts either a raw PyTorch state_dict.pt or a HuggingFace-style .safetensors — prefixed HF keys are auto-remapped to the training-script layout.

End-to-end β-aware quantization flow:

# 1. Measure β on a raw checkpoint.
tinyloop-measure-beta \
    --model /path/to/model.safetensors \
    --calib-text /path/to/wikitext.txt \
    --output /tmp/betas.json \
    --L-values 1,2,4,8,16 \
    --trials 3

# 2. Feed the measured β values into GPTQ.
python tools/gptq_convert.py \
    --model /path/to/model.safetensors \
    --calib-text /path/to/wikitext.txt \
    --beta /tmp/betas.json \
    --bits 4 \
    --output model_int4.tinyloop

The output JSON shape is {"per_matrix": {"pre.0.attn_qkv": {"beta": 0.43, "intercept": 2.98}, …}} plus an arch and a measurement block recording the CLI flags for reproducibility.

tinyloop-measure-beta requires CUDA — the perturbation forward passes run on the GPU. For A100-class hardware set --precision fp16 (default); for fp32 debug runs use --precision fp32.

A development-path equivalent python tools/measure_beta.py ... is preserved as a thin shim around the same module so repo-relative scripts keep working.

Remaining open work

• Acceptance validation: "β-GPTQ beats vanilla GPTQ by ≥ 0.3 PPL at L=8" is the target in CHECKLIST.md §3 and is not yet measured on a real model. It requires the recovered 407M checkpoint and WikiText-103 calibration; both are pending pod availability.
• Mixed-precision bit allocation across matrices using measured β values is tracked as CHECKLIST.md §16.3 P1 — β-aware mixed precision and is still open.

Format Compatibility

The public runtime surface currently documents .tinyloop format versions 1 through 3.

If you change artifact semantics:

1. bump the format version intentionally
2. keep loader compatibility rules explicit
3. update conversion docs and regression coverage together

What Conversion Does Not Solve Automatically

Conversion is not a proof of quality.

Always validate:

• output parity or behavior sanity
• memory footprint
• benchmark behavior
• downstream task quality