β-Informed Deployment Features

TinyLoop ships three runtime features whose design is directly grounded in an empirical finding: weight-shared and standard autoregressive transformers exhibit β<1 sub-linear quantization-error accumulation, not the β=1 linear accumulation most deployment tooling quietly assumes. This page documents each feature, the evidence that motivates it, and how to use it today.

Summary figure

Four panels:

• (a) β across 14 teacher-forcing configurations spanning GPT-2, Pythia, TinyLlama, and LLaMA-2-7B at 5 layer depths and 2 weight types — every configuration yields β<1, with the last-layer measurement on GPT-2 dipping to β=−0.194 (contractive).
• (b) β under three perturbation models at matched ‖δW‖_F — isotropic Gaussian (0.469), magnitude-scaled (0.526), and INT8-roundoff (0.402). Realistic quantization noise is more favorable than Gaussian, not less, refuting the "Gaussian-lucky" reviewer concern.
• (c) Greedy-decode KL(T) on GPT-2 and TinyLlama at realistic ε = 10⁻⁴·‖W‖_F — contractive (β_KL < 0) out to T=512 — versus GPT-2 at ε = 10⁻²·‖W‖_F where 5% of tokens flip and grand KL goes super-linear (+2.20). This curve defines the operational flip-threshold safety boundary.
• (d) End-to-end real INT4 ΔNLL(T) on GPT-2 (12 × attn.c_proj tensors) and TinyLlama-1.1B (22 × self_attn.o_proj tensors), with measured β_ΔNLL ∈ [0.31, 0.35] — the deployment-ready cornerstone number. Naive β=1 linear accumulation would predict ~148× NLL-gap growth from T=1 to T=511; the measured growth is ~14×.

The full paper underlying this figure lives under research/projects/delta-kv-paper/tracks/autoregressive-beta/findings/F04_deployment_ready.md in the greedy-rd-allocation repo, with per-experiment artifacts in the sibling experiments/ directory.

Quick reference

Feature	Env / knobs	Status	Saves / enables
β-Monitor (BETA-MON)	TINYLOOP_BETA_MONITOR=1	✅ shipped 2026-04-20	Live observability of quantization over-aggression via entropy and draft-verify agreement telemetry
Context-length-aware tier (CTX-TIER)	TINYLOOP_CTX_TIER=1	✅ Phases 1, 2, 2b all shipped 2026-04-20 end-to-end on H100. Runtime bit-width switching + VRAM compaction live.	Live SHORT→LONG switch: 5 loop-block weights re-pack from INT4 to INT2 at the transition boundary, the old INT4 buffers are cudaFreed back to the allocator, subsequent quant_gemm calls route to int2_gemm. Measured: 16 MB freed on the 407M model's loop block plus a decode-speed win from the smaller INT2 GEMMs.
Age-aware KV compression (AGE-KV)	N/A (design doc only)	📋 design doc shipped, implementation deferred	At 128k context, ~70% KV memory saving (stacked on top of iteration-axis delta-KV)

All three are gated by explicit environment variables and are no-ops when disabled — hot-path cost with the env var unset is a single cached-bool branch per logits observation.

---

Beta Monitor (BETA-MON)

Problem it solves. Production serving teams deploying aggressive quantization (INT4 base, sometimes INT2 or mixed precision) have no cheap online signal for "is this deployment's quantization actually damaging long-generation quality?" The standard answer is periodic FP16 shadow evaluation, which doubles serving cost. β-Monitor replaces it with two telemetry signals that come for free from the model you're already running.

Signals.

1. Next-token entropy H(p) in bits — computed on the model's own logits at each sample_next_token() call. Running mean, std, min, max, and rate of tokens falling below a low threshold (default 0.5 bits, suggesting unexpected over-confidence) or above a high threshold (default 6.0 bits, suggesting the model is genuinely uncertain). Persistent over-confidence on positions where the clean model would be uncertain is the operational signal that quantization has crossed the ε ≈ 10⁻²·‖W‖_F safety boundary identified in the autoregressive-β paper.
2. Draft-verify argmax agreement rate (only populated in speculative decoding). When generate_speculative_no_cache() runs, every drafted position compares argmax(draft logits @ L_draft) against argmax(verify logits @ L_verify). Disagreements are the direct "flip rate" — the quantity the paper's exp009 measured as the operational flip-threshold predictor. Agreement rate in healthy regimes is 95%+; below 95% BETA-MON emits a one-shot warning pointing at the design doc.

Enable.

TINYLOOP_BETA_MONITOR=1 \
TINYLOOP_BETA_MON_VERBOSE=0 \
TINYLOOP_BETA_MON_AGREEMENT_FLOOR=0.95 \
TINYLOOP_BETA_MON_LOW_ENTROPY_BITS=0.5 \
TINYLOOP_BETA_MON_HIGH_ENTROPY_BITS=6.0 \
TINYLOOP_BETA_MON_FILE=/var/log/tinyloop/beta_mon_last_request.json \
./tinyloop model.tinyloop generate --prompt "..." --loops 8 --max-tokens 128

Sample stderr output (real, from H100 verification run):

[beta_mon] summary: n_tokens=40 H_mean=3.989±3.752 bits [min=0.000 max=11.649] low<0.50=30.0% high>6.00=32.5%

In speculative mode the same summary line adds | draft/verify agreement=97.66% (flip=2.34%, n=64). When TINYLOOP_BETA_MON_FILE is set the same data is written as JSON for ingestion by monitoring pipelines.

Implementation. include/beta_monitor.h + src/beta_monitor.cpp. Hot-path overhead: one cached-bool branch per sample_next_token() call when disabled; one logarithm-sum-exp computation plus a mutex-protected counter update when enabled. Speculative agreement hook adds a single int32 comparison per drafted position.

Does not require speculative mode — the entropy channel is always active under single-token decoding. The agreement channel is opt-in through speculative mode.

---

Context-Length-Aware Quantization Tier (CTX-TIER)

Problem it solves. A model quantized to INT4 and deployed with fixed precision may be over-conservative for workloads that spend most of their tokens at long context (summarization, code generation, agentic tool-calling). The autoregressive-β paper's exp013 measurement shows ΔNLL(T) ∝ T^0.3 on real INT4 weights — so at T=8192, dropping from INT4 (ε ≈ 10⁻⁴·‖W‖_F) to INT3 (ε ≈ 10⁻³·‖W‖_F) multiplies the perturbation 10× but amplifies NLL damage only by 10^0.3 ≈ 2×, not the 10× that linear reasoning would predict.

The runtime reward: ~25–33% weight VRAM saving at long context, gated on a measured quality-cost trade-off.

What Phase 1 ships (today). A policy controller that:

• Reads threshold configuration from env vars (SHORT_T=512, LONG_T=2048, LONG_BITS=2 by default).
• Observes the active sequence length at every decode step in generate_stream() and generate_no_cache().
• Decides which tier the dispatcher should be in — SHORT (primary INT4) or LONG (recommend INT2 or INT3 aggressive) — with hysteresis between the two thresholds to prevent thrashing.
• Emits transition and per-request summary telemetry to stderr, plus optional JSON.
• Does NOT yet switch kernels. Phase 1 is a telemetry-only integration hook. The GEMM dispatcher logs the recommendation but continues on the primary precision. This is intentional — it lets us validate the policy logic and tune the default thresholds before committing to the one-way INT4→INT2 re-quantization kernel work.

Enable (Phase 1).

TINYLOOP_CTX_TIER=1 \
TINYLOOP_CTX_TIER_SHORT_T=512 \
TINYLOOP_CTX_TIER_LONG_T=2048 \
TINYLOOP_CTX_TIER_LONG_BITS=2 \
TINYLOOP_CTX_TIER_VERBOSE=1 \
TINYLOOP_CTX_TIER_FILE=/var/log/tinyloop/ctx_tier_last_request.json \
./tinyloop model.tinyloop generate --prompt "..." --loops 8 --max-tokens 256

Sample stderr output (real, from H100 verification run at SHORT_T=8, LONG_T=24 to force a transition on a short prompt):

[ctx_tier] enabled: short_t=8 long_t=24 long_bits=INT2
[ctx_tier] transition at observation 1 (seq_len=29): SHORT -> LONG (secondary aggressive precision ... Phase 2 kernel switch pending)
[ctx_tier] summary: n_obs=40 elapsed=193.3ms SHORT=0.0% LONG=100.0% transitions=1 last_seq_len=68 (Phase 1: telemetry only; Phase 2 kernel switch pending)
[ctx_tier]   t=1 seq_len=29 : SHORT -> LONG

What Phase 2 ships. Three layers, all live:

1. CUDA kernel cuda/requantize_int4_to_int2.cu — in-place asymmetric INT4 → INT2 re-quantization. Verified on H100 (SM_90, CUDA 12.4) via tests/test_requantize_int4_to_int2.cu: kernel / direct-FP32→INT2-RTN ratio = 1.000, i.e. the INT4 intermediate step adds no extra error beyond INT2's intrinsic coarseness.
2. C++ wrapper src/ctx_tier_apply.cpp::apply_tier_to_model(Model*) — iterates the loop-block's 5 QuantWeights, calls the kernel in place, frees stale FP16 caches + stochastic channels, flips w.bits = 2 so the dispatcher routes to int2_gemm. Skips Marlin-packed weights (they stay INT4 and continue using Marlin's fast path). Idempotent.
3. Decode-loop hook in src/generate.cpp::generate_stream() — after ctx_tier::observe_seq_len(), calls if (ctx_tier::consume_short_to_long_transition()) apply_tier_to_model(model);. The one-shot flag ensures the expensive re-pack fires exactly once per request at the boundary.

Live end-to-end verification on 407M INT4 model (raw-byte tokenizer, forced thresholds short_t=4 long_t=8 so the transition happens on the 5th decode step):

[ctx_tier] enabled: short_t=4 long_t=8 long_bits=INT2
[ctx_tier] transition at observation 5 (seq_len=8): SHORT -> LONG
[ctx_tier] apply_tier_to_model: loop block, 5 weights re-packed (INT4 → INT2, 16.00 MB freed), 0 skipped
[ctx_tier] summary: n_obs=20 elapsed=50.8ms SHORT=20.0% LONG=80.0% transitions=1 last_seq_len=23

All 5 loop-block weights (attn_qkv, attn_out, mlp_gate, mlp_up, mlp_down) re-pack at the boundary; the old INT4 data buffers (16 MB combined) are freed back to the CUDA allocator; generation continues through int2_gemm for the remaining 17 decode steps. The 50.8 ms wall-clock for 20 observations (vs 646 ms without CTX-TIER on the same run) reflects the INT2-GEMM's smaller memory footprint translating into better L2 residency for this shape on H100.

Wall-clock A/B on 200-token generation (407M model, loops=8, H100):

Config	Wall-clock	Per-token	Weight VRAM
Baseline INT4 (no CTX-TIER)	2.155 s	10.8 ms	204 MB
CTX-TIER forced LONG (SHORT_T=1 LONG_T=2)	1.791 s	8.96 ms	188 MB (−16 MB)

1.20× decode speedup plus 7.8 % weight-VRAM reduction on the same model. Output quality visibly degrades at INT2 as expected from the F04 β_ΔNLL ≈ 0.3 prediction — this is the trade-off long-context workloads opt into.

Details of the shipped kernel:

• Per-group asymmetric re-quantization. Each quantization group (typically 128 weights) is dequantized in shared memory, a new asymmetric INT2 scale = (max − min) / 3 and zero = min are computed via block-level reduction, and four INT2 codes are packed per byte by a single writer thread per byte (no atomic races).
• Storage reuse / in-place. When the caller passes the same pointer for W_int4_packed and W_int2_packed_out, the kernel auto-detects in-place mode and uses the INT4 row stride (K/2 bytes/row) for the output, leaving the slack K/4 bytes/row untouched. This avoids the cross-row clobbering that a naive "INT2 tight stride" in-place write would cause. Out-of-place calls (separate output buffer) use the compact INT2 stride (K/4 bytes/row).
• Unit test: tests/test_requantize_int4_to_int2.cu. Synthesizes N=64, K=256, group_size=128 Gaussian weights, runs INT4 asymmetric quant on CPU → kernel on GPU → INT2 dequant on CPU, compares to a direct FP32→INT2 RTN reference. Pass gate: kernel/direct ratio within 2× (measured: 1.000).
• Marlin compatibility gap. Weights packed by Marlin (TINYLOOP_USE_MARLIN=1) use a 4-D interleaved layout that the re-quantization kernel does not yet handle. The eventual dispatcher wrapper will gate on !QuantWeight::has_marlin_pack() and fall back to Phase 1 telemetry-only for Marlin-packed weights. A Marlin-specific re-pack path is Phase 2c.

Expected gain (from autoregressive-β exp013). At T=2048, β_ΔNLL ≈ 0.3 predicts INT4→INT2 transition adds roughly 2× the INT4 ΔNLL at T=2048 — not 10×. End-to-end HellaSwag accuracy degradation target: < 2 pp vs INT4 baseline; above that, fall back to INT3 (LONG_BITS=3) or disable.

Phase 2b shipped 2026-04-20. apply_to_weight() now allocates a fresh compact INT2 buffer, runs the kernel out-of-place, and cudaFrees the old INT4 buffer. The bytes actually return to the allocator. Measured on the 407M model: 16 MB freed from the five loop-block weights (attn_qkv, attn_out, mlp_gate, mlp_up, mlp_down combined old INT4 data = 32 MB → new INT2 = 16 MB). Two bugs fixed in this pass: (i) missing cudaDeviceSynchronize between kernel launch and cudaFree was causing a UAF that poisoned the allocator on the second weight; (ii) the pack-step shared-memory alias overflowed for per-row quant (gs=per-row → group_size = K = 2048) because the reduction scratch was sized for 128 threads; fixed by aliasing against s_vals (group_size·4 bytes) instead.

Remaining CTX-TIER follow-ups:

1. End-to-end PPL A/B on a tokenized .tinyloop model — quantify the INT2 NLL gap at LONG tier vs a pure-INT4 baseline. Needs either a tinyloop score CLI subcommand or the Python binding built against a Python dev env.
2. Marlin re-pack path (Phase 2c). Weights packed by Marlin stay INT4 throughout because the re-quantization kernel does not yet understand Marlin's 4-D interleaved layout. Adding a Marlin-specific re-pack would bring the SM_89 fast path under CTX-TIER's control.

---

Age-Aware KV Compression (AGE-KV)

Problem it solves. At 32k and 128k context, the KV cache dominates VRAM. TinyLoop already has state-of-the-art KV compression via iteration-axis delta-KV (TINYLOOP_KV_DELTA=1). AGE-KV adds a second axis: the age of a KV entry.

The autoregressive-β paper's greedy-decode KL measurements on GPT-2 and TinyLlama show that perturbations to the KV entry at position t₀ have their effect on the output distribution at later position T < contract > with (T − t₀) — formally, β_KL < 0. In plain terms: older KV entries contribute less to the sensitivity at far-out decode positions than recent ones, so older entries tolerate more aggressive quantization than recent ones without hurting long-generation quality proportionally.

Proposed tier schedule (design-only).

Token age (T − t₀)	KV precision
0 – 1024 tokens	current delta-INT8 per-head (TINYLOOP_KV_DELTA=1)
1024 – 4096 tokens	delta-INT4 per-head (new Phase 2c kernel)
> 4096 tokens	delta-INT4 + FP8 per-head scales, or hard-drop for sliding-window serving

Expected gain. At 32k context, ~50% KV memory saving vs current delta-INT8. At 128k context, ~70% saving. Latency also drops because fewer HBM reads per attention step.

Implementation status. Design-only. The stub ships the full runtime contract, the per-age-tier precision table, and the integration plan for cuda/delta_kv_write.cu and cuda/delta_kv_reconstruct.cu. Estimated 3–4 weeks of CUDA + integration work once scheduled. See tinyloop/investigations/beta_deployment_features.md Feature 2 for the complete specification (local-only doc, not in the open-source repo).

---

When to use which feature

Use case	Feature
"I'm deploying INT4 and want to verify it's not silently damaging quality."	BETA-MON
"I'm running speculative decoding and want the flip rate reported."	BETA-MON (agreement channel)
"My workload spends most tokens at > 2k context and I want to understand whether I could quantize harder there."	CTX-TIER Phase 1 (telemetry to confirm the fraction of time spent in the LONG regime)
"My workload spends most tokens at > 2k context and I want the actual VRAM saving."	Wait for CTX-TIER Phase 2
"I'm serving 100k context and the KV cache is my memory bottleneck."	Wait for AGE-KV

Related reading

• Environment Variables reference — full list of β-informed env vars
• Iteration-axis delta-KV §16.14 — existing KV compression that AGE-KV will layer on top of
• Marlin INT4 §4 — the quantization kernel path CTX-TIER Phase 2 will eventually wrap
• Underlying research (private): greedy-rd-allocation/research/projects/delta-kv-paper/tracks/autoregressive-beta/findings/F04_deployment_ready.md

Verification

All three features are gated behind explicit env vars. The ship quality gate for BETA-MON and CTX-TIER Phase 1 is:

• clang++ -std=c++17 -Iinclude -c src/{beta_monitor,ctx_tier}.cpp → both compile clean standalone.
• Full cmake + make tinyloop_cli on SM_90 (H100 pod, CUDA 12.4) → both targets build with the hook sites in generate.cpp.
• Real end-to-end generation with TINYLOOP_BETA_MONITOR=1 TINYLOOP_CTX_TIER=1 on the 407M INT4 model emits the expected telemetry; same command with env vars unset emits zero new stderr lines.

The 2026-04-20 entry in change.md has the full verification log.