CLI Reference

TinyLoop ships a single CLI binary:

tinyloop <model.tinyloop> <command> [options]

Current commands:

• inspect
• benchmark
• generate
• speculate

Global Form

./tinyloop/build/tinyloop /path/to/model.tinyloop <command> ...

The model path is always the first argument after the binary name.

inspect

Read metadata and memory estimates without loading full weights onto the GPU.

Usage

./tinyloop/build/tinyloop model.tinyloop inspect

What It Prints

• format version
• quantization bit-width
• dimensions and head count
• embedding mode
• weight size estimate
• buffer estimates at common sequence lengths
• KV-cache estimates
• compression comparison against a rough FP16 baseline

Use Cases

• quick artifact validation
• memory planning
• converter sanity checks

benchmark

Run repeated forward passes and report latency and throughput.

Usage

./tinyloop/build/tinyloop model.tinyloop benchmark \
  --loops 8 \
  --seq-len 128 \
  --repeat 10

Options

Flag	Meaning	Default
--loops	Runtime loop count	8
--seq-len	Synthetic token length	512
--repeat	Number of timed iterations	20

Notes

• this path benchmarks forward scoring, not generation throughput
• warmup runs are skipped when TINYLOOP_CUDA_PROFILE=1 is set
• the benchmark path uses dummy token ids and is meant for runtime measurement only

generate

Generate a continuation from a prompt.

Usage

./tinyloop/build/tinyloop model.tinyloop generate \
  --prompt "Looped transformers are" \
  --loops 8 \
  --max-tokens 32 \
  --temperature 0.8 \
  --top-k 50 \
  --cache-window 0 \
  --tokenizer /path/to/gpt2 \
  --repetition-penalty 1.3 \
  --stop "." \
  --stream

Options

Flag	Meaning	Default
--prompt	Input prompt string	"The meaning of life is"
--loops / --L	Loop count for generation (aliases)	8
--max-tokens	Number of new tokens to append	128
--temperature	Sampling temperature (0 = greedy)	0.8
--top-k	Top-k cutoff	50
--cache-window	Sliding-window KV size, 0 means full cache	0
--tokenizer	Directory containing vocab.json + merges.txt (GPT-2 BPE). Also read from TINYLOOP_TOKENIZER_DIR. Without it, prompt/output fall back to raw-byte encoding.	unset
--repetition-penalty	Standard HF-style penalty applied to logits of tokens already present in the running output. >1 suppresses repeats, 1.0 disables. Applied before temperature / top-k.	1.0
--stop TEXT	Stop generation as soon as the tail of the output matches the tokenization of TEXT. Repeatable; any match halts. Matching tokens are included in the returned output.	[]
--stream	Emit tokens as they are generated via generate_stream(). After completion prints a stats line with prefill_ms, first_token_ms, decode_steady_ms, tokens_emitted.	off
--early-exit-patience N	Enable confidence-based early exit in cached decode. Stop the loop iteration when the hidden state has been stable (relative L2 delta below --early-exit-tau) for N consecutive iters past --early-exit-min-L. 0 disables. Cache stays consistent across tokens via monotonic clamping (once reduced, L never goes back up within the generation).	0
--early-exit-min-L N	Minimum loop count before the stability check even starts considering early exit.	2
--early-exit-tau F	Relative-L2-delta threshold for calling a hidden state "stable". Lower is stricter. Current measurements on 1B: hidden rarely stabilizes tighter than 0.1, so small tau values effectively disable early exit.	1e-3
--early-exit-peek-every N	Only probe the stability check every N loop iterations past min_L. Larger values amortize the per-iter host-memcpy + norm cost. peek_every=4 cuts the EE overhead by ~4× on the 1B-effective benchmark (2.9% → 0.7%).	1
--early-exit-logit-margin F	Opt-in: when > 0, switches the stability metric from hidden-state L2 delta to top-1 vs top-2 logit margin. Stops the loop when the model is confidently predicting one token. Costs a full head projection per peek (combine with --early-exit-peek-every 2 to amortize). On 1B-effective, hidden-L2 mode doesn't fire reliably at any tau; logit-margin mode fires correctly and yields a real speedup.	0.0 (disabled)
--l-warmup-tokens N	L scheduling (architecture-exclusive): first N decode steps run at --l-warmup-L depth instead of --loops. Amortizes compute — easy initial tokens get cheap shallow passes, later tokens get full depth. Only possible because depth is a runtime knob with shared weights.	0 (disabled)
--l-warmup-L N	Loop depth for the warmup phase of L scheduling.	2

Current Behavior

• cached decode is enabled by default
• prompt text is BPE-encoded when --tokenizer is passed, raw-byte otherwise
• the runtime allocates max_seq = prompt_tokens + max_tokens

Example: Fixing Greedy Loop Degeneration

Without a repetition penalty, greedy decoding on a small model often locks into repetitive output:

# no penalty
Output: The cat sat on the floor and the cat sat on the floor.
        - The cat sat on the floor and the cat

Adding --repetition-penalty 1.3 turns it into coherent English:

# --repetition-penalty 1.3
Output: The cat sat on the floor and looked at me. I was very happy to
        see him, but he didn't know what to do with it. He had a

Combining with --stop "." produces a single-sentence completion:

# --repetition-penalty 1.3 --stop "."
Output: The cat sat on the floor and looked at me.

Example: Tuning Confidence-Based Early Exit

On a well-trained looped transformer, the hidden-state L2 delta rarely stabilizes below a useful threshold, so the default hidden-L2 early-exit rarely fires. The logit-margin mode is the working design — enable it by passing a positive --early-exit-logit-margin.

Measured on the 1B-effective INT4 artifact ("The meaning of life is", 100 generated tokens, L=8 base):

# baseline (no early exit)
decode=493 ms → "that it involves a lot more work than just the sum total cost."

# margin=3.0, patience=1, peek_every=2, min_L=4 — the measured sweet spot
--early-exit-patience 1 --early-exit-min-L 4 --early-exit-peek-every 2 \
--early-exit-logit-margin 3.0
decode=428 ms (13% faster) → "involves a lot of work, and that it can be done with little forethought."

More aggressive configurations (lower min_L, lower margin) fire more often and save more time but degrade quality — margin=0.5 patience=1 drops decode to 295 ms but produces repetitive / off-topic output. Conservative configurations (margin>=3.0, min_L>=L_train/2) fire rarely enough to keep the output near the baseline.

Debugging The Reference Path

To force uncached generation:

TINYLOOP_DISABLE_KV_CACHE=1 \
./tinyloop/build/tinyloop model.tinyloop generate --prompt "Hello"

To disable the newer tiled prefill attention path:

TINYLOOP_DISABLE_FLASH2_PREFILL=1 \
./tinyloop/build/tinyloop model.tinyloop generate --prompt "Hello"

speculate

Run self-speculative decoding with the same model used for both draft and verify passes.

Usage

./tinyloop/build/tinyloop model.tinyloop speculate \
  --prompt "Looped transformers are" \
  --draft-loops 2 \
  --verify-loops 8 \
  --draft-ahead 4 \
  --max-tokens 32 \
  --temperature 0.0 \
  --top-k 50 \
  --top-p 0.9 \
  --cache-window 0

Options

Flag	Meaning	Default
--prompt	Input prompt string	"The meaning of life is"
--draft-loops	Draft loop count	2
--verify-loops	Verify loop count	8
--draft-ahead	Number of drafted tokens per round	4
--max-tokens	Max generated tokens	50
--temperature	Sampling temperature	0.0
--top-k	Top-k cutoff	50
--top-p	Top-p cutoff	0.9
--cache-window	Sliding-window KV size	0

Environment Variables

Variable	Meaning
TINYLOOP_LOG_LEVEL=N	Runtime log verbosity. 0 silences all informational output (errors still print to stderr); 1 (default) keeps the model header, VRAM summary, and L2-residency line; 2 adds per-component load detail, buffer allocator notes, and the cache-warmup line. Values outside [0, 2] fall back to 1.
TINYLOOP_CUDA_PROFILE=1	Print phase timings during benchmark
TINYLOOP_DISABLE_KV_CACHE=1	Force uncached generation
TINYLOOP_EXPERIMENTAL_FP16_BODY=1	Dequantize the body weights to an FP16 cache at load and route every body GEMM through cuBLAS tensor cores (WMMA on Ampere, WGMMA on Hopper). Measured 2026-04-17 on H100 / 407M INT4: prefill 30-65× faster (e.g. seq=128 183→2.8 ms, seq=1024 1431→48 ms) and decode ~2.4× faster (1241→514 ms for 128 tokens). Weight VRAM grows ~2.9× (216 → 619 MB on 407M) — keep this flag off when VRAM is tight, on when it is not. Numerical output can diverge after a few tokens because FP16 tensor-core accumulation has a different rounding path than our INT4 scalar-FP32 reference; both outputs are valid samples from the quantized model.
TINYLOOP_KV_INT8=1	Store the KV cache as per-token per-head symmetric INT8 with one FP16 scale per (token, head). Measured 2026-04-17 on H100 / 407M INT4 at max_seq=2048: KV footprint drops from 167.8 MB → 85.2 MB (−49 %), so the same VRAM budget now holds ~4 k tokens of context instead of ~2 k. Greedy output is byte-identical with the FP16 KV path through the first ~40 tokens, then diverges as INT8 rounding accumulates — both continuations are valid samples. Decode regresses +3.6 % / +9.1 % / +13.6 % at seq ∈ 2048. Composes cleanly with TINYLOOP_EXPERIMENTAL_FP16_BODY=1.
TINYLOOP_CROSS_ITER_ROUND=1	Enable §16.9 cross-iteration stochastic rounding. The dispatcher routes INT4 GEMMs through int4_gemm_stoch_frac4 (preferred) or int4_gemm_stoch_1bit (fallback) when the loaded .tinyloop file is v5+ and has the matching sign-bit or 4-bit-frac channel on the weights. Each loop iteration draws a fresh PCG-hashed seed per matmul so the same weight gives a slightly different effective value across the L iterations — errors are zero-mean and accumulate as O(√L) rather than naïve INT4's O(L). Requires a v5 file produced by tools/tinyloop_add_sign_bit.py --from-pt PATH --loop-only [--frac-bits {1,4}]. Measured on 407M / WikiText-103 val: at L=16 stoch_frac4 beats naïve INT4 by −0.32 PPL, at L=32 by −0.67 PPL (paper §R.10). Off by default; zero cost to models without the sign-bit channel.
TINYLOOP_KV_H_MODE=1	§16.14.b — iteration-axis KV compression (store-h cache). When enabled, the runtime does not allocate per-layer K/V buffers; it allocates one FP16 h_cache per layer (stores the pre-QKV LayerNorm output) plus one shared RuntimeKVCache-level reconstruction scratch (K + V + QKV). At each cached-attention call the scratch is filled by running the block's attn_qkv over h_cache and splitting out K and V. K/V reconstruction is bit-exact against the FP16 KV baseline (verified by tests/test_kv_h_mode). Memory: 50 % KV saving asymptote (46 % at L=32). Latency on H100 / 1B-effective / L=8 / prefix=512 / decode=64: decode 2118 ms vs 675 ms baseline = +214 % — the reconstruction quant_gemm per cached-attention call is the dominant extra cost. This is a memory-for-compute trade; use when KV VRAM is the scarce resource. See the dedicated KV Cache Modes page for composability, memory math, and the full latency table. Sliding-window (current_len == max_seq_len) is supported end-to-end. Off by default.
TINYLOOP_KV_H_MODE=1 + TINYLOOP_KV_INT8=1	§16.14.b Phase 1D-INT8 — store h as per-head INT8 + FP16 scales. Composes the two env vars. Per-token per-layer footprint drops from 4096 bytes (FP16 h) to 2080 bytes at D=2048, n_heads=16 — ~66 % KV saving vs the FP16 KV baseline at L=32, max_seq=1024 (asymptotes toward 75 % at high L). Reconstruction quality: max_abs ≤ 0.014, L2_rel ≤ 0.006 against the bit-exact FP16 baseline K (same accept thresholds as the INT8 KV path). Combined with TINYLOOP_EXPERIMENTAL_FP16_BODY=1, decode is −37 % vs FP16 KV baseline on 1B-effective / H100. Use when memory is tight (edge / many-tenant) and ~0.6 % K/V reconstruction error is acceptable.
TINYLOOP_KV_H_MODE=1 + TINYLOOP_KV_INT4=1	§16.14.b Phase 1G-INT4 — store h as packed signed INT4 nibbles + FP16 per-head scales. Takes precedence over TINYLOOP_KV_INT8 for h storage (the more aggressive compression wins); TINYLOOP_KV_INT4=1 without TINYLOOP_KV_H_MODE=1 is silently ignored (there is no INT4 K/V path in TinyLoop today). Per-token per-layer footprint: 1056 bytes at D=2048, n_heads=16 (D/2 byte-packed nibbles + n_heads × 2 bytes of scales), giving 78 % KV saving vs FP16 KV at L=32, max_seq=1024 — 37 % smaller than INT8-h. Packing is two's-complement signed 4-bit in [-7, 7]; each byte holds an even-indexed element in its low nibble and an odd-indexed element in its high nibble. dim / n_heads must be even (the alloc path warns and falls back to FP16 h otherwise). Reconstruction quality on real 407M prefill: max_abs = 0.23, mean_rel = 0.12, L2_rel(K) = 0.11, L2_rel(V) = 0.10 — roughly 17× looser than INT8-h (reflects INT4's ~16× larger step size; 15 symmetric levels vs 255). Latency with TINYLOOP_EXPERIMENTAL_FP16_BODY=1: decode 425 ms on 1B-effective — matches INT8-h's 423 ms within noise while using ~37 % less KV. Recommended for long-context / VRAM-starved workloads where ~10 % K/V L2-rel noise is acceptable. Off by default.
TINYLOOP_KV_H_FUSED=1	§16.14.b Phase 1E — opt-in fused INT4-GEMM + INT8-h unpack kernel. When enabled and TINYLOOP_KV_H_MODE=1 TINYLOOP_KV_INT8=1 are also set and the attn_qkv weight is on the plain INT4 path (no fp16_cache, no sign-bit channel, no cross-iter rounding), the attention-time reconstruction routes through a single fused kernel that skips the FP16 unpack scratch and the Q third of the projection. Correctness-validated, currently performance-negative (~2× slower than the unfused path on H100 / INT4 / D=2048) because scalar INT8 byte loads underperform the simple INT4 GEMM's FP16-X cache pattern. Not yet wired for INT4-h — Phase 1H tracks the INT4-h fused variant (int4_gemm_fused_kv_from_int4_h). Kept opt-in for future vectorised-load + shared-memory-scale optimisation. Do not enable for latency-sensitive workloads. Off by default.
TINYLOOP_DISABLE_FLASH2_PREFILL=1	Disable the newer tiled prefill attention path
TINYLOOP_DISABLE_CUBLAS_FP16=1	Disable cuBLAS-backed FP16 GEMM
TINYLOOP_USE_CUDNN_SDPA=1	Route prefill attention (T > 1) through the cuDNN SDPA graph. Internally resolves to FlashAttention-3 on Hopper (SM90) and FA-2 on Ampere (SM80). Requires cuDNN 9+ at build time (auto-detected from the nvidia-cudnn-cu12 / nvidia-cudnn-frontend pip wheels). Silently falls back to attention_flash2 if plan build or execute fails. Off by default — end-to-end gain on 407M INT4 is marginal because INT4 GEMMs dominate; the win scales with attention's share of the forward (larger seq_len, BF16/FP8 bodies, etc.).
TINYLOOP_TOKEN_EXIT=N	Enable token-stable early-exit probing in scoring
TINYLOOP_TOKEN_EXIT_VERBOSE=1	Print early-exit diagnostics

What The CLI Is Not Yet

Current CLI limits:

• not a tokenizer-first interface
• not a server
• not a batching surface
• not a final compatibility contract for serving multiple checkpoint families