Testing and CI

TinyLoop validation currently has three layers:

1. CUDA-side unit tests
2. generation and parity regressions
3. self-hosted CUDA CI wiring

CTest Coverage

Current tests include:

Test	Purpose
tinyloop.test_int2_kernel	INT2 kernel correctness
tinyloop.test_ops	LayerNorm, residual, embedding, and fused op coverage
tinyloop.test_attention	Attention correctness including production-shape prefill check
tinyloop.test_kv_cache	KV-cache helper correctness
tinyloop.test_fp16_gemm	FP16 GEMM correctness
tinyloop.test_decode_parity	Hidden/logit parity between cached and uncached decode
tinyloop.test_generate_regression	Raw-byte CLI generation parity
tinyloop.test_generate_tokenizer_regression	Tokenizer-aware Python generation parity
tinyloop.test_generate_eval_slice	Small eval-slice parity coverage
tinyloop.test_speculative_greedy_regression	Speculative greedy parity
tinyloop.test_speculative_sampling_regression	Speculative sampling parity
tinyloop.test_prefix_cache_regression	Prefix-cache parity
tinyloop.test_int4_kernel	INT4 kernel correctness
tinyloop.test_rope	RoPE kernel parity (prefill, decode, V-slice invariance)
tinyloop.test_gqa	GQA attention (MHA parity, fewer KV heads, cached decode, cache append)
tinyloop.test_fused_ln_qkv	Fused LN+QKV kernel parity and norm output
tinyloop.test_variable_l_parity	Variable-L logit-lens parity at L=16
tinyloop.test_memory_leak	5-cycle load/unload GPU memory growth detection
tinyloop.test_fuzz	Random seq_len/L/tokens fuzzing (50 iterations)
tinyloop.test_kv_cache_regression	27-point cached-vs-uncached multi-token parity sweep
tinyloop.test_constrained_decoding	Regex/choice/JSON-shape grammar-constrained generation
tinyloop.test_paged_attention	Paged KV pool, page table, paged vs contiguous parity

Tooling:

Tool	Purpose
tools/perf_regression.py	Benchmark baseline/compare with regression threshold
tools/trace_to_chrome.py	CUDA profile → chrome://tracing JSON visualization
tools/trace_tokens.py	Token trajectory — argmax flips across loop iterations
tools/convert.py	Unified PyTorch → .tinyloop converter

Local Commands

Run the suite:

ctest --test-dir tinyloop/build --output-on-failure

Run with a model artifact:

TINYLOOP_TEST_MODEL_PATH=/absolute/path/to/model.tinyloop \
ctest --test-dir tinyloop/build --output-on-failure

Skip Semantics

Model-dependent tests use skip code 77 when prerequisites are missing.

That means:

• missing model artifacts do not poison the whole test run
• missing Python binding support does not turn optional tests into false failures

Python-Dependent Regressions

The tokenizer-aware and eval-oriented regressions require:

• pybind11
• Python module build support
• numpy
• transformers
• sometimes datasets

If those are unavailable, the tests skip cleanly.

Direct Regression Scripts

Run the raw-byte CLI regression:

TINYLOOP_TEST_BINARY=/path/to/tinyloop \
TINYLOOP_TEST_MODEL_PATH=/path/to/model.tinyloop \
python tinyloop/tests/test_generate_regression.py

Run tokenizer-aware parity:

TINYLOOP_TEST_BUILD_DIR=/path/to/tinyloop/build \
TINYLOOP_TEST_MODEL_PATH=/path/to/model.tinyloop \
python tinyloop/tests/test_generate_tokenizer_regression.py

Run prefix-cache parity:

TINYLOOP_TEST_BUILD_DIR=/path/to/tinyloop/build \
TINYLOOP_TEST_MODEL_PATH=/path/to/model.tinyloop \
python tinyloop/tests/test_prefix_cache_regression.py

Microbenchmarks

bench_hot_ops is not just a convenience binary. It is the fastest way to verify hot-path changes in:

• QKV projection
• attention output projection
• prefill attention
• cached decode attention
• SwiGLU
• down projection
• head projection

GitHub Actions

The repository includes a self-hosted CUDA workflow:

.github/workflows/cuda-ci.yml

It currently:

• installs Python dependencies
• configures the build
• builds the project
• runs CTest

What CI Still Does Not Do

Current CI does not yet provide:

• automated performance regression tracking
• multiple GPU-family coverage
• release artifact generation
• end-to-end serving tests

Those are still roadmap items, not present guarantees.