FP16 Body Cache & Kernel Fusion
FP16 body cache
Promote quantised body weights to FP16 caches at load time. Trades ~2.9× weight VRAM for large speedup.
| Variable | Value | Effect |
|---|
TINYLOOP_EXPERIMENTAL_FP16_BODY | 1 | Prefill 30-65× faster, decode ~2.4× faster. Weight VRAM 216 → 619 MB on 407M. |
TINYLOOP_H_MODE_FP16_ATTN_QKV | 1 | Keep attn_qkv FP16 copy resident under h-mode. |
TINYLOOP_H_MODE_FP16_ATTN_QKV_EPHEMERAL | 1 | Materialise FP16 attn_qkv only during attention, free after. |
FP16 residual stream
| Variable | Value | Effect |
|---|
TINYLOOP_FP16_RESIDUAL | 1 | Use FP16 buf.main instead of FP32. Halves residual bandwidth. All paths wired: decode, prefill, chunked prefill, logit-lens, speculative, safety check. Verified on H100. |
The residual stream (buf.main) is the persistent hidden state that carries data across all L loop iterations. In FP32 mode it's 4 bytes/element; FP16 mode cuts that to 2 bytes/element. The numerical impact is minimal at L≤16 because each iteration's contribution is computed in FP32 (via cuBLAS) and only the accumulation is FP16.
Fused LN+QKV
| Variable | Value | Effect |
|---|
TINYLOOP_FUSED_LN_QKV | 1 | Fused LN + INT4 QKV GEMM for decode. Experimental — currently 6% regression. INT4 per-row only. |