Store-h KV Cache
What It Does
Instead of storing K and V per layer (2 × D per token per layer), store the pre-QKV hidden state h (1 × D per token per layer). Reconstruct K and V at attention time by running K = W_k · h, V = W_v · h.
Compression Modes
| Mode | Per-token footprint (D=2048) | KV saving | Quality |
|---|---|---|---|
| FP16 K/V | 8,192 B | — | Baseline |
| FP16-h | 4,096 B | 50% | Bit-exact |
| INT8-h | 2,080 B | ~66% | Greedy-preserving |
| INT4-h | 1,056 B | 78% | Eval/ranking-safe |
How Reconstruction Works
The reconstruction uses the block's existing attn_qkv weight — no extra projection matrices needed.
Usage
# FP16-h (bit-exact, 50% saving)
TINYLOOP_KV_H_MODE=1 tinyloop model.tinyloop generate --loops 8
# INT8-h (66% saving)
TINYLOOP_KV_H_MODE=1 TINYLOOP_KV_INT8=1 tinyloop model.tinyloop generate --loops 8
# INT4-h (78% saving, use for eval/ranking)
TINYLOOP_KV_H_MODE=1 TINYLOOP_KV_INT4=1 tinyloop model.tinyloop generate --loops 8
# Recommended production stack (fastest + most compression)
TINYLOOP_KV_H_MODE=1 TINYLOOP_KV_INT4=1 TINYLOOP_EXPERIMENTAL_FP16_BODY=1 \
tinyloop model.tinyloop generate --loops 8
Why Only Looped
Standard transformers could store a "latent" like DeepSeek MLA, but MLA requires trained compression and decompression matrices (extra parameters). Store-h needs nothing extra — the reconstruction uses the existing QKV weight that the model already has. This works because the weight is shared across iterations, so the same projection is valid at every cache layer.