Skip to main content

KV Cache Environment Variables

Stack these to select one of the storage modes. See KV Cache Modes for full memory math.

VariableValueEffect
TINYLOOP_KV_INT81INT8 K/V with per-head scales. −49% KV footprint.
TINYLOOP_KV_H_MODE1Store-h cache: reconstruct K/V from stored hidden state. Bit-exact. 50% saving.
TINYLOOP_KV_INT8 + TINYLOOP_KV_H_MODE1 + 1h-as-INT8. ~66% saving.
TINYLOOP_KV_INT4 + TINYLOOP_KV_H_MODE1 + 1h-as-INT4. 78% saving. Recommended for long-context.
TINYLOOP_KV_H_FUSED1Fused INT4+unpack kernel. Currently performance-negative.
TINYLOOP_PAGED_KV1Paged KV cache. 16-token pages, pool-based. ~5.6% overhead. Foundation for continuous batching.

Composability​

GoalStack
Long-context low memoryTINYLOOP_KV_H_MODE=1 TINYLOOP_KV_INT4=1 TINYLOOP_H_MODE_FP16_ATTN_QKV=1
Near-lossless compressionTINYLOOP_KV_H_MODE=1 TINYLOOP_KV_INT8=1 TINYLOOP_EXPERIMENTAL_FP16_BODY=1