Prefix cache

When many requests share the same leading prompt (RAG system prompts, chat templates, few-shot preambles), prefilling the prefix once and reusing it across decode calls avoids the L-iteration prefill on every request.

The primitive is a PrefixCache — an opaque object holding the KV state at all L iterations of the loop block for the shared prompt. A single PrefixCache survives across many generate_from_prefix_cache calls and several concurrent / serial decodes off the same prefix.

Model.build_prefix_cache(prompt, loops, cache_window=0)

prefix = model.build_prefix_cache(prompt, loops=8, cache_window=0)

Runs the full prefill for prompt at loops iterations and returns a PrefixCache handle whose K/V tensors span every loop iteration of the shared loop block. No decode is performed — the handle is ready to be consumed by one or more generate_from_prefix_cache calls.

Parameters

Name	Type	Default	Description
prompt	np.ndarray[int32]	—	The shared prefix token ids, shape [seq_len]. Every subsequent generate_from_prefix_cache call must pass the same loops value — the prefix is depth-specialized.
loops	int	model.config()["default_loops"]	Loop depth the prefix is built at.
cache_window	int	0	Sliding-window cap. 0 means full cache (no eviction). >0 truncates the prefix's KV cache to the last cache_window tokens, losing information from earlier positions.

Returns

tinyloop_py.PrefixCache — an opaque object. Its internal state is:

• A full RuntimeKVCache sized for seq_len + (max_new_tokens from callers), populated up to seq_len at every one of the n_pre_blocks + loops cache layers
• The prompt token array (copied, so the caller's buffer is free to mutate)
• The loop depth (used to validate subsequent generate_from_prefix_cache calls)

Memory cost scales with seq_len × D × (n_pre_blocks + loops) × 2 bytes for FP16 KV. At seq_len=1024, D=2048, L=8: 80 MB. With TINYLOOP_KV_H_MODE=1 TINYLOOP_KV_INT4=1: 14 MB (see KV cache modes).

Raises

• RuntimeError — model has been closed; out-of-memory during cache allocation; seq_len > max_seq_len.
• ValueError — loops < 1; cache_window < 0.

Lifetime

Hold the PrefixCache for the lifetime of your shared workload. It is owned by Python reference-count; the underlying GPU memory is freed when the handle goes out of scope or is explicitly replaced. There is no explicit close() method (unlike Model) — GC handles it.

Two common patterns:

1. Module-level cache. Build once at startup, reuse for every request. Simplest when the prefix never changes (fixed system prompt).
2. Cache-per-prefix. Maintain a dict {prefix_hash: PrefixCache}. Evict on LRU when count / VRAM exceeds a budget.

Model.generate_from_prefix_cache(prefix, max_tokens=32, loops=8, ...)

Decodes starting from the state cached in prefix. The prefill skip is implicit — the first decoded token is produced without any re-run of the prompt.

generated = model.generate_from_prefix_cache(
    prefix,
    max_tokens=32,
    loops=8,
    temperature=0.0,
    top_k=50,
    top_p=1.0,
    cache_window=0,
    repetition_penalty=1.0,
    stop_sequences=[],
)

Parameters

Name	Type	Default	Description
prefix	PrefixCache	—	Returned by build_prefix_cache.
loops	int	—	Must match the loops used when building prefix. The binding validates this and raises if they differ.
max_tokens, temperature, top_k, top_p, cache_window, repetition_penalty, stop_sequences			Same semantics as Model.generate.

Returns

List[int] — the full token stream: prefix + decoded tokens. Length is len(prefix.prompt) + emitted.

Raises

• RuntimeError — model has been closed; out-of-memory; loops mismatch between call and prefix.
• ValueError — max_tokens < 1.

What actually happens

Internally this:

1. Clones prefix's RuntimeKVCache (cheap — per-layer pointer copy in the h modes, full cudaMemcpy in the K/V modes). This gives the call its own scratch-safe copy so concurrent calls off the same prefix don't race.
2. Decodes max_tokens tokens as usual, reusing the cloned cache.
3. Drops the clone at end of call. The original prefix is unchanged.

Step (1) is why you can use one PrefixCache across many concurrent calls: each call has its own KV scratch copy. The main cost is the clone's memory allocation — small relative to the prefill work it avoids.

Measured multi-request throughput

Because the loop block is applied L times per forward, a TinyLoop prefix cache stores K/V at every one of those L iterations — sharing a single PrefixCache across N decode calls avoids re-doing the full L-iteration prefill N − 1 times. The absolute savings are (N − 1) × prefill_cost; the observed speedup asymptotes at (prefill + decode) / (prefill / N + decode).

Measured 2026-04-17 on model_407m_gptq_int4.tinyloop / H100 via tests/bench_prefix_cache.py (NO_CACHE rebuilds the full prefix each request; SHARED builds the prefix once and reuses it for all N continuations; both run at the same loop depth L = 8):

backend	prefix	N	NO_CACHE	SHARED	speedup
INT4 scalar (default)	500 tok	10	12 847 ms	5 415 ms	2.37×
INT4 scalar (default)	1024 tok	10	23 852 ms	8 329 ms	2.86×
INT4 scalar (default)	1024 tok	50	104 605 ms	34 531 ms	3.03×
FP16 body tensor core ¹	1024 tok	10	5 123 ms	4 732 ms	1.08×

¹ TINYLOOP_EXPERIMENTAL_FP16_BODY=1 — prefill is already close to memory-bound on tensor cores, so the prefix-share savings shrink.

Practical guidance

• Expect 2-3× throughput on RAG / multi-user same-prefix workloads with a long prefix (>~500 tokens) on the default INT4-scalar body.
• FP16-body tensor-core path delivers smaller relative gains — prefill is already cheap, so the share isn't as leveraged. Prefix cache is still correct and cheap, just not dramatic.
• Reuse one PrefixCache across requests. Do not call build_prefix_cache per request — that defeats the purpose.
• Match loops. A prefix built at L=8 cannot serve a decode call at L=16. If you need multiple depths, use warm-start mid-loop instead, or maintain one PrefixCache per depth.

Relationship to other APIs

• Prefix cache vs warm-start mid-loop: prefix cache shares a prompt across many requests at the same L; warm-start shares a prompt across different L values in one request. Both can compose — in principle you could build a PrefixCache and then warm-start a ResumeHandle off it — but the APIs are not yet wired for that combination (tracked as a follow-up).
• Prefix cache vs speculative decode: speculative decode uses the same weights at two depths within one generation; prefix cache reuses state across separate generations at the same depth. Orthogonal — can be stacked.
• Prefix cache and KV cache modes: prefix cache uses the same KV storage mode as generate (FP16, INT8, or one of the three h modes). The h modes make the prefix cache ~2× smaller; with INT4-h + fp16_body, the prefix share speedup and the latency-cost reduction stack.

Error conditions

Error	Cause	Fix
RuntimeError: model was closed	Called on a closed Model	Construct a new Model
RuntimeError: loops mismatch	generate_from_prefix_cache's loops differs from build-time	Align the kwargs or rebuild at the target depth
RuntimeError: out of memory	Prefix too long / cache budget exceeded	Reduce seq_len, lower L, enable INT4-h, or cap cache_window
ValueError: max_tokens < 1	Self-explanatory	Use a positive value