Cache and State Reuse APIs

TinyLoop exposes three reuse primitives, each optimized for a different reuse axis. All three are explicit, caller-managed handles: they own GPU memory, must be freed, and are not safe to share across concurrent generate* calls on the same Model* without external synchronization.

Handle	Reuse axis	Typical use
PrefixCache	Same prefix, same loop depth, many requests	Shared system prompt across user sessions.
ResumeHandle	Same prefix, different loop depth	Upgrade a low-depth draft to a deeper final generation.
PrefixPool	Multiple prefixes, content-addressable dispatch	Multi-tenant serving with several known scaffolds.

PrefixCache

A PrefixCache captures the post-prefill hidden state + RuntimeKVCache for a fixed (prompt, loops) pair. Every subsequent generate_from_prefix_cache call clones this state into fresh working buffers and continues from the clone, skipping prefill entirely.

build_prefix_cache

PrefixCache* build_prefix_cache(Model* model,
                                const int32_t* tokens,
                                int seq_len,
                                int n_loops = 8,
                                int cache_window = 0);

Construct a prefix cache from a prompt. Runs one full prefill and stores the result.

Parameters

Name	Type	Default	Description
model	Model*	—	Loaded model pointer. Must not be nullptr.
tokens	const int32_t*	—	Prompt token ids, length seq_len.
seq_len	int	—	1 <= seq_len <= max_seq_len. The cache is sized exactly for seq_len; subsequent decode will grow beyond this point.
n_loops	int	8	Loop depth at which the prefill is computed. Must match any downstream generate_from_prefix_cache call's config.loops.
cache_window	int	0	Sliding-window cap on the cache layers. 0 = unlimited.

Returns

PrefixCache* — opaque handle. nullptr on failure (rare; most failure paths abort).

Notes

• Ownership. Caller owns the returned pointer. Must call free_prefix_cache exactly once.
• VRAM. A prefix cache holds a full RuntimeKVCache at seq_len plus a residual snapshot. Memory scales with seq_len * n_layers * 2 * head_dim bytes (plus overhead); allocate with disk/VRAM budget in mind.
• Immutable once built. You cannot mutate a built PrefixCache. To extend the prefix, build a new one.

free_prefix_cache

void free_prefix_cache(PrefixCache* prefix_cache);

Release all GPU memory held by the prefix cache.

Notes

• Safe to pass nullptr (no-op).
• Must not be called while a generate_from_prefix_cache call is in flight using this handle.

generate_from_prefix_cache

std::vector<int32_t> generate_from_prefix_cache(Model* model,
                                                const PrefixCache* prefix_cache,
                                                const GenerateConfig& config);

Decode starting from the state captured in prefix_cache. Equivalent to calling generate with the same prompt, minus the prefill cost.

Parameters

Name	Type	Default	Description
model	Model*	—	Must be the same model instance used to build the prefix cache; mixing is undefined.
prefix_cache	const PrefixCache*	—	Prebuilt prefix handle.
config	const GenerateConfig&	—	Decode config. config.loops must equal the n_loops used at build time; mismatch is a contract error.

Returns

std::vector<int32_t> — full token stream: prompt tokens (recovered from the prefix cache) followed by decoded tokens.

Error conditions

• config.loops does not match build-time n_loops — aborts with a stderr diagnostic.
• config.grammar is set but incompatible with the prefix's token history — aborts.
• seq_len + config.max_tokens > max_seq_len — aborts.

Example

// Build once for a shared system prompt
auto* prefix = tinyloop::build_prefix_cache(model, sys_prompt.data(), sys_prompt.size(), 8);

// Reuse across many requests
tinyloop::GenerateConfig cfg;
cfg.loops = 8;
cfg.max_tokens = 128;
for (const auto& user_turn : batch) {
    cfg.stop_sequences = user_turn.stops;
    auto out = tinyloop::generate_from_prefix_cache(model, prefix, cfg);
    // prefix is unchanged; safe to reuse
}

tinyloop::free_prefix_cache(prefix);

ResumeHandle

A ResumeHandle captures state at loop depth L_used but pre-allocates cache capacity for a deeper max_loops. You build it by running prefill at a shallow depth, then can "resume" generation at any new_L > L_used without replaying the first L_used iterations. This is the warm-start path — the architecture-specific upgrade primitive.

build_resume_handle

ResumeHandle* build_resume_handle(Model* model,
                                  const int32_t* tokens,
                                  int seq_len,
                                  int L_used,
                                  int max_loops = 32,
                                  int cache_window = 0);

Build a resume handle with space for loop extension.

Parameters

Name	Type	Default	Description
model	Model*	—	Loaded model pointer.
tokens	const int32_t*	—	Prompt token ids.
seq_len	int	—	1 <= seq_len <= max_seq_len.
L_used	int	—	Loop depth the handle is initialized at. Cache layers beyond iteration L_used are zero-filled placeholders.
max_loops	int	32	Upper bound on the loop depth resume_generate can be called at. Handle allocates cache capacity for this depth.
cache_window	int	0	Sliding-window cap.

Returns

ResumeHandle* — opaque handle. nullptr on failure.

Error conditions

• L_used < 1 or L_used > max_loops — aborts.
• max_loops > 64 — aborts.
• OOM during over-allocation — aborts.

Notes

• Over-allocation is deliberate. The handle's cache is sized for max_loops iterations of cache layers even though only L_used are filled. This pays memory upfront to avoid reallocation on resume.
• Residual snapshot. The handle also stores a snapshot of the residual stream at iteration L_used, required for correctness of the skip-ahead resume path.

resume_generate

std::vector<int32_t> resume_generate(Model* model,
                                     const ResumeHandle* handle,
                                     int new_L,
                                     const GenerateConfig& config);

Resume decoding at deeper loop depth new_L, running only the iterations [L_used, new_L) before starting the decode loop.

Parameters

Name	Type	Default	Description
model	Model*	—	Same model that built the handle.
handle	const ResumeHandle*	—	Prebuilt resume handle.
new_L	int	—	Target loop depth. Must satisfy handle.L_used <= new_L <= handle.max_loops.
config	const GenerateConfig&	—	Decode config. config.loops is ignored for the prefill portion (overridden by new_L); still used for decode per-step loop count.

Returns

std::vector<int32_t> — full output sequence.

Internal state machine

1. Clone the handle's cache into scratch working buffers.
2. Restore the residual snapshot into buffers.main.
3. Run run_loop_iters_range(L_used, new_L) on the shared loop block to bring the residual from depth L_used to depth new_L.
4. Synthesize a temporary prefix cache from the advanced state.
5. Run the standard decode loop at config.loops iterations per step.
6. Free scratch caches at exit; the original handle is unchanged.

Measured benefit

On the 407M INT4 artifact, resuming from L_used = 4 to new_L = 8 saves 32.5 % wall-clock vs. running a fresh generate(loops=8) from scratch. Break-even is N = 1 follow-up, i.e., the build cost is amortized after one resume (see paper §6).

free_resume_handle

void free_resume_handle(ResumeHandle* handle);

Release the handle's GPU memory.

Notes

• Safe on nullptr.
• Must not overlap with in-flight resume_generate calls using the handle.

PrefixPool

A content-addressable pool of PrefixCache instances. When generate_with_pool is called, the pool finds the longest registered prefix that is a prefix of the incoming prompt and uses that prefix's cache for decode; the remaining (suffix) tokens are scored through a mini-prefill.

create_prefix_pool

PrefixPool* create_prefix_pool(size_t capacity = 16);

Construct an empty pool.

Parameters

Name	Type	Default	Description
capacity	size_t	16	Maximum number of registered prefixes. On overflow, the LRU prefix is evicted and freed.

Returns

PrefixPool*.

register_prefix

int register_prefix(PrefixPool* pool,
                    Model* model,
                    const int32_t* tokens,
                    int seq_len,
                    int n_loops = 8,
                    int cache_window = 0);

Build and register a prefix cache into the pool. Returns an integer id; the same id can be used to track the prefix's lifetime if needed.

Parameters

Name	Type	Default	Description
pool	PrefixPool*	—	Target pool.
model	Model*	—	Model used to build.
tokens	const int32_t*	—	Prefix token ids.
seq_len	int	—	Prefix length.
n_loops	int	8	Loop depth of the prefix cache.
cache_window	int	0	Optional cache window cap.

Returns

int — an identifier for the registered prefix. Values are small positive integers that may be reused across evictions.

Notes

• Hashing. Prefix lookup uses token-sequence equality. Only exact-prefix matches count — a prompt starting with the same 200 tokens as a registered 100-token prefix matches the 100 tokens; one off-by-one token disqualifies.
• Same-model invariant. All prefixes registered into a pool must be built with the same Model*; mixing is undefined.

generate_with_pool

std::vector<int32_t> generate_with_pool(Model* model,
                                        PrefixPool* pool,
                                        const int32_t* prompt,
                                        int prompt_len,
                                        const GenerateConfig& config);

Decode a prompt, reusing the longest registered prefix from pool.

Parameters

Name	Type	Default	Description
model	Model*	—	Same as pool's model.
pool	PrefixPool*	—	Prebuilt pool.
prompt	const int32_t*	—	Full prompt ids.
prompt_len	int	—	Number of prompt tokens.
config	const GenerateConfig&	—	Decode config. config.loops must match the pool's registered prefix n_loops.

Returns

std::vector<int32_t> — full output.

Behavior

• If any registered prefix is a prefix of prompt, the longest is chosen; its PrefixCache clone is used as the decode starting point.
• If no prefix matches, the call degrades cleanly to a standard generate — no performance penalty over a baseline call without the pool.
• After the call, the matched prefix's LRU position is bumped; stale prefixes are evicted if the pool hit capacity.

Measured benefit

On the 407M INT4 artifact, generate_with_pool matches tree-speculative decoding's quality in closed-book completions (bit-exact at temperature == 0) while delivering wall-clock speedups proportional to the prefix-sharing ratio. See paper §S for the full benchmark.

free_prefix_pool

void free_prefix_pool(PrefixPool* pool);

Free the pool and all registered prefixes in one call.

Reuse boundary summary

The three handles partition the reuse landscape by what can change from call to call:

Dimension	PrefixCache	ResumeHandle	PrefixPool
Prefix (prompt prefix)	fixed	fixed	variable (longest match)
Loop depth	fixed (n_loops)	upgradeable (L_used → new_L)	fixed per registered entry
Decode config	variable	variable	variable
Per-call cost	decode only	loop-range + decode	match + mini-prefill + decode

Error conditions shared across all three handles

• Loop-depth mismatch. Using a handle built at n_loops = 8 with a decode config at config.loops = 16 is a hard contract error — the cache layers beyond iteration 8 are unpopulated (for PrefixCache) or allocated but zero (for ResumeHandle outside [L_used, max_loops]).
• Cross-model misuse. Using a handle built with one Model* against a different Model* is undefined. The handles carry a reference to the originating model but verification is caller-side.
• Concurrent forward on the same model while a handle is in use — the model's mutable working buffers can be clobbered by an interleaving call, invalidating the handle's correctness guarantees. Serialize externally.

Trade-offs

• Handles are stateful — stronger reuse increases operational complexity. An API that does not use handles is simpler to reason about; handles pay for themselves only when the reuse ratio is meaningfully high.
• All handles consume VRAM for the lifetime they exist. Build policies should cap the total number of live handles, not just the pool capacity.
• Prefix matching is exact on tokens, not on semantics. Equivalent paraphrased prompts do not share a cache; this is the correct semantics for a deterministic runtime but may differ from what application-layer users expect.
• Greedy is the well-characterized path for reuse. Stochastic sampling with reused prefixes is correct but may produce different sample streams than a fresh generate with the same seed — the RNG advances from the cached-state starting point.

See also

• Generation APIs — the non-reuse decode entry points.
• Lifecycle and Memory Model — handle memory accounting in context.
• Batching and Integration Patterns — how to compose handles with concurrent serving.