Skip to main content

Lifecycle and Memory Model

This page is the authoritative memory/ownership contract for TinyLoop C++ callers.

Lifecycle entry points​

Model* load_model(const std::string& path, int max_seq_len = 2048, int prefill_chunk = 0);
void   free_model(Model* model);
size_t vram_usage_bytes(Model* model);

Internal allocation phases (load_model)​

load_model is not a lightweight constructor. Internally it performs:

  1. header parse + format validation (MAGIC, VERSION_MAX, dims)
  2. quantized weight upload (QuantWeight: packed bytes + FP16 scales/zeros)
  3. optional FP16 caches for hot GEMM paths (env-controlled)
  4. BufferPool allocation:
    • main (FP32 residual, always seq-wide)
    • norm, scratch1, attn_down (FP16 scratch; seq-wide or chunk-wide)
    • lazy logit_row
  5. optional reconstruction scratch for h-mode paths

Any allocation failure is fatal to model creation. Caller receives nullptr and must not continue with that model pointer.

Model-owned memory​

Model owns long-lived GPU memory:

  • quantized weights + metadata
  • output/head tensors
  • BufferPool reusable activations
  • optional FP16 weight caches

Model does not permanently own per-request decode caches. Those are created by generation/prefix/resume APIs.

Per-request / handle-owned memory​

RuntimeKVCache​

Allocated by generation-family calls and by prefix/warm-start builders. Layer storage mode depends on env/runtime flags:

  • FP16 K/V
  • INT8 K/V
  • store-h FP16
  • store-h INT8
  • store-h INT4

PrefixCache / ResumeHandle / PrefixPool​

Opaque handles that own cache state across calls.

  • PrefixCache: prefilled prefix state at a fixed loop depth.
  • ResumeHandle: residual snapshot + over-allocated cache for depth upgrades.
  • PrefixPool: multiple prefix caches + LRU behavior.

Mutable state and thread-safety implications​

Model has mutable working buffers (buffers.main, buffers.norm, scratch buffers). Every forward mutates them.

Consequences:

  • Concurrent forward calls on the same Model* are unsafe unless externally serialized.
  • Handles tied to one model (especially ResumeHandle) assume no foreign forward clobbers their expected residual/cache evolution.
  • Safer concurrency patterns:
    • lock-per-model
    • model-per-worker (higher VRAM)
    • process-per-replica

prefill_chunk and scratch residency​

prefill_chunk > 0 changes memory behavior:

  • main: still max_seq_len × D × sizeof(float)
  • scratch buffers (norm, scratch1, attn_down): allocated at chunk rows instead of full sequence rows

Trade-off:

  • lower VRAM footprint for long context
  • higher launch/driver overhead during prefill due to chunked execution
  • score-style APIs are constrained by chunk row budget

Why this memory design exists​

TinyLoop prioritizes:

  • stable high throughput after one model load
  • avoiding repeated allocation churn in hot paths
  • explicit control of VRAM-vs-latency tradeoffs via runtime mode flags

Compared with fully stateless APIs, this yields higher sustained performance but requires stronger caller discipline around ownership and concurrency.

Practical integration checklist​

Before integrating in production C++:

  1. Decide one concurrency model (lock vs model-per-worker).
  2. Decide max sequence budget + chunk strategy from VRAM envelope.
  3. Decide cache mode (FP16/INT8/store-h variants) and document quality/latency policy.
  4. Treat all opaque handles as VRAM-bearing objects; release deterministically.
  5. Wire health probes around load_model failures and OOM paths.