Batching and Integration

This page covers production-level integration concerns: how to batch requests, how to structure a service, and how to compose runtime modes without stepping on undefined behaviour. Batching in TinyLoop is static — N prompts decode in lockstep, one Model* at a time. Continuous batching (paged attention) is not currently supported.

Integration decision matrix

You want...	Use
Lowest latency per request	Single-request generate_stream
Highest throughput, same shape	generate_batch with lockstep prompts
Shared system prompt across many requests	PrefixCache + generate_from_prefix_cache
Multi-tenant with several known scaffolds	PrefixPool + generate_with_pool
Low-depth draft → high-depth final	ResumeHandle + resume_generate
Zero-aux-param decode speedup	generate_speculative

Each entry is an explicit runtime API — there is no hidden scheduler in TinyLoop that will pick one for you.

Static batched generation

std::vector<int32_t> generate_batch(Model* model,
                                    const std::vector<std::vector<int32_t>>& prompts,
                                    const GenerateConfig& config);

Static lockstep batched generation over N = prompts.size() lanes.

Execution shape

1. Per-lane prefill (sequential). Each prompt is prefilled one at a time, filling its dedicated cache region. Prefill is not parallelized across lanes in the current implementation.
2. Lockstep decode. All lanes step one token at a time in parallel via batched attention and GEMM kernels. Lanes that have stopped (EOS, stop-sequence, max-tokens) continue to participate in the kernel shape but their outputs are discarded.
3. Per-lane host-side sampling. After each decode step, sampling, repetition penalty, and grammar masking are applied per-lane on host-visible logits.

This is a throughput-oriented primitive, not full continuous batching. Lanes cannot be added mid-decode; the batch shape is fixed at call entry.

Current constraints (enforced by src/generate.cpp)

These are explicit runtime contracts, not incidental behavior. Violations abort with stderr diagnostics.

Constraint	Enforced at	Rationale
All prompts must satisfy prompt_len + max_tokens <= max_seq_len	Call entry	Static cache allocation per lane.
cache_window > 0 is rejected in batched mode	Call entry	Ring-buffer semantics do not compose cleanly with lockstep kernels at present.
beam_size > 1 is rejected in batched mode	Call entry	Beam × batch would multiply cache memory by beam_size * batch; not currently supported.
All lanes must share loops, temperature, top_k, top_p, repetition_penalty	Call entry	Per-lane divergent sampling is not batched efficiently.
Per-lane stop_sequences and grammar ARE allowed	Runtime	Masking is per-lane on host.
Store-h KV cache modes: fully supported	Runtime	All lanes use the same cache mode.

When static batching helps vs hurts

Helps:

• Similar-length prompts (e.g., classification, short generation).
• Steady-state high-throughput scenarios where lane divergence is low.

Hurts:

• Highly variable prompt lengths — short lanes wait for long lanes to reach EOS.
• Interactive workloads — a single slow lane blocks first-token latency for others.

A typical production pattern is to maintain multiple batch sizes and route requests to the smallest batch whose shape accommodates them.

Integration patterns

TinyLoop is an execution primitive, not a serving framework. The patterns below are known-good orchestrations that compose the runtime APIs safely.

Pattern 1 — Low-complexity single-tenant service

The simplest correct structure:

• One Model* per process (no sharing across threads).
• External request queue (e.g., stdin, HTTP endpoint, message queue).
• One worker thread drains the queue and invokes generate_stream per request.

// Pseudocode
tinyloop::Model* model = tinyloop::load_model("model.tinyloop", 2048);
std::queue<Request> queue;

std::thread worker([&]() {
    while (running) {
        Request req = queue.pop_blocking();
        tinyloop::GenerateConfig cfg = build_config(req);
        auto out = tinyloop::generate_stream(model, req.prompt.data(),
                                             req.prompt.size(), cfg,
                                             [&](int32_t tok, int pos) {
                                                 req.response.push_token(tok);
                                                 return !req.cancelled;
                                             });
        req.response.finish(out);
    }
});

tinyloop::free_model(model);

Trade-offs: simplest to reason about; max throughput is 1× single-request throughput; no prefix reuse; latency is predictable.

Pattern 2 — Throughput-oriented service with bucketing

When you have sustained traffic and can afford VRAM for multiple model instances:

• Worker pool with N processes, each with its own Model*.
• Dispatcher that buckets requests by compatible shape (loops, max_tokens band, cache_mode, beam_size).
• Per-bucket: if bucket has >= min_batch_size ready requests, call generate_batch; else fall back to generate_stream.

Key parameters to tune:

• min_batch_size — below this, static batching's prefill cost exceeds its decode win.
• Bucket bin boundaries on max_tokens — tight bins reduce wasted decode cycles on short lanes.
• Worker count — balance VRAM budget against queue depth.

Pattern 3 — Shared-scaffold multi-tenant service

When many requests share a common system prompt / scaffold:

• Register the scaffold into a PrefixPool at process startup.
• Serve requests through generate_with_pool — pool matches longest registered prefix and skips its prefill cost.
• Fresh prompts (no match) fall through to a standard prefill with no penalty.

auto* pool = tinyloop::create_prefix_pool(32);
tinyloop::register_prefix(pool, model, system_prompt.data(),
                          system_prompt.size(), /*n_loops=*/8);

// In request handler
auto out = tinyloop::generate_with_pool(model, pool, prompt.data(),
                                         prompt.size(), cfg);

Trade-offs: measurable wins when prefix-share ratio is >50 %; below that, the hash lookup and potential eviction churn may offset gains.

Pattern 4 — Quality-tiered service with warm-start upgrades

For workloads that want "fast draft, upgrade on demand":

1. Build a ResumeHandle at L_used = 4 from the prompt.
2. Serve the draft from resume_generate(handle, 4, cfg).
3. If the user clicks "refine", call resume_generate(handle, 16, cfg) — the first 4 iterations are skipped.

This is the operational form of the warm-start architectural primitive. Measured wall-clock savings are 32.5 % on the 407M artifact for one upgrade; the more upgrades per build, the better the amortization.

Pattern 5 — Speculative with a β-aware fallback

If your checkpoint has β ≤ 1 (transparent or bottleneck regime), self-speculative decode is a near-free 2× throughput gain:

tinyloop::SpeculativeConfig sc;
sc.draft_loops = 2;
sc.verify_loops = 8;
sc.draft_ahead = 4;
auto out = tinyloop::generate_speculative(model, prompt.data(),
                                          prompt.size(), sc);

On high-β checkpoints (β > 1) acceptance rates drop, potentially making speculative slower than baseline. Always benchmark before committing to speculative in production. A safe pattern is:

• At startup, run a small benchmark (5–10 representative prompts) comparing generate and generate_speculative wall-clock.
• If speculative is not >1.3× faster, fall back to standard decode for this checkpoint.

Composing runtime modes

TinyLoop does not enforce exclusivity across every combination of runtime modes. Some combinations work, some are rejected with explicit errors, and some are undefined. The matrix below is the currently verified surface:

Mode	Single decode	Batched decode	Speculative	Tree-speculative	Prefix reuse	Warm-start
FP16 cache	✅	✅	✅	✅	✅	✅
INT8 cache	✅	✅	✅	✅	✅	⚠ partial
INT4 cache (store-h INT4)	✅	✅	⚠ partial	⚠ partial	✅	⚠ partial
Grammar	✅	✅ (per-lane)	❌	❌	✅	✅
cache_window > 0	✅	❌	⚠ partial	❌	❌	⚠ partial
Beam search	✅	❌	❌	❌	❌	❌

✅ = fully tested. ⚠ partial = works in current commits but not exhaustively tested. ❌ = explicitly rejected at runtime.

Treat this matrix as load-bearing — the rejected combinations are guarded in src/generate.cpp and throwing past them is undefined behavior.

Thread-safety and concurrency summary

• One Model* is not reentrant. Any two forward passes on the same Model* must be externally serialized. The mutable working buffers (buffers.main, buffers.norm, scratch) would otherwise race.
• Handles are bound to a model. A PrefixCache, ResumeHandle, or PrefixPool built with model A must not be used with model B. No runtime enforcement — contract error.
• Handles are not reentrant. Two generate_from_prefix_cache calls on the same handle would race each other on the cloning path. Serialize externally.
• Concurrent build + decode (building a new PrefixCache while another thread decodes) is also unsafe: the build runs a full forward on the model's mutable buffers.

Safe patterns:

• Per-worker model. One Model* per thread/process, each with its own handle set.
• Lock-per-model. Single Model* protected by a mutex; build and decode calls serialize.
• Replica pool. N processes, each with its own loaded model, balanced via an external dispatcher.

Why TinyLoop exposes this as API, not hidden scheduler logic

Scheduler policy is product-dependent:

• Latency-sensitive products choose generate_stream with small batch sizes.
• Throughput-sensitive products use generate_batch with bucketing.
• Cost-sensitive products add PrefixPool to reduce duplicated prefill.
• Quality-tiered products compose ResumeHandle + speculative.

A framework that hides these choices behind a single serve() API would make strong implicit assumptions (e.g., uniform latency budget, single tenant) that are wrong for most real deployments. TinyLoop's position is: the runtime primitives should be explicit, and integration code — at the product layer — is where SLO policy lives.

Practical integration checklist

Before deploying TinyLoop as a C++ service:

1. Decide the concurrency model — lock-per-model vs model-per-worker. Document it in code.
2. Set a VRAM budget — per-model + per-handle + decode cache. Abort startup if budget cannot be honored.
3. Cap live handles — total of all PrefixCache + ResumeHandle + PrefixPool entries. Evict on cap, not on memory pressure (eviction under memory pressure is latency-destroying).
4. Benchmark speculative per checkpoint — do not assume speedup; always verify.
5. Add mode guardrails — reject combinations your product doesn't need at the dispatcher layer (defense-in-depth over the runtime's own rejections).
6. Wire health probes — load_model failures, OOM during build, and repeated decode aborts should page operators, not just log.
7. Explicitly choose greedy vs stochastic per deployment — production deterministic reproducibility depends on explicit seeds and greedy modes.
8. Instrument GenerateStats — log prefill_ms, first_token_ms, decode_ms, stop_reason per request. These are the only signals that catch runtime mode misconfiguration quickly.

Trade-offs

• Static batching improves throughput but can hurt tail latency. Short lanes wait for long lanes.
• Explicit integration gives control but requires orchestration engineering. Product teams carry the scheduling policy, not TinyLoop.
• Mode combinatorics require policy guardrails in caller code — the runtime guards the obvious impossibilities, not every product-specific invariant.
• VRAM budgeting is manual. No built-in memory pressure eviction; handle lifetimes are a product responsibility.

See also

• Generation APIs — the decode primitives composed here.
• Cache Reuse and State Transition APIs — PrefixCache, ResumeHandle, PrefixPool details.
• Runtime Modes — env-flag surface for cache modes, speculative, early exit.
• Performance and Memory — VRAM budget worked examples.