Runtime Modes

TinyLoop is easier to reason about if you separate the runtime into distinct modes.

Inspect

Goal:

• validate artifact metadata
• estimate memory footprint
• avoid full weight upload

Use when:

• checking a new .tinyloop file
• planning VRAM budget
• validating conversion output

Benchmark

Goal:

• measure runtime latency and throughput
• avoid full logits work when not needed

Use when:

• profiling kernels
• comparing runtime modes
• measuring loop-count effects

Important notes:

• benchmark uses synthetic token ids
• benchmark is not a quality evaluation
• TINYLOOP_CUDA_PROFILE=1 prints phase timings

Score

Goal:

• materialize logits for every sequence position

Use when:

• running task evaluation loops
• comparing outputs across implementations
• inspecting position-wise behavior

Cost tradeoff:

• highest output-memory path
• not the preferred decode loop helper

Score Last Token

Goal:

• compute only the final logits row

Use when:

• implementing generation
• doing next-token ranking
• minimizing output allocation

This is the preferred low-memory scoring path for decode-style workloads.

Generate

Goal:

• run ordinary autoregressive decoding

Current behavior:

• cached decode is the default
• TINYLOOP_DISABLE_KV_CACHE=1 forces the uncached reference path
• cache_window can bound the KV cache
• the CLI supports optional GPT-2 BPE (--tokenizer DIR or TINYLOOP_TOKENIZER_DIR), with raw-byte fallback only when tokenizer loading is not configured
• generation core executes via generate_stream state machine: prefill once, then per-token logit transform (repetition + grammar mask + temperature) and decode-cache append
• streaming mode uses synchronous callback backpressure (on_token -> bool) and can abort cleanly after current token emission

Speculate

Goal:

• use the same model for fast draft and slower verify passes

Current behavior:

• shared-cache speculative runtime exists
• accept-or-resample logic exists
• speculative regressions exist
• broader serving-grade validation is still an open task

Execution shape:

1. draft proposes up to draft_ahead tokens at draft_loops
2. verify scores at verify_loops
3. each drafted token is accepted or replaced
4. accepted prefix is committed and next round continues

Trade-offs:

• lower draft_loops improves draft speed but often lowers acceptance
• larger draft_ahead increases best-case amortization and worst-case wasted work
• verify path quality dominates final output distribution

Prefix Cache Reuse

Goal:

• prefill once on a shared prefix
• reuse that prefix across repeated requests

This is separate from ordinary generate() because it is about workload reuse rather than one-shot generation.

Warm-start Mid-loop

Goal:

• reuse an earlier generation's cached state when the follow-up request wants more loop iterations at the same prompt

Behaviour:

• the first request runs at L_used and populates cache layers [0, n_pre + L_used)
• a follow-up request at L_new > L_used resumes from the residual state left after iter L_used - 1, runs iters L_used ... L_new - 1 in place, extends the cache with the new layers, and produces output bit-exact to running L_new from scratch on the same prompt
• savings: the first L_used iterations of the loop block never re-run (saves L_used × prefix-length worth of INT4 GEMMs + attention)

Architecture:

• the primitive is run_loop_iters_range(model, seq_len, start_iter, end_iter, cache) in src/runtime_internal.h
• the parity test tests/test_warmstart_parity.cu verifies bit-exact output on 407M INT4 at seq_len=16 across all 10 cache layers and 32 768 residual elements
• public C++ and Python APIs (a ResumeHandle type with build_resume_handle + resume_generate) are pending

Why this is weight-sharing-exclusive:

• a standard deep transformer has different weights per layer, so "resume at layer k" is not a well-defined operation — the k-th layer has seen its input through a specific weight matrix that doesn't repeat
• TinyLoop's looped block is the same W applied L times, so the residual stream after iter k is a valid starting point for applying W a further L_new - k times

Use cases:

• interactive UIs where a user asks "try harder on that" — route the follow-up to resume_generate(handle, new_L) instead of a fresh generate
• adaptive quality levels that cache intermediate depths and upgrade on demand
• offline eval pipelines that score the same prompt at several depths — run once at L_max and reuse intermediate states

This mode is distinct from prefix caching (that reuses shared prompts; warm-start reuses the same prompt at different depths) and distinct from speculative decoding (which uses the same weights at different depths within one generation; warm-start reuses state across separate generations).

Practical Advice

Use:

• inspect first
• benchmark for raw runtime measurement
• score_last for decode helpers
• generate for normal autoregressive output
• prefix caching for repeated shared prompts
• warm-start mid-loop when a user upgrades to a deeper L on the same prompt
• speculative decode only when you intentionally want that runtime behavior