Generation APIs

Generation APIs are where runtime state transitions are most visible: cache growth, stop conditions, and per-token control flow. They are the autoregressive decode surface.

Function	Purpose
generate	Blocking batch decode. Returns full prompt+continuation when done.
generate_stream	Decode with per-token callback; caller observes and can abort mid-stream.
generate_batch	Decode a batch of prompts sequentially on one Model*.
generate_speculative	Chain self-speculative decoding — draft at L_draft, verify at L_verify, zero auxiliary weights.
generate_tree_speculative	Tree-branching self-speculative with K-wide draft candidates per step.

All generation functions share a common substrate:

• Allocate a fresh RuntimeKVCache sized for prompt_len + max_tokens.
• Prefill the prompt at the configured loop depth.
• Run a per-token decode loop with an explicit stop policy (EOS, stop-sequence tail match, callback abort, grammar dead-end, max_tokens).
• Free the cache before returning.

None of the generation functions take ownership of the cache — it is always cleaned up internally at function exit. PrefixCache / ResumeHandle (see Cache Reuse APIs) are the opt-in path for reusing state across calls.

generate

std::vector<int32_t> generate(Model* model,
                              const int32_t* prompt,
                              int prompt_len,
                              const GenerateConfig& config);

Standard blocking autoregressive decode. Runs prefill on the prompt, then samples tokens one at a time until a stop condition fires.

Parameters

Name	Type	Default	Description
model	Model*	—	Loaded model pointer. Must not be nullptr.
prompt	const int32_t*	—	Prompt token-id array, length prompt_len. Token ids must satisfy 0 <= t < vocab_size.
prompt_len	int	—	Number of tokens in the prompt. Must satisfy 1 <= prompt_len + config.max_tokens <= max_seq_len.
config	const GenerateConfig&	—	Decode configuration; see GenerateConfig below.

Returns

std::vector<int32_t> — the full token stream, prompt followed by decoded tokens. Length is prompt_len + emitted, where emitted <= config.max_tokens depending on stop policy.

Raises / Error conditions

On fatal error the runtime writes a message to stderr and aborts. Conditions:

• model == nullptr.
• prompt_len < 1 or prompt_len + config.max_tokens > max_seq_len.
• Out-of-memory during RuntimeKVCache allocation.
• Invalid sampling parameters (temperature < 0, top_k <= 0 when sampling is active, beam_size > 1 with inconsistent beam-path knobs).

Example

tinyloop::GenerateConfig cfg;
cfg.max_tokens = 64;
cfg.loops = 8;
cfg.temperature = 0.8f;
cfg.top_k = 50;
cfg.stop_sequences = {{13}};            // stop on "." (GPT-2 id for ".")
auto out = tinyloop::generate(model, prompt.data(), prompt.size(), cfg);
// out = concat(prompt, generated_tokens)

generate_stream

std::vector<int32_t> generate_stream(Model* model,
                                     const int32_t* prompt,
                                     int prompt_len,
                                     const GenerateConfig& config,
                                     const GenerateTokenCallback& on_token,
                                     GenerateStats* stats = nullptr);

Same decode semantics as generate, but calls on_token synchronously after each emitted token and optionally populates a GenerateStats struct with per-phase timing. Return value is still the full token stream at end of decode.

Parameters

Name	Type	Default	Description
model	Model*	—	Loaded model pointer.
prompt	const int32_t*	—	Prompt token ids.
prompt_len	int	—	1 <= prompt_len + max_tokens <= max_seq_len.
config	const GenerateConfig&	—	See GenerateConfig.
on_token	const GenerateTokenCallback&	—	Callback invoked once per emitted token. Signature: bool(int32_t token, int position). Return false to abort decode after including this token.
stats	GenerateStats*	nullptr	Optional stats output. Fields include prefill_ms, first_token_ms, decode_ms, tokens_emitted, stop_reason.

Returns

std::vector<int32_t> — same as generate.

Callback semantics

• on_token is called synchronously from the decode thread. It runs inside the tight decode loop; avoid blocking work (disk I/O, network sends) in the callback or wrap it in a bounded queue.
• Callback is called in order, once per emitted token.
• Returning false aborts cleanly: the last token is still included in the returned vector; cache cleanup still runs.
• The callback does not receive logits. If you need logit-level observation, use score family outside the decode loop.

GenerateStats fields

Field	Type	Description
prefill_ms	double	Wall-clock time for prefill phase (prompt → first-token-ready).
first_token_ms	double	Wall-clock time from call entry to first token emitted.
decode_ms	double	Wall-clock time for the steady-state decode loop.
tokens_emitted	int	Number of tokens emitted (excludes prompt).
stop_reason	enum	One of EOS, STOP_SEQUENCE, MAX_TOKENS, CALLBACK_ABORT, GRAMMAR_DEAD_END.
loops_used	int	Effective loop depth used (relevant for adaptive modes).

Example

tinyloop::GenerateStats stats;
auto out = tinyloop::generate_stream(
    model, prompt.data(), prompt.size(), cfg,
    [](int32_t tok, int pos) {
        std::cout << detokenize(tok) << std::flush;
        return true;                  // keep going; return false to abort
    },
    &stats);
std::cerr << "decode rate: "
          << stats.tokens_emitted / (stats.decode_ms / 1000.0)
          << " tok/s\n";

generate_batch

std::vector<int32_t> generate_batch(Model* model,
                                    const std::vector<std::vector<int32_t>>& prompts,
                                    const GenerateConfig& config);

Sequential batch decode. Each prompt is decoded to completion with a fresh cache, then the next begins. The return value is the flat concatenation of all outputs (prompt + continuation for each, joined).

Parameters

Name	Type	Default	Description
model	Model*	—	Loaded model pointer.
prompts	const std::vector<std::vector<int32_t>>&	—	Batch of prompts. Each inner vector is one prompt.
config	const GenerateConfig&	—	Shared config applied to every prompt. Per-prompt config is not currently supported — for that, call generate directly in a loop.

Returns

std::vector<int32_t> — flat concatenation of all outputs. To recover per-prompt boundaries, the caller must track each prompt's length and each resulting emission count (available via the lengths of the prompts plus the config's max_tokens bound, minus early stops).

Notes

• This is not continuous batching. TinyLoop processes prompts one at a time on the same Model*. Real batched prefill + paged attention is a multi-week refactor and is currently out of scope.
• Per-prompt early stops (EOS, stop sequences) work as in generate.
• For concurrent batch serving, use one Model* per worker thread (higher VRAM) or an external lock-per-model scheme — same guidance as Lifecycle and Memory Model.

generate_speculative

std::vector<int32_t> generate_speculative(Model* model,
                                          const int32_t* prompt,
                                          int prompt_len,
                                          const SpeculativeConfig& config);

Chain self-speculative decoding. Drafts draft_ahead tokens at the cheaper L_draft, verifies them in a single forward at L_verify, accepts the longest matching prefix, and rolls the cache forward. No auxiliary draft model is required — draft and verify use the same shared loop block at different depths.

Parameters

Name	Type	Default	Description
model	Model*	—	Loaded model pointer.
prompt	const int32_t*	—	Prompt token ids.
prompt_len	int	—	1 <= prompt_len + max_tokens <= max_seq_len.
config	const SpeculativeConfig&	—	See SpeculativeConfig.

Returns

std::vector<int32_t> — same shape as generate: prompt followed by decoded tokens.

SpeculativeConfig

Field	Type	Default	Description
max_tokens	int	64	Total emission cap (excludes prompt).
draft_loops	int	2	Cheap draft-pass loop depth. Lower = faster drafts, more rejections.
verify_loops	int	8	Verify-pass loop depth. Typically equal to model.default_loops.
draft_ahead	int	4	Number of tokens drafted per verify cycle. Larger = more parallelism per verify, more wasted work on miss.
temperature	float	0.0	Greedy acceptance is the well-characterized path; stochastic speculative is experimental.
seed	uint64_t	0	RNG seed for stochastic mode. Ignored when temperature == 0.

Acceptance semantics

• Draft generates draft_ahead tokens greedily at draft_loops.
• Verify runs once at verify_loops across the entire drafted sequence.
• The longest prefix whose argmax at verify_loops matches the draft is accepted.
• First mismatch contributes one bonus token from the verify distribution; the cache is rolled forward to that point and the next cycle begins.

Notes

• Forward-pass reduction, not wall-clock. The measured acceptance rate translates to fewer full-depth forward_with_cache calls, not a proportional wall-clock speedup — the draft passes still cost compute. Realised wall-clock speedup on well-tuned configs is 2.0–2.8× on 407M checkpoints (see paper §6).
• Greedy is the safe mode. Stochastic speculative decode introduces additional subtleties (draft/verify distribution mismatch under sampling) that the current implementation does not guarantee across all parameterizations.
• β-dependent. On high-β models (β > 1) the draft and verify distributions diverge faster, collapsing acceptance rate and potentially producing slower-than-baseline decode. Test empirically before committing to speculative for a given checkpoint.

generate_tree_speculative

std::vector<int32_t> generate_tree_speculative(Model* model,
                                               const int32_t* prompt,
                                               int prompt_len,
                                               const TreeSpeculativeConfig& config);

Tree-branching speculative decoding. Instead of a linear chain of drafted tokens, draft K candidates per step forming a tree, then verify the whole tree with a single tree-mask-gated forward. Accepts the longest matching path.

Parameters

Name	Type	Default	Description
model	Model*	—	Loaded model pointer.
prompt	const int32_t*	—	Prompt token ids.
prompt_len	int	—	1 <= prompt_len + max_tokens <= max_seq_len.
config	const TreeSpeculativeConfig&	—	Extends SpeculativeConfig with K_branches and tree_depth.

Returns

std::vector<int32_t> — full output sequence.

TreeSpeculativeConfig

Field	Type	Default	Description
max_tokens	int	64	Total emission cap.
draft_loops	int	2	Draft-pass loop depth.
verify_loops	int	8	Verify-pass loop depth.
draft_ahead	int	4	Tree depth — number of sequential draft steps per cycle.
K_branches	int	2	Number of candidate tokens kept per draft step. Total tree size is approximately K_branches^draft_ahead.
temperature	float	0.0	Greedy is the bit-exact path (matches non-speculative output); sampling is experimental.

Notes

• Architecture-exclusive. Tree speculative decoding with the same weights at draft and verify depths is only possible with a shared loop block. Mainstream runtimes require an auxiliary draft head (Medusa) or small draft model (EAGLE) to achieve similar throughput.
• Bit-exact at temperature == 0. Given identical seeds and K_branches == 2, greedy tree-speculative produces the same tokens as generate_speculative with draft_ahead=1 and chain decode. This is test-verified across several prompts.
• Throughput sweet spot on the 407M artifact is K_branches=2, draft_ahead=2, draft_loops=2, verify_loops=8. Higher K quickly stops paying off (tree mass explodes with little quality gain).

GenerateConfig

Shared across generate, generate_stream, and generate_batch.

Field	Type	Default	Description
max_tokens	int	64	Upper bound on emitted tokens.
loops	int	model.default_loops	Loop depth. Set explicitly to override the checkpoint's trained default.
temperature	float	0.0	Softmax temperature. 0.0 (or <= 0.01) means greedy argmax. Must be >= 0.
top_k	int	50	Retain top-k tokens before sampling. Must be >= 1 when sampling is active. Ignored when temperature == 0.
top_p	float	1.0	Nucleus sampling threshold. 1.0 disables. Applied after top_k.
cache_window	int	0	Sliding-window cap on KV cache. 0 means unlimited (bounded by max_seq_len).
repetition_penalty	float	1.0	HuggingFace-convention penalty applied before temperature/top-k/top-p. 1.0 disables.
stop_sequences	std::vector<std::vector<int32_t>>	{}	Token-id sequences that terminate decode when matched at the tail of emitted output. Matching tokens are included in the returned stream.
beam_size	int	0	0 or 1 = standard sampling. >= 2 switches to deterministic beam search; sampling knobs are then ignored.
length_penalty	float	1.0	Beam-search only. Ranks beams by log_prob / max(1, len)^length_penalty.
early_exit_patience	int	0	0 disables adaptive-depth early exit. >0 enables: exit after patience consecutive stable probes past min_L.
early_exit_min_L	int	2	First-probe floor for early exit.
early_exit_peek_every	int	1	Probe cadence past min_L.
early_exit_logit_margin	float	0.0	>0 switches from hidden-L2 to logit-margin stability. Empirically the right signal for high-β models.
sample_seed	uint64_t	0	RNG seed for stochastic sampling. Deterministic given seed + config + model.
grammar	Grammar*	nullptr	Optional constrained-decoding grammar. See the Constrained decoding reference for grammar construction.

Canonical decode state machine

For generate and generate_stream, the internal state machine is:

1. Allocate RuntimeKVCache sized for prompt_len + config.max_tokens (plus cache_window adjustments if set).
2. Prefill the prompt with prefill_with_kv_cache at config.loops.
3. Produce first-token logits from the post-prefill hidden state.
4. Per token, in a loop: a. Apply repetition_penalty to logits at already-emitted positions. b. Apply grammar mask if configured; if the mask is all-zero, stop with GRAMMAR_DEAD_END. c. Sample the next token (greedy or stochastic). d. Append the token to the output vector. e. Advance the cache via decode_with_kv_cache* at config.loops. f. Check stop conditions: EOS, stop-sequence tail match, callback abort, max_tokens cap.
5. Return the full vector.

Speculative variants replace the single decode step with a draft-then-verify sub-loop; cache advancement is done in multi-token chunks rather than one token at a time.

Trade-offs and limitations

• No continuous batching. Batch size is logical, not concurrent. For true throughput batching a paged-attention refactor is required.
• Not reentrant on the same Model*. Mutable working buffers preclude concurrent generate calls on the same loaded model without external synchronization.
• Deterministic quality guarantees are strongest in greedy settings. Sampling reproducibility depends on explicit sample_seed configuration; cross-platform numeric reproducibility is not guaranteed (float accumulation order depends on CUDA stream scheduling).
• Speculative modes are β-sensitive. See the per-function notes above; empirical testing per checkpoint is required before committing to speculative decode in production.

See also

• Lifecycle and Memory Model — who owns caches and buffers.
• Cache Reuse and State Transition APIs — PrefixCache, ResumeHandle, PrefixPool.
• Scoring APIs — forward without sampling.
• Constrained decoding — Grammar reference (Python API page; C++ surface mirrors it).