Generation

Decode APIs on Model. All three share the same sampling kwargs (temperature, top_k, cache_window, repetition_penalty, stop_sequences) — pick the one whose return shape matches your use case.

Conventions used on this page:

• D = model.config()["dim"]
• V = model.config()["vocab_size"]
• Token ids are int32, bounded by V.
• Loop depth L is the number of applications of the shared loop block; total effective depth is n_pre_blocks + L.

Model.generate(...)

Standard batch autoregressive decode. Runs prefill on the prompt, then samples tokens one at a time until max_tokens or a stop sequence.

generated = model.generate(
    prompt,
    max_tokens=64,
    loops=8,
    temperature=0.8,
    top_k=50,
    cache_window=0,
    repetition_penalty=1.3,
    stop_sequences=[[13], [198, 198]],  # [".", "\n\n"] encoded as BPE ids
    early_exit_patience=0,
    early_exit_min_L=2,
    early_exit_tau=1e-3,
    early_exit_peek_every=1,
    early_exit_logit_margin=0.0,
)

Parameters

Name	Type	Default	Description
prompt	np.ndarray[int32]	—	Prompt token ids, shape [seq_len]. Converted via np.asarray(..., dtype=np.int32) inside the binding; lists of Python ints work but add one copy. Must satisfy seq_len + max_tokens <= model.max_seq_len.
max_tokens	int	64	Upper bound on generated token count (excludes the prompt). Generation may stop earlier via stop_sequences or early exit. Must be >= 1.
loops	int	model.config()["default_loops"]	Loop depth. Higher L = more compute per position, generally better quality up to saturation.
temperature	float	0.0	Softmax temperature. 0.0 selects argmax (greedy); any <= 0.01 is also treated as greedy. Must be >= 0.
top_k	int	50	Retain only the top-k tokens before sampling. TinyLoop-specific constraint: sampling paths expect top_k >= 1; top_k=0 is not a supported "disable top-k" sentinel in current runtime code. Ignored when temperature == 0.
top_p	float	1.0	Nucleus sampling — retain smallest set of tokens whose cumulative probability exceeds top_p. 1.0 disables. Applied after top_k.
cache_window	int	0	Sliding-window cap on the KV cache. 0 means full cache (no eviction). >0 bounds the cache to the last cache_window tokens; earlier positions are evicted FIFO. Useful for long-context workloads where memory is tight; costs quality on information outside the window.
repetition_penalty	float	1.0	HuggingFace-convention penalty applied before temperature / top_k / top_p. 1.0 disables. Values > 1.0 scale down logits for already-emitted tokens; typical range 1.05–1.30.
stop_sequences	List[List[int]]	[]	List of token-id sequences that terminate generation when matched at the tail of the emitted output. Matching tokens are included in the returned output. Empty list disables. Each sequence is matched left-to-right; shorter matches win (first match terminates).
early_exit_patience	int	0	Early-exit probe count. 0 disables the entire feature (falls through to decoding all loops iterations). >0 enables: after early_exit_min_L iterations, probe stability every early_exit_peek_every iterations; exit after patience consecutive stable probes.
early_exit_min_L	int	2	Minimum iterations before the first probe. Guards against false-positive exits in the high-noise early iterations.
early_exit_tau	float	1e-3	Hidden-L2 mode threshold. Relative change `rho =
early_exit_peek_every	int	1	Probe cadence past min_L. Larger values amortize probe cost at the expense of slightly later exits.
early_exit_logit_margin	float	0.0	Logit-margin mode threshold. > 0 switches from hidden-L2 to logit-margin — the top-1 logit's lead over top-2 must exceed margin for a stable probe. Costs one full head projection per probe but fires on well-trained models. Measured sweet spot on 1B-effective: margin=3.0, min_L=4, peek_every=2, patience=1 gives −13 % decode at no visible quality loss.
beam_size	int	0	Beam search. 0 or 1 disables beam search (standard sampling). >= 2 enables deterministic beam-search decoding. temperature / top_k / top_p are ignored when beam search is active. See the Beam search section below.
length_penalty	float	1.0	Beam-search-only. Final beams are ranked by log_prob / max(1, new_len)^length_penalty so short outputs don't dominate. 1.0 = unnormalised log-prob; 0.6–0.8 typical for translation; >1.0 favours shorter outputs. Ignored when beam_size <= 1.
early_stopping	bool	True	Beam-search-only. When true, beam search returns as soon as the top-ranked beam has emitted EOS. When false, beam search runs to max_tokens on every beam. Ignored when beam_size <= 1.
grammar	tl.Grammar / None	None	Constrained decoding. When non-None, the grammar masks the sampling distribution at every step so only tokens whose byte sequences keep the grammar alive can be sampled. Composable with standard sampling, beam search, and every early-exit / repetition-penalty setting. See Constrained decoding for the full Grammar reference.

Returns

List[int] — the full token stream, prompt followed by decoded tokens. Length is seq_len + emitted, where emitted <= max_tokens.

Raises

• RuntimeError — model has been closed; seq_len + max_tokens > max_seq_len; out-of-memory during KV cache allocation.
• ValueError / RuntimeError — invalid shapes/ranges (max_tokens < 1, non-positive loops, invalid sampling bounds such as top_k <= 0 when sampling is active) or internal guard failures.

Semantics notes

• Cached decode is the default. The runtime allocates a RuntimeKVCache sized for seq_len + max_tokens, prefills the prompt at loops iterations, then produces one token per decode step reusing the cache. The uncached reference path (set TINYLOOP_DISABLE_KV_CACHE=1) is ~10× slower and exists only for parity testing.
• cache_window > 0 allocates a smaller cache of cache_window tokens and runs it as a ring buffer. cache.start_index advances once per eviction. All h-mode reconstruction paths handle the wrap correctly (K/V are unpacked in attention-visit order).
• stop_sequences matching. The match window is the last max(len(s) for s in stop_sequences) emitted tokens. Non-overlapping with the prompt — the emitter cannot "stop on" a token that was already in the prompt.
• GIL. Released during the CUDA forward and sampler. Python callbacks (not applicable here; see generate_stream) would run with the GIL held.

Internal execution flow

Model.generate(...) in Python is a thin wrapper over tinyloop::generate, which dispatches to generate_stream(..., on_token=nullptr) for non-beam decoding and to a dedicated beam path when beam_size > 1.

Non-beam decode loop (implementation-level):

1. allocate RuntimeKVCache in selected mode (FP16/INT8/store-h variants)
2. prefill prompt once (prefill_with_kv_cache)
3. score last hidden (score_hidden_vector) to get logits
4. apply penalties/masks in order:
   • repetition penalty
   • grammar legality mask (if enabled)
   • temperature scaling
5. sample next token
6. append decode state (decode_with_kv_cache*) and repeat

This ordering is intentional: grammar is an absolute legality mask in logit space, and repetition penalty should operate on untempered logits to preserve conventional scaling behavior.

Beam search

When beam_size >= 2, generate switches to deterministic beam search: maintain beam_size top-log-prob hypotheses per step, expand each by its top beam_size next-token candidates, globally keep the best beam_size survivors. Temperature / top_k / top_p are ignored (beam is deterministic). Repetition penalty, stop sequences, grammar masks, and all early-exit settings still apply — per-beam.

Cache management

Each beam owns its own RuntimeKVCache. At each step, decoding a beam's last token leaves its cache in exactly the state a child beam needs as its starting state. When multiple candidates select the same parent, the first child takes ownership of the parent's cache; subsequent siblings call clone_runtime_kv_cache_prefix. Parents that don't donate to any child are freed.

Peak cache VRAM: beam_size × cache_bytes. At beam=4, D=2048, seq=512, L=10 that's ~320 MB. Beam ≥ 16 gets expensive fast.

Length normalization

Final beams are ranked by log_prob / max(1, new_len)^length_penalty:

• length_penalty = 1.0 → unnormalised log-prob (default; beam will prefer short outputs).
• length_penalty = 0.6–0.8 → mild preference for longer outputs (typical for translation / summarisation).
• length_penalty > 1.0 → explicit preference for shorter outputs.

Measured

On H100 / 407M GPTQ-INT4 / prefix=16 / max=12 / L=4:

greedy:      [11, 564, 250, 464, 968, 1971, 3782, 447, ...]
beam_size=3: [11, 564, 250,  40, 836,  447,  247,  83, ...]

Beam diverges from greedy at position 3 — exactly the case beam search is designed for: the deeper lookahead exposes a higher-cumulative-log-prob trajectory that greedy's local argmax missed.

Example

# Translation-style decoding with mild long-output preference.
tokens = model.generate(
    prompt, max_tokens=128, loops=8,
    beam_size=4, length_penalty=0.7, early_stopping=True,
)

See tests/test_beam_search.py for a ready-made parity + length-penalty sweep.

Constrained decoding

Pass a compiled tl.Grammar via the grammar kwarg and every sampling step will mask out tokens that don't fit the grammar. Works under greedy, sampling, and beam search — the mask is applied in logit space, so temperature / top_k / top_p ignore the illegal tokens naturally.

import tinyloop_py as tl

g = tl.Grammar(r"\{\"name\":\"[a-z]+\"\}")
g.set_token_vocabulary(token_bytes)  # one bytes object per token id
out = model.generate(prompt, grammar=g, max_tokens=32, temperature=0.0)
# out decodes to a string shaped like '{"name":"abc"}'

See the dedicated Constrained decoding page for:

• Supported regex syntax
• Building the token_bytes map from a Hugging Face GPT-2 tokenizer
• Composition with beam search
• JSON / function-calling patterns
• Common gotchas (BPE tokens with leading spaces, special tokens)

Model.generate_stream(...)

Streaming version of generate with a per-token callback and a timing breakdown.

result = model.generate_stream(
    prompt,
    on_token=lambda tok: print(tok, end="", flush=True) or True,
    max_tokens=64,
    loops=8,
    temperature=0.8,
    top_k=50,
    cache_window=0,
    repetition_penalty=1.3,
    stop_sequences=[],
)
# result = {"tokens": List[int],
#           "prefill_ms": float, "first_token_ms": float,
#           "decode_steady_ms": float, "tokens_emitted": int}

Parameters

Same as generate, plus:

Name	Type	Default	Description
on_token	Callable[[int], bool]	—	Invoked synchronously once per emitted token. Returning False aborts generation after that token is included (useful for UI cancel buttons, early-stop heuristics in chat). The GIL is briefly acquired for the call; keep the callback work short.

Returns

A dict with:

Key	Type	Meaning
tokens	List[int]	Full token stream including prompt. Same shape as generate's return.
prefill_ms	float	Wall-clock time spent on the prompt prefill (setting up the KV cache). Dominated by quantized GEMMs.
first_token_ms	float	Wall-clock time from the start of the call to the first on_token callback. Includes prefill + first decode step + sampler.
decode_steady_ms	float	Wall-clock time for tokens 2..N inclusive (exclusive of prefill). Divide by tokens_emitted - 1 for per-token steady-state decode latency.
tokens_emitted	int	Count of new tokens generated (does not include the prompt).

Use cases

• Chat-style UIs where first-token latency needs to be reported separately from steady-state decode rate.
• Interactive CLIs with live token echoing.
• Streaming wrappers around external protocols (SSE, WebSocket, gRPC server-streaming) — the callback bridges the TinyLoop sync loop to the external async sink.

Timing detail

The three latency fields partition the wall clock as:

• first_token_ms = prefill + first decode iter + sampler + first callback
• decode_steady_ms = sum over decode iters 2..N of (loop block × loops + sampler + callback)

So the total wall clock of the call is approximately first_token_ms + decode_steady_ms. Minor overhead for the on_token callback is counted in whichever window contains it.

Example: cancellable streaming

import queue, threading

output_q = queue.Queue()
cancel = threading.Event()

def on_tok(tok):
    output_q.put(tok)
    return not cancel.is_set()

# Runs in worker thread; main thread consumes from output_q or sets cancel
t = threading.Thread(target=lambda: model.generate_stream(
    prompt, on_tok, max_tokens=256, loops=8))
t.start()

Model.generate_speculative(...)

Self-speculative decoding — the same checkpoint acts as draft and verifier at different loop depths.

result = model.generate_speculative(
    prompt,
    max_tokens=64,
    draft_loops=2,
    verify_loops=8,
    draft_ahead=4,
    temperature=0.0,
    top_k=50,
    top_p=0.9,
    cache_window=0,
)

tokens = result["tokens"]

Parameters

Name	Type	Default	Description
prompt	np.ndarray[int32]	—	As generate.
max_tokens	int	64	As generate.
draft_loops	int	2	Loop depth for the draft forward. Shorter = faster draft, lower accept rate.
verify_loops	int	8	Loop depth for the verification forward. The verified output is indistinguishable from generate(prompt, loops=verify_loops) under greedy decoding.
draft_ahead	int	4	Number of speculative tokens the draft proposes per cycle. Longer = more parallelism when the draft is accurate, more rework when it is not.
temperature, top_k, top_p, cache_window			As generate. Temperature applies to the verifier's sampling distribution.

Returns

{"tokens": List[int]}

Current Python binding returns only tokens (no accept_rate, draft_calls, verify_calls fields yet).

The earlier wiki version that listed those extra fields was aspirational; the shipped binding in python/tinyloop_py.cpp currently exposes only {"tokens": ...} for this API.

When to use

Only when you are intentionally testing self-speculative decoding behavior. On current TinyLoop builds the speculative path has known regressions in some configurations; broader serving-grade validation is on the production roadmap. See runtime modes → speculate for current limitations.

Internal flow and trade-offs

Speculative decode alternates draft/verify rounds:

1. draft proposes up to draft_ahead tokens at draft_loops
2. verifier scores at verify_loops
3. each drafted token is accepted or replaced (residual resampling path under stochastic decode)

Trade-off envelope:

• lower draft_loops → faster draft, typically lower acceptance
• higher draft_ahead → better amortization when acceptance is high, more wasted work when low
• verify path dominates quality and often latency once acceptance drops

Model.generate_tree_speculative(...)

Tree speculative decoding — architecture-exclusive primitive that replaces the K sequential verify decodes of chain spec with one tree-mask-gated forward pass. Zero auxiliary parameters (no Medusa / EAGLE heads); uses the weight-shared loop to run draft and verify on the same checkpoint at different depths.

result = model.generate_tree_speculative(
    prompt,
    max_tokens=64,
    draft_loops=2,
    verify_loops=8,
    draft_ahead=2,      # depth of each branch
    K_branches=2,       # 1 = pure chain; >1 = top-K multi-branch tree
    temperature=0.0,    # only greedy is wired today
    top_k=50,
    top_p=0.9,
    cache_window=0,
)

tokens = result["tokens"]

Parameters

Name	Type	Default	Description
prompt	np.ndarray[int32]	—	As generate.
max_tokens	int	64	Soft cap on new tokens — may overshoot by up to draft_ahead.
draft_loops	int	2	Loop depth for the draft forward.
verify_loops	int	8	Loop depth for the verify forward.
draft_ahead	int	4	Depth of each branch.
K_branches	int	1	Number of parallel draft branches. K_branches=1 is chain mode (one linear speculation). K_branches >= 2 samples top-K from the draft's root distribution and chain-extends each — this is where the "tree" structure kicks in. K_branches * draft_ahead <= 64.
temperature, top_k, top_p, cache_window			As generate. Only temperature = 0 is implemented today; sampling is a follow-up.

Returns

{"tokens": [...]}

Guarantees

Under greedy decoding (temperature = 0), the output token sequence is bit-exact to model.generate(prompt, loops=verify_loops, temperature=0). This holds for both chain (K_branches=1) and multi-branch (K_branches >= 2) modes. The speculative choice of drafts changes which forwards we batch, never what tokens come out.

When to use

Today, primarily for the correctness guarantee and for experimenting with draft / verify loop depths. The wall-clock speedup over generate_speculative depends on draft-verify agreement rate at the root position (how often the top-K draft includes the verify's argmax). On well-trained checkpoints this tends to be high, giving a measurable tree-over-chain win; on under-trained or mismatched checkpoints, tree mode can be slower than chain because extra drafting cost doesn't pay back in extra accepts.

Limitations

• Greedy only. temperature > 0 is rejected at the API boundary.
• Standard FP16 KV cache only. TINYLOOP_KV_H_MODE / TINYLOOP_KV_INT8 / TINYLOOP_KV_INT4 are rejected.
• No ring-buffer wrap in the verify kernel (requires start_index = 0).
• Kernel caps K_branches * draft_ahead <= 64.

Architectural note

Self-speculative decoding is exclusive to weight-shared looped transformers at single-checkpoint deployment. Standard decoders need a separate smaller draft model (Medusa, EAGLE) and its own training — same behaviour costs you a second checkpoint plus the training budget to align it. TinyLoop gets draft + verifier from the same weights at different L.

Common patterns

Greedy decode with early exit

out = model.generate(
    prompt, max_tokens=32, loops=16, temperature=0.0,
    early_exit_patience=1, early_exit_min_L=4, early_exit_peek_every=2,
    early_exit_logit_margin=3.0,
)

The logit-margin mode fires on well-trained models and typically gives −10 to −13 % decode at no visible quality loss.

Top-p sampling with repetition penalty

out = model.generate(
    prompt, max_tokens=128, loops=8,
    temperature=0.8, top_k=50, top_p=0.9,
    repetition_penalty=1.15,
)

repetition_penalty=1.15 is a moderate setting that suppresses verbatim repeats without flattening the distribution. Apply higher values (1.3) for workloads with strong repetition bias, but watch for degeneration where the model avoids correct answers because they overlap with the prompt.

Streaming with timing breakdown

r = model.generate_stream(prompt, on_token=lambda t: True,
                          max_tokens=64, loops=8, temperature=0.0)
print(f"TTFT {r['first_token_ms']:.1f} ms   "
      f"decode {(r['decode_steady_ms'] / (r['tokens_emitted'] - 1)):.1f} ms/tok")