Scoring APIs

Scoring calls execute forward computation without autoregressive sampling loops. They are the C++ entry points for evaluation, ranking, uncertainty estimation, and any workload where you want logits without a decode loop.

Function	Purpose
`score`	Full `[seq_len, vocab]` logits at a fixed loop depth.
`score_last_token`	Final-position logits only `[vocab]` — lowest memory traffic.
`score_logit_lens`	Per-loop-depth final logits `[n_loops, vocab]` for depth-evolution analysis.
`score_with_uncertainty`	Two-depth forward with KL uncertainty signal per token.
`score_with_consistency_escalation`	Adaptive-depth forward: escalate loop count when low-depth disagrees with high.

Unless otherwise noted, every scoring function in this section:

Is defined in include/tinyloop.h under namespace tinyloop.
Mutates internal Model working buffers (buffers.main, buffers.norm, scratch); concurrent invocation on the same Model* is unsafe without external serialization.
Does not advance any decode cache. No RuntimeKVCache is allocated or appended to unless the API explicitly uses cache helpers.
Aborts with a stderr diagnostic on invariant violation (null pointer, out-of-range dimensions, GPU OOM). Callers should treat these as fatal: the runtime does not unwind.
Writes results to a caller-allocated output buffer; no heap allocation is done for the logits themselves.

`score`

void score(Model* model,
           const int32_t* tokens,
           int seq_len,
           float* logits_out,
           int n_loops = 8);

Compute the full [seq_len, vocab_size] logit tensor for a given prompt at a fixed loop depth.

Parameters

Name	Type	Default	Description
`model`	`Model*`	—	Loaded model pointer. Must not be `nullptr`.
`tokens`	`const int32_t*`	—	Token-id array, length `seq_len`. Token ids must satisfy `0 <= t < vocab_size`.
`seq_len`	`int`	—	Number of tokens to score. Must satisfy `1 <= seq_len <= max_seq_len`.
`logits_out`	`float*`	—	Output buffer, contiguous, size `seq_len * vocab_size * sizeof(float)`. Caller allocates; no pre-zeroing required.
`n_loops`	`int`	`8`	Number of applications of the shared loop block. Must satisfy `1 <= n_loops <= 64`. Higher values improve quality up to saturation at the cost of linear compute.

Returns

void. Results are written to logits_out in row-major order: element (t, v) at offset t * vocab_size + v.

Raises / Error conditions

The runtime writes a message to stderr and aborts on:

model == nullptr.
seq_len < 1 or seq_len > max_seq_len passed at load_model.
n_loops < 1 or n_loops > 64.
GPU out-of-memory during prefill or output buffer allocation for internal scratch.

Notes

Memory footprint. score has the largest output footprint of any scoring variant. At vocab_size = 50257 and seq_len = 2048, the output tensor alone is ~393 MB in FP32. For decode hot paths prefer score_last_token.
Compute dominates. For typical shapes (d = 2048, L = 8, seq_len = 1024) the loop-block GEMMs account for >90 % of the wall-clock; the head projection is a second-order cost.
prefill_chunk interaction. When the model was loaded with prefill_chunk > 0, score is constrained by the chunk row budget: calls with seq_len larger than the chunk size execute in multiple passes, keeping peak VRAM low but trading off kernel-launch overhead.

Example

std::vector<int32_t> tokens = tokenize("The quick brown fox");
std::vector<float> logits(tokens.size() * vocab_size);
tinyloop::score(model, tokens.data(), tokens.size(), logits.data(), /*n_loops=*/8);
// logits[(t * vocab_size) + v] is the unnormalized logit for token v at position t

`score_last_token`

void score_last_token(Model* model,
                      const int32_t* tokens,
                      int seq_len,
                      float* logits_out,
                      int n_loops = 8);

Compute logits only at the final token position. Use this for ranking, scoring candidate continuations, and decode helpers where per-position logits are wasted.

Parameters

Name	Type	Default	Description
`model`	`Model*`	—	Loaded model pointer. Must not be `nullptr`.
`tokens`	`const int32_t*`	—	Token-id array, length `seq_len`.
`seq_len`	`int`	—	Number of tokens, `1 <= seq_len <= max_seq_len`.
`logits_out`	`float*`	—	Output buffer, size `vocab_size * sizeof(float)`.
`n_loops`	`int`	`8`	Shared-block iterations, `1 <= n_loops <= 64`.

Returns

void. logits_out[v] is the unnormalized logit for token v at the last position.

Raises / Error conditions

Same error paths as score. Out-of-range n_loops, null model, or seq_len outside [1, max_seq_len] is fatal.

Notes

Memory win. Output buffer is vocab_size floats (a few hundred KB), vs seq_len * vocab_size for score. This is the single biggest lever for reducing host↔device transfer in scoring workloads.
Same forward cost as score (the savings are purely on output shape and post-forward copies); if you need both full logits and a final-token shortcut, run score once rather than both.
Commonly used to implement custom decode loops outside TinyLoop's built-in Generation APIs.

Example

std::vector<float> next_logits(vocab_size);
tinyloop::score_last_token(model, prompt.data(), prompt.size(), next_logits.data(), 8);
int next_token = std::distance(next_logits.begin(),
                               std::max_element(next_logits.begin(), next_logits.end()));

`score_logit_lens`

void score_logit_lens(Model* model,
                      const int32_t* tokens,
                      int seq_len,
                      float* logits_out,
                      int n_loops = 8);

Produce final-token logits at every loop depth from 1 through n_loops, laid out as [n_loops, vocab_size]. This is the logit-lens primitive: it lets callers observe how the model's prediction evolves as more loop iterations are applied.

Parameters

Name	Type	Default	Description
`model`	`Model*`	—	Loaded model pointer.
`tokens`	`const int32_t*`	—	Token ids, length `seq_len`.
`seq_len`	`int`	—	`1 <= seq_len <= max_seq_len`.
`logits_out`	`float*`	—	Output buffer, size `n_loops * vocab_size * sizeof(float)`.
`n_loops`	`int`	`8`	Upper loop depth. Logits are written for depths `1, 2, ..., n_loops`.

Returns

void. logits_out[(l-1) * vocab_size + v] is the last-position logit for token v after l loop iterations, for l = 1, ..., n_loops.

Raises / Error conditions

Same as score. Additionally, an n_loops of 0 is treated as invalid (at least one depth must be requested).

Notes

Cost is superlinear in n_loops. The underlying implementation does not memoize hidden states across depths in all configurations; treat this as an analysis primitive, not a hot-path serving primitive.
Pairs with score_with_uncertainty. Logit-lens output is the raw material for KL-based uncertainty heuristics.
Depth-evolution analysis on a trained looped model typically shows non-monotonic argmax trajectories — a token can appear "decided" at l=3, flip at l=4, and restabilize by l=8. See the project paper for discussion.

`score_with_uncertainty`

void score_with_uncertainty(Model* model,
                            const int32_t* tokens,
                            int seq_len,
                            float* logits_hi_out,
                            float* kl_out,
                            int L_lo = 4,
                            int L_hi = 8);

Run two forward passes at different loop depths (L_lo and L_hi) and emit both the high-depth logits and the per-position KL divergence between the two distributions as an uncertainty signal.

Parameters

Name	Type	Default	Description
`model`	`Model*`	—	Loaded model pointer.
`tokens`	`const int32_t*`	—	Token ids, length `seq_len`.
`seq_len`	`int`	—	`1 <= seq_len <= max_seq_len`.
`logits_hi_out`	`float*`	—	Output buffer for `L_hi` logits, shape `[seq_len, vocab_size]`.
`kl_out`	`float*`	—	Output buffer for per-position KL divergence, shape `[seq_len]` floats. Each value is `KL(softmax(logits_hi) \|\| softmax(logits_lo))`.
`L_lo`	`int`	`4`	Low-depth loop count. `1 <= L_lo < L_hi`.
`L_hi`	`int`	`8`	High-depth loop count. `L_lo < L_hi <= 64`.

Returns

void. High-depth logits in logits_hi_out; per-position KL values in kl_out.

Raises / Error conditions

Same as score, plus:

L_lo >= L_hi is invalid.
Either L_lo or L_hi outside [1, 64] is invalid.

Notes

Two forward passes. Compute cost is approximately L_lo + L_hi loop-block applications vs. L_hi for a single-depth score. Budget for the extra compute.
KL is high-to-low. By convention this call reports KL(p_hi || p_lo) — high KL means the low-depth distribution is "surprised" by the high-depth one, which is a useful "needs more thinking" signal.
For workloads that want adaptive compute without explicit KL inspection, see score_with_consistency_escalation.

`score_with_consistency_escalation`

void score_with_consistency_escalation(Model* model,
                                       const int32_t* tokens,
                                       int seq_len,
                                       float* logits_out,
                                       int* escalated_out,
                                       int* L_used_out,
                                       int L_lo = 8,
                                       int L_hi = 16,
                                       int escalate_L = 32);

Adaptive-depth scoring: first run at L_lo. For each position, if a cheap consistency check against L_hi disagrees, rerun only that position at escalate_L. Intended for workloads that want high quality on hard tokens without paying escalate_L cost everywhere.

Parameters

Name	Type	Default	Description
`model`	`Model*`	—	Loaded model pointer.
`tokens`	`const int32_t*`	—	Token ids, length `seq_len`.
`seq_len`	`int`	—	`1 <= seq_len <= max_seq_len`.
`logits_out`	`float*`	—	Output buffer, shape `[seq_len, vocab_size]`. Contains the logits actually used per position (`L_lo` if consistent, `escalate_L` if escalated).
`escalated_out`	`int*`	—	Output buffer, shape `[seq_len]`. Written as `0`/`1` per position: `1` if position was escalated to `escalate_L`, `0` otherwise.
`L_used_out`	`int*`	—	Output buffer, shape `[seq_len]`. Written as the loop depth actually used at each position.
`L_lo`	`int`	`8`	First-pass loop depth.
`L_hi`	`int`	`16`	Consistency-check loop depth. Must satisfy `L_lo < L_hi`.
`escalate_L`	`int`	`32`	Loop depth used for escalated positions. Must satisfy `L_hi < escalate_L <= 64`.

Returns

void. Primary logits in logits_out; per-position escalation flags in escalated_out; per-position loop depth used in L_used_out.

Raises / Error conditions

Same as score, plus:

Any of L_lo, L_hi, escalate_L outside [1, 64].
Ordering violation: must have L_lo < L_hi < escalate_L.

Notes

Worst-case cost is (L_hi + escalate_L) * seq_len loop-block applications if every position escalates. In practice, escalation rate is usually low (under 20 %) on well-trained models.
Non-determinism is inherited from sampling-free forward: the escalation decision is deterministic given tokens, L_lo, L_hi, but the practical escalation rate depends on the model.
The check is on argmax agreement between L_lo and L_hi distributions at each position; richer consistency signals (full KL) are available via score_with_uncertainty.

State transitions in the scoring path

For all score variants:

Token embedding into model buffer space.
Pre-block execution (n_pre_blocks).
Shared loop block repeated n_loops times.
Final LN + head projection according to requested output mode.

No decode cache advancement occurs unless an API explicitly uses cache helpers. The model's buffers.main is overwritten each call; callers who want the residual stream preserved across scoring calls must snapshot it via lower-level APIs (not part of the public scoring surface).

Performance model

Output mode	Output bytes	Forward cost	Typical use
`score`	`seq_len * vocab * 4`	1× forward	Offline evaluation
`score_last_token`	`vocab * 4`	1× forward	Decode helpers, ranking
`score_logit_lens`	`n_loops * vocab * 4`	~1.5× forward	Interpretability analysis
`score_with_uncertainty`	`(seq_len + 1) * vocab * 4`	~2× forward	Active learning, RLHF scoring
`score_with_consistency_escalation`	`seq_len * vocab * 4 + 2 * seq_len * 4`	`L_hi + β * escalate_L` forward	Production adaptive-compute

β is the empirical escalation rate; on a well-trained 1B-effective model it is typically 0.05 to 0.20.

Trade-offs and caveats

The API is token-id based; tokenizer semantics are external to TinyLoop.
Consistency/uncertainty APIs add extra forward passes by design — they trade throughput for an adaptive quality signal. Treat them as opt-in.
For prefill_chunk > 0 configurations, score-style APIs must obey the chunk row budget (see Lifecycle and Memory Model).
Output buffer ownership is caller-managed throughout — TinyLoop never allocates the logit tensor itself.

score​

Parameters​

Returns​

Raises / Error conditions​

Notes​

Example​

See also​

score_last_token​

Parameters​

Returns​

Raises / Error conditions​

Notes​

Example​

score_logit_lens​

Parameters​

Returns​

Raises / Error conditions​

Notes​

score_with_uncertainty​

Parameters​

Returns​

Raises / Error conditions​

Notes​

score_with_consistency_escalation​

Parameters​

Returns​

Raises / Error conditions​

Notes​

State transitions in the scoring path​

Performance model​

Trade-offs and caveats​

`score`

Parameters

Returns

Raises / Error conditions

Notes

Example

See also

`score_last_token`

Parameters

Returns

Raises / Error conditions

Notes

Example

`score_logit_lens`

Parameters

Returns

Raises / Error conditions

Notes

`score_with_uncertainty`

Parameters

Returns

Raises / Error conditions

Notes

`score_with_consistency_escalation`

Parameters

Returns

Raises / Error conditions

Notes

State transitions in the scoring path

Performance model

Trade-offs and caveats