Config Reference

This page documents the public configuration types in include/tinyloop.h as runtime contracts, not just field lists. The goal is to explain how each field changes execution, memory behavior, and output semantics.

Where configs are consumed

• Model load path: load_model(path, max_seq_len, prefill_chunk)
• Scoring paths: score* family (primarily n_loops)
• Decode paths: generate* family (GenerateConfig, SpeculativeConfig, TreeSpeculativeConfig)
• Cache reuse paths: prefix cache / warm-start / prefix pool

Most configs are evaluated at call-time and can vary per request. A few (notably model load sizing and env-controlled cache modes) define long-lived process state.

ModelConfig (artifact header)

struct ModelConfig {
    uint32_t dim;
    uint32_t n_heads;
    uint32_t ffn_dim;
    uint32_t vocab_size;
    uint32_t embed_factor_dim;
    uint32_t n_pre_blocks;
    uint32_t default_loops;
    bool     use_rope = false;
    float    rope_theta = 10000.0f;
};

ModelConfig is read from the .tinyloop artifact and describes structural model shape.

Execution impact

• dim, n_heads, ffn_dim determine all projection widths and attention layout.
• n_pre_blocks + loops determines active cache layer count at runtime.
• default_loops is the suggested baseline depth, but callers can override loops per request.
• use_rope enables rotary position embeddings on Q and K vectors after QKV projection. When enabled, the runtime precomputes cos/sin frequency tables at load time and applies them in all forward paths (prefill, decode, chunked prefill, tree-verify, batched decode). Set via TINYLOOP_USE_ROPE=1.
• rope_theta controls the RoPE base frequency (default 10000.0, matching Llama-2). Override via TINYLOOP_ROPE_THETA=N for models trained with different theta values (e.g. Llama-3 uses 500000.0).
• post_norm switches from pre-norm (LayerNorm before attention/MLP, the default) to post-norm (LayerNorm after). Set via TINYLOOP_POST_NORM=1.
• use_gelu switches from SwiGLU (the default) to GeLU activation in the MLP. For GPT-2/BERT-style models. Set via TINYLOOP_USE_GELU=1.

Why this separation exists

Artifact shape is immutable per model file, while runtime depth (loops) remains a request-time control knob.

InferenceConfig

struct InferenceConfig {
    int n_loops = 8;
    int max_seq_len = 2048;
};

A light helper type; most production flows call APIs directly with explicit args/config structs.

GenerateConfig

struct GenerateConfig {
    int n_loops = 8;
    int max_tokens = 128;
    float temperature = 1.0f;
    int top_k = 50;
    float top_p = 0.9f;
    int eos_token = 50256;
    int cache_window = 0;

    float repetition_penalty = 1.0f;
    std::vector<std::vector<int32_t>> stop_sequences;

    int early_exit_patience = 0;
    int early_exit_min_L = 2;
    float early_exit_tau = 1e-3f;
    int early_exit_peek_every = 1;
    float early_exit_logit_margin = 0.0f;

    int beam_size = 0;
    float length_penalty = 1.0f;
    bool early_stopping = true;

    uint64_t sample_seed = 0;
    Grammar* grammar = nullptr;
};

How fields map to runtime phases

Depth and length controls

• n_loops: compute depth per token (quality/latency axis)
• max_tokens: decode horizon
• cache_window: memory bound for KV cache (0 = full cache, >0 ring-buffer)

Sampling controls

• temperature, top_k, top_p apply to stochastic path
• temperature <= 0.01 behaves as greedy
• sample_seed != 0 enables reproducible RNG stream

Logit-space post-processing

• repetition_penalty is applied before temperature/top-k/top-p
• grammar mask is then applied as hard legality filter
• resulting logits feed sampling/argmax

Termination controls

• eos_token
• stop_sequences tail match
• early exit family (confidence-based loop truncation)

Search-mode controls

• beam_size > 1 switches from sampling decode to deterministic beam path
• length_penalty and early_stopping apply only in beam mode

Internal state transitions affected by config

Within non-beam decode (generate_stream core path):

1. allocate cache with layer count n_pre_blocks + n_loops
2. prefill once at n_loops
3. for each token:
   • score last hidden
   • apply penalties/masks
   • sample
   • append decode cache rows

When early exit is enabled, effective loop depth can shrink monotonically during generation (effective_loops clamps down after stable probes).

Trade-offs

• larger n_loops improves quality envelope but linearly increases per-token compute
• smaller cache_window lowers VRAM but truncates long-context memory
• beam mode improves search but multiplies cache memory and compute
• grammar constraints improve output validity but can force early stop on dead-end states

SpeculativeConfig

struct SpeculativeConfig {
    int draft_loops = 2;
    int verify_loops = 8;
    int draft_ahead = 4;
    int max_tokens = 128;
    float temperature = 0.0f;
    int top_k = 50;
    float top_p = 0.9f;
    int eos_token = 50256;
    int cache_window = 0;
};

Runtime meaning

• draft uses draft_loops
• verify uses verify_loops
• draft_ahead controls per-round speculative depth

Trade-off surface

• lower draft depth + larger ahead window increases potential speedup when acceptance is high
• low acceptance shifts cost back toward verifier and can erase gains

TreeSpeculativeConfig

struct TreeSpeculativeConfig {
    int draft_loops  = 2;
    int verify_loops = 8;
    int draft_ahead  = 4;
    int K_branches   = 1;
    int max_tokens   = 128;
    float temperature = 0.0f;
    int top_k = 50;
    float top_p = 0.9f;
    int eos_token = 50256;
    int cache_window = 0;
};

Runtime meaning

• K_branches=1 behaves as chain-like path
• K_branches>=2 enables multi-branch tree verify
• verify is performed through tree-mask-gated forward path

Practical limitations

• strongest parity claims are in greedy mode
• mode composition depends on cache mode and kernel guards

Load-time sizing configs (load_model)

max_seq_len

Defines sequence budget for persistent runtime buffers and upper bounds decode/prefill capacity.

prefill_chunk

When >0, scratch buffers are chunk-sized instead of seq-sized.

• reduces VRAM pressure for long context
• increases prefill control-flow overhead
• score-style APIs must obey chunk constraints

Assumptions and boundaries

• Tokenization and text-format concerns are intentionally out-of-scope for these structs.
• Some operational behavior is env-controlled (TINYLOOP_*) and composes with these structs.
• If struct defaults differ from wrapper defaults (CLI/Python/server), wrapper docs should be treated as integration-specific overrides.