Skip to main content

Config Reference

This page documents the public configuration types in include/tinyloop.h as runtime contracts, not just field lists. The goal is to explain how each field changes execution, memory behavior, and output semantics.

Where configs are consumed​

  • Model load path: load_model(path, max_seq_len, prefill_chunk)
  • Scoring paths: score* family (primarily n_loops)
  • Decode paths: generate* family (GenerateConfig, SpeculativeConfig, TreeSpeculativeConfig)
  • Cache reuse paths: prefix cache / warm-start / prefix pool

Most configs are evaluated at call-time and can vary per request. A few (notably model load sizing and env-controlled cache modes) define long-lived process state.

ModelConfig (artifact header)​

struct ModelConfig {
    uint32_t dim;
    uint32_t n_heads;
    uint32_t ffn_dim;
    uint32_t vocab_size;
    uint32_t embed_factor_dim;
    uint32_t n_pre_blocks;
    uint32_t default_loops;
    bool     use_rope = false;
    float    rope_theta = 10000.0f;
};

ModelConfig is read from the .tinyloop artifact and describes structural model shape.

Execution impact​

  • dim, n_heads, ffn_dim determine all projection widths and attention layout.
  • n_pre_blocks + loops determines active cache layer count at runtime.
  • default_loops is the suggested baseline depth, but callers can override loops per request.
  • use_rope enables rotary position embeddings on Q and K vectors after QKV projection. When enabled, the runtime precomputes cos/sin frequency tables at load time and applies them in all forward paths (prefill, decode, chunked prefill, tree-verify, batched decode). Set via TINYLOOP_USE_ROPE=1.
  • rope_theta controls the RoPE base frequency (default 10000.0, matching Llama-2). Override via TINYLOOP_ROPE_THETA=N for models trained with different theta values (e.g. Llama-3 uses 500000.0).
  • post_norm switches from pre-norm (LayerNorm before attention/MLP, the default) to post-norm (LayerNorm after). Set via TINYLOOP_POST_NORM=1.
  • use_gelu switches from SwiGLU (the default) to GeLU activation in the MLP. For GPT-2/BERT-style models. Set via TINYLOOP_USE_GELU=1.

Why this separation exists​

Artifact shape is immutable per model file, while runtime depth (loops) remains a request-time control knob.

InferenceConfig​

struct InferenceConfig {
    int n_loops = 8;
    int max_seq_len = 2048;
};

A light helper type; most production flows call APIs directly with explicit args/config structs.

GenerateConfig​

struct GenerateConfig {
    int n_loops = 8;
    int max_tokens = 128;
    float temperature = 1.0f;
    int top_k = 50;
    float top_p = 0.9f;
    int eos_token = 50256;
    int cache_window = 0;

    float repetition_penalty = 1.0f;
    std::vector<std::vector<int32_t>> stop_sequences;

    int early_exit_patience = 0;
    int early_exit_min_L = 2;
    float early_exit_tau = 1e-3f;
    int early_exit_peek_every = 1;
    float early_exit_logit_margin = 0.0f;

    int beam_size = 0;
    float length_penalty = 1.0f;
    bool early_stopping = true;

    uint64_t sample_seed = 0;
    Grammar* grammar = nullptr;
};

How fields map to runtime phases​

Depth and length controls​

  • n_loops: compute depth per token (quality/latency axis)
  • max_tokens: decode horizon
  • cache_window: memory bound for KV cache (0 = full cache, >0 ring-buffer)

Sampling controls​

  • temperature, top_k, top_p apply to stochastic path
  • temperature <= 0.01 behaves as greedy
  • sample_seed != 0 enables reproducible RNG stream

Logit-space post-processing​

  • repetition_penalty is applied before temperature/top-k/top-p
  • grammar mask is then applied as hard legality filter
  • resulting logits feed sampling/argmax

Termination controls​

  • eos_token
  • stop_sequences tail match
  • early exit family (confidence-based loop truncation)

Search-mode controls​

  • beam_size > 1 switches from sampling decode to deterministic beam path
  • length_penalty and early_stopping apply only in beam mode

Internal state transitions affected by config​

Within non-beam decode (generate_stream core path):

  1. allocate cache with layer count n_pre_blocks + n_loops
  2. prefill once at n_loops
  3. for each token:
    • score last hidden
    • apply penalties/masks
    • sample
    • append decode cache rows

When early exit is enabled, effective loop depth can shrink monotonically during generation (effective_loops clamps down after stable probes).

Trade-offs​

  • larger n_loops improves quality envelope but linearly increases per-token compute
  • smaller cache_window lowers VRAM but truncates long-context memory
  • beam mode improves search but multiplies cache memory and compute
  • grammar constraints improve output validity but can force early stop on dead-end states

SpeculativeConfig​

struct SpeculativeConfig {
    int draft_loops = 2;
    int verify_loops = 8;
    int draft_ahead = 4;
    int max_tokens = 128;
    float temperature = 0.0f;
    int top_k = 50;
    float top_p = 0.9f;
    int eos_token = 50256;
    int cache_window = 0;
};

Runtime meaning​

  • draft uses draft_loops
  • verify uses verify_loops
  • draft_ahead controls per-round speculative depth

Trade-off surface​

  • lower draft depth + larger ahead window increases potential speedup when acceptance is high
  • low acceptance shifts cost back toward verifier and can erase gains

TreeSpeculativeConfig​

struct TreeSpeculativeConfig {
    int draft_loops  = 2;
    int verify_loops = 8;
    int draft_ahead  = 4;
    int K_branches   = 1;
    int max_tokens   = 128;
    float temperature = 0.0f;
    int top_k = 50;
    float top_p = 0.9f;
    int eos_token = 50256;
    int cache_window = 0;
};

Runtime meaning​

  • K_branches=1 behaves as chain-like path
  • K_branches>=2 enables multi-branch tree verify
  • verify is performed through tree-mask-gated forward path

Practical limitations​

  • strongest parity claims are in greedy mode
  • mode composition depends on cache mode and kernel guards

Load-time sizing configs (load_model)​

max_seq_len​

Defines sequence budget for persistent runtime buffers and upper bounds decode/prefill capacity.

prefill_chunk​

When >0, scratch buffers are chunk-sized instead of seq-sized.

  • reduces VRAM pressure for long context
  • increases prefill control-flow overhead
  • score-style APIs must obey chunk constraints

Assumptions and boundaries​

  • Tokenization and text-format concerns are intentionally out-of-scope for these structs.
  • Some operational behavior is env-controlled (TINYLOOP_*) and composes with these structs.
  • If struct defaults differ from wrapper defaults (CLI/Python/server), wrapper docs should be treated as integration-specific overrides.