Skip to main content

Runtime Architecture

TinyLoop has a deliberately narrow architecture. It is easier to understand if you separate it into model loading, inference orchestration, generation, kernels, and validation.

1. Model Loading​

Primary implementation:

  • src/model.cpp
  • include/model.h

Responsibilities:

  • parse .tinyloop headers
  • validate dimensions and compatibility constraints
  • upload tensors to GPU
  • allocate runtime buffers
  • report weight and buffer memory

Important current behavior:

  • tensors can stream from disk to GPU in chunks instead of requiring full host copies
  • optional buffers such as logit_row are lazy
  • FP16 body caches can be allocated at load time when TINYLOOP_EXPERIMENTAL_FP16_BODY=1

2. Buffer Pool​

The runtime reuses a small number of persistent buffers:

  • main: FP32 persistent hidden state (or main_fp16 when TINYLOOP_FP16_RESIDUAL=1)
  • norm: FP16 layernorm output
  • scratch1: shared QKV or tiled-MLP workspace
  • scratch2: optional FP16-body MLP temp
  • attn_down: shared attention-output / down-projection buffer
  • logit_row: lazily allocated final-row logits buffer

The res_* dispatch helpers (res_layernorm, res_add, res_zero, etc.) abstract the FP32/FP16 branching so each call site doesn't need manual if/else.

That buffer strategy is central to TinyLoop's memory profile.

3. Inference Orchestration​

Primary implementation:

  • src/inference.cpp

Responsibilities:

  • embedding path selection
  • pre-block execution
  • shared loop-block execution
  • final norm
  • logits materialization
  • hidden-trace and profiling helpers

Core execution shape:

embed
-> pre.0
-> pre.1
-> loop block x L
-> final norm
-> optional logits

Important split:

  • full scoring uses full-sequence paths
  • generation-style workloads can use score_last_token(...)

4. KV Cache​

Primary implementation:

  • include/kv_cache.h
  • cuda/kv_cache.cu
  • src/inference.cpp
  • src/generate.cpp

Current capabilities:

  • prefill-time cache population
  • single-token decode against cached K/V
  • default-on cached generation
  • sliding-window cache via cache_window
  • prefix-cache reuse
  • shared-cache speculative runtime

Current missing pieces:

  • paged attention
  • KV-cache quantization
  • broader serving-oriented batching behavior

5. Generation Layer​

Primary implementation:

  • src/generate.cpp

Responsibilities:

  • prompt validation
  • sampling
  • EOS handling
  • prefix-cache helpers
  • speculative draft/verify orchestration

Current generation split:

  • generate(...) for ordinary autoregressive output
  • generate_speculative(...) for self-speculative decode
  • prefix-cache helpers for repeated shared prompts

6. CUDA Kernels​

Primary implementation:

  • cuda/int2_gemm.cu
  • cuda/ops.cu
  • cuda/attention.cu
  • cuda/kv_cache.cu

Current kernel groups:

  • quantized GEMM
  • FP16 GEMM fallback/fast paths
  • fused SwiGLU and residual ops
  • full prefill attention
  • cached single-query decode attention
  • KV-cache append/materialization helpers
  • rotary position embeddings (RoPE) — cuda/rope.cu

Current attention state:

  • direct cached single-query decode attention is in place
  • a safer tiled prefill attention path now exists
  • the roadmap item for full FlashAttention-2 integration remains open
  • GQA head mapping for models with fewer KV heads than Q heads
  • sliding window via ring-buffer KV cache (cache_window > 0)
  • paged attention with virtual memory pages (TINYLOOP_PAGED_KV=1)

7. Validation Layer​

Primary implementation:

  • tests/
  • CMakeLists.txt

This layer matters because TinyLoop is still moving quickly and performance work is easy to overclaim.

Current validation includes:

  • CUDA unit tests
  • decode parity tests
  • generation regressions
  • tokenizer-aware regressions
  • eval-slice regressions
  • hot-op microbenchmarks

Architectural Principle​

TinyLoop stays useful only if it remains opinionated:

  • one model family
  • explicit runtime tradeoffs
  • narrow public surface
  • measurable behavior

If it tries to become a universal runtime too early, it loses the specialization that makes it interesting.