Runtime Architecture

TinyLoop has a deliberately narrow architecture. It is easier to understand if you separate it into model loading, inference orchestration, generation, kernels, and validation.

1. Model Loading

Primary implementation:

• src/model.cpp
• include/model.h

Responsibilities:

• parse .tinyloop headers
• validate dimensions and compatibility constraints
• upload tensors to GPU
• allocate runtime buffers
• report weight and buffer memory

Important current behavior:

• tensors can stream from disk to GPU in chunks instead of requiring full host copies
• optional buffers such as logit_row are lazy
• FP16 body caches can be allocated at load time when TINYLOOP_EXPERIMENTAL_FP16_BODY=1

2. Buffer Pool

The runtime reuses a small number of persistent buffers:

• main: FP32 persistent hidden state (or main_fp16 when TINYLOOP_FP16_RESIDUAL=1)
• norm: FP16 layernorm output
• scratch1: shared QKV or tiled-MLP workspace
• scratch2: optional FP16-body MLP temp
• attn_down: shared attention-output / down-projection buffer
• logit_row: lazily allocated final-row logits buffer

The res_* dispatch helpers (res_layernorm, res_add, res_zero, etc.) abstract the FP32/FP16 branching so each call site doesn't need manual if/else.

That buffer strategy is central to TinyLoop's memory profile.

3. Inference Orchestration

Primary implementation:

• src/inference.cpp

Responsibilities:

• embedding path selection
• pre-block execution
• shared loop-block execution
• final norm
• logits materialization
• hidden-trace and profiling helpers

Core execution shape:

embed
-> pre.0
-> pre.1
-> loop block x L
-> final norm
-> optional logits

Important split:

• full scoring uses full-sequence paths
• generation-style workloads can use score_last_token(...)

4. KV Cache

Primary implementation:

• include/kv_cache.h
• cuda/kv_cache.cu
• src/inference.cpp
• src/generate.cpp

Current capabilities:

• prefill-time cache population
• single-token decode against cached K/V
• default-on cached generation
• sliding-window cache via cache_window
• prefix-cache reuse
• shared-cache speculative runtime

Current missing pieces:

• paged attention
• KV-cache quantization
• broader serving-oriented batching behavior

5. Generation Layer

Primary implementation:

• src/generate.cpp

Responsibilities:

• prompt validation
• sampling
• EOS handling
• prefix-cache helpers
• speculative draft/verify orchestration

Current generation split:

• generate(...) for ordinary autoregressive output
• generate_speculative(...) for self-speculative decode
• prefix-cache helpers for repeated shared prompts

6. CUDA Kernels

Primary implementation:

• cuda/int2_gemm.cu
• cuda/ops.cu
• cuda/attention.cu
• cuda/kv_cache.cu

Current kernel groups:

• quantized GEMM
• FP16 GEMM fallback/fast paths
• fused SwiGLU and residual ops
• full prefill attention
• cached single-query decode attention
• KV-cache append/materialization helpers
• rotary position embeddings (RoPE) — cuda/rope.cu

Current attention state:

• direct cached single-query decode attention is in place
• a safer tiled prefill attention path now exists
• the roadmap item for full FlashAttention-2 integration remains open
• GQA head mapping for models with fewer KV heads than Q heads
• sliding window via ring-buffer KV cache (cache_window > 0)
• paged attention with virtual memory pages (TINYLOOP_PAGED_KV=1)

7. Validation Layer

Primary implementation:

• tests/
• CMakeLists.txt

This layer matters because TinyLoop is still moving quickly and performance work is easy to overclaim.

Current validation includes:

• CUDA unit tests
• decode parity tests
• generation regressions
• tokenizer-aware regressions
• eval-slice regressions
• hot-op microbenchmarks

Architectural Principle

TinyLoop stays useful only if it remains opinionated:

• one model family
• explicit runtime tradeoffs
• narrow public surface
• measurable behavior

If it tries to become a universal runtime too early, it loses the specialization that makes it interesting.