Skip to main content

Production Roadmap

TinyLoop becomes credible as infrastructure only if the work stays sequenced.

The order still matters:

  1. prove decode and runtime correctness
  2. harden performance and serving behavior
  3. prove the unique research advantage with measurements

If those happen in order, TinyLoop can move from "interesting runtime" to "serious specialized framework."

Stage 1: Decode And Runtime Correctness​

Goal​

Make autoregressive execution real and testable.

What is already done​

  • default-on KV-cache decode
  • cached-vs-uncached regression coverage
  • tokenizer-aware Python regressions
  • prefix caching
  • sliding-window cache
  • shared-cache speculative runtime
  • CTest and self-hosted CUDA CI wiring

What is still open​

  • broader regression coverage across more prompts and longer runs
  • first-class tokenizer-backed CLI behavior
  • cleaner public tokenizer story

Exit condition​

TinyLoop should be able to say:

  • cached decode is default
  • parity coverage is easy to rerun
  • published numbers correspond to checked runtime states

Stage 2: Performance And Serving Hardening​

Goal​

Turn TinyLoop into an operable runtime, not just a validated library.

Key workstreams​

Attention​

  • finish the FlashAttention-2 roadmap item properly
  • keep the direct cached single-query path for decode
  • treat the current safer tiled prefill kernel as an intermediate step, not the final story

Memory Profiling By Mode​

  • separate prefill and decode expectations more clearly
  • make runtime memory reporting easier to reason about

Serving Surface​

  • HTTP API
  • model lifecycle handling
  • health endpoints
  • structured logs and metrics

Batching​

  • paged attention
  • static batching first
  • more advanced scheduling later

Exit condition​

TinyLoop should be able to say:

  • it exposes a usable service interface
  • it has diagnostics for runtime failures
  • it has mode-aware performance and memory expectations

Stage 3: Prove The Unique Advantage​

Goal​

Show the parts mainstream runtimes do not have:

  • adaptive loop depth
  • self-speculative decoding with the same weights
  • β-aware quantization advantages

Main workstreams​

Early Exit​

  • add configurable exit logic
  • log per-token loop counts
  • compare quality and latency against fixed-loop baselines

β-aware Quantization​

  • make GPTQ stable first
  • implement β-weighted GPTQ
  • show measurable benefit at fixed bit-width

Scheduling For Mixed Loop Counts​

  • only after adaptive loop depth is real
  • avoid building a scheduler that destroys the very gains adaptive compute is supposed to provide

Exit condition​

TinyLoop should be able to say:

  • adaptive depth is measurable
  • speculative decode is more than a demo
  • the paper-specific quantization story is real, not aspirational

Current Priority Call​

From a pure framework/runtime perspective, the biggest operational gaps are:

  • serving
  • batching
  • tokenizer-first UX
  • full FlashAttention-2 integration

From the paper perspective, the biggest remaining research gap is still:

  • stable GPTQ followed by β-weighted GPTQ

Those are not the same priority stack. Keep them distinct.