Python API

TinyLoop exposes a small pybind11 module named tinyloop_py. The Python surface is intended for evaluation, scripting, tokenizer integration, and service-side orchestration. It is not meant to be a training framework.

Installation

The Python module is built alongside the C++ library by the top-level CMakeLists.txt. After cmake --build . --target tinyloop_py you will find tinyloop_py.cpython-*.so in the build directory. Make it importable by either:

PYTHONPATH=/path/to/tinyloop/build python3 -c "import tinyloop_py; print(tinyloop_py.__doc__)"

or copying the .so into a Python site-packages directory.

A PyPI distribution named loopformer (confusingly, this is TinyLoop under a different name for ecosystem reasons) is published via the repository's trusted-publishing workflow — see installation for the pip install loopformer path.

Quickstart

import numpy as np
import tinyloop_py as tl

with tl.Model("model.tinyloop", max_seq_len=2048) as model:
    prompt = np.array([12, 34, 567], dtype=np.int32)
    tokens = model.generate(prompt, max_tokens=32, loops=8, temperature=0.0, top_k=50)
    print(tokens)

Reference

The Python surface is split into the following sections. Each page is a self-contained reference for the APIs it covers — pick the one that matches what you need.

Section	What it covers
Model	Load a model, manage its lifetime, inspect its config and VRAM usage. Context manager, thread safety, determinism.
Scoring	score, score_last, score_logit_lens, score_trajectory, score_with_uncertainty, score_with_consistency.
Generation	generate, generate_stream, generate_speculative. Sampling kwargs, streaming timing breakdown, self-speculative decode.
Prefix cache	build_prefix_cache, generate_from_prefix_cache — reuse a prefilled prompt across many requests. Measured 2.4–3.0× throughput on same-prefix workloads.
Prefix pool	PrefixPool, register_prefix, generate_with_pool — content-addressable registry of many prebuilt PrefixCache entries with automatic longest-prefix matching. The multi-tenant-serving building block.
Warm-start mid-loop	build_resume_handle, resume_generate — upgrade the same prompt to a deeper L without re-prefilling. Measured −32.5 % wall-clock saving with N=1 break-even.
KV cache modes	How storage modes (FP16 / INT8 / store-h FP16 / store-h INT8 / store-h INT4) interact with the Python API, selection guide, composability.
Constrained decoding	tl.Grammar(pattern) + Model.generate(grammar=...) — regex-level byte grammars for digits-only / multiple-choice / JSON-shape / regex-pattern constrained output. Composes with beam search.

Architecture map

tinyloop_py
├─ Model                                (lifecycle + scoring + generation dispatch)
│  ├─ score / score_last                 → framework-level full / last logits
│  ├─ score_logit_lens                   → per-iteration logits, O(L) single-forward
│  ├─ score_trajectory                   → hidden-state trajectory (embed + pre + loops)
│  ├─ score_with_uncertainty             → KL between two depth logits
│  ├─ score_with_consistency             → argmax-agreement gate w/ escalation
│  ├─ generate / generate_stream         → autoregressive decode
│  ├─ generate_speculative               → self-speculative (same weights, two L)
│  ├─ build_prefix_cache                 → PrefixCache (shared-prefix reuse)
│  ├─ generate_from_prefix_cache         → decode off a PrefixCache
│  ├─ register_prefix                    → insert a PrefixCache into a PrefixPool
│  ├─ generate_with_pool                 → auto-match the longest pooled prefix
│  ├─ build_resume_handle                → ResumeHandle (warm-start mid-loop)
│  ├─ resume_generate                    → extend + decode on a ResumeHandle
│  ├─ config / vram_usage_mb             → metadata + memory reporting
│  └─ close / __enter__ / __exit__        → lifecycle
│
├─ Grammar (regex NFA)                   → constrained decoding (compose with generate)
├─ PrefixCache (opaque)                  → used by generate_from_prefix_cache
├─ PrefixPool (opaque, LRU)              → registry of PrefixCache entries
├─ ResumeHandle (opaque)                 → seq_len / loops_used / max_loops accessors
└─ load (alias for Model constructor)

All opaque types are ref-counted by pybind11; GPU memory is released when the last Python reference drops.

Recommended usage pattern

For most Python-side applications:

1. Load one Model instance and reuse it across the process lifetime.
2. Tokenize outside TinyLoop (e.g. with transformers). All APIs take np.ndarray[int32].
3. Prefer score_last() when you only need the final logits — smallest output allocation.
4. Use build_prefix_cache() for workloads with repeated shared prompts (expect 2–3× throughput).
5. Use build_resume_handle() + resume_generate() when the same prompt needs to be served at multiple loop depths (32.5 % wall-clock saving on the follow-up call).
6. Set KV cache mode env vars before import tinyloop_py when memory is tight — see KV cache modes.

Common error patterns

Error	Typical cause	Fix
RuntimeError: model was closed	Called after close() or after with block exit	Construct a fresh Model
RuntimeError: out of memory	max_seq_len too large, too many cache handles alive, or fp16_body on a small GPU	Reduce max_seq_len, drop handles, or disable TINYLOOP_EXPERIMENTAL_FP16_BODY
RuntimeError: loops mismatch	generate_from_prefix_cache(loops=X) differs from the loops used at build	Match the kwargs or rebuild the prefix
ValueError on kwarg	Negative / zero value where positive required	Check the per-API parameter table
Import failure	.so not on PYTHONPATH, pybind11 ABI mismatch	Rebuild against the current interpreter; verify with python3 -c "import tinyloop_py"

See troubleshooting for deeper failure mode references.

Limits

Python integration today is still:

• single-model, single-process oriented (no cross-process sharing)
• explicit-tokenization by caller
• not a full serving framework on its own

If you need multi-model orchestration, batching across tenants, or cross-process sharing, build that layer on top of the primitives on these pages. All Model methods release the GIL during CUDA kernel execution, so orchestration layers can parallelise freely — the bottleneck is GPU time, not Python interpreter time.

Continuous batching within a single Model is tracked on the production roadmap.