メインコンテンツまでスキップ

Async API

AsyncModel wraps the synchronous tinyloop_py.Model bindings so every blocking CUDA call runs in a thread pool, letting you await inference from any asyncio coroutine without stalling the event loop.

Quick start​

import asyncio
from tinyloop_tools.async_model import AsyncModel

async def main():
    async with AsyncModel("model.tinyloop") as model:
        tokens = await model.generate([1, 2, 3], max_tokens=32, loops=8)
        print(tokens)

asyncio.run(main())

Concurrency model​

The underlying C++ model is not thread-safe. AsyncModel serialises all calls behind an asyncio.Lock, so concurrent awaits queue automatically — no manual synchronisation needed.

A single-thread ThreadPoolExecutor runs the blocking CUDA work. This means:

  • The event loop stays responsive during GPU compute.
  • Multiple concurrent await model.generate(...) calls are serialised, not parallelised.
  • The GIL is released during C++ execution via pybind11's gil_scoped_release.

API reference​

Constructor​

AsyncModel(path, max_seq_len=2048, prefill_chunk=0, max_workers=1)

Scoring​

logits = await model.score(tokens, loops=8)         # [seq_len, vocab]
logits = await model.score_last(tokens, loops=8)     # [vocab]
logits = await model.score_logit_lens(tokens, loops=8)  # [loops, vocab]
traj   = await model.score_trajectory(tokens, loops=8)  # dict

Generation​

tokens = await model.generate(prompt, max_tokens=128, loops=8,
                               temperature=0.0, top_k=50)

All generate parameters from the sync API are supported: cache_window, repetition_penalty, sample_seed, l_warmup_tokens, l_warmup_L, safety_check, safety_norm_threshold.

Streaming​

async for token_id in model.generate_stream(prompt, max_tokens=128, loops=8):
    print(token_id, end=" ")

The stream is bridged from the C++ callback via asyncio.Queue — tokens are yielded as soon as they are produced.

Speculative decode​

result = await model.generate_speculative(prompt, draft_loops=2,
                                           verify_loops=8, draft_ahead=4)

Batch generation​

outputs = await model.generate_batch(
    [prompt1, prompt2, prompt3],
    max_tokens=64, loops=8,
    per_lane_loops=[4, 8, 16]  # optional L-aware batching
)

Context manager​

async with AsyncModel("model.tinyloop") as model:
    ...  # model.close() called automatically

Utilities​

cfg = model.config()           # sync — returns dict
vram = await model.vram_usage_mb()