Skip to main content

Streaming & Clients

Request lifecycle​

  1. Accept: FastAPI deserialises and validates via Pydantic (server/schemas/*.py). Validation errors → HTTP 422.
  2. Tokenise: GPT-2 BPE via HuggingFace gpt2 tokenizer. Text prompts become int32 arrays.
  3. Length check: len(prompt_tokens) + max_tokens <= TINYLOOP_MAX_SEQ_LEN or HTTP 400.
  4. Stop encoding: Stop strings are encoded to their first token id (single-token approximation).
  5. Engine dispatch: Engine.generate or Engine.generate_stream. The asyncio.Lock serialises access to the underlying Model.
  6. Decode: TinyLoop's forward passes run on the CUDA stream; tokens are detokenised in the server thread (keeps the GIL-bound HF tokenizer off the worker).
  7. Return: single JSON (non-streaming) or close the SSE stream with [DONE].

Timeout handling: there is no server-side timeout today. Clients that want one should enforce it on the transport layer (timeout= in requests / httpx / openai).

Client examples​

Python openai SDK​

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="not-used")

resp = client.chat.completions.create(
    model="tl",
    messages=[{"role": "user", "content": "Capital of France?"}],
    max_tokens=16, temperature=0.0,
)
print(resp.choices[0].message.content)

# Streaming
stream = client.chat.completions.create(
    model="tl",
    messages=[{"role": "user", "content": "Count to 5."}],
    max_tokens=24, stream=True,
)
for chunk in stream:
    delta = chunk.choices[0].delta.content or ""
    print(delta, end="", flush=True)

curl​

# Non-streaming chat
curl -s http://localhost:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"tl","messages":[{"role":"user","content":"Hi"}],"max_tokens":16}' | jq

# Streaming chat (raw SSE)
curl -N http://localhost:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"tl","messages":[{"role":"user","content":"Count"}],"max_tokens":16,"stream":true}'

# LM Studio with tree speculative
curl -s http://localhost:8000/api/v0/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"tl","messages":[{"role":"user","content":"Hi"}],"max_tokens":32,"mode":"tree_speculative","K_branches":2,"draft_loops":2,"verify_loops":8}' | jq '.stats'

# TinyLoop-native with explicit loop depth
curl -s http://localhost:8000/generate \
  -H 'Content-Type: application/json' \
  -d '{"prompt":"The meaning of life is","max_tokens":32,"loops":16,"temperature":0.0,"cache_window":512}' | jq

# Pre-tokenised prompt (bypass server tokenizer)
curl -s http://localhost:8000/generate \
  -H 'Content-Type: application/json' \
  -d '{"prompt":[464,3139,286,4881],"max_tokens":16,"loops":8}' | jq

httpx / raw HTTP​

import httpx

with httpx.Client(base_url="http://localhost:8000") as c:
    r = c.post("/generate/tree_speculative", json={
        "prompt": "The meaning of life is",
        "max_tokens": 32, "loops": 8,
        "draft_loops": 2, "verify_loops": 8, "draft_ahead": 4, "K_branches": 2,
        "temperature": 0.0,
    })
    r.raise_for_status()
    data = r.json()
    print(data["text"])
    print(f"stats: {data['stats']['wall_ms']:.1f} ms, "
          f"{data['stats']['tokens_per_second']:.1f} tok/s")

Smoke test​

server/smoke_test.py runs every endpoint against an in-process FastAPI TestClient — no actual HTTP server needed:

PYTHONPATH=$(pwd)/build:$(pwd) \
TINYLOOP_MODEL_PATH=/path/to/model.tinyloop \
python3 server/smoke_test.py

Expected output ends with === smoke test PASS ===. The script exits non-zero if any endpoint returns an unexpected status code, the streaming framing is broken, or the generated content count is empty when it shouldn't be.

This is the recommended way to verify a build after KV mode env-var changes — it is fast (~1 second once the model is loaded) and covers every router end-to-end.

Running in Docker​

FROM nvidia/cuda:12.4.0-runtime-ubuntu22.04
RUN apt-get update && apt-get install -y python3 python3-pip git
WORKDIR /app
COPY . .
RUN pip3 install -r server/requirements.txt
# Assumes you've already built tinyloop_py.so into ./build on the host
# or via a multi-stage build with the CUDA devel image.
ENV PYTHONPATH=/app/build
EXPOSE 8000
CMD ["uvicorn", "server.app:app", "--host", "0.0.0.0", "--port", "8000"]

The devcontainer needs matching CUDA userspace (driver + runtime) for libcuda.so to load. Build tinyloop_py.so either in a multi-stage builder image (nvidia/cuda:12.4.0-devel-ubuntu22.04) or on the host with a matching compiler toolchain.

What is not implemented​

FeatureStatusTracking
Continuous batchingNot implementedMulti-week paged-attention refactor — production roadmap
Authentication / API keysNot implementedDeploy behind a reverse proxy (nginx, Traefik, Caddy) for auth
Rate limitingNot implementedSame — use the reverse proxy
Multi-model servingNot implementedOne Model per process; run multiple uvicorn instances for multi-model
Tool / function callingRequest fields ignoredTracked as a pre-v1.0 item
n > 1 (multiple samples)Rejected at validationWould require batched decode across duplicate prompts
logprobsNot implementedRequires engine-side logits capture, not wired
JSON mode (response_format)Not implementedConstrained decoding exists at the Python API layer (tl.Grammar) but isn't wired through the server
WebSocket transportNot implementedSSE covers the streaming use case today
/generate/speculative streamingSilently non-streamingCallback plumbing in the speculative path is a follow-up

See also​