Setup & Architecture

HTTP server

server/app.py is a FastAPI application that wraps a TinyLoop Model and exposes it over HTTP. It ships three endpoint families sharing one underlying engine, so the same model can be served to OpenAI clients, LM Studio clients, and TinyLoop-native callers concurrently.

/v1/* — OpenAI-compatible (openai Python/Node SDKs, curl scripts targeting api.openai.com work unchanged)
/api/v0/* — LM Studio-style (OpenAI shape plus runtime stats, model info, and TinyLoop-specific knobs)
/generate* — TinyLoop-native (direct access to tree speculative, warm-start, KV cache modes — no OpenAI translation layer)
/healthz, /metrics — ops endpoints (plain text health probe, Prometheus exposition)

Every request flows through a proper request Scheduler, not a bare lock: request lifecycle state machine, priority queue, admission control, PrefixPool auto-dispatch, and full Prometheus telemetry. The engine still processes one request at a time for now — the scheduler is the abstraction the next iteration's batched-homogeneous dispatch / iteration interleave / continuous batching will slot into. See Scheduler for the complete reference. For high throughput today, use the prefix cache or register prefix pool entries to amortise shared-prefix prefill across tenants.

Running

Minimal

pip install -r server/requirements.txt
TINYLOOP_MODEL_PATH=/path/to/model.tinyloop \
TINYLOOP_MAX_SEQ_LEN=2048 \
PYTHONPATH=$(pwd)/build \
uvicorn server.app:app --host 0.0.0.0 --port 8000

PYTHONPATH=$(pwd)/build makes the compiled tinyloop_py.so importable — the engine.py also honours the explicit TINYLOOP_BUILD_DIR env var if you want to point at an out-of-tree build.

Long-context / multi-tenant KV compression

Stack the KV cache mode flags on the same uvicorn command:

TINYLOOP_KV_H_MODE=1 \
TINYLOOP_KV_INT4=1 \
TINYLOOP_H_MODE_FP16_ATTN_QKV=1 \
TINYLOOP_MODEL_PATH=... \
PYTHONPATH=$(pwd)/build \
uvicorn server.app:app --host 0.0.0.0 --port 8000

On 1B-effective this cuts per-token KV VRAM by 78 % at roughly baseline latency, so a 40 GB H100 holds ~5× as many concurrent conversations. The server reports the currently active mode in the LM Studio model_info.kv_cache_mode field.

Detached background launch

server/start.sh starts uvicorn in a detached process group that survives the launching SSH session:

TINYLOOP_MODEL_PATH=/workspace/model.tinyloop PORT=8000 ./server/start.sh
# → TinyLoop server pid=12345 on 0.0.0.0:8000 — log /tmp/tinyloop_server.log

Kill it with kill -TERM <pid>. The script respects HOST, PORT, and LOG env vars.

Architecture

┌──────────────────────── FastAPI app ───────────────────────────┐
│                                                                │
│   openai router       lmstudio router      native router       │
│    /v1/*               /api/v0/*            /generate*          │
│         │                    │                    │            │
│         └────────────────────┴────────────────────┘            │
│                              │                                 │
│                   scheduler.submit / submit_stream             │
│                              │                                 │
│                   ┌──────────▼───────────┐                     │
│                   │  Scheduler           │                     │
│                   │  - PriorityQueue     │                     │
│                   │  - Admission (SLO)   │                     │
│                   │  - PrefixPool (LRU)  │                     │
│                   │  - Histograms        │                     │
│                   │  - Dispatcher task   │                     │
│                   └──────────┬───────────┘                     │
│                              │ engine.generate(_stream)        │
│                   ┌──────────▼───────────┐                     │
│                   │  Engine              │                     │
│                   │  tl.Model            │                     │
│                   │  asyncio.Lock        │                     │
│                   └──────────┬───────────┘                     │
└──────────────────────────────┼─────────────────────────────────┘
                               │
                       [ One CUDA Model ]

Scheduler in front of Engine. Since v0.5, all routers go through server.scheduler.Scheduler.submit(...) / submit_stream(...). The scheduler owns request lifecycle, priority, admission, the shared PrefixPool, and Prometheus telemetry. Engine never serves a router directly in production. Full reference: Scheduler.
Single singleton engine. server.engine.Engine owns the tl.Model and an asyncio.Lock. The scheduler's dispatcher task is the only approved caller of engine methods. The lock stays as a defensive layer in case a future integration path needs to talk to Engine directly.
Streaming marshaling. tl.Model.generate_stream runs blocking CUDA work and invokes a callback per token. Engine.generate_stream runs that blocking work in an executor thread, bridges tokens to an asyncio.Queue, and does UTF-8-safe detokenisation in the async consumer loop. The scheduler then forwards each chunk to the router's submit_stream iterator. See Streaming format for the wire-level detail.
Tokenizer. All three router families tokenise prompt text with the HuggingFace gpt2 BPE. The server detokenises via server.tokenizer.StreamingDetokenizer, which buffers incomplete UTF-8 sequences across emitted tokens so multi-byte characters never emit mojibake mid-stream.

Environment variables

Server-specific env vars — for CUDA / KV compression flags see the Environment variables reference.

Variable	Required	Default	Purpose
`TINYLOOP_MODEL_PATH`	yes	—	Absolute path to the `.tinyloop` artifact. Engine constructor `tl.Model(TINYLOOP_MODEL_PATH, max_seq_len=TINYLOOP_MAX_SEQ_LEN)`.
`TINYLOOP_MAX_SEQ_LEN`	no	`2048`	Max prompt + generated length. Sizes the residual stream and (unless `prefill_chunk` is used) all scratch buffers. Requests with `len(prompt_tokens) + max_tokens > TINYLOOP_MAX_SEQ_LEN` get HTTP 400.
`TINYLOOP_MAX_QUEUE_DEPTH`	no	`128`	Scheduler queue cap. Submissions exceeding this get HTTP 503.
`TINYLOOP_DEFAULT_SLO_MS`	no	unset	If set, every submission inherits this SLO (ms). Scheduler rejects with HTTP 503 when its EMA-based predictor says the request can't meet it.
`TINYLOOP_PREFIX_POOL_CAPACITY`	no	`16`	Capacity of the scheduler's shared `PrefixPool`. See Prefix pool.
`TINYLOOP_BUILD_DIR`	no	`./build`	Path prepended to `sys.path` so `tinyloop_py.so` imports. Relative to the server module.
`TINYLOOP_COMMIT`	no	`null`	Opaque commit / version string surfaced in `model_info.runtime.commit` (LM Studio). Set at CI build time.
`HF_HOME`	no	HuggingFace default	Cache directory for the `gpt2` tokenizer download.

Endpoint catalogue

Endpoint	Method	Family	Streaming	Purpose
`/healthz`	GET	ops	—	Plain-text health probe (`ok\n`)
`/metrics`	GET	ops	—	Prometheus exposition
`/v1/models`	GET	openai	—	List served model(s)
`/v1/chat/completions`	POST	openai	SSE	Chat with role-labelled messages
`/v1/completions`	POST	openai	SSE	Raw text completion
`/api/v0/models`	GET	lmstudio	—	Same as `/v1/models` with LM Studio shape
`/api/v0/chat/completions`	POST	lmstudio	SSE	Chat + `stats` + `model_info` + `runtime` + TinyLoop knobs
`/api/v0/completions`	POST	lmstudio	—	Completion + same extended fields
`/generate`	POST	native	SSE	TinyLoop generate — direct access to `loops`, `sample_seed`, `cache_window`, `stop`
`/generate/speculative`	POST	native	—	Chain self-speculative decode (`draft_loops` + `verify_loops` + `draft_ahead`)
`/generate/tree_speculative`	POST	native	—	Tree speculative decode (adds `K_branches` branching factor)

`/healthz`

GET /healthz HTTP/1.1

HTTP/1.1 200 OK
Content-Type: text/plain

ok

Returns 503 Unavailable with an error description when the engine is not initialised or failed to load the model.

`/metrics`

Prometheus text-exposition. The base engine gauges (max_seq_len, default_loops, up_timestamp_seconds) are joined by the full scheduler metric surface — counters, histograms, and queue-depth / in-flight gauges. See Scheduler → Prometheus metrics for every series definition.

GET /metrics HTTP/1.1

HTTP/1.1 200 OK
Content-Type: text/plain

# HELP tinyloop_max_seq_len Maximum sequence length the model was loaded with.
# TYPE tinyloop_max_seq_len gauge
tinyloop_max_seq_len 2048
# HELP tinyloop_default_loops Default loop depth.
# TYPE tinyloop_default_loops gauge
tinyloop_default_loops 8
# HELP tinyloop_up_timestamp_seconds Unix timestamp at server start.
# TYPE tinyloop_up_timestamp_seconds gauge
tinyloop_up_timestamp_seconds 1713475200
# HELP tinyloop_requests_submitted_total Total requests accepted by the scheduler.
# TYPE tinyloop_requests_submitted_total counter
tinyloop_requests_submitted_total 42
# HELP tinyloop_queue_wait_ms Seconds-in-queue from submit to dispatch, in ms.
# TYPE tinyloop_queue_wait_ms histogram
tinyloop_queue_wait_ms_bucket{le="1.0"} 40
tinyloop_queue_wait_ms_bucket{le="5.0"} 41
tinyloop_queue_wait_ms_bucket{le="+Inf"} 42
tinyloop_queue_wait_ms_sum 18.3
tinyloop_queue_wait_ms_count 42
...

Emitted series (see Scheduler metrics for bucket definitions and semantics):

Counters: tinyloop_requests_submitted_total, ..._rejected_total, ..._completed_total, ..._errored_total, ..._cancelled_total, tinyloop_prefix_pool_hits_total
Histograms: tinyloop_queue_wait_ms, tinyloop_dispatch_ms, tinyloop_first_token_ms, tinyloop_request_wall_ms
Gauges: tinyloop_queue_depth, tinyloop_requests_in_flight

HTTP server

Running​

Minimal​

Long-context / multi-tenant KV compression​

Detached background launch​

Architecture​

Environment variables​

Endpoint catalogue​

/healthz​

/metrics​