Endpoints

OpenAI-compatible (/v1/*)

Any openai client speaks to /v1/* unchanged — pass base_url="http://<host>:<port>/v1" and any api_key (the server does not authenticate today).

GET /v1/models

{
  "object": "list",
  "data": [
    {
      "id": "/workspace/model_407m_gptq_int4.tinyloop",
      "object": "model",
      "created": 1713475200,
      "owned_by": "tinyloop"
    }
  ]
}

The id is the TINYLOOP_MODEL_PATH — use it as the model field in subsequent requests if you want a stable identifier, or pass any string (it's ignored by the engine beyond being echoed into responses).

POST /v1/chat/completions

Request body (subset of the OpenAI spec — extras are documented under Not implemented):

Field	Type	Required	Default	Notes
model	string	yes	—	Echoed into the response
messages	array of {role, content}	yes	—	role ∈ {"system", "user", "assistant"}. Rendered via a simple "Role: content" template — TinyLoop's current models use raw GPT-2 BPE, not a chat-tuned tokenizer
max_tokens	int	no	128	1 <= max_tokens <= 8192
temperature	float	no	1.0	0.0 <= t <= 2.0. 0.0 is greedy
top_p	float	no	1.0	Nucleus sampling cutoff. Currently approximated — top_k=50 is also always active (server default)
n	int	no	1	Must be 1 — multiple samples per request not implemented; n>1 fails validation
stop	string | string[]	no	null	Server currently approximates multi-token stops by matching the first token of each stop string
stream	bool	no	false	true returns an SSE stream of chat.completion.chunk events
seed	int	no	null	Deterministic sampling; forwarded as sample_seed to the engine
presence_penalty	float	no	0.0	-2.0 <= p <= 2.0. Collapsed with frequency_penalty into a single repetition_penalty = 1 + max(0, p+f)/2
frequency_penalty	float	no	0.0	See above
loops	int	no	null	TinyLoop extension — loop depth 1..64. Ignored by strict OpenAI clients; honoured when present

Non-streaming response (stream=false):

{
  "id": "chatcmpl-0f4c8e1d1c2a4e1f80d8123a",
  "object": "chat.completion",
  "created": 1713475200,
  "model": "tinyloop-407m",
  "choices": [
    {
      "index": 0,
      "message": {"role": "assistant", "content": "Paris is the capital of France."},
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 12,
    "completion_tokens": 8,
    "total_tokens": 20
  }
}

Streaming response (stream=true) is a sequence of text/event-stream events. See Streaming format for framing.

Error responses:

Status	Cause
400	prompt_tokens + max_tokens > TINYLOOP_MAX_SEQ_LEN; invalid shape (n != 1, out-of-range temperature)
422	Pydantic validation error (missing messages, wrong types)
503	Engine not loaded — likely a model load failure at startup

POST /v1/completions

Raw text completion. Same sampling fields as chat; the difference is the prompt shape:

Field	Type	Required	Default	Notes
model	string	yes	—	Echoed into the response
prompt	string | string[]	yes	—	Array is supported but only the first prompt is used; multi-prompt requires n>1 which is not implemented

Response shape mirrors OpenAI text_completion:

{
  "id": "cmpl-0f4c8e1d1c2a4e1f80d8123a",
  "object": "text_completion",
  "created": 1713475200,
  "model": "tinyloop-407m",
  "choices": [{"text": "…", "index": 0, "finish_reason": "stop"}],
  "usage": {"prompt_tokens": 4, "completion_tokens": 16, "total_tokens": 20}
}

LM Studio-style extended (/api/v0/*)

Identical endpoints to /v1/* with two additions:

1. Request schema is a superset. Optional fields: mode ("generate" | "speculative" | "tree_speculative"), draft_loops, verify_loops, draft_ahead, K_branches, cache_window, repetition_penalty. Strict OpenAI clients that don't send these still work; the engine falls back to defaults.
2. Response adds three sections — stats, model_info, runtime — attached to the same ChatCompletionResponse / CompletionResponse shape.

/api/v0/chat/completions response extras

{
  "id": "chatcmpl-…",
  "choices": [...],
  "usage": {...},

  "stats": {
    "wall_ms": 152.4,
    "tokens_per_second": 210.1,
    "prompt_tokens": 8,
    "completion_tokens": 32,
    "mode": "tree_speculative",
    "loops_used": 8
  },
  "model_info": {
    "arch": "weight-shared-looped-transformer",
    "dim": 2048,
    "n_heads": 16,
    "ffn_dim": 8192,
    "vocab_size": 50257,
    "default_loops": 8,
    "quant": "int4",
    "kv_cache_mode": "int4-h"
  },
  "runtime": {
    "name": "tinyloop",
    "version": "0.5-dev",
    "commit": null
  }
}

Field sources:

• stats.wall_ms / tokens_per_second — wall-clock elapsed between the engine receiving the request and returning the result.
• stats.mode — echoes the selected generation mode ("generate" / "speculative" / "tree_speculative").
• stats.loops_used — verify_loops for the speculative modes, loops for plain generate.
• model_info.quant — inferred heuristically from the model path string ("int4" / "int2" / "fp16"). A future release may read the quant bits from the model header directly.
• model_info.kv_cache_mode — derived from the env var stack at server start: TINYLOOP_KV_INT4=1 + TINYLOOP_KV_H_MODE=1 yields "int4-h"; see the KV cache modes page for the full env → mode mapping.
• runtime.commit — from TINYLOOP_COMMIT if set, else null.

LM Studio-only knobs

Set on the request to route generation through an architecture-exclusive path while keeping the OpenAI wrapper:

Field	Type	Default	Applies when	Notes
mode	"generate" | "speculative" | "tree_speculative"	"generate"	all	Selects the internal dispatcher
draft_loops	int	2	mode ∈ {speculative, tree_speculative}	Loop depth for the draft forward
verify_loops	int	8	mode ∈ {speculative, tree_speculative}	Loop depth for the verifier forward (also reported as loops_used)
draft_ahead	int	4	mode ∈ {speculative, tree_speculative}	Draft tokens per round (chain) or tree depth (tree)
K_branches	int	1	mode == tree_speculative	Tree branching factor; K=1 is chain mode, bit-exact to mode=speculative
cache_window	int	0	all	Sliding-window KV cap (0 = full)
repetition_penalty	float	1.0	all	0.5 <= p <= 2.0

TinyLoop-native (/generate*)

No OpenAI wrapping — the request and response are a tight JSON shape that exposes every architecture-exclusive runtime knob.

POST /generate

Field	Type	Required	Default	Notes
prompt	string | int[]	yes	—	Text (encoded with GPT-2 BPE server-side) or pre-tokenised int ids
max_tokens	int	no	128	1..8192
loops	int	no	8	1..64
temperature	float	no	0.0	0.0..2.0
top_k	int	no	50	0..1000
top_p	float	no	0.9	0..1
repetition_penalty	float	no	1.0	0.5..2.0
sample_seed	int	no	0	Deterministic sampling
stream	bool	no	false	true streams one JSON event per token
stop	string | string[] | int[]	no	null	String(s) encoded to first-token ids, or raw int ids
cache_window	int	no	0	Sliding-window cap

Non-streaming response:

{
  "tokens": [464, 3139, 286, 4881, ...],
  "text": "Paris is the capital of France.",
  "finish_reason": "stop",
  "stats": {
    "wall_ms": 88.3,
    "tokens_per_second": 90.6,
    "prompt_tokens": 6,
    "completion_tokens": 8,
    "mode": "generate",
    "loops_used": 8
  }
}

tokens contains only the generated tokens (prompt is stripped). Use the prompt_tokens count if you need to reconstruct the full sequence.

Streaming response (stream=true): one SSE event per emitted token — see Native SSE format.

POST /generate/speculative

Extends /generate with the self-speculative triplet:

Field	Type	Default	Notes
draft_loops	int	2	1..32
verify_loops	int	8	1..64. Verified output is bit-exact to generate(prompt, loops=verify_loops) under greedy
draft_ahead	int	4	1..16 — tokens the draft proposes per cycle

Response shape is identical to /generate with stats.mode="speculative" and stats.loops_used=verify_loops. Streaming is not wired for this endpoint yet (stream=true is silently ignored — future release will enable token-by-token streaming of verified tokens).

POST /generate/tree_speculative

Extends /generate/speculative with the tree branching factor:

Field	Type	Default	Notes
K_branches	int	1	1..16. K=1 = pure chain (bit-exact to /generate/speculative); K>=2 = top-K tree exploration at each depth

Response shape as above; stats.mode="tree_speculative". Streaming not wired.

Constraint: tree speculative currently requires FP16 KV cache — TINYLOOP_KV_H_MODE / TINYLOOP_KV_INT8 / TINYLOOP_KV_INT4 are rejected at the kernel boundary. Greedy only (temperature=0.0); K_branches * draft_ahead <= 64 (enforced by the verify kernel).

Streaming format

OpenAI SSE (/v1/* + /api/v0/chat/completions + /api/v0/completions)

Standard OpenAI server-sent-events framing. One data: line per event, blank line between events, [DONE] terminator.

data: {"id":"…","object":"chat.completion.chunk","created":1713475200,"model":"tl","choices":[{"index":0,"delta":{"role":"assistant"},"finish_reason":null}]}

data: {"id":"…","object":"chat.completion.chunk","created":1713475200,"model":"tl","choices":[{"index":0,"delta":{"content":"Paris"},"finish_reason":null}]}

data: {"id":"…","object":"chat.completion.chunk","created":1713475200,"model":"tl","choices":[{"index":0,"delta":{"content":" is"},"finish_reason":null}]}

…

data: {"id":"…","object":"chat.completion.chunk","created":1713475200,"model":"tl","choices":[{"index":0,"delta":{},"finish_reason":"stop"}]}

data: [DONE]

Notes:

• First event carries {"role": "assistant"} (OpenAI convention).
• Content events carry {"content": "<text delta>"}. Deltas are emitted as complete UTF-8 characters only — the StreamingDetokenizer buffers partial byte sequences (e.g. one byte of a 3-byte emoji) until the character is whole. Zero-content deltas are dropped.
• Final event carries {} delta with finish_reason="stop" or "length".
• Terminator is a literal data: [DONE]\n\n with no JSON.

Native SSE format

Used by the /generate streaming endpoint.

One event per token, no role/done framing. Useful for clients that do their own display logic.

data: {"token_id":464,"text":"Paris"}

data: {"token_id":318,"text":" is"}

data: {"token_id":262,"text":" the"}

…

data: [DONE]

text is the UTF-8-safe decoded delta for this token — same buffering as the OpenAI path. A final token_id: -1 event carries any flushed tail bytes (replacement characters for incomplete multi-byte sequences).

Finish reasons

Reason	Trigger
"stop"	Stop token matched, or end-of-generation before max_tokens
"length"	Hit max_tokens without a stop
"error"	Internal engine error (surfaced in non-streaming paths; rarely used)

The non-streaming code path's "stop" vs "length" is computed from whether stop_idx fired inside the generated slice — see server/engine.py::_run_blocking_full for the exact logic.