Architecture & API

---

Request scheduler

server/scheduler.py is the single approved caller of Engine in production. HTTP routers no longer talk to the Engine directly — they submit requests to the scheduler and get back a future (non-streaming) or an async iterator (streaming).

Why the scheduler exists:

The previous design serialised every request through a bare asyncio.Lock inside Engine. That was fine for a demo but painted the architecture into a corner — continuous batching, L-aware batching, cross-request iteration interleaving, SLO-aware admission, and multi-tenant prefix reuse all need a request abstraction the engine lock cannot provide.

The scheduler's first iteration does not improve throughput. Engine is still one-request-at-a-time. What the scheduler provides is the abstraction, the telemetry, and the hooks that (a) L-aware batching, (b) iteration interleaving, and (c) continuous batching will slot into without changing any router code.

What you get today

• Request lifecycle state machine — queued → dispatched → decoding → done | cancelled | error — with wall-clock timestamps on every transition.
• Priority queue with FIFO within a priority level.
• Admission control: reject requests whose predicted wall-clock (queue depth × EMA dispatch latency) would exceed an optional per-request SLO, or when queue depth exceeds max_queue_depth.
• Content-addressable PrefixPool ownership. Any mode="generate" request whose prompt starts with a registered prefix automatically routes through generate_with_pool — prefill skipped on the matched tokens.
• Prometheus histograms + counters + gauges on /metrics (all tinyloop_* series below).
• Graceful cancellation: cancel a future while still queued → the scheduler drops the request without dispatching. Cancel after dispatch → CUDA kernel runs to completion but result is discarded.
• Graceful shutdown: stop accepting new work, wait up to drain_timeout_s for the in-flight request to finish.

What you get next

Already slotted into the abstraction, shipping in follow-up iterations:

• Batched-homogeneous dispatch — the scheduler will auto-batch same-L, same-cache_window, mode="generate" requests through generate_batch (2× throughput at N=8 today on matched prompts).
• Iteration-granular interleave — dispatch one loop iteration of request A, then request B, then A again. Requires the engine to expose a step-level primitive.
• Continuous batching — add and remove sequences mid-generation. Requires paged attention (multi-week).
• Downgraded-L on admission pressure — instead of 503, accept at a lower L when the SLO can't be met at requested depth.

Architecture

┌──────────────────── FastAPI app ───────────────────────┐
│                                                        │
│   openai router   lmstudio router   native router      │
│        │                │                 │            │
│        └────────────────┴─────────────────┘            │
│                         │                              │
│                  scheduler.submit / submit_stream      │
│                         │                              │
│                ┌────────▼─────────┐                    │
│                │   Scheduler      │                    │
│                │ - PriorityQueue  │                    │
│                │ - Admission      │                    │
│                │ - PrefixPool     │                    │
│                │ - Histograms     │                    │
│                │ - Dispatcher task│                    │
│                └────────┬─────────┘                    │
│                         │ engine.generate(_stream)     │
│                ┌────────▼─────────┐                    │
│                │  Engine          │                    │
│                │  tl.Model        │                    │
│                │  asyncio.Lock    │                    │
│                └────────┬─────────┘                    │
└─────────────────────────┼──────────────────────────────┘
                          │
                   [ CUDA Model ]

The dispatcher is a single long-lived coroutine (_dispatcher_loop) started at app boot. It races self._queue.get() against a stop asyncio.Event so shutdown() breaks the loop promptly without the Python 3.11 wait_for cancellation edge cases.

Request lifecycle

submit()  ──►  QUEUED ──►  DISPATCHED ──►  DECODING ──►  DONE
                │               │               │
                ▼               ▼               ▼
            CANCELLED        ERROR           ERROR

ScheduledRequest is the internal object holding the request through its entire lifecycle. Fields:

Field	Type	When set
request_id	str (24-char hex)	At submit
req	GenerationRequest	At submit
priority	int	At submit
slo_ms	float | None	At submit
state	RequestState	Each transition
queued_at	float	At submit
dispatched_at	float | None	When dispatcher picks up
first_token_at	float | None	Streaming: on first emitted chunk
completed_at	float | None	On any terminal transition
result	GenerationResult | None	On DONE (non-streaming)
error	BaseException | None	On ERROR
cancelled	bool	When the caller cancels
streaming	bool	At submit

Terminal states resolve the caller's future:

• DONE — future.set_result(result); for streaming, the chunk queue closes with None after the last chunk.
• CANCELLED — future.cancel(); for streaming, the chunk queue closes with None and the iteration ends without raising.
• ERROR — future.set_exception(error); for streaming, the chunk queue closes with None and the iteration re-raises the captured error after the sentinel.

Public API

from server.scheduler import (
    Scheduler, get_scheduler, set_scheduler,
    AdmissionRejected, QueueFullError, SchedulerShutdown,
    RequestState,
)

Scheduler(engine, *, max_queue_depth=128, default_slo_ms=None, prefix_pool_capacity=16, ema_alpha=0.2)

Parameter	Type	Default	Description
engine	Engine	—	The TinyLoop Engine the scheduler dispatches against. The scheduler is the only approved caller of engine methods in production.
max_queue_depth	int	128	Maximum number of queued (not yet dispatched) requests. Submissions beyond this raise QueueFullError.
default_slo_ms	float | None	None	Optional per-server SLO applied when a submission does not set its own slo_ms. None disables admission-by-SLO (only max_queue_depth gates).
prefix_pool_capacity	int	16	Capacity passed to the underlying tl.PrefixPool. Evicts the shortest entry when full (rough LRU).
ema_alpha	float	0.2	Smoothing factor for the EMA of dispatch latency. Used by the admission predictor. Higher = faster reaction to latency swings, noisier.

await scheduler.start()

Launch the dispatcher task. Must be called before any submit(). Idempotent.

await scheduler.shutdown(drain_timeout_s=30.0)

Set the stop flag, wait up to drain_timeout_s for the dispatcher to finish current work, then cancel if still running. Any queued requests get CANCELLED futures.

await scheduler.submit(req, *, priority=0, slo_ms=None) -> GenerationResult

Submit a non-streaming request; await the final result.

• priority — higher values dispatched first. FIFO within the same priority level.
• slo_ms — overrides default_slo_ms. If the predicted wait exceeds this, AdmissionRejected is raised at submit time (before queueing).

Raises:

• AdmissionRejected — predicted wait exceeds SLO.
• QueueFullError — queue depth would exceed max_queue_depth.
• SchedulerShutdown — scheduler is shutting down.
• Any exception the engine raises during generation (caller sees it via await).

scheduler.submit_stream(req, *, priority=0, slo_ms=None) -> AsyncIterator[GenerationChunk]

Same admission semantics as submit; yields GenerationChunk(token_id, text) tuples as the engine emits tokens. Errors are raised after the final chunk is consumed (standard "close" behaviour for async generators).

scheduler.register_prefix(tokens, loops=8, cache_window=0) -> int

Register a tokenized prefix in the scheduler's shared PrefixPool. Returns the number of tokens cached (usually len(tokens)). Subsequent submit() calls in mode="generate" whose prompt starts with this prefix will use generate_with_pool automatically.

scheduler.prefix_pool_stats() -> dict

{"entries": int, "total_cached_tokens": int}

scheduler.snapshot() -> dict

{
    "queue_depth": int,          # requests waiting (not yet dispatched)
    "in_flight": int,            # requests currently DISPATCHED or DECODING
    "ema_dispatch_ms": float | None,  # EMA of recent dispatch latencies
    "prefix_pool": {"entries": int, "total_cached_tokens": int},
}

scheduler.metrics_text(extra_lines=None) -> str

Returns the full Prometheus exposition text for the scheduler. /metrics calls this with its own base gauges prepended via extra_lines.

Priority queue semantics

Requests are popped in (priority_desc, submit_order_asc) order:

• Higher priority wins. priority=10 is dispatched before priority=5.
• FIFO within a priority level. Two requests both at priority=5 dispatch in submission order.
• One request in flight at a time (for now). Priority only reorders the queue; it does not preempt a running request.

No starvation protection today. If a steady stream of high-priority requests arrives, low-priority ones wait forever. Aging-based priority bumps are tracked as a follow-up.

Admission control

Two gates, in order:

1. Queue-full. If the queue already holds max_queue_depth entries, new submissions raise QueueFullError immediately. Routers return HTTP 503.

2. SLO prediction. If the request (or server default) sets slo_ms and there is an in-flight request (so ema_dispatch_ms is populated), the scheduler computes:

predicted_wait = (queue_depth + 1) * ema_dispatch_ms

The +1 accounts for the currently-in-flight request the submission must wait behind. If predicted_wait > slo_ms, the submission raises AdmissionRejected(reason, predicted_wait_ms). Routers return HTTP 503 with the reason.

Failure modes to know about: