Admission & Metrics

• The first request after startup has no EMA and always admits regardless of SLO. That's by design — we don't have enough information to predict yet.
• SLO prediction does not include decode time, only dispatch wait. A request with a short max_tokens dispatched when the queue is shallow may still overshoot its SLO if the model is slow. This is fine for admission — the purpose is triage, not exact bound.

PrefixPool auto-dispatch

The scheduler owns one tl.PrefixPool (capacity from prefix_pool_capacity). scheduler.register_prefix(tokens, loops, cache_window) adds an entry.

When a request with mode == "generate" is dispatched and the pool has at least one entry, the scheduler uses generate_with_pool instead of generate:

1. tl.PrefixPool.find scans entries for the longest (tokens, loops, cache_window)-keyed match to the incoming prompt.
2. Match → clone the cached KV, decode the unmatched suffix tokens, decode max_tokens new tokens.
3. No match → fall through to plain engine.generate semantically identically.

The tinyloop_prefix_pool_hits_total counter increments on every dispatch where pool.size() > 0, regardless of whether an actual match was found inside generate_with_pool. This is a rough but useful metric — expensive to measure actual hit rate in the current binding.

Speculative / tree-speculative / beam paths are not routed through the pool. They have their own cache lifecycle requirements and would need separate primitives. Tracked as a follow-up.

Prometheus metrics

All emitted on /metrics in Prometheus text-exposition format. Buckets for histograms are tuned for inference latencies:

1, 5, 10, 25, 50, 100, 250, 500,
1000, 2500, 5000, 10000, 30000, 60000  (milliseconds; plus +Inf)

Counters

Name	Meaning
tinyloop_requests_submitted_total	Admitted requests (before any terminal state).
tinyloop_requests_rejected_total	Rejected by QueueFullError or AdmissionRejected.
tinyloop_requests_completed_total	Reached DONE.
tinyloop_requests_errored_total	Reached ERROR.
tinyloop_requests_cancelled_total	Cancelled before or during dispatch.
tinyloop_prefix_pool_hits_total	Dispatched through generate_with_pool (pool had entries).

Histograms

Name	Observation
tinyloop_queue_wait_ms	Ms between submit() and first engine call (queue time only).
tinyloop_dispatch_ms	Ms from engine call start to terminal state (prefill + decode).
tinyloop_first_token_ms	Streaming only: ms from submit to first emitted chunk.
tinyloop_request_wall_ms	End-to-end ms from submit to terminal state.

Each histogram emits _bucket{le="..."}, _sum, and _count lines (standard Prometheus).

Gauges

Name	Meaning
tinyloop_queue_depth	Requests currently queued.
tinyloop_requests_in_flight	Requests currently DISPATCHED or DECODING.

Error model

Exception	Trigger	HTTP status (in routers)
AdmissionRejected	Predicted wait > SLO	503
QueueFullError	Queue depth at max	503
SchedulerShutdown	Submission after shutdown	503
Engine exception	Raised during generation	500 (default FastAPI handling)
asyncio.CancelledError	Caller cancelled the future	No HTTP response (connection aborted)

AdmissionRejected carries reason: str and predicted_wait_ms: float | None so clients can build retry-after logic.

Configuration

Environment variables read on server startup (server/app.py::startup):

Variable	Default	Effect
TINYLOOP_MAX_QUEUE_DEPTH	128	Passed as max_queue_depth.
TINYLOOP_DEFAULT_SLO_MS	unset	If set, applied as the default SLO for every submission that doesn't specify one.
TINYLOOP_PREFIX_POOL_CAPACITY	16	Capacity of the shared PrefixPool.

See the environment variables reference for the full list.

Testing pattern

Scheduler logic is pure Python and tests without CUDA or tinyloop_py. Use a FakeEngine to exercise the state machine, priority queue, admission, cancellation, and error propagation:

from dataclasses import dataclass, field
from server.engine import GenerationChunk, GenerationRequest, GenerationResult
from server.scheduler import Scheduler

@dataclass
class FakeEngine:
    latency_ms: float = 10.0
    fail_with: Exception | None = None
    stream_tokens: list[int] = field(default_factory=lambda: [1, 2, 3])
    max_seq_len: int = 2048
    default_loops: int = 8

    def __post_init__(self):
        import asyncio
        self._lock = asyncio.Lock()
        self._model = None

    async def generate(self, req):
        async with self._lock:
            import asyncio
            await asyncio.sleep(self.latency_ms / 1000)
            if self.fail_with: raise self.fail_with
            return GenerationResult(
                tokens=[42] * req.max_tokens, text="ok",
                finish_reason="stop",
                prompt_tokens=len(req.prompt_tokens),
                completion_tokens=req.max_tokens,
                wall_ms=self.latency_ms)

    async def generate_stream(self, req):
        async with self._lock:
            if self.fail_with: raise self.fail_with
            for t in self.stream_tokens:
                yield GenerationChunk(token_id=t, text=f"<{t}>")

scheduler = Scheduler(FakeEngine())
await scheduler.start()
try:
    result = await scheduler.submit(req, priority=5, slo_ms=200)
finally:
    await scheduler.shutdown()

The shipped suite (server/tests/test_scheduler.py) covers 13 cases including priority ordering, FIFO-within-priority, queue-full, SLO rejection, cancel-while-queued, engine errors, streaming, shutdown drain, and the Prometheus text output.

Run from the repo root:

python3 -m unittest server.tests.test_scheduler -v

No CUDA required — runs on macOS laptops in under 2 seconds.

Limits and follow-ups

Deliberately not in this first cut — each has a clean place to slot in:

• Batched-homogeneous dispatch. The dispatcher loop currently pops one entry at a time. Grouping compatible requests before dispatch and calling engine.generate_batch is the next throughput win.
• Iteration interleaving. Requires engine.step_one_iter(req_id) — not yet exposed.
• Continuous batching. Needs paged attention. Multi-week.
• Downgrade-on-admission. Accept at lower L instead of rejecting when predicted wait exceeds SLO.
• Priority aging. Low-priority requests starve under sustained high-priority load.
• PrefixPool on speculative / beam. Tree speculative and beam both allocate their own caches; wiring the pool through those paths is a separate change.
• Multi-engine scheduler. One scheduler over N engines on N GPUs.

See also

• HTTP server — how the scheduler is exposed over REST
• Python API: prefix pool — the primitive the scheduler wraps
• Environment variables — server-side configuration
• Production roadmap — batching, continuous batching, and related serving targets