Skip to main content

L-Aware Batching

What It Does​

Multiple sequences in the same generate_batch call run at different loop depths. Free tier at L=4, pro at L=16, enterprise at L=32 — one model, one GPU, one batch.

How It Works​

Each lane runs its own full forward pass at its own loop depth. The KV cache is allocated per-lane to match each lane's L.

Verified​

On H100 / 407M, verified bit-exact:

  • Mixed lane 0 (L=4) produces identical output to standalone generate(loops=4)
  • Mixed lane 1 (L=8) produces identical output to standalone generate(loops=8)

Usage​

results = model.generate_batch(
    [free_prompt, pro_prompt, enterprise_prompt],
    per_lane_loops=[4, 16, 32],
    max_tokens=64
)
# results[0] = free tier output (L=4, fast)
# results[1] = pro tier output (L=16, balanced)
# results[2] = enterprise output (L=32, highest quality)

Why Only Looped​

Standard transformers can't vary depth per request. Every sequence must run all layers. To serve different quality tiers, the industry deploys separate models — 3 models for 3 tiers. TinyLoop serves all tiers from one set of weights by varying the loop count per request.