Skip to main content

Quickstart

This is the fastest path from a built repo to a verified runtime.

1. Inspect A Model​

./tinyloop/build/tinyloop model.tinyloop inspect

Use inspect before anything else. It validates:

  • file magic and format version
  • dimensions and head count
  • embedding mode
  • estimated weight, buffer, and KV-cache memory

2. Benchmark The Runtime​

./tinyloop/build/tinyloop model.tinyloop benchmark --loops 8 --seq-len 128 --repeat 10

This exercises the runtime without full logits materialization.

Current documented benchmark states:

  • default low-bit path: about 30.48 ms
  • experimental FP16-body path: about 2.82 ms

Both numbers refer to the validated H100 target workload at seq_len=128, loops=8, no logits.

To print CUDA-event phase timings for one run:

TINYLOOP_CUDA_PROFILE=1 \
./tinyloop/build/tinyloop model.tinyloop benchmark --loops 8 --seq-len 128 --repeat 1

3. Generate Text​

./tinyloop/build/tinyloop model.tinyloop generate \
  --prompt "Looped transformers are" \
  --loops 8 \
  --max-tokens 32 \
  --temperature 0.8 \
  --top-k 50

Important current behavior:

  • cached decode is the default path
  • TINYLOOP_DISABLE_KV_CACHE=1 forces the uncached reference path
  • prompt text is currently tokenized as raw bytes in the CLI

For the uncached reference:

TINYLOOP_DISABLE_KV_CACHE=1 \
./tinyloop/build/tinyloop model.tinyloop generate --prompt "Looped transformers are"

4. Try Self-Speculative Decoding​

./tinyloop/build/tinyloop model.tinyloop speculate \
  --prompt "Looped transformers are" \
  --draft-loops 2 \
  --verify-loops 8 \
  --draft-ahead 4 \
  --max-tokens 32

This uses the same model for draft and verify passes.

Current state:

  • cache-aware speculation exists
  • accept-or-resample logic exists
  • regression coverage exists
  • broader workload validation is still an open roadmap item

5. Use The Python Binding For Real Tokenization​

import numpy as np
import tinyloop_py

model = tinyloop_py.Model("model.tinyloop", max_seq_len=2048)
tokens = np.asarray([15496, 995], dtype=np.int32)

logits = model.score_last(tokens, loops=8)
generated = model.generate(tokens, max_tokens=32, loops=8, temperature=0.0, top_k=50)

The Python path is the right place for:

  • GPT-style tokenizer integration
  • evaluation scripts
  • service-side orchestration

6. Run The Test Surface​

ctest --test-dir tinyloop/build --output-on-failure

For model-dependent checks:

TINYLOOP_TEST_MODEL_PATH=/absolute/path/to/model.tinyloop \
ctest --test-dir tinyloop/build --output-on-failure

Current Caveats​

caution

The CLI is still a raw-byte prompt interface. For realistic tokenizer-backed generation, use Python or your own integration layer.

The current performance claims are backed by the checked validation paths, not by universal cross-model guarantees.