Skip to main content

Troubleshooting

This page focuses on the current failure modes that actually show up in TinyLoop workflows.

Build Problems​

Failed to find nvcc​

Meaning:

  • the active environment cannot find the CUDA toolkit

Check:

  • nvcc exists
  • CUDA binaries are on PATH
  • CUDAToolkit_ROOT is correct

Python module did not build​

Meaning:

  • either pybind11 was not found
  • or Python3 Development.Module was missing

Check the CMake configure output for:

  • pybind11_FOUND
  • Python3_Development.Module_FOUND

Model Load Problems​

Invalid magic or unsupported version​

Meaning:

  • the file is not a valid .tinyloop artifact
  • or the artifact version is outside the runtime's supported range

First action:

  • run inspect
  • confirm the converter wrote the expected version

dim must be divisible by n_heads​

Meaning:

  • the artifact header is inconsistent or corrupted

Check:

  • source checkpoint config
  • converter logic
  • written header fields

Unsupported n_pre_blocks​

Meaning:

  • the current runtime assumes the narrow looped-transformer family it was built for

This is not a one-line patch. It is a deliberate runtime-extension task.

Runtime Problems​

CUDA allocation failure​

Common causes:

  • max_seq_len too large
  • model too large for the device
  • FP16 body cache mode enabled on a too-small GPU

Try:

  • smaller max_seq_len
  • inspect before loading
  • lower-memory runtime mode
  • disable TINYLOOP_EXPERIMENTAL_FP16_BODY=1

Generation output looks wrong​

First isolate the source:

  1. compare cached vs uncached generation with TINYLOOP_DISABLE_KV_CACHE=1
  2. compare new prefill path vs reference with TINYLOOP_DISABLE_FLASH2_PREFILL=1
  3. if you are using the CLI, remember it tokenizes prompt text as raw bytes

If the Python/tokenizer-backed path looks right and the CLI does not, the problem may be text preprocessing rather than the runtime.

Benchmark numbers look inconsistent​

Check:

  • whether the run is cold or warmed
  • whether TINYLOOP_CUDA_PROFILE=1 changed warmup behavior
  • whether TINYLOOP_EXPERIMENTAL_FP16_BODY=1 was enabled
  • whether another process was using the same GPU

KV cache mode env vars appear ignored​

Symptom: you set TINYLOOP_KV_H_MODE=1 (or TINYLOOP_KV_INT8 / TINYLOOP_KV_INT4) but the decode latency and VRAM match the baseline FP16 KV path.

Cause: the env vars are probed on the first cache-allocation call and cached in a module-level static. If they were set after the first Model construction or the first generate call in the same process, they have no effect.

Fix:

  • Set all TINYLOOP_KV_* env vars before import tinyloop_py in Python, or before load_model() in C++.
  • For in-process mode switching (benchmarking multiple modes), use subprocess isolation — see Python API → KV cache modes for the template.

INT4-h falls back to FP16-h with a warning​

Symptom: load log prints TINYLOOP_KV_INT4 requires head_dim to be even.

Cause: dim / n_heads is odd. INT4-h packs two nibbles per byte, so head_dim must be even.

Fix: use a model with an even head_dim. All reference TinyLoop checkpoints (407M, 1B-effective) satisfy head_dim = 128 and are unaffected.

INT4-h argmax differs from FP16-h on specific prompts​

Symptom: greedy-decoded tokens diverge between FP16-h and INT4-h on some prompts.

Cause: INT4-h introduces ~10 % L2-relative K/V reconstruction error. On close-call logits (two candidates within ~0.1 in FP16), the reconstruction noise can flip the argmax. This is expected behaviour, not a bug — the error envelope is documented in the KV cache modes page.

Mitigation:

  • Use INT8 h instead (~0.6 % L2-rel, argmax flips are rare).
  • Use FP16 h for quality-sensitive workloads.
  • If you have measured quality loss you consider unacceptable, please file an issue with the prompt + tokens.

Warm-start decode disagrees with fresh generate​

Symptom: resume_generate(build_resume_handle(prompt, L_used=k), new_L=L) produces different tokens than generate(prompt, loops=L).

Under greedy decoding (temperature=0.0) this is a bug — the two paths should be bit-exact. Report it.

Under non-greedy sampling (temperature>0) this is expected: the warm-start path does not share the sampling PRNG state with the fresh-generate path. Nonzero-temperature parity is an open follow-up.

Prefix cache loops mismatch​

Symptom: RuntimeError: loops mismatch on generate_from_prefix_cache.

Cause: the loops kwarg on generate_from_prefix_cache differs from the loops value used at build_prefix_cache time. A prefix is depth-specialized — you cannot reuse an L=8 prefix to serve an L=16 decode.

Fix:

  • Match the kwargs. Use the same loops on both calls.
  • If you need multiple depths on the same prompt, use warm-start mid-loop instead, or maintain one PrefixCache per depth.

OOM during long prefill with TINYLOOP_EXPERIMENTAL_FP16_BODY=1​

Symptom: cudaMalloc failure during model load or early generation; GPU has enough memory for the model's INT4 size but not for the FP16 cache.

Cause: the FP16 body cache triples weight VRAM (1B-effective: 222 → 633 MB).

Mitigation options, in order of preference:

  1. Leave TINYLOOP_EXPERIMENTAL_FP16_BODY off and accept slower prefill.
  2. Enable the combination TINYLOOP_KV_H_MODE=1 TINYLOOP_KV_INT4=1 TINYLOOP_EXPERIMENTAL_FP16_BODY=1 — the INT4-h KV savings offset the FP16 body cost for long-context workloads.
  3. Reduce max_seq_len so runtime buffers + KV cache are smaller.

head_dim > 128 in alloc warning​

The INT8 / INT4 pack/unpack kernels are templated on MAX_HEAD_DIM=128. If your model has head_dim > 128, the kernels silently return without packing.

Fix: bump the template bound in cuda/kv_cache.cu, rebuild, confirm the test harness still passes.

Test Problems​

Model-dependent tests are skipping​

Set:

TINYLOOP_TEST_MODEL_PATH=/absolute/path/to/model.tinyloop

Tokenizer-aware tests are skipping​

Check:

  • the Python binding was built
  • numpy is installed
  • transformers is installed

Python regression cannot import tinyloop_py​

Set:

TINYLOOP_TEST_BUILD_DIR=/path/to/tinyloop/build

or add the build directory to PYTHONPATH.

When To Stop Guessing​

If a bug is not obvious:

  1. reproduce it with a small command
  2. compare against the closest reference path
  3. preserve the exact env flags used
  4. run the smallest relevant test or microbenchmark

TinyLoop is moving fast enough that vague debugging usually wastes time.