Skip to main content

Troubleshooting

Common errors during install and first run, with the quickest fix.

If your issue isn't here, please file a bug at github.com/casys-kaist/LLMServingSim/issues with the full command, the error output, and your OS / Docker / GPU versions.

Submodules are missing​

Symptom: Build fails with errors about missing files under astra-sim/extern/graph_frontend/chakra/ or astra-sim/build/.

Cause: You cloned without --recurse-submodules.

Fix:

git submodule update --init --recursive

Then re-run ./scripts/compile.sh.

docker: permission denied​

Symptom:

docker: Got permission denied while trying to connect to the
Docker daemon socket

Cause: Your user isn't in the docker group.

Fix:

sudo usermod -aG docker $USER
newgrp docker
# or log out and back in

GPU not detected​

Symptom: Inside the vLLM container, nvidia-smi says command not found or no devices found.

Cause: NVIDIA Container Toolkit isn't installed or Docker isn't configured to use it.

Fix: install / re-configure the toolkit (see Prerequisites) and restart Docker:

sudo systemctl restart docker

Then verify:

docker run --rm --gpus all nvidia/cuda:12.4.0-base-ubuntu22.04 nvidia-smi

If the host's nvidia-smi works but the container's doesn't, the toolkit is the problem. If the host's nvidia-smi fails too, install the NVIDIA driver first.

A successful CUDA 12.4 visibility probe does not verify the CUDA 13 runtime required by the default vLLM image. Check that image separately using the vLLM setup instructions.

Hugging Face: gated model / 401 / 403​

Symptom: When profiling a Llama 3.x or gated Qwen variant:

huggingface_hub.utils._errors.GatedRepoError: Access to model
meta-llama/Llama-3.1-8B is restricted...

Fix:

  1. Accept the license on the model page (one-time, on huggingface.co).

  2. Set HF_TOKEN in your shell before launching the vLLM container:

    export HF_TOKEN="hf_xxxxxxxxxxxxxxxxxxxxxxxxxx"
    ./scripts/docker-vllm.sh

The token gets forwarded into the container automatically. Check that it is set without printing the secret:

test -n "$HF_TOKEN" && echo "HF_TOKEN is set"

ASTRA-Sim build fails​

Symptom: ./scripts/compile.sh errors out partway through, often with a CMake or compiler message.

Common causes & fixes:

  • Missing build deps inside the container. The official astrasim/tutorial-micro2024 image has them by default. If you customized the image, ensure cmake, g++, protobuf-compiler, libprotobuf-dev, and libboost-dev are installed.

  • Stale build state. Wipe the build directories and retry:

    rm -rf astra-sim/build/astra_analytical/build/
    ./scripts/compile.sh
  • Outside the container. compile.sh is meant to run inside the simulator container, not on the host. Use ./scripts/docker-sim.sh first.

model_parallel_NPU_group <= 0 or a Chakra VersionError​

Symptom: the Chakra conversion step aborts the run with ValueError: model_parallel_NPU_group <= 0, or with google.protobuf.runtime_version.VersionError: Detected incompatible Protobuf Gencode/Runtime versions.

Cause: Chakra is installed into the container's site-packages by scripts/compile.sh, so an existing container keeps whatever version was installed when it was first built. After pulling a change to the converter or to the trace format, that stale copy can no longer read the traces the simulator writes.

Fix: reinstall Chakra inside the simulator container:

cd astra-sim/extern/graph_frontend/chakra && pip3 install .

Rerunning ./scripts/compile.sh does the same thing along with the C++ rebuild.

Container name already in use​

Symptom:

docker: Error response from daemon: Conflict. The container name
"/servingsim_docker" is already in use by container "abc123..."

Cause: A previous run left the container around.

Fix: either re-attach or remove and recreate.

# re-attach to existing
docker start -ai servingsim_docker

# or wipe and recreate
docker rm -f servingsim_docker
./scripts/docker-sim.sh

Same idea for vllm_docker.

Missing profile data​

Symptom: Running the simulator with a hardware / model combination that doesn't have profile data:

FileNotFoundError: Profile variant folder not found:
../profiler/perf/RTXPRO6000/meta-llama/Llama-3.1-8B/bf16-kvfp8. The
variant name is derived from the checkpoint (weight dtype, plus a
-kv<dtype> suffix when it declares a quantized KV cache), ...

Cause: The (hardware, model, variant) triple has no profiled bundle. The check is on the variant folder, so the message names the directory rather than a specific CSV. ls the parent path it prints to see which variants you do have.

The variant is derived, and there is no flag to change it: the weight dtype comes from quantization_config.quant_method or torch_dtype, and a -kvfp8 suffix appears when the checkpoint declares a quantized KV cache. Compare that expected label with the acquired bundle's source config and recorded engine settings; automatic dtype resolution has config-specific limitations, so labels alone do not prove compatible data. Re-profile when the required acquisition is missing rather than relabel incompatible rows. The profiler can also write intentionally different variants. For quantized checkpoints a matching floating dtype keeps the quantization name, while a different one uses -act...; those alternate measurements are not automatically selected by the simulator. See profiler naming.

Two neighbouring errors with different fixes:

FileNotFoundError: meta.yaml missing at <variant>/meta.yaml. Re-run the
profiler to produce it.

The folder exists but the bundle is incomplete — usually a profile run that was interrupted.

FileNotFoundError: Architecture yaml not found for model_type='gemma2'
at profiler/models/gemma2.yaml. Add profiler/models/gemma2.yaml
describing the architecture.

The model's family has no catalog yet. See Adding a model architecture.

Fix: either

  • pick a hardware / model / precision combo that's already profiled (ls profiler/perf/), or
  • run the Profiler to generate the missing bundle yourself.

--max-num-batched-tokens warning at startup​

Symptom:

max-num-batched-tokens=4096 exceeds profiled 2048 for
RTXPRO6000/meta-llama/Llama-3.1-8B/bf16; attention/dense lookups will
extrapolate

There is a matching one for the sequence cap:

max-num-seqs=256 exceeds profiled 128 for
RTXPRO6000/meta-llama/Llama-3.1-8B/bf16; per-sequence lookups will
extrapolate

Cause: You're running the simulator past the bounds the profiler swept, taken from meta.yaml::engine_effective. Latency lookups linearly extrapolate past the measured range.

Both are emitted once per (hardware, model, variant), not once per iteration, so seeing them a single time does not mean it happened only once. And they are silent when meta.yaml has no engine_effective entry for the field, so a hand-authored bundle gets no warning at all.

Fix:

  • For best accuracy, re-profile at the higher --max-num-batched-tokens (MAX_NUM_BATCHED_TOKENS=4096 ./profiler/profile.sh).
  • Or stay at the profiled bound. Extrapolation is usually fine for small overshoots; large ones can drift.

Simulator stuck / very slow on big workloads​

Symptom: Simulation runs but takes much longer than expected, especially with MoE + EP or large prefix caches.

Common causes & fixes:

  • Read-only backend output paths. The analytical backend writes to astra-sim/log/ and astra-sim/tmp__mem/, in addition to the selected inputs directory. A read-only source mount needs writable mounts for these output directories; --inputs-root alone does not relocate the backend's logs or temporary memory files. Check backend startup errors before treating an apparent wait as a slow simulation.
  • Block-copy disabled. For MoE, keep --enable-block-copy on (the default) to reuse rows for matching block/site sequences. This is exact for deterministic BALANCED/RR routing at a fixed batch shape. RAND reuses a draw and suppresses per-layer variation; turn copying off when that variation matters. Contextual profiles disable copying.
  • Verbose logging. --log-level DEBUG writes a lot. Drop to --log-level INFO or WARNING.
  • --log-interval too small. Setting it to 0.1 makes the logger run every 100 ms; raise to 1.0 (default) or higher.

masked_mha_available crash partway through a profile run​

Symptom: profiling DeepSeek-V3.2 or GLM-5 dies after some minutes, with the sweep's progress bar part-way through a category:

Exception: Call to collective_rpc method failed:
'FlashInferMLASparseSM120Impl' object has no attribute 'masked_mha_available'

Cause: a vLLM 0.28.0 bug on Blackwell. FlashInferMLASparseSM120Impl is the only sparse-MLA backend whose supports_compute_capability accepts major == 12, so every Blackwell card uses it, and in 0.28.0 it does not override supports_dense_mha_prefill (which attention/backend.py defaults to True). mla_attention.py then builds a prefill backend and reads self.impl.masked_mha_available, an attribute only SparseMLACommonImpl sets — and that class is not in this impl's MRO.

Fix: the default vLLM 0.31.0 image's registered SM120 implementation already inherits the sparse common implementation and sets supports_dense_mha_prefill to False. The obsolete repository backport of PR #51395 has been removed. Recreating the container with the current launcher uses this upstream fix; an old 0.28.0 container must use the upstream fix or its matching historical repository checkout. This specific fix does not certify all sparse-MLA paths on the newer runtime.

profiler coverage does not catch this one

Coverage fires three fixed shots. In the affected 0.28.0 run, the failure needed a batch that took the dense-MHA prefill path, which first appeared at shot 89 of 152 in the dense category — so a clean coverage report is not evidence that a full sweep will finish.

Out of memory inside the vLLM container​

Symptom: Profiler crashes with CUDA OOM, usually partway through the attention sweep — the sweep's largest shots come late, so a run can look healthy for hours first.

Fix: cut how far the KV axes reach, in profiler/profile.sh:

ATTENTION_MAX_KV=16384 # top of the kv_decode / prefill_key axes

Unset, that bound is the model's own context window, so a long-context checkpoint sweeps to 131k or beyond and the largest decode shots ask the engine for n_decode × kv_decode tokens of KV at once. Capping it bounds the biggest requested history and can reduce runtime. The shot count depends on all grid factors, query lengths and live KV capacity; see profiling cost planning.

If that is not enough, work down this list — each one trades away something different:

KnobWhat it boundsWhat you lose
ATTENTION_MAX_KVthe top of the kv axeslong-context coverage; the simulator extrapolates past the last profiled kv
MAX_MODEL_LENthe engine's context, so the KV cache it reservesthe same reach, and it caps ATTENTION_MAX_KV too (the grid runs to min(the two))
MAX_NUM_SEQSn_decode per shotlarge-batch decode coverage
MAX_NUM_BATCHED_TOKENStokens per shot, i.e. the activation peaklarge prefill-chunk coverage
GPU_MEMORY_UTILIZATIONthe split between KV cache and everything elsesee the caution below
These are not free knobs

Every shot-feasibility filter is measured against the KV cache the engine actually resolved, so GPU_MEMORY_UTILIZATION, MAX_MODEL_LEN and MAX_NUM_SEQS change which shots the sweep contains, not just whether it survives. A bundle profiled under one paging regime does not describe another — keep them in step with what you simulate at, and record what you used. Lowering GPU_MEMORY_UTILIZATION in particular cuts both ways: it leaves more room for activations but shrinks the KV cache, which filters out more of the large-kv shots.

MiniMax-M3 needs both a lower MAX_MODEL_LEN and a smaller GPU_MEMORY_UTILIZATION to fit at all; profiler/profile.sh says so inline.

To skip whole categories instead, see Profiler → Running.

Still stuck?​

When you file a bug, please include:

  1. The exact command you ran
  2. The full error output
  3. Your OS, Docker version, NVIDIA driver, GPU model
  4. Whether you're inside the simulator container or the vLLM container (or bare metal)