Troubleshooting
Common errors during install and first run, with the quickest fix.
If your issue isn't here, please file a bug at github.com/casys-kaist/LLMServingSim/issues with the full command, the error output, and your OS / Docker / GPU versions.
Submodules are missing
Symptom: Build fails with errors about missing files under
astra-sim/extern/graph_frontend/chakra/ or astra-sim/build/.
Cause: You cloned without --recurse-submodules.
Fix:
git submodule update --init --recursive
Then re-run ./scripts/compile.sh.
docker: permission denied
Symptom:
docker: Got permission denied while trying to connect to the
Docker daemon socket
Cause: Your user isn't in the docker group.
Fix:
sudo usermod -aG docker $USER
newgrp docker
# or log out and back in
GPU not detected
Symptom: Inside the vLLM container, nvidia-smi says
command not found or no devices found.
Cause: NVIDIA Container Toolkit isn't installed or Docker isn't configured to use it.
Fix: install / re-configure the toolkit (see Prerequisites) and restart Docker:
sudo systemctl restart docker
Then verify:
docker run --rm --gpus all nvidia/cuda:12.4.0-base-ubuntu22.04 nvidia-smi
If the host's nvidia-smi works but the container's doesn't, the
toolkit is the problem. If the host's nvidia-smi fails too, install
the NVIDIA driver first.
A successful CUDA 12.4 visibility probe does not verify the CUDA 13 runtime required by the default vLLM image. Check that image separately using the vLLM setup instructions.
Hugging Face: gated model / 401 / 403
Symptom: When profiling a Llama 3.x or gated Qwen variant:
huggingface_hub.utils._errors.GatedRepoError: Access to model
meta-llama/Llama-3.1-8B is restricted...
Fix:
-
Accept the license on the model page (one-time, on huggingface.co).
-
Set
HF_TOKENin your shell before launching the vLLM container:export HF_TOKEN="hf_xxxxxxxxxxxxxxxxxxxxxxxxxx"./scripts/docker-vllm.sh
The token gets forwarded into the container automatically. Check that it is set without printing the secret:
test -n "$HF_TOKEN" && echo "HF_TOKEN is set"
ASTRA-Sim build fails
Symptom: ./scripts/compile.sh errors out partway through, often
with a CMake or compiler message.
Common causes & fixes:
-
Missing build deps inside the container. The official
astrasim/tutorial-micro2024image has them by default. If you customized the image, ensurecmake,g++,protobuf-compiler,libprotobuf-dev, andlibboost-devare installed. -
Stale build state. Wipe the build directories and retry:
rm -rf astra-sim/build/astra_analytical/build/./scripts/compile.sh -
Outside the container.
compile.shis meant to run inside the simulator container, not on the host. Use./scripts/docker-sim.shfirst.
model_parallel_NPU_group <= 0 or a Chakra VersionError
Symptom: the Chakra conversion step aborts the run with
ValueError: model_parallel_NPU_group <= 0, or with
google.protobuf.runtime_version.VersionError: Detected incompatible Protobuf Gencode/Runtime versions.
Cause: Chakra is installed into the container's site-packages by
scripts/compile.sh, so an existing container keeps whatever version
was installed when it was first built. After pulling a change to the
converter or to the trace format, that stale copy can no longer read
the traces the simulator writes.
Fix: reinstall Chakra inside the simulator container:
cd astra-sim/extern/graph_frontend/chakra && pip3 install .
Rerunning ./scripts/compile.sh does the same thing along with the
C++ rebuild.
Container name already in use
Symptom:
docker: Error response from daemon: Conflict. The container name
"/servingsim_docker" is already in use by container "abc123..."
Cause: A previous run left the container around.
Fix: either re-attach or remove and recreate.
# re-attach to existing
docker start -ai servingsim_docker
# or wipe and recreate
docker rm -f servingsim_docker
./scripts/docker-sim.sh
Same idea for vllm_docker.
Missing profile data
Symptom: Running the simulator with a hardware / model combination that doesn't have profile data:
FileNotFoundError: Profile variant folder not found:
../profiler/perf/RTXPRO6000/meta-llama/Llama-3.1-8B/bf16-kvfp8. The
variant name is derived from the checkpoint (weight dtype, plus a
-kv<dtype> suffix when it declares a quantized KV cache), ...
Cause: The (hardware, model, variant) triple has no profiled
bundle. The check is on the variant folder, so the message names the
directory rather than a specific CSV. ls the parent path it prints to
see which variants you do have.
The variant is derived, and there is no flag to change it: the weight
dtype comes from quantization_config.quant_method or torch_dtype,
and a -kvfp8 suffix appears when the checkpoint declares a quantized
KV cache. Compare that expected label with the acquired bundle's source config
and recorded engine settings; automatic dtype resolution has config-specific
limitations, so labels alone do not prove compatible data. Re-profile when the
required acquisition is missing rather than relabel incompatible rows.
The profiler can also write intentionally
different variants. For quantized checkpoints a matching floating dtype keeps
the quantization name, while a different one uses -act...; those alternate
measurements are not automatically selected by the simulator.
See profiler naming.
Two neighbouring errors with different fixes:
FileNotFoundError: meta.yaml missing at <variant>/meta.yaml. Re-run the
profiler to produce it.
The folder exists but the bundle is incomplete — usually a profile run that was interrupted.
FileNotFoundError: Architecture yaml not found for model_type='gemma2'
at profiler/models/gemma2.yaml. Add profiler/models/gemma2.yaml
describing the architecture.
The model's family has no catalog yet. See Adding a model architecture.
Fix: either
- pick a hardware / model / precision combo that's already profiled
(
ls profiler/perf/), or - run the Profiler to generate the missing bundle yourself.
--max-num-batched-tokens warning at startup
Symptom:
max-num-batched-tokens=4096 exceeds profiled 2048 for
RTXPRO6000/meta-llama/Llama-3.1-8B/bf16; attention/dense lookups will
extrapolate
There is a matching one for the sequence cap:
max-num-seqs=256 exceeds profiled 128 for
RTXPRO6000/meta-llama/Llama-3.1-8B/bf16; per-sequence lookups will
extrapolate
Cause: You're running the simulator past the bounds the profiler
swept, taken from meta.yaml::engine_effective. Latency lookups
linearly extrapolate past the measured range.
Both are emitted once per (hardware, model, variant), not once per
iteration, so seeing them a single time does not mean it happened only
once. And they are silent when meta.yaml has no
engine_effective entry for the field, so a hand-authored bundle gets
no warning at all.
Fix:
- For best accuracy, re-profile at the higher
--max-num-batched-tokens(MAX_NUM_BATCHED_TOKENS=4096 ./profiler/profile.sh). - Or stay at the profiled bound. Extrapolation is usually fine for small overshoots; large ones can drift.
Simulator stuck / very slow on big workloads
Symptom: Simulation runs but takes much longer than expected, especially with MoE + EP or large prefix caches.
Common causes & fixes:
- Read-only backend output paths. The analytical backend writes to
astra-sim/log/andastra-sim/tmp__mem/, in addition to the selected inputs directory. A read-only source mount needs writable mounts for these output directories;--inputs-rootalone does not relocate the backend's logs or temporary memory files. Check backend startup errors before treating an apparent wait as a slow simulation. - Block-copy disabled. For MoE, keep
--enable-block-copyon (the default) to reuse rows for matching block/site sequences. This is exact for deterministic BALANCED/RR routing at a fixed batch shape. RAND reuses a draw and suppresses per-layer variation; turn copying off when that variation matters. Contextual profiles disable copying. - Verbose logging.
--log-level DEBUGwrites a lot. Drop to--log-level INFOorWARNING. --log-intervaltoo small. Setting it to0.1makes the logger run every 100 ms; raise to1.0(default) or higher.
masked_mha_available crash partway through a profile run
Symptom: profiling DeepSeek-V3.2 or GLM-5 dies after some minutes, with the sweep's progress bar part-way through a category:
Exception: Call to collective_rpc method failed:
'FlashInferMLASparseSM120Impl' object has no attribute 'masked_mha_available'
Cause: a vLLM 0.28.0 bug on Blackwell. FlashInferMLASparseSM120Impl
is the only sparse-MLA backend whose supports_compute_capability
accepts major == 12, so every Blackwell card uses it, and in 0.28.0 it
does not override supports_dense_mha_prefill (which
attention/backend.py defaults to True). mla_attention.py then
builds a prefill backend and reads self.impl.masked_mha_available, an
attribute only SparseMLACommonImpl sets — and that class is not in
this impl's MRO.
Fix: the default vLLM 0.31.0 image's registered SM120 implementation already
inherits the sparse common implementation and sets supports_dense_mha_prefill
to False. The obsolete repository backport of
PR #51395 has been removed.
Recreating the container with the current launcher uses this upstream fix;
an old 0.28.0 container must use the upstream fix or its matching historical
repository checkout. This specific fix does not certify all sparse-MLA paths
on the newer runtime.
profiler coverage does not catch this oneCoverage fires three fixed shots. In the affected 0.28.0 run, the failure needed a batch that took the dense-MHA prefill path, which first appeared at shot 89 of 152 in the dense category — so a clean coverage report is not evidence that a full sweep will finish.
Out of memory inside the vLLM container
Symptom: Profiler crashes with CUDA OOM, usually partway through the attention sweep — the sweep's largest shots come late, so a run can look healthy for hours first.
Fix: cut how far the KV axes reach, in profiler/profile.sh:
ATTENTION_MAX_KV=16384 # top of the kv_decode / prefill_key axes
Unset, that bound is the model's own context window, so a long-context
checkpoint sweeps to 131k or beyond and the largest decode shots ask the
engine for n_decode × kv_decode tokens of KV at once. Capping it bounds the
biggest requested history and can reduce runtime. The shot count depends on
all grid factors, query lengths and live KV capacity; see
profiling cost planning.
If that is not enough, work down this list — each one trades away something different:
| Knob | What it bounds | What you lose |
|---|---|---|
ATTENTION_MAX_KV | the top of the kv axes | long-context coverage; the simulator extrapolates past the last profiled kv |
MAX_MODEL_LEN | the engine's context, so the KV cache it reserves | the same reach, and it caps ATTENTION_MAX_KV too (the grid runs to min(the two)) |
MAX_NUM_SEQS | n_decode per shot | large-batch decode coverage |
MAX_NUM_BATCHED_TOKENS | tokens per shot, i.e. the activation peak | large prefill-chunk coverage |
GPU_MEMORY_UTILIZATION | the split between KV cache and everything else | see the caution below |
Every shot-feasibility filter is measured against the KV cache the engine
actually resolved, so GPU_MEMORY_UTILIZATION, MAX_MODEL_LEN and
MAX_NUM_SEQS change which shots the sweep contains, not just whether it
survives. A bundle profiled under one paging regime does not describe another —
keep them in step with what you simulate at, and record what you used. Lowering
GPU_MEMORY_UTILIZATION in particular cuts both ways: it leaves more room for
activations but shrinks the KV cache, which filters out more of the large-kv
shots.
MiniMax-M3 needs both a lower MAX_MODEL_LEN and a smaller
GPU_MEMORY_UTILIZATION to fit at all; profiler/profile.sh says so inline.
To skip whole categories instead, see Profiler → Running.
Still stuck?
- GitHub Issues: casys-kaist/LLMServingSim/issues
- Discussions: casys-kaist/LLMServingSim/discussions
When you file a bug, please include:
- The exact command you ran
- The full error output
- Your OS, Docker version, NVIDIA driver, GPU model
- Whether you're inside the simulator container or the vLLM container (or bare metal)