Skip to main content

vLLM setup

This step is optional. You only need the vLLM environment if you plan to:

  • Profile a new GPU or model (python -m profiler)
  • Benchmark real vLLM end-to-end (python -m bench run)
  • Generate ShareGPT-style workload datasets (python -m workloads.generators)

If you only want to run pre-profiled simulations on the bundled hardware (RTXPRO6000 or RTX 4090), skip this page. Comparing stored benchmarks with python -m bench validate also runs in the CPU-only simulator environment.

Choose an install method​

The Docker path uses the official vllm/vllm-openai:v0.31.0 image, which already contains vLLM, PyTorch, and all CUDA dependencies.

That tag uses CUDA 13 and requires a compatible NVIDIA driver. A different CUDA image or vLLM version needs separate compatibility checks. The pin matters beyond the driver: every module under profiler/core/hooks/ binds to a vLLM internal API — the MoE router, the model runner's KV cache groups and execution interfaces — and those move between releases, so a different vLLM can break profiling silently. An image upgrade does not establish every model/backend's acquisition coverage or fresh simulation accuracy. Existing profile bundles and benchmark truths retain their recorded versions; see validation.

On a shared machine, set VLLM_GPUS to keep the container off GPUs someone else is using. The inner quotes are part of the value:

VLLM_GPUS='"device=2,3"' ./scripts/docker-vllm.sh

Without them Docker splits on the comma and reads the second field as a GPU count, failing with cannot set both Count and DeviceIDs.

1. (Optional) Set HF_TOKEN​

Some model configs (Llama 3.x, gated Qwen variants) need a Hugging Face token to auto-fetch:

export HF_TOKEN="hf_xxxxxxxxxxxxxxxxxxxxxxxxxx"

Skip this if you only profile open models or use locally-stored configs.

2. Launch the vLLM container​

From the repo root:

./scripts/docker-vllm.sh

This:

  • Requests the GPUs selected by VLLM_GPUS (all GPUs if unset)
  • Forwards HF_TOKEN from your shell
  • Mounts the repo root to /workspace (so python -m profiler, python -m bench, python -m workloads.generators all work)
  • Mounts the Hugging Face and vLLM compile caches to share them with the host
  • Sets --shm-size=16g (vLLM needs this for inter-process tensors)
  • Pre-installs pandas, matplotlib and datasets (the alpha fit, the bench plots, and the workload generators), and pins nvidia-nccl-cu13==2.29.7 to the image's Torch 2.13.0+cu130 requirement. This package pin does not force NCCL-only communication
  • Applies the MiniMax-M3 MTP layer-name fix and two optional observers in scripts/patches/. The SM120 sparse-MLA fix is already upstream in 0.31.0, so its backport has been removed. Recheck registered implementations and patch anchors when changing vLLM versions
  • Drops you into bash at /workspace

The container is named vllm_docker. Re-attach later with:

docker start -ai vllm_docker

3. Verify the install​

Inside the container:

python -c "import vllm; print(vllm.__version__)"
nvidia-smi

You should see vLLM 0.31.0 and your GPU listed.

Optional diagnostics​

Neither observer is enabled by the launcher. VLLM_STEP_SHAPE_LOG records scheduled shapes and CPU execute_model entry timestamps, including when model work uses graphs. The intervals are not GPU step durations; its legacy prefill labels use a query-length heuristic that does not distinguish one-token prefills from decode or speculative multi-token decode from prefill.

bench --record-gate-stats enables VLLM_MOE_ACTIVATED_LOG and requires --enforce-eager. This observer measures distinct experts on modular router paths; monolithic backends can bypass it. It adds work and synchronization, so do not use an enabled diagnostic run as uninstrumented latency ground truth. The retained MTP namespace patch is a startup fix, not certification of every quantized checkpoint or speculative-decoding backend.

What's next​

  • Profiler guide: capture per-layer CUDA kernel timings into the per-category CSV bundle the simulator consumes.
  • Bench CLI: run vLLM end-to-end and validate the simulator against ground truth, with every flag for both subcommands.
  • Validation: the accuracy numbers those runs produced.