Prerequisites
LLMServingSim runs on Linux with Docker. The simulator side runs on CPU, but the profiler and the vLLM benchmark need an NVIDIA GPU.
System
| Required for Simulator | Required for Profiler / Bench | |
|---|---|---|
| OS | Linux (Ubuntu 22.04+ tested) | Linux (Ubuntu 22.04+ tested) |
| Docker | ✓ | ✓ (or bare-metal install) |
| NVIDIA GPU | ✓ | |
| NVIDIA Container Toolkit | ✓ (for GPU passthrough into Docker) | |
| CUDA driver | 13.x or compatible | |
| Disk | ~3 GB | ~10 GB additional (vLLM image + HF model cache) |
| RAM | 16 GB | 32 GB+ recommended |
If you only plan to run pre-profiled simulations (e.g., the bundled RTXPRO6000 profiles), you do not need a GPU.
Install Docker
If you don't already have Docker:
# Ubuntu, official quick-install script
curl -fsSL https://get.docker.com | sh
sudo usermod -aG docker $USER
newgrp docker
Verify:
docker run --rm hello-world
Install NVIDIA Container Toolkit
Required only for GPU containers (profiler / bench). On Ubuntu:
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | \
sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
sudo apt-get update
sudo apt-get install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker
Verify:
docker run --rm --gpus all nvidia/cuda:12.4.0-base-ubuntu22.04 nvidia-smi
You should see your GPU listed. If not, see Troubleshooting → GPU not detected.
Hugging Face token (optional)
Some model configs (e.g., Llama 3.x, gated Qwen variants) live behind HF authentication. The profiler can auto-fetch these if you set:
export HF_TOKEN="hf_xxxxxxxxxxxxxxxxxxxxxxxxxx"
You only need this if you plan to profile new models. Running pre-profiled simulations does not require an HF token.
Get a token from huggingface.co/settings/tokens.
Next
You're ready to install. Continue with Simulator setup
- this is the main install path that everyone needs.