Profiler

The profiler is a vLLM-based layerwise profiler. It drives a real vLLM engine with synthetic batches and records per-layer CUDA kernel latencies into per-category CSV files. Those CSVs are exactly what the simulator's trace_generator reads at run time, the profiler's output IS the simulator's input.

When you need to run it

You don't need to run the profiler if your hardware × model combination is already in the bundled profile data. Otherwise:

Scenario	Profile?
Running a bundled (hardware, model) combo (e.g., RTXPRO6000 + Llama-3.1-8B)	No, just simulate
New GPU (e.g., H100, A100) with a bundled model	Yes, see Adding new hardware
Bundled GPU with a new model (Mistral-7B, Phi-3.5-MoE, …)	Maybe, see Adding model architecture
Non-GPU accelerator (TPU, custom NPU)	Yes, but a different workflow, see Adding non-GPU hardware

What it produces

For each (hardware, model, variant) profiled, the profiler writes a folder under profiler/perf/<hardware>/<model>/<variant>/ with one tp<N>/ subfolder per profiled tensor-parallel degree:

perf/<hardware>/<model>/<variant>/
├── meta.yaml                       # engine flags, sweep specs, skew_fit summary
└── tp<N>/
    ├── dense.csv                   # token-count → latency
    ├── per_sequence.csv            # seq-count → latency
    ├── attention.csv               # 4D: (pc, kv_pre, n_dec, kv_dec) → latency
    ├── moe.csv                     # MoE only: (tokens, experts) → latency
    ├── skew.csv                    # raw heterogeneous-decode shots
    └── skew_fit.csv                # fitted per-bucket alpha table

Times are stored in microseconds (time_us column); the simulator multiplies by 1000 and rounds to ns at load time.

Schema details on Output bundle.

How it fits the bigger picture

The profiler runs on the vLLM Docker container (or bare metal via scripts/install-vllm.sh). The simulator runs on the simulator container (astrasim/tutorial-micro2024). They share the profiler/perf/ directory, that's the only thing they exchange.

Bundled profile data

Hardware	Model	Variant	TP degrees
RTXPRO6000	meta-llama/Llama-3.1-8B	bf16	1, 2
RTXPRO6000	Qwen/Qwen3-32B	bf16	1, 2
RTXPRO6000	Qwen/Qwen3-30B-A3B-Instruct-2507	bf16	1, 2
RTX4090	meta-llama/Llama-3.1-8B	bf16	1

That is the whole set: two hardware targets, three models, bf16 only, and only the RTXPRO6000 bundles carry TP=2. If your (hardware, model, variant, tp) combo is in this table you can skip the profiler entirely; anything else needs a profile run. The RTX 4090 bundle was contributed in #59 and is the one validated end-to-end inside 1% on every metric — see Validation.

Two consequences that catch people out:

• No -kvfp8 bundle ships. --kv-cache-dtype fp8 resolves to a bf16-kvfp8 variant folder that does not exist, so it fails at startup until you profile it with KV_CACHE_DTYPE=fp8. See Examples → FP8 KV cache.
• TP=4 and above need a profile run, even on RTXPRO6000 with a bundled model, and so does TP=2 on RTX4090 — that bundle is TP=1 only. _load_perf_db hard-errors on a missing tp<N>/ folder rather than extrapolating.

Prerequisites

• vLLM Docker container running at /workspace (mounts repo root). See Installation → vLLM setup.
• NVIDIA GPU (only for the profiler, the simulator runs on CPU).
• HF_TOKEN environment variable for gated model configs (Llama 3.x, etc.). Set this in scripts/docker-vllm.sh before launching.
• A few GB of GPU memory for the model variant you're profiling (TP=1 needs the full model; TP=N needs model_size / N).

Where to go next

Running

Edit profile.sh, pick options for your sweep, hit go.

Output bundle

Schema reference for every CSV the profiler emits.

Skew & alpha fit

How the heterogeneous-decode correction is profiled and fit.

Adding new hardware

GPU (vLLM-supported) or non-GPU (TPU, custom accelerator).

Adding a model architecture

When to write a new architecture YAML, and what to put in it.