Profiler

The profiler is a vLLM-based layerwise profiler. It drives a real vLLM engine with synthetic batches and records per-layer CUDA kernel latencies into per-category CSV files. Those CSVs are exactly what the simulator's trace_generator reads at run time, the profiler's output IS the simulator's input.

When you need to run it

You don't need to run the profiler if your hardware × model combination and required execution geometry are already covered by the bundled profile data. For native MoE, that includes target TP/DP/EP, backend and rank ownership, not just a matching model name. Otherwise:

Scenario	Profile?
Running a bundled (hardware, model) combo (e.g., RTXPRO6000 + Llama-3.1-8B)	No, just simulate
New GPU (e.g., H100, A100) with a bundled model	Yes, see Adding new hardware
Bundled GPU with a new model (Mistral-7B, Phi-3.5-MoE, …)	Maybe, see Adding model architecture
Non-GPU accelerator (TPU, custom NPU)	Yes, but a different workflow, see Adding non-GPU hardware

What it produces

For each (hardware, model, variant) profiled, the profiler writes a folder under profiler/perf/<hardware>/<model>/<variant>/ with one tp<N>/ subfolder per profiled tensor-parallel degree:

perf/<hardware>/<model>/<variant>/
├── meta.yaml                       # engine flags, sweep specs, skew_fit summary
└── tp<N>/
    ├── dense.csv                   # token-count → latency
    ├── per_sequence.csv            # seq-count → latency
    ├── attention.csv               # per-layer, per-q: (pc, kv_pre, n_dec, kv_dec)
    ├── linear_attention.csv        # hybrid only: per-layer (prefill_tokens, n_dec)
    ├── moe.csv                     # whole-block MoE: (ep, tokens, experts)
    ├── skew.csv                    # raw heterogeneous-decode shots
    └── skew_fit.csv                # fitted per-bucket alpha table

Times are stored in microseconds (time_us column); the simulator multiplies by 1000 and rounds to ns at load time.

Schema details on Output bundle.

Explicit-DP native MoE profiling adds a versioned component index and deployment-specific tables alongside this bundle. It preserves global top-k and separates local routing, gathered expert work and local finalization, with independently measured eager/graph GPU timings.

How it fits the bigger picture

The profiler runs on the vLLM Docker container (or bare metal via scripts/install-vllm.sh). The simulator runs on the simulator container (astrasim/tutorial-micro2024). They share the profiler/perf/ directory, that's the only thing they exchange.

Bundled profile data

The following bundles drive the headline Llama and Qwen validation examples:

Hardware	Model	Variant	TP degrees
RTXPRO6000	meta-llama/Llama-3.1-8B	bf16	1, 2
RTXPRO6000	Qwen/Qwen3-32B	bf16	1, 2
RTXPRO6000	Qwen/Qwen3-30B-A3B-Instruct-2507	bf16	1, 2
RTX4090	meta-llama/Llama-3.1-8B	bf16	1

Additional sparse-attention and hybrid-model bundles live under profiler/perf/. The tracked folders and metadata are the authoritative model/variant/TP inventory; the table above is not exhaustive. Existing data avoids a new acquisition only when its shapes, backend and coverage match the target. Native MoE also requires a matching deployment contract, not merely an existing TP folder.

Updating profiler code does not rewrite these measurements. Their metadata and row identities determine acquisition provenance; reproducing a stored example does not validate a complete fresh-only bundle from the current protocol.

The RTX 4090 bundle was contributed in #59; it retains vLLM 0.19 measurements. See Validation for current per-statistic results and the distinction between profile availability and measured accuracy.

Two consequences that catch people out:

• KV precision must match the bundle. The simulator derives it from the model config, not a --kv-cache-dtype flag, and rejects a missing variant. The profiler's KV_CACHE_DTYPE=fp8 option can acquire that variant; it does not change the simulator's model config. See Examples → FP8 KV cache.
• TP=4 and above need a profile run, even on RTXPRO6000 with a bundled model, and so does TP=2 on RTX4090 — that bundle is TP=1 only. _load_perf_db hard-errors on a missing tp<N>/ folder rather than extrapolating.

Prerequisites

• vLLM Docker container running at /workspace (mounts repo root). See Installation → vLLM setup.
• NVIDIA GPU (only for the profiler, the simulator runs on CPU).
• HF_TOKEN environment variable for gated model configs (Llama 3.x, etc.). Export it in your shell before launching.
• Enough GPU memory for the category's shrunken per-rank stack, KV cache, activations and backend workspaces. Profiling normally uses dummy weights and the minimum layer prefix needed to expose the category, not the full checkpoint. Native MoE has its own deployment contract. Real bench run recordings normally load the full model.

Where to go next

Running

Edit profile.sh, pick options for your sweep, hit go.

Output bundle

Schema reference for every CSV the profiler emits.

Skew & alpha fit

How the heterogeneous-decode correction is profiled and fit.

Adding new hardware

GPU (vLLM-supported) or non-GPU (TPU, custom accelerator).

Adding a model architecture

When to write a new architecture YAML, and what to put in it.