Skip to main content

Validation

LLMServingSim is validated end-to-end against real vLLM on the bundled (hardware, model) combos. The numbers below come from running a 300-request ShareGPT replay through both real vLLM and the simulator, then comparing the per-request and per-tick metrics with python -m bench validate. The tables reproduce values from the committed bench/examples/<hardware>/<model>/validation/summary.txt; the example wrappers regenerate the comparisons on CPU without starting vLLM.

Validation belongs to the recorded deployment

The RTXPRO6000 Llama and both Qwen bundles contain refreshed vLLM 0.31.0 TP1 and TP2 acquisition data. Accuracy here covers Llama TP1, Qwen3-32B TP2 and Qwen3-30B TP1/DP2/EP2. Generated metadata retains measured values and settings; only environment-dependent path identifiers are made portable. Other acquired TP and MoE deployment scopes require their own end-to-end validation. The RTX 4090 example and retained historical MoE fallback data keep their original provenance.

The largest absolute errors across all fifteen statistics for the RTXPRO6000 Llama, Qwen3-32B and Qwen3-30B examples are 0.682%, 0.456% and 1.197%.

The additional stored DeepSeek diagnostic has larger errors and is reported separately below; it is not included in the four headline configurations.

Want to validate your own change? See For Contributors → Validating your changes for the regression workflow.

Setup​

KnobValue
Workload300 ShareGPT-derived requests, ~10 sps Poisson arrivals
HardwareRTXPRO6000 and RTX 4090, single node (profile bundles in profiler/perf/<hardware>/)
vLLM versionv0.31.0 for the three RTXPRO6000 headline references; v0.19.0 for the retained RTX 4090 reference
Block size16 for the headline examples; 64 for the additional DeepSeek diagnostic
Engine flagsRecorded in each example's vllm/meta.json; match its effective settings rather than assuming current CLI defaults
Cluster configsbench/examples/<hardware>/<model>/config.json
InterconnectQwen3-32B and Qwen3-30B inherit primitive-calibrated per-operation BW and common latency from profiler/perf/RTXPRO6000/hardware.yaml; recorded backend settings remain in each reference
Target graphsExplicit V2 FULL_AND_PIECEWISE policy and capture grid in the RTXPRO6000 example configs
KV capacityLlama and Qwen3-32B use mem_util: 0.92; Qwen3-30B and RTX 4090 match recorded block counts as described below

The strict-replay path fixes each request's input token IDs and requested output length, and schedules it at the dataset's arrival offset. Generated output token content and per-request timing are measured outcomes, not pinned values. These comparisons use flat request workloads; the benchmark skips agentic-session rows.

Match capacity from recorded engine observations

npu_mem.mem_util sizes the simulated KV cache. The simulator does not model vLLM's activation peak or CUDA context, so equal utilization fractions need not produce equal block counts. The RTX 4090 example sets mem_util to reproduce the capacity in its historical single-engine meta.json:

"kv_cache": { "num_gpu_blocks": 2588, "block_size": 16, "num_kv_tokens": 41408 }

The example uses mem_util: 0.833919 to reproduce those 2,588 blocks. This matches a recorded capacity, not a latency target; its current errors are listed in the result tables below.

That JSON is the historical single-engine reference. For new DP benchmarks, use complete kv_cache.capacity.per_engine observations and match each instance to its native rank. vLLM 0.31's raw client block count sums connected engines; do not use it as one pool or divide it to infer unobserved capacities. See the capacity metadata contract.

Qwen3-30B records 38,256 blocks for each of its two DP engines. Its simulator instances use mem_util: 0.9043190002441406 to match those observations; the benchmark metadata retains its actual utilization of 0.92. Llama and Qwen3-32B use 0.92 without claiming an observed per-engine capacity match.

Check preemption in the truth's timeseries.csv and the simulator log when assessing memory pressure. Occupancy percentages have different definitions: the simulator's heartbeat includes cached-but-free blocks, while vLLM's kv_cache_pct counts pinned blocks. Compare block capacities and scheduling behavior before interpreting a difference in those percentages.

Metric definitions​

Use python -m bench validate for the comparison. Its arrival timestamp is converted from wall-clock epoch seconds to the engine's monotonic domain using the run's minimum queued_ts - arrival_time offset. Raw timestamps from those two domains cannot be subtracted directly. Using queued_ts as the arrival instead would omit frontend-to-engine waiting from TTFT; legacy records use the validator's documented fallback when arrival information is missing.

MetricDefinition in the aligned clock domain
TTFTfirst_token_ts - arrival
TPOT(last_token_ts - first_token_ts) / (output_toks - 1), for requests with more than one output token
Latencylast_token_ts - arrival

Report mean, median, P90, P95 and P99 for each metric. Benchmark accuracy is separate from simulator execution speed: Total clocks (ns) is the modeled makespan, while the log's Total simulation time is elapsed host time inside the simulator. A wrapper's wall time also includes startup and teardown. Compare wall times under the same CPU placement, dependencies, storage and logging settings; identical simulated results need not take identical host time.

Headline numbers​

Mean error vs. real vLLM, per metric, on the four configurations summarized here:

HardwareModelParallelismTTFT meanTPOT meanLatency meanworst of 15
RTX 4090Llama-3.1-8BTP=1 dense+0.5%+0.4%+0.5%+1.2%
RTXPRO6000Llama-3.1-8BTP=1 dense+0.7%+0.3%+0.4%+0.7%
RTXPRO6000Qwen3-32BTP=2 dense-0.2%+0.2%+0.1%+0.5%
RTXPRO6000Qwen3-30B-A3B-Instruct-2507DP=2 x EP=2 MoE-0.2%-0.3%-0.3%+1.2%

These workloads were used during development, including skew-axis selection. Their results are useful regression checks, not an independent estimate of generalization to other models, hardware or request distributions.

The last column is the largest absolute error across all fifteen metrics (TTFT / TPOT / latency x mean / median / P90 / P95 / P99), which is the honest summary of a run: a mean can be small because two errors cancelled. Small errors on these references do not cover arbitrary deployments or eliminate execution-to-execution TTFT variation.

Queueing can amplify small step-cost differences into larger TTFT errors. The signs of aggregate errors do not identify a unique kernel-level cause or establish generalization. Target CUDA graph padding applies before DP synchronization; these examples use measured kernel tables without a fitted per-step graph-time correction.

Per-percentile numbers are in the same summary.txt files under bench/examples/.

One recorded run does not establish generalization

The simulator is deterministic, while real DP batch pairing and queueing can vary between executions. A stored truth is a reproducible reference, not a guarantee of the same error on another run. Report all fifteen statistics against each fixed repeat when repeated references are available; TTFT and failing percentiles remain part of the accuracy assessment.

Per-configuration results​

RTX 4090 — Llama-3.1-8B (TP=1 dense)​

Throughput timeline, vLLM (blue) vs. simulator (orange, dashed):

RTX 4090 Llama-3.1-8B throughput

MetricvLLMSimDiff
TTFT mean65.49 s65.81 s+0.5%
TTFT median60.13 s60.46 s+0.6%
TTFT P99137.36 s138.15 s+0.6%
TPOT mean32.4 ms32.6 ms+0.4%
TPOT P9956.0 ms56.7 ms+1.2%
Latency mean86.61 s87.02 s+0.5%
Latency P99153.63 s154.52 s+0.6%

The 24 GB configuration saturates its KV cache. Its memory utilization setting matches the recorded KV block count, so memory-pressure behavior can be compared at the same capacity. Compute latencies are supplied by the profile bundle rather than fitted to the end-to-end results.

This is a retained vLLM v0.19.0 reference, not a v0.28 validation. Its profile bundle is the matching 0.19 one and its cluster config carries link_bw / link_latency explicitly, since there is no interconnect measurement to inherit. Those explicit link values are configuration assumptions, not measured interconnect parameters; this TP=1 example does not validate collective transport.

RTXPRO6000 — Llama-3.1-8B (TP=1 dense)​

Llama-3.1-8B throughput

MetricvLLM (ms)Sim (ms)Diff
TTFT mean6515.46559.8+0.7%
TTFT median8366.78415.1+0.6%
TTFT P9015304.415408.5+0.7%
TTFT P9517076.917088.9+0.1%
TTFT P9918387.118501.1+0.6%
TPOT mean31.431.5+0.3%
TPOT median32.632.7+0.3%
TPOT P9035.635.7+0.3%
TPOT P9535.936.0+0.3%
TPOT P9936.436.6+0.5%
Latency mean26946.427057.9+0.4%
Latency median28357.928429.1+0.3%
Latency P9033840.933950.1+0.3%
Latency P9535091.335251.9+0.5%
Latency P9936064.936158.1+0.3%

This TP1 scope uses refreshed vLLM 0.31.0 eager CUDA measurements with the target serving graph policy declared separately. Local graph padding changes forward rows without inventing requests or KV history. The reference records disable_custom_all_reduce: false; TP1 has no TP collective to exercise. A good result on this stored workload does not establish the same accuracy for another model, GPU or execution contract.

RTXPRO6000 — Qwen3-32B (TP=2 dense)​

Qwen3-32B throughput

MetricvLLM (ms)Sim (ms)Diff
TTFT mean27758.127707.5-0.2%
TTFT median31314.231323.0+0.0%
TTFT P9065070.165063.4-0.0%
TTFT P9567848.267823.0-0.0%
TTFT P9972188.372151.1-0.1%
TPOT mean67.567.7+0.2%
TPOT median70.570.8+0.5%
TPOT P9078.378.5+0.2%
TPOT P9579.679.8+0.2%
TPOT P9980.080.2+0.3%
Latency mean72714.072769.0+0.1%
Latency median79224.679287.4+0.1%
Latency P9094933.694847.6-0.1%
Latency P9599410.699249.0-0.2%
Latency P99102554.6102303.0-0.2%

TP=2 exercises the decoder all-reduces, plus the shared embedding all-reduce and logits all-gather. Head tensor sizes now use the same per-sequence row count as the profile lookup, with padded vocabulary shards and full-vocabulary sampler inputs. This deployment uses refreshed vLLM 0.31.0 TP2 measurements and the required TP1 stable-entry source.

Profile coverage alone does not establish generalization. Communication parameters and profiled compute times are not fitted to these results. This dense TP example does not validate MoE dispatch/combine or alternative heads.

This reference records custom all-reduce disabled for its NCCL communication baseline with the calibrated interconnect; the general bench default is unchanged. Its config omits link_bw, link_latency and collective_links, inheriting the hardware bundle's AllReduce and AllGather bandwidths and common latency. No benchmark-fitted bandwidth or model-specific override is applied.

RTXPRO6000 — Qwen3-30B-A3B-Instruct-2507 (DP=2 × EP=2 MoE)​

Qwen3-30B-A3B throughput

MetricvLLM (ms)Sim (ms)Diff
TTFT mean929.3927.1-0.2%
TTFT median158.0157.7-0.2%
TTFT P904202.24252.5+1.2%
TTFT P957163.77098.1-0.9%
TTFT P998796.78888.1+1.0%
TPOT mean45.945.7-0.3%
TPOT median47.347.3-0.1%
TPOT P9050.550.7+0.3%
TPOT P9551.151.0-0.1%
TPOT P9951.651.7+0.2%
Latency mean31244.631157.3-0.3%
Latency median30449.030338.4-0.4%
Latency P9037224.337200.7-0.1%
Latency P9538401.038417.5+0.0%
Latency P9942605.442462.8-0.3%

This is DP+EP, not prefill/decode disaggregation: two data-parallel members share experts and execute wave-synchronized collectives. The simulator first resolves each member's local graph padding and then the common DP mode. The head and attention request geometry remain separate from padded forward rows.

This example uses refreshed vLLM 0.31.0 eager native MoE component tables for TP1/DP2/EP2, with target serving graph padding resolved separately. Its reference retains disable_custom_all_reduce: false; TP1 does not invoke the TP custom-all-reduce path. Hidden states, top-k weights and IDs have separate collective payloads. Transport bandwidths come from primitive NCCL measurements, not these end-to-end timings; the calibration retains the existing common latency. The bundle also contains TP2 and native TP1/DP1/EP1 acquisition data; this benchmark does not validate those deployments. Its retained whole-block moe.csv is historical fallback data.

The transport path still uses a worst-rank Ring approximation for unequal contributions and does not reproduce grouped NCCL kernel execution exactly. The measured native contract covers this TP1/DP2/EP2 deployment, not arbitrary TP/DP degrees, routing histograms, quantized backends or other model families.

Additional diagnostic: DeepSeek-V3.2-Exp-16L64E​

This stored dummy-weight checkpoint reduces depth and expert count so the sparse model fits one card. It is not a full-size DeepSeek deployment. Its simulator-side refresh uses the recorded block size of 64.

MetricvLLMSimDiff
TTFT mean29.55 s24.83 s-16.0%
TTFT median32.09 s27.18 s-15.3%
TTFT P9964.81 s60.55 s-6.6%
TPOT mean55.7 ms57.3 ms+2.8%
TPOT P9977.3 ms71.6 ms-7.4%
Latency mean66.80 s63.03 s-5.6%

It remains an accuracy limitation even when its deterministic regression digest matches. All fifteen statistics are retained in its summary.txt.

Reproducing locally​

The bench module ships with reproduction scripts that re-run the simulator side and re-run the comparison against the committed vLLM artifacts:

Published examples use repository-relative dataset paths and equivalent canonical model identifiers instead of machine-local paths. All other emitted values and rows remain unchanged, including measurement settings, validation results and logs.

# Sim side: writes bench/examples/<hardware>/<model>/outputs/sim.csv
./bench/examples/run.sh # all discovered examples
./bench/examples/run.sh RTX4090/Llama-3.1-8B # or one at a time

# Compare: writes bench/examples/<hardware>/<model>/validation/{summary.txt, *.png}
./bench/examples/validate.sh
./bench/examples/validate.sh RTX4090/Llama-3.1-8B

Both scripts take <hardware>/<model> and discover the examples from the directory layout, so every number on this page comes back from the committed artifacts without editing a script.

The validation step regenerates the throughput / latency / requests plots and the headline summary. To rerun vLLM itself (instead of reusing the committed artifacts under bench/examples/<hardware>/<model>/vllm/), use python -m bench run from inside the vLLM container; see bench/README.md for the full layout.

What's next​