Validating your changes
The simulator is deterministic — the same cluster config, workload and flags
reproduce the same makespan exactly — so validation is equality against
recorded results in addition to focused local checks. serving/validate.sh
runs that comparison for you. Temporary scripts and tests written to diagnose
or verify a fix remain local under the commit policy.
1. Run the validation script (every PR)
For resource-bounded execution, wrap a command with the shared watchdog:
python3 scripts/monitor_run.py --output outputs/resources/run.csv \
--max-rss-gib 8 --min-available-gib 16 --max-swap-growth-gib 0.5 \
--timeout 1800 -- ./serving/validate.sh
These are example limits; set them for your host. The CSV records sampled
process-tree RSS and host memory, and a sibling JSON records completion or the
stop reason. Use a new log path for each run. Shared pages may be counted more
than once, and sampling is not a hard memory limit; container limits remain
the backstop. Only the launched command and its descendants are stopped.
Optional GPU telemetry requires both --gpu-uuid and --max-gpu-temp-c;
it does not allocate a GPU or check other users' ownership. Confirm availability
separately before launching GPU work.
./serving/validate.sh
Two stages, about eight minutes total:
- Behaviour — every scenario in
serving/validate-baselines.txt, compared against its recordedTotal clocks (ns). Every cluster config, every parallelism shape (TP, PP, DP and their combinations, EP), prefix caching and the tiers below it, the scheduler flags, both routing policies, PIM, CXL, P/D disaggregation, agentic sessions, two hardware profiles — and the model families: linear attention, both sparse shapes, MLA, heterogeneous stacks and speculative decoding, which nothing else reaches because the rest of the list runs Llama-3.1-8B/70B, Qwen3-30B-A3B or Qwen3-32B. - Accuracy — regenerates each
bench/examplesentry'soutputs/sim.csvandvalidation/summary.txtand checks both md5s.sim.csvis the per-request TTFT / TPOT / latency against a recorded real vLLM run;summary.txtis the error table this site quotes. Digesting both catches drift a total clock could hide, and a summary that no longer describes its ownsim.csv. The three plots are regenerated but not digested — matplotlib output is not stable across versions, and a check that fails for the wrong reason stops being read.
A clean run ends with:
Behaviour: 70/70 scenarios match their baselines.
Accuracy: all 10 sim.csv + summary.txt files are byte-identical.
That is the bar for a change that claims to be behaviour-preserving. A refactor that moves any of these numbers is not behaviour-preserving.
Useful variations:
./serving/validate.sh --clocks-only # skip the slow accuracy stage
./serving/validate.sh dp moe_dp_pp # just these scenarios, while iterating
./serving/validate.sh --list # scenario names
./serving/validate.sh --help # all options
Run it from the repo root inside the simulator container. In a fresh
container run ./scripts/compile.sh first — it installs the Chakra converter
into the container's site-packages, which does not persist, and without it
every scenario reports "did not finish". Pass LOG_DIR=<a mounted path> too,
or the per-scenario logs die with the container.
profiler/ are simulator inputsThe simulator shares data and CPU-only helpers with profiler/; changes there
can move simulation clocks without changing serving/. Important inputs include:
profiler/models/*.yaml— the layer order. Merging two catalogs into one broke all 16 MoE scenarios exactly this way.profiler/core/stack.py— which block each decoder layer runs, resolved from the checkpoint's config. Shared with the profiler on purpose.profiler/core/catalog_path.py—model_type→ yaml resolution. Also shared; adding aliasing to only one side is what broke those 16 scenarios.- Attention shape, skew calibration and MoE contract helpers — geometry, correction-table loading and deployment compatibility affect lookup.
profiler/perf/— measured data and hardware defaults.
Audit actual imports when deciding the regression scope rather than treating this list as exhaustive. Shared helpers must remain usable without importing vLLM or the GPU acquisition environment.
2. If something changed, report it
The script prints a markdown table of everything that moved, and writes
the same thing to report.md in its log directory:
## Validation report
Behaviour -- 1/58 scenarios changed:
| scenario | baseline | now | delta |
| --- | --- | --- | --- |
| `moe_dp_pp` | 1435561517 | 1435559904 | -0.0001% |
A difference is not automatically a bug — but it is never self-explanatory. Paste the table into the PR and add, per row, what in your change moved it and why the new number is the right one. Reviewers cannot tell an intended fix from an accidental regression by looking at the diff.
If the change is intended, land the new truth in the same PR:
./serving/validate.sh --update, then commitserving/validate-baselines.txt.- If a
sim.csvchanged, also run./bench/examples/validate.shand commit the regeneratedoutputs/sim.csv,validation/summary.txtand the three plots for each affected example. A changedsim.csvmakes those plots and that summary stale — leaving them behind publishes accuracy numbers for a simulator that no longer exists.
workloads/example_trace.jsonl has 2–22 token prompts, so most scenarios
never fill the KV cache and their DP members always drain together. That is
why issue #65 survived a green moe_dp_pp: the bug needed one DP member to
go idle while another was still busy. The *_uneven and saturated_*
scenarios exist for exactly those regimes. If your change targets a regime
no scenario reaches, see
the last section.
3. Bench validation (changes that affect end-to-end accuracy)
Step 1's accuracy stage tells you whether sim.csv moved. This step
tells you by how much — run it when that digest check fails, or when your
change could move the simulator's output relative to real vLLM (anything in
scheduler.py, trace_generator.py, memory_model.py, profile lookup, MoE
accounting) and you want the error numbers before opening the PR.
The bench module captures a real vLLM execution, then compares the simulator's output for the same dataset:
# 1. Rerun the sim side of an existing example
./bench/examples/run.sh RTXPRO6000/Llama-3.1-8B
# 2. Compare against the committed vLLM reference
./bench/examples/validate.sh RTXPRO6000/Llama-3.1-8B
Output lands in bench/examples/RTXPRO6000/Llama-3.1-8B/validation/:
summary.txt: mean, median, P90, P95 and P99 errors for TTFT, TPOT and latency.- Three PNGs:
latency.png(per-request latency CDF),throughput.png(throughput timeline),requests.png(running / waiting curves).
Compare all fifteen statistics against
bench/examples/<hardware>/<model>/validation/summary.txt; do not use only a
mean or the figure in the abstract. The headline Llama and Qwen examples meet
the 5% absolute-error target on all fifteen statistics. The additional reduced
DeepSeek diagnostic does not. See Validation for the
current per-configuration results and remaining limitations; these results
do not establish accuracy for unmeasured deployments or workloads.
A matching regression digest establishes reproducibility, not agreement with vLLM. Explain every intentional movement, including regressions; correcting execution geometry can expose an independent timing error. Update the simulator-side examples and their public summaries together, preserving the recorded vLLM references. When fixed repeated references are available, report all fifteen statistics against each one. Engine-side variation is not a reason to omit TTFT or a failing percentile.
For deeper detail on the validation methodology, see
bench/README.md.
4. Profiler-side changes (if you touched profiler/)
Acquisition-only changes affect simulated timings when new tables are adopted. Shared lookup, layout, geometry and metadata helpers can affect the simulator immediately, so inspect their consumers and run the corresponding CPU regression checks. For acquisition changes, run a profile with explicitly bounded grids and resource limits to confirm the pipeline still works. The command below uses CLI default sweep sizes; it is not a small smoke test by default:
# Inside the vLLM container
python -m profiler profile meta-llama/Llama-3.1-8B --hardware RTXPRO6000
If using profiler/profile.sh instead, edit its MODEL and HARDWARE
assignments first; those assignments override exported environment values.
Then verify the simulator still loads it cleanly:
./serving/validate.sh --clocks-only single.
For a calibration-only change (skew_calibration.py), rebuild a copy of
the existing bundle with python -m profiler refit-skew MODEL --hardware HW --out PROFILE_ROOT. This CPU-only command does not remeasure skew or alter
attention. --only-skew, in contrast, is an acquisition command.
Run the committed bench examples and compare every reported latency statistic;
keep temporary tests, scripts and validation artifacts out of commits.
If you touched hardware.yaml
It feeds the simulator, so a change there can move every clock in
validate.sh even though the file lives under profiler/.
python -m profiler hardware prints the fitted link_bw / link_latency,
the mean residual and the worst one, plus a per-collective breakdown; the
residual per size is kept in the file. Two things to check:
- Check each collective against the backend, not just the fit equation. The calibration contract specifies Ring phases, GiB/s network units and local reduction costs. At two ranks, AllReduce has two network phases but AllGather/ReduceScatter have one. A common pair may leave substantial residuals even with the correct formula; report them rather than tuning against a model benchmark. CPU single-collective traces can verify the equation against the ASTRA binary.
- Check the timing scope.
fit.timingrecords graphed, isolated or mixed inputs. For a captured execution target, verify that graph capture succeeded; isolated timings include host launch/synchronisation overhead. Individual primitives also do not validate grouped multi-tensor or uneven-rank MoE dispatch/combine, which require their own execution-path checks.
Two GPUs are needed for the second one. Without them the command writes the
spec section, records interconnect: null, and exits non-zero.
What "this should reproduce" looks like in a PR
In your PR description, include the exact command you ran and the key number from the output. Examples:
Validation:
./bench/examples/validate.sh RTXPRO6000/Llama-3.1-8B→ Report the signed Diff% for all fifteen TTFT / TPOT / latency statistics fromsummary.txt, together with changes from the prior fixed reference.
Validation:
./serving/validate.sh→ all recorded scenarios match their baselines; all discoveredsim.csvandsummary.txtdigests match.
This gives the reviewer something to rerun, and gives you (and future readers of the git log) a record of what was checked.
When the existing scenarios don't cover what you changed
If your contribution adds a feature that no bundled scenario
exercises, add a scenario as part of the PR. Add a line to the
SCENARIOS list in serving/validate.sh (and a
configs/cluster/<your_scenario>.json if no bundled config fits), then
record its baseline with ./serving/validate.sh --update <name> and commit
both. That makes the feature reproducible for the next contributor instead
of relying on them to think of it.
Prefer a scenario that would fail without your change. A case whose clock
matches an existing scenario exercises the flag's parsing and nothing else —
check the new number differs from the closest existing one, and if it does
not, find a configuration where the flag actually bites (turning the KV
cache saturated with --npu-memory-utilization is usually enough).
serving/run.sh is a menu of one example per feature, not a test suite —
adding to it does not get your case validated.
For features that need a custom workload (a new agentic dataset, a
specific prompt distribution), commit a small JSONL under
workloads/ and reference it from the cluster config example.
Don't commit anything over a few MB.
What's next
- PR workflow: how to package the change up.
- Reading the output: what the per-request CSV columns mean (useful when validating).