Skip to main content

Adding new hardware

This page is the workflow for bringing up a brand-new hardware target that doesn't have a profile bundle in profiler/perf/<HARDWARE>/ yet. The bundled acquisition path targets NVIDIA CUDA and vLLM 0.31 internals. vLLM support alone does not establish that the profiler's CUDA attribution, cache preparation and hardware probes support another accelerator:

The CSV bundle format described on Output bundle is the contract. Once you produce one, the simulator works the same way regardless of how the data was collected.

Adding a new GPU​

For a compatible NVIDIA target, start with the workflow below. Check live kernel coverage and backend support before committing to a full sweep.

0. Measure the machine​

python -m profiler hardware --hardware <LABEL> --npus 2

The command defaults to INFO logging; no verbosity flag is required. Use --log-level DEBUG, INFO, WARNING or ERROR to override it. Invalid levels are rejected before probing the GPU.

Writes profiler/perf/<LABEL>/hardware.yaml: the card's spec, queried from the device, and NCCL AllReduce, AllGather and ReduceScatter sweeps used to fit shared link_bw / link_latency parameters for the analytical backend, plus per-operation bandwidths at the shared fitted latency. Cluster configs on this hardware then inherit those instead of carrying a guess, and every inherited value is logged with its provenance (measured / spec / assumed).

Do this first — it characterises the machine, not a model, and the same file serves every model bundle in the folder. Re-measure after interconnect or communication-software changes.

It needs two of the cards. With one, the spec section is still written, interconnect is null with the reason, and the command exits non-zero; a cluster config then has to set link_bw and link_latency explicitly. That is the honest outcome: a link has two ends, and the simulator will not substitute a number nobody measured.

Calibration contract​

The fit models a single-chunk Ring in one local dimension on a one-hop FullyConnected topology. It uses the actual measured rank count N. The raw samples[].bytes field, denoted S below, has these meanings:

CollectiveS meansRing message bytes CNetwork phases PReduction steps R
AllReduceFull input per rankfloor(S/N)2(N-1)N-1
AllGatherLocal input per rankSN-10
ReduceScatterLocal output per rank; input is N*SSN-1N-1

The prediction, in nanoseconds, follows Ring.cc, PacketBundle.cc, MemBus.cc and the analytical network backend:

time_ns = P * floor(L + C * 1e9 / (B * 2^30))
+ 3 * R * floor(C / M)
+ (P + 1) * E

B is network bandwidth in GiB/s; L is link latency in ns. M is local-memory bandwidth in decimal GB/s, using the integer spec value that config_builder.py supplies to ASTRA. E is the local endpoint event cost defined by the backend's MemBus::Transmition::Fast path. The local reduction and endpoint terms are accounted for separately, not mistaken for network transmission. At two ranks, AllReduce pays two network phases but AllGather and ReduceScatter pay only one.

The fitter minimises relative squared error over the primitive sweep, with nonnegative latency and inverse bandwidth. It uses the continuous network term to solve the two-variable fit; saved residuals use the serialized parameters and the backend's integer-nanosecond truncation. No model benchmark or model-specific correction coefficient participates.

The hardware file retains raw graphed and isolated timings, the timing source, fit version and assumptions, and per-size/per-collective residuals. The legacy key bandwidth_gbps is retained, but bandwidth_unit: GiB/s makes its actual unit explicit. Inherited defaults also record their units. The bundled hardware file preserves the measured output unchanged, including its raw samples and residual arrays.

A retained-data bandwidth refresh can instead hold an independently calibrated latency fixed. The bundled post-configuration-change RTXPRO6000 calibration uses that contract: the existing 6,600 ns common latency is retained, and common and per-operation bandwidths are recomputed from NCCL primitives. Its file records fixed_latency_ns, the source of that constraint, repetitions and residuals. This is not the same optimization as a fresh profiler hardware run, which estimates both common parameters. Do not silently relabel one as the other.

Historical examples can pin their original transport values with explicit link_bw, link_latency and collective_links: {}. Their compute profiles and benchmark truths then remain comparable when shared hardware defaults change.

The same primitive samples also feed collective_fits: one bandwidth fit per operation, with latency fixed to the saved shared fit. This solves the remaining one-variable problem in inverse bandwidth using the same relative objective and local costs. Per-operation residuals remain visible; a fit that cannot identify positive finite bandwidth records unavailable rather than publishing a value. The shared fit is retained as the fallback.

Usable operation fits populate defaults.collective_links.<operation>.link_bw. The bundled RTXPRO6000 Qwen3-32B TP2 and Qwen3-30B DP2/EP2 examples consume these values directly, with recorded NCCL references for that interconnect. They are inherited when the entire link is inherited, not when a user specifies a hypothetical common link. An explicit collective_links map can override bandwidth and latency or disable the operation defaults. See the configuration precedence. Updating code alone does not add these values to existing hardware files.

Calibration is not an exact NCCL model

A shared BW/latency pair need not match all three NCCL curves. Inspect the residuals rather than assuming a successful fit proves accuracy. Different rank counts, topologies, dataset splitting or local-memory overrides change the model contract. Grouped multi-tensor MoE dispatch, uneven rank sizes and their broadcast/reduce implementations are not measured by this primitive sweep; summing bytes does not reproduce their execution costs.

Updating profiler code does not migrate existing hardware.yaml values. Old fit residuals do not certify the corrected backend contract. Refit retained raw samples with their original rank count and hardware spec, or re-measure, then validate before adopting new defaults.

1. Confirm the acquisition backend​

The profiler targets vLLM 0.31.0 by default (scripts/docker-vllm.sh pulls vllm/vllm-openai:v0.31.0). Its launcher, hardware probes, cache initialization and timing attribution target NVIDIA CUDA. A device being supported by vLLM alone does not validate this profiler.

TargetBundled acquisition path
NVIDIA CUDAIntended target; verify the selected model/backend, live coverage and measurement contract on the device
AMD ROCmRequires an acquisition port; changing the Docker image is insufficient
Intel HPU and other acceleratorsRequires an acquisition adapter or an independently produced compatible bundle

Choose the CUDA image for the host driver and installed dependencies, not the GPU family name alone. The launcher defaults to the CUDA 13 image; another image requires its own model/backend compatibility and acquisition checks. This upgrade does not remeasure existing bundles. See vLLM installation for environment setup.

2. Edit profile.sh​

HARDWARE="H100" # or whatever you want as the folder name
TP_DEGREES="1,2,4,8"
MEASUREMENT_ITERATIONS=3
# ... other knobs as needed

HARDWARE is just a label, pick something memorable. The simulator later references this via cluster_config.hardware.

For unusual GPU types, you may need to adjust:

  • MAX_NUM_BATCHED_TOKENS and MAX_NUM_SEQS for memory limits
  • ATTENTION_MAX_KV if KV cache memory is much smaller than HBM GPUs of similar generation
  • DTYPE if the GPU lacks bf16 support (rare on modern GPUs)

3. Run​

./profiler/profile.sh

Wait. Drink coffee. Output lands in profiler/perf/<HARDWARE>/<MODEL>/<variant>/. See Running → Expected runtime for ballpark times.

Once it's done, the simulator is ready to use, no further changes. Update your cluster_config.json to set "hardware": "<HARDWARE>" and run.

Other accelerator backends​

ROCm and HPU are not drop-in replacements for this CUDA acquisition path. Audit and port device discovery, cache layouts, kernel attribution and communication measurement before publishing a bundle from another backend. The free-form HARDWARE label selects an output folder; it does not select or implement an accelerator adapter.

Adding non-GPU hardware​

For a target outside the bundled acquisition path, use an appropriate measurement adapter or analytical model. The simulator consumes the profile bundle contract, independently of the acquisition environment; skew calibration and native MoE data also require their matching metadata.

The strategy: synthesize CSVs in the Output bundle format from your own measurement source.

Three sources for the data​

1. Vendor analytical / cycle-accurate model​

Most vendors maintain an internal performance model for their hardware. If you have access:

  • Use the vendor's model to compute kernel-level latencies for the layer types the simulator's architecture YAML declares (qkv_proj, attention, down_proj, etc.).
  • Sweep the same axes the GPU profiler does (tokens, sequences, and attention's (prefill_chunk, prefill_key, n_decode, kv_decode) for each kernel and decode_q_len). Use the category-specific schema for linear attention, MTP and MoE; native MoE component tables require their matching contracts.
  • Write CSVs in the schema documented on Output bundle.

This produces the most accurate simulator predictions because the relative latencies between layers reflect your hardware's actual behavior.

2. External simulator​

If you have an analytical compute simulator (GEMM-perf, roofline, or a cycle-accurate model from a published paper), feed it the shapes the profiler would have profiled and dump the same CSV format.

The architecture YAMLs at profiler/models/<model_type>.yaml declare which kernels you need to time. For each entry in the catalog: section you need:

  • For dense category: latency as a function of tokens.
  • For per_sequence: latency as a function of sequences.
  • For attention: a table per kernel and decode_q_len, interpolated over (prefill_chunk, prefill_key, n_decode, kv_decode).
  • For linear_attention, MTP and MoE: follow the matching output schema. Include the linear-attention regimes, drafter entries or MoE contracts required by the model and deployment.

3. Hand-authored from datasheets / public benchmarks​

Last resort. If you only have peak FLOPs / memory bandwidth / latency numbers for your hardware:

  1. Compute roofline-style latencies per layer type.
  2. Write the CSVs. Keep it coarse, a few rows per axis is enough for first-pass sanity checks.
  3. Validate against any public benchmark you can find for the same hardware × model combo.

This produces optimistic predictions (no realistic kernel overhead), so use cautiously. The other two paths are strongly preferred.

What to put in meta.yaml​

Even when synthesizing, write a meta.yaml so the simulator's runtime warnings work properly:

profiler_version: "synthetic"
vllm_version: "n/a"
gpu: "<driver device name, or n/a>"
hardware: "<HARDWARE>" # must equal the folder name
variant: "<VARIANT>"
model: "<org>/<name>"
tp_degrees: [1]
profiled_at: "<date>"

engine_effective:
max_num_batched_tokens: <whatever your CSVs cover>
max_num_seqs: <ditto>

skew_fit:
enabled: false # no measured heterogeneous correction

Do not supply guessed inline alpha coefficients or invent runtime-skew-calibration entries or their hashes: generate them from measured skew data and matching attention references with profiler refit-skew. See Skew & alpha fit.

hardware is the folder name a cluster config's hardware field must match; gpu is free-form provenance. They are separate fields — don't put the label in both.

engine_effective takes only the two batching bounds; there is no dtype / kv_cache_dtype key there, since the dtypes are encoded in variant. Only those two are read, and only to emit the sweep-bound warning.

attention_grid and skew_profile are provenance for humans and are not read at run time, so you can omit them from a synthetic bundle or fill them in however you like.

Enable only measured, compiled calibration

An absent or disabled skew_fit applies zero correction. Enabled entries must use a recognized calibration format from the compiler; the old constant-alpha and bucket formats are rejected. profiler refit-skew compiles your own raw measurements against matching attention references. A missing TP or query slice does not borrow another slice's correction.

Omit skew.csv and skew_fit.csv when you have no heterogeneous-decode measurements.

What you can skip​

  • skew.csv and skew_fit.csv if you don't have heterogeneous-decode data — see the warning above about what that costs you.

  • moe.csv if you're not modeling MoE on this hardware (only needed when running MoE models).

  • tp<N>/ folders for TP degrees you don't need to simulate. Note the mechanism: _load_perf_db discovers available tp*/ folders and checks that the degrees your cluster config needs are present. Category tables are loaded lazily on lookup; TP1 can still be needed for stable entries and MoE even when the deployment uses another TP degree. A missing one is a hard error, not a fallback:

    FileNotFoundError: No profile data for tp=[4] under
    perf/<hw>/<model>/<variant>/. Re-run the profiler with TP_DEGREES
    including 4.

Required inputs​

  • The category tables and canonical entries required by the model's resolved catalog and deployment, including linear-attention or MTP tables when used. See the output schema for category coordinates and metadata.
  • meta.yaml, which the simulator requires to load the bundle's metadata. The variant folder name itself is derived from the model configuration.

Validation​

Once you've synthesized a CSV bundle:

  1. Smoke test: run the simulator with a small workload (workloads/example_trace.jsonl) and a single-instance config pointing at your new HARDWARE.
  2. Compare against a known reference: if your hardware has published latency numbers for a public model, run a workload that matches and check TTFT / TPOT match within reason.
  3. Sanity-check the throughput log: the per-iteration prompt_t and decode_t values should make rough sense (not 10× too high or too low).
  4. Watch for the "extrapolation" warning at startup. If your CSVs are too coarse, the simulator warns; densify the relevant axes if accuracy matters.

Where this gets used​

Once your CSV bundle lives at profiler/perf/<HARDWARE>/<MODEL>/<variant>/, the simulator picks it up automatically when the cluster config names matching values:

{
"hardware": "<HARDWARE>",
"model_name": "<MODEL>",
"tp_size": <N>
}

The simulator's resolve_variant() derives the folder from the checkpoint; it has no runtime dtype-selection flags. Profiler --dtype, --kv-cache-dtype and --variant can create alternative measurements that are not automatically selected. See Simulator → Trace generation and profiler naming.

What's next​

  • Output bundle: schema reference for what you need to produce (or have the profiler produce).
  • Adding a model architecture - separate concern, only when the model's model_type isn't already in profiler/models/.