Skip to main content

Output bundle

Each profile run produces a directory tree under profiler/perf/<HARDWARE>/<MODEL>/<variant>/. This is the contract between the profiler and the simulator. Tables must match their schemas, model geometry and execution contract. Skew calibration retains reference invalidation checks, and native MoE requires matching semantic deployment contracts; filenames alone do not establish compatibility.

Folder layout​

profiler/perf/<HARDWARE>/
├── hardware.yaml # the card's spec + the measured interconnect;
│ # one per hardware folder, shared by every model
└── <MODEL>/<variant>/
├── meta.yaml # all profile metadata, including per-TP records
└── tp<N>/ # one folder per profiled TP degree
├── dense.csv
├── per_sequence.csv
├── attention.csv
├── linear_attention.csv # mamba / gated-DeltaNet models only
├── moe.csv # historical whole-block fallback only
├── mtp.csv # one drafter pass, only with --profile-mtp
├── skew.csv # skew-enabled runs only
├── skew_fit.csv # skew-enabled runs only
└── moe_components/ # deployment-specific CSVs and optional raw checkpoints

<variant> uses the established checkpoint label plus explicit activation/KV suffixes (bf16, fp8, fp8-actfp16, fp8-kvfp8, …): see Running → Output naming. Multiple variants for the same hardware × model live as siblings. The label is not a complete resolved-precision record, and alternate variants are not automatically selected by the simulator; check the naming limitations in the running guide.

tp<N>/ exists for each TP in TP_DEGREES. Layers tagged tp_stable: true in the architecture YAML (layernorms, sampler) are profiled once at TP=1. The writer replicates stable dense and per_sequence rows into other TP folders. Stable MTP rows stay in tp1/mtp.csv; the simulator selects TP1 for their lookup.

hardware.yaml sits one level up because it answers a different question. The CSVs are per (model, hardware); the interconnect and the card's memory are per hardware, so one file serves every model bundle underneath. Written by python -m profiler hardware, and read by cluster configs that omit link_bw / link_latency / npu_mem.* — see Cluster config schema.

Times are microseconds​

All time_us columns are in microseconds. Ordinary category tables are converted to rounded nanoseconds at load time. Native MoE component tables retain microseconds for interpolation, then multiply by 1000 and round the lookup result to nanoseconds. If you're hand-authoring CSVs (see Adding non-GPU hardware), remember to use μs.

CUDA activity and acquisition identity​

New ordinary layerwise measurements record a descriptive device-active-union method. The common PyTorch/Kineto collector records explicit dynamic module scopes and native CUDA activities without Python stacks, tensor shapes or memory snapshots. For each raw module invocation, merge overlapping CUDA intervals on the same device, then apply the existing per-call normalization. Distinct calls remain separate before averaging, even when their device execution overlaps. Bindings to raw CUDA kernel names retain the existing sum of matching leaf durations; they are distinct from a module's per-call interval union. CPU launch scopes establish ownership only; CPU duration, GPU annotations and gaps between device activities do not contribute latency. The result is device-active time, not whole-step elapsed time. Separate catalog components can still overlap, so adding their costs is not an exact global interval union.

Common direct model methods, declared native operators and explicitly declared callables can supply additional runtime ownership contexts. An operator: binding selects an exact native CPU operator event, but its cost is the union of its launch-correlated CUDA activity intervals, not the CPU event's duration. A callable: binding uses the same CUDA accounting at an exact function/method boundary. Common method contexts remain context-only; independently measured methods require an explicit callable binding. All these ownership contexts are transparent to descendants' parent-call normalization, so adding ancestry does not create extra execution counts. Missing or ambiguous ownership is rejected; a static module alias is not a substitute for observed runtime ancestry.

Catalogs may explicitly choose normalization: parent or invocations; omission keeps ordinary parent/occurrence normalization and whole-MoE invocation means. Unavailable callable targets are reported, never stored as zero measurements. The full architecture includes inactive alternatives, so availability and positive selected-layout coverage are separate checks. See the callable contract.

New category CSVs contain only their coordinate columns and time_us, as shown below. Acquisition metadata lives in the variant-root meta.yaml, under per_tp[tp].measurements.<category>. Each entry retains the acquisition identity, its semantic measurement token and the observed acquisitions. It records the PyTorch/Kineto collector, per-invocation device-active interval union and normalization, dummy weights, assigned-page KV preparation, zero-start recurrent state, deterministic queries and repetitions, together with actual library versions. Callable and contextual tables keep their structural manifests and completion records in the same root metadata; those are not extra timing columns or sidecars.

Host/worker agreement and resume compare catalog, contexts, method, geometry and execution semantics. Exact Torch/vLLM version strings are observational provenance, not execution vetoes. Historical source hashes remain readable observations; new admission does not collect source-file digests. Native API, selected backend, shape, dtype, layout, finite-output and coverage checks still apply. Table hashes may seal completed data; they are not implementation-version locks.

Historical data keep their original metadata and timing columns. Existing files without compatible acquisition metadata remain valid historical simulation inputs where supported, but cannot be extended as a current acquisition. Use a separate output root or explicitly refresh the category with --force; never restamp old rows. No-op sweeps preserve prior provenance. Compatible additions retain earlier acquisition observations; TP-stable replication rejects mixed timing methods.

Each new ordinary CSV publication also records per_tp[tp].measurements.<category>.publication with a state and csv_sha256. The writer first serializes and flushes the replacement CSV, records the intended checksum as pending, atomically replaces the CSV, then marks that publication complete. Later checkpoints and TP-stable replication use the same ordering. This prevents an interrupted forced refresh from presenting old CSV bytes as a new acquisition.

Resume checks the saved CSV against its publication checksum. A matching pending record means the saved bytes match the intended generation and can be reused; reading it does not finalize the record. A mismatched pending or complete record is rejected: use fresh output or explicitly remeasure with --force. Metadata without a publication record remains readable under its existing compatibility rules and is not retrospectively stamped by a no-op read. These checks bind saved data to metadata, not source files or runtime versions, and change no measured timings. Publication complete certifies that CSV generation, not completion of the requested acquisition grid or any separate category-coverage contract.

Skew uses the same CUDA interval accounting normalized per measured forward. Root per_tp[tp].skew retains its semantic identity, capture contexts, descriptive method and observed runtime versions; new skew CSVs hold coordinates and timings. Native DP+EP component bundles retain their separate recorded measurement contract. The skew fitter's own input/code invalidation rules are unchanged. Catalog coverage accounts for each native kernel, copy and memset exactly once, including work outside module scopes. Such work is reported as a gap, not assigned to a convenient layer or called CPU overhead. Overlapping canonical claims remain errors visible to the coverage CLI; ordinary measurements reject overlaps within their category slice before saving. Cross-category overlaps require full-catalog coverage. Unscoped work requires an ownership-boundary review, not an arbitrary raw-kernel binding. Coverage is a device-work accounting check, not a latency union or proof that every declared catalog alternative executed.

Coverage's canonical_invocations audit records physical scope calls and positive-CUDA calls for each explicitly measured forward, with raw-kernel activity counts kept separate. Each normalization group retains its actual denominator; do not infer physical calls from that divisor or sum divisors across groups. Warmup is excluded and missing canonical entries remain explicit. This metadata changes neither timing values nor the normalization formula.

Every timed forward must contain the same positive active canonical set. Inactive catalog alternatives need not execute, but a component disappearing in only one forward invalidates the shot. Warmup validates geometry and finite output outside timing. A failed check cannot produce a successful acquisition.

Recurrent-only prefixes also verify query/history pairs and their assigned state pages, including metadata reused between cache groups. With cache mode none, the guard accepts ordinary non-speculative GDN metadata and a separately native vLLM 0.31 KDA metadata layout. KDA consumes the complete non-speculative query/state view, including mixed batches' decode prefix, rather than filling GDN's prefill-only views. Both paths retain query/history, state-page, initial-state and token-conservation checks; neither changes the native forward. Unsupported APIs/schemas and active speculative, checkpoint, RecoverSSM or optional FlashInfer metadata paths fail explicitly. This check is outside timing and alone does not establish model support; Kimi acquisition has the additional native contract below. Its preparation semantics belong in acquisition metadata: old rows remain readable, but an incompatible acquisition cannot resume them by relabeling metadata. MTP acquisition requires every entry in its category slice: partially captured drafter work cannot be saved as a complete table.

Both acquisition identities include assigned-page dummy KV initialization through vLLM's dummy-weight helper. Initialization is outside timing and is repeated per context. See preparation and limitations.

Synthetic requests allocate disjoint physical page spans across all KV groups from one shared pool, reserving null block 0. Capacity checks include every group's allocation, not just the largest group. Initialization touches only assigned spans and respects the backend's block axis and kernel subdivisions. Typed logical cache views are checked against the engine's shape contract; the packed formats below have explicit byte-layout checks. Unsupported packed layouts and state modes fail before acquisition.

Allocation and initialization are described in acquisition metadata. Changes to their semantics require fresh measurements even if the CSV schema is unchanged. Existing bundles are not rewritten or certified by this fix; use a separate output root or explicitly remeasure with --force.

Both protocols preserve the prompt/output boundary of history-bearing decode requests, not just their query lengths and KV positions. This allows vLLM to select the intended prefill/decode kernels for each mixed batch.

Dummy model weight preparation​

Model weights and dummy KV history are separate inputs. The profiler still uses vLLM's native floating-weight initialization. On the admitted native MXFP4/SITU path, it additionally encodes expert checkpoint bytes with the native MXFP4 quantizer before unchanged native postload processing. A local CUDA generator supplies BF16 values in [-0.001, 0.001]; bounded row chunks use at most 8 MiB of source staging, not a full floating copy of every expert. CUDA and CPU random streams are not claimed to be identical.

Native KDA's nonpersistent decode buffers are normally filled by checkpoint callbacks that dummy loading skips. The adapter copies the convolution layout and FP32 norm from the initialized native parameters, preserving their relation instead of randomizing the derived buffers independently. These checks and copies occur outside timed forwards.

The packed path requires the native auto-selected DEEPGEMM_MXFP4 backend, BF16 activations and unsharded TP/PP/DP/EP1 without MTP or offload. Unsupported APIs, backends, layouts or synthetic sharding fail before publication. Ordinary models call the original loader unchanged; this contract does not add a universal quantized-weight initializer or complete model support.

An authoritative main engine retains dummy_weight_preparation under engine_resolved.per_tp. Its completed initialization and postload-layout checks do not certify finite forwards, catalog coverage or benchmark accuracy. Those remain independent acquisition gates. Ordinary and skew method metadata describe this preparation; incompatible old rows require remeasurement.

Packed attention history​

The NVIDIA vLLM 0.31 acquisition path supports these exact opaque cache formats:

FormatPayload and scale layout
fp8_ds_mlaEach 656-byte token record contains 512 FP8 NoPE values, four FP32 scales (one per 128 values), then 64 BF16 RoPE values
DEEPSEEK_V32_INDEXEREach token contributes 128 FP8 values and one FP32 scale, stored in separate planes within its block; the explicit backend shuffle flag selects plain or 16-by-16 tiled values

The cache spec, role, dtype, dimensions, block geometry and byte strides must match the supported format. A uint8 allocation alone is not enough to infer it. The indexer's 132 bytes per token are a storage budget, not interleaved 128-byte-value/4-byte-scale records. Null pages, unassigned pages and per-block padding remain untouched, including when layer views share a backing allocation.

Logical FP8 payloads are initialized as FP16 through the installed initialize_single_dummy_weight, using its unmodified distribution and per-tensor seed; RoPE payloads use BF16. Native NVIDIA E4M3 quantization then writes the data and deterministic positive power-of-two scales. For a group maximum amax, the main cache uses 2^ceil(log2(max(amax / 448, 1e-4))), while the index cache uses 2^ceil(log2(max(amax, 1e-4) / 448)). The distinct clamp positions are intentional. Scale bytes are neither random dummy values nor arbitrary unit scales.

Floating sources are created in bounded chunks independent of total cache size; FP32 conversion and quantization intermediates add a bounded multiple of that source storage. The source-chunk limit is not a peak-memory limit. Initialization is outside warmup and measured query work. No native forward or trained-weight KV generation is used; the result is synthetic history, not trained KV statistics or a guarantee of representative data-dependent sparse selection.

Floating MLA indexer caches use the ordinary typed-cache path. A three-dimensional [blocks, tokens, channels] view may omit the head axis when the spec declares one KV head and the same floating dtype; shape validation restores that axis before checking the backend's logical geometry. Existing four-dimensional typed views remain supported. Squeezed views must also pass a non-overlap stride check: in ascending stride order, each non-singleton axis starts beyond the extent of the preceding axes. Padded and permuted views that satisfy this rule are allowed. This does not relax opaque uint8 layout checks. Older runtimes without the sparse-role capability retain the typed path; this does not enable their unsupported opaque formats. NVFP4, other packed layouts and ROCm/FNUZ packed acquisition remain unsupported. Preparation support alone does not certify model catalog coverage, trace composition or benchmark accuracy. Changed initialization semantics require remeasurement; a source-file hash alone does not determine compatibility.

Runtime-selected component layouts​

A catalog may declare alternative block decompositions. Qwen supports historical GDN conv/scan/norm components or a packed native core including the gated norm; projections and outer glue stay separate. Both use the same GPU-only timing rule.

Positive prefill/decode/mixed CUDA coverage must select a unique variant on every worker; every acquired shot rechecks it across category boundaries. Model/version names, CPU events or absent kernels are not evidence. Small probes and acquired coordinates do not certify unmeasured shapes or backend transitions.

Discovery currently reuses the fixed _COVERAGE_REGIMES in profiler/core/runner.py. The engine's token, request, context and cache limits must accommodate those probes; they are not reduced with the requested grid. An otherwise valid small engine envelope can therefore fail discovery before category acquisition. The fixed probes require at least 5 requests, context length 258 and an acquisition query-token buffer of 68 (logical token budget plus max_num_seqs). The host validates every probe before the first worker call; it refuses an undersized envelope without resizing or skipping probes. Synthetic batch assembly also rejects request, token and context overflow before GPU work.

meta.yaml.component_layouts is the single authority: schema component-layouts, with a per_tp record containing variant selections, layout_sha256, a shared acquisition identity and each affected category's state/measurement identity. The layout hash binds declarations, choices and affected catalog bindings, not CSV timing values. Each CSV retains its usual row-level measurement identity.

Category stateMeaningUsable by simulation
pendingPrevious acquisition invalidated; no category rows certifiedNo
writingCan resume before the first measured shot or from compatible whole-shot checkpointsNo
completeCategory finished; saved rows passed layout/identity checksOnly when every coupled category is complete and valid

Replacement makes pending durable before clearing measured rows. Metadata writers preserve the record rather than inferring completion from file presence. Unrelated TP-stable replicas retain their original TP1 identities; they do not establish measured coverage at another degree or become new samples. Replicas are validated before exemption from local coverage. When a replica-only table starts local acquisition, its category metadata adopts the local method while retaining each replicated layer's source-TP acquisition record. Local resume checks copied rows against that record, not the local measurement identity. An empty or replica-only writing category can resume, but cannot become complete. Readers reject stale layout contracts, inconsistent row identities, malformed/duplicate rows, mixed component sets and completed attention/linear-attention tables missing an entire required regime. This is not a certificate for every grid coordinate.

Unmarked historical bundles keep their old block order and stored timings; alternative-only rows without metadata are refused. Validated choices and expanded sequences are cached per TP without mutating the shared catalog. See the trace guide for sizing and communication, which are separate from measured latency.

For the Qwen hybrid catalog, dense and linear_attention are coupled: removing the separate norm cost requires acquiring its replacement inside the fused core. Use a fresh output bundle for a different representation or changed layout definition. A coordinated forced refresh of historical tables is also possible, but the bundle is unusable until every affected category completes. See resume and force for interruption and refresh rules.

Use one writer per variant bundle, even for different TP/category slices, and do not mutate a bundle while a simulation reads it. Atomic file replacement and directory synchronization provide interruption safety, not locking or a live multi-file transaction. These checks do not establish end-to-end model accuracy.

Native DSA acquisition​

deepseek_v32.yaml retains the historical legacy-modules representation and declares native-eager-tail for the native DeepSeek-V3.2 path. Selection requires actual positive CUDA coverage, site invocation counts and runtime admission; a catalog alias does not extend this contract to GLM-5.

The native sequence replaces fused units together. Its shared embedding endpoint already includes the first input norm; only absolute target layer 0 consumes that matching norm prefix. Later norms use invocation normalization. The combined dsa_attention_tail includes sparse indexing and attention, so its history axis remains uncapped even beyond index_topk. Historical attention/indexer rows are not summed with the new tail or relabeled as it.

The native-dsa-eager-tail contract targets the reviewed native vLLM 0.31 path: original-width FP8 or unquantized BF16 weights with BF16 activations, physical/logical TP1 and EP1, PP1/DP1, eager execution and decode query length 1. It rejects MTP, offload, sequence parallelism, shared/skipped indexers, dense-MHA prefill, quantized query input and unsupported cache geometry. Source-config fields are compared with the resolved native config, including supported alias/RoPE normalization. Reduced prefix depth is checked separately, not confused with model width or identity.

The existing per-TP record additionally stores runtime.runtime_sha256, runtime.invariant_facts and runtime.category_engines. The shared invariant binds source geometry, backend/cache modes and the actual native runner/sampler route. The profiler's transparent sampler facade does not change that route. Its reviewed source determines returned ID width (V1 int32, V2 int64); this is source-backed metadata, not a tensor-content observation. Category evidence retains actual reduced depths and cache capacities separately. Runtime versions are recorded provenance; historical source hashes are not exact-match execution admission. Actual native APIs, dtypes, layouts and backend modes must still satisfy the contract.

New publication requires dense, per_sequence and attention to be complete with matching runtime identities, per-category engine evidence and valid whole shots. Native MoE components have a separate deployment contract and publication record; a fresh bundle requires that complete acquisition too, without waiting for a removed moe.csv acquisition. Completing attention alone cannot certify the head or MoE. Historical records that include a whole-block moe category retain their original completion checks. Compatible interrupted shots resume; incompatible old rows require fresh acquisition, never a metadata rewrite. Skew uses the same selected attention components and runtime identity.

These tables currently support only explicit duration trace replay. The native 656-byte main cache plus 132-byte index cache is not silently passed to the legacy 708-byte capacity model. This does not certify full-grid coverage, serving memory feasibility, distributed execution or benchmark accuracy.

Native Kimi acquisition​

kimi_k3.yaml selects native_profile: hybrid-residual. The shared NativeProfile host orchestration uses an adapter in hooks/runtime_admission.py for the runtime-admitted vLLM 0.31 Kimi-K3 text path: BF16 activations, native MXFP4/SITU experts, eager execution and local TP/PP/DP/EP1. Backend/ABI checks and observed layouts remain mandatory; exact library versions and historical source hashes are provenance rather than execution locks. Removing a fixed checkpoint or device-capability lock does not certify arbitrary devices or backends; a catalog alias alone cannot admit another native implementation.

The independently supplied nested configuration defines widths, expert count, top-k, the decoder schedule and residual block geometry. Shared stack helpers derive the smallest prefix covering each category's axes. Native MoE acquisition uses the MLP axis. For the shipped configuration these are four decoder layers, or two for the MoE slice, without reducing widths or experts. No fixed checkpoint digest or decoder depth stands in for validating the supplied configuration and observed runtime.

Ordinary category slice commands use the existing grids. Full profiles require complete ordinary/contextual data and independently published native local MoE components, including measured wrapper work around the processed MXFP4 experts. The live adapter must admit the actual backend and capabilities; catalog presence does not establish support on another device. Explicit local DP1/EP1 selectors are allowed. Skew and MTP remain unsupported; full profiles require --skip-skew. Historical whole-block measurements remain readable under their original contract. See the bounded invocation. Each TP directory contains the ordinary CSVs; root per_tp[tp].categories.<category> stores its contract, publication status and completed csv_sha256. Shared CategoryRecord publication binds planned row keys, the whole declared architecture, native runtime, capture identity, completion status and CSV hash. Loading independently rebuilds the catalog bindings and requires every binding at each recorded coordinate; completion does not claim unmeasured coordinates or a full production grid. Malformed, ragged or duplicate partial rows are refused before preload or deduplication. An incomplete shot is remeasured as a whole. A completed table with changed data or incompatible semantic bindings is refused. Runtime versions are recorded per acquisition, not compared as exact execution-admission constraints. Changed execution semantics require fresh measurements in a separate output root or an explicit remeasurement. Historical metadata has narrow read fallbacks; relocation alone neither remeasures timings nor relabels an incomplete acquisition as successful. Dummy-weight preparation elapsed times remain telemetry in engine metadata, not execution invariants or GPU latency inputs. The intended scope is bounded native acquisition and saved-duration trace replay. Fresh GPU validation is required before claiming coverage for a new bundle; TP2, full profiling, native capacity admission and end-to-end benchmark accuracy are not established by this adapter.

Two boundaries need particular care:

  • MLA: the q-LoRA phase joins native attention and gate work that may execute concurrently on separate streams. Its measured device interval union is one attention component; the gate application and output projection are separate dense components. Adding separate costs for the concurrent branches would double-count their overlap.
  • AttnRes: original layer position changes the number of residual blocks and optional inputs. A reduced engine cannot measure every original-stack context. The dense pass therefore calls the unchanged native AttnRes function separately, with ordered roles and structural contexts derived from the supplied depth, residual block width and source schedule at each dense token point. This does not execute the full original model on the GPU.

Root per_tp[tp].contexts holds ordinary and callables context records, each with its contract and completion receipts. per_tp[tp].callables.callables holds acquisition options, the explicit residual plan and token-keyed shots. Options retain the selected profile, tokens, iterations, warmups and runtime versions. The plan retains workloads, catalog, contexts and semantic stamps; compact shot summaries bind scope, measurement stamp, CSV-shot checksum, call counts, native kernel routes and device observations.

tp<N>/callables/dense_context.csv still stores context-specific microseconds. Full native CUDA ledgers are validated during acquisition, not saved as new per-token evidence JSON files. Historical sidecars remain read-only fallbacks. The generic collector is the same lightweight PyTorch/Kineto path; no CPU scope duration enters the table. Explicit finite synthetic tensors supply the standalone function's inputs; native API, shape, strides, optional inputs and physical invocation counts are checked. Allocation is bounded to 512 MiB of declared input tensors per shot, with at most 16 timed iterations and four warmups. Larger plans fail before engine startup.

ContextualProfileBundle in serving/core/component_sizes.py requires every category and residual context to be complete and compatible with the independently loaded source configuration. Callable receipts bind compatible runtime semantics and the validated per-shot summaries to the saved timing rows. Ordinary tables retain their existing interpolation. AttnRes uses an immutable context lookup and linear interpolation in tokens within that exact context; it neither searches neighboring contexts nor extrapolates beyond the measured token range. This supports local duration replay, not resident-memory admission, whole-workload accuracy or CUDA-graph equivalence. Coverage still reports unscoped runner GPU work, including input/state updates and host/device copies, separately from these layer components. It can therefore exit nonzero even when all declared model boundaries executed. Do not suppress those gaps or interpret model-boundary coverage as complete step-time accounting.

Native attention lookup contract​

New MiniMax-M3 attention acquisitions record meta.yaml.attention_lookup with schema native-attention-lookup and an independent per_tp record for each measured TP. This is acquisition-owned backend evidence, not a user-supplied cap or a reinterpretation of old data.

The supported contract checks the native vLLM 0.31.0 NVIDIA Triton pair MiniMaxM3IndexerTritonImpl / MiniMaxM3SparseTritonImpl, using BF16 indexer and attention KV storage, max-score selection, and a contiguous int32 selection buffer whose width equals the configured sparse_topk_blocks. The reviewed layout requires 128 tokens per block. Initial/local blocks occupy the selected-block total; they do not enlarge it. Acquisition inspects the actual selected objects, buffer layout, callable API and model/catalog parameters, and requires semantic worker agreement. Observed versions and any historical source hashes are provenance, not execution locks; new acquisition checks the actual API and layout. Another implementation, layout, storage precision or speculative path needs a separately reviewed contract; a matching model name or version label alone is insufficient.

The resulting sparse_topk_blocks × sparse_block_size cap (2048 for the shipped configuration) applies only to sparse_attention prefill lookup: cap each request's effective key before taking the query-weighted mean. Dense attention and the indexer remain uncapped, as do decode-history coordinates. Producer rows keep their raw coordinates and measured timings, including above the cap; those rows still describe real acquisition work, not the chosen lookup anchor. A cap does not assert identical latency at every larger history. It is also separate from logical trace-byte accounting and physical KV capacity.

Each per-TP record contains semantic, its semantic_sha256, the ordinary measurement identity, completed, and attention_sha256. Acquisition metadata binds the semantic digest while CSVs retain coordinates and timings. Publication marks the record pending before accepting rows and seals the completed attention CSV. Consumers reject pending records, unknown schemas, missing metadata for marked rows, altered checksums, mixed identities, or model/catalog/TP mismatches. Completion describes the selected acquisition plan, not proof of a full grid or arbitrary context range.

The reader accepts current measurement metadata and explicitly supported historical row protocols. This does not make acquisition identities interchangeable: semantic contracts and completion seals must still match. Source-file changes or an exact version difference alone are not a veto. Reading a historical protocol never rewrites or upgrades its stored rows.

Unmarked historical tables retain their existing lookup semantics: 2176 for the shipped MiniMax-M3 configuration. Do not add a contract to historical rows, rewrite their keys or automatically change their timings/fits. Use a fresh root, or explicitly remeasure incompatible attention data. A compatible resume reuses rows without timing them again; interrupted native publication must be resumed before consumption. Unrelated category refreshes preserve the contract. Keep one writer per variant root and published bundles immutable.

Serving, skew fitting, support completion and plan-skew share the per-TP resolver. Native skew rows bind the same semantics and require a matching completed root per_tp[tp].skew record and table checksum before fitting. A zero-case skew plan creates no new skew file or completion claim, does not itself trigger a fit, and leaves existing artifacts intact, even with --force; it contributes no new skew provenance. Other nonempty TP acquisitions are still recorded.

The bounded native validation covers TP1 BF16 prefix acquisition on SM120 and consumer behavior around the selection boundary. It does not establish full-grid coverage, multi-TP deployment, model-capacity admission or end-to-end benchmark accuracy. Inspect the actual bundle's plan and measured support before use.

dense.csv​

layer,tokens,time_us
act_fn,1,4.21367
act_fn,2,5.36533
...
qkv_proj,1,20.4373
qkv_proj,2,20.4813
...
ColumnMeaning
layerCanonical layer name (must match the architecture YAML's catalog)
tokenstotal_len for this shot
time_usMeasured kernel latency, microseconds

The simulator does 1D linear interpolation over tokens when looking up.

Layers it covers: embedding, layernorm, qkv_proj, qk_norm, rotary_emb, o_proj, gate_up_proj, act_fn, down_proj, final_layernorm. (Anything in the YAML's catalog with category dense.)

DeepSeek/GLM indexer_glue measures only residual work outside the indexer's LayerNorm and SparseAttnIndexer subtrees. Their copy/fill kernels already belong to indexer_k_norm and indexer; wildcard kernel bindings must not charge them a second time, including when profiling the dense category alone. Refresh older glue rows with profiler slice --group dense --force for the affected bundle. Updating the catalog does not rewrite stored latency tables.

per_sequence.csv​

layer,sequences,time_us
lm_head,1,1075.13
lm_head,2,1044.52
...
sampler,1,25.9333
...
ColumnMeaning
layerlm_head or sampler
sequencesnum_requests for this shot (decode rounds operate per-sequence)
time_usMeasured kernel latency

Simulator: 1D linear interpolation over sequences.

attention.csv​

The attention table covers pure-prefill, pure-decode, and mixed kernel shapes:

layer,prefill_chunk,prefill_key,n_decode,kv_decode,decode_q_len,time_us
attention,0,0,1,16,1,8.08533
attention,0,0,1,32,1,8.17033
...
attention,512,2304,4,128,1,...
...
ColumnMeaning
prefill_chunkTokens of the prefill chunk in this iteration. 0 = pure decode
prefill_keyQuery-weighted effective causal key length across prefills
n_decodeNumber of concurrent decode requests in this iteration. 0 = pure prefill
kv_decodeKV cache history length the decode requests attend to
decode_q_lenQuery tokens per decode sequence; normally 1, larger for speculative verification
time_usMeasured attention kernel latency

For chunks c_i with histories k_i, the shared profiling/runtime coordinate is sum(c_i * (k_i + c_i / 2)) / sum(c_i) (zero without prefills). Equal chunks and single-request shots keep their previous values. Legacy kv_prefill tables held one prefill per shot and are relabelled on load as kv_prefill + prefill_chunk / 2; this does not create missing grid coverage.

The simulator selects the table for the requested decode_q_len, then uses 4D linear interpolation over the remaining shape axes. An unprofiled query length selects the nearest available query-length table with a warning; it is not interpolated. Profile the query lengths needed by the intended deployment. Within that table, coordinates are bracketed and blended on a linear scale, even when measurement points are geometrically spaced. Queries below an axis minimum are clamped, and those above its maximum extrapolate from the top two samples. Missing prefill-token/decode-count corners use the nearest available corner before the blend.

This lookup does not round batches up to a profiled size. CUDA graph padding is a separate execution-shape contract, and the simulator's attention lookup retains the real query/KV lists. It does not separately represent FULL-graph padded attention metadata/empty-slot cost; see the per-layer graph contract. When enabled, skew calibration selects a compiled alpha bucket to correct the mean-KV estimate toward the maximum-KV estimate; alpha itself is not interpolated between neighboring buckets.

The base grid is geometric, controlled by --attention-chunk-factor, --attention-kv-factor and --attention-n-factor. Read the CLI's --help for current defaults; the editable profile.sh supplies explicit overrides. Applicable pure-regime slices also include target serving graph anchors: FULL-capable decode points at each selected query length, and prefill points only for FULL. Mixed shots keep their geometric axes. These remain eager measurements; see sampling policy. The KV factor controls two axes. Smaller factors increase the Cartesian product of measurements, not just one additive set of points. Finer sampling does not guarantee better end-to-end accuracy; compare completed bundles using the same benchmark reference. See cost planning.

linear_attention.csv (mamba / gated-DeltaNet models only)​

ColumnMeaning
layerCanonical layer name
prefill_tokensTokens in the batch's prefill chunk
n_decodeNumber of decoding sequences
time_usLatency for one trace node, microseconds

Two axes rather than attention's four, and no kv axis at all: a gated-DeltaNet layer keeps a fixed-size conv state and a fixed-size recurrent state per sequence, neither a function of position. Cost therefore does not depend on sequence length — measured on Qwen3.8-27B, a 64x spread in kv length moves it 1.1% — and no skew correction applies, unlike attention.csv.

Two axes rather than one because which kernel runs depends on the mix. A pure-decode batch runs a recurrent kernel; add a prefill chunk and vLLM switches to a fused-gating one, and the conv switches too. That is why this file, like attention.csv, includes a layer column: one block can run several non-interchangeable kernels on the same axes, so each gets its own rows, and a cell is empty when the batch shape never reaches that kernel.

This describes the historical separate-kernel decomposition. A catalog with a recorded component layout may instead emit one fused-core row on these axes, with corresponding changes to its coupled dense.csv. Do not mix representations or infer a new choice from row names.

The prefill_tokens axis is sampled chunk-aware — inside the first chunk, at every chunk boundary, and at the token just past each boundary. Cost is a staircase, not a line: one token past a 64-boundary costs 13.5% more than the boundary itself, and the interval to the next boundary is nearly flat. A plain geometric grid lands only on boundaries, so interpolating between two samples underestimates most of the token counts a chunked-prefill scheduler actually produces.

Native DP+EP components​

Native MoE acquisition defaults to DP1; --dp selects other target degrees. Optional --ep filters the supported TP*DP targets without changing their geometry. It publishes root per_tp[tp].moe_components metadata and deployment-specific components.csv tables. Contracts and coverage live in that root record, not standalone index/contract/coverage JSON files. Conditioning coverage is a compact count and maximum-change summary; individual control coordinates and timings stay in raw resume checkpoints. Publication still requires the configured quality checks and preserves the completed table checksum. These are separate from the legacy whole-block moe.csv: local gate/routing, gathered experts and local finalization have different token domains and eager measurements. New acquisition does not capture isolated CUDA graphs. Historical tables retain their recorded modes. Global expert count and top-k are preserved. Modular DP1 records only gate_routing and experts, with verified identity preparation and aliased no-op finalization in contract.local_execution; no synthetic copy or zero-latency row is stored. Target-specific backend selection is retained. The separate live TP1/DP1/EP1 contract records gate_routing, experts and wrapper. It preserves native processed weights, backend and expert prepare/finalize work; the wrapper retains shared experts, latent transforms and combination work after excluding gate/router and routed-kernel activities. Exact native-method references and unchanged state are required outside timing. The metric is a sum of unique CUDA activity durations, not overlap-aware elapsed latency. Recorded gate dtype/width can differ from expert width. Physical expert weight bytes and cloned-bank allocation bytes are distinct, with wrapper weights counted outside gate/expert storage. The isolated engine's private KV-memory cap preserves requested limits and requires native capacity validation; its actual settings stay in the MoE contract's observations, never replacing main-engine cache facts. New expert tables sample physical tokens N, active local experts A and local assignments M using --moe-grid-factor (default 2) and mandatory feasible geometry corners. The recorded expert_sampling contract identifies these axes and their spacing, plus local/expert token anchors transformed from target serving graph coordinates. capture_tokens holds those anchors; geometric_token_ranges holds uncovered local/expert intervals (empty means capture-covered, absent preserves an existing plan). logical_token_budget retains the requested limit; local_token_budget in the contract includes any MRV1 compiler-SP rounding needed for the physical buffer. Endpoints and feasible corners remain part of the plan. sampling_graph records the capture source; capture sizes do not determine active experts or assignments. The simulator precomputes Delaunay support at load time and caches barycentric interpolation in physical coordinates, using at most four measured rows per query. It fits no latency predictor. Historical balanced-assignment tables retain their existing lookup. The live contract admits quantized/shared execution by actual capability; unsupported layouts or deployments are rejected rather than mapped to an old table.

Matching tables are selected automatically, with deployment, dtype, coverage and checksum validation. Missing coverage can use legacy fallback only when a historical whole-block table exists; fresh bundles require complete matching components. See Native MoE components for the full schema, raw repetitions, resume and unsupported-path behavior.

moe.csv (legacy whole-block MoE profiles)​

ep,tokens,activated_experts,time_us
1,1,8,47.8
2,1,4,36.1
4,1,2,30.2
ColumnMeaning
epEP degree used by the legacy expert/top-k-shrunk acquisition
tokensInput rows presented to the profiled rank; the AgRs consumer uses gathered rows
activated_expertsDistinct experts touched on that rank
time_usMeasured whole-block latency in microseconds

These are historical measurements. Whole-block acquisition, expert/top-k slicing, forced routing and --moe-ep-degrees have been removed. A new slice --group moe measures native components at DP1 by default and does not rewrite moe.csv; native contracts require a fresh output root to remeasure.

The legacy consumer retains two-dimensional interpolation over tokens and active experts within an EP grid. An unprofiled EP degree falls back to the nearest with a warning. A bundle without the EP column is read as EP1.

The old acquisition changed the checkpoint expert/top-k shape to approximate a local rank. Its EP column does not establish native global-top-k routing, DP/backend coverage or separate local and gathered measurements. Existing files remain readable until supported components are remeasured, with warnings for unverified DP coverage. Stored timings are neither corrected nor relabelled by a code or metadata update. Unsupported native acquisition fails explicitly; there is no whole-block acquisition fallback.

skew.csv (skew-enabled runs)​

One measured target per attention kernel and ordered request shape.

ColumnMeaning
layerExact attention-category kernel
requests_jsonOrdered query/history pairs
n_prefill, decode_q_lenExplicit query roles
case_idGeometry-derived key
round_timings_us_jsonIndividual forward times, grouped by context
t_skew_usMedian of the context-level forward medians

Acquisition families, method identity, resolved block size, repetition counts and runtime observations live in root per_tp[tp].skew. Historical CSVs may retain these columns; new acquisitions do not repeat them on every timing row.

Historical bimodal rows can be reconstructed; general distributions need their full request lists. Measured controls, when present, remain diagnostics. The compiler obtains mean/max references through the unchanged serving attention lookup, never by relabelling measured control values.

Kernel ownership is verified through native CUDA launch correlation IDs and the innermost launching CPU module scope, after mapping profiler thread identities to native OS threads. This can repair misplaced CUDA leaves in the upstream profile tree without shifting timestamps, changing GPU durations, or adding CPU overhead. Missing or ambiguous launch ownership rejects the shot. Method metadata describes this attribution and aggregation; older rows remain historical measurements, not completed current acquisitions.

Root per_tp[tp].skew records new acquisitions with the actual per-TP plan, resolved page size/capacity, seed, family counts and completion status. Interrupted or failed runs retain an incomplete status and checkpointed rows. If the final rows were saved but the completion update was interrupted, resume validates the completed plan and seals its record without measuring again or adding acquisition provenance.

skew_fit.csv (skew-enabled runs)​

New fits use runtime-skew-calibration and write:

layer,decode_q_len,pc_label,lev_label,n_anchor,alpha,direct_rows,pooled_rows
ColumnMeaning
layer, decode_q_lenSeparate kernel/query slice; no cross-slice borrowing
pc_label, lev_labelPrefill-token and relative-reference-gap partition
n_anchorMeasured decode count with enough direct support in this partition
alphaClipped relative-latency weighted median
direct_rowsDistinct supported cases at the anchor itself
pooled_rowsDirect cases plus nearby unsupported-N cases pooled into this anchor

The simulator picks the nearest supported N on a log scale, with lower-anchor ties. It does not interpolate neighboring alpha values. The support-adaptive N anchors differ between partitions; prefill cutoffs scale with the measured prefill envelope and leverage cutoffs are dimensionless. See Skew & alpha fit for the objective and fallback rules.

Each meta.yaml::skew_fit.per_tp[tp] entry records the schema, bundle/TP identity, axes, min_rows, alpha_clip, n_samples, dropped-row counts, alpha_default_by_kernel, n_range_by_kernel, reference, bucket_table and bucket_table_sha256. Kernel/query keys have the form attention|q=1. reference records attention and raw-skew checksums, the lookup fingerprint and key-saturation semantics. The fitted CSV contains the cells, not raw shots.

The lookup fingerprint covers batch-context construction as well as attention lookup. A head-row change can therefore invalidate a fit without changing its attention reference values. Run the CPU-only profiler refit-skew command to refresh the bundle, rather than replacing fingerprints manually. A successful refit can leave all numerical CSV values unchanged.

Legacy files have layer, n_label, pc_label, lev_label, alpha, n_samples and no schema field. They are no longer accepted when skew is enabled: run profiler refit-skew first. Disabled bundles do not read these tables.

meta.yaml​

Sibling of the tp<N>/ folders and the only metadata document written for a new profile bundle. per_tp[tp] owns measurements, category publication, context/callable records, skew acquisition and native MoE entries. Already-central engine, attention-lookup, component-layout and calibration sections retain their existing locations and separate authority. Updates preserve unrelated records through the shared strict metadata reader/writer. Moving metadata between locations does not alter numerical behavior; adopting a different serving graph policy can alter simulated execution shapes. Metadata writes are serialized under an empty .meta.lock; this lock file is synchronization, not measurement metadata. It does not make concurrent acquisition into one table safe: honor each acquisition path's writer exclusion and do not change a bundle while it is being read.

Historical sidecars have narrow read fallbacks, not new write destinations; skew.meta.yaml is no longer loaded. Optional raw repetition checkpoints support acquisition resume; failure-only diagnostic records can support debugging. Neither is required to load published timing tables, and neither replaces the root metadata contract. Moving existing metadata preserves its observations and timing-table bytes; it does not certify a fresh acquisition. This schematic omits generated hashes; use the profiler to create enabled entries, not hand-written placeholders.

hardware: RTXPRO6000
model: Qwen/Qwen3-32B
variant: bf16
per_tp:
'2':
measurements: {} # category identities, observations and CSV publication seals
categories: {} # native category contracts, status and completed CSV hashes
contexts: {} # ordinary/callables contracts and completion receipts
callables: {} # callable options, plans and compact per-shot summaries
skew: {} # actual acquisition plan and completion, when acquired
moe_components: {} # deployment entries with contracts, coverage and table paths
engine_resolved:
per_tp:
'2':
block_size: 16
max_model_len: 40960
num_cache_tokens: 43523488
skew_fit:
enabled: true
per_tp:
2:
schema: runtime-skew-calibration
identity: {hardware: RTXPRO6000, model: Qwen/Qwen3-32B, variant: bf16, tp: 2}
estimator: supported_n_relative_latency_l1
min_rows: 20
bucket_table: tp2/skew_fit.csv
# axes, fallbacks, measured ranges and fingerprints are compiler output

serving_execution​

Serving graph observations are separate from eager acquisition settings. The benchmark records completed worker startup under meta.json.execution; import it explicitly into an existing bundle:

python -m profiler import-execution Qwen/Qwen3-32B --hardware RTXPRO6000 \
--bench-dir bench/results/my-run

The root serving_execution list keeps one current observation per deployment scope, not an execution history. Each entry records the effective runner/mode, complete base and actual FULL/PIECEWISE capture sizes, compiler sequence parallelism, graph implementation, model configuration, device and precision. Scope includes TP/PP/DP/EP, scheduler limits, speculative token count, block size, maximum model length, chunked prefill and prefix caching. Runtime version and timestamp are provenance, not locks. The importer checks model/device/precision compatibility and does not accept reduced-model or perturbed diagnostic runs as full-model references. When a local model declaration is available, workers resolve it through their installed native configuration class before comparing it with the effective model. This handles renamed fields and nested RoPE parameters without an offline model-specific alias list. Only the compact declaration is retained after a match. A custom variant label requires explicit matching acquisition dtype and KV dtype; its name alone is not a precision contract.

An explicit instance cudagraph object wins. Otherwise the simulator uses a scope-matching observation, or derives its historical defaults; an incompatible saved observation warns before fallback. Worker observation currently supports the fixed-query MRV2 contract without LoRA, DBO or context parallelism. Unsupported or incomplete startup, conflicting workers and partial rank coverage remain unknown and cannot be imported. Frontend requested settings alone are not proof of worker capture. Observing all DP engines requires collecting their individual responses rather than retaining only the load-balancing client's first result.

Import changes no latency CSV or acquisition method. Graph-shaped axes do not certify eager timings as graph timings: attention capture metadata and kernel selection can differ even with the same real requests. This metadata supports dispatch modeling; matched timing validation remains separate.

Identity and provenance​

Stored symbolic method, policy and format labels use descriptive hyphen-separated names, without old-name aliases. Remeasure obsolete method contracts in a separate output root rather than relabeling them. Legacy file-layout fallbacks still require current method contracts; unmarked historical mainstream tables remain readable. Numeric structural schemas and actual library/model versions are unchanged.

KeyMeaning
profiler_version / vllm_version / cuda_versionVersions the bundle was produced with. Kernel timings shift a few percent across CUDA driver versions, so this is the field to check before trusting a mixed comparison
gpuThe driver's device name, verbatim
hardwareThe --hardware label, i.e. the folder name and the value a cluster config's hardware field must match. Distinct from gpu
architecture / architecture_sha256Which profiler/models/*.yaml was used, and its hash — so you can tell whether a catalog edit invalidates the bundle
model / variant / tp_degreesWhat was profiled
measurement_iterationsTimed forwards averaged per shot

engine_effective​

The engine kwargs vLLM actually ran with, not what was requested. Notable entries:

  • max_num_batched_tokens / max_num_seqs — the logical values. The engine is booted with max_num_batched_tokens + max_num_seqs for shot-bypass headroom, and the bump is subtracted back before recording, so what you see here is the sweep bound.
  • hf_overrides — the overrides still forwarded after local config materialization, not a complete inventory of effective model fields. Nested backbone-only fields, including reduced depth and per-rank shapes, are written into text_config and consumed before calling vLLM. Root-only and shared root/text keys remain forwarded. Flat model overrides are unchanged. Category engines retain every block type they measure; hybrid stacks are not universally reduced to one layer.
  • load_format: dummy — checkpoint weights are not loaded. Synthetic weights support shape-controlled acquisition, not equivalence to trained routing or KV contents; see the measurement contracts above.
  • model — the tmpdir the model config was written to, so vLLM needed no Hub access. The path is dead after the run.

An empty hf_overrides does not mean the original full-depth model ran. Consumed values are applied in the temporary local config; their resolved text fields are retained separately under engine_resolved, below. A configuration fix does not certify new backend coverage or migrate existing tables; incompatible acquisition resumes remain rejected.

Explicit dtype and kv_cache_dtype engine arguments are retained here when passed. Their absence means vLLM's defaults were used, not that a particular dtype was resolved. A quantized model can record dtype: bfloat16 while keeping FP8 weights. Check these arguments, the source config and any recorded runtime facts together; variant alone does not encode every effective dtype.

engine_resolved​

per_tp[tp] retains the engine's block_size, max_model_len and num_cache_tokens. New authoritative main-engine records also include model_text_config: a deep-copied snapshot of the native resolved text config, including reduced depth, per-rank fields and consumed nested overrides. The snapshot survives deletion of the temporary config directory. It is not the multimodal wrapper config, backend selection or a parameter-layout inventory.

A refresh updates only the recorded TP entries. MTP and non-authoritative short-stack engines do not replace main-engine snapshots. Older limits and metadata without a snapshot keep that absence; missing values are not reconstructed from today's config. Read the source and engine arguments with these facts rather than treating the snapshot as full execution provenance.

Grid specs are compact, not enumerated​

attention_grid.decode_q_lens lists the requested decode query lengths. Row counts alone do not establish that coverage. Pure-prefill rows are emitted once at q=1 regardless of the requested decode query lengths, so their presence does not establish ordinary q=1 decode coverage.

category_provenance records the vLLM version and timestamp per category. A bundle is not necessarily one measurement session — a slice refresh rewrites one category and leaves the rest — and the top-level vllm_version / profiled_at describe only the most recent refresh. Which rows came from which version is not a detail: vLLM 0.28 restructured MoE, and the two versions agree to within noise on decode-sized batches while differing by 16–26% at 2048 tokens.

A refresh only rewrites what it measured. engine_effective and engine_resolved describe the deepest main engine. An ordinary category slice updates them for its selected TP when its required depth equals the all-axis depth. A shallower slice or an MTP slice preserves them; the drafter engine has different cache requirements. attention_grid may only be rewritten by a run that swept attention.

attention_grid uses a shorthand rather than listing every point:

SpecReads as
0, 16-2048 x2the value 0, then 16 doubling to 2048
2-256 x22 doubling to 256, no zero point
[0.0625, 0.125, …]an explicit list, used where the axis is not geometric

The root per_tp[tp].skew record retains explicit dynamic acquisition axes, resolved limits and completion. Older root summaries may remain historical records, but acquisition, refitting and lookup do not load skew.meta.yaml. Current repeat-level skew data requires complete root acquisition metadata; unmarked historical skew tables retain their existing behavior.

Sampling plans​

Root per_tp[tp].sampling[category] records ordinary acquisition plans: eager execution, geometric factor, feasible shot count and applicable target graph coordinates. Targets record runner, mode, query length, TP/DP, capture sizes and their source: observed worker capture, explicit target options or derived capture candidates. Derived plans retain assumptions and acquisition backend capability scope; an unmatched observation records its fallback reason. TP-stable forward anchors are recorded when TP1 supplies multiple TP degrees. Excluded dense capture coordinates are reported when runtime bounds prevent their measurement.

Attention records its independent factors and pure-regime anchor restriction. Per-sequence metadata states that logits and sampler use actual output rows outside the graph. MTP records request anchors and the missing first-pass query/token dimension. Native MoE records sampling_graph and transformed expert_sampling.capture_tokens in each deployment contract. Skew retains its independent acquisition plan.

A sampling plan describes requested feasible coordinates, not completed rows or graph capture by the acquisition engine. Completion and integrity remain the responsibility of each category's publication records. Sampling changes do not relabel historical measurements as graph timings.

What the simulator actually reads​

KeyUsed for
engine_effective.max_num_batched_tokens / .max_num_seqsOne-shot warning when the runtime CLI exceeds the sweep bounds, since lookups will extrapolate
skew_fit.enabledWhether to apply any skew correction at all
attention_lookup.per_tp[tp]Validate native attention semantics, row identity and completion checksum; select the matching TP's sparse-prefill cap
skew_fit.per_tp[tp].schemaRequired runtime-skew-calibration format
skew_fit.per_tp[tp].identity, .reference, .bucket_table_sha256Validate bundle identity, attention/reference implementation, saturation contract and table bytes
skew_fit.per_tp[tp].axes, .bucket_tableLoad partitions and supported N anchors once
skew_fit.per_tp[tp].alpha_default_by_kernel, .n_range_by_kernelSame-kernel/query fallback and measured N bounds
Legacy inline alpha / bucket-axis fieldsNot supported for enabled calibration

Other version and acquisition fields describe provenance. Raw skew data is required for rebuilding, not for runtime lookup. A changed attention table, lookup implementation, fitted table or saturation contract invalidates new calibration and requires profiler refit-skew.

How the simulator consumes this​

For the simulator-side mechanics, see Simulator → Trace generation.

Gotchas​

  1. Don't edit CSVs by hand to "tune" simulation results. The simulator interpolates linearly across rows; bogus values produce non-monotonic behavior that's hard to debug.
  2. time_us is microseconds. A common mistake when synthesizing CSVs from external tools is to put nanoseconds. Triple-check.
  3. Layer names in dense.csv must match the architecture YAML. If you add a layer to the YAML and don't profile it, the simulator one-shot-warns (and uses 0 latency for that layer, silently corrupting results). Re-run profile after YAML edits.
  4. tp<N>/ folders aren't symlinks. TP-stable layers are physically copied by the writer. Editing tp1/dense.csv doesn't propagate to tp2/.

What's next​