Cluster config explained

Every simulation in LLMServingSim is driven by one JSON file: a cluster config. It captures the entire hardware topology, how many nodes, how many instances per node, what GPU each instance runs on, how memory is laid out, and how the model is parallelized.

Once you understand this file, every example in this section is a small variation on the same shape.

The minimum viable config

This is configs/cluster/single_node_single_instance.json: the smallest config that runs:

configs/cluster/single_node_single_instance.json

{
  "num_nodes": 1,
  "link_bw": 16,
  "link_latency": 20000,
  "nodes": [
    {
      "num_instances": 1,
      "cpu_mem": {
        "mem_size": 512,
        "mem_bw": 256,
        "mem_latency": 0
      },
      "instances": [
        {
          "model_name": "meta-llama/Llama-3.1-8B",
          "hardware": "RTXPRO6000",
          "npu_mem": {
            "mem_size": 96,
            "mem_bw": 1597,
            "mem_latency": 0
          },
          "num_npus": 1,
          "tp_size": 1,
          "pd_type": null
        }
      ]
    }
  ]
}

That's: one node, one instance, running Llama-3.1-8B on one RTXPRO6000 GPU with TP=1 (no parallelism).

The file has three nested levels. We'll walk through them top-down.

1. Top level, the cluster

{
  "num_nodes": 1,
  "link_bw": 16,
  "link_latency": 20000,
  "nodes": [...]
}

Field	Type	Meaning
num_nodes	int	Number of physical nodes in the cluster
link_bw	float or float[]	ASTRA-Sim topology link bandwidth in GB/s
link_latency	float or float[]	ASTRA-Sim topology link latency in ns
nodes	array	One entry per node (length must match num_nodes)

For multi-node setups (e.g., two boxes in a rack), set num_nodes: 2 and add a second node entry. link_bw / link_latency can be either:

• a scalar, broadcast to every ASTRA-Sim topology dimension
• an array, with one value per final network.yml::npus_count dimension

Optional top-level fields:

Field	Used for
cxl_mem	CXL memory expansion config (see CXL memory tier)

2. Per-node level

{
  "num_instances": 1,
  "cpu_mem": {
    "mem_size": 512,
    "mem_bw": 256,
    "mem_latency": 0
  },
  "instances": [...]
}

Field	Type	Meaning
num_instances	int	How many serving instances live on this node
cpu_mem.mem_size	float	Host CPU memory capacity (GB)
cpu_mem.mem_bw	float	CPU memory bandwidth (GB/s)
cpu_mem.mem_latency	float	CPU memory latency (ns)
instances	array	One entry per instance (length = num_instances)

Optional per-node fields:

Field	Used for
cpu_mem.pim_config	Name of a PIM device config in configs/pim/ (see PIM attention offload)
power	Power model coefficients (see Power modeling)

3. Per-instance level

This is where the real work happens. An instance is one independent LLM serving replica, a model, a parallelism strategy, and a chunk of GPUs.

{
  "model_name": "meta-llama/Llama-3.1-8B",
  "hardware": "RTXPRO6000",
  "npu_mem": {"mem_size": 96, "mem_bw": 1597, "mem_latency": 0},
  "num_npus": 1,
  "tp_size": 1,
  "pd_type": null
}

Required fields

Field	Type	Meaning
model_name	string	Hugging Face model id. Must match a config in configs/model/{model_name}.json
hardware	string	Hardware tag. Must match profiler/perf/{hardware}/ (e.g. RTXPRO6000, H100)
npu_mem	object	Per-GPU memory: mem_size (GB), mem_bw (GB/s), mem_latency (ns)
pd_type	string|null	"prefill", "decode", or null for combined prefill+decode

Parallelism fields (at least one required)

Field	Type	Default	Meaning
num_npus	int	inferred	Total GPUs for this instance, equals tp_size * pp_size
tp_size	int	inferred	Tensor-parallel degree
pp_size	int	1	Pipeline-parallel degree
ep_size	int	tp_size (MoE) / 1 (dense)	Expert-parallel degree
dp_group	string|null	null	Instances with the same string form one data-parallel group, wave-synchronized per iteration; for MoE they also share experts across the group

You only need to provide one of num_npus or tp_size. The other gets inferred. So:

• tp_size: 4 → num_npus = 4 * pp_size (PP defaults to 1, so 4)
• num_npus: 4, pp_size: 2 → tp_size = 2

Parallelism rules to remember:

• num_npus == tp_size * pp_size (always)
• pp_size <= num_hidden_layers: stages are cut on transformer-block boundaries, so a stage cannot be empty
• TP and EP share the same GPUs: dense layers do TP-ALLREDUCE, MoE layers do EP-ALLTOALL
• Without dp_group: ep_size <= tp_size
• With dp_group: EP can scale beyond a single instance's GPUs (see DP+EP example)
• For MoE models: ep_size must divide num_local_experts

Optional advanced fields

Field	Used for
placement	Per-layer / per-block weight + KV-cache placement (see CXL memory)
any of the 14 runtime overrides	Per-instance scheduler / memory / dtype settings (see below)

Per-instance runtime overrides

Everything so far describes hardware. An instance can also carry runtime settings, and that is what makes heterogeneous clusters possible: 14 of the python -m serving flags can be re-specified per instance, so a prefill instance and a decode instance in the same run can be scheduled completely differently.

The precedence is one level deep:

instances[i].<field>   >   --<field> on the CLI   >   built-in default

A field written in the instance wins for that instance. Every other instance keeps whatever the CLI said. There is no merging and no inheritance between instances.

The 14 overridable fields

Field	Overrides	Typical reason to set it per instance
max_num_seqs	--max-num-seqs	Narrow batches on a prefill instance, wide ones on decode
max_num_batched_tokens	--max-num-batched-tokens	Big prefill chunks, small decode steps
long_prefill_token_threshold	--long-prefill-token-threshold	Stop one long prompt monopolising a prefill instance
block_size	--block-size	Coarser blocks where fragmentation matters less
dtype	--dtype	Serve the same model at two precisions
kv_cache_dtype	--kv-cache-dtype	FP8 KV on the decode instance only
enable_chunked_prefill	--enable-chunked-prefill	Chunk on prefill, never on decode
enable_prefix_caching	--enable-prefix-caching	Cache where prefixes repeat, skip where they don't
npu_mem.mem_util	--npu-memory-utilization	Leave headroom on one instance only
reserve_full_isl	--reserve-full-isl	Loosen admission on a decode-only instance
enable_local_offloading	--enable-local-offloading	Weight offload on one instance
enable_attn_offloading	--enable-attn-offloading	PIM attention on one instance
enable_sub_batch_interleaving	--enable-sub-batch-interleaving	Overlap NPU and PIM on the offloading instance
enable_block_copy	--enable-block-copy	Faithful per-layer expert variance on one instance

Note where mem_util sits: it is the only override nested inside another object, because it exists to scale npu_mem.mem_size and follows that block's mem_* naming.

"npu_mem": {"mem_size": 96, "mem_bw": 1597, "mem_latency": 0, "mem_util": 0.8}

A heterogeneous example

configs/cluster/single_node_heterogeneous.json runs one Qwen3-32B prefill instance and one decode instance, both TP=2, with opposite scheduler settings:

configs/cluster/single_node_heterogeneous.json (instances only)

"instances": [
  {
    "model_name": "Qwen/Qwen3-32B",
    "hardware": "RTXPRO6000",
    "npu_mem": {"mem_size": 96, "mem_bw": 1597, "mem_latency": 0},
    "num_npus": 2,
    "tp_size": 2,
    "pd_type": "prefill",
    "max_num_seqs": 32,
    "max_num_batched_tokens": 8192,
    "enable_chunked_prefill": true,
    "enable_prefix_caching": true
  },
  {
    "model_name": "Qwen/Qwen3-32B",
    "hardware": "RTXPRO6000",
    "npu_mem": {"mem_size": 96, "mem_bw": 1597, "mem_latency": 0},
    "num_npus": 2,
    "tp_size": 2,
    "pd_type": "decode",
    "max_num_seqs": 256,
    "max_num_batched_tokens": 0,
    "enable_chunked_prefill": false,
    "enable_prefix_caching": false
  }
]

Reading it: the prefill instance takes few sequences with a large token budget, because a prefill step wants long chunks. The decode instance takes many sequences and no token cap, because each decode step contributes one token per sequence.

configs/cluster/single_node_pd_per_instance_config.json is the Llama-3.1-8B version of the same idea, and additionally overrides long_prefill_token_threshold, block_size, dtype, and kv_cache_dtype.

max_num_batched_tokens: 0 is not infinity

0 maps to "unlimited", but the scheduler then applies min(max_num_batched_tokens, max_position_embeddings). On Qwen3-32B that is 40960, so the decode instance above runs at 40960, not unbounded. See Model config.

What you cannot set per instance

The other 15 flags are cluster-wide. Writing one inside an instance object does nothing at all: no key reads it, and no error is raised.

Scope	Flags
Cluster / backend	--cluster-config, --network-backend
Router, cross-instance by definition	--request-routing-policy, --expert-routing-policy
Shared lower KV tier	--enable-prefix-sharing, --prefix-storage
Workload, one per run	--dataset, --num-reqs, --skip-prefill
Run plumbing	--output, --run-id, --inputs-root, --save-trace-text, --keep-inputs, --log-interval, --log-level

Two rejected combinations

Both are checked per instance, against the effective values, at config-load time:

• enable_sub_batch_interleaving without enable_attn_offloading - there is nothing to overlap the NPU sub-batch against.
• enable_sub_batch_interleaving with pp_size > 1: an interleaved trace leaves both sub-batches mid-block at every stage edge, so a pipeline stage has no single hidden state to hand on.

Full semantics, including dtype's three-level fallback and the mem_util range check, are in Reference → Cluster config.

DP+EP, the topology that needs more explanation

When multiple instances share the same dp_group, they form a multi-dimensional ASTRA-Sim topology, innermost dimension first: [tp_size, dp_group_size], or [tp_size, pp_size, dp_group_size] when pp_size > 1. Collectives are scoped per dimension:

• TP ALLREDUCE runs on the TP dim only (within an instance)
• EP runs on the DP dim, plus the TP dim when EP spans past one instance's GPUs. The PP dim is never involved — vLLM's EP group pins the pipeline stage.

All instances in a DP group share one ASTRA-Sim process with wave-synchronized scheduling. MoE expert weights are sharded by ep_size: each instance holds num_local_experts / ep_size experts.

Concrete example: Qwen3-30B-A3B has 128 experts. With tp_size=1, ep_size=2, dp_group="A" and two instances, each holds 64 experts. Per-token activation crosses the DP group via ALLTOALL.

This is the DP+EP MoE example.

What config_builder.py does with this file

When you launch the simulator, serving/core/config_builder.py reads the cluster config and generates three ASTRA-Sim input files under astra-sim/inputs/:

Generated file	Driven by
network/network.yml	link_bw, link_latency, [tp_size, (pp_size,) dp_group_size] topology
system/system.json	Memory bandwidths, scheduling policy, per-dim collective implementations
memory/memory_expansion.json	CXL devices and any extended memory tiers

You don't write these by hand, they're regenerated on every run from the cluster config.

Provided configs

The repo ships 27 worked configs under configs/cluster/. Those without a link are runnable as-is but have no dedicated example page:

Config	Shape	Used by
single_node_single_instance.json	1 node, 1 instance, Llama-3.1-8B, TP=1	the default; Tensor parallel raises it to tp_size=2
single_node_single_instance_H100.json	Llama-3.1-70B on H100, TP=4	—
single_node_multi_instance.json	1 node, 2 instances	Multi-instance LOAD routing
single_node_4_instance_2TP.json	1 node, 4 instances at TP=2	—
single_node_heterogeneous.json	P/D pair with opposite runtime settings	Per-instance runtime overrides
single_node_pd_instance.json	prefill/decode disaggregation	Prefill/decode split
single_node_pd_per_instance_config.json	P/D with per-instance runtime limits	Per-instance runtime overrides
single_node_pp_instance.json	4 GPUs as pp=4	Pipeline parallel
single_node_tp_pp_instance.json	4 GPUs as tp=2 x pp=2	Pipeline parallel
single_node_moe_single_instance.json	Qwen3-MoE, TP=2 EP=2	Expert parallel
single_node_moe_dp_ep_instance.json	2 MoE instances in one DP group, EP=2	DP+EP MoE
single_node_dp_instance.json	DP=2 x TP=2, dense model, 4 GPUs	—
rtx4090_single_instance.json	RTX 4090, Llama-3.1-8B TP=1, mem_util calibrated	Validation
rtx4090_tp2_instance.json	2x RTX 4090 as tp=2, Llama-3.1-8B. Needs an RTX4090 tp=2 profile first — only tp1 is bundled, so it raises FileNotFoundError as shipped	Profiler
rtx4090_multi_instance.json	2 independent TP=1 RTX 4090 instances	—
single_node_moe_dp_tp_instance.json	DP=2 x TP=2 MoE, EP=2, 4 GPUs	—
single_node_moe_dp_pp_instance.json	DP=2 x PP=2 MoE, EP=2, 4 GPUs	—
single_node_moe_dp_tp_pp_instance.json	DP=2 x TP=2 x PP=2 MoE, EP=4, 8 GPUs	—
single_node_moe_multi_instance.json	2 MoE instances, no DP group	—
single_node_moe_pd_instance.json	MoE with P/D disaggregation	—
single_node_moe_pp_instance.json	MoE on 4 GPUs, tp=2 x pp=2, ep=2	Pipeline parallel
single_node_cxl_instance.json	CXL memory expansion	CXL memory tier
single_node_memory_instance.json	weight / KV placement control	CXL memory tier
single_node_pim_instance.json	PIM-enabled memory + power model	PIM attention offload
single_node_power_instance.json	power modeling enabled	Power modeling
dual_node_multi_instance.json	2 nodes, 2 instances each	multi-node setups
dual_node_moe_dp_ep_intra_inter_instance.json	2-node DP+EP MoE, per-dimension link_bw / link_latency	DP+EP MoE

What's next

Now that you can read a cluster config, pick an example and see how the same shape produces very different topologies:

• Tensor parallel: simplest non-trivial: TP=2 on one instance.
• Multi-instance LOAD routing - shows what num_instances > 1 does.
• DP+EP MoE: the most interesting topology this simulator can model.