DP+EP MoE
What this demonstrates: stretching expert parallelism across multiple instances using
dp_group. Two instances form a single 2D ASTRA-Sim topology with TP on one axis and EP on the other.
This is the most interesting topology LLMServingSim can model. Two serving instances each run their own TP group, but they share experts via a cross-instance all-to-all. The 2D ASTRA-Sim network mesh routes TP-ALLREDUCE on dim 0 and the EP collectives on dim 1.
Prerequisites
- Simulator container set up
- Bundled RTXPRO6000 profile for
Qwen/Qwen3-30B-A3B-Instruct-2507 - An agentic dataset (or any workload with enough requests to keep both instances busy)
Cluster config
configs/cluster/single_node_moe_dp_ep_instance.json:
{
"num_nodes": 1,
"link_bw": 16,
"link_latency": 20000,
"nodes": [
{
"num_instances": 2,
"cpu_mem": {"mem_size": 512, "mem_bw": 256, "mem_latency": 0},
"instances": [
{
"model_name": "Qwen/Qwen3-30B-A3B-Instruct-2507",
"hardware": "RTXPRO6000",
"npu_mem": {"mem_size": 96, "mem_bw": 1597, "mem_latency": 0},
"num_npus": 1, "tp_size": 1, "ep_size": 2, "dp_group": "A",
"pd_type": null
},
{
"model_name": "Qwen/Qwen3-30B-A3B-Instruct-2507",
"hardware": "RTXPRO6000",
"npu_mem": {"mem_size": 96, "mem_bw": 1597, "mem_latency": 0},
"num_npus": 1, "tp_size": 1, "ep_size": 2, "dp_group": "A",
"pd_type": null
}
]
}
]
}
The two pieces that turn this from "two independent instances" into a DP+EP cluster:
dp_group: "A"on both instances, instances with the same string form one DP group.ep_size: 2whiletp_size: 1: EP spans the DP group (ep_size > tp_sizeonly allowed when adp_groupis set).
config_builder.py sees the DP group and emits a topology sized
[tp_size, dp_group_size] = [1, 2] — innermost dimension first, with a
pp_size dimension inserted in the middle when pp_size > 1.
Collectives are scoped per dimension via the involved_dim BoolList:
- TP-ALLREDUCE:
[True, False]: dim 0 only (within instance, no-op here sincetp_size=1) - EP dispatch / combine:
[False, True]: dim 1 only (across the two instances)
Run
python -m serving \
--cluster-config 'configs/cluster/single_node_moe_dp_ep_instance.json' \
--block-size 16 \
--dataset 'workloads/swe-bench-qwen3-30b-a3b-50-sps0.2.jsonl' \
--output 'outputs/dp_ep_moe_run.csv' \
--num-reqs 1
Notes on the flags:
--num-reqs 1means one session with serial sub-requests. Only one real request from that session is eligible at a time; the other DP member participates with dummy batches.- Increase the session count to overlap independent sessions and exercise real work on both instances.
Reading the heartbeat
The following excerpt is from the separate committed ShareGPT validation run, which uses its own workload and cluster settings. It is not expected output from the one-session command above.
[4.0s] Avg prompt throughput: 7965.0 tokens/s, Avg generation throughput: 1095.0 tokens/s
├─Running Instance[0]: 23 reqs, Waiting: 0 reqs, Total # 1 NPUs, Each NPU Memory Usage 32628.90 MB (33.541 % Used), Prefix Cache Hit ratio 0.00 %, (0 / 20285)
├─Running Instance[1]: 22 reqs, Waiting: 1 reqs, Total # 1 NPUs, Each NPU Memory Usage 32976.90 MB (33.899 % Used), Prefix Cache Hit ratio 0.00 %, (0 / 24147)
└─Node[0]: Total CPU Memory Usage 0.00 MB, 0.000 % Used (Instance[0]: 0.00 %, Instance[1]: 0.00 %)
[5.0s] Avg prompt throughput: 6440.0 tokens/s, Avg generation throughput: 1499.0 tokens/s
├─Running Instance[0]: 26 reqs, Waiting: 1 reqs, Total # 1 NPUs, Each NPU Memory Usage 32948.40 MB (33.870 % Used), Prefix Cache Hit ratio 0.00 %, (0 / 22913)
├─Running Instance[1]: 26 reqs, Waiting: 0 reqs, Total # 1 NPUs, Each NPU Memory Usage 33305.40 MB (34.237 % Used), Prefix Cache Hit ratio 0.00 %, (0 / 26861)
└─Node[0]: Total CPU Memory Usage 0.00 MB, 0.000 % Used (Instance[0]: 0.00 %, Instance[1]: 0.00 %)
Those two lines are copied from
bench/examples/RTXPRO6000/Qwen3-30B-A3B-Instruct-2507/outputs/sim.log,
the committed DP+EP validation run. The heartbeat reports one
Running Instance[i] line per member but does not identify EP collectives
or DP waves. Similar running-request counts in this excerpt do not establish
wave synchronization; members can have unequal real workloads.
To see the synchronized message size you have to read the trace, not
the log: run with --save-trace-text and look at the EXPERT 0 /
EXPERT END markers, whose comm_size is computed from
max(total_len_per_instance) across the group.
What's interesting
- Expert weights split across instances, not just GPUs. With
ep_size=2and 128 total experts, each instance holds 64. Per-GPU weight memory is roughly halved compared to single-instance EP=1. - One ASTRA-Sim process for both instances. Wave-synchronized
scheduling across the DP group means the simulator generates
.etfiles for both instances that share stream IDs on the EP collectives, forcing ASTRA-Sim to block until both NPUs reach them. - Idle instance gets a dummy batch. When one instance has no pending work, the scheduler synthesizes a 1-decode-token batch so it can still participate in the wave's collective. Same when one instance finishes early, it keeps generating dummies until the group does.
comm_sizeis synchronized. Even if instance A has a much bigger batch than instance B, both pass the same size to ASTRA-Sim, set to the max, so the network model sees the heavier side.
Related examples
- Expert parallel: the same MoE model on a single instance (EP within one TP group).
- Multi-instance LOAD routing - the non-DP version of multi-instance: independent replicas without expert sharing.
- Cluster config explained -
the field-level walkthrough of how
dp_grouplights up the 2D topology.