Skip to main content

DP+EP MoE

What this demonstrates: stretching expert parallelism across multiple instances using dp_group. Two instances form a single 2D ASTRA-Sim topology with TP on one axis and EP on the other.

This is the most interesting topology LLMServingSim can model. Two serving instances each run their own TP group, but they share experts via a cross-instance all-to-all. The 2D ASTRA-Sim network mesh routes TP-ALLREDUCE on dim 0 and the EP collectives on dim 1.

Prerequisites​

  • Simulator container set up
  • Bundled RTXPRO6000 profile for Qwen/Qwen3-30B-A3B-Instruct-2507
  • An agentic dataset (or any workload with enough requests to keep both instances busy)

Cluster config​

configs/cluster/single_node_moe_dp_ep_instance.json:

configs/cluster/single_node_moe_dp_ep_instance.json
{
"num_nodes": 1,
"link_bw": 16,
"link_latency": 20000,
"nodes": [
{
"num_instances": 2,
"cpu_mem": {"mem_size": 512, "mem_bw": 256, "mem_latency": 0},
"instances": [
{
"model_name": "Qwen/Qwen3-30B-A3B-Instruct-2507",
"hardware": "RTXPRO6000",
"npu_mem": {"mem_size": 96, "mem_bw": 1597, "mem_latency": 0},
"num_npus": 1, "tp_size": 1, "ep_size": 2, "dp_group": "A",
"pd_type": null
},
{
"model_name": "Qwen/Qwen3-30B-A3B-Instruct-2507",
"hardware": "RTXPRO6000",
"npu_mem": {"mem_size": 96, "mem_bw": 1597, "mem_latency": 0},
"num_npus": 1, "tp_size": 1, "ep_size": 2, "dp_group": "A",
"pd_type": null
}
]
}
]
}

The two pieces that turn this from "two independent instances" into a DP+EP cluster:

  • dp_group: "A" on both instances, instances with the same string form one DP group.
  • ep_size: 2 while tp_size: 1: EP spans the DP group (ep_size > tp_size only allowed when a dp_group is set).

config_builder.py sees the DP group and emits a topology sized [tp_size, dp_group_size] = [1, 2] — innermost dimension first, with a pp_size dimension inserted in the middle when pp_size > 1. Collectives are scoped per dimension via the involved_dim BoolList:

  • TP-ALLREDUCE: [True, False]: dim 0 only (within instance, no-op here since tp_size=1)
  • EP dispatch / combine: [False, True]: dim 1 only (across the two instances)

Run​

python -m serving \
--cluster-config 'configs/cluster/single_node_moe_dp_ep_instance.json' \
--block-size 16 \
--dataset 'workloads/swe-bench-qwen3-30b-a3b-50-sps0.2.jsonl' \
--output 'outputs/dp_ep_moe_run.csv' \
--num-reqs 1

Notes on the flags:

  • --num-reqs 1 means one session with serial sub-requests. Only one real request from that session is eligible at a time; the other DP member participates with dummy batches.
  • Increase the session count to overlap independent sessions and exercise real work on both instances.

Reading the heartbeat​

The following excerpt is from the separate committed ShareGPT validation run, which uses its own workload and cluster settings. It is not expected output from the one-session command above.

[4.0s] Avg prompt throughput: 7965.0 tokens/s, Avg generation throughput: 1095.0 tokens/s
├─Running Instance[0]: 23 reqs, Waiting: 0 reqs, Total # 1 NPUs, Each NPU Memory Usage 32628.90 MB (33.541 % Used), Prefix Cache Hit ratio 0.00 %, (0 / 20285)
├─Running Instance[1]: 22 reqs, Waiting: 1 reqs, Total # 1 NPUs, Each NPU Memory Usage 32976.90 MB (33.899 % Used), Prefix Cache Hit ratio 0.00 %, (0 / 24147)
└─Node[0]: Total CPU Memory Usage 0.00 MB, 0.000 % Used (Instance[0]: 0.00 %, Instance[1]: 0.00 %)
[5.0s] Avg prompt throughput: 6440.0 tokens/s, Avg generation throughput: 1499.0 tokens/s
├─Running Instance[0]: 26 reqs, Waiting: 1 reqs, Total # 1 NPUs, Each NPU Memory Usage 32948.40 MB (33.870 % Used), Prefix Cache Hit ratio 0.00 %, (0 / 22913)
├─Running Instance[1]: 26 reqs, Waiting: 0 reqs, Total # 1 NPUs, Each NPU Memory Usage 33305.40 MB (34.237 % Used), Prefix Cache Hit ratio 0.00 %, (0 / 26861)
└─Node[0]: Total CPU Memory Usage 0.00 MB, 0.000 % Used (Instance[0]: 0.00 %, Instance[1]: 0.00 %)

Those two lines are copied from bench/examples/RTXPRO6000/Qwen3-30B-A3B-Instruct-2507/outputs/sim.log, the committed DP+EP validation run. The heartbeat reports one Running Instance[i] line per member but does not identify EP collectives or DP waves. Similar running-request counts in this excerpt do not establish wave synchronization; members can have unequal real workloads.

To see the synchronized message size you have to read the trace, not the log: run with --save-trace-text and look at the EXPERT 0 / EXPERT END markers, whose comm_size is computed from max(total_len_per_instance) across the group.

What's interesting​

  • Expert weights split across instances, not just GPUs. With ep_size=2 and 128 total experts, each instance holds 64. Per-GPU weight memory is roughly halved compared to single-instance EP=1.
  • One ASTRA-Sim process for both instances. Wave-synchronized scheduling across the DP group means the simulator generates .et files for both instances that share stream IDs on the EP collectives, forcing ASTRA-Sim to block until both NPUs reach them.
  • Idle instance gets a dummy batch. When one instance has no pending work, the scheduler synthesizes a 1-decode-token batch so it can still participate in the wave's collective. Same when one instance finishes early, it keeps generating dummies until the group does.
  • comm_size is synchronized. Even if instance A has a much bigger batch than instance B, both pass the same size to ASTRA-Sim, set to the max, so the network model sees the heavier side.