Skip to main content

python -m serving CLI flags

Complete reference for every command-line flag accepted by python -m serving. For the conceptual side of each flag (what it does internally), see Simulator.

Runtime overrides can be set per instance

Flags marked (per-instance) below can also be written into an individual instances[i] object in the cluster config, which wins over the CLI value for that instance only. That is how one run serves heterogeneous instances. Flags without that marker remain run-wide. See Cluster config → Runtime overrides.

Cluster topology​

FlagTypeDefaultDescription
--cluster-configpathconfigs/cluster/single_node_single_instance.jsonPath to a cluster-config JSON. See Cluster config
--network-backendchoiceanalyticalNetwork simulation backend. analytical (fast) or ns3 (detailed, WIP)

Batching and scheduling​

These flags are deployment defaults. A cluster config can override the matching runtime knobs per instances[i]; see Cluster config.

FlagTypeDefaultDescription
--max-num-seqs (per-instance)int128Max sequences in a batch. 0 = unlimited
--max-num-batched-tokens (per-instance)int2048Max tokens per iteration across all requests (token budget). Clamped to the model config's max_position_embeddings, so 0 ("unlimited") resolves to the context length, not infinity
--long-prefill-token-threshold (per-instance)int0Per-request token cap per step for chunked prefill. 0 = disabled
--enable-chunked-prefill (per-instance)boolTrueSplit long prefill across iterations. Use --no-enable-chunked-prefill to disable
--npu-memory-utilization (per-instance, as npu_mem.mem_util)float0.9Fraction of NPU memory usable for weights plus KV cache. Corresponds to vLLM's --gpu-memory-utilization; KV capacity is npu_mem.mem_size * this - model weight. Override per instance with npu_mem.mem_util
--reserve-full-isl / --no-reserve-full-isl (per-instance)flagonAdmit a request only if its whole sequence fits, not merely its first chunk. Mirrors vLLM's scheduler_reserve_full_isl; without it chunked prefill over-admits and thrashes the KV cache
--async-scheduling / --no-async-scheduling (per-instance)flagonModel next-batch admission lookahead while the current batch runs. V1 pipeline parallelism already supplies the modeled in-flight window; V2 can add async lookahead. Idle instances admit current arrivals. The TTFT effect depends on workload and batching, not a fixed per-request delay
--block-size (per-instance)intthe profiled value, else 16KV cache block size in tokens. vLLM treats this as a floor and an alignment unit, not the answer: it takes max(backend minimum, your value) as the alignment, derives the smallest multiple of that whose attention page covers one mamba page, and raises the block size to it — never lowering it. On Qwen3.8-27B, asking for 16 gives 784 and asking for 64 gives 832. The profiler records what the engine settled on in meta.yaml::engine_resolved.per_tp[tp] — per TP degree, since both the mamba page and the attention page scale with the rank's shard — and the simulator reads back the entry for the instance's tp_size, so lookups match the block size the latencies were measured at. An explicit value that disagrees is allowed but warned about. Bundles profiled before the field existed do not carry it and fall back to 16; ones written before it was split by TP carry a flat value, which is read as a fallback
--skip-prefillflagoffLegacy initialization flag: marks requests past first-token bookkeeping, but does not skip prompt computation or prepare cached KV. Not a decode-only workload generator

Routing​

FlagChoicesDefaultDescription
--request-routing-policyLOAD / RR / RAND / CUSTOMLOADCross-instance request routing. CUSTOM requires implementing the hook in serving/core/router.py; its shipped stub raises NotImplementedError
--expert-routing-policyBALANCED / RR / RAND / CUSTOMBALANCEDMoE expert token routing. CUSTOM reads the measured distinct-expert count from --gate-stats instead of deriving it from a uniform gate
--gate-statspathNoneA gate_stats.json (or its benchmark directory) recorded by bench run --record-gate-stats, used only with --expert-routing-policy CUSTOM. It supplies observed distinct-expert counts, not a latency coefficient or arbitrary load histogram. Relative paths resolve against the repo root. Missing, unreadable or mismatched files warn and fall back to BALANCED; see MoE expert routing
--enable-block-copy / --no-enable-block-copy (per-instance)boolTrueReuse rows across matching block/site sequences. Exact for deterministic BALANCED/RR routing at a fixed shape; RAND copying suppresses per-layer variation. Contextual profiles disable reuse

Precision​

There are no precision flags. Every dtype is read from the model config, because that is where each one is actually decided:

CacheConfig fieldRule
Weightsquantization_config.quant_method, then torch_dtype / dtypeOn a quantized checkpoint the dtype fields describe the activation dtype, so DeepSeek-V3.2 (quant_method: fp8, torch_dtype: bfloat16) is fp8, not bf16. Same rule as the profiler's, because it also picks which perf/.../<variant>/ folder is read
KV cachequantization_config.kv_cache_scheme (compressed-tensors) or kv_cache_quant_algo (ModelOpt)Either one present means fp8. This is vLLM's own promotion at attention.py:281, and the direction its source states for itself: "kv cache dtype should be specified in the FP8 checkpoint config and become the 'auto' behavior". Otherwise the cache takes the floating dtype (torch_dtype / dtype), as vLLM's auto does, so a weight-only quantized checkpoint keeps a 16-bit cache: DeepSeek-R1 (quant_method: fp8, torch_dtype: bfloat16) caches bf16. Sparse-MLA models (index_topk, DeepSeek-V3.2 and GLM-5) keep the weight dtype instead, because vLLM picks their cache format per attention backend
Mamba conv statemamba_cache_dtypeauto falls back to the weight dtype
Mamba recurrent statemamba_ssm_dtypeauto falls back to the conv dtype. Qwen3.8-27B declares float32, so its recurrent state is 4 bytes where its conv state is 2
Sparse-indexer side cachenone — fixed by the modelDeepSeek/GLM store fp8 keys plus fp32 scales as uint8; MiniMax-M3 stores bf16. Neither follows the KV cache dtype

A dtype is a property of the checkpoint, and once a model carries five of them a flag per dtype is both unusable and unfaithful — it describes a model nobody can serve. To simulate a different precision, profile it: the profiler can write alternate perf/.../<variant>/ bundles, but the simulator reads the checkpoint-default one. A quantized checkpoint's --dtype changes floating activations, not its weight quantization; different activations get an -act... suffix. See profiler naming.

Speculative decoding​

FlagChoicesDefaultDescription
--num-speculative-tokens (per-instance)int0 (off when no acceptance rate is supplied)Positive values choose draft length N; -1 requests the published N from configs/spec_decode.json. Omitting both flags leaves speculation off. Supplying a rate with N=0 also requests the published N; without a published N that case remains off
--spec-acceptance-rate (per-instance)floatthe model's published valueFraction of drafted tokens the target accepts, so the mean accept length is 1 + rate * N. Marginal, which is what every published source reports — not Leviathan's conditional per-position alpha. A model with no published figure must be given one
--spec-position-acceptance (per-instance)comma-separated floatsunsetPer-position rates P(accepted >= i) for draft positions 1..N, as vLLM's per-position acceptance counters report them and bench run records as position_acceptance. Must be nonincreasing and within [0, 1]. Sets N when --num-speculative-tokens is 0 or -1; given explicitly, N must match its length. Takes precedence over --spec-acceptance-rate and the published curve: its mean replaces the aggregate rate, with a warning when the two differ
--spec-acceptance-policy (per-instance)FIXED / DECAY / CUSTOMFIXEDHow the accepted count is drawn. FIXED accepts each position independently at the pooled rate. DECAY draws a prefix of the draft from per-position rates, one uniform draw per step, so P(accepted >= i) follows the curve: same mean, different spread. Its curve is the explicit per-position input, else the published curve when no explicit rate is given, else a ramp around the rate

Model-owned MTP drafter time is charged from its profiles; missing required catalog entries or measurements are errors. vLLM runs the drafter N times per step — once, then num_speculative_tokens - 1 more. Each pass is a norm pair, an eh_proj, a full decoder layer, and (DeepSeek/GLM) a norm plus lm_head — the decoder layer dominating by roughly 4:1 over the wrapper. Reporting any of it as free would claim a speedup no engine can deliver, so a model with MTP modules (num_nextn_predict_layers, num_mtp_modules, mtp_num_hidden_layers) is charged for all N passes, emitted after the target's head — which is where vLLM runs them, from sample_tokens(). It raises if the architecture catalog is missing mtp.prologue or mtp.decoder_block, or if the bundle has no profiled mtp.csv. All four modern families ship both; a new family needs a profile run with --profile-mtp.

A model with no MTP modules drafts with a separate model or with n-gram — a serving choice rather than a checkpoint property, and the simulator has no second model to charge. That case warns instead of refusing, and says plainly that the reported speedup is an upper bound.

The drafter's KV cache is charged either way: an MTP module wraps a real decoder layer, so it publishes a cache spec of its own. That is +1.6% bytes/token on DeepSeek-V3.2's one module, +11.7% on MiniMax-M3's seven, +6.2% on Qwen3.8-27B.

Prefix caching and offloading​

FlagDefaultDescription
--enable-prefix-caching (per-instance)TruePrefix caching over a per-tier block pool with chained block hashes. Use --no-enable-prefix-caching to disable
--enable-prefix-sharingoffSecond-tier prefix pool shared across instances within a node
--prefix-storageNoneWhere the second-tier pool lives. None / CPU / CXL
--enable-local-offloading (per-instance)offWeight offloading to NPU (counts weight reads in profiling)
--enable-attn-offloading (per-instance)offAttention computation offloading to PIM. Requires the node's cpu_mem.pim_config; target CUDA graph replay is unsupported
--enable-sub-batch-interleaving (per-instance)offOverlap GPU compute with PIM attention. Requires attention offloading and pp_size: 1; invalid combinations are rejected

Dataset and output​

FlagTypeDefaultDescription
--datasetpathNoneJSONL workload file. See Workloads → JSONL format
--num-reqsint0Entries to load from the dataset (0 = all). For agentic, each entry is a session
--outputpathNonePer-request CSV output path. Stdout only if None. The literal {run_id} is replaced with the active run id

Run isolation​

Each invocation writes ASTRA-Sim intermediates under a run-specific input root so parallel simulations do not overwrite each other's generated configs, traces, or Chakra workloads. Text traces are not generated by default, and the run-specific input root is removed after a successful simulation by default.

FlagTypeDefaultDescription
--run-idstringauto-generatedPath-safe id for this simulation run. Used in astra-sim/inputs/runs/<run-id> and the {run_id} output placeholder
--inputs-rootpathastra-sim/inputs/runs/<run-id>Override the generated ASTRA-Sim input root, for example to place intermediates on local SSD or tmpfs
--save-trace-text / --no-save-trace-textboolfalseWrite each batch's trace as text, for inspection. Nothing in the pipeline reads it — the Chakra converter takes the trace rows directly — so it is produced only on request, and it is the only human-readable form of what the simulator emitted. Implies --keep-inputs
--keep-inputs / --no-keep-inputsboolfalseKeep the generated ASTRA-Sim inputs under astra-sim/inputs/runs/<run-id> after a successful simulation: the Chakra .et workloads and the generated network / system / memory configs, so a run can be replayed through ASTRA-Sim by hand. Without it, each batch's workload folder is also deleted during the run once pp_size + 1 newer ones of the same instance or DP group have been handed out

Logging​

FlagTypeDefaultDescription
--log-intervalfloat1.0Seconds between throughput / memory log lines
--log-levelchoiceWARNINGWARNING (default) / INFO / DEBUG

Quick reference: which flag for which feature​

FeatureFlag(s)
Multi-instance (parallelism via cluster config)(cluster config num_instances)
Tensor parallel(cluster config tp_size)
MoE expert parallel(cluster config ep_size)
DP+EP MoE(cluster config dp_group)
Prefix caching--enable-prefix-caching (default on), --enable-prefix-sharing, --prefix-storage
Chunked prefill--enable-chunked-prefill (default on), --long-prefill-token-threshold
PIM attention offload--enable-attn-offloading (cluster config sets pim_config)
FP8 KV cache(model config quantization_config.kv_cache_scheme)
ns3 backend--network-backend ns3
Heterogeneous instances in one run(cluster config per-instance overrides; see the tip above)

For the full conceptual treatment of each feature, browse the Simulator section. For runnable examples, see Examples.