Skip to main content

Continuous batching

The scheduler is the heart of each serving instance. Every iteration of the main loop calls scheduler.schedule(current, sys) and gets back a Batch (or None). The scheduler enforces the same constraints vLLM does: token budget, sequence count cap, and optionally chunked prefill. This page walks through the rules.

Need the configuration knobs? See Reference → CLI flags for the flag list. This page explains what each flag does internally.

Two phases, one scheduler​

Scheduler.schedule() in serving/core/scheduler.py follows vLLM V1's shape and runs in two phases per step:

PhaseQueueBehaviour
Aself.running (persistent across steps)Serve every running request. If one cannot get a block, preempt from the tail of running and retry.
Bself.waiting (arrival-ordered)Admit while budget and sequence slots remain. Stops at the first request that cannot be allocated — phase B never preempts.

Phase B is skipped entirely on any step that preempted. That anti-thrash rule is what keeps the running set from oscillating preempt → refill → preempt.

Admission is one step stale​

Phase B's arrival horizon lags by one step, because vLLM's scheduler_config.async_scheduling is on by default. That makes vLLM's max_concurrent_batches 2 at pp_size 1, and EngineCore.step_with_batch_queue composes batch k+1 right after submitting k, then blocks on k's future. The batch that runs next was therefore built while the previous one was still on the GPU, and a request arriving after that composition cannot join it — it waits for the batch after.

Following vLLM's loop for a request arriving at a during batch A: it misses B (composed when A was submitted) and lands in C (composed when A finished), so TTFT = (t_A_end − a) + dt_B + dt_C. _arrival_cutoff reproduces this by lagging the horizon to the completed batch's own composition clock.

The lag is dropped when the instance was not busy right up to the current clock — an idle engine's batch queue is empty, and vLLM admits a new arrival immediately — and schedule() retries phase B with the full horizon whenever the lagged pass admitted nothing.

At pp_size > 1, the scheduler applies this additional lag only when its _uses_v2_model_runner classification is true; the modeled V1 pipeline already supplies its in-flight window. This classification currently treats non-MoE, non-hybrid stacks as V2 and is separate from the cudagraph.runner setting used for graph dispatch. The TTFT effect depends on workload and batch composition, not a fixed fraction of a step. Turn it off with --no-async-scheduling to model an engine with async scheduling disabled.

--enable-prefix-caching does not change the code path, only whether blocks get indexed for reuse. There is one scheduler for both.

The constraints are:

  • Sequence cap: len(batch) <= --max-num-seqs. Default 128. Set to 0 for unbounded.
  • Token budget: sum(tokens_to_run_this_step) <= --max-num-batched-tokens. Default 2048.
  • Per-request cap (chunked prefill): tokens_for_this_request_this_step <= --long-prefill-token-threshold. Default 0 = disabled.

Blocks come from the per-instance NPU BlockPool, whose num_free_blocks is exact — an allocation either succeeds or reports failure in the same call, which is what decides whether to preempt. Details on Prefix caching.

What the scheduler picks each step​

Three things about the shape are load-bearing. Phase B never preempts — every failure path there leaves the loop rather than freeing someone else's blocks. Phase B is skipped entirely on any step that preempted, which is what stops the running set oscillating preempt → refill → preempt. And the whole thing is gated on a free pipeline slot, so with pp_size > 1 a step can return None purely because every stage already has a batch in flight. All three mirror vLLM V1's schedule().

Note also what is not in the diagram: there is no prefill branch and no decode branch. A request simply catches up to num_tokens_reached, so num_new is 1 in steady-state decode and the whole remainder for a resumed request. The trace classifies by scheduled token count after the fact.

Conceptually, the loop is:

budget = max_num_batched_tokens
scheduled, preempted = [], []

# Phase A: requests already running
for request in running:
if budget <= 0: break
cap = tokens_to_catch_up(request, budget)
while allocate_blocks(request, cap) failed:
victim = running.pop() # tail = lowest FCFS priority
preempt(victim); preempted.append(victim)
if victim is request: break
if allocation still failed: break
scheduled.append((request, cap)); budget -= cap

# Phase B: admit from waiting -- skipped entirely if anything was preempted
if not preempted:
while waiting and budget > 0 and len(running) < max_num_seqs:
request = waiting[0]
hit = look_up_prefix(request) # NPU blocks, then lower tiers
cap = tokens_to_catch_up(request, budget, from=hit)
if allocate_blocks(request, cap) failed: break # never preempts
waiting.pop(0); running.append(request)
scheduled.append((request, cap)); budget -= cap

return Batch(scheduled) if scheduled else None

tokens_to_catch_up is one expression for every request state:

min(req.num_tokens_reached - req.num_computed_tokens,
long_prefill_token_threshold or infinity,
budget)
  • Prefill, no chunk yet: prompt length minus any prefix cache hit.
  • Prefill, mid-chunk: remaining prompt tokens.
  • Decode: 1, because num_tokens_reached == num_computed_tokens + 1.
  • Resuming after preemption: whatever neither tier could return.

Chunked prefill​

--long-prefill-token-threshold N (or --enable-chunked-prefill which sets a sensible default) lets the scheduler split a long prefill across multiple iterations. Without it, a single 32k-token request hogs the whole budget and TPOT for other in-flight requests collapses.

Concretely, a request whose remaining prefill is 8000 tokens with --long-prefill-token-threshold 1024 runs as eight separate 8x1024-token chunks across eight scheduler iterations. The Request.num_computed_tokens field tracks progress; on each iteration the scheduler bumps it by however many tokens were just processed.

Decode steps continue to run concurrently in the same batch, the chunked prefill just keeps long prompts from monopolizing.

On a hybrid model, chunks must end on a block boundary​

A gated-DeltaNet layer keeps its state in slots where slot p holds the state after exactly (p + 1) * block_size tokens, and the state is written at a chunk end. So a chunk that ends mid-block would leave a slot holding a state no position can name. With prefix caching on — vLLM's mamba_cache_mode: "align" — the scheduler therefore clips the chunk, which is Scheduler._mamba_block_aligned_split:

  • a chunk that is not the prompt's last is floored to a block boundary — unless flooring would leave nothing and the block is wider than one chunk's budget, in which case it advances sub-block and realigns at the next boundary;
  • a chunk starting mid-block stops at the next boundary;
  • no chunk runs past the last block-aligned position in the sequence.

This changes batch composition on every hybrid run. Qwen3.8-27B's state forces vLLM to resolve block_size 784, so with a 2048-token budget a chunk is 784 or 1568 — never 2048.

The split can legitimately return zero tokens: that is vLLM's "insufficient budget for a block-aligned chunk", and the request waits for a step whose budget covers a whole block. It is not the deadlock the scheduler's num_new <= 0 guard catches, because the split only floors to zero when block_size <= max_prefill_tokens — so a fresh step's budget does cover one and the request always moves.

The rule is off for every non-hybrid model and whenever prefix caching is disabled (mamba_cache_mode: "none", where nothing is checkpointed and there is no invariant to protect).

No prefill phase, no decode phase​

There is no prefill-vs-decode branch in the scheduler, exactly as in vLLM. A request simply catches up to the length it has reached:

num_new = req.num_tokens_reached - req.num_computed_tokens

which is 1 in steady-state decode, a chunk during prefill, and the whole sequence for a request that was preempted and is recovering. The trace classifies by the scheduled token count instead: more than one token is a prefill chunk, exactly one is a decode. That is also how the attention kernel sees the batch.

With speculative decoding the target is num_tokens_with_spec = num_tokens + spec_tokens, so a verification step schedules 1 + N tokens for one sequence. That is classified by why it has many tokens, not by the count: req.num_spec_scheduled > 0 makes it a speculative decode, not a prefill chunk, because its 1 + N queries share one sequence's KV read where a prefill chunk of the same size does not. Rejection rolls back with num_computed_tokens -= num_rejected, and the rollback happens before the prefix cache hashes anything, so a block holding a rejected token is never indexed.

Pipeline depth (PP)​

For pp_size > 1 instances, the scheduler also keeps an inflight list of batches currently traversing the pipeline. Its length is capped at pp_size: when the pipeline is full, the scheduler returns None until ASTRA-Sim drains a stage.

This makes the simulator's PP behavior match production training frameworks (e.g., Megatron) where micro-batches stream through the pipeline.

Where the scheduler stops​

The simulator exits when, simultaneously:

  • Every scheduler returns None (no eligible requests).
  • Router.has_pending_requests() returns False (no future arrivals).
  • Router.has_deferred_sessions() returns False (no agentic sessions waiting on tool calls).

If only the third is non-empty, the main loop fast-forwards current to the next pending arrival time and resumes.

What the scheduler hands back​

scheduler.add_done(npu_id, sys, current) is called once per iteration when ASTRA-Sim reports completion. It returns:

(prompt_t, gen_t, end_reqs)
  • prompt_t counts all input tokens including prefix cache hits, matching vLLM's reporting, which also counts cached tokens. The line in add_done is literally prompt_t += num_new + req.prefix_cache_hit, and it fires once per request, on the step its prefill completes.
  • gen_t counts only newly generated tokens, incremented when a request catches up to num_tokens_reached. A resumed request recomputing its history contributes nothing here, so recomputation is never counted as generation.
  • end_reqs is the list of requests that completed during this iteration.

For prefill instances under P/D disaggregation, the main loop hands end_reqs to router.transfer_prefill_request so the decode instance picks them up.

Gotchas​

  1. Prefill plus prefix caching doesn't double-count: hit_len is subtracted from the tokens the scheduler actually runs, but added to prompt_throughput. So a 1000-token request with 600 tokens of prefix hit consumes 400 tokens of budget and reports 1000 tokens of prompt throughput.

  2. --max-num-seqs 0 means unlimited, not zero. Useful when you want pure token-budget gating, but watch memory.

  3. The token budget is shared across prefill + decode. A batch with 64 in-progress decodes and a 1500-token prefill chunk runs 1564 tokens this step. Decode contributions count.

  4. Pipeline parallelism caps inflight at pp_size. Each iteration's layers are split across stages on transformer-block boundaries, with send/recv between them, so inter-stage P2P latency is modeled.

What's next​