PD Disaggregation¶

Prefill-Decode (PD) disaggregation runs the two phases of LLM inference on separate workers. Prefill workers process the prompt and build its KV cache. Decode workers take over that cache and generate the output tokens. For every request, SMG picks a prefill/decode pair, runs the KV handoff the way the engine expects, and returns the decode worker's output to the client.

Overview¶

Why Disaggregate?¶

Traditional LLM inference has two distinct phases with different characteristics:

Running both phases on the same worker creates inefficiencies:

• Prefill batches are delayed waiting for decode slots
• Decode batches are small due to prefill memory pressure
• Neither phase is optimally configured

PD disaggregation solves this by dedicating workers to each phase.

Supported Engines¶

• The worker URL scheme selects the path. http:// workers go through the HTTP PD router, which forwards the client's JSON body to both legs and adds the handoff fields. grpc:// workers go through the gRPC pipeline, where SMG tokenizes the prompt and parses the output itself.
• Other runtimes have no PD protocol. TensorRT-LLM and MLX cannot serve as PD legs. A gRPC PD request that lands on another runtime fails with 400 runtime_pd_not_supported.
• Direct ZMQ workers are rejected. An ipc:// worker registered as prefill, decode, or encode fails registration, because the ZMQ wire carries no KV-transfer metadata. See ZMQ Workers.
• EPD adds an encode leg. For multimodal models on TokenSpeed, EPD moves the vision tower onto separate encode workers.

How It Works¶

Parallel Dispatch (SGLang, TokenSpeed)¶

1. Select a pair. The prefill policy picks a prefill worker, then the decode policy picks one of the decode workers that can pair with it (see Pairing).
2. Mint a rendezvous. SMG generates a bootstrap room and points both legs at the prefill worker's bootstrap server: its host and the bootstrap port the worker was registered with. Over HTTP, SMG adds bootstrap_host, bootstrap_port, and bootstrap_room to the JSON body of both legs. Over gRPC, SGLang receives them as disaggregation parameters and TokenSpeed as KV bootstrap fields.
3. Dispatch both legs at once. The engines use the room to find each other, and the prefill engine transfers the KV cache it computes to the decode engine.
4. Stream the decode output back to the client.

Register every prefill worker with its bootstrap port (--prefill <url> <port>). Over gRPC, a prefill worker without one is addressed on port 8998. Over HTTP, SMG forwards the registered value as it is. With --dp-aware, SGLang legs carry the DP rank of the selected worker. TokenSpeed places both legs by bootstrap_room % dp_size, so SMG mints the room to land on the prefill worker's rank.

Sequential Dispatch (vLLM)¶

1. Send prefill a one-token request. The prefill leg carries the prompt with the output capped at one token, streaming off, and n set to 1. Over HTTP, SMG sets the endpoint's own cap field (max_tokens, max_completion_tokens when the chat request used it, or max_output_tokens on /v1/responses) and drops min_tokens and stream_options. It also tags the leg with the connector's kv_transfer_params (see below).
2. Wait for prefill to finish. The prefill engine computes the KV cache and holds it for the handoff.
3. Relay the handoff. SMG adds kv_transfer_params to the original request and sends it to decode.
4. Decode receives the KV cache from prefill and streams tokens back to the client.

Connector Modes¶

The prefill worker's KV connector decides how SMG tags the prefill leg and what the decode leg carries:

The handoff falls back to a local recompute in these cases:

• NIXL prefill returns nothing. SMG increments smg_pd_kv_transfer_failures_total and decode recomputes the prompt. The usual causes are an outdated servicer or a worker without --kv-transfer-config.
• Mooncake without an engine id. Minting needs the prefill's KV engine id. Without it, gRPC falls back to injecting the bootstrap host and port, and over HTTP decode recomputes the prompt.
• n>1. The handoff is single-consumer, so no KV is handed off and decode computes the prompt itself. Over HTTP, SMG skips the prefill leg entirely. Over gRPC, the prefill leg still runs, untagged, and a Mooncake decode leg still receives the legacy bootstrap host and port.

A MoRI-IO decode engine never recomputes the prompt, so MoRI-IO has no such fallback: SMG fails the request instead (see MoRI-IO).

Where the Connector Comes From¶

A restarted vLLM process without a pinned engine_id comes back with a new KV engine id. When a gRPC prefill worker recovers from the failed or not-ready state, SMG re-reads its engine id before routing to it again. Workers registered over HTTP keep the engine id they were registered with.

MoRI-IO (HTTP)¶

vLLM's MoRIIOConnector moves the KV cache over RDMA with MoRI-IO in one of two modes. In READ mode ("read_mode": true in kv_connector_extra_config on both engines), the decode engine pulls the KV after the prefill leg returns. In WRITE mode, the default, the prefill engine pushes the KV into blocks the decode engine allocates. SMG sends the legs sequentially by default in both modes; a WRITE pair can opt into concurrent dispatch with the decode worker's moriio_write_dispatch label (see below).

A MoRI-IO decode engine never recomputes the prompt. Without a valid handoff it answers with text computed over KV that never arrived (READ) or waits for a push that never comes (WRITE). SMG therefore takes the MoRI-IO path whenever the decode worker is a MoRIIOConnector worker, and fails a request rather than sending a decode leg without its handoff.

HTTP vLLM workers report none of the MoRI-IO settings, so set them as worker labels when you register each leg, matching the engine's kv_connector_extra_config:

curl -X POST http://localhost:30000/workers \
  -H "Content-Type: application/json" \
  -d '{"url": "http://prefill-0:8000", "worker_type": "prefill", "runtime_type": "vllm",
       "kv_connector": "MoRIIOConnector", "labels": {"moriio_mode": "write"}}'
curl -X POST http://localhost:30000/workers \
  -H "Content-Type: application/json" \
  -d '{"url": "http://decode-0:8000", "worker_type": "decode", "runtime_type": "vllm",
       "kv_connector": "MoRIIOConnector", "labels": {"moriio_mode": "write"}}'

The engines do not need proxy_ip and proxy_ping_port, which only register them with vllm-router.

Before either leg is contacted, SMG refuses:

• A misconfigured pair with 503 moriio_pair_misconfigured: a leg that is not a MoRIIOConnector worker, legs in different modes, a worker with DP>1, a missing, unparsable, or invalid label (including a moriio_write_dispatch value other than sequential or concurrent), or, in WRITE mode, a decode side channel the prefill cannot reach: a loopback or unspecified decode host (localhost, 127.0.0.1, 0.0.0.0, [::]) while the prefill is not, or the prefill's own side channel. Under concurrent dispatch the decode engine dials the prefill's side channel as labeled, so the same refusal applies to a loopback or unspecified prefill host while the decode is not. The reason is in the gateway log. Pairing already keeps READ and WRITE workers apart, because their transports are moriio-read and moriio-write.
• Requests that would share one handoff with 400 moriio_fanout_unsupported: n>1, a batched prompt or prompt_embeds list, and use_beam_search.
• Request ids carrying peer addresses with 400 moriio_request_id_reserved: the connector reads its peer from markers in the request id (___prefill_addr_, ___decode_addr_) before the explicit fields, and vLLM takes the request id from X-Request-Id or, without that header, from the body's request_id. SMG checks every X-Request-Id value it forwards and the body field.
• Other routes with 501 moriio_route_unsupported: only /v1/chat/completions and /v1/completions are supported.

The connector returns a handoff only when the prefill leg ends at its one-token cap. That token is discarded, so SMG sets ignore_eos and removes stop conditions on the prefill leg; the decode leg keeps the client's. A prefill response whose handoff is missing, carries a transfer_id other than the minted one, or has a field the decode engine cannot parse fails the request with 502 moriio_handoff_invalid: under sequential dispatch no decode leg is sent, and under concurrent dispatch SMG drops the decode leg already in flight. The decode engine's connector parses these fields in its scheduler step, where a value of the wrong shape raises instead of failing one request — a relayed malformed handoff could break the decode engine's scheduling, not just this request. SMG therefore refuses a handoff unless:

• remote_engine_id is a non-empty string.
• remote_block_ids is a flat list of integer block ids, or one list per KV cache group.
• remote_host is a non-empty string without whitespace. It is not compared with the prefill worker's moriio_host label: a hostname or an address on another interface is accepted, and a sequential decode leg dials the address the prefill engine reports. A concurrent decode leg was already told the label.
• remote_handshake_port and remote_notify_port are non-zero ports, each written as a number or a string.
• remote_dp_size, remote_dp_size_local, remote_dp_rank, tp_size, and remote_tp_size, when present and not null, parse as integer counts, as a number or a decimal string.

The gRPC pipeline does not speak MoRI-IO and refuses such pairs with 501 moriio_grpc_pd_unsupported.

Under concurrent dispatch, SMG mints the decode leg's kv_transfer_params up front instead of relaying the prefill's handoff (see Connector Modes): the decode engine dials the side channel named by the prefill worker's moriio_host, moriio_handshake_port, and moriio_notify_port labels, which must therefore match the prefill engine's configuration. SMG still validates the handoff the prefill returns, with one extra check: it must report the same handshake and notify ports the decode leg was told. The decode response — even one that has already arrived — is forwarded only after the handoff passes; otherwise the request fails with 502 moriio_handoff_invalid and SMG drops the decode leg, whose engine would wait for a push that never comes.

With both legs in flight, the first leg to fail ends the request, and SMG drops the other leg, which makes its engine abort it. A prefill error status passes through as prefill_upstream_error, a decode transport error answers 502 decode_request_failed, and a decode engine that refuses the request takes no push, so its error status is returned without waiting for the prefill.

Failures, Cancellation, and Streaming¶

• Retries re-select both legs. Each attempt picks a new pair under the router's retry settings.
• A failed leg fails fast (gRPC, parallel dispatch). The first leg that fails to start answers the client right away. SMG drops the other leg as soon as its dispatch lands, which aborts its bootstrap room instead of leaving it to the engine's deadline. Over HTTP, a transport error on either leg cancels the other.
• Prefill failures stay on the prefill worker (sequential dispatch). A failed prefill leg never reaches the decode worker's circuit breaker, because decode was never contacted. Under concurrent MoRI-IO WRITE dispatch (see MoRI-IO) both legs are in flight: the first leg to fail answers the client, and the leg SMG drops records no outcome on its worker.
• Decode aborts wait for the handoff (gRPC). When the client disconnects, a prefill leg still running is aborted at once. The decode leg's abort waits for the leg's first response or a terminal event, for at most 30 seconds, to avoid tearing a decode engine down mid-transfer.
• HTTP streams start on the response heads (parallel dispatch). For a streamed request that does not ask for logprobs, SMG starts streaming decode output as soon as both legs return a 2xx response head. It drains the prefill body in the background, because closing it early would abort the KV transfer, and records the prefill worker's outcome when the drain ends. Requests with logprobs wait for the prefill body.
• Non-streaming HTTP responses keep a JSON content type. Streamed HTTP PD responses are sent as text/event-stream. A non-streaming decode body without logprobs is relayed with the decode worker's response headers, so its Content-Type comes from the engine. When the request asks for logprobs, SMG builds the response body itself — the logprob-merged body, or the plain decode body when the prefill body is missing or the merge fails — and sets Content-Type: application/json on it, rather than the application/octet-stream default of a raw byte response.

Parallel Sampling (n>1)¶

On the gRPC path, a text-only n>1 request to SGLang or TokenSpeed fans out into n single-sample PD dispatches. Each sample gets its own bootstrap room and request id (<id>-<i>); on TokenSpeed, a pinned seed is also offset by the sample index. The dispatches fail fast together, and their responses merge back into one response with one choice per sample. Multimodal requests keep a single dispatch.

The HTTP PD router has no per-sample fan-out: a /v1/completions request with one prompt and n>1 sends every sample to the same rendezvous, and on SGLang the extra samples stall. Serve n>1 SGLang traffic through gRPC workers. vLLM needs no fan-out, because n>1 skips the KV handoff (see above).

Multimodal Requests¶

Over HTTP, both legs receive the request's media as sent. On the gRPC path:

• The decode leg is a copy of the request without pixel tensors, so pixels travel only to prefill. TokenSpeed decode keeps the per-item metadata it needs.
• vLLM decode keeps each image's identity and its M-RoPE grid tensors (image_grid_thw, video_grid_thw, and the video timing tensor), so it computes the same positions as prefill.
• With worker-side media processing, vLLM processes each image or video once, on the prefill leg. Prefill returns a media identity without pixels, and SMG passes it to decode. With n>1 there is no handoff, so decode processes the media itself.
• vLLM decode workers started with --language-model-only receive only the prefill-expanded token ids, the KV handoff, and per-image content hashes. Requests such a pool cannot serve fail with a non-retryable 400: M-RoPE models (pd_decode_language_model_only_mrope), worker-side media references (pd_decode_language_model_only_media_refs), and multimodal n>1 (pd_decode_language_model_only_n_samples).

See Multimodal for media processing modes and tensor transport.

Supported APIs¶

Token counting generates nothing. Over HTTP, SMG sends it to a single prefill worker and adds no bootstrap fields. Over gRPC, SMG answers it from the model's tokenizer and contacts no leg at all. See the Messages API.

Gateway-side token counting on gRPC PD is newer than v1.12.0: v1.12.0 and earlier answer /v1/messages/count_tokens on gRPC PD with 501 Not Implemented.

Configuration¶

Flags¶

The startup worker lists may be empty: add legs at runtime with POST /workers and "worker_type": "prefill" or "decode" (see Worker Management). For every option, see the configuration reference.

Examples¶

smg launch \
  --pd-disaggregation \
  --prefill http://prefill-0:8000 8998 \
  --prefill http://prefill-1:8000 8998 \
  --decode http://decode-0:8000 \
  --decode http://decode-1:8000 \
  --prefill-policy cache_aware \
  --decode-policy power_of_two

# Workers run with --smg-grpc-mode (SGLang 0.5.16+; --grpc-mode before)
smg launch \
  --pd-disaggregation \
  --prefill grpc://prefill-0:50051 8998 \
  --decode grpc://decode-0:50061 \
  --model-path meta-llama/Llama-3.1-8B-Instruct

# NIXL: no bootstrap port
smg launch \
  --pd-disaggregation \
  --prefill grpc://prefill-0:50051 \
  --decode grpc://decode-0:50052 \
  --model-path meta-llama/Llama-3.1-8B-Instruct
# Mooncake: pass each prefill worker's VLLM_MOONCAKE_BOOTSTRAP_PORT
smg launch 
  --pd-disaggregation 
  --prefill grpc://prefill-0:50051 8998 
  --prefill grpc://prefill-1:50051 8999 
  --decode grpc://decode-0:50052 
  --model-path meta-llama/Llama-3.1-8B-Instruct

# Start PD mode without startup workers...
smg launch --pd-disaggregation --host 0.0.0.0 --port 30000

# ...then register each leg with its KV connector
curl -X POST http://localhost:30000/workers 
  -H "Content-Type: application/json" 
  -d '{"url": "http://prefill-0:8000", "worker_type": "prefill", "kv_connector": "NixlConnector"}'
curl -X POST http://localhost:30000/workers 
  -H "Content-Type: application/json" 
  -d '{"url": "http://decode-0:8000", "worker_type": "decode", "kv_connector": "NixlConnector"}'

smg launch 
  --pd-disaggregation 
  --prefill grpc://prefill-0:50051 8998 
  --decode grpc://decode-0:50052 
  --model-path meta-llama/Llama-3.1-8B-Instruct

Worker launch commands for each engine are in Getting Started: PD Disaggregation.

Per-Phase Policies¶

The prefill and decode legs each run their own policy instance, so a stateful policy such as round_robin keeps a separate rotation per leg. The prefill policy chooses among prefill workers that have an available compatible decode worker, then the decode policy chooses among that prefill's compatible decode workers. A leg without its own policy uses --policy. Both legs take the same tuning flags as --policy, such as --cache-threshold or the --least-load-* flags.

smg launch \
  --pd-disaggregation \
  --prefill http://prefill-0:8000 8998 \
  --prefill http://prefill-1:8000 8998 \
  --decode http://decode-0:8000 \
  --decode http://decode-1:8000 \
  --prefill-policy cache_aware \
  --decode-policy least_load

Prefill/Decode Pairing¶

A prefill worker can hand its KV cache only to a decode worker that speaks the same KV transfer protocol. SMG gives each prefill and decode worker a pairing descriptor and pairs a prefill only with decode workers whose descriptor is compatible. Compatibility is worked out when workers join or leave, not on every request.

What Is Compared¶

gRPC workers report these facts at registration. SGLang reports its transfer backend, kv_cache_dtype, page_size, attention_backend, and version. TokenSpeed reports the same except page_size. vLLM reports its connector, kv_cache_dtype, block_size, attention_backend, and model_dtype, but no version. HTTP workers report no transport or KV layout, only the runtime and engine version, so an HTTP vLLM worker's transport comes from its registered kv_connector.

GET /workers shows each prefill and decode worker's key in pd_pairing: the explicit protocol if one is set, otherwise runtime/transport/layout with ? for an unknown part, for example vllm/nixl/dtype=auto,page=16,attn=flash_attn,model=torch.bfloat16. The version is not part of the key.

Pairing Modes¶

lenient keeps a fleet that reports nothing pairing as before, and lets a rolling engine upgrade mix versions. Under strict, HTTP workers need an explicit protocol (or, for vLLM, a registered kv_connector), because they report no transport.

Explicit Pairing Protocol¶

Set a pairing protocol to declare compatibility yourself. Two legs that both carry one pair only when the values are equal, and nothing except the runtime is compared. When only one leg carries one, lenient falls back to the derived facts and strict refuses the pair. SMG reads it from these sources, highest precedence first:

1. A pairing_protocol worker label, for example in the labels of POST /workers.
2. The SMG_PAIRING_PROTOCOL environment variable of the engine process. The vLLM, SGLang, and TokenSpeed gRPC servicers report it in their server info.

The worker spec also has a pairing_protocol field, but v1.11.0 does not apply it when it registers the worker; use the label instead.

On Kubernetes, set SMG_PAIRING_PROTOCOL in the engine container; there is no pod annotation for it.

When no prefill shares a protocol with any decode, requests fail with 503 no_compatible_pd_pair. The message names the components that differ and each leg's keys, and SMG logs the same at warn level. A prefill worker with no compatible decode is skipped as long as other prefill workers can pair.

Admission and Errors¶

Decode Admission Window (gRPC)¶

Before a gRPC PD or EPD dispatch goes out, SMG claims one room for each backend request on the decode worker, within the running window that engine reports: SGLang's max_running_requests or TokenSpeed's max_num_seqs (the engines' --max-running-requests and --max-num-seqs flags). When the window is full, the request waits up to --pd-admission-wait-secs (default 30) and is then shed with 503 worker_overload_protection_shed and a Retry-After header. 0 sheds immediately. Keep the wait well under the engine's bootstrap deadline (120 seconds on TokenSpeed). Decode workers that report no window, such as vLLM, are not gated. smg_pd_admission_waits_total counts dispatches admitted after a wait, and smg_pd_admission_sheds_total counts the sheds. See Overload Protection.

Context Length (gRPC)¶

On the gRPC path, SMG counts the prompt tokens itself and rejects a prompt longer than the model's context window with 400 context_length_exceeded before dispatch. In PD mode the smaller window of the two legs applies. Workers that advertise no window are not checked, and the HTTP path leaves the check to the engine.

Error Codes¶

The moriio_* codes are newer than v1.11.0.

/readiness reports ready only when at least one prefill worker and one decode worker are healthy (and an encode worker in EPD mode).

EPD (Encode-Prefill-Decode)¶

EPD adds a third worker role for multimodal models. Encode workers run the vision tower, and prefill and decode workers run the language model. SMG assigns each media item to an encode worker, the encode worker ships the resulting embeddings to the prefill worker over Mooncake, and prefill and decode then proceed as in PD. EPD requires gRPC workers running TokenSpeed; SMG refuses to start an EPD router for HTTP workers.

smg launch \
  --epd-disaggregation \
  --encode grpc://encode-0:50060 8995 \
  --prefill grpc://prefill-0:50061 8998 \
  --decode grpc://decode-0:50062 \
  --model-path "$MODEL"

• Each media item gets its own encode worker and rendezvous, so the items of one request can be encoded on different workers.
• Text-only requests skip the encode leg.
• SMG preprocesses the media for the encode workers. Encode workers never take raw media references, so with --mm-processing worker an EPD request that carries media fails with 400 multimodal_not_supported.
• EPD mode does not serve /v1/responses (501).
• EPD keeps all three legs on one runtime but does not apply --pd-pairing-mode.

Worker launch commands are in Getting Started: PD Disaggregation.

Kubernetes Service Discovery¶

smg launch \
  --service-discovery \
  --pd-disaggregation \
  --prefill-selector app=sglang role=prefill \
  --decode-selector app=sglang role=decode \
  --service-discovery-namespace inference \
  --service-discovery-port 8000 \
  --prefill-policy cache_aware \
  --decode-policy least_load

SMG assigns each pod the role of the selector it matches. Prefill (and encode) pods advertise their bootstrap port with the sglang.ai/bootstrap-port annotation. vLLM pods declare their KV connector and engine id with the smg.ai/kv-connector and smg.ai/kv-engine-id annotations. Service discovery turns on IGW mode automatically; PD and EPD mode and the per-role policies are kept. See PD Disaggregation Discovery for the annotation formats and a pod example.

Sizing Guidelines¶

Prefill Workers¶

Prefill is compute-bound. Its cost grows with prompt length, and every request's time to first token includes the prefill and the KV transfer. Add prefill capacity when time to first token (smg_pd_ttft_seconds) climbs while the decode workers have headroom.

Decode Workers¶

Decode is memory-bandwidth-bound. Its throughput depends on how many sequences fit in the decode batch, which the KV cache capacity limits. Add decode capacity when dispatches wait for or are shed at the decode admission window (smg_pd_admission_waits_total, smg_pd_admission_sheds_total), or when time per output token grows.

Prefill:Decode Ratio¶

Long prompts with short outputs need relatively more prefill capacity. Short prompts with long outputs need relatively more decode capacity. Start from your workload's prompt and output lengths, then scale each pool independently based on the signals above.

Monitoring¶

PD Metrics¶

Related series:

• smg_worker_selection_total counts picks per leg (worker_type is prefill, decode, or encode), with connection_mode, model, and policy labels.
• smg_worker_requests_active shows in-flight requests per worker (worker label).
• smg_router_ttft_seconds with backend_type="pd" (gRPC) measures only the decode leg. Use smg_pd_ttft_seconds for the time the client waits.
• smg_engine_pd_kv_transfer_latency_ms, smg_engine_pd_kv_transfer_speed_gb_s, smg_engine_pd_prefill_queue_reqs, and smg_engine_pd_decode_queue_reqs (labels worker, role, dp_rank) are published when an engine's load report includes disaggregation statistics, as the SGLang gRPC servicer's does.

See the Metrics Reference for every metric.

PromQL Queries¶

# p95 PD time to first token by model
histogram_quantile(0.95,
  sum by (le, model) (rate(smg_pd_ttft_seconds_bucket[5m])))

# Share of vLLM PD dispatches running passthrough
sum(rate(smg_pd_kv_connector_mode_total{mode="passthrough"}[5m]))
  / sum(rate(smg_pd_kv_connector_mode_total[5m]))

# gRPC PD dispatches shed at the decode window
rate(smg_pd_admission_sheds_total[5m])

# In-flight requests per worker
smg_worker_requests_active

Troubleshooting¶

Debug Logging¶

RUST_LOG=smg=info,smg::routers=debug smg launch --pd-disaggregation ...

At debug level SMG logs each selected pair and, on the gRPC vLLM path, whether it relayed the prefill's kv_transfer_params.

Verify Configuration¶

# Role, health, and pairing key of every worker
curl -s http://localhost:30000/workers | jq '.workers[] | {url, worker_type, is_healthy, pd_pairing}'

# Ready only when both legs have a healthy worker
curl -i http://localhost:30000/readiness

What's Next?¶