ZMQ Direct Workers¶
SMG can connect straight to the engine core over ZMQ. No engine HTTP server or gRPC servicer sits in the path: the engine runs headless (scheduling and model execution only), SMG does everything else — tokenization, chat templates, parsing, stop handling, and routing — and the two exchange token IDs over ZMQ sockets: local ipc:// sockets when they share a host, or TCP sockets for an engine on another host (see Engines on Another Host).
When to Use ZMQ¶
| HTTP | gRPC | ZMQ | |
|---|---|---|---|
| Worker URL | http://host:port |
grpc://host:port |
ipc:///path or tcp://host:port |
| Engine-side process | Engine's OpenAI-compatible server | Engine with a gRPC servicer | Headless engine core only |
| Tokenization, chat templates, parsers | Engine | Gateway | Gateway |
| Engine location | Any reachable host | Any reachable host | Same host (ipc://), or any host that can dial the gateway (tcp://) |
| Prefill-decode disaggregation | Yes | Yes | TokenSpeed only |
| KV-event stream for cache-aware routing | No | Yes | No |
Compared with gRPC, the direct path drops the servicer process, a serialization round-trip, and a context switch per message. Use gRPC when you need KV-event routing, embeddings, disaggregation with vLLM or SGLang, or vLLM engine admin operations (see Limits).
How It Works¶
For a worker URL ipc://<path>:
- SMG binds two data-plane sockets,
ipc://<path>-in.sock(requests) andipc://<path>-out.sock(outputs), plus a handshake listener attcp://127.0.0.1:<port>, where the port is derived from<path>(see Handshake Address). Atcp://worker binds TCP sockets instead (see Engines on Another Host). - The headless engine dials the handshake address. SMG answers with the data-plane addresses (a HELLO, INIT, READY exchange); the engine connects to them and reports its context length and data-parallel size.
- SMG marks the worker ready as soon as the handshake completes.
- Requests go to the engine as token IDs. Outputs come back as token IDs, with the engine's scheduler load piggybacked on the output batches.
The engine never sees text, so SMG also does the work that the engine's API server or gRPC servicer would otherwise do:
| Task | What SMG does |
|---|---|
| Tokenization, chat templates, reasoning and tool-call parsing | Same as in gRPC mode |
| String stops | Sends a stop string that encodes to a single token as a stop token ID, and matches longer stop strings on the decoded text and trims them. For Harmony (gpt-oss) models, stops are matched on the parsed channel text |
| EOS (vLLM) | Attaches the model's EOS token IDs to every request: from config.json and generation_config.json when --model-path is a local directory, otherwise from the tokenizer. TokenSpeed and SGLang stop at EOS on their own |
Default max_tokens (vLLM) |
Uses the context length the engine reported at handshake minus the prompt length |
n > 1 |
Sends n single-sample requests and merges the results; an explicit seed becomes seed + i for sample i, so samples differ but stay reproducible (vLLM and TokenSpeed; the SGLang wire carries no seed) |
| Data-parallel ranks | Picks the least-loaded engine inside a grouped worker (see Data-Parallel Engines) |
Supported Engines¶
| Engine | Runtime | Headless engine command | Notes |
|---|---|---|---|
| vLLM | vllm |
vllm serve <model> --headless ... |
Assumed when a ZMQ worker declares no runtime |
| TokenSpeed | tokenspeed |
python -m tokenspeed.cli serve --headless ... |
Must be declared. Needs --grammar-backend xgrammar for structured outputs on checkpoints it doesn't default to xgrammar, and --enable-output-logprobs for logprobs |
| SGLang | sglang |
python -m smg_grpc_servicer.sglang.headless ... |
Must be declared. Needs smg-grpc-servicer[sglang] (SGLang 0.5.21 or newer): its plugin dials SMG from inside the scheduler, and SGLang itself is unchanged |
All of these engines use the same handshake, so SMG can't tell which one dialed in. Declare the runtime with --backend (startup workers) or runtime_type (worker API). A ZMQ worker without a runtime is treated as vLLM, and SMG logs runtime_type unspecified for ZMQ worker; defaulting to vLLM EngineCore.
These are the only three ZMQ runtimes: smg serve accepts --connection-mode zmq only with --backend vllm, --backend tokenspeed, or --backend sglang, and the gateway rejects any other runtime with ZMQ worker ... has unsupported runtime ...: only vllm, tokenspeed and sglang are supported over the ZMQ direct backend. TensorRT-LLM and MLX can't use ZMQ; connect those engines over gRPC.
SMG doesn't check engine versions at connect time, but it does check wire layout. An engine can declare the field order of its positional (array-encoded) request structs in its ready response — the SGLang launcher does — and SMG compares the fields it encodes against that order once per engine at connect, before the first request. If a field sits in a different slot than the one SMG encodes it in, the connection fails with a wire layout mismatch error naming the engine, the struct, and the field that differs (for example, that SamplingParams field 2 is stop_token_ids on SMG's side and watermark on the engine's). An engine release whose wire moved under SMG is refused at connect rather than failing every request one by one.
Only the fields SMG sends are compared, so an engine release that appends new fields after them still connects, and an engine that declares no layouts connects unchecked. SMG's decoders likewise accept fields that newer engine releases append. SMG's CI runs the ZMQ suites against vLLM 0.31.0, SGLang 0.5.21, and a pinned TokenSpeed revision.
Quick Start with smg serve¶
smg serve --connection-mode zmq starts headless engines and the gateway together:
smg serve \
--backend vllm \
--connection-mode zmq \
--model meta-llama/Llama-3.1-8B-Instruct \
--router-model-path meta-llama/Llama-3.1-8B-Instruct \
--host 0.0.0.0 \
--port 30000smg serve \
--backend tokenspeed \
--connection-mode zmq \
--model meta-llama/Llama-3.1-8B-Instruct \
--router-model-path meta-llama/Llama-3.1-8B-Instruct \
--host 0.0.0.0 \
--port 30000smg serve \
--backend sglang \
--connection-mode zmq \
--model-path meta-llama/Llama-3.1-8B-Instruct \
--host 0.0.0.0 \
--port 30000What smg serve sets up in ZMQ mode:
| Item | Behavior |
|---|---|
| Worker URL | ipc://<socket dir>/engine-<worker port> per replica. The socket directory is $SMG_ZMQ_SOCKET_DIR, default /tmp/smg-zmq-<uid> |
| Engine command | A headless engine that dials the handshake port derived from its worker URL (commands below) |
| TokenSpeed defaults | Adds --grammar-backend xgrammar and --enable-output-logprobs unless you pass your own value for either flag |
| Engine health wait | Skipped. The gateway starts at once, and /readiness returns 503 until a worker has completed its handshake and the tokenizer has loaded |
| Gateway runtime | --backend is forwarded so the gateway speaks the engine's wire protocol |
| Replicas | smg serve's own --data-parallel-size N (before any --) starts N independent single-engine workers, each with its own socket path and GPU slice. The launch stops early if two replicas' paths derive the same handshake port (ZMQ handshake port collision on ...); change --worker-base-port |
The launcher builds these engine commands. If you pass a flag the launcher sets, your copy is dropped, except for the two TokenSpeed defaults above, where your value is used:
# vLLM
python -m vllm.entrypoints.cli.main serve <model> --headless \
--data-parallel-size 1 --data-parallel-size-local 1 \
--data-parallel-address 127.0.0.1 --data-parallel-rpc-port <handshake port>
# TokenSpeed
python -m tokenspeed.cli serve --headless --model <model> \
--port <worker port> --dist-init-addr 127.0.0.1:<store port> \
--data-parallel-address 127.0.0.1 --data-parallel-rpc-port <handshake port> \
--zmq-engine-index 0 --grammar-backend xgrammar --enable-output-logprobs
# SGLang
python -m smg_grpc_servicer.sglang.headless \
--zmq-handshake-address tcp://127.0.0.1:<handshake port> --zmq-engine-index 0 \
--model-path <model> --port <worker port>To put several data-parallel TokenSpeed engines behind one worker, pass the engine-level --data-parallel-size N after a -- separator on the smg serve command line: the flag passes through to the headless engine command, and the gateway awaits N engines on the worker's one socket set, so no --zmq-engine-count is needed. You can also start the engines yourself and register them as a grouped worker with smg launch (see Data-Parallel Engines).
The -- passthrough works only with --backend tokenspeed. A vLLM or SGLang smg serve command exits with an unrecognized arguments error on anything after a --, and a --data-parallel-size (or --dp-size) before any -- is always smg serve's own replica count, so neither engine takes a grouped launch through smg serve: a grouped vLLM worker is set up with smg launch only, and the SGLang ZMQ wire has no data parallelism at all (use gRPC for SGLang data parallelism).
Manual Setup with smg launch¶
Socket Path¶
Pick a path for each worker, for example ipc:///tmp/smg-zmq/engine-0. SMG creates the parent directory with mode 0700 if it doesn't exist. An existing directory must be a real directory (not a symlink) owned by the user SMG runs as, or SMG refuses to bind, so don't place sockets directly in a shared directory such as /tmp. The engine must be able to reach the directory: run it as the same user, or create the directory yourself with the access the engine needs.
Before binding, SMG deletes any socket files already at <path>-in.sock and <path>-out.sock, treating them as leftovers from an earlier gateway run. Give each gateway process its own socket paths: a second gateway configured with the same worker URL would delete the first one's live sockets.
Handshake Address¶
SMG binds the handshake on 127.0.0.1 at a port in 20000–29999, computed from the path after ipc:// as 20000 + FNV-1a-64(path) mod 10000. Compute it before starting the engine:
python3 - <<'EOF'
path = "/tmp/smg-zmq/engine-0" # the worker URL without "ipc://"
h = 0xCBF29CE484222325
for b in path.encode():
h = ((h ^ b) * 0x100000001B3) & 0xFFFFFFFFFFFFFFFF
print(20000 + h % 10000)
EOFFor this path it prints 22670. SMG also logs the address when it binds:
Binding ZMQ client for worker ipc:///tmp/smg-zmq/engine-0 (handshake=tcp://127.0.0.1:22670, engines=Declared)engines= shows how many engines the handshake waits for: Exactly(N) for a grouped worker with an explicit engine count, or Declared when the gateway takes the count from the first engine to dial (see Data-Parallel Engines).
Two paths can hash to the same port; SMG rejects the second worker at registration. Workers added through the worker API can set a fixed zmq_handshake_address instead.
Engines on Another Host¶
A worker URL tcp://<host>:<port> serves engines that run on other machines. The scheme selects ZMQ like ipc:// does, but everything rides TCP: SMG binds the handshake at the URL itself and its request and output sockets on ephemeral ports of the same host, and hands those addresses to the engines during the handshake. <host> must therefore be an address of the gateway's machine that the engines can reach — a wildcard such as 0.0.0.0 is refused, and a bare IPv6 host needs brackets (tcp://[fd00::5]:31000).
Start the engine with the worker URL as its dial target: --data-parallel-address <host> --data-parallel-rpc-port <port> for vLLM and TokenSpeed, or --zmq-handshake-address tcp://<host>:<port> for the SGLang headless launcher. A tcp:// worker binds no socket files and takes no zmq_handshake_address (its URL already is the handshake address).
These sockets carry no authentication, so put tcp:// workers only on a trusted network. Everything else on this page — runtimes, grouped workers, lifecycle, features and limits — applies to ipc:// and tcp:// workers alike.
Start the Engine¶
vllm serve meta-llama/Llama-3.1-8B-Instruct \
--headless \
--data-parallel-size 1 \
--data-parallel-size-local 1 \
--data-parallel-address 127.0.0.1 \
--data-parallel-rpc-port 22670python -m tokenspeed.cli serve \
--headless \
--model meta-llama/Llama-3.1-8B-Instruct \
--data-parallel-address 127.0.0.1 \
--data-parallel-rpc-port 22670 \
--zmq-engine-index 0 \
--grammar-backend xgrammar \
--enable-output-logprobsWithout --grammar-backend, TokenSpeed turns its grammar backend on only for checkpoints it defaults to xgrammar; the rest run with none and reject requests that need constrained decoding, such as tool_choice: "required" or a json_schema response format — the request fails with the engine's refusal instead of coming back unenforced. --grammar-backend xgrammar turns enforcement on for any model. Output logprobs are off by default. When several TokenSpeed engines share a host, give each its own --port; TokenSpeed derives its internal control ports from it.
python -m smg_grpc_servicer.sglang.headless \
--zmq-handshake-address tcp://127.0.0.1:22670 \
--zmq-engine-index 0 \
--model-path meta-llama/Llama-3.1-8B-Instruct \
--port 31000The launcher comes with pip install "smg-grpc-servicer[sglang]" and starts SGLang's own scheduler ranks with no HTTP server, tokenizer manager, or detokenizer; the package's plugin dials SMG's handshake from inside the scheduler once the model has loaded. Every flag other than the two --zmq-* options is an ordinary SGLang server arg. When several SGLang engines share a host, give each its own --port; SGLang derives its internal control ports from it.
Add your usual engine flags, such as --tensor-parallel-size.
Start the Gateway¶
smg launch \
--worker-urls ipc:///tmp/smg-zmq/engine-0 \
--backend vllm \
--model-path meta-llama/Llama-3.1-8B-Instruct \
--host 0.0.0.0 \
--port 30000Use --backend tokenspeed or --backend sglang to match the engine. --model-path is required: it names the model and loads the tokenizer.
SMG binds the sockets as soon as it registers the worker, then waits up to --zmq-startup-timeout-secs (default 3600 seconds; with smg serve, --router-zmq-startup-timeout-secs) for each handshake message. SMG sees nothing from the engine except those messages, so this one bound must cover everything the engine does between its HELLO and its READY — loading weights, profiling the KV cache, capturing graphs. --worker-startup-timeout-secs bounds registration, not the handshake. If the handshake fails, the next health probe starts a new attempt. An engine that dialed in during an earlier attempt still connects: the new attempt accepts its READY.
To serve several engines, list one URL per engine. They are independent workers balanced by --policy.
Register Workers at Runtime¶
Add a ZMQ worker to a running gateway with POST /workers. The ipc:// and tcp:// schemes select ZMQ, so connection_mode can be left out:
curl -X POST http://localhost:30000/workers \
-H "Content-Type: application/json" \
-d '{
"url": "ipc:///tmp/smg-zmq/engine-1",
"runtime_type": "tokenspeed"
}'ZMQ-related worker fields:
| Field | Description |
|---|---|
url |
ipc://<path> (the path is required), or tcp://<host>:<port> for engines on another host |
runtime_type |
vllm, tokenspeed, or sglang (runtime is accepted as an alias). Unset means vLLM |
zmq_handshake_address |
A tcp:// address to bind for the handshake instead of the one derived from an ipc:// path. Rejected on non-ZMQ workers, and on tcp:// workers, whose URL already is the handshake address |
dp_size |
Number of engines in a grouped worker; if omitted, the worker waits for the group size declared by the first engine to dial (one for an engine that declares none). Leave dp_rank unset |
labels.model_path |
Model ID and tokenizer source, for gateways started without --model-path |
zmq_handshake_address pairs a worker with an engine that dials a fixed address. A TokenSpeed engine started without --data-parallel-rpc-port dials tcp://127.0.0.1:30500, so this worker pairs with it:
curl -X POST http://localhost:30000/workers \
-H "Content-Type: application/json" \
-d '{
"url": "ipc:///tmp/smg-zmq/tokenspeed-0",
"runtime_type": "tokenspeed",
"zmq_handshake_address": "tcp://127.0.0.1:30500"
}'Registration runs in the background (the API answers 202 Accepted), so a rejected worker shows up only in the gateway log; see Troubleshooting. For the full request schema, see Worker Management.
Data-Parallel Engines¶
There are two ways to run data parallelism over ZMQ:
- Replicas: several single-engine workers, each with its own socket path. The gateway's routing policy spreads requests across them.
smg serve --data-parallel-size Nsets this up. - Grouped worker: one worker URL and one socket set, with
Ndata-parallel engines behind it. The gateway sees one worker and picks the engine (rank) for each request itself. For TokenSpeed,smg servesets this up when an engine-level--data-parallel-size Nfollows--on its command line; a vLLM group is set up only manually withsmg launch. Configure a Grouped Worker shows the manual setup for both.
Replicas work with every ZMQ engine. Grouped workers are a vLLM and TokenSpeed setup: data parallelism over the SGLang ZMQ wire isn't supported yet (use gRPC for SGLang data parallelism).
Configure a Grouped Worker¶
Start the engine group with its data-parallel size. A TokenSpeed attention-DP group declares its size when it dials in, so --zmq-engine-count can stay unset and the gateway waits for the whole declared group. A vLLM engine declares no group size, so a vLLM group needs the explicit count:
vllm serve meta-llama/Llama-3.1-8B-Instruct \
--headless \
--data-parallel-size 2 \
--data-parallel-size-local 2 \
--data-parallel-address 127.0.0.1 \
--data-parallel-rpc-port 22670smg launch
--worker-urls ipc:///tmp/smg-zmq/engine-0
--backend vllm
--model-path meta-llama/Llama-3.1-8B-Instruct
--zmq-engine-count 2
python -m tokenspeed.cli serve
--headless
--model meta-llama/Llama-3.1-8B-Instruct
--data-parallel-size 2
--dist-init-addr 127.0.0.1:31233
--data-parallel-address 127.0.0.1
--data-parallel-rpc-port 22670
--zmq-engine-index 0
--grammar-backend xgrammar
--enable-output-logprobs
smg launch
--worker-urls ipc:///tmp/smg-zmq/engine-0
--backend tokenspeed
--model-path meta-llama/Llama-3.1-8B-Instruct
--zmq-engine-count 2
Each TokenSpeed rank dials in with its own identity (--zmq-engine-index plus its rank). TokenSpeed requires an explicit --dist-init-addr when --data-parallel-size is above 1; pick a free port outside the 20000–29999 handshake range.
--zmq-engine-countapplies to every ZMQ (ipc://ortcp://) URL in--worker-urls; an explicit value must be a positive integer, and--zmq-engine-count 1behaves like leaving the flag unset. For a worker added through the API, set"dp_size"on the worker instead; adp_sizeof 1 likewise behaves like leaving the field out.- Left unset, the count settles on what the first engine to dial declares, and an engine that declares no group size counts as one. A vLLM group with the count unset therefore settles at a single engine, and the rest of its ranks dial in as extras.
- The count the gateway waits for must match the number of engines that dial in. An extra engine can fail the handshake with
duplicate HELLO ... after INIT phase; a missing one leaves the handshake waiting until it times out. - An engine may declare a group size that differs from the count the handshake awaits, whether that count was set explicitly or settled by the first engine. The engine still connects; the gateway logs
engine declares a group of a different size than this handshake awaitsand keeps the count it was waiting for. - The engine process needs
Ntimes its tensor-parallel size in GPUs. - Grouped workers can't be combined with
--dp-aware, which tries to expand the group into one worker per rank and fails withcannot be dp-aware expanded. Single-engine ZMQ workers register normally under--dp-aware.
Rank Selection¶
Output batches carry the producing rank's scheduler load: running requests, waiting requests, and KV-cache usage (vLLM's scheduler stats, or TokenSpeed's load snapshot). For each request, SMG scores every rank in the group and sends the request to the lowest score:
score = max(requests SMG has in flight on the rank, running + waiting)
+ waiting × 6 × max(0, kv_cache_usage − 0.5)- The in-flight floor keeps a stale report from making a busy rank look idle, so a burst of requests between reports spreads across ranks.
- Engines report load only while they produce output. When a rank's last in-flight request finishes, SMG resets that rank's running and waiting counts to zero and keeps its last reported KV usage.
- Ties rotate across ranks. A TokenSpeed build that sends no load snapshot is scored on SMG's in-flight counts alone.
- vLLM MoE models run their data-parallel ranks in lockstep and pause the whole group when idle. SMG sends the wake-up that vLLM's data-parallel coordinator process would otherwise send, so no coordinator runs. Ranks of dense models run independently.
Rank selection happens inside the connection. Routing policies see the group as one worker, and a plain request is never pinned to a rank. A disaggregated TokenSpeed request follows its KV rendezvous instead: a prefill group places it on the rank its rendezvous room selects, and a decode group takes it on the matching rank, so the KV transfer lands where the decode side looks it up.
Worker Lifecycle¶
| Stage | What happens |
|---|---|
| Registered | The worker starts as pending. SMG binds its sockets and starts the handshake right away |
| Connected | SMG promotes the worker to ready the moment the handshake completes, without waiting for the health-check success threshold |
| Serving | Health probes read a local liveness flag; the engine has no health RPC. /readiness stays at 503 with tokenizer not yet registered until the model's tokenizer has loaded |
| Engine lost | SMG marks the connection dead when the engine signals that it died (vLLM does so on a fatal error, TokenSpeed also on shutdown, and SGLang whenever its scheduler's event loop exits), the socket fails, three outputs in a row can't be decoded, the engine leaves a frame unaccepted on its input for 300 seconds, or in-flight requests spend a combined 300 seconds — past the prefill allowance for the prompt tokens the engine holds — without any output from the engine. In-flight requests fail. The next health probe drops the dead connection, and a later probe binds the sockets again so a restarted engine can reconnect |
| Removed | DELETE /workers/{worker_id} drains the worker, then removes it; a handshake still in progress is cancelled and its sockets are released |
- Health checks stay on for ZMQ workers even with
--disable-health-checkor a per-workerdisable_health_check, because the probe is what reconnects a restarted engine. SMG logsIgnoring disabled health checks for ZMQ worker .... - With
--remove-unhealthy-workers, a ZMQ worker whose engine stays down is removed and nothing re-adds it. The flag is off by default (left unset with service discovery, removal covers only the workers discovery itself re-adds); leave it off for ZMQ workers so they rejoin when their engine restarts. - An engine that stops reading its input (a paused or frozen engine core) fails requests, not the connection. A request the engine does not accept within 10 seconds fails with a retryable
503, and a retry can select a different worker (see Retries); the requests the engine already holds keep streaming. The unaccepted frame is still delivered when the engine reads again, followed by an abort, so the engine drops the failed request instead of generating output nobody is waiting for. While the input is blocked, 256 requests or 32 MiB of request payload may wait behind it; a request that would take the waiting requests past either bound gets the503immediately instead of after the 10-second wait. - The 10-second and 300-second windows are fixed, and so is the prefill allowance's rate: the prompt tokens of requests the engine has accepted but not yet produced output for extend the 300-second bound at 2,000 tokens per second — 500 extra seconds per million pending prompt tokens, and none when no prefill is pending. The TokenSpeed and SGLang wires carry token batches only, as does vLLM's with
--disable-log-stats: such a wire says nothing for a prompt until its first token, so a batch of long prompts is silent for as long as its prefill takes. The allowance keeps that silence from counting, so a single request whose prefill outlasts 300 seconds is not failed as a dead engine. A vLLM engine with stats logging on reports every scheduler step, so it is never silent through a prefill. The rate is a bound on how slowly a healthy engine may prefill, not an estimate of how fast it does, so the verdict on a genuinely dead engine comes later in proportion to the prompt it was handed. - Only silence past the allowance counts, and it accumulates across requests: the in-flight time since the engine's last output, beyond its pending prefill work, adds up over consecutive busy periods, and only an output of any kind — or 300 seconds with nothing in flight — resets the count. A request cancelled before its first output leaves its prefill allowance in place until it has passed: the engine may still be mid-prefill on the cancelled prompt.
- Updating a ZMQ worker's properties, such as labels or priority, keeps its live connection.
Features and Limits¶
Supported over ZMQ¶
| Feature | vLLM | TokenSpeed | SGLang |
|---|---|---|---|
| Chat Completions, Completions, Responses, and Messages, including streaming | Yes | Yes | Yes |
| Tool-call and reasoning parsing | Yes | Yes | Yes |
Structured outputs (response_format, constrained tool_choice, regex, grammar) |
Yes | Yes, with --grammar-backend xgrammar set on the engine (without the flag, only checkpoints TokenSpeed defaults to xgrammar enforce them) |
Yes |
| String stops and EOS | Yes | Yes | Yes |
| Harmony (gpt-oss) stop strings | Yes | Yes | Yes |
| Multimodal inputs | Yes, one modality per request | Yes; media whose image_grid_thw/video_grid_thw grids feed MRoPE positions are served only for the Qwen3.5 lineage (Qwen3.5, Qwen3.5-MoE, Qwen4-Exp), and only when the engine's handshake declares the checkpoint's spatial_merge_size, image_token_id, and video_token_id; patch-layout grids (Muse Spark) pass |
Images, assembled as over gRPC; video and audio are refused |
| Output logprobs | Yes, including top_logprobs |
Sampled-token logprobs, with --enable-output-logprobs; top_logprobs above 1 only when every connected engine declares the top-k column at its handshake, rejected otherwise |
Yes, including top_logprobs |
| Prompt logprobs | Forwarded to the engine, but no API field requests them | Only when every connected engine declares the prompt-logprob column at its handshake, rejected otherwise; token_ids_logprob is rejected |
Yes, requested through /generate's logprob_start_len; token_ids_logprob is rejected |
n > 1 and sampling seed |
Yes | Yes | n > 1 yes; the SGLang wire carries no seed |
| Prefill, decode, and encode worker roles | No | Yes (see Limits) | No |
| Engine admin (cache flush, profiling) and RL control routes | No | Yes, when every rank's handshake declares the control reply | Cache flush and profiling yes; RL control routes no |
Limits¶
| Limit | What you see |
|---|---|
| Host scope and mesh | ipc:// workers are same-host: local socket paths with a loopback handshake. tcp:// workers serve engines on another host over unauthenticated sockets — trusted networks only. ZMQ workers of either scheme are never synced to high-availability mesh peers; each belongs to the gateway that registered it |
| Disaggregated roles are TokenSpeed-only | A vLLM or SGLang ZMQ worker registered as prefill, decode, or encode fails with ZMQ worker ... cannot serve worker type .... A TokenSpeed ZMQ worker serves every role, with conditions checked at connect: the engine must declare the matching role, a prefill or encode engine must declare its bootstrap address in the handshake, and an encode engine must list the encode request — a connection failing any of these ends with an error naming the gap. PD and EPD routing select gRPC workers and TokenSpeed ZMQ workers |
| No KV-event stream | cache_aware tracks ZMQ workers with its approximate prefix tree, and SMG logs ... the ZMQ transport has no KV-event stream ... |
| No embeddings | 501 with code unsupported_backend: ZMQ backend does not support embeddings yet |
| Engine admin operations | vLLM EngineCore carries no control calls: POST /flush_cache skips such workers, and profiling on one fails with ... is not supported over this engine's ZMQ wire. SGLang schedulers take the cache flush and the profiler; TokenSpeed schedulers take both — plus the RL control routes (weight updates, pause/continue, and the engine's other POST control routes) — when every rank's handshake declared the control reply. vLLM's management RPCs (LoRA add/remove, sleep/wake, weight updates) stay unported over ZMQ |
| No tokenizer from the engine | The gateway loads it from --model-path or --tokenizer-path (or the worker's model_path/tokenizer_path labels) |
| No dp-aware rank pinning | See Rank Selection |
| vLLM multimodal | Mixed image and video in one request: 400, the vLLM ZMQ backend takes one modality per request .... Worker-side media processing (media references) needs a gRPC vLLM worker: 400, multimodal_not_supported |
| TokenSpeed logprobs | top_logprobs above 1 or prompt logprobs on an engine that declares no column for them, or token_ids_logprob on any: 400, ... not supported over the TokenSpeed ZMQ backend |
| TokenSpeed MRoPE models | Outside the Qwen3.5 lineage: 400, ... MRoPE position tensors of this model are not derivable over the TokenSpeed ZMQ wire (the Qwen3.5 lineage's are); use the gRPC transport for this model. Any multimodal request to a Qwen3.5-lineage engine whose handshake omits the layout facts: 400, ... need its MRoPE position table, which the launcher's model facts (spatial_merge_size, image_token_id, video_token_id) did not provide ... |
| SGLang non-image media | Video or audio in a request to an SGLang ZMQ worker: 400, multimodal_not_supported (backend sglang does not support video inputs). Video needs SGLang gRPC workers |
SGLang token_ids_logprob |
501, selected-token logprobs require the SGLang gRPC servicer |
| SGLang data parallelism | The wire has no DP yet (see Data-Parallel Engines); use gRPC for SGLang data parallelism |
Verify¶
# Ready once the handshake has completed and the tokenizer has loaded
curl http://localhost:30000/readiness
# Transport, runtime, and status of each worker
curl -s http://localhost:30000/workers | jq '.workers[] | {url, connection_mode, runtime_type, status}'Expected worker entry:
{
"url": "ipc:///tmp/smg-zmq/engine-0",
"connection_mode": "zmq",
"runtime_type": "vllm",
"status": "ready"
}Send a request:
curl http://localhost:30000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "meta-llama/Llama-3.1-8B-Instruct",
"messages": [{"role": "user", "content": "Hello!"}]
}'Gateway metrics label ZMQ workers with connection_mode="zmq":
curl -s http://localhost:29000/metrics | grep 'connection_mode="zmq"'Troubleshooting¶
Worker stays pending
Symptoms: /readiness returns 503 with insufficient healthy workers, and /workers shows "status": "pending".
Solutions:
- Check that the engine dials the address SMG bound: compare the engine's
--data-parallel-rpc-portand--data-parallel-addresswith thehandshake=value in SMG'sBinding ZMQ client for worker ...log line. With no engine, once--zmq-startup-timeout-secsexpires, SMG logsZMQ backend handshake failed for ...: ... startup handshake timed out while waiting for HELLO after 3600s; will retry on the next health probe. - For a grouped worker, check that an explicit
--zmq-engine-count(ordp_size) matches the engine's--data-parallel-size. With no explicit count, the gateway waits for the group size declared by the first engine to dial, so the worker stays pending until that many engines have dialed in. - A worker added through the API to a gateway started with
--disable-health-checkand no ZMQ workers at startup is never promoted. Restart the gateway without that flag.
Socket bind errors or stale socket files
Symptoms: The worker stays pending, and the gateway log shows ZMQ backend handshake failed for ... followed by a socket or directory error.
Solutions:
SMG deletes leftover socket files at <path>-in.sock and <path>-out.sock before binding, so a crashed gateway doesn't block a restart. Other causes:
ipc socket path ... exists but is not a socket; refusing to unlink: a regular file sits at the socket path. Move it away.ipc socket dir ... is owned by uid ...oripc socket dir ... exists but is not a directory: use a directory owned by SMG's user, not/tmpitself or a symlink.- Another process holds the handshake port. Choose a different socket path, or register the worker with
zmq_handshake_address. - The socket path is too long. Unix socket paths have a small length limit (108 bytes on Linux), so keep
<path>-out.sockshort.
Registration rejected
Symptoms: The worker never appears in /workers, and the gateway log shows Failed job: type=AddWorker, worker=ipc://... with the reason.
Solutions:
has no model identity: start SMG with--model-path(withsmg serve,--router-model-path), or set amodel_pathlabel on the worker.has unsupported runtime: usevllm,tokenspeed, orsglang.would bind handshake address ..., already claimed by worker ...: two socket paths hash to the same port. Rename one, or setzmq_handshake_address.zmq_handshake_address must be a tcp:// addressorZMQ worker URL must be ipc://<path> or tcp://<host>:<port>: fix the address or URL.cannot serve worker type: vLLM and SGLang ZMQ workers must beregular; TokenSpeed engines can serve prefill, decode, and encode (see Limits).cannot be dp-aware expanded: drop--dp-aware, or use single-engine workers.
TokenSpeed or SGLang requests fail or hang
Symptoms: The worker connects, but requests error or time out.
Solutions:
- Look for
runtime_type unspecified for ZMQ worker; defaulting to vLLM EngineCorein the gateway log. A TokenSpeed or SGLang worker must be declared with--backend tokenspeed/--backend sglangor an explicitruntime_type, or SMG speaks the vLLM wire protocol to it. - If a TokenSpeed engine rejects requests that need constrained decoding (its grammar backend is off) or logprobs are missing, start it with
--grammar-backend xgrammarand--enable-output-logprobs.
Readiness reports tokenizer not yet registered
Symptoms: /readiness returns 503 with tokenizer not yet registered after the worker is ready.
Solutions:
- Wait for the tokenizer download from
--model-pathto finish, and check the gateway log for load errors. - Set
--tokenizer-pathif the tokenizer lives somewhere other than the model path.
Next Steps¶
- gRPC Workers — Engines behind a gRPC servicer, vLLM and SGLang disaggregation, and KV-event routing
- Multiple Workers — Mix worker types and register workers at runtime
- Load Balancing — Policies for spreading traffic across ZMQ replicas
- Architecture Overview — Where the ZMQ path sits in the gateway
- Configuration Reference —
--worker-urlsand related worker flags - Monitoring — Gateway and engine-load metrics