ZMQ Direct Workers¶
When SMG and the inference engine share a host, SMG can connect straight to the engine core over ZMQ. No engine HTTP server or gRPC servicer sits in the path: the engine runs headless (scheduling and model execution only), SMG does everything else — tokenization, chat templates, parsing, stop handling, and routing — and the two exchange token IDs over local ipc:// sockets. A worker URL of tcp://<host>:<port> runs the same wire over TCP for an engine on another host (see Engines on Another Host).
When to Use ZMQ¶
| HTTP | gRPC | ZMQ | |
|---|---|---|---|
| Worker URL | http://host:port |
grpc://host:port |
ipc:///path or tcp://host:port |
| Engine-side process | Engine's OpenAI-compatible server | Engine with a gRPC servicer | Headless engine core only |
| Tokenization, chat templates, parsers | Engine | Gateway | Gateway |
| Engine location | Any reachable host | Any reachable host | Same host as SMG (ipc://), or any host that can dial the gateway (tcp://) |
| Prefill-decode disaggregation | Yes | Yes | TokenSpeed only |
| KV-event stream for cache-aware routing | No | Yes | No |
Compared with gRPC, the direct path drops the servicer process, a serialization round-trip, and a context switch per message. Use it when the gateway and engine run on one machine; a tcp:// worker URL extends the same wire to an engine on another host over a trusted network. Use gRPC when you need KV-event routing, embeddings, disaggregation on vLLM or SGLang, or engine admin operations on vLLM (see Limits).
How It Works¶
For a worker URL ipc://<path>:
- SMG binds two data-plane sockets,
ipc://<path>-in.sock(requests) andipc://<path>-out.sock(outputs), plus a handshake listener attcp://127.0.0.1:<port>, where the port is derived from<path>(see Handshake Address). - The headless engine dials the handshake address. SMG answers with the data-plane addresses (a HELLO, INIT, READY exchange); the engine connects to them and reports its context length and data-parallel size.
- SMG marks the worker ready as soon as the handshake completes.
- Requests go to the engine as token IDs. Outputs come back as token IDs, with the engine's scheduler load piggybacked on the output batches.
The engine never sees text, so SMG also does the work that the engine's API server or gRPC servicer would otherwise do:
| Task | What SMG does |
|---|---|
| Tokenization, chat templates, reasoning and tool-call parsing | Same as in gRPC mode |
| String stops | Sends a stop string that encodes to a single token as a stop token ID, and matches longer stop strings on the decoded text and trims them. For Harmony (gpt-oss) models, stops are matched on the parsed channel text |
| EOS (vLLM) | Attaches the model's EOS token IDs to every request: from config.json and generation_config.json when --model-path is a local directory, otherwise from the tokenizer. TokenSpeed and SGLang stop at EOS on their own |
Default max_tokens (vLLM) |
Uses the context length the engine reported at handshake minus the prompt length |
n > 1 |
Sends n single-sample requests and merges the results; an explicit seed becomes seed + i for sample i, so samples differ but stay reproducible (vLLM and TokenSpeed; the SGLang wire carries no seed) |
| Data-parallel ranks | Picks the least-loaded engine inside a grouped worker (see Data-Parallel Engines) |
Engines on Another Host¶
For a worker URL tcp://<host>:<port>, SMG binds the handshake at that address itself and the two data-plane sockets on ephemeral ports of the same host; the handshake hands the engine the data-plane addresses, so <host> must be an address of the gateway host that the engine can reach (a wildcard such as 0.0.0.0 is rejected at registration). Start the engine with --data-parallel-address <host> and --data-parallel-rpc-port <port>. No socket files are created, no handshake port is derived, and the worker takes no zmq_handshake_address — its URL is the handshake address. The sockets carry no authentication, so use tcp:// workers on a trusted network only. The rest of the ZMQ contract — handshake flow, lifecycle, grouped workers, features and limits — is the same on both transports.
Supported Engines¶
| Engine | Runtime | Headless engine command | Notes |
|---|---|---|---|
| vLLM | vllm |
vllm serve <model> --headless ... |
Assumed when a ZMQ worker declares no runtime |
| TokenSpeed | tokenspeed |
python -m tokenspeed.cli serve --headless ... |
Must be declared. Needs --grammar-backend for structured outputs and --enable-output-logprobs for logprobs |
| SGLang | sglang |
python -m smg_grpc_servicer.sglang.headless ... |
Must be declared. Needs smg-grpc-servicer[sglang] (SGLang 0.5.21 or newer): its plugin dials SMG from inside the scheduler, and SGLang itself is unchanged |
All of these engines use the same handshake, so SMG can't tell which one dialed in. Declare the runtime with --backend (startup workers) or runtime_type (worker API). A ZMQ worker without a runtime is treated as vLLM, and SMG logs runtime_type unspecified for ZMQ worker; defaulting to vLLM EngineCore.
These are the only three ZMQ runtimes: smg serve accepts --connection-mode zmq only with --backend vllm, --backend tokenspeed, or --backend sglang, and the gateway rejects any other runtime with ZMQ worker ... has unsupported runtime ...: only vllm, tokenspeed and sglang are supported over the ZMQ direct backend. TensorRT-LLM and MLX can't use ZMQ; connect those engines over gRPC.
SMG doesn't check engine versions at connect time, and its decoders accept fields that newer engine releases append. SMG's CI runs the ZMQ suites against vLLM 0.31.0, SGLang 0.5.21, and a pinned TokenSpeed revision.
Quick Start with smg serve¶
smg serve --connection-mode zmq starts headless engines and the gateway together:
smg serve \
--backend vllm \
--connection-mode zmq \
--model meta-llama/Llama-3.1-8B-Instruct \
--router-model-path meta-llama/Llama-3.1-8B-Instruct \
--host 0.0.0.0 \
--port 30000smg serve \
--backend tokenspeed \
--connection-mode zmq \
--model meta-llama/Llama-3.1-8B-Instruct \
--router-model-path meta-llama/Llama-3.1-8B-Instruct \
--host 0.0.0.0 \
--port 30000smg serve \
--backend sglang \
--connection-mode zmq \
--model-path meta-llama/Llama-3.1-8B-Instruct \
--host 0.0.0.0 \
--port 30000What smg serve sets up in ZMQ mode:
| Item | Behavior |
|---|---|
| Worker URL | ipc://<socket dir>/engine-<worker port> per replica. The socket directory is $SMG_ZMQ_SOCKET_DIR, default /tmp/smg-zmq-<uid> |
| Engine command | A headless engine that dials the handshake port derived from its worker URL (commands below) |
| TokenSpeed defaults | Adds --grammar-backend xgrammar and --enable-output-logprobs unless you pass your own value for either flag |
| Engine health wait | Skipped. The gateway starts at once, and /readiness returns 503 until a worker has completed its handshake and the tokenizer has loaded |
| Gateway runtime | --backend is forwarded so the gateway speaks the engine's wire protocol |
| Replicas | smg serve's own --data-parallel-size N (before any --) starts N independent single-engine workers, each with its own socket path and GPU slice. The launch stops early if two replicas' paths derive the same handshake port (ZMQ handshake port collision on ...); change --worker-base-port |
The launcher builds these engine commands. If you pass a flag the launcher sets, your copy is dropped, except for the two TokenSpeed defaults above, where your value is used:
# vLLM
python -m vllm.entrypoints.cli.main serve <model> --headless \
--data-parallel-size 1 --data-parallel-size-local 1 \
--data-parallel-address 127.0.0.1 --data-parallel-rpc-port <handshake port>
# TokenSpeed
python -m tokenspeed.cli serve --headless --model <model> \
--port <worker port> --dist-init-addr 127.0.0.1:<store port> \
--data-parallel-address 127.0.0.1 --data-parallel-rpc-port <handshake port> \
--zmq-engine-index 0 --grammar-backend xgrammar --enable-output-logprobs
# SGLang
python -m smg_grpc_servicer.sglang.headless \
--zmq-handshake-address tcp://127.0.0.1:<handshake port> --zmq-engine-index 0 \
--model-path <model> --port <worker port>To put several data-parallel TokenSpeed engines behind one worker, pass the engine-level --data-parallel-size N after a -- separator on the smg serve command line: the flag passes through to the headless engine command, and the gateway waits for N engines on the worker's one socket set. You can also start the engines yourself and register them as a grouped worker with smg launch (see Data-Parallel Engines).
The -- passthrough works only with --backend tokenspeed. A vLLM or SGLang smg serve command exits with an unrecognized arguments error on anything after a --, and a --data-parallel-size (or --dp-size) before any -- is always smg serve's own replica count, so neither engine takes a grouped launch through smg serve: a grouped vLLM worker is set up with smg launch only, and the SGLang ZMQ wire has no data parallelism at all (use gRPC for SGLang data parallelism).
Manual Setup with smg launch¶
Socket Path¶
Pick a path for each ipc:// worker, for example ipc:///tmp/smg-zmq/engine-0. SMG creates the parent directory with mode 0700 if it doesn't exist. An existing directory must be a real directory (not a symlink) owned by the user SMG runs as, or SMG refuses to bind, so don't place sockets directly in a shared directory such as /tmp. The engine must be able to reach the directory: run it as the same user, or create the directory yourself with the access the engine needs.
Before binding, SMG deletes any socket files already at <path>-in.sock and <path>-out.sock, treating them as leftovers from an earlier gateway run. Give each gateway process its own socket paths: a second gateway configured with the same worker URL would delete the first one's live sockets.
Handshake Address¶
SMG binds the handshake on 127.0.0.1 at a port in 20000–29999, computed from the path after ipc:// as 20000 + FNV-1a-64(path) mod 10000. Compute it before starting the engine:
python3 - <<'EOF'
path = "/tmp/smg-zmq/engine-0" # the worker URL without "ipc://"
h = 0xCBF29CE484222325
for b in path.encode():
h = ((h ^ b) * 0x100000001B3) & 0xFFFFFFFFFFFFFFFF
print(20000 + h % 10000)
EOFFor this path it prints 22670. SMG also logs the address when it binds:
Binding ZMQ client for worker ipc:///tmp/smg-zmq/engine-0 (handshake=tcp://127.0.0.1:22670, engines=Declared)engines= shows how many engines the handshake waits for: Exactly(N) for a grouped worker with an explicit engine count, or Declared when the gateway takes the count from the first engine to dial (see Data-Parallel Engines).
Two paths can hash to the same port; SMG rejects the second worker at registration. Workers added through the worker API can set a fixed zmq_handshake_address instead. A tcp:// worker derives nothing: its URL is the handshake address the engine dials (see Engines on Another Host).
Start the Engine¶
vllm serve meta-llama/Llama-3.1-8B-Instruct \
--headless \
--data-parallel-size 1 \
--data-parallel-size-local 1 \
--data-parallel-address 127.0.0.1 \
--data-parallel-rpc-port 22670python -m tokenspeed.cli serve \
--headless \
--model meta-llama/Llama-3.1-8B-Instruct \
--data-parallel-address 127.0.0.1 \
--data-parallel-rpc-port 22670 \
--zmq-engine-index 0 \
--grammar-backend xgrammar \
--enable-output-logprobsTokenSpeed's grammar backend defaults to none, and then the engine ignores structured-output constraints: a tool_choice: "required" or json_schema request comes back as free text. Output logprobs are off by default. When several TokenSpeed engines share a host, give each its own --port; TokenSpeed derives its internal control ports from it.
python -m smg_grpc_servicer.sglang.headless \
--zmq-handshake-address tcp://127.0.0.1:22670 \
--zmq-engine-index 0 \
--model-path meta-llama/Llama-3.1-8B-Instruct \
--port 31000The launcher comes with pip install "smg-grpc-servicer[sglang]" and starts SGLang's own scheduler ranks with no HTTP server, tokenizer manager, or detokenizer; the package's plugin dials SMG's handshake from inside the scheduler once the model has loaded. Every flag other than the two --zmq-* options is an ordinary SGLang server arg. When several SGLang engines share a host, give each its own --port; SGLang derives its internal control ports from it.
Add your usual engine flags, such as --tensor-parallel-size.
Start the Gateway¶
smg launch \
--worker-urls ipc:///tmp/smg-zmq/engine-0 \
--backend vllm \
--model-path meta-llama/Llama-3.1-8B-Instruct \
--host 0.0.0.0 \
--port 30000Use --backend tokenspeed or --backend sglang to match the engine. --model-path is required: it names the model and loads the tokenizer.
SMG binds the sockets as soon as it registers the worker, then waits up to --zmq-startup-timeout-secs (default 3600 seconds; with smg serve, --router-zmq-startup-timeout-secs) for each handshake message. SMG sees nothing from the engine except those messages, so this one bound must cover everything the engine does between its HELLO and its READY — loading weights, profiling the KV cache, capturing graphs. --worker-startup-timeout-secs bounds registration, not the handshake. If the handshake fails, the next health probe starts a new attempt. An engine that dialed in during an earlier attempt still connects: the new attempt accepts its READY.
To serve several engines, list one ipc:// URL per engine. They are independent workers balanced by --policy.
Register Workers at Runtime¶
Add a ZMQ worker to a running gateway with POST /workers. The ipc:// and tcp:// schemes select ZMQ, so connection_mode can be left out:
curl -X POST http://localhost:30000/workers \
-H "Content-Type: application/json" \
-d '{
"url": "ipc:///tmp/smg-zmq/engine-1",
"runtime_type": "tokenspeed"
}'ZMQ-related worker fields:
| Field | Description |
|---|---|
url |
ipc://<path> (the path is required), or tcp://<host>:<port> for an engine on another host (see Engines on Another Host) |
runtime_type |
vllm, tokenspeed, or sglang (runtime is accepted as an alias). Unset means vLLM |
zmq_handshake_address |
A tcp:// address to bind for the handshake instead of the one derived from an ipc:// path. Rejected on non-ZMQ workers, and on tcp:// workers, whose URL is already the handshake address |
dp_size |
Number of engines in a grouped worker; if omitted, the worker waits for the group size declared by the first engine to dial (one for an engine that declares none). Leave dp_rank unset |
labels.model_path |
Model ID and tokenizer source, for gateways started without --model-path |
zmq_handshake_address pairs a worker with an engine that dials a fixed address. A TokenSpeed engine started without --data-parallel-rpc-port dials tcp://127.0.0.1:30500, so this worker pairs with it:
curl -X POST http://localhost:30000/workers \
-H "Content-Type: application/json" \
-d '{
"url": "ipc:///tmp/smg-zmq/tokenspeed-0",
"runtime_type": "tokenspeed",
"zmq_handshake_address": "tcp://127.0.0.1:30500"
}'Registration runs in the background (the API answers 202 Accepted), so a rejected worker shows up only in the gateway log; see Troubleshooting. For the full request schema, see Worker Management.
Data-Parallel Engines¶
There are two ways to run data parallelism over ZMQ:
- Replicas: several single-engine workers, each with its own socket path. The gateway's routing policy spreads requests across them.
smg serve --data-parallel-size Nsets this up. - Grouped worker: one worker URL and one socket set, with
Ndata-parallel engines behind it. The gateway sees one worker and picks the engine (rank) for each request itself. For TokenSpeed,smg servesets this up when an engine-level--data-parallel-size Nfollows--on its command line; a vLLM group is set up only manually withsmg launch. Configure a Grouped Worker shows the manual setup for both.
Replicas work with every ZMQ engine. Grouped workers are a vLLM and TokenSpeed setup: data parallelism over the SGLang ZMQ wire isn't supported yet (use gRPC for SGLang data parallelism).
Configure a Grouped Worker¶
Start the engine group with its data-parallel size. A TokenSpeed attention-DP group declares its size when it dials in, so --zmq-engine-count can stay unset and the gateway waits for the whole declared group. A vLLM engine declares no group size, so a vLLM group needs the explicit count:
vllm serve meta-llama/Llama-3.1-8B-Instruct \
--headless \
--data-parallel-size 2 \
--data-parallel-size-local 2 \
--data-parallel-address 127.0.0.1 \
--data-parallel-rpc-port 22670smg launch
--worker-urls ipc:///tmp/smg-zmq/engine-0
--backend vllm
--model-path meta-llama/Llama-3.1-8B-Instruct
--zmq-engine-count 2
python -m tokenspeed.cli serve
--headless
--model meta-llama/Llama-3.1-8B-Instruct
--data-parallel-size 2
--dist-init-addr 127.0.0.1:31233
--data-parallel-address 127.0.0.1
--data-parallel-rpc-port 22670
--zmq-engine-index 0
--grammar-backend xgrammar
--enable-output-logprobs
smg launch
--worker-urls ipc:///tmp/smg-zmq/engine-0
--backend tokenspeed
--model-path meta-llama/Llama-3.1-8B-Instruct
--zmq-engine-count 2
Each TokenSpeed rank dials in with its own identity (--zmq-engine-index plus its rank). TokenSpeed requires an explicit --dist-init-addr when --data-parallel-size is above 1; pick a free port outside the 20000–29999 handshake range.
--zmq-engine-countapplies to every ZMQ (ipc://ortcp://) URL in--worker-urls; an explicit value must be a positive integer, and--zmq-engine-count 1behaves like leaving the flag unset. For a worker added through the API, set"dp_size"on the worker instead; adp_sizeof 1 likewise behaves like leaving the field out.- Left unset, the count settles on what the first engine to dial declares, and an engine that declares no group size counts as one. A vLLM group with the count unset therefore settles at a single engine, and the rest of its ranks dial in as extras.
- The count the gateway waits for must match the number of engines that dial in. An extra engine can fail the handshake with
duplicate HELLO ... after INIT phase; a missing one leaves the handshake waiting until it times out. - An engine may declare a group size that differs from the count the handshake awaits, whether that count was set explicitly or settled by the first engine. The engine still connects; the gateway logs
engine declares a group of a different size than this handshake awaitsand keeps the count it was waiting for. - The engine process needs
Ntimes its tensor-parallel size in GPUs. - Grouped workers can't be combined with
--dp-aware, which tries to expand the group into one worker per rank and fails withcannot be dp-aware expanded. Single-engine ZMQ workers register normally under--dp-aware.
Rank Selection¶
Output batches carry the producing rank's scheduler load: running requests, waiting requests, and KV-cache usage (vLLM's scheduler stats, or TokenSpeed's load snapshot). For each request, SMG scores every rank in the group and sends the request to the lowest score:
score = max(requests SMG has in flight on the rank, running + waiting)
+ waiting × 6 × max(0, kv_cache_usage − 0.5)- The in-flight floor keeps a stale report from making a busy rank look idle, so a burst of requests between reports spreads across ranks.
- Engines report load only while they produce output. When a rank's last in-flight request finishes, SMG resets that rank's running and waiting counts to zero and keeps its last reported KV usage.
- Ties rotate across ranks. A TokenSpeed build that sends no load snapshot is scored on SMG's in-flight counts alone.
- vLLM MoE models run their data-parallel ranks in lockstep and pause the whole group when idle. SMG sends the wake-up that vLLM's data-parallel coordinator process would otherwise send, so no coordinator runs. Ranks of dense models run independently.
Rank selection happens inside the connection. Routing policies see the group as one worker, and a plain request is never pinned to a rank. A disaggregated TokenSpeed request follows its KV rendezvous instead: a prefill group places it on the rank its rendezvous room selects, and a decode group takes it on the matching rank, so the KV transfer lands where the decode side looks it up.
Worker Lifecycle¶
| Stage | What happens |
|---|---|
| Registered | The worker starts as pending. SMG binds its sockets and starts the handshake right away |
| Connected | SMG promotes the worker to ready the moment the handshake completes, without waiting for the health-check success threshold |
| Serving | Health probes read a local liveness flag; the engine has no health RPC. /readiness stays at 503 with tokenizer not yet registered until the model's tokenizer has loaded |
| Engine lost | SMG marks the connection dead when the engine signals that it died (vLLM does so on a fatal error, TokenSpeed also on shutdown, and SGLang whenever its scheduler's event loop exits), the socket fails, three outputs in a row can't be decoded, the engine leaves a frame unaccepted on its input for 300 seconds, or in-flight requests spend a combined 300 seconds — past the prefill allowance for the prompt tokens the engine holds — without any output from the engine. In-flight requests fail. The next health probe drops the dead connection, and a later probe binds the sockets again so a restarted engine can reconnect |
| Removed | DELETE /workers/{worker_id} drains the worker, then removes it; a handshake still in progress is cancelled and its sockets are released |
- Health checks stay on for ZMQ workers even with
--disable-health-checkor a per-workerdisable_health_check, because the probe is what reconnects a restarted engine. SMG logsIgnoring disabled health checks for ZMQ worker .... - With
--remove-unhealthy-workers, a ZMQ worker whose engine stays down is removed and nothing re-adds it. The flag is off by default unless service discovery is enabled; leave it off for ZMQ workers so they rejoin when their engine restarts. - An engine that stops reading its input (a paused or frozen engine core) fails requests, not the connection. A request the engine does not accept within 10 seconds fails with a retryable
503, and a retry can select a different worker (see Retries); the requests the engine already holds keep streaming. The unaccepted frame is still delivered when the engine reads again, followed by an abort, so the engine drops the failed request instead of generating output nobody is waiting for. While the input is blocked, 256 requests or 32 MiB of request payload may wait behind it; a request that would take the waiting requests past either bound gets the503immediately instead of after the 10-second wait. - The 10-second and 300-second windows are fixed, and so is the prefill allowance's rate: the prompt tokens of requests the engine has accepted but not yet produced output for extend the 300-second bound at 2,000 tokens per second — 500 extra seconds per million pending prompt tokens, and none when no prefill is pending. The TokenSpeed and SGLang wires carry token batches only, as does vLLM's with
--disable-log-stats: such a wire says nothing for a prompt until its first token, so a batch of long prompts is silent for as long as its prefill takes. The allowance keeps that silence from counting, so a single request whose prefill outlasts 300 seconds is not failed as a dead engine. A vLLM engine with stats logging on reports every scheduler step, so it is never silent through a prefill. The rate is a bound on how slowly a healthy engine may prefill, not an estimate of how fast it does, so the verdict on a genuinely dead engine comes later in proportion to the prompt it was handed. - Only silence past the allowance counts, and it accumulates across requests: the in-flight time since the engine's last output, beyond its pending prefill work, adds up over consecutive busy periods, and only an output of any kind — or 300 seconds with nothing in flight — resets the count. A request cancelled before its first output leaves its prefill allowance in place until it has passed: the engine may still be mid-prefill on the cancelled prompt.
- Updating a ZMQ worker's properties, such as labels or priority, keeps its live connection.
Features and Limits¶
Supported over ZMQ¶
| Feature | vLLM | TokenSpeed | SGLang |
|---|---|---|---|
| Chat Completions, Completions, Responses, and Messages, including streaming | Yes | Yes | Yes |
| Tool-call and reasoning parsing | Yes | Yes | Yes |
Structured outputs (response_format, constrained tool_choice, regex, grammar) |
Yes | Yes, with --grammar-backend set on the engine |
Yes |
| String stops and EOS | Yes | Yes | Yes |
| Harmony (gpt-oss) stop strings | Yes | Yes | Yes |
| Multimodal inputs | Yes, one modality per request | Yes; SMG computes the Qwen3.5 lineage's MRoPE position table itself and refuses other MRoPE models | Images; video and audio are refused |
| Output logprobs | Yes, including top_logprobs |
Sampled-token logprobs, with --enable-output-logprobs; top_logprobs above 1 needs an engine whose handshake declares the top-k columns |
Yes, including top_logprobs |
| Prompt logprobs | Forwarded to the engine, but no API field requests them | With an engine whose handshake declares the prompt-logprob columns; token_ids_logprob is rejected |
Yes, requested through /generate's logprob_start_len; token_ids_logprob is rejected |
n > 1 and sampling seed |
Yes | Yes | n > 1 yes; the SGLang wire carries no seed |
Limits¶
| Limit | What you see |
|---|---|
No authentication on tcp:// workers |
ipc:// sockets are local paths with a loopback handshake; a tcp:// worker accepts an engine from another host, but its sockets carry no authentication, so keep it on a trusted network. ZMQ workers are never synced to high-availability mesh peers; each belongs to the gateway that registered it |
| Prefill, decode, and encode workers are TokenSpeed-only | A vLLM or SGLang ZMQ worker of a disaggregated role is rejected with ZMQ worker ... cannot serve worker type .... A TokenSpeed ZMQ worker serves every role — its engine must declare the matching role at the handshake, and a prefill or encode engine also its bootstrap address — and PD and EPD routing pair it with workers of the same runtime |
| No KV-event stream | cache_aware tracks ZMQ workers with its approximate prefix tree, and SMG logs ... the ZMQ transport has no KV-event stream ... |
| No embeddings | 501 with code unsupported_backend: ZMQ backend does not support embeddings yet |
| Engine admin needs a control-call wire | POST /flush_cache and profiling reach an SGLang ZMQ worker, and a TokenSpeed one whose every rank declared the control reply at its handshake; the flush skips other ZMQ workers, and profiling them fails with ... is not supported over this engine's ZMQ wire. RL control calls such as weight updates ride a control-wire TokenSpeed worker's own connection; a vLLM ZMQ engine carries no control calls |
| No tokenizer from the engine | The gateway loads it from --model-path or --tokenizer-path (or the worker's model_path/tokenizer_path labels) |
| No dp-aware rank pinning | See Rank Selection |
| vLLM multimodal | Mixed image and video in one request: 400, the vLLM ZMQ backend takes one modality per request .... Worker-side media processing (media references) needs a gRPC vLLM worker: 400, multimodal_not_supported |
| TokenSpeed logprobs | top_logprobs above 1 or prompt logprobs on an engine that declares no column for them, or any token_ids_logprob: 400, ... not supported over the TokenSpeed ZMQ backend |
| TokenSpeed MRoPE models outside the Qwen3.5 lineage | 400: MRoPE position tensors of this model are not derivable over the TokenSpeed ZMQ wire (the Qwen3.5 lineage's are); use the gRPC transport for this model |
| SGLang non-image media | 400: multimodal modality ... is not carried over the SGLang ZMQ backend; the scheduler takes preprocessed image tensors on this wire |
| SGLang selected-token logprobs | token_ids_logprob on a ZMQ worker: 501, selected-token logprobs require the SGLang gRPC servicer — use a gRPC SGLang worker to request them |
| SGLang data parallelism | The wire has no DP yet; run SGLang replicas, or use gRPC for engine-level DP (see Data-Parallel Engines) |
Verify¶
# Ready once the handshake has completed and the tokenizer has loaded
curl http://localhost:30000/readiness
# Transport, runtime, and status of each worker
curl -s http://localhost:30000/workers | jq '.workers[] | {url, connection_mode, runtime_type, status}'Expected worker entry:
{
"url": "ipc:///tmp/smg-zmq/engine-0",
"connection_mode": "zmq",
"runtime_type": "vllm",
"status": "ready"
}Send a request:
curl http://localhost:30000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "meta-llama/Llama-3.1-8B-Instruct",
"messages": [{"role": "user", "content": "Hello!"}]
}'Gateway metrics label ZMQ workers with connection_mode="zmq":
curl -s http://localhost:29000/metrics | grep 'connection_mode="zmq"'Troubleshooting¶
Worker stays pending
Symptoms: /readiness returns 503 with insufficient healthy workers, and /workers shows "status": "pending".
Solutions:
- Check that the engine dials the address SMG bound: compare the engine's
--data-parallel-rpc-portand--data-parallel-addresswith thehandshake=value in SMG'sBinding ZMQ client for worker ...log line. With no engine, once--zmq-startup-timeout-secsexpires, SMG logsZMQ backend handshake failed for ...: ... startup handshake timed out while waiting for HELLO after 3600s; will retry on the next health probe. - For a grouped worker, check that an explicit
--zmq-engine-count(ordp_size) matches the engine's--data-parallel-size. With no explicit count, the gateway waits for the group size declared by the first engine to dial, so the worker stays pending until that many engines have dialed in. - A TokenSpeed worker registered as prefill, decode, or encode needs an engine that declares the same role — and, for prefill and encode, its bootstrap address — at its handshake; a mismatch fails the handshake with the engine's declared role named in the gateway log.
- A worker added through the API to a gateway started with
--disable-health-checkand no ZMQ workers at startup is never promoted. Restart the gateway without that flag.
Socket bind errors or stale socket files
Symptoms: The worker stays pending, and the gateway log shows ZMQ backend handshake failed for ... followed by a socket or directory error.
Solutions:
SMG deletes leftover socket files at <path>-in.sock and <path>-out.sock before binding, so a crashed gateway doesn't block a restart. Other causes:
ipc socket path ... exists but is not a socket; refusing to unlink: a regular file sits at the socket path. Move it away.ipc socket dir ... is owned by uid ...oripc socket dir ... exists but is not a directory: use a directory owned by SMG's user, not/tmpitself or a symlink.- Another process holds the handshake port. Choose a different socket path, or register the worker with
zmq_handshake_address. - The socket path is too long. Unix socket paths have a small length limit (108 bytes on Linux), so keep
<path>-out.sockshort.
Registration rejected
Symptoms: The worker never appears in /workers, and the gateway log shows Failed job: type=AddWorker, worker=ipc://... with the reason.
Solutions:
has no model identity: start SMG with--model-path(withsmg serve,--router-model-path), or set amodel_pathlabel on the worker.has unsupported runtime: usevllm,tokenspeed, orsglang.would bind handshake address ..., already claimed by worker ...: two socket paths hash to the same port. Rename one, or setzmq_handshake_address.zmq_handshake_address must be a tcp:// address,ZMQ worker URL must be ipc://<path> or tcp://<host>:<port>,names a wildcard host, ortakes no separate handshake address: fix the address or URL — atcp://worker must name a reachable address of the gateway host and takes no handshake override.cannot serve worker type: only TokenSpeed engines serve prefill, decode, or encode over ZMQ; vLLM and SGLang ZMQ workers must beregular.cannot be dp-aware expanded: drop--dp-aware, or use single-engine workers.
TokenSpeed or SGLang requests fail or hang
Symptoms: The worker connects, but requests error or time out.
Solutions:
- Look for
runtime_type unspecified for ZMQ worker; defaulting to vLLM EngineCorein the gateway log. A TokenSpeed or SGLang worker must be declared with--backend tokenspeed/--backend sglangor an explicitruntime_type, or SMG speaks the vLLM wire protocol to it. - If structured outputs come back unconstrained or logprobs are missing from a TokenSpeed engine, start it with
--grammar-backend xgrammarand--enable-output-logprobs.
Readiness reports tokenizer not yet registered
Symptoms: /readiness returns 503 with tokenizer not yet registered after the worker is ready.
Solutions:
- Wait for the tokenizer download from
--model-pathto finish, and check the gateway log for load errors. - Set
--tokenizer-pathif the tokenizer lives somewhere other than the model path.
Next Steps¶
- gRPC Workers — Engines behind a gRPC servicer: KV-event routing, embeddings, and disaggregation on vLLM, SGLang, and TokenSpeed
- Multiple Workers — Mix worker types and register workers at runtime
- Load Balancing — Policies for spreading traffic across ZMQ replicas
- Architecture Overview — Where the ZMQ path sits in the gateway
- Configuration Reference —
--worker-urlsand related worker flags - Monitoring — Gateway and engine-load metrics