ZMQ Direct Workers

When SMG and the inference engine share a host, SMG can connect straight to the engine core over ZMQ. No engine HTTP server or gRPC servicer sits in the path: the engine runs headless (scheduling and model execution only), SMG does everything else — tokenization, chat templates, parsing, stop handling, and routing — and the two exchange token IDs over local ipc:// sockets. A worker URL of tcp://<host>:<port> runs the same wire over TCP for an engine on another host (see Engines on Another Host).


When to Use ZMQ

HTTP gRPC ZMQ
Worker URL http://host:port grpc://host:port ipc:///path or tcp://host:port
Engine-side process Engine's OpenAI-compatible server Engine with a gRPC servicer Headless engine core only
Tokenization, chat templates, parsers Engine Gateway Gateway
Engine location Any reachable host Any reachable host Same host as SMG (ipc://), or any host that can dial the gateway (tcp://)
Prefill-decode disaggregation Yes Yes TokenSpeed only
KV-event stream for cache-aware routing No Yes No

Compared with gRPC, the direct path drops the servicer process, a serialization round-trip, and a context switch per message. Use it when the gateway and engine run on one machine; a tcp:// worker URL extends the same wire to an engine on another host over a trusted network. Use gRPC when you need KV-event routing, embeddings, disaggregation on vLLM or SGLang, or engine admin operations on vLLM (see Limits).


How It Works

For a worker URL ipc://<path>:

  1. SMG binds two data-plane sockets, ipc://<path>-in.sock (requests) and ipc://<path>-out.sock (outputs), plus a handshake listener at tcp://127.0.0.1:<port>, where the port is derived from <path> (see Handshake Address).
  2. The headless engine dials the handshake address. SMG answers with the data-plane addresses (a HELLO, INIT, READY exchange); the engine connects to them and reports its context length and data-parallel size.
  3. SMG marks the worker ready as soon as the handshake completes.
  4. Requests go to the engine as token IDs. Outputs come back as token IDs, with the engine's scheduler load piggybacked on the output batches.

The engine never sees text, so SMG also does the work that the engine's API server or gRPC servicer would otherwise do:

Task What SMG does
Tokenization, chat templates, reasoning and tool-call parsing Same as in gRPC mode
String stops Sends a stop string that encodes to a single token as a stop token ID, and matches longer stop strings on the decoded text and trims them. For Harmony (gpt-oss) models, stops are matched on the parsed channel text
EOS (vLLM) Attaches the model's EOS token IDs to every request: from config.json and generation_config.json when --model-path is a local directory, otherwise from the tokenizer. TokenSpeed and SGLang stop at EOS on their own
Default max_tokens (vLLM) Uses the context length the engine reported at handshake minus the prompt length
n > 1 Sends n single-sample requests and merges the results; an explicit seed becomes seed + i for sample i, so samples differ but stay reproducible (vLLM and TokenSpeed; the SGLang wire carries no seed)
Data-parallel ranks Picks the least-loaded engine inside a grouped worker (see Data-Parallel Engines)

Engines on Another Host

For a worker URL tcp://<host>:<port>, SMG binds the handshake at that address itself and the two data-plane sockets on ephemeral ports of the same host; the handshake hands the engine the data-plane addresses, so <host> must be an address of the gateway host that the engine can reach (a wildcard such as 0.0.0.0 is rejected at registration). Start the engine with --data-parallel-address <host> and --data-parallel-rpc-port <port>. No socket files are created, no handshake port is derived, and the worker takes no zmq_handshake_address — its URL is the handshake address. The sockets carry no authentication, so use tcp:// workers on a trusted network only. The rest of the ZMQ contract — handshake flow, lifecycle, grouped workers, features and limits — is the same on both transports.


Supported Engines

Engine Runtime Headless engine command Notes
vLLM vllm vllm serve <model> --headless ... Assumed when a ZMQ worker declares no runtime
TokenSpeed tokenspeed python -m tokenspeed.cli serve --headless ... Must be declared. Needs --grammar-backend for structured outputs and --enable-output-logprobs for logprobs
SGLang sglang python -m smg_grpc_servicer.sglang.headless ... Must be declared. Needs smg-grpc-servicer[sglang] (SGLang 0.5.21 or newer): its plugin dials SMG from inside the scheduler, and SGLang itself is unchanged

All of these engines use the same handshake, so SMG can't tell which one dialed in. Declare the runtime with --backend (startup workers) or runtime_type (worker API). A ZMQ worker without a runtime is treated as vLLM, and SMG logs runtime_type unspecified for ZMQ worker; defaulting to vLLM EngineCore.

These are the only three ZMQ runtimes: smg serve accepts --connection-mode zmq only with --backend vllm, --backend tokenspeed, or --backend sglang, and the gateway rejects any other runtime with ZMQ worker ... has unsupported runtime ...: only vllm, tokenspeed and sglang are supported over the ZMQ direct backend. TensorRT-LLM and MLX can't use ZMQ; connect those engines over gRPC.

SMG doesn't check engine versions at connect time, and its decoders accept fields that newer engine releases append. SMG's CI runs the ZMQ suites against vLLM 0.31.0, SGLang 0.5.21, and a pinned TokenSpeed revision.


Quick Start with smg serve

smg serve --connection-mode zmq starts headless engines and the gateway together:

smg serve \
  --backend vllm \
  --connection-mode zmq \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --router-model-path meta-llama/Llama-3.1-8B-Instruct \
  --host 0.0.0.0 \
  --port 30000
smg serve \
  --backend tokenspeed \
  --connection-mode zmq \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --router-model-path meta-llama/Llama-3.1-8B-Instruct \
  --host 0.0.0.0 \
  --port 30000
smg serve \
  --backend sglang \
  --connection-mode zmq \
  --model-path meta-llama/Llama-3.1-8B-Instruct \
  --host 0.0.0.0 \
  --port 30000

What smg serve sets up in ZMQ mode:

Item Behavior
Worker URL ipc://<socket dir>/engine-<worker port> per replica. The socket directory is $SMG_ZMQ_SOCKET_DIR, default /tmp/smg-zmq-<uid>
Engine command A headless engine that dials the handshake port derived from its worker URL (commands below)
TokenSpeed defaults Adds --grammar-backend xgrammar and --enable-output-logprobs unless you pass your own value for either flag
Engine health wait Skipped. The gateway starts at once, and /readiness returns 503 until a worker has completed its handshake and the tokenizer has loaded
Gateway runtime --backend is forwarded so the gateway speaks the engine's wire protocol
Replicas smg serve's own --data-parallel-size N (before any --) starts N independent single-engine workers, each with its own socket path and GPU slice. The launch stops early if two replicas' paths derive the same handshake port (ZMQ handshake port collision on ...); change --worker-base-port

The launcher builds these engine commands. If you pass a flag the launcher sets, your copy is dropped, except for the two TokenSpeed defaults above, where your value is used:

# vLLM
python -m vllm.entrypoints.cli.main serve <model> --headless \
  --data-parallel-size 1 --data-parallel-size-local 1 \
  --data-parallel-address 127.0.0.1 --data-parallel-rpc-port <handshake port>

# TokenSpeed
python -m tokenspeed.cli serve --headless --model <model> \
  --port <worker port> --dist-init-addr 127.0.0.1:<store port> \
  --data-parallel-address 127.0.0.1 --data-parallel-rpc-port <handshake port> \
  --zmq-engine-index 0 --grammar-backend xgrammar --enable-output-logprobs

# SGLang
python -m smg_grpc_servicer.sglang.headless \
  --zmq-handshake-address tcp://127.0.0.1:<handshake port> --zmq-engine-index 0 \
  --model-path <model> --port <worker port>

To put several data-parallel TokenSpeed engines behind one worker, pass the engine-level --data-parallel-size N after a -- separator on the smg serve command line: the flag passes through to the headless engine command, and the gateway waits for N engines on the worker's one socket set. You can also start the engines yourself and register them as a grouped worker with smg launch (see Data-Parallel Engines).

The -- passthrough works only with --backend tokenspeed. A vLLM or SGLang smg serve command exits with an unrecognized arguments error on anything after a --, and a --data-parallel-size (or --dp-size) before any -- is always smg serve's own replica count, so neither engine takes a grouped launch through smg serve: a grouped vLLM worker is set up with smg launch only, and the SGLang ZMQ wire has no data parallelism at all (use gRPC for SGLang data parallelism).


Manual Setup with smg launch

Socket Path

Pick a path for each ipc:// worker, for example ipc:///tmp/smg-zmq/engine-0. SMG creates the parent directory with mode 0700 if it doesn't exist. An existing directory must be a real directory (not a symlink) owned by the user SMG runs as, or SMG refuses to bind, so don't place sockets directly in a shared directory such as /tmp. The engine must be able to reach the directory: run it as the same user, or create the directory yourself with the access the engine needs.

Before binding, SMG deletes any socket files already at <path>-in.sock and <path>-out.sock, treating them as leftovers from an earlier gateway run. Give each gateway process its own socket paths: a second gateway configured with the same worker URL would delete the first one's live sockets.

Handshake Address

SMG binds the handshake on 127.0.0.1 at a port in 20000–29999, computed from the path after ipc:// as 20000 + FNV-1a-64(path) mod 10000. Compute it before starting the engine:

python3 - <<'EOF'
path = "/tmp/smg-zmq/engine-0"  # the worker URL without "ipc://"
h = 0xCBF29CE484222325
for b in path.encode():
    h = ((h ^ b) * 0x100000001B3) & 0xFFFFFFFFFFFFFFFF
print(20000 + h % 10000)
EOF

For this path it prints 22670. SMG also logs the address when it binds:

Binding ZMQ client for worker ipc:///tmp/smg-zmq/engine-0 (handshake=tcp://127.0.0.1:22670, engines=Declared)

engines= shows how many engines the handshake waits for: Exactly(N) for a grouped worker with an explicit engine count, or Declared when the gateway takes the count from the first engine to dial (see Data-Parallel Engines).

Two paths can hash to the same port; SMG rejects the second worker at registration. Workers added through the worker API can set a fixed zmq_handshake_address instead. A tcp:// worker derives nothing: its URL is the handshake address the engine dials (see Engines on Another Host).

Start the Engine

vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --headless \
  --data-parallel-size 1 \
  --data-parallel-size-local 1 \
  --data-parallel-address 127.0.0.1 \
  --data-parallel-rpc-port 22670
python -m tokenspeed.cli serve \
  --headless \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --data-parallel-address 127.0.0.1 \
  --data-parallel-rpc-port 22670 \
  --zmq-engine-index 0 \
  --grammar-backend xgrammar \
  --enable-output-logprobs

TokenSpeed's grammar backend defaults to none, and then the engine ignores structured-output constraints: a tool_choice: "required" or json_schema request comes back as free text. Output logprobs are off by default. When several TokenSpeed engines share a host, give each its own --port; TokenSpeed derives its internal control ports from it.

python -m smg_grpc_servicer.sglang.headless \
  --zmq-handshake-address tcp://127.0.0.1:22670 \
  --zmq-engine-index 0 \
  --model-path meta-llama/Llama-3.1-8B-Instruct \
  --port 31000

The launcher comes with pip install "smg-grpc-servicer[sglang]" and starts SGLang's own scheduler ranks with no HTTP server, tokenizer manager, or detokenizer; the package's plugin dials SMG's handshake from inside the scheduler once the model has loaded. Every flag other than the two --zmq-* options is an ordinary SGLang server arg. When several SGLang engines share a host, give each its own --port; SGLang derives its internal control ports from it.

Add your usual engine flags, such as --tensor-parallel-size.

Start the Gateway

smg launch \
  --worker-urls ipc:///tmp/smg-zmq/engine-0 \
  --backend vllm \
  --model-path meta-llama/Llama-3.1-8B-Instruct \
  --host 0.0.0.0 \
  --port 30000

Use --backend tokenspeed or --backend sglang to match the engine. --model-path is required: it names the model and loads the tokenizer.

SMG binds the sockets as soon as it registers the worker, then waits up to --zmq-startup-timeout-secs (default 3600 seconds; with smg serve, --router-zmq-startup-timeout-secs) for each handshake message. SMG sees nothing from the engine except those messages, so this one bound must cover everything the engine does between its HELLO and its READY — loading weights, profiling the KV cache, capturing graphs. --worker-startup-timeout-secs bounds registration, not the handshake. If the handshake fails, the next health probe starts a new attempt. An engine that dialed in during an earlier attempt still connects: the new attempt accepts its READY.

To serve several engines, list one ipc:// URL per engine. They are independent workers balanced by --policy.

Register Workers at Runtime

Add a ZMQ worker to a running gateway with POST /workers. The ipc:// and tcp:// schemes select ZMQ, so connection_mode can be left out:

curl -X POST http://localhost:30000/workers \
  -H "Content-Type: application/json" \
  -d '{
    "url": "ipc:///tmp/smg-zmq/engine-1",
    "runtime_type": "tokenspeed"
  }'

ZMQ-related worker fields:

Field Description
url ipc://<path> (the path is required), or tcp://<host>:<port> for an engine on another host (see Engines on Another Host)
runtime_type vllm, tokenspeed, or sglang (runtime is accepted as an alias). Unset means vLLM
zmq_handshake_address A tcp:// address to bind for the handshake instead of the one derived from an ipc:// path. Rejected on non-ZMQ workers, and on tcp:// workers, whose URL is already the handshake address
dp_size Number of engines in a grouped worker; if omitted, the worker waits for the group size declared by the first engine to dial (one for an engine that declares none). Leave dp_rank unset
labels.model_path Model ID and tokenizer source, for gateways started without --model-path

zmq_handshake_address pairs a worker with an engine that dials a fixed address. A TokenSpeed engine started without --data-parallel-rpc-port dials tcp://127.0.0.1:30500, so this worker pairs with it:

curl -X POST http://localhost:30000/workers \
  -H "Content-Type: application/json" \
  -d '{
    "url": "ipc:///tmp/smg-zmq/tokenspeed-0",
    "runtime_type": "tokenspeed",
    "zmq_handshake_address": "tcp://127.0.0.1:30500"
  }'

Registration runs in the background (the API answers 202 Accepted), so a rejected worker shows up only in the gateway log; see Troubleshooting. For the full request schema, see Worker Management.


Data-Parallel Engines

There are two ways to run data parallelism over ZMQ:

  • Replicas: several single-engine workers, each with its own socket path. The gateway's routing policy spreads requests across them. smg serve --data-parallel-size N sets this up.
  • Grouped worker: one worker URL and one socket set, with N data-parallel engines behind it. The gateway sees one worker and picks the engine (rank) for each request itself. For TokenSpeed, smg serve sets this up when an engine-level --data-parallel-size N follows -- on its command line; a vLLM group is set up only manually with smg launch. Configure a Grouped Worker shows the manual setup for both.

Replicas work with every ZMQ engine. Grouped workers are a vLLM and TokenSpeed setup: data parallelism over the SGLang ZMQ wire isn't supported yet (use gRPC for SGLang data parallelism).

Configure a Grouped Worker

Start the engine group with its data-parallel size. A TokenSpeed attention-DP group declares its size when it dials in, so --zmq-engine-count can stay unset and the gateway waits for the whole declared group. A vLLM engine declares no group size, so a vLLM group needs the explicit count:

vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --headless \
  --data-parallel-size 2 \
  --data-parallel-size-local 2 \
  --data-parallel-address 127.0.0.1 \
  --data-parallel-rpc-port 22670

smg launch
--worker-urls ipc:///tmp/smg-zmq/engine-0
--backend vllm
--model-path meta-llama/Llama-3.1-8B-Instruct
--zmq-engine-count 2

python -m tokenspeed.cli serve 
--headless
--model meta-llama/Llama-3.1-8B-Instruct
--data-parallel-size 2
--dist-init-addr 127.0.0.1:31233
--data-parallel-address 127.0.0.1
--data-parallel-rpc-port 22670
--zmq-engine-index 0
--grammar-backend xgrammar
--enable-output-logprobs

smg launch
--worker-urls ipc:///tmp/smg-zmq/engine-0
--backend tokenspeed
--model-path meta-llama/Llama-3.1-8B-Instruct
--zmq-engine-count 2

Each TokenSpeed rank dials in with its own identity (--zmq-engine-index plus its rank). TokenSpeed requires an explicit --dist-init-addr when --data-parallel-size is above 1; pick a free port outside the 20000–29999 handshake range.

  • --zmq-engine-count applies to every ZMQ (ipc:// or tcp://) URL in --worker-urls; an explicit value must be a positive integer, and --zmq-engine-count 1 behaves like leaving the flag unset. For a worker added through the API, set "dp_size" on the worker instead; a dp_size of 1 likewise behaves like leaving the field out.
  • Left unset, the count settles on what the first engine to dial declares, and an engine that declares no group size counts as one. A vLLM group with the count unset therefore settles at a single engine, and the rest of its ranks dial in as extras.
  • The count the gateway waits for must match the number of engines that dial in. An extra engine can fail the handshake with duplicate HELLO ... after INIT phase; a missing one leaves the handshake waiting until it times out.
  • An engine may declare a group size that differs from the count the handshake awaits, whether that count was set explicitly or settled by the first engine. The engine still connects; the gateway logs engine declares a group of a different size than this handshake awaits and keeps the count it was waiting for.
  • The engine process needs N times its tensor-parallel size in GPUs.
  • Grouped workers can't be combined with --dp-aware, which tries to expand the group into one worker per rank and fails with cannot be dp-aware expanded. Single-engine ZMQ workers register normally under --dp-aware.

Rank Selection

Output batches carry the producing rank's scheduler load: running requests, waiting requests, and KV-cache usage (vLLM's scheduler stats, or TokenSpeed's load snapshot). For each request, SMG scores every rank in the group and sends the request to the lowest score:

score = max(requests SMG has in flight on the rank, running + waiting)
        + waiting × 6 × max(0, kv_cache_usage − 0.5)
  • The in-flight floor keeps a stale report from making a busy rank look idle, so a burst of requests between reports spreads across ranks.
  • Engines report load only while they produce output. When a rank's last in-flight request finishes, SMG resets that rank's running and waiting counts to zero and keeps its last reported KV usage.
  • Ties rotate across ranks. A TokenSpeed build that sends no load snapshot is scored on SMG's in-flight counts alone.
  • vLLM MoE models run their data-parallel ranks in lockstep and pause the whole group when idle. SMG sends the wake-up that vLLM's data-parallel coordinator process would otherwise send, so no coordinator runs. Ranks of dense models run independently.

Rank selection happens inside the connection. Routing policies see the group as one worker, and a plain request is never pinned to a rank. A disaggregated TokenSpeed request follows its KV rendezvous instead: a prefill group places it on the rank its rendezvous room selects, and a decode group takes it on the matching rank, so the KV transfer lands where the decode side looks it up.


Worker Lifecycle

Stage What happens
Registered The worker starts as pending. SMG binds its sockets and starts the handshake right away
Connected SMG promotes the worker to ready the moment the handshake completes, without waiting for the health-check success threshold
Serving Health probes read a local liveness flag; the engine has no health RPC. /readiness stays at 503 with tokenizer not yet registered until the model's tokenizer has loaded
Engine lost SMG marks the connection dead when the engine signals that it died (vLLM does so on a fatal error, TokenSpeed also on shutdown, and SGLang whenever its scheduler's event loop exits), the socket fails, three outputs in a row can't be decoded, the engine leaves a frame unaccepted on its input for 300 seconds, or in-flight requests spend a combined 300 seconds — past the prefill allowance for the prompt tokens the engine holds — without any output from the engine. In-flight requests fail. The next health probe drops the dead connection, and a later probe binds the sockets again so a restarted engine can reconnect
Removed DELETE /workers/{worker_id} drains the worker, then removes it; a handshake still in progress is cancelled and its sockets are released
  • Health checks stay on for ZMQ workers even with --disable-health-check or a per-worker disable_health_check, because the probe is what reconnects a restarted engine. SMG logs Ignoring disabled health checks for ZMQ worker ....
  • With --remove-unhealthy-workers, a ZMQ worker whose engine stays down is removed and nothing re-adds it. The flag is off by default unless service discovery is enabled; leave it off for ZMQ workers so they rejoin when their engine restarts.
  • An engine that stops reading its input (a paused or frozen engine core) fails requests, not the connection. A request the engine does not accept within 10 seconds fails with a retryable 503, and a retry can select a different worker (see Retries); the requests the engine already holds keep streaming. The unaccepted frame is still delivered when the engine reads again, followed by an abort, so the engine drops the failed request instead of generating output nobody is waiting for. While the input is blocked, 256 requests or 32 MiB of request payload may wait behind it; a request that would take the waiting requests past either bound gets the 503 immediately instead of after the 10-second wait.
  • The 10-second and 300-second windows are fixed, and so is the prefill allowance's rate: the prompt tokens of requests the engine has accepted but not yet produced output for extend the 300-second bound at 2,000 tokens per second — 500 extra seconds per million pending prompt tokens, and none when no prefill is pending. The TokenSpeed and SGLang wires carry token batches only, as does vLLM's with --disable-log-stats: such a wire says nothing for a prompt until its first token, so a batch of long prompts is silent for as long as its prefill takes. The allowance keeps that silence from counting, so a single request whose prefill outlasts 300 seconds is not failed as a dead engine. A vLLM engine with stats logging on reports every scheduler step, so it is never silent through a prefill. The rate is a bound on how slowly a healthy engine may prefill, not an estimate of how fast it does, so the verdict on a genuinely dead engine comes later in proportion to the prompt it was handed.
  • Only silence past the allowance counts, and it accumulates across requests: the in-flight time since the engine's last output, beyond its pending prefill work, adds up over consecutive busy periods, and only an output of any kind — or 300 seconds with nothing in flight — resets the count. A request cancelled before its first output leaves its prefill allowance in place until it has passed: the engine may still be mid-prefill on the cancelled prompt.
  • Updating a ZMQ worker's properties, such as labels or priority, keeps its live connection.

Features and Limits

Supported over ZMQ

Feature vLLM TokenSpeed SGLang
Chat Completions, Completions, Responses, and Messages, including streaming Yes Yes Yes
Tool-call and reasoning parsing Yes Yes Yes
Structured outputs (response_format, constrained tool_choice, regex, grammar) Yes Yes, with --grammar-backend set on the engine Yes
String stops and EOS Yes Yes Yes
Harmony (gpt-oss) stop strings Yes Yes Yes
Multimodal inputs Yes, one modality per request Yes; SMG computes the Qwen3.5 lineage's MRoPE position table itself and refuses other MRoPE models Images; video and audio are refused
Output logprobs Yes, including top_logprobs Sampled-token logprobs, with --enable-output-logprobs; top_logprobs above 1 needs an engine whose handshake declares the top-k columns Yes, including top_logprobs
Prompt logprobs Forwarded to the engine, but no API field requests them With an engine whose handshake declares the prompt-logprob columns; token_ids_logprob is rejected Yes, requested through /generate's logprob_start_len; token_ids_logprob is rejected
n > 1 and sampling seed Yes Yes n > 1 yes; the SGLang wire carries no seed

Limits

Limit What you see
No authentication on tcp:// workers ipc:// sockets are local paths with a loopback handshake; a tcp:// worker accepts an engine from another host, but its sockets carry no authentication, so keep it on a trusted network. ZMQ workers are never synced to high-availability mesh peers; each belongs to the gateway that registered it
Prefill, decode, and encode workers are TokenSpeed-only A vLLM or SGLang ZMQ worker of a disaggregated role is rejected with ZMQ worker ... cannot serve worker type .... A TokenSpeed ZMQ worker serves every role — its engine must declare the matching role at the handshake, and a prefill or encode engine also its bootstrap address — and PD and EPD routing pair it with workers of the same runtime
No KV-event stream cache_aware tracks ZMQ workers with its approximate prefix tree, and SMG logs ... the ZMQ transport has no KV-event stream ...
No embeddings 501 with code unsupported_backend: ZMQ backend does not support embeddings yet
Engine admin needs a control-call wire POST /flush_cache and profiling reach an SGLang ZMQ worker, and a TokenSpeed one whose every rank declared the control reply at its handshake; the flush skips other ZMQ workers, and profiling them fails with ... is not supported over this engine's ZMQ wire. RL control calls such as weight updates ride a control-wire TokenSpeed worker's own connection; a vLLM ZMQ engine carries no control calls
No tokenizer from the engine The gateway loads it from --model-path or --tokenizer-path (or the worker's model_path/tokenizer_path labels)
No dp-aware rank pinning See Rank Selection
vLLM multimodal Mixed image and video in one request: 400, the vLLM ZMQ backend takes one modality per request .... Worker-side media processing (media references) needs a gRPC vLLM worker: 400, multimodal_not_supported
TokenSpeed logprobs top_logprobs above 1 or prompt logprobs on an engine that declares no column for them, or any token_ids_logprob: 400, ... not supported over the TokenSpeed ZMQ backend
TokenSpeed MRoPE models outside the Qwen3.5 lineage 400: MRoPE position tensors of this model are not derivable over the TokenSpeed ZMQ wire (the Qwen3.5 lineage's are); use the gRPC transport for this model
SGLang non-image media 400: multimodal modality ... is not carried over the SGLang ZMQ backend; the scheduler takes preprocessed image tensors on this wire
SGLang selected-token logprobs token_ids_logprob on a ZMQ worker: 501, selected-token logprobs require the SGLang gRPC servicer — use a gRPC SGLang worker to request them
SGLang data parallelism The wire has no DP yet; run SGLang replicas, or use gRPC for engine-level DP (see Data-Parallel Engines)

Verify

# Ready once the handshake has completed and the tokenizer has loaded
curl http://localhost:30000/readiness

# Transport, runtime, and status of each worker
curl -s http://localhost:30000/workers | jq '.workers[] | {url, connection_mode, runtime_type, status}'

Expected worker entry:

{
  "url": "ipc:///tmp/smg-zmq/engine-0",
  "connection_mode": "zmq",
  "runtime_type": "vllm",
  "status": "ready"
}

Send a request:

curl http://localhost:30000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "meta-llama/Llama-3.1-8B-Instruct",
    "messages": [{"role": "user", "content": "Hello!"}]
  }'

Gateway metrics label ZMQ workers with connection_mode="zmq":

curl -s http://localhost:29000/metrics | grep 'connection_mode="zmq"'

Troubleshooting

Worker stays pending

Symptoms: /readiness returns 503 with insufficient healthy workers, and /workers shows "status": "pending".

Solutions:

  1. Check that the engine dials the address SMG bound: compare the engine's --data-parallel-rpc-port and --data-parallel-address with the handshake= value in SMG's Binding ZMQ client for worker ... log line. With no engine, once --zmq-startup-timeout-secs expires, SMG logs ZMQ backend handshake failed for ...: ... startup handshake timed out while waiting for HELLO after 3600s; will retry on the next health probe.
  2. For a grouped worker, check that an explicit --zmq-engine-count (or dp_size) matches the engine's --data-parallel-size. With no explicit count, the gateway waits for the group size declared by the first engine to dial, so the worker stays pending until that many engines have dialed in.
  3. A TokenSpeed worker registered as prefill, decode, or encode needs an engine that declares the same role — and, for prefill and encode, its bootstrap address — at its handshake; a mismatch fails the handshake with the engine's declared role named in the gateway log.
  4. A worker added through the API to a gateway started with --disable-health-check and no ZMQ workers at startup is never promoted. Restart the gateway without that flag.
Socket bind errors or stale socket files

Symptoms: The worker stays pending, and the gateway log shows ZMQ backend handshake failed for ... followed by a socket or directory error.

Solutions:

SMG deletes leftover socket files at <path>-in.sock and <path>-out.sock before binding, so a crashed gateway doesn't block a restart. Other causes:

  1. ipc socket path ... exists but is not a socket; refusing to unlink: a regular file sits at the socket path. Move it away.
  2. ipc socket dir ... is owned by uid ... or ipc socket dir ... exists but is not a directory: use a directory owned by SMG's user, not /tmp itself or a symlink.
  3. Another process holds the handshake port. Choose a different socket path, or register the worker with zmq_handshake_address.
  4. The socket path is too long. Unix socket paths have a small length limit (108 bytes on Linux), so keep <path>-out.sock short.
Registration rejected

Symptoms: The worker never appears in /workers, and the gateway log shows Failed job: type=AddWorker, worker=ipc://... with the reason.

Solutions:

  1. has no model identity: start SMG with --model-path (with smg serve, --router-model-path), or set a model_path label on the worker.
  2. has unsupported runtime: use vllm, tokenspeed, or sglang.
  3. would bind handshake address ..., already claimed by worker ...: two socket paths hash to the same port. Rename one, or set zmq_handshake_address.
  4. zmq_handshake_address must be a tcp:// address, ZMQ worker URL must be ipc://<path> or tcp://<host>:<port>, names a wildcard host, or takes no separate handshake address: fix the address or URL — a tcp:// worker must name a reachable address of the gateway host and takes no handshake override.
  5. cannot serve worker type: only TokenSpeed engines serve prefill, decode, or encode over ZMQ; vLLM and SGLang ZMQ workers must be regular.
  6. cannot be dp-aware expanded: drop --dp-aware, or use single-engine workers.
TokenSpeed or SGLang requests fail or hang

Symptoms: The worker connects, but requests error or time out.

Solutions:

  1. Look for runtime_type unspecified for ZMQ worker; defaulting to vLLM EngineCore in the gateway log. A TokenSpeed or SGLang worker must be declared with --backend tokenspeed / --backend sglang or an explicit runtime_type, or SMG speaks the vLLM wire protocol to it.
  2. If structured outputs come back unconstrained or logprobs are missing from a TokenSpeed engine, start it with --grammar-backend xgrammar and --enable-output-logprobs.
Readiness reports tokenizer not yet registered

Symptoms: /readiness returns 503 with tokenizer not yet registered after the worker is ready.

Solutions:

  1. Wait for the tokenizer download from --model-path to finish, and check the gateway log for load errors.
  2. Set --tokenizer-path if the tokenizer lives somewhere other than the model path.

Next Steps