gRPC Workers¶
When workers connect via gRPC instead of HTTP, SMG becomes a full OpenAI-compatible server: it handles tokenization, chat templates, reasoning extraction, and tool calling at the gateway level. Workers run raw inference only.
For vLLM, TokenSpeed, or SGLang engines, the ZMQ direct backend is an alternative: it runs the same gateway pipeline over ipc:// sockets to a same-host engine, or over tcp:// to an engine on another host.
What gRPC Mode Enables¶
| Capability | HTTP Mode (worker handles) | gRPC Mode (gateway handles) |
|---|---|---|
| Chat templates | Worker | Gateway |
| Tokenization | Worker | Gateway (with optional caching) |
| Cache-aware routing | On request text, or token IDs for pre-tokenized requests | On the token IDs the gateway produced |
| Reasoning extraction | Worker | Gateway |
| Tool call parsing | Worker | Gateway, with per-model parser overrides |
| Constrained tool calls | Worker | Gateway builds the grammar, worker enforces it |
| Image and video preprocessing | Worker | Gateway; vLLM workers can take it over (--mm-processing, see Multimodal Pipeline) |
| MCP tool execution (Responses API) | Not handled by the gateway | Gateway |
In HTTP mode, SMG is a smart proxy that handles routing and failover only. In gRPC mode, SMG takes over the full request processing pipeline.
Start a gRPC Worker¶
The vLLM, SGLang, TokenSpeed and MLX gRPC servers come from the smg-grpc-servicer Python package. Install the extra for your engine: pip install "smg-grpc-servicer[vllm]", [sglang] or [mlx]. For TokenSpeed, install TokenSpeed first, then smg-grpc-servicer. SMG's engine container images already include the package. TensorRT-LLM serves gRPC from its own serve command, or from the package's python -m smg_grpc_servicer.trtllm launcher.
python -m vllm.entrypoints.grpc_server \
--model meta-llama/Llama-3.1-8B-Instruct \
--host 0.0.0.0 \
--port 50051This starts the default Python servicer. Setting SMG_VLLM_SERVICER_IMPL=rust in the worker's environment serves the same contract from Rust instead; see Rust vLLM Servicer.
After the engine loads, the worker holds its gRPC health at NOT_SERVING while a planned warm-up runs over the engine — about a minute by default; SMG_VLLM_SERVICER_WARMUP tunes or disables it (see Planned Warm-Up Before Serving).
python -m sglang.launch_server \
--model-path meta-llama/Llama-3.1-8B-Instruct \
--host 0.0.0.0 \
--port 50051 \
--grpc-mode--grpc-mode serves the worker through smg-grpc-servicer, which needs SGLang 0.5.21 or later. In this mode SGLang also serves an HTTP sidecar (profiling, plus /metrics with --enable-metrics).
This starts the default Python servicer. Adding --servicer-impl rust — or setting SMG_SGLANG_SERVICER_IMPL=rust in the worker's environment — serves the same contract from Rust instead; see Rust SGLang Servicer.
After the engine loads, the worker holds its gRPC health at NOT_SERVING while a planned warm-up runs over the engine — about a minute by default; SMG_SGLANG_SERVICER_WARMUP tunes or disables it (see Planned Warm-Up Before Serving).
# Enable XGrammar guided decoding for forced tool calls and response_format
echo "guided_decoding_backend: xgrammar" > trtllm-config.yamlpython -m tensorrt_llm.commands.serve serve
meta-llama/Llama-3.1-8B-Instruct
--grpc
--host 0.0.0.0
--port 50051
--backend pytorch
--extra_llm_api_options trtllm-config.yaml
This runs the engine's own Python servicer. The smg-grpc-servicer package carries a TensorRT-LLM launcher too: python -m smg_grpc_servicer.trtllm --model <model> --host 0.0.0.0 --port 50051, with the LLM API's keyword arguments on --llm-args (JSON text, or @path to a JSON or YAML file — the shape of --extra_llm_api_options).
Adding --servicer-impl rust on that launcher — or setting SMG_TRTLLM_SERVICER_IMPL=rust, which the engine's own gRPC entrypoint reads as well — serves the same contract from Rust (smg.servicer.TrtllmGrpcServer, which needs the smg wheel): the engine runs headless in a spawned child behind a shim the launcher finds by import (tensorrt_llm.grpc.smg.shim in the engine's tree, or trtllm_smg_shim installed beside it), over a same-host ZMQ connection inside the worker, and the gateway cannot tell the two implementations apart. The flag defaults to python and wins over the environment variable. What the wire does not carry, the Rust path refuses by name — beam search, the logits-returning flags, KV-cache retention, prompt-tuning and lookahead configs — and Embed and SubscribeKvEvents answer UNIMPLEMENTED on both implementations.
python -m smg_grpc_servicer.tokenspeed \
--model meta-llama/Llama-3.1-8B-Instruct \
--host 0.0.0.0 \
--port 50051 \
--grammar-backend xgrammarTokenSpeed's grammar backend defaults to none, so without --grammar-backend xgrammar it doesn't enforce forced tool calls or response_format. Add --enable-output-logprobs if clients request logprobs; TokenSpeed leaves output logprobs off by default.
This starts the default Python servicer. Adding --servicer-impl rust — or setting SMG_TOKENSPEED_SERVICER_IMPL=rust in the worker's environment — serves the same contract from Rust instead; see Rust TokenSpeed Servicer.
After the engine loads, the worker holds its gRPC health at NOT_SERVING while a planned warm-up runs over the engine. A one-process engine takes about a minute, as on the other engines; an engine of more than one process takes ten minutes or more, since the pass then also warms every prompt length up to the chunk size. SMG_TOKENSPEED_SERVICER_WARMUP tunes or disables it (see Planned Warm-Up Before Serving).
python -m smg_grpc_servicer.mlx.server \
--model mlx-community/Qwen3-0.6B-4bit \
--host 0.0.0.0 \
--port 50051MLX workers are gRPC-only and run on Apple Silicon. They don't support constrained decoding: chat requests with a forced tool_choice or a response_format get a 400, as do requests with n greater than 1 or string stop sequences.
Planned Warm-Up Before Serving¶
vLLM, SGLang and TokenSpeed gRPC workers run a planned warm-up over the engine before they report SERVING on the standard gRPC health service (grpc.health.v1) — the signal Kubernetes gRPC startup and readiness probes read. Both the Python and Rust servicer implementations run the pass; MLX and TensorRT-LLM workers have no such pass, and a model that generates nothing (a pooling or embedding model) takes none.
TokenSpeed workers skip the pass on a disaggregation role. An encoder-only SGLang worker skips it too: the engine runs that role on its own encoder server, never on a servicer. A language-only SGLang worker runs the pass as on a single engine — its text requests carry no media, so the encoders see nothing of them — and on the Rust path the plan also ends with one image through the encoders (the media leg below). On an SGLang prefill or decode leg the Python servicer skips the pass, while the Rust path runs it as on a single engine: each request carries a stand-in transfer to nobody, so the leg warms up without its peer.
A worker whose health stays NOT_SERVING for a while after the engine loaded is warming up, not stuck. The pass logs Planned warm-up starts (...): readiness is held until it ran, one INFO line per shape with its time (TokenSpeed's chunk remainders log at DEBUG, with an INFO progress line every 500 remainders), and Planned warm-up over: N of M steps covered ... at the end.
Engines select and compile kernels lazily: the first scheduler step at a token count the engine has not seen pays a kernel search or a compile of up to several seconds, and every request in flight during that step waits with it. The warm-up visits those token counts before the worker reports SERVING; without it, the first traffic ramp pays them — on every worker of a fleet at once when the workers start together. The pass covers, in order:
- a decode staircase: as many concurrent requests as the running window holds, each asking for one more output token than the last, so every running-batch size runs once;
- a prefill ladder: one request per 128-token prompt-length class up to the engine's admission step (its chunk size or prefill token budget), plus the kernel tuners' bucket grid, the engine's graph-capture sizes, and lengths one above each bucket;
- a second pass over the bucket grid, whose logged timings show what the first pass paid;
- on a language-only SGLang worker on the Rust path, the media leg: one request carrying one image through the encoders — the engine's own minimal warm-up image, sent as a data URL with the prompt the model's chat template renders around it — because no text request reaches the scheduler's media path, whose first use costs a fresh worker a few seconds. A chat template that renders no image placeholder leaves the leg out (a logged line says why), and the leg runs at the
fastlevel too; - on a TokenSpeed engine of more than one process, the chunk remainders: every remaining prompt length up to the chunk size, because on this engine a prompt length's first use during serving can deadlock the forward pass under a collective still in flight. A one-process engine has no peer to wait on and keeps to the length classes, like the other engines. The remainders run last, in an order whose every prefix spreads evenly over the range, so a budget cut leaves even coverage rather than a missing tail.
Every request carries its own token IDs, so prefix caching never hides a prefill, and the pass flushes the engine's prefix cache at the end. A servicer that cannot carry that flush to its engine (the Rust TokenSpeed servicer, on a TokenSpeed whose handshake declares no control reply) refuses it, and the end line then says the warm-up's blocks stay cached.
The gateway's health checks read a different signal: each engine's own HealthCheck RPC, not the standard health service. While the pass runs, a Rust servicer mirrors the hold there — not healthy, with the message Warming up, distinct from Draining — and admits only the warm-up's own requests, refusing anything else with gRPC UNAVAILABLE ("servicer is warming up"), so the gateway neither finishes registering a warming worker nor routes traffic to it.
A Python servicer answers that RPC healthy throughout (vLLM answers from engine liveness; SGLang and TokenSpeed serve the check's one-token probe like any other request) and refuses nothing, so it registers and passes the gateway's health checks while it warms: a request routed to it in that window is served, sharing the engine with the warm-up. On both implementations the hold is released when the pass ends, whatever its outcome — health flips to SERVING, and a failed warm-up request is a logged line, never a worker kept out of rotation.
SMG_VLLM_SERVICER_WARMUP, SMG_SGLANG_SERVICER_WARMUP and SMG_TOKENSPEED_SERVICER_WARMUP tune the pass per engine:
| Value | Effect |
|---|---|
full |
The default: the whole pass above, every 128-token length class included — about a minute at a 16,384-token step. On a multi-process TokenSpeed engine, also the chunk remainders — about 12 minutes for the whole pass, measured at a 4,096-token chunk size on a four-GPU node |
fast |
Skips the 128-token length classes and keeps the rest of the pass (the staircase, the bucket grid with its one-above lengths and second pass, the capture sizes), finishing in seconds. On a multi-process TokenSpeed engine it also skips the chunk remainders and leaves their first-use hazard open, which the start log warns about |
off |
No planned warm-up: the worker reports SERVING once the engine is up and answering |
A number (120) |
The pass's time budget in seconds, at the full level |
<level>:<seconds> (fast:30) |
Level and budget together |
The budget defaults to 300 seconds — 900 for TokenSpeed, whose plan can include the chunk remainders. When it runs out, the pass ends between shapes (the end line names the shape it stopped before), and each request's timeout is clamped to what is left of the budget (never below one second), so a shape that hangs cannot hold readiness past it. A multi-process TokenSpeed worker with no chunk size set plans about 23 minutes of remainders at its default 8,192-token step, and the budget then cuts the pass about halfway through them (evenly, thanks to the spread order): set a chunk size or a larger budget to cover every length.
A value the knob does not take fails a Rust-path worker at startup with an error naming the variable; a Python servicer logs a warning and serves without the warm-up.
Under data parallelism, a vLLM worker — and, on the Rust path, an SGLang worker — runs the staircase once per rank and every prefill shape on every rank at once, each request pinned to its rank, so the log's step count is per rank. The Python SGLang servicer sends no rank pin (the engine's own load balancing places each request), and a TokenSpeed worker routes each warm-up request to the least-loaded rank on both paths; either way each prefill shape warms one rank, the other ranks pay its first-time cost on their first request of that shape, and a logged warning says so.
Rust vLLM Servicer¶
vLLM gRPC workers can serve the same gRPC contract from a Rust implementation instead of the default Python one. The switch lives in the smg-grpc-servicer package and needs no vLLM change: vLLM's gRPC launcher imports the package's servicer classes before it defines serve_grpc, and that import installs the switch over serve_grpc.
The selection requires SMG v1.12.0 or newer wheels and engine images.
Select the implementation with --servicer-impl on vllm serve --grpc — a flag the package adds to vLLM's parser as a vllm.general_plugins plugin — or with the SMG_VLLM_SERVICER_IMPL environment variable:
vllm serve meta-llama/Llama-3.1-8B-Instruct \
--grpc \
--host 0.0.0.0 \
--port 50051 \
--servicer-impl rust
# The environment variable selects it for python -m vllm.entrypoints.grpc_server too:
SMG_VLLM_SERVICER_IMPL=rust python -m vllm.entrypoints.grpc_server \
--model meta-llama/Llama-3.1-8B-Instruct \
--host 0.0.0.0 \
--port 50051When both are set, the flag wins: the parsed --servicer-impl is read ahead of the environment, and the decision is written back to SMG_VLLM_SERVICER_IMPL so the worker's own guard agrees with it. A VLLM_PLUGINS setting that leaves out the smg-servicer plugin only removes the flag; the environment variable works regardless.
In Rust mode, the vLLM engine runs headless in a spawned child process and the servicer talks to it over a same-host ZMQ connection; Python keeps only the process lifecycle. That ZMQ hop is inside the worker — the gateway still connects over grpc://, so this is not the ZMQ direct backend. The gateway cannot tell the two implementations apart.
With smg serve, select it with --servicer-impl rust. The flag defaults to python, is valid only with --backend vllm --connection-mode grpc (any other backend or connection mode is rejected at startup), and is authoritative: it sets SMG_VLLM_SERVICER_IMPL=rust in each worker's environment, and with --servicer-impl python it removes an SMG_VLLM_SERVICER_IMPL inherited from your shell.
smg serve \
--backend vllm \
--connection-mode grpc \
--servicer-impl rust \
--model meta-llama/Llama-3.1-8B-Instruct \
--data-parallel-size 2For the selection to take effect, the worker's environment needs:
- the
smgwheel, which carries the Rust servicer binding (smg.servicer.VllmGrpcServer) smg-grpc-servicer[vllm], which carries the switch and the Rust request path- a vLLM whose gRPC launcher imports the package's servicer classes at module level — the stock shape, so an unmodified vLLM qualifies — or carries the check (
smg_grpc_servicer.vllm.resolve_servicer_impl) in its ownserve_grpc
Misconfiguration fails loudly instead of silently serving Python: smg serve --servicer-impl rust refuses to start when smg-grpc-servicer is not importable or the installed vLLM's launcher would not be switched, and when SMG_VLLM_SERVICER_IMPL=rust reaches a worker where the switch never ran (a launcher file executed directly as __main__, or one that didn't import the package's servicer before defining serve_grpc), the Python servicer itself refuses to start.
SubscribeKvEvents relays vLLM's ZMQ KV-event publisher, which the servicer turns on by itself — on this path as on the Python one — when the launcher is given no --kv-events-config, at a port found free on the host rather than vLLM's fixed default, so two engines on one host do not collide. An explicit --kv-events-config is kept exactly as given, whether it names an endpoint or turns publishing off; without one, SMG_VLLM_SERVICER_KV_EVENTS=0 in the worker's environment leaves the publisher off. With the publisher off, SubscribeKvEvents answers UNIMPLEMENTED. See KV Events Cache-Aware Routing.
Rust SGLang Servicer¶
SGLang gRPC workers have the same choice: a Rust implementation serving the same gRPC contract as the default Python servicer.
Like the vLLM switch, the selection requires SMG v1.12.0 or newer wheels and engine images.
Select the implementation with --servicer-impl on python -m sglang.launch_server — a flag the smg-grpc-servicer package adds to SGLang's parser as an sglang.srt.plugins plugin — or with the SMG_SGLANG_SERVICER_IMPL environment variable:
python -m sglang.launch_server \
--model-path meta-llama/Llama-3.1-8B-Instruct \
--host 0.0.0.0 \
--port 50051 \
--grpc-mode \
--servicer-impl rust
# The environment variable selects it on the same entrypoint:
SMG_SGLANG_SERVICER_IMPL=rust python -m sglang.launch_server \
--model-path meta-llama/Llama-3.1-8B-Instruct \
--host 0.0.0.0 \
--port 50051 \
--grpc-modeWhen both are set, the flag wins: --servicer-impl python overrides an exported rust. The decision a flag makes is written back to SMG_SGLANG_SERVICER_IMPL, so the processes the worker spawns inherit the same choice. The selection defaults to python, and a value other than python or rust fails startup. At startup the worker logs which implementation it chose and whether the flag, the environment or the default decided it.
python -m sglang.launch_server --help lists the flag. SGLang's sglang serve wrapper prints its help before plugins load, so the flag is missing from that listing but still accepted. A SGLANG_PLUGINS setting that leaves out the smg-servicer plugin only removes the flag; the environment variable works regardless.
smg serve's --servicer-impl stays scoped to vLLM: it rejects rust for every other backend. Exporting the environment variable before smg serve --backend sglang --connection-mode grpc reaches its workers too, since worker environments inherit the shell's, and no smg serve flag clears an inherited value the way --servicer-impl python does for vLLM.
In Rust mode, the entrypoint hands the process to the Rust servicer (smg.servicer.SglangGrpcServer, which needs the smg wheel installed alongside smg-grpc-servicer[sglang]) before any Python gRPC machinery exists: the Rust server speaks the gRPC contract on the worker's host and port, the scheduler runs headless in a spawned child over a same-host msgpack ZMQ connection, and Python keeps only the process lifecycle. As with vLLM, that ZMQ hop is inside the worker — the gateway still connects over grpc://, so this is not the ZMQ direct backend, and the gateway cannot tell the two implementations apart.
What the scheduler's msgpack wire does not carry, the Rust path reports rather than emulates:
- Three launches refuse to start on the Rust path: an encoder-only worker (SGLang runs that role on its own encoder server, not on a servicer), a language-only worker on the
zmq_to_tokenizerencoder transfer backend (the frontend itself receives and processes the embeddings there, which neither servicer runs), and a prefill leg of several plain data-parallel replicas —--dp-size Nwithout attention DP — where every replica reports attention-DP rank 0 to the engine's KV sender, so the decode side cannot bootstrap against the requests the other replicas ran; run attention DP (--tp-size N --attn-dp-size N) or one rank per leg. Any other prefill or decode leg, and a language-only worker on the backends whose scheduler ranks receive the embeddings, is served as on the Python servicer. - A multimodal payload of image references without preprocessed tensors — the encoders' input in encode/prefill disaggregation — is refused per request on every worker but a language-only one, whose encode dispatch hands the references to the encoders; the tensors the gateway preprocesses in gRPC mode reach the scheduler as its own processor's output on every worker. LoRA loading answers UNIMPLEMENTED (the Python servicer does not serve it either).
- SGLang's HTTP sidecar (profiling,
/metrics) is not started.
Embed, FlushCache and profiling answer as on the Python servicer.
SubscribeKvEvents relays SGLang's ZMQ KV-event publisher, which the servicer turns on by itself — on this path as on the Python one — when the launcher is given no --kv-events-config, at a port found free on the host rather than SGLang's fixed default. An explicit configuration is kept exactly as given, whether it names an endpoint or turns publishing off; without one, SMG_SGLANG_SERVICER_KV_EVENTS=0 leaves the publisher off. With the publisher off, SubscribeKvEvents answers UNIMPLEMENTED.
Rust TokenSpeed Servicer¶
TokenSpeed gRPC workers have the same choice: a Rust implementation serving the same gRPC contract as the default Python servicer.
Like the vLLM and SGLang switches, the selection requires SMG v1.12.0 or newer wheels and engine images.
Select the implementation with --servicer-impl on python -m smg_grpc_servicer.tokenspeed — this launcher's own flag, which --help lists ahead of TokenSpeed's ServerArgs flags — or with the SMG_TOKENSPEED_SERVICER_IMPL environment variable:
python -m smg_grpc_servicer.tokenspeed \
--model meta-llama/Llama-3.1-8B-Instruct \
--host 0.0.0.0 \
--port 50051 \
--servicer-impl rust
# The environment variable selects it on the same entrypoint:
SMG_TOKENSPEED_SERVICER_IMPL=rust python -m smg_grpc_servicer.tokenspeed \
--model meta-llama/Llama-3.1-8B-Instruct \
--host 0.0.0.0 \
--port 50051When both are set, the flag wins: --servicer-impl python overrides an exported rust. The decision a flag makes is written back to SMG_TOKENSPEED_SERVICER_IMPL, so the processes the worker spawns inherit the same choice. The selection defaults to python, and a value other than python or rust fails startup. At startup the worker logs which implementation it chose and whether the flag, the environment or the default decided it.
This launcher is also the process ts serve spawns for its gRPC worker. smg serve offers no path here: its --servicer-impl stays scoped to vLLM, and it drives TokenSpeed workers over --connection-mode zmq only — start the worker directly and connect it with smg launch.
In Rust mode, the entrypoint hands the process to the Rust servicer (smg.servicer.TokenSpeedGrpcServer, which needs the smg wheel installed alongside smg-grpc-servicer) before any TokenSpeed engine or Python gRPC machinery exists: the Rust server speaks the gRPC contract on the worker's host and port, the scheduler(s) run headless in a spawned child — TokenSpeed's own headless launch, what ts serve --headless runs — over a same-host msgpack ZMQ connection, and Python keeps only the process lifecycle. As with vLLM and SGLang, that ZMQ hop is inside the worker: the gateway still connects over grpc://, so this is not the ZMQ direct backend, and the gateway cannot tell the two implementations apart.
What the scheduler's msgpack wire does not carry, the Rust path reports rather than emulates:
FlushCacheand profiling answer UNIMPLEMENTED on a TokenSpeed whose handshake declares no control reply; one that declares it gets them as control calls, each scheduler rank's own handler answering.- Ranked
top_logprobsand prompt logprobs are served on a TokenSpeed whose handshake declares their output columns; a request asking for one the handshake does not declare is refused. - Prefill/decode disaggregation is served — the request's bootstrap fields ride the wire, and a prefill launch starts the engine's KV bootstrap rendezvous the decode side discovers its KV at, as the engine's own frontend does — and so is the EPD prefill side. The EPD encode role runs on an engine whose wire carries the encode request; a launch on one whose wire predates it is refused by name rather than left waiting in silence.
- The RL control plane extras stay with the Python servicer: the
rl.*capability advertisement, the liveweight_versionon generate responses, andis_paused.
Credentials are redacted from the server_args the worker reports exactly as on the Python servicer.
SubscribeKvEvents relays TokenSpeed's ZMQ KV-event publisher, which the servicer turns on by itself — on this path as on the Python one — when the launcher is given no --kv-events-config, at a port found free on the host rather than TokenSpeed's fixed default. An explicit configuration is kept exactly as given, whether it names an endpoint or turns publishing off; without one, SMG_TOKENSPEED_SERVICER_KV_EVENTS=0 leaves the publisher off. With the publisher off, SubscribeKvEvents answers UNIMPLEMENTED.
On a hybrid model (Gated DeltaNet, Mamba or KDA layers) the servicer leaves the publisher off and logs why once: such a model keeps its recurrent state in State groups of TokenSpeed's cache, and TokenSpeed's KV-event drain aborts the engine on such a group. An explicit --kv-events-config turns the publisher on anyway.
Tuning environment variables: SMG_TOKENSPEED_SERVICER_HANDSHAKE_PORT (default: a free port) pins the port the headless scheduler dials, and SMG_TOKENSPEED_SERVICER_DRAIN_SECS (default 5) is how long in-flight streams get to finish after a shutdown signal.
Two bounds cover startup. SMG_TOKENSPEED_SERVICER_STARTUP_TIMEOUT_SECS (default 1800) bounds the wait for a sign of life from the scheduler during the handshake: the launcher reports the scheduler process alive each time it polls it, and the handshake's own messages count too, so a slow but living start — model load and kernel JIT compilation on a cold cache — can outlast this bound and still complete, while a scheduler process that died fails fast instead of waiting it out. SMG_TOKENSPEED_SERVICER_STARTUP_CEILING_SECS (default 14400, four hours) is the most a start may take however alive the scheduler is; 0 lifts the ceiling.
Once the scheduler is connected, SMG_TOKENSPEED_SERVICER_SILENCE_TIMEOUT_SECS (default 300) is how long the scheduler may hold requests in flight, in total since its last output, with nothing pending prefill, before the servicer treats it as dead, fails them and closes the link for the launcher to restart the pair. Prompt tokens pending prefill — those of requests the scheduler has accepted and produced nothing for yet — extend that bound at SMG_TOKENSPEED_SERVICER_PREFILL_TOKENS_PER_SEC (default 2000 tokens per second: 500 s per million pending prompt tokens). The scheduler's wire carries token batches only, so a batch of long prompts is silent for as long as its prefill takes; the allowance keeps that from reading as a dead scheduler. The rate is a bound on how slowly a healthy scheduler may prefill, not an estimate of how fast it does; 0 allows nothing.
The vLLM and SGLang Rust servicers take the same startup and silence knobs under their SMG_VLLM_SERVICER_* and SMG_SGLANG_SERVICER_* names, and the TensorRT-LLM launcher under SMG_TRTLLM_SERVICER_*.
After the handshake, the planned warm-up runs before the worker reports SERVING; SMG_TOKENSPEED_SERVICER_WARMUP tunes or disables it.
Connect SMG¶
Point SMG at the gRPC worker with a grpc:// URL. SMG detects the engine on its own by probing for SGLang, vLLM, TensorRT-LLM, TokenSpeed and MLX in turn:
smg launch \
--worker-urls grpc://localhost:50051 \
--model-path meta-llama/Llama-3.1-8B-Instruct \
--host 0.0.0.0 \
--port 30000The API is still OpenAI-compatible, so clients send the same requests as with HTTP workers:
curl http://localhost:30000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "meta-llama/Llama-3.1-8B-Instruct",
"messages": [
{"role": "user", "content": "Hello!"}
]
}'Multiple gRPC Workers¶
smg launch \
--worker-urls grpc://worker1:50051 grpc://worker2:50052 \
--model-path meta-llama/Llama-3.1-8B-Instruct \
--policy round_robinReasoning Extraction¶
For thinking models (DeepSeek-R1, Qwen3, and others), SMG moves chain-of-thought content into a separate field:
smg launch \
--worker-urls grpc://worker:50051 \
--model-path deepseek-ai/DeepSeek-R1 \
--reasoning-parser deepseek_r1By default SMG picks the reader per model — a worker label, else the table of a checkpoint Symphony's registry knows, else --reasoning-parser, else detection from the model name — so the flag reads only the models the registry does not know. Separation is on by default, so a plain request is enough:
curl http://localhost:30000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "deepseek-ai/DeepSeek-R1",
"messages": [{"role": "user", "content": "What is 25 * 37?"}]
}'The response includes both fields:
{
"choices": [{
"message": {
"role": "assistant",
"content": "925",
"reasoning_content": "Let me calculate 25 * 37 step by step..."
}
}]
}Send "separate_reasoning": false to keep the reasoning in content.
Supported Reasoning Parsers¶
| Parser | Models |
|---|---|
deepseek_r1 |
DeepSeek-R1 |
deepseek_v31 |
DeepSeek-V3.1 |
deepseek_v4 |
DeepSeek-V4 |
deepseek_v41 |
DeepSeek-V4.1 |
qwen3 |
Qwen3 and later hybrid-thinking Qwen models |
qwen3_thinking |
Qwen3 -Thinking checkpoints |
gemma4 |
Gemma 4 |
glm45 |
GLM-4.5, GLM-4.7, GLM-5.x |
kimi |
Kimi-K2 (Instruct) |
kimi_thinking |
Kimi-K2-Thinking |
kimi_k25 |
Kimi-K2.5 |
kimi_k3 |
Kimi-K3 |
hy_v4 |
Hy4 (tencent/Hy4-preview) |
minimax |
MiniMax-M2 |
minimax_m3 |
MiniMax-M3 |
nano_v3 |
Nemotron Nano, Nemotron Super, Nemotron-3 family |
cohere_cmd |
Command-R, Command-A |
step3 |
Step-3 |
inkling |
Inkling |
base and passthrough are also registered, but only for explicit use. The gRPC Pipeline page lists the name patterns each parser is auto-detected from.
Tool Calling¶
In gRPC mode, SMG parses function calls from model output:
smg launch \
--worker-urls grpc://worker:50051 \
--model-path meta-llama/Llama-3.2-3B-Instruct \
--tool-call-parser llamaTo have the gateway execute MCP tools on /v1/responses, add --mcp-config-path; see MCP in Responses API.
Supported Tool Call Parsers¶
| Parser | Models |
|---|---|
json |
GPT, Claude, Gemini, Gemma other than Gemma 4, and other Llama and GLM models that emit plain JSON |
gemma4 |
Gemma 4 |
llama |
Llama 3.2 |
pythonic |
Llama 4 |
mistral |
Mistral, Mixtral |
qwen |
Qwen2.5, Qwen3 |
qwen_xml |
Qwen3-Coder, Qwen3.5, Qwen3.6, Qwen3.8, Nemotron-3 family (aliases qwen_coder, nemotron) |
deepseek |
DeepSeek-V3 |
deepseek31 |
DeepSeek-V3.1, DeepSeek-V3.2-Exp |
deepseek32 |
DeepSeek-V3.2 |
deepseek_v4 |
DeepSeek-V4 |
deepseek_v41 |
DeepSeek-V4.1 |
glm45_moe |
GLM-4.5, GLM-4.6 |
glm47_moe |
GLM-4.7, GLM-5.x |
kimik2 |
Kimi-K2 |
kimi_k3 |
Kimi-K3 |
hy_v4 |
Hy4 (tencent/Hy4-preview) |
minimax_m2 |
MiniMax-M2 |
minimax_m3 |
MiniMax-M3 |
cohere |
Command-R, Command-A |
step3 |
Step-3 |
sarashina |
Sarashina |
inkling |
Inkling |
passthrough (no parsing) is also registered. Formats and name patterns are in the gRPC Pipeline reference.
Per-Model Parsers¶
--tool-call-parser and --reasoning-parser are gateway-wide, but they read only the models Symphony's registry does not know. When one gateway serves several model families, set parsers per model through the worker API instead. Labels on the worker registration override both the flags and the registry for that worker's model:
curl -X POST http://localhost:30000/workers \
-H "Content-Type: application/json" \
-H "Authorization: Bearer ${ADMIN_TOKEN}" \
-d '{
"url": "grpc://glm-worker:50051",
"labels": {"tool_parser": "glm47_moe", "reasoning_parser": "glm45"}
}'An unknown parser name fails the registration. See Parser Selection for the full precedence rules.
gpt-oss (Harmony) Vocab¶
Serving gpt-oss models over gRPC uses the Harmony encoding, whose vocab
(o200k_base.tiktoken) is fetched when a gpt-oss worker registers — from a
local directory if TIKTOKEN_ENCODINGS_BASE is set, otherwise downloaded and
cached (TIKTOKEN_RS_CACHE_DIR, default $TMPDIR/tiktoken-rs-cache). If the
vocab can't be loaded, that worker's registration fails and retries; other
models are unaffected.
For air-gapped deployments, pre-seed the file and point SMG at it:
export TIKTOKEN_ENCODINGS_BASE=/opt/tiktoken # contains o200k_base.tiktokenHTTP vs gRPC: When to Use Which¶
| Use Case | Recommended Mode |
|---|---|
| Workers already run OpenAI-compatible HTTP servers | HTTP |
| You need gateway-level tool parsing, reasoning extraction, or Responses MCP | gRPC |
| You serve several model families and want parsers chosen per model | gRPC |
| You want cache-aware routing on token IDs | gRPC |
| Simplest possible setup | HTTP |
Troubleshooting¶
| Symptom | Cause | Solution |
|---|---|---|
404 model_not_found |
No registered worker serves the requested model name | Check GET /v1/models and send a served model name or alias |
503 no_available_workers |
Every worker for the model is unhealthy or has an open circuit breaker | Check worker health and logs |
A grpcs:// worker never registers; the log repeats grpc health check failed ... gRPC connection failed: transport error |
The gateway's gRPC channel has no TLS | Use grpc://, and a service mesh when the hop must be encrypted (Gateway to gRPC Workers) |
A worker's gRPC health stays NOT_SERVING after the engine loaded; on the Rust path requests get UNAVAILABLE servicer is warming up |
The planned warm-up is still running — about a minute by default, ten minutes or more on a multi-process TokenSpeed engine | Wait for the worker's Planned warm-up over log line, or set its warm-up variable to fast or off |
503 tokenizer_not_ready |
The model's tokenizer is still loading (the gateway loads it after the worker registers) | Retry; the response carries Retry-After: 1, and /readiness reports the same state |
503 tokenizer_load_failed |
The model's tokenizer won't load on its own — no source to load from (no path label, flag, or model id), the load failed, or the tokenizer was removed; the message names the reason, and the gateway log has the details | Pass --model-path or set a tokenizer_path label and re-register the worker, or register a tokenizer with POST /v1/tokenizers; retrying the request changes nothing until a tokenizer registers |
500 tokenizer_not_found |
The model has no tokenizer and --disable-tokenizer-autoload is set, so the gateway won't load one itself |
Register a tokenizer with POST /v1/tokenizers |
Tool calls or reasoning left in content |
No parser matched the model name | Set --tool-call-parser / --reasoning-parser or a per-model label |
Next Steps¶
- gRPC Pipeline Concepts — Pipeline architecture, parser selection, and all supported parsers
- ZMQ Workers — vLLM, TokenSpeed, and SGLang engines over the ZMQ direct backend
- Multimodal Pipeline — Image and video processing on the router or the worker
- Tokenizer Caching — Two-level cache for reduced CPU overhead
- MCP in Responses API — Configure Model Context Protocol servers for
/v1/responses - PD Disaggregation — Separate prefill and decode with gRPC workers