gRPC Workers

When workers connect via gRPC instead of HTTP, SMG becomes a full OpenAI-compatible server: it handles tokenization, chat templates, reasoning extraction, and tool calling at the gateway level. Workers run raw inference only.

For vLLM, TokenSpeed, or SGLang engines, the ZMQ direct backend is an alternative: it runs the same gateway pipeline straight against the headless engine core, over local ipc:// sockets on a shared host or tcp:// for an engine on another host.


What gRPC Mode Enables

Capability HTTP Mode (worker handles) gRPC Mode (gateway handles)
Chat templates Worker Gateway
Tokenization Worker Gateway (with optional caching)
Cache-aware routing On request text, or token IDs for pre-tokenized requests On the token IDs the gateway produced
Reasoning extraction Worker Gateway
Tool call parsing Worker Gateway, with per-model parser overrides
Constrained tool calls Worker Gateway builds the grammar, worker enforces it
Image and video preprocessing Worker Gateway; vLLM workers — and SGLang workers running SMG's own pipeline — can take it over (--mm-processing, see Multimodal Pipeline)
MCP tool execution (Responses API) Not handled by the gateway Gateway

In HTTP mode, SMG is a smart proxy that handles routing and failover only. In gRPC mode, SMG takes over the full request processing pipeline.


Start a gRPC Worker

The vLLM, SGLang, TokenSpeed and MLX gRPC servers come from the smg-grpc-servicer Python package. Install the extra for your engine: pip install "smg-grpc-servicer[vllm]", [sglang] or [mlx]. For TokenSpeed, install TokenSpeed first, then smg-grpc-servicer. SMG's engine container images already include the package. TensorRT-LLM serves gRPC from its own serve command, or from the package's python -m smg_grpc_servicer.trtllm launcher.

python -m vllm.entrypoints.grpc_server \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --host 0.0.0.0 \
  --port 50051

This starts the Rust servicer, the default. Setting SMG_VLLM_SERVICER_IMPL=python in the worker's environment serves the same contract from the deprecated Python servicer instead; see Rust vLLM Servicer.

After the engine loads, the worker holds its gRPC health at NOT_SERVING while a planned warm-up runs over the engine — about a minute by default; SMG_VLLM_SERVICER_WARMUP tunes or disables it (see Planned Warm-Up Before Serving).

python -m sglang.launch_server \
  --model-path meta-llama/Llama-3.1-8B-Instruct \
  --host 0.0.0.0 \
  --port 50051 \
  --grpc-mode

--grpc-mode serves the worker through smg-grpc-servicer, which needs SGLang 0.5.21 or later.

This starts the Rust servicer, the default. Adding --servicer-impl python — or setting SMG_SGLANG_SERVICER_IMPL=python in the worker's environment — serves the same contract from the deprecated Python servicer instead; see Rust SGLang Servicer. On the Python servicer, SGLang also serves an HTTP sidecar (profiling, plus /metrics with --enable-metrics), which the Rust servicer does not start.

After the engine loads, the worker holds its gRPC health at NOT_SERVING while a planned warm-up runs over the engine — about a minute by default; SMG_SGLANG_SERVICER_WARMUP tunes or disables it (see Planned Warm-Up Before Serving).

# Enable XGrammar guided decoding for forced tool calls and response_format
echo "guided_decoding_backend: xgrammar" > trtllm-config.yaml

python -m tensorrt_llm.commands.serve serve
meta-llama/Llama-3.1-8B-Instruct
--grpc
--host 0.0.0.0
--port 50051
--backend pytorch
--extra_llm_api_options trtllm-config.yaml

This runs the engine's own Python servicer. The smg-grpc-servicer package carries a TensorRT-LLM launcher too: python -m smg_grpc_servicer.trtllm --model <model> --host 0.0.0.0 --port 50051, with the LLM API's keyword arguments on --llm-args (JSON text, or @path to a JSON or YAML file — the shape of --extra_llm_api_options).

Adding --servicer-impl rust on that launcher — or setting SMG_TRTLLM_SERVICER_IMPL=rust in its environment — serves the same contract from Rust (smg.servicer.TrtllmGrpcServer, which needs the smg wheel): the engine runs headless in a spawned child behind a shim the launcher finds by import (tensorrt_llm.grpc.smg.shim in the engine's tree, or trtllm_smg_shim installed beside it), over a same-host ZMQ connection inside the worker, and the gateway cannot tell the two implementations apart. The flag defaults to python and wins over the environment variable. The engine's own serve command carries no servicer switch, so the variable selects nothing there: serving from Rust means starting the worker with the package launcher. What the wire does not carry, the Rust path refuses by name — beam search, the logits-returning flags, KV-cache retention, prompt-tuning and lookahead configs — and Embed answers UNIMPLEMENTED on both implementations. On the Rust path, a loads_only SubscribeKvEvents — the stream the gateway's load monitoring reads — streams the servicer's own load records, while a full subscription answers UNIMPLEMENTED naming why: this wire has no KV-event publisher yet.

python -m smg_grpc_servicer.tokenspeed \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --host 0.0.0.0 \
  --port 50051 \
  --grammar-backend xgrammar

Without --grammar-backend, TokenSpeed turns its grammar backend on only for checkpoints it defaults to xgrammar; the rest run with none and reject requests that need constrained decoding, such as a forced tool_choice or a JSON response_format. --grammar-backend xgrammar turns enforcement on for any model. Add --enable-output-logprobs if clients request logprobs; TokenSpeed leaves output logprobs off by default.

This starts the Rust servicer, the default. Adding --servicer-impl python — or setting SMG_TOKENSPEED_SERVICER_IMPL=python in the worker's environment — serves the same contract from the deprecated Python servicer instead; see Rust TokenSpeed Servicer.

After the engine loads, the worker holds its gRPC health at NOT_SERVING while a planned warm-up runs over the engine. A one-process engine takes about a minute, as on the other engines; an engine of more than one process takes ten minutes or more, since the pass then also warms every prompt length up to the chunk size. SMG_TOKENSPEED_SERVICER_WARMUP tunes or disables it (see Planned Warm-Up Before Serving).

python -m smg_grpc_servicer.mlx.server \
  --model mlx-community/Qwen3-0.6B-4bit \
  --host 0.0.0.0 \
  --port 50051

MLX workers are gRPC-only and run on Apple Silicon. They don't support constrained decoding: chat requests with a forced tool_choice or a response_format get a 400, as do requests with n greater than 1 or string stop sequences.

Planned Warm-Up Before Serving

vLLM, SGLang and TokenSpeed gRPC workers run a planned warm-up over the engine before they report SERVING on the standard gRPC health service (grpc.health.v1) — the signal Kubernetes gRPC startup and readiness probes read. Both the Python and Rust servicer implementations run the pass; MLX and TensorRT-LLM workers have no such pass, and a model that generates nothing (a pooling or embedding model) takes none.

TokenSpeed workers skip the pass on a disaggregation role. An encoder-only SGLang worker skips it too: the engine runs that role on its own encoder server, never on a servicer. A language-only SGLang worker runs the pass as on a single engine — its text requests carry no media, so the encoders see nothing of them — and on the Rust path the plan also ends with one image through the encoders (the media leg below).

On an SGLang prefill or decode leg the Python servicer skips the pass; the Rust path runs it, every request carrying a stand-in transfer to nobody, so the leg warms up without its peer. A decode leg runs the whole pass, as on a single engine. A prefill leg's decode steps run on its peer, so its plan is one request of the smallest shape per rank, plus the media leg when the worker has one: the first forward and the transfer path are paid before SERVING, and each prompt-length class is paid by its first request during serving.

A worker whose health stays NOT_SERVING for a while after the engine loaded is warming up, not stuck. The pass logs Planned warm-up starts (...): readiness is held until it ran, one INFO line per shape with its time (TokenSpeed's chunk remainders log at DEBUG, with an INFO progress line every 500 remainders), and Planned warm-up over: N of M steps covered ... at the end.

Engines select and compile kernels lazily: the first scheduler step at a token count the engine has not seen pays a kernel search or a compile of up to several seconds, and every request in flight during that step waits with it. The warm-up visits those token counts before the worker reports SERVING; without it, the first traffic ramp pays them — on every worker of a fleet at once when the workers start together. The pass covers, in order:

  • a decode staircase: as many concurrent requests as the running window holds, each asking for one more output token than the last, so every running-batch size runs once;
  • a prefill ladder: one request per 128-token prompt-length class up to the engine's admission step (its chunk size or prefill token budget), plus the kernel tuners' bucket grid, the engine's graph-capture sizes, and lengths one above each bucket;
  • a second pass over the bucket grid, whose logged timings show what the first pass paid;
  • on a language-only SGLang worker on the Rust path, the media leg: one request carrying one image through the encoders — the engine's own minimal warm-up image, sent as a data URL with the prompt the model's chat template renders around it — because no text request reaches the scheduler's media path, whose first use costs a fresh worker a few seconds. A chat template that renders no image placeholder leaves the leg out (a logged line says why), and the leg runs at the fast level too;
  • on a TokenSpeed engine of more than one process, the chunk remainders: every remaining prompt length up to the chunk size, because on this engine a prompt length's first use during serving can deadlock the forward pass under a collective still in flight. A one-process engine has no peer to wait on and keeps to the length classes, like the other engines. The remainders run last, in an order whose every prefix spreads evenly over the range, so a budget cut leaves even coverage rather than a missing tail.

Every request carries its own token IDs, so prefix caching never hides a prefill, and the pass flushes the engine's prefix cache at the end. A servicer that cannot carry that flush to its engine (the Rust TokenSpeed servicer, on a TokenSpeed whose handshake declares no control reply) refuses it, and the end line then says the warm-up's blocks stay cached.

The gateway's health checks read a different signal: each engine's own HealthCheck RPC, not the standard health service. While the pass runs, a Rust servicer mirrors the hold there — not healthy, with the message Warming up, distinct from Draining — and admits only the warm-up's own requests, refusing anything else with gRPC UNAVAILABLE ("servicer is warming up"), so the gateway neither finishes registering a warming worker nor routes traffic to it.

A Python servicer answers that RPC healthy throughout (vLLM answers from engine liveness; SGLang and TokenSpeed serve the check's one-token probe like any other request) and refuses nothing, so it registers and passes the gateway's health checks while it warms: a request routed to it in that window is served, sharing the engine with the warm-up. On both implementations the hold is released when the pass ends, whatever its outcome — health flips to SERVING, and a failed warm-up request is a logged line, never a worker kept out of rotation.

SMG_VLLM_SERVICER_WARMUP, SMG_SGLANG_SERVICER_WARMUP and SMG_TOKENSPEED_SERVICER_WARMUP tune the pass per engine:

Value Effect
full The default: the whole pass above, every 128-token length class included — about a minute at a 16,384-token step. On a multi-process TokenSpeed engine, also the chunk remainders — about 12 minutes for the whole pass, measured at a 4,096-token chunk size on a four-GPU node
fast Skips the 128-token length classes and keeps the rest of the pass (the staircase, the bucket grid with its one-above lengths and second pass, the capture sizes), finishing in seconds. On a multi-process TokenSpeed engine it also skips the chunk remainders and leaves their first-use hazard open, which the start log warns about
off No planned warm-up: the worker reports SERVING once the engine is up and answering
A number (120) The pass's time budget in seconds, at the full level
<level>:<seconds> (fast:30) Level and budget together

The budget defaults to 300 seconds — 900 for TokenSpeed, whose plan can include the chunk remainders. When it runs out, the pass ends between shapes (the end line names the shape it stopped before), and each request's timeout is clamped to what is left of the budget (never below one second), so a shape that hangs cannot hold readiness past it. A multi-process TokenSpeed worker with no chunk size set plans about 23 minutes of remainders at its default 8,192-token step, and the budget then cuts the pass about halfway through them (evenly, thanks to the spread order): set a chunk size or a larger budget to cover every length.

A value the knob does not take fails a Rust-path worker at startup with an error naming the variable; a Python servicer logs a warning and serves without the warm-up.

Under data parallelism, a vLLM worker — and, on the Rust path, an SGLang worker — runs the staircase once per rank and every prefill shape on every rank at once, each request pinned to its rank, so the log's step count is per rank. The Python SGLang servicer sends no rank pin (the engine's own load balancing places each request), and a TokenSpeed worker routes each warm-up request to the least-loaded rank on both paths; either way each prefill shape warms one rank, the other ranks pay its first-time cost on their first request of that shape, and a logged warning says so.

Rust vLLM Servicer

vLLM gRPC workers serve the gRPC contract from a Rust implementation by default; the deprecated Python implementation serves the same contract and runs only when selected explicitly. The switch lives in the smg-grpc-servicer package and needs no vLLM change: vLLM's gRPC launcher imports the package's servicer classes before it defines serve_grpc, and that import installs the switch over serve_grpc.

The selection requires SMG v1.12.0 or newer wheels and engine images.

Select an implementation explicitly with --servicer-impl on vllm serve --grpc — a flag the package adds to vLLM's parser as a vllm.general_plugins plugin — or with the SMG_VLLM_SERVICER_IMPL environment variable. rust names the default; selecting python runs the deprecated Python servicer, and the launcher logs a deprecation warning:

vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --grpc \
  --host 0.0.0.0 \
  --port 50051 \
  --servicer-impl python

# The environment variable selects it for python -m vllm.entrypoints.grpc_server too:
SMG_VLLM_SERVICER_IMPL=python python -m vllm.entrypoints.grpc_server \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --host 0.0.0.0 \
  --port 50051

When both are set, the flag wins: the parsed --servicer-impl is read ahead of the environment, and the decision is written back to SMG_VLLM_SERVICER_IMPL so the worker's own guard agrees with it. A VLLM_PLUGINS setting that leaves out the smg-servicer plugin only removes the flag; the environment variable works regardless.

In Rust mode, the vLLM engine runs headless in a spawned child process and the servicer talks to it over a same-host ZMQ connection; Python keeps only the process lifecycle. That ZMQ hop is inside the worker — the gateway still connects over grpc://, so this is not the ZMQ direct backend. The gateway cannot tell the two implementations apart.

With smg serve, an unset --servicer-impl leaves the choice to the workers: an SMG_VLLM_SERVICER_IMPL exported in your shell reaches them unchanged, and without one they run the servicer package's own default, the Rust servicer. An explicit --servicer-impl value — rust, or the deprecated python — is valid only with --backend vllm --connection-mode grpc (any other backend or connection mode is rejected at startup) and is authoritative: it is set as SMG_VLLM_SERVICER_IMPL in each worker's environment, overriding an inherited value. A worker on another connection mode (http or zmq) gets no servicer variable: smg serve removes it from that worker's environment.

smg serve \
  --backend vllm \
  --connection-mode grpc \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --data-parallel-size 2

For the Rust servicer to run, the worker's environment needs:

  • the smg wheel, which carries the Rust servicer binding (smg.servicer.VllmGrpcServer)
  • smg-grpc-servicer[vllm], which carries the switch and the Rust request path
  • a vLLM whose gRPC launcher imports the package's servicer classes at module level — the stock shape, so an unmodified vLLM qualifies — or carries the check (smg_grpc_servicer.vllm.resolve_servicer_impl) in its own serve_grpc

Misconfiguration fails loudly instead of silently serving Python. smg serve refuses to start when the installed vLLM's launcher would not be switched; the check runs whenever the Rust servicer is the effective choice — the default, an explicit --servicer-impl rust, or an inherited SMG_VLLM_SERVICER_IMPL=rust — and only a python chosen by the flag or the variable skips it. An explicit or inherited rust also fails startup when smg-grpc-servicer is not importable; on the default, a missing package is left to the worker, whose gRPC entrypoint needs the package for either servicer and says so itself. On a worker where the switch never ran (a launcher file executed directly as __main__, or one that didn't import the package's servicer before defining serve_grpc), the Python servicer itself refuses to start unless python was selected explicitly.

SubscribeKvEvents relays vLLM's ZMQ KV-event publisher, which the servicer turns on by itself — on this path as on the Python one — when the launcher is given no --kv-events-config, at a port found free on the host rather than vLLM's fixed default, so two engines on one host do not collide. An explicit --kv-events-config is kept exactly as given, whether it names an endpoint or turns publishing off; without one, SMG_VLLM_SERVICER_KV_EVENTS=0 in the worker's environment leaves the publisher off. With the publisher off, SubscribeKvEvents answers UNIMPLEMENTED. See KV Events Cache-Aware Routing.

Rust SGLang Servicer

SGLang gRPC workers make the same choice: the Rust servicer is the default, and the deprecated Python servicer serves the same gRPC contract when selected explicitly.

Like the vLLM switch, the selection requires SMG v1.12.0 or newer wheels and engine images.

Select an implementation explicitly with --servicer-impl on python -m sglang.launch_server — a flag the smg-grpc-servicer package adds to SGLang's parser as an sglang.srt.plugins plugin — or with the SMG_SGLANG_SERVICER_IMPL environment variable. rust names the default; selecting python runs the deprecated Python servicer, and the launcher logs a deprecation warning:

python -m sglang.launch_server \
  --model-path meta-llama/Llama-3.1-8B-Instruct \
  --host 0.0.0.0 \
  --port 50051 \
  --grpc-mode \
  --servicer-impl python

# The environment variable selects it on the same entrypoint:
SMG_SGLANG_SERVICER_IMPL=python python -m sglang.launch_server \
  --model-path meta-llama/Llama-3.1-8B-Instruct \
  --host 0.0.0.0 \
  --port 50051 \
  --grpc-mode

When both are set, the flag wins: --servicer-impl python overrides an exported rust. The decision a flag makes is written back to SMG_SGLANG_SERVICER_IMPL, so the processes the worker spawns inherit the same choice. The selection defaults to rust, and a value other than python or rust fails startup. At startup the worker logs which implementation it chose and whether the flag, the environment or the default decided it.

python -m sglang.launch_server --help lists the flag. SGLang's sglang serve wrapper prints its help before plugins load, so the flag is missing from that listing but still accepted. A SGLANG_PLUGINS setting that leaves out the smg-servicer plugin only removes the flag; the environment variable works regardless.

smg serve's --servicer-impl stays scoped to vLLM: it rejects an explicit value for every other backend. An SMG_SGLANG_SERVICER_IMPL exported before smg serve --backend sglang --connection-mode grpc reaches its workers too, since worker environments inherit the shell's, and no smg serve flag overrides an inherited value the way --servicer-impl does for vLLM.

In Rust mode, the entrypoint hands the process to the Rust servicer (smg.servicer.SglangGrpcServer, which needs the smg wheel installed alongside smg-grpc-servicer[sglang]) before any Python gRPC machinery exists: the Rust server speaks the gRPC contract on the worker's host and port, the scheduler runs headless in a spawned child over a same-host msgpack ZMQ connection, and Python keeps only the process lifecycle. As with vLLM, that ZMQ hop is inside the worker — the gateway still connects over grpc://, so this is not the ZMQ direct backend, and the gateway cannot tell the two implementations apart.

What the scheduler's msgpack wire does not carry, the Rust path reports rather than emulates:

  • Three launches refuse to start on the Rust path: an encoder-only worker (SGLang runs that role on its own encoder server, not on a servicer), a language-only worker on the zmq_to_tokenizer encoder transfer backend (the frontend itself receives and processes the embeddings there, which neither servicer runs), and a prefill leg of several plain data-parallel replicas — --dp-size N without attention DP — where every replica reports attention-DP rank 0 to the engine's KV sender, so the decode side cannot bootstrap against the requests the other replicas ran; run attention DP (--tp-size N --attn-dp-size N) or one rank per leg. Any other prefill or decode leg, and a language-only worker on the backends whose scheduler ranks receive the embeddings, is served as on the Python servicer.
  • A multimodal payload of image references without preprocessed tensors — the encoders' input in encode/prefill disaggregation — is refused per request on every worker but a language-only one, whose encode dispatch hands the references to the encoders; the tensors the gateway preprocesses in gRPC mode reach the scheduler as its own processor's output on every worker. LoRA loading answers UNIMPLEMENTED (the Python servicer does not serve it either).
  • SGLang's HTTP sidecar (profiling, /metrics) is not started.

Embed, FlushCache, profiling and prompt logprobs answer as on the Python servicer; hidden states are served on this path only (the Python servicer refuses them by name).

SubscribeKvEvents relays SGLang's ZMQ KV-event publisher, which the servicer turns on by itself — on this path as on the Python one — when the launcher is given no --kv-events-config, at a port found free on the host rather than SGLang's fixed default. An explicit configuration is kept exactly as given, whether it names an endpoint or turns publishing off; without one, SMG_SGLANG_SERVICER_KV_EVENTS=0 leaves the publisher off. With the publisher off, SubscribeKvEvents answers UNIMPLEMENTED.

Rust TokenSpeed Servicer

TokenSpeed gRPC workers make the same choice: the Rust servicer is the default, and the deprecated Python servicer serves the same gRPC contract when selected explicitly.

Like the vLLM and SGLang switches, the selection requires SMG v1.12.0 or newer wheels and engine images.

Select an implementation explicitly with --servicer-impl on python -m smg_grpc_servicer.tokenspeed — this launcher's own flag, which --help lists ahead of TokenSpeed's ServerArgs flags — or with the SMG_TOKENSPEED_SERVICER_IMPL environment variable. rust names the default; selecting python runs the deprecated Python servicer, and the launcher logs a deprecation warning:

python -m smg_grpc_servicer.tokenspeed \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --host 0.0.0.0 \
  --port 50051 \
  --servicer-impl python

# The environment variable selects it on the same entrypoint:
SMG_TOKENSPEED_SERVICER_IMPL=python python -m smg_grpc_servicer.tokenspeed \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --host 0.0.0.0 \
  --port 50051

When both are set, the flag wins: --servicer-impl python overrides an exported rust. The decision a flag makes is written back to SMG_TOKENSPEED_SERVICER_IMPL, so the processes the worker spawns inherit the same choice. The selection defaults to rust, and a value other than python or rust fails startup. At startup the worker logs which implementation it chose and whether the flag, the environment or the default decided it.

This launcher is also the process ts serve spawns for its gRPC worker. smg serve offers no path here: its --servicer-impl stays scoped to vLLM, and it drives TokenSpeed workers over --connection-mode zmq only — start the worker directly and connect it with smg launch.

In Rust mode, the entrypoint hands the process to the Rust servicer (smg.servicer.TokenSpeedGrpcServer, which needs the smg wheel installed alongside smg-grpc-servicer) before any TokenSpeed engine or Python gRPC machinery exists: the Rust server speaks the gRPC contract on the worker's host and port, the scheduler(s) run headless in a spawned child — TokenSpeed's own headless launch, what ts serve --headless runs — over a same-host msgpack ZMQ connection, and Python keeps only the process lifecycle. As with vLLM and SGLang, that ZMQ hop is inside the worker: the gateway still connects over grpc://, so this is not the ZMQ direct backend, and the gateway cannot tell the two implementations apart.

What the scheduler's msgpack wire does not carry, the Rust path reports rather than emulates:

  • FlushCache and profiling answer UNIMPLEMENTED on a TokenSpeed whose handshake declares no control reply; one that declares it gets them as control calls, each scheduler rank's own handler answering.
  • Ranked top_logprobs and prompt logprobs each ride to schedulers whose handshakes declare that output's columns, and are refused per request on an engine whose handshake does not.
  • The msgpack wire bypasses the engine's input processor, which computes a multimodal item's MRoPE position table on the Python path. Items of the Qwen3.5 lineage (Qwen3.5, Qwen3.5-MoE, Qwen4-Exp) are served anyway: the Rust servicer computes the table itself from the checkpoint's vision_config.spatial_merge_size, image_token_id, and video_token_id; on a lineage checkpoint missing any of those facts, every multimodal request is refused. Items of other MRoPE families (for example Qwen3-VL or Qwen3-Omni) are refused per request; the Python servicer serves those models through the engine's own input processor.
  • The RL control plane extras stay with the Python servicer: the rl.* capability advertisement, the live weight_version on generate responses, and is_paused.

Prefill/decode disaggregation is served as on the Python servicer: a request's bootstrap fields ride the wire, and a prefill worker starts the engine's KV bootstrap server ahead of its scheduler ranks (or leaves it to a headless launcher that hosts its own). The EPD prefill side is served the same way, and the EPD encode role on a TokenSpeed whose wire carries the encode request; an engine whose wire predates it fails the start by name.

Credentials are redacted from the server_args the worker reports exactly as on the Python servicer.

SubscribeKvEvents relays TokenSpeed's ZMQ KV-event publisher, which the servicer turns on by itself — on this path as on the Python one — when the launcher is given no --kv-events-config, at a port found free on the host rather than TokenSpeed's fixed default. An explicit configuration is kept exactly as given, whether it names an endpoint or turns publishing off; without one, SMG_TOKENSPEED_SERVICER_KV_EVENTS=0 leaves the publisher off. With the publisher off, SubscribeKvEvents answers UNIMPLEMENTED.

On a hybrid model (Gated DeltaNet, Mamba or KDA layers) the default follows the installed engine. An engine whose KV events name their cache group publishes such a model's history chain beside its State checkpoints, so the publisher stays on. On an engine whose events carry no group, the servicer leaves the publisher off and logs why once: such a model keeps its recurrent state in State groups of TokenSpeed's cache, and a feed without groups publishes a boundary only while every group holds it, so the model's history chain would never reach the router. An explicit --kv-events-config turns the publisher on anyway.

Tuning environment variables: SMG_TOKENSPEED_SERVICER_HANDSHAKE_PORT (default: a free port) pins the port the headless scheduler dials, and SMG_TOKENSPEED_SERVICER_DRAIN_SECS (default 5) is how long in-flight streams get to finish after a shutdown signal.

Two bounds cover startup. SMG_TOKENSPEED_SERVICER_STARTUP_TIMEOUT_SECS (default 1800) bounds the wait for a sign of life from the scheduler during the handshake: the launcher reports the scheduler process alive each time it polls it, and the handshake's own messages count too, so a slow but living start — model load and kernel JIT compilation on a cold cache — can outlast this bound and still complete, while a scheduler process that died fails fast instead of waiting it out. SMG_TOKENSPEED_SERVICER_STARTUP_CEILING_SECS (default 14400, four hours) is the most a start may take however alive the scheduler is; 0 lifts the ceiling.

Once the scheduler is connected, SMG_TOKENSPEED_SERVICER_SILENCE_TIMEOUT_SECS (default 300) is how long the scheduler may hold requests in flight, in total since its last sign of life, with nothing pending prefill, before the servicer treats it as dead, fails them and closes the link for the launcher to restart the pair. An output of any kind is a sign of life, and so is a KV-event batch the scheduler's publisher sends: storing or freeing KV blocks is the scheduler's own work, so a chunked prefill that publishes its blocks chunk by chunk is not read as a dead scheduler, even though this wire carries token batches only and says nothing for a prompt until its first token. Only a batch read live off the publisher, in the publisher's sequence, counts — a batch filled in from gap replay, a repeat of a sequence already relayed, a state-snapshot chunk and the relay's load-only heartbeats are not the scheduler's current work and end no silence. Prompt tokens pending prefill — those of requests the scheduler has accepted and produced nothing for yet — extend that bound at SMG_TOKENSPEED_SERVICER_PREFILL_TOKENS_PER_SEC (default 2000 tokens per second: 500 s per million pending prompt tokens); that allowance is what keeps a long prefill from reading as a dead scheduler when it publishes no KV events. The rate is a bound on how slowly a healthy scheduler may prefill, not an estimate of how fast it does; 0 allows nothing.

The vLLM and SGLang Rust servicers take the same startup and silence knobs under their SMG_VLLM_SERVICER_* and SMG_SGLANG_SERVICER_* names, and the TensorRT-LLM launcher's Rust path under SMG_TRTLLM_SERVICER_*.

After the handshake, the planned warm-up runs before the worker reports SERVING; SMG_TOKENSPEED_SERVICER_WARMUP tunes or disables it.


Connect SMG

Point SMG at the gRPC worker with a grpc:// URL. SMG detects the engine on its own by probing for SGLang, vLLM, TensorRT-LLM, TokenSpeed and MLX in turn:

smg launch \
  --worker-urls grpc://localhost:50051 \
  --model-path meta-llama/Llama-3.1-8B-Instruct \
  --host 0.0.0.0 \
  --port 30000

The API is still OpenAI-compatible, so clients send the same requests as with HTTP workers:

curl http://localhost:30000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "meta-llama/Llama-3.1-8B-Instruct",
    "messages": [
      {"role": "user", "content": "Hello!"}
    ]
  }'

Multiple gRPC Workers

smg launch \
  --worker-urls grpc://worker1:50051 grpc://worker2:50052 \
  --model-path meta-llama/Llama-3.1-8B-Instruct \
  --policy round_robin

Reasoning Extraction

For thinking models (DeepSeek-R1, Qwen3, and others), SMG moves chain-of-thought content into a separate field:

smg launch \
  --worker-urls grpc://worker:50051 \
  --model-path deepseek-ai/DeepSeek-R1

Checkpoints SMG knows are read with its built-in output tables; for a model it does not know, SMG picks the parser from the model name, and --reasoning-parser sets one for those models. Worker labels override both per model (see Per-Model Parsers). Separation is on by default, so a plain request is enough:

curl http://localhost:30000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "deepseek-ai/DeepSeek-R1",
    "messages": [{"role": "user", "content": "What is 25 * 37?"}]
  }'

The response includes both fields:

{
  "choices": [{
    "message": {
      "role": "assistant",
      "content": "925",
      "reasoning_content": "Let me calculate 25 * 37 step by step..."
    }
  }]
}

Send "separate_reasoning": false to keep the reasoning in content.

Supported Reasoning Parsers

Parser Models
deepseek_r1 DeepSeek-R1
deepseek_v31 DeepSeek-V3.1
deepseek_v4 DeepSeek-V4
deepseek_v41 DeepSeek-V4.1
qwen3 Qwen3 and later hybrid-thinking Qwen models
qwen3_thinking Qwen3 -Thinking checkpoints
glm45 GLM-4.5, GLM-4.7, GLM-5.x
gemma4 Gemma 4
hy_v4 Hy4
kimi Kimi-K2 (Instruct)
kimi_thinking Kimi-K2-Thinking
kimi_k25 Kimi-K2.5
kimi_k3 Kimi-K3
minimax MiniMax-M2
minimax_m3 MiniMax-M3
nano_v3 Nemotron Nano, Nemotron Super, Nemotron-3 family
cohere_cmd Command-R, Command-A
step3 Step-3
inkling Inkling

base and passthrough are also registered, but only for explicit use. The gRPC Pipeline page lists the name patterns each parser is auto-detected from.


Tool Calling

In gRPC mode, SMG parses function calls from model output. Checkpoints SMG knows are read with its built-in output tables, so a plain launch is enough:

smg launch \
  --worker-urls grpc://worker:50051 \
  --model-path meta-llama/Llama-3.2-3B-Instruct

For a model SMG does not know — a renamed or quantized checkpoint, for example — set --tool-call-parser to a name from the table below.

To have the gateway execute MCP tools on /v1/responses, add --mcp-config-path; see MCP in Responses API.

Supported Tool Call Parsers

Parser Models
json GPT, Claude, Gemini, Gemma, and other Llama and GLM models that emit plain JSON
gemma4 Gemma 4
llama Llama 3.2
pythonic Llama 4
mistral Mistral, Mixtral
qwen Qwen2.5, Qwen3
qwen_xml Qwen3-Coder, Qwen3.5, Qwen3.6, Qwen3.8, Nemotron-3 family (aliases qwen_coder, nemotron)
deepseek DeepSeek-V3
deepseek31 DeepSeek-V3.1
deepseek32 DeepSeek-V3.2, DeepSeek-V3.2-Exp
deepseek_v4 DeepSeek-V4
deepseek_v41 DeepSeek-V4.1
glm45_moe GLM-4.5, GLM-4.6
glm47_moe GLM-4.7, GLM-5.x
kimik2 Kimi-K2
kimi_k3 Kimi-K3
hy_v4 Hy4
minimax_m2 MiniMax-M2
minimax_m3 MiniMax-M3
cohere Command-R, Command-A
step3 Step-3
sarashina Sarashina
inkling Inkling

passthrough (no parsing) is also registered. Formats and name patterns are in the gRPC Pipeline reference.

Per-Model Parsers

--tool-call-parser and --reasoning-parser are gateway-wide fallbacks: they decide for the models whose checkpoints SMG's registry does not know. When one gateway serves several model families, set parsers per model through the worker API instead. Labels on the worker registration override the registry and the flags for that worker's model:

curl -X POST http://localhost:30000/workers \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer ${ADMIN_TOKEN}" \
  -d '{
    "url": "grpc://glm-worker:50051",
    "labels": {"tool_parser": "glm47_moe", "reasoning_parser": "glm45"}
  }'

An unknown parser name fails the registration. See Parser Selection for the full precedence rules.


gpt-oss (Harmony) Vocab

Serving gpt-oss models over gRPC uses the Harmony encoding, whose vocab (o200k_base.tiktoken) is fetched when a gpt-oss worker registers — from a local directory if TIKTOKEN_ENCODINGS_BASE is set, otherwise downloaded and cached (TIKTOKEN_RS_CACHE_DIR, default $TMPDIR/tiktoken-rs-cache). If the vocab can't be loaded, that worker's registration fails and retries; other models are unaffected.

For air-gapped deployments, pre-seed the file and point SMG at it:

export TIKTOKEN_ENCODINGS_BASE=/opt/tiktoken   # contains o200k_base.tiktoken

HTTP vs gRPC: When to Use Which

Use Case Recommended Mode
Workers already run OpenAI-compatible HTTP servers HTTP
You need gateway-level tool parsing, reasoning extraction, or Responses MCP gRPC
You serve several model families and want parsers chosen per model gRPC
You want cache-aware routing on token IDs gRPC
Simplest possible setup HTTP

Troubleshooting

Symptom Cause Solution
404 model_not_found No registered worker serves the requested model name Check GET /v1/models and send a served model name or alias
503 no_available_workers Every worker for the model is unhealthy or has an open circuit breaker Check worker health and logs
A grpcs:// worker never registers; the log repeats grpc health check failed ... gRPC connection failed: transport error The gateway's gRPC channel has no TLS Use grpc://, and a service mesh when the hop must be encrypted (Gateway to gRPC Workers)
A worker's gRPC health stays NOT_SERVING after the engine loaded; on the Rust path requests get UNAVAILABLE servicer is warming up The planned warm-up is still running — about a minute by default, ten minutes or more on a multi-process TokenSpeed engine Wait for the worker's Planned warm-up over log line, or set its warm-up variable to fast or off
503 tokenizer_not_ready The model's tokenizer is still loading (the gateway loads it after the worker registers) Retry; the response carries Retry-After: 1, and /readiness reports the same state
503 tokenizer_load_failed The model's tokenizer won't load on its own — no tokenizer_path or model_path to load from, the load failed, or the tokenizer was removed; the message names the reason, and the gateway log has the details Pass --model-path or set a tokenizer_path label and re-register the worker, or register a tokenizer with POST /v1/tokenizers; retrying the request changes nothing until a tokenizer registers
500 tokenizer_not_found The model has no tokenizer and --disable-tokenizer-autoload is set, so the gateway won't load one itself Register a tokenizer with POST /v1/tokenizers
Tool calls or reasoning left in content The checkpoint is not one SMG knows (a renamed or quantized checkpoint, for example) and no parser matched the model name Set --tool-call-parser / --reasoning-parser or a per-model label

Next Steps