gRPC Workers¶

When workers connect via gRPC instead of HTTP, SMG becomes a full OpenAI-compatible server: it handles tokenization, chat templates, reasoning extraction, and tool calling at the gateway level. Workers run raw inference only.

For vLLM or TokenSpeed engines on the same host as the gateway, the ZMQ direct backend is an alternative: it runs the same gateway pipeline over local IPC sockets.

What gRPC Mode Enables¶

In HTTP mode, SMG is a smart proxy that handles routing and failover only. In gRPC mode, SMG takes over the full request processing pipeline.

Start a gRPC Worker¶

The vLLM, SGLang, TokenSpeed and MLX gRPC servers come from the smg-grpc-servicer Python package. Install the extra for your engine: pip install "smg-grpc-servicer[vllm]", [sglang] or [mlx]. For TokenSpeed, install TokenSpeed first, then smg-grpc-servicer. SMG's engine container images already include the package. TensorRT-LLM serves gRPC from its own serve command.

python -m vllm.entrypoints.grpc_server \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --host 0.0.0.0 \
  --port 50051

python -m sglang.launch_server \
  --model-path meta-llama/Llama-3.1-8B-Instruct \
  --host 0.0.0.0 \
  --port 50051 \
  --smg-grpc-mode

# Enable XGrammar guided decoding for forced tool calls and response_format
echo "guided_decoding_backend: xgrammar" > trtllm-config.yaml
python -m tensorrt_llm.commands.serve serve 
  meta-llama/Llama-3.1-8B-Instruct 
  --grpc 
  --host 0.0.0.0 
  --port 50051 
  --backend pytorch 
  --extra_llm_api_options trtllm-config.yaml

python -m smg_grpc_servicer.tokenspeed 
  --model meta-llama/Llama-3.1-8B-Instruct 
  --host 0.0.0.0 
  --port 50051 
  --grammar-backend xgrammar

python -m smg_grpc_servicer.mlx.server \
  --model mlx-community/Qwen3-0.6B-4bit \
  --host 0.0.0.0 \
  --port 50051

Rust vLLM Servicer¶

vLLM gRPC workers can serve the same gRPC contract from a Rust implementation instead of the default Python one. The switch lives in the smg-grpc-servicer package and needs no vLLM change: vLLM's gRPC launcher imports the package's servicer classes before it defines serve_grpc, and that import installs the switch over serve_grpc.

The selection requires SMG v1.12.0 or newer wheels and engine images.

Select the implementation with --servicer-impl on vllm serve --grpc — a flag the package adds to vLLM's parser as a vllm.general_plugins plugin — or with the SMG_VLLM_SERVICER_IMPL environment variable:

vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --grpc \
  --host 0.0.0.0 \
  --port 50051 \
  --servicer-impl rust

# The environment variable selects it for python -m vllm.entrypoints.grpc_server too:
SMG_VLLM_SERVICER_IMPL=rust python -m vllm.entrypoints.grpc_server \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --host 0.0.0.0 \
  --port 50051

When both are set, the flag wins: the parsed --servicer-impl is read ahead of the environment, and the decision is written back to SMG_VLLM_SERVICER_IMPL so the worker's own guard agrees with it. A VLLM_PLUGINS setting that leaves out the smg-servicer plugin only removes the flag; the environment variable works regardless.

In Rust mode, the vLLM engine runs headless in a spawned child process and the servicer talks to it over a same-host ZMQ connection; Python keeps only the process lifecycle. That ZMQ hop is inside the worker — the gateway still connects over grpc://, so this is not the ZMQ direct backend. The gateway cannot tell the two implementations apart.

With smg serve, select it with --servicer-impl rust. The flag defaults to python, is valid only with --backend vllm --connection-mode grpc (any other backend or connection mode is rejected at startup), and is authoritative: it sets SMG_VLLM_SERVICER_IMPL=rust in each worker's environment, and with --servicer-impl python it removes an SMG_VLLM_SERVICER_IMPL inherited from your shell.

smg serve \
  --backend vllm \
  --connection-mode grpc \
  --servicer-impl rust \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --data-parallel-size 2

For the selection to take effect, the worker's environment needs:

• the smg wheel, which carries the Rust servicer binding (smg.servicer.VllmGrpcServer)
• smg-grpc-servicer[vllm], which carries the switch and the Rust request path
• a vLLM whose gRPC launcher imports the package's servicer classes at module level — the stock shape, so an unmodified vLLM qualifies — or carries the check (smg_grpc_servicer.vllm.resolve_servicer_impl) in its own serve_grpc

Misconfiguration fails loudly instead of silently serving Python: smg serve --servicer-impl rust refuses to start when smg-grpc-servicer is not importable or the installed vLLM's launcher would not be switched, and when SMG_VLLM_SERVICER_IMPL=rust reaches a worker where the switch never ran (a launcher file executed directly as __main__, or one that didn't import the package's servicer before defining serve_grpc), the Python servicer itself refuses to start.

Rust SGLang Servicer¶

SGLang gRPC workers have the same choice: a Rust implementation serving the same gRPC contract as the default Python servicer.

Like the vLLM switch, the selection requires SMG v1.12.0 or newer wheels and engine images.

One environment variable selects it on the unchanged entrypoint — every SGLang server flag stays the same:

SMG_SGLANG_SERVICER_IMPL=rust python -m sglang.launch_server \
  --model-path meta-llama/Llama-3.1-8B-Instruct \
  --host 0.0.0.0 \
  --port 50051 \
  --smg-grpc-mode

SMG_SGLANG_SERVICER_IMPL defaults to python, and a value other than python or rust fails startup. smg serve's --servicer-impl stays scoped to vLLM: it rejects rust for every other backend. Exporting the variable before smg serve --backend sglang --connection-mode grpc reaches its workers too, since worker environments inherit the shell's, and no smg serve flag clears an inherited value the way --servicer-impl python does for vLLM.

In Rust mode, the entrypoint hands the process to the Rust servicer (smg.servicer.SglangGrpcServer, which needs the smg wheel installed alongside smg-grpc-servicer[sglang]) before any Python gRPC machinery exists: the Rust server speaks the gRPC contract on the worker's host and port, the scheduler runs headless in a spawned child over a same-host msgpack ZMQ connection, and Python keeps only the process lifecycle. As with vLLM, that ZMQ hop is inside the worker — the gateway still connects over grpc://, so this is not the ZMQ direct backend, and the gateway cannot tell the two implementations apart.

What the scheduler's msgpack wire does not carry, the Rust path reports rather than emulates:

• PD and EPD disaggregation stay with the Python servicer: a worker with a disaggregation_mode set, or with language_only or encoder_only, refuses to start on the Rust path.
• Multimodal inputs and hidden states are refused per request, and LoRA loading answers UNIMPLEMENTED (the Python servicer does not serve it either).
• SGLang's HTTP sidecar (profiling, /metrics) is not started, so nothing listens on --port + 1.

Embed, FlushCache and profiling answer as on the Python servicer, and SubscribeKvEvents relays the ZMQ publisher SGLang was told to run with --kv-events-config (UNIMPLEMENTED when events are off).

Rust TokenSpeed Servicer¶

TokenSpeed gRPC workers have the same choice: a Rust implementation serving the same gRPC contract as the default Python servicer.

Like the vLLM and SGLang switches, the selection requires SMG v1.12.0 or newer wheels and engine images.

One environment variable selects it on the unchanged entrypoint — every flag is still TokenSpeed's ServerArgs:

SMG_TOKENSPEED_SERVICER_IMPL=rust python -m smg_grpc_servicer.tokenspeed \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --host 0.0.0.0 \
  --port 50051

SMG_TOKENSPEED_SERVICER_IMPL defaults to python, and a value other than python or rust fails startup. This launcher is also the process ts serve spawns for its gRPC worker. smg serve offers no path here: its --servicer-impl stays scoped to vLLM, and it drives TokenSpeed workers over --connection-mode zmq only — start the worker directly and connect it with smg launch.

In Rust mode, the entrypoint hands the process to the Rust servicer (smg.servicer.TokenSpeedGrpcServer, which needs the smg wheel installed alongside smg-grpc-servicer) before any TokenSpeed engine or Python gRPC machinery exists: the Rust server speaks the gRPC contract on the worker's host and port, the scheduler(s) run headless in a spawned child — TokenSpeed's own headless launch, what ts serve --headless runs — over a same-host msgpack ZMQ connection, and Python keeps only the process lifecycle. As with vLLM and SGLang, that ZMQ hop is inside the worker: the gateway still connects over grpc://, so this is not the ZMQ direct backend, and the gateway cannot tell the two implementations apart.

What the scheduler's msgpack wire does not carry, the Rust path reports rather than emulates:

• FlushCache and profiling answer UNIMPLEMENTED.
• Ranked top_logprobs and prompt logprobs are refused per request.
• PD and EPD disaggregation stay with the Python servicer.
• The RL control plane extras stay with the Python servicer: the rl.* capability advertisement, the live weight_version on generate responses, and is_paused.

Credentials are redacted from the server_args the worker reports exactly as on the Python servicer. SubscribeKvEvents relays the ZMQ KV-event publisher TokenSpeed was told to run with --kv-events-config (UNIMPLEMENTED when events are off).

Tuning environment variables: SMG_TOKENSPEED_SERVICER_HANDSHAKE_PORT (default: a free port) pins the port the headless scheduler dials, SMG_TOKENSPEED_SERVICER_DRAIN_SECS (default 5) is how long in-flight streams get to finish after a shutdown signal, and SMG_TOKENSPEED_SERVICER_STARTUP_TIMEOUT_SECS (default 1800) bounds the wait for the scheduler's handshake. Model load and kernel JIT compilation count against that bound, so raise it when a first start on a cold cache needs longer; a scheduler process that died is noticed on the supervisor's poll rather than waiting out the bound.

Connect SMG¶

Point SMG at the gRPC worker with a grpc:// URL. SMG detects the engine on its own by probing for SGLang, vLLM, TensorRT-LLM, TokenSpeed and MLX in turn:

smg launch \
  --worker-urls grpc://localhost:50051 \
  --model-path meta-llama/Llama-3.1-8B-Instruct \
  --host 0.0.0.0 \
  --port 30000

The API is still OpenAI-compatible, so clients send the same requests as with HTTP workers:

curl http://localhost:30000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "meta-llama/Llama-3.1-8B-Instruct",
    "messages": [
      {"role": "user", "content": "Hello!"}
    ]
  }'

Multiple gRPC Workers¶

smg launch \
  --worker-urls grpc://worker1:50051 grpc://worker2:50052 \
  --model-path meta-llama/Llama-3.1-8B-Instruct \
  --policy round_robin

Reasoning Extraction¶

For thinking models (DeepSeek-R1, Qwen3, and others), SMG moves chain-of-thought content into a separate field:

smg launch \
  --worker-urls grpc://worker:50051 \
  --model-path deepseek-ai/DeepSeek-R1 \
  --reasoning-parser deepseek_r1

SMG picks the parser from the model name by default; --reasoning-parser overrides that for every model. Separation is on by default, so a plain request is enough:

curl http://localhost:30000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "deepseek-ai/DeepSeek-R1",
    "messages": [{"role": "user", "content": "What is 25 * 37?"}]
  }'

The response includes both fields:

{
  "choices": [{
    "message": {
      "role": "assistant",
      "content": "925",
      "reasoning_content": "Let me calculate 25 * 37 step by step..."
    }
  }]
}

Send "separate_reasoning": false to keep the reasoning in content.

Supported Reasoning Parsers¶

base and passthrough are also registered, but only for explicit use. The gRPC Pipeline page lists the name patterns each parser is auto-detected from.

Tool Calling¶

In gRPC mode, SMG parses function calls from model output:

smg launch \
  --worker-urls grpc://worker:50051 \
  --model-path meta-llama/Llama-3.2-3B-Instruct \
  --tool-call-parser llama

To have the gateway execute MCP tools on /v1/responses, add --mcp-config-path; see MCP in Responses API.

Supported Tool Call Parsers¶

passthrough (no parsing) is also registered. Formats and name patterns are in the gRPC Pipeline reference.

Per-Model Parsers¶

--tool-call-parser and --reasoning-parser apply to every model. When one gateway serves several model families, set parsers per model through the worker API instead. Labels on the worker registration override the flags for that worker's model:

curl -X POST http://localhost:30000/workers \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer ${ADMIN_TOKEN}" \
  -d '{
    "url": "grpc://glm-worker:50051",
    "labels": {"tool_parser": "glm47_moe", "reasoning_parser": "glm45"}
  }'

An unknown parser name fails the registration. See Parser Selection for the full precedence rules.

gpt-oss (Harmony) Vocab¶

Serving gpt-oss models over gRPC uses the Harmony encoding, whose vocab (o200k_base.tiktoken) is fetched when a gpt-oss worker registers — from a local directory if TIKTOKEN_ENCODINGS_BASE is set, otherwise downloaded and cached (TIKTOKEN_RS_CACHE_DIR, default $TMPDIR/tiktoken-rs-cache). If the vocab can't be loaded, that worker's registration fails and retries; other models are unaffected.

For air-gapped deployments, pre-seed the file and point SMG at it:

export TIKTOKEN_ENCODINGS_BASE=/opt/tiktoken   # contains o200k_base.tiktoken

HTTP vs gRPC: When to Use Which¶

Troubleshooting¶

Next Steps¶

• gRPC Pipeline Concepts — Pipeline architecture, parser selection, and all supported parsers
• ZMQ Workers — Same-host vLLM and TokenSpeed engines over the ZMQ direct backend
• Multimodal Pipeline — Image and video processing on the router or the worker
• Tokenizer Caching — Two-level cache for reduced CPU overhead
• MCP in Responses API — Configure Model Context Protocol servers for /v1/responses
• PD Disaggregation — Separate prefill and decode with gRPC workers