Getting Started

Shepherd Model Gateway (SMG) routes and manages LLM traffic across workers. This page gives you a fast path to a working gateway, then points you to feature-specific setup guides.

Install

pip install smg

The wheels on PyPI target the stable ABI (abi3), so each one works on CPython 3.9 and newer:

Platform Wheels
Linux x86_64 manylinux2014 (glibc 2.17+), musllinux 1.1
Linux aarch64 manylinux 2.28 (glibc 2.28+), musllinux 1.1
macOS Intel (10.12+), Apple Silicon (11.0+)
Windows x86_64

The package installs the smg command:

  • smg launch starts the gateway alone, for workers you run yourself
  • smg serve starts engine workers and then the gateway; the engine (vLLM, SGLang, TensorRT-LLM, or TokenSpeed) must be installed in the same environment

smg launch parses its flags in Python (smg.launch_router, which is also the gateway Docker image's entrypoint) and runs the Rust gateway in-process. It takes the Rust binary's full flag set; a few flags differ, such as audit logging: the binary writes control-plane audit logs unless --disable-audit-logging turns them off, while the launcher writes them only when --control-plane-audit-enabled turns them on.

Nightly Linux x86_64 wheels are attached to dated prereleases in smg-project/artifacts. Install one with pip install <wheel URL>.

cargo install --locked smg

This compiles the gateway from crates.io and installs the smg binary (and its amg alias) into ~/.cargo/bin. --locked builds with the Cargo.lock published with the release. The binary takes the full Rust flag set through smg launch (or plain smg); smg serve is only in the Python package.

The build needs protoc, a C/C++ toolchain, and OpenSSL headers:

# Debian/Ubuntu
sudo apt-get install -y build-essential pkg-config libssl-dev protobuf-compiler

# macOS (Homebrew) brew install protobuf openssl@3

To compile OpenSSL from source instead of linking the system library, add --features vendored-openssl (needs perl and make).

Gateway only (no inference engine), for linux/amd64 and linux/arm64:

docker pull lightseekorg/smg:latest

# Same image on GitHub Container Registry docker pull ghcr.io/smg-project/smg:latest

Release tags are the bare version with no v prefix, for example lightseekorg/smg:1.15.0, and latest points at the newest release. GHCR carries gateway releases from 1.10.1 on; older release tags are on Docker Hub. Nightly builds are on GHCR only: ghcr.io/smg-project/smg:nightly, or nightly-<YYYYMMDD>-<short-sha> to pin one night. Dated nightly tags are pruned after a short retention window.

Gateway + engine (all-in-one images that route and serve), for linux/amd64 only:

docker pull ghcr.io/smg-project/smg:<version>-<engine>-<engine_version>

Each release builds these engine images from the listed base images:

Engine Tag suffix Base image Nightly tag
vLLM vllm-v0.31.0, vllm-v0.27.1, vllm-v0.26.0 vllm/vllm-openai nightly-vllm
SGLang sglang-v0.5.21 lmsysorg/sglang nightly-sglang
TensorRT-LLM trtllm-1.3.0rc24, trtllm-1.3.0rc23, trtllm-1.3.0rc22 nvcr.io/nvidia/tensorrt-llm/release nightly-trtllm
TokenSpeed tokenspeed-<x.y.z.postYYYYMMDD> dated TokenSpeed nightly build nightly-tokenspeed

For example, this release publishes ghcr.io/smg-project/smg:1.15.0-vllm-v0.31.0 and ghcr.io/smg-project/smg:1.15.0-sglang-v0.5.21, both built from SMG 1.15.0. Browse every tag on GHCR or Docker Hub. Engine images for 1.9.0 and earlier were published as ghcr.io/lightseekorg/smg.

Each release's TokenSpeed image is built from the dated TokenSpeed nightly pinned in the SMG repository at build time, and its tag suffix carries that nightly's version. The older tokenspeed-tml release images belong to 1.11.0 and earlier, and they do not contain that release's SMG: they were built from SMG 1.7.0, so 1.11.0-tokenspeed-tml contains SMG 1.7.0. nightly-tokenspeed is built from current SMG.

# Install Rust
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
source "$HOME/.cargo/env"

# Clone and build (needs the protoc and OpenSSL packages from the Cargo tab) git clone https://github.com/smg-project/smg.git cd smg cargo build --release

The gateway binary is at ./target/release/smg. To build and install the Python package (which adds smg serve) from the same checkout:

pip install maturin
make python-install

Step 1: Start SMG

Choose one of these startup paths.

Option A: All-in-one with smg serve

smg serve launches backend worker process(es) and then starts SMG with generated worker URLs.

smg serve \
  --backend vllm \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --data-parallel-size 2 \
  --host 0.0.0.0 \
  --port 30000
smg serve \
  --backend trtllm \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --data-parallel-size 2 \
  --host 0.0.0.0 \
  --port 30000
smg serve \
  --backend tokenspeed \
  --connection-mode zmq \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --router-model-path meta-llama/Llama-3.1-8B-Instruct \
  --data-parallel-size 2 \
  --host 0.0.0.0 \
  --port 30000

TokenSpeed runs headless, and SMG connects to its engine core over ZMQ. --router-model-path gives the gateway the model name and tokenizer, which the engine doesn't report over ZMQ.

smg serve \
  --backend sglang \
  --model-path meta-llama/Llama-3.1-8B-Instruct \
  --data-parallel-size 2 \
  --connection-mode grpc \
  --host 0.0.0.0 \
  --port 30000

This starts --data-parallel-size worker replicas, waits for each to become healthy, then starts the gateway. With --connection-mode zmq there is no wait: the gateway starts at once, and /readiness returns 503 until a worker has completed its handshake and the tokenizer has loaded.

To connect vLLM to its engine core directly instead of through gRPC, add --connection-mode zmq and --router-model-path <model>. See ZMQ Direct Workers.

Option Default Description
--backend sglang Inference backend: sglang, vllm, trtllm, or tokenspeed. The SMG_DEFAULT_BACKEND environment variable changes the default
--connection-mode grpc Worker connection mode: grpc, http, or zmq. TensorRT-LLM supports only grpc and TokenSpeed only zmq; zmq also works with vllm and sglang
--servicer-impl rust Implementation of the vLLM gRPC servicer: rust or the deprecated python. When unset, the workers follow an SMG_VLLM_SERVICER_IMPL exported in your shell, else the servicer package's own default, the Rust servicer. An explicit value requires --backend vllm --connection-mode grpc; see Rust vLLM Servicer
--host 127.0.0.1 Router host
--port 8080 Router port (30000 with --backend sglang; see below)
--data-parallel-size, --dp-size 1 Number of worker replicas, each on its own GPU slice
--worker-host 127.0.0.1 Host for worker processes
--worker-base-port 31000 Base port for worker processes
--worker-startup-timeout 300 Seconds to wait for each worker to become healthy
--enable-token-usage-details off In http mode, start the engine with cached-token reporting (--enable-cache-report for SGLang, --enable-prompt-tokens-details for vLLM)

Gateway options take a --router- prefix (for example --router-policy or --router-model-path); other flags are passed to the engine. When the backend's own CLI also defines --host or --port, its definition replaces the one above: with --backend sglang the router defaults to 127.0.0.1:30000, and with vLLM in http mode to 0.0.0.0:8000. Pass both flags to be explicit.

Option B: Launch gateway only with smg launch

Use this when workers are already running or managed by another platform.

For gRPC workers:

smg launch \
  --worker-urls grpc://localhost:50051 \
  --model-path meta-llama/Llama-3.1-8B-Instruct \
  --policy round_robin \
  --host 0.0.0.0 \
  --port 30000

For HTTP workers:

smg launch \
  --worker-urls http://localhost:8000 \
  --policy round_robin \
  --host 0.0.0.0 \
  --port 30000

Step 2: Verify Core Endpoints

Health:

curl http://localhost:30000/health
curl http://localhost:30000/readiness

OpenAI-compatible chat completions:

curl http://localhost:30000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "meta-llama/Llama-3.1-8B-Instruct",
    "messages": [{"role": "user", "content": "Say hello in one sentence."}]
  }'

Responses API:

curl http://localhost:30000/v1/responses \
  -H "Content-Type: application/json" \
  -d '{
    "model": "meta-llama/Llama-3.1-8B-Instruct",
    "input": "Say hello in one sentence."
  }'

Step 3: Choose Your Setup Track

Core Deployment

Operations and Security

Reliability and Data

Advanced Features


Worker Startup Recipes (Standalone)

Use these when workers are not started via smg serve. Each command starts one worker; register it with smg launch --worker-urls using a grpc:// or http:// URL, as in Option B. The SMG engine images for vLLM, SGLang, TensorRT-LLM, and TokenSpeed already contain the engine and the SMG gRPC servicer (see Install).

gRPC mode needs the SMG servicer: pip install "vllm[grpc]" adds smg-grpc-servicer[vllm].

# gRPC worker (grpc://<host>:50051)
python -m vllm.entrypoints.grpc_server \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --host 0.0.0.0 \
  --port 50051 \
  --tensor-parallel-size 1

# HTTP worker (http://<host>:8000) python -m vllm.entrypoints.openai.api_server
--model meta-llama/Llama-3.1-8B-Instruct
--host 0.0.0.0
--port 8000

The gRPC worker runs the Rust servicer (which needs the smg wheel) by default. Setting SMG_VLLM_SERVICER_IMPL=python in its environment serves the same contract from the deprecated Python servicer instead; see Rust vLLM Servicer.

SGLang depends on smg-grpc-servicer, so gRPC mode works after pip install sglang. The servicer supports SGLang 0.5.21 and later.

# gRPC worker (grpc://<host>:50051)
python -m sglang.launch_server \
  --model-path meta-llama/Llama-3.1-8B-Instruct \
  --host 0.0.0.0 \
  --port 50051 \
  --grpc-mode

# HTTP worker (http://<host>:8000) python -m sglang.launch_server
--model-path meta-llama/Llama-3.1-8B-Instruct
--host 0.0.0.0
--port 8000

The gRPC worker runs the Rust servicer (which needs the smg wheel) by default. Setting SMG_SGLANG_SERVICER_IMPL=python in its environment serves the same contract from the deprecated Python servicer instead; on that path SGLang also serves an HTTP sidecar (profiling, plus /metrics with --enable-metrics), which the Rust servicer does not start. See Rust SGLang Servicer.

gRPC serving is built into TensorRT-LLM, so this command needs no SMG package:

# gRPC worker (grpc://<host>:50051)
python -m tensorrt_llm.commands.serve serve \
  meta-llama/Llama-3.1-8B-Instruct \
  --grpc \
  --host 0.0.0.0 \
  --port 50051 \
  --backend pytorch \
  --tp_size 1

This runs the engine's own Python servicer, the TensorRT-LLM default. The smg-grpc-servicer package carries a TensorRT-LLM launcher too, python -m smg_grpc_servicer.trtllm, which takes the LLM API's keyword arguments on --llm-args. Adding --servicer-impl rust on that launcher — or setting SMG_TRTLLM_SERVICER_IMPL=rust in its environment, the fallback the flag overrides — serves the same contract from the Rust servicer (which needs the smg wheel) instead. The engine's own entrypoint above has no servicer switch, so running the Rust servicer means running the package launcher; see gRPC Workers.

Install TokenSpeed from source, then pip install "smg-grpc-servicer[tokenspeed]".

# gRPC worker (grpc://<host>:50051)
python -m smg_grpc_servicer.tokenspeed \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --host 0.0.0.0 \
  --port 50051 \
  --grammar-backend xgrammar

Without --grammar-backend, TokenSpeed turns its grammar backend on only for checkpoints it defaults to xgrammar; the rest run with none and reject requests that need constrained decoding, such as tool_choice set to required or a named function, or a JSON response_format. --grammar-backend xgrammar turns enforcement on for any model. Add --enable-output-logprobs if clients request logprobs; without it they come back empty.

The gRPC worker runs the Rust servicer (which needs the smg wheel) by default. Setting SMG_TOKENSPEED_SERVICER_IMPL=python in its environment serves the same contract from the deprecated Python servicer instead; see Rust TokenSpeed Servicer.

Apple Silicon only. Install the servicer with pip install "smg-grpc-servicer[mlx]".

# gRPC worker (grpc://<host>:50051)
python -m smg_grpc_servicer.mlx.server \
  --model mlx-community/Qwen3-0.6B-4bit \
  --host 0.0.0.0 \
  --port 50051

For ZMQ direct workers (ipc:// or tcp:// URLs), the engine runs headless and connects to the gateway instead; see ZMQ Workers.

PD Disaggregation Workers

For prefill-decode disaggregation, start separate prefill and decode workers, each on its own GPUs (on one host, pin them with CUDA_VISIBLE_DEVICES):

# Prefill worker
python -m sglang.launch_server \
  --model-path meta-llama/Llama-3.1-8B-Instruct \
  --host 0.0.0.0 \
  --port 50051 \
  --grpc-mode \
  --disaggregation-mode prefill \
  --disaggregation-bootstrap-port 8998

# Decode worker python -m sglang.launch_server
--model-path meta-llama/Llama-3.1-8B-Instruct
--host 0.0.0.0
--port 50052
--grpc-mode
--disaggregation-mode decode

Start SMG with the prefill worker's bootstrap port after its URL:

smg launch \
  --pd-disaggregation \
  --prefill grpc://localhost:50051 8998 \
  --decode grpc://localhost:50052 \
  --model-path meta-llama/Llama-3.1-8B-Instruct \
  --host 0.0.0.0 \
  --port 30000
# Prefill worker
python -m sglang.launch_server \
  --model-path meta-llama/Llama-3.1-8B-Instruct \
  --host 0.0.0.0 \
  --port 8000 \
  --disaggregation-mode prefill \
  --disaggregation-bootstrap-port 8998

# Decode worker python -m sglang.launch_server
--model-path meta-llama/Llama-3.1-8B-Instruct
--host 0.0.0.0
--port 8001
--disaggregation-mode decode

Start SMG with the prefill worker's bootstrap port after its URL:

smg launch \
  --pd-disaggregation \
  --prefill http://localhost:8000 8998 \
  --decode http://localhost:8001 \
  --host 0.0.0.0 \
  --port 30000

vLLM uses NIXL for KV cache transfer between prefill and decode workers:

# Prefill worker
VLLM_NIXL_SIDE_CHANNEL_PORT=5600 \
python -m vllm.entrypoints.grpc_server \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --host 0.0.0.0 \
  --port 50051 \
  --kv-transfer-config '{"kv_connector":"NixlConnector","kv_role":"kv_producer"}'

# Decode worker VLLM_NIXL_SIDE_CHANNEL_PORT=5601
python -m vllm.entrypoints.grpc_server
--model meta-llama/Llama-3.1-8B-Instruct
--host 0.0.0.0
--port 50052
--kv-transfer-config '{"kv_connector":"NixlConnector","kv_role":"kv_consumer"}'

Start SMG (no bootstrap ports needed — NIXL handles KV transfer):

smg launch \
  --pd-disaggregation \
  --prefill grpc://localhost:50051 \
  --decode grpc://localhost:50052 \
  --model-path meta-llama/Llama-3.1-8B-Instruct \
  --host 0.0.0.0 \
  --port 30000

SMG drives vLLM disaggregation over the OpenAI-compatible HTTP server too, with no gRPC servicer:

# Prefill worker
VLLM_NIXL_SIDE_CHANNEL_PORT=5600 \
vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --host 0.0.0.0 \
  --port 8000 \
  --kv-transfer-config '{"kv_connector":"NixlConnector","kv_role":"kv_producer"}'

# Decode worker VLLM_NIXL_SIDE_CHANNEL_PORT=5601
vllm serve meta-llama/Llama-3.1-8B-Instruct
--host 0.0.0.0
--port 8001
--kv-transfer-config '{"kv_connector":"NixlConnector","kv_role":"kv_consumer"}'

Start SMG in PD mode without startup workers:

smg launch --pd-disaggregation --host 0.0.0.0 --port 30000

The vLLM HTTP server does not report its KV connector, so register both workers with it. Without kv_connector, requests still succeed but decode recomputes every prompt:

curl -X POST http://localhost:30000/workers \
  -H "Content-Type: application/json" \
  -d '{"url": "http://localhost:8000", "worker_type": "prefill", "kv_connector": "NixlConnector"}'

curl -X POST http://localhost:30000/workers
-H "Content-Type: application/json"
-d '{"url": "http://localhost:8001", "worker_type": "decode", "kv_connector": "NixlConnector"}'

See PD Disaggregation for the Mooncake backend, TokenSpeed and EPD, Kubernetes discovery, and scaling.

Send a Request

curl http://localhost:30000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "meta-llama/Llama-3.1-8B-Instruct",
    "messages": [
      {"role": "user", "content": "What is the capital of France?"}
    ],
    "max_tokens": 50
  }'

Expected response:

{
  "id": "chatcmpl-abc123",
  "object": "chat.completion",
  "model": "meta-llama/Llama-3.1-8B-Instruct",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "The capital of France is Paris."
      },
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 14,
    "completion_tokens": 8,
    "total_tokens": 22
  }
}

Verify Health

# Gateway health
curl http://localhost:30000/health

# Worker status
curl http://localhost:30000/workers

Deploy with Docker

For local deployment, run the gateway image and point it at your worker. Container arguments go to the image's entrypoint, python3 -m smg.launch_router, so they are the same flags as smg launch from pip:

docker run -d \
  --name smg \
  -p 30000:30000 \
  -p 29000:29000 \
  --add-host=host.docker.internal:host-gateway \
  lightseekorg/smg:latest \
  --worker-urls http://host.docker.internal:8000 \
  --policy cache_aware

The gateway listens on 0.0.0.0:30000 and serves Prometheus metrics on 0.0.0.0:29000 by default. --add-host makes host.docker.internal resolve to the Docker host on Linux; Docker Desktop provides that name already. The container runs as a non-root user (UID 65532), so files you mount into it must be readable by that user. ghcr.io/smg-project/smg:latest is the same image.

Verify:

docker ps | grep smg
curl http://localhost:30000/health

All-in-one with engine images

Engine images use smg as their entrypoint, so pass serve to start the worker and the gateway in one container. Set --host 0.0.0.0: smg serve binds the gateway to 127.0.0.1 by default, which a published port cannot reach. The Docker tab under Install lists the engine tags each release builds.

docker run -d --gpus all \
  --name smg \
  -p 30000:30000 \
  -v /path/to/models:/models \
  ghcr.io/smg-project/smg:1.15.0-sglang-v0.5.21 \
  serve \
  --backend sglang \
  --model-path /models/meta-llama/Llama-3.1-8B-Instruct \
  --host 0.0.0.0 \
  --port 30000
docker run -d --gpus all \
  --name smg \
  -p 30000:30000 \
  -v /path/to/models:/models \
  ghcr.io/smg-project/smg:1.15.0-vllm-v0.31.0 \
  serve \
  --backend vllm \
  --model /models/meta-llama/Llama-3.1-8B-Instruct \
  --host 0.0.0.0 \
  --port 30000

Each engine image sets SMG_DEFAULT_BACKEND to its engine, so --backend is optional. Workers connect over gRPC by default. TensorRT-LLM images take --backend trtllm and support only gRPC. TokenSpeed needs --connection-mode zmq and --router-model-path (see ZMQ Workers).

The TokenSpeed release images from 1.11.0 and earlier cannot serve TokenSpeed this way: the SMG 1.7.0 they contain has no TokenSpeed backend or ZMQ mode in smg serve.

Verify:

curl http://localhost:30000/health
curl http://localhost:30000/v1/models

Deploy to Kubernetes (Quick Start)

Run SMG in the cluster with the Helm chart, or with your own manifests.

Helm

The chart is published to GHCR as an OCI artifact, versioned with SMG releases (1.9.0 and later):

helm install smg oci://ghcr.io/smg-project/charts/smg \
  --version 1.15.0 \
  --namespace inference --create-namespace \
  -f values.yaml

A minimal values.yaml either lists workers or discovers them:

router:
  policy: cache_aware
  workerUrls:
    - http://worker-1.inference.svc:8000
    - http://worker-2.inference.svc:8000

For gRPC workers, use grpc:// URLs. The gateway loads the tokenizer from the tokenizer or model path each worker reports, and fetches it from the worker if that path doesn't load on the gateway. TensorRT-LLM workers report only a model ID, which the gateway uses the same way; set router.model to the model's Hugging Face ID or path when that ID is not one the gateway can load.

router:
  policy: cache_aware
  serviceDiscovery:
    enabled: true
    selector: "app=sglang-worker"
    port: 8000

The gateway watches pods with this label in the release namespace. Set router.serviceDiscovery.namespace to watch another namespace (the chart creates its Role and RoleBinding in the watched namespace, so the router can read those pods), or router.serviceDiscovery.clusterWide: true to watch all of them. router.serviceDiscovery is the legacy spelling of router.discovery.provider: kubernetes (set one, not both); the provider value also takes file, slurm, or consul — the chart README covers those.

The chart creates these resources (names assume the release is called smg):

Resource When
Deployment smg-router running docker.io/lightseekorg/smg:<chart appVersion> Always; a StatefulSet plus a headless Service when router.mesh.enabled
Service smg-router (ClusterIP): port 80 to the gateway's 30000, and 29000 for metrics Always
Service smg-router-admin exposing the gateway's admin listener inside the cluster router.adminPort is set
ServiceAccount smg serviceAccount.create (default true)
Secret smg-hf-token, passed to the router and workers as HF_TOKEN huggingface.token is set
Secret smg-auth holding the shared API key, mounted into the router pod and read with --api-key-file auth.apiKey is set and no auth.existingSecret replaces it
Secret smg-control-plane holding the control-plane API keys, mounted into the router pod and read with --control-plane-api-keys-file auth.controlPlane.apiKeys is set and no auth.controlPlane.existingSecret replaces it
Secret smg-history holding the history store's connection URL, or smg-secrets holding the Oracle credentials history.backend is postgres or redis with url set, or oracle with dsn, user, or password set, and no existingSecret replaces it
Role and RoleBinding allowing get, list, and watch on pods router.serviceDiscovery.enabled (or router.discovery.provider: kubernetes), router.mesh.enabled without peerUrls, or rbac.create; a ClusterRole and ClusterRoleBinding when discovery is cluster-wide
ServiceMonitor; a PrometheusRule with serving alerts router.metrics.serviceMonitor.enabled; prometheusRule.enabled (both need the Prometheus Operator CRDs)
Ingress, HorizontalPodAutoscaler, PodDisruptionBudget, Grafana dashboard ConfigMap router.ingress.enabled, router.autoscaling.enabled, router.podDisruptionBudget.enabled, grafana.dashboard.enabled
Deployment and Service for an engine worker One per workers[] entry

A few chart behaviors to know:

  • The router container runs the gateway image, so it accepts the same flags as smg launch from pip. router.extraArgs appends flags the values file does not cover.
  • The chart's schema limits router.policy to the launcher's --policy choices: cache_aware (the default), round_robin, power_of_two, least_load, passthrough, consistent_hashing, prefix_hash, manual, and random. bucket is a prefill-side policy, so the schema rejects it for router.policy; set it as router.disaggregation.prefillPolicy instead.
  • router.disaggregation deploys a prefill/decode or encode/prefill/decode fleet from values. mode: pd renders --pd-disaggregation and mode: epd renders --epd-disaggregation; empty, the default, is a regular fleet. The role policies prefillPolicy, decodePolicy, and encodePolicy render --prefill-policy, --decode-policy, and --encode-policy, each limited by the schema to the launcher's choices for that role (bucket only for prefillPolicy; encodePolicy takes random, round_robin, or consistent_hashing). The role workers are either listed or discovered. The static lists prefill, decode, and encode take entries with a url and, for prefill and encode, an optional bootstrapPort, rendered as --prefill URL [BOOTSTRAP_PORT], --decode URL, and --encode URL [BOOTSTRAP_PORT]. Under Kubernetes discovery, the pod selectors router.discovery.kubernetes.prefillSelector, decodeSelector, and encodeSelector find each role's pods instead; under Slurm discovery, the role job names router.discovery.slurm.prefillJobNames, decodeJobNames, and encodeJobNames name the jobs whose nodes serve each role. Values a disaggregated gateway would ignore at startup fail the render instead: a role given without its mode, the encode role outside epd, router.workerUrls or the regular Slurm jobNames under a mode, and workers[] engines when no role list or discovery selector addresses them. A mistyped role key fails the schema. PD Disaggregation covers setting up the role workers and their bootstrap ports; the PD routing concept explains what the modes do.
  • router.parser, rendered as --parser, picks the parser that reads a chat completion's reasoning and tool calls: symphony (one table per model family) or legacy (the per-model parsers). Empty, the default, renders no flag and is safe on every image: the image's own default applies (Symphony on releases after 1.13.0, the legacy parsers on a 1.13.0 image). A set value needs a gateway image of 1.13.0 or later; an earlier image rejects the flag and exits at startup. The choice applies to gRPC and ZMQ workers; an HTTP worker's engine parses its own output. A request's X-SMG-Parser header overrides the chosen parser for that request. If your values already pass --parser through router.extraArgs, remove it there when setting this value: the chart renders extraArgs last and the gateway takes the last occurrence of a repeated flag, so the leftover extraArgs value silently overrides router.parser. Parser Selection covers the two parsers and what router.toolCallParser and router.reasoningParser mean under each.
  • global.image.tag overrides the gateway version, which defaults to the chart's appVersion.
  • Engine images for workers[] resolve to ghcr.io/smg-project/smg:<tag> by default; workers[].image.registry and image.repository override the path. Set each worker's image.tag to the engine-specific tag (for example 1.15.0-vllm-v0.31.0); without it the tag falls back to global.image.tag or the chart's appVersion, a bare gateway version rather than an engine image tag.
  • history.backend selects the conversation-history backend: memory (the default), none (history off), postgres, redis, or oracle. For postgres and redis, set the store's url under history.<backend>, or name a Secret you manage in existingSecret; for oracle, set history.oracle.dsn, user, and password, or existingSecret. The connection URL and credentials reach the router from a Secret through environment variables, never in the pod's arguments. Data Connections covers each backend, including schema migrations.

Verify:

helm test smg -n inference
kubectl -n inference port-forward svc/smg-router 30000:80 &
curl http://localhost:30000/workers

The chart's README, values.yaml, and examples, under deploy/helm/smg in the smg repository, cover engine workers, multiple models, mesh HA, monitoring, and ingress.

Without Helm

Start SMG with service discovery:

smg launch \
  --service-discovery \
  --selector app=sglang-worker \
  --service-discovery-namespace inference \
  --service-discovery-port 8000 \
  --policy cache_aware

In a pod, pass the same flags as container args to the gateway image. SMG needs permission to watch pods in the namespace:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: smg-discovery
  namespace: inference
rules:
  - apiGroups: [""]
    resources: ["pods"]
    verbs: ["get", "list", "watch"]

Bind the Role to the gateway pod's service account with a RoleBinding; Service Discovery has the full manifests. Without --service-discovery-namespace, SMG watches pods in all namespaces and needs a ClusterRole instead.

Verify:

kubectl get pods -n inference -l app=sglang-worker
curl http://localhost:30000/workers

Core Setup

Operations

  • Monitoring — Prometheus metrics, tracing, and alerts
  • Logging — structured logs and aggregation patterns
  • TLS — HTTPS gateway configuration
  • Control Plane Auth — secure worker/tokenizer/WASM management endpoints

Reliability and Data

Advanced Features

Troubleshooting

Gateway starts but can't connect to worker

Symptoms: Gateway logs show connection errors.

Solutions:

  1. Verify the worker is running: curl http://localhost:8000/health
  2. Check network connectivity between gateway and worker
  3. If using Docker, ensure proper network configuration (--network host or Docker network)
Request times out

Symptoms: Requests hang or return 504 errors.

Solutions:

  1. Check worker health: curl http://localhost:30000/workers
  2. Raise the gateway's per-attempt timeout if long generations exceed it: --request-timeout-secs (default 1800) bounds each attempt at an HTTP worker or over a ZMQ link; gRPC workers have no deadline unless --grpc-request-timeout-secs sets one
  3. Check worker logs for errors
Model not found error

Symptoms: a 404 response with error code model_not_found and the message No worker available for model '<model>'.

Solutions:

  1. The model field in requests should match the model loaded on the worker
  2. Check available models: curl http://localhost:30000/v1/models