Getting Started¶
Shepherd Model Gateway (SMG) routes and manages LLM traffic across workers. This page gives you a fast path to a working gateway, then points you to feature-specific setup guides.
Install¶
pip install smgThe wheels on PyPI target the stable ABI (abi3), so each one works on CPython 3.9 and newer:
| Platform | Wheels |
|---|---|
| Linux x86_64 | manylinux2014 (glibc 2.17+), musllinux 1.1 |
| Linux aarch64 | manylinux 2.28 (glibc 2.28+), musllinux 1.1 |
| macOS | Intel (10.12+), Apple Silicon (11.0+) |
| Windows | x86_64 |
The package installs the smg command:
smg launchstarts the gateway alone, for workers you run yourselfsmg servestarts engine workers and then the gateway; the engine (vLLM, SGLang, TensorRT-LLM, or TokenSpeed) must be installed in the same environment
smg launch parses its flags in Python (smg.launch_router, which is also the gateway Docker image's entrypoint) and runs the Rust gateway in-process. It takes the Rust binary's full flag set; only a few spellings differ, such as the launcher's --control-plane-audit-enabled for the binary's --disable-audit-logging.
Nightly Linux x86_64 wheels are attached to dated prereleases in smg-project/artifacts. Install one with pip install <wheel URL>.
cargo install --locked smgThis compiles the gateway from crates.io and installs the smg binary (and its amg alias) into ~/.cargo/bin. --locked builds with the Cargo.lock published with the release. The binary takes the same flags through smg launch (or plain smg); smg serve is only in the Python package.
The build needs protoc, a C/C++ toolchain, and OpenSSL headers:
# Debian/Ubuntu
sudo apt-get install -y build-essential pkg-config libssl-dev protobuf-compiler# macOS (Homebrew)
brew install protobuf openssl@3
To compile OpenSSL from source instead of linking the system library, add --features vendored-openssl (needs perl and make).
Gateway only (no inference engine), for linux/amd64 and linux/arm64:
docker pull lightseekorg/smg:latest# Same image on GitHub Container Registry
docker pull ghcr.io/smg-project/smg:latest
Release tags are the bare version with no v prefix, for example lightseekorg/smg:1.14.0, and latest points at the newest release. GHCR carries gateway releases from 1.10.1 on; older release tags are on Docker Hub. Nightly builds are on GHCR only: ghcr.io/smg-project/smg:nightly, or nightly-<YYYYMMDD>-<short-sha> to pin one night. Dated nightly tags are pruned after a short retention window.
Gateway + engine (all-in-one images that route and serve), for linux/amd64 only:
docker pull ghcr.io/smg-project/smg:<version>-<engine>-<engine_version>Each release builds these engine images from the listed base images:
| Engine | Tag suffix | Base image | Nightly tag |
|---|---|---|---|
| vLLM | vllm-v0.31.0, vllm-v0.27.1, vllm-v0.26.0 |
vllm/vllm-openai |
nightly-vllm |
| SGLang | sglang-v0.5.21 |
lmsysorg/sglang |
nightly-sglang |
| TensorRT-LLM | trtllm-1.3.0rc24, trtllm-1.3.0rc23, trtllm-1.3.0rc22 |
nvcr.io/nvidia/tensorrt-llm/release |
nightly-trtllm |
| TokenSpeed | tokenspeed-<x.y.z.postYYYYMMDD> |
dated TokenSpeed nightly build | nightly-tokenspeed |
For example, this release publishes ghcr.io/smg-project/smg:1.14.0-vllm-v0.31.0 and ghcr.io/smg-project/smg:1.14.0-sglang-v0.5.21, both built from SMG 1.14.0. Browse every tag on GHCR or Docker Hub. Engine images for 1.9.0 and earlier were published as ghcr.io/lightseekorg/smg.
Each release's TokenSpeed image is built from the dated TokenSpeed nightly pinned in the SMG repository at build time, and its tag suffix carries that nightly's version. The older tokenspeed-tml release images belong to 1.11.0 and earlier, and they do not contain that release's SMG: they were built from SMG 1.7.0, so 1.11.0-tokenspeed-tml contains SMG 1.7.0. nightly-tokenspeed is built from current SMG.
# Install Rust
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
source "$HOME/.cargo/env"# Clone and build (needs the protoc and OpenSSL packages from the Cargo tab)
git clone https://github.com/smg-project/smg.git
cd smg
cargo build --release
The gateway binary is at ./target/release/smg. To build and install the Python package (which adds smg serve) from the same checkout:
pip install maturin
make python-installStep 1: Start SMG¶
Choose one of these startup paths.
Option A: All-in-one with smg serve¶
smg serve launches backend worker process(es) and then starts SMG with generated worker URLs.
smg serve \
--backend vllm \
--model meta-llama/Llama-3.1-8B-Instruct \
--data-parallel-size 2 \
--host 0.0.0.0 \
--port 30000smg serve \
--backend trtllm \
--model meta-llama/Llama-3.1-8B-Instruct \
--data-parallel-size 2 \
--host 0.0.0.0 \
--port 30000smg serve \
--backend tokenspeed \
--connection-mode zmq \
--model meta-llama/Llama-3.1-8B-Instruct \
--router-model-path meta-llama/Llama-3.1-8B-Instruct \
--data-parallel-size 2 \
--host 0.0.0.0 \
--port 30000TokenSpeed runs headless, and SMG connects to its engine core over ZMQ. --router-model-path gives the gateway the model name and tokenizer, which the engine doesn't report over ZMQ.
smg serve \
--backend sglang \
--model-path meta-llama/Llama-3.1-8B-Instruct \
--data-parallel-size 2 \
--connection-mode grpc \
--host 0.0.0.0 \
--port 30000This starts --data-parallel-size worker replicas, waits for each to become healthy, then starts the gateway. With --connection-mode zmq there is no wait: the gateway starts at once, and /readiness returns 503 until a worker has completed its handshake and the tokenizer has loaded.
To connect vLLM to its engine core directly instead of through gRPC, add --connection-mode zmq and --router-model-path <model>. See ZMQ Direct Workers.
| Option | Default | Description |
|---|---|---|
--backend |
sglang |
Inference backend: sglang, vllm, trtllm, or tokenspeed. The SMG_DEFAULT_BACKEND environment variable changes the default |
--connection-mode |
grpc |
Worker connection mode: grpc, http, or zmq. TensorRT-LLM supports only grpc and TokenSpeed only zmq; zmq also works with vllm and sglang |
--servicer-impl |
python |
Implementation of the vLLM gRPC servicer: python or rust. rust requires --backend vllm --connection-mode grpc; see Rust vLLM Servicer |
--host |
127.0.0.1 |
Router host |
--port |
8080 |
Router port (30000 with --backend sglang; see below) |
--data-parallel-size, --dp-size |
1 |
Number of worker replicas, each on its own GPU slice |
--worker-host |
127.0.0.1 |
Host for worker processes |
--worker-base-port |
31000 |
Base port for worker processes |
--worker-startup-timeout |
300 |
Seconds to wait for each worker to become healthy |
--enable-token-usage-details |
off | In http mode, start the engine with cached-token reporting (--enable-cache-report for SGLang, --enable-prompt-tokens-details for vLLM) |
Gateway options take a --router- prefix (for example --router-policy or --router-model-path); other flags are passed to the engine. When the backend's own CLI also defines --host or --port, its definition replaces the one above: with --backend sglang the router defaults to 127.0.0.1:30000, and with vLLM in http mode to 0.0.0.0:8000. Pass both flags to be explicit.
Option B: Launch gateway only with smg launch¶
Use this when workers are already running or managed by another platform.
For gRPC workers:
smg launch \
--worker-urls grpc://localhost:50051 \
--model-path meta-llama/Llama-3.1-8B-Instruct \
--policy round_robin \
--host 0.0.0.0 \
--port 30000For HTTP workers:
smg launch \
--worker-urls http://localhost:8000 \
--policy round_robin \
--host 0.0.0.0 \
--port 30000Step 2: Verify Core Endpoints¶
Health:
curl http://localhost:30000/health
curl http://localhost:30000/readinessOpenAI-compatible chat completions:
curl http://localhost:30000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "meta-llama/Llama-3.1-8B-Instruct",
"messages": [{"role": "user", "content": "Say hello in one sentence."}]
}'Responses API:
curl http://localhost:30000/v1/responses \
-H "Content-Type: application/json" \
-d '{
"model": "meta-llama/Llama-3.1-8B-Instruct",
"input": "Say hello in one sentence."
}'Step 3: Choose Your Setup Track¶
Core Deployment¶
Operations and Security¶
Reliability and Data¶
Advanced Features¶
- Load Balancing
- KV Events Cache-Aware Routing
- Tokenizer Caching
- MCP in Responses API
- External Providers
- RL Control Plane
Worker Startup Recipes (Standalone)¶
Use these when workers are not started via smg serve. Each command starts one worker; register it with smg launch --worker-urls using a grpc:// or http:// URL, as in Option B. The SMG engine images for vLLM, SGLang, TensorRT-LLM, and TokenSpeed already contain the engine and the SMG gRPC servicer (see Install).
gRPC mode needs the SMG servicer: pip install "vllm[grpc]" adds smg-grpc-servicer[vllm].
# gRPC worker (grpc://<host>:50051)
python -m vllm.entrypoints.grpc_server \
--model meta-llama/Llama-3.1-8B-Instruct \
--host 0.0.0.0 \
--port 50051 \
--tensor-parallel-size 1# HTTP worker (http://<host>:8000)
python -m vllm.entrypoints.openai.api_server
--model meta-llama/Llama-3.1-8B-Instruct
--host 0.0.0.0
--port 8000
Setting SMG_VLLM_SERVICER_IMPL=rust in the gRPC worker's environment serves the same contract from the Rust servicer instead of the default Python one; see Rust vLLM Servicer.
SGLang depends on smg-grpc-servicer, so gRPC mode works after pip install sglang. The servicer supports SGLang 0.5.21 and later.
# gRPC worker (grpc://<host>:50051)
python -m sglang.launch_server \
--model-path meta-llama/Llama-3.1-8B-Instruct \
--host 0.0.0.0 \
--port 50051 \
--grpc-mode# HTTP worker (http://<host>:8000)
python -m sglang.launch_server
--model-path meta-llama/Llama-3.1-8B-Instruct
--host 0.0.0.0
--port 8000
In gRPC mode SGLang also serves an HTTP sidecar (profiling, plus /metrics with --enable-metrics).
Setting SMG_SGLANG_SERVICER_IMPL=rust in the gRPC worker's environment serves the same contract from the Rust servicer instead of the default Python one (needs the smg wheel; the HTTP sidecar stays off on that path); see Rust SGLang Servicer.
gRPC serving is built into TensorRT-LLM, so it needs no SMG servicer.
# gRPC worker (grpc://<host>:50051)
python -m tensorrt_llm.commands.serve serve \
meta-llama/Llama-3.1-8B-Instruct \
--grpc \
--host 0.0.0.0 \
--port 50051 \
--backend pytorch \
--tp_size 1The smg-grpc-servicer package serves the same gRPC contract through python -m smg_grpc_servicer.trtllm --model <model> --host 0.0.0.0 --port 50051, with the LLM API's keyword arguments passed as JSON through --llm-args. Its --servicer-impl rust (or SMG_TRTLLM_SERVICER_IMPL=rust in the environment, which the flag overrides) serves that contract from the Rust servicer instead of the engine's own (needs the smg wheel).
Install TokenSpeed from source, then pip install "smg-grpc-servicer[tokenspeed]".
# gRPC worker (grpc://<host>:50051)
python -m smg_grpc_servicer.tokenspeed \
--model meta-llama/Llama-3.1-8B-Instruct \
--host 0.0.0.0 \
--port 50051 \
--grammar-backend xgrammarTokenSpeed's default grammar backend is none, which makes it reject requests that need constrained decoding, such as tool_choice set to required or a named function, or a JSON response_format. Add --enable-output-logprobs if clients request logprobs; without it they come back empty.
Setting SMG_TOKENSPEED_SERVICER_IMPL=rust in the gRPC worker's environment serves the same contract from the Rust servicer instead of the default Python one (needs the smg wheel); see Rust TokenSpeed Servicer.
Apple Silicon only. Install the servicer with pip install "smg-grpc-servicer[mlx]".
# gRPC worker (grpc://<host>:50051)
python -m smg_grpc_servicer.mlx.server \
--model mlx-community/Qwen3-0.6B-4bit \
--host 0.0.0.0 \
--port 50051For ZMQ direct workers (ipc:// or tcp:// URLs), the engine runs headless and connects to the gateway instead; see ZMQ Workers.
PD Disaggregation Workers¶
For prefill-decode disaggregation, start separate prefill and decode workers, each on its own GPUs (on one host, pin them with CUDA_VISIBLE_DEVICES):
# Prefill worker
python -m sglang.launch_server \
--model-path meta-llama/Llama-3.1-8B-Instruct \
--host 0.0.0.0 \
--port 50051 \
--grpc-mode \
--disaggregation-mode prefill \
--disaggregation-bootstrap-port 8998# Decode worker
python -m sglang.launch_server
--model-path meta-llama/Llama-3.1-8B-Instruct
--host 0.0.0.0
--port 50052
--grpc-mode
--disaggregation-mode decode
Start SMG with the prefill worker's bootstrap port after its URL:
smg launch \
--pd-disaggregation \
--prefill grpc://localhost:50051 8998 \
--decode grpc://localhost:50052 \
--model-path meta-llama/Llama-3.1-8B-Instruct \
--host 0.0.0.0 \
--port 30000# Prefill worker
python -m sglang.launch_server \
--model-path meta-llama/Llama-3.1-8B-Instruct \
--host 0.0.0.0 \
--port 8000 \
--disaggregation-mode prefill \
--disaggregation-bootstrap-port 8998# Decode worker
python -m sglang.launch_server
--model-path meta-llama/Llama-3.1-8B-Instruct
--host 0.0.0.0
--port 8001
--disaggregation-mode decode
Start SMG with the prefill worker's bootstrap port after its URL:
smg launch \
--pd-disaggregation \
--prefill http://localhost:8000 8998 \
--decode http://localhost:8001 \
--host 0.0.0.0 \
--port 30000vLLM uses NIXL for KV cache transfer between prefill and decode workers:
# Prefill worker
VLLM_NIXL_SIDE_CHANNEL_PORT=5600 \
python -m vllm.entrypoints.grpc_server \
--model meta-llama/Llama-3.1-8B-Instruct \
--host 0.0.0.0 \
--port 50051 \
--kv-transfer-config '{"kv_connector":"NixlConnector","kv_role":"kv_producer"}'# Decode worker
VLLM_NIXL_SIDE_CHANNEL_PORT=5601
python -m vllm.entrypoints.grpc_server
--model meta-llama/Llama-3.1-8B-Instruct
--host 0.0.0.0
--port 50052
--kv-transfer-config '{"kv_connector":"NixlConnector","kv_role":"kv_consumer"}'
Start SMG (no bootstrap ports needed — NIXL handles KV transfer):
smg launch \
--pd-disaggregation \
--prefill grpc://localhost:50051 \
--decode grpc://localhost:50052 \
--model-path meta-llama/Llama-3.1-8B-Instruct \
--host 0.0.0.0 \
--port 30000SMG drives vLLM disaggregation over the OpenAI-compatible HTTP server too, with no gRPC servicer:
# Prefill worker
VLLM_NIXL_SIDE_CHANNEL_PORT=5600 \
vllm serve meta-llama/Llama-3.1-8B-Instruct \
--host 0.0.0.0 \
--port 8000 \
--kv-transfer-config '{"kv_connector":"NixlConnector","kv_role":"kv_producer"}'# Decode worker
VLLM_NIXL_SIDE_CHANNEL_PORT=5601
vllm serve meta-llama/Llama-3.1-8B-Instruct
--host 0.0.0.0
--port 8001
--kv-transfer-config '{"kv_connector":"NixlConnector","kv_role":"kv_consumer"}'
Start SMG in PD mode without startup workers:
smg launch --pd-disaggregation --host 0.0.0.0 --port 30000The vLLM HTTP server does not report its KV connector, so register both workers with it. Without kv_connector, requests still succeed but decode recomputes every prompt:
curl -X POST http://localhost:30000/workers \
-H "Content-Type: application/json" \
-d '{"url": "http://localhost:8000", "worker_type": "prefill", "kv_connector": "NixlConnector"}'curl -X POST http://localhost:30000/workers
-H "Content-Type: application/json"
-d '{"url": "http://localhost:8001", "worker_type": "decode", "kv_connector": "NixlConnector"}'
See PD Disaggregation for the Mooncake backend, TokenSpeed and EPD, Kubernetes discovery, and scaling.
Send a Request¶
curl http://localhost:30000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "meta-llama/Llama-3.1-8B-Instruct",
"messages": [
{"role": "user", "content": "What is the capital of France?"}
],
"max_tokens": 50
}'Expected response:
{
"id": "chatcmpl-abc123",
"object": "chat.completion",
"model": "meta-llama/Llama-3.1-8B-Instruct",
"choices": [
{
"index": 0,
"message": {
"role": "assistant",
"content": "The capital of France is Paris."
},
"finish_reason": "stop"
}
],
"usage": {
"prompt_tokens": 14,
"completion_tokens": 8,
"total_tokens": 22
}
}Verify Health¶
# Gateway health
curl http://localhost:30000/health
# Worker status
curl http://localhost:30000/workersDeploy with Docker¶
For local deployment, run the gateway image and point it at your worker. Container arguments go to the image's entrypoint, python3 -m smg.launch_router, so they are the same flags as smg launch from pip:
docker run -d \
--name smg \
-p 30000:30000 \
-p 29000:29000 \
--add-host=host.docker.internal:host-gateway \
lightseekorg/smg:latest \
--worker-urls http://host.docker.internal:8000 \
--policy cache_awareThe gateway listens on 0.0.0.0:30000 and serves Prometheus metrics on 0.0.0.0:29000 by default. --add-host makes host.docker.internal resolve to the Docker host on Linux; Docker Desktop provides that name already. The container runs as a non-root user (UID 65532), so files you mount into it must be readable by that user. ghcr.io/smg-project/smg:latest is the same image.
Verify:
docker ps | grep smg
curl http://localhost:30000/healthAll-in-one with engine images¶
Engine images use smg as their entrypoint, so pass serve to start the worker and the gateway in one container. Set --host 0.0.0.0: smg serve binds the gateway to 127.0.0.1 by default, which a published port cannot reach. The Docker tab under Install lists the engine tags each release builds.
docker run -d --gpus all \
--name smg \
-p 30000:30000 \
-v /path/to/models:/models \
ghcr.io/smg-project/smg:1.14.0-sglang-v0.5.21 \
serve \
--backend sglang \
--model-path /models/meta-llama/Llama-3.1-8B-Instruct \
--host 0.0.0.0 \
--port 30000docker run -d --gpus all \
--name smg \
-p 30000:30000 \
-v /path/to/models:/models \
ghcr.io/smg-project/smg:1.14.0-vllm-v0.31.0 \
serve \
--backend vllm \
--model /models/meta-llama/Llama-3.1-8B-Instruct \
--host 0.0.0.0 \
--port 30000Each engine image sets SMG_DEFAULT_BACKEND to its engine, so --backend is optional. Workers connect over gRPC by default. TensorRT-LLM images take --backend trtllm and support only gRPC. TokenSpeed needs --connection-mode zmq and --router-model-path (see ZMQ Workers).
The TokenSpeed release images from 1.11.0 and earlier cannot serve TokenSpeed this way: the SMG 1.7.0 they contain has no TokenSpeed backend or ZMQ mode in smg serve.
Verify:
curl http://localhost:30000/health
curl http://localhost:30000/v1/modelsDeploy to Kubernetes (Quick Start)¶
Run SMG in the cluster with the Helm chart, or with your own manifests.
Helm¶
The chart is published to GHCR as an OCI artifact, versioned with SMG releases (1.9.0 and later):
helm install smg oci://ghcr.io/smg-project/charts/smg \
--version 1.14.0 \
--namespace inference --create-namespace \
-f values.yamlA minimal values.yaml either lists workers or discovers them:
router:
policy: cache_aware
workerUrls:
- http://worker-1.inference.svc:8000
- http://worker-2.inference.svc:8000For gRPC workers, use grpc:// URLs. The gateway loads the tokenizer from the tokenizer or model path each worker reports, and fetches it from the worker if that path doesn't load on the gateway. TensorRT-LLM workers report no path, so for them set router.model to the model's Hugging Face ID or path.
router:
policy: cache_aware
serviceDiscovery:
enabled: true
selector: "app=sglang-worker"
port: 8000The gateway watches pods with this label in the release namespace. Set router.serviceDiscovery.namespace to watch another namespace (the chart creates its Role and RoleBinding in the watched namespace, so the router can read those pods), or router.serviceDiscovery.clusterWide: true to watch all of them. router.serviceDiscovery is the legacy spelling of router.discovery.provider: kubernetes (set one, not both); the provider value also takes file, slurm, or consul — the chart README covers those.
The chart creates these resources (names assume the release is called smg):
| Resource | When |
|---|---|
Deployment smg-router running docker.io/lightseekorg/smg:<chart appVersion> |
Always; a StatefulSet plus a headless Service when router.mesh.enabled |
Service smg-router (ClusterIP): port 80 to the gateway's 30000, and 29000 for metrics |
Always |
ServiceAccount smg |
serviceAccount.create (default true) |
Secret smg-hf-token, passed to the router and workers as HF_TOKEN |
huggingface.token is set |
Secret smg-auth holding the shared API key, mounted into the router pod and read with --api-key-file |
auth.apiKey is set and no auth.existingSecret replaces it |
Secret smg-history holding the history store's connection URL, or smg-secrets holding the Oracle credentials |
history.backend is postgres or redis with url set, or oracle with dsn, user, or password set, and no existingSecret replaces it |
Role and RoleBinding allowing get, list, and watch on pods |
router.serviceDiscovery.enabled (or router.discovery.provider: kubernetes), router.mesh.enabled without peerUrls, or rbac.create; a ClusterRole and ClusterRoleBinding when discovery is cluster-wide |
ServiceMonitor; a PrometheusRule with serving alerts |
router.metrics.serviceMonitor.enabled; prometheusRule.enabled (both need the Prometheus Operator CRDs) |
Ingress, HorizontalPodAutoscaler, PodDisruptionBudget, Grafana dashboard ConfigMap |
router.ingress.enabled, router.autoscaling.enabled, router.podDisruptionBudget.enabled, grafana.dashboard.enabled |
Deployment and Service for an engine worker |
One per workers[] entry |
A few chart behaviors to know:
- The router container runs the gateway image, so it accepts the same flags as
smg launchfrom pip.router.extraArgsappends flags the values file does not cover. - The chart's schema limits
router.policytocache_aware,round_robin,power_of_two,manual,random, andprefix_hash. - The gateway reads a chat completion's reasoning and tool calls with the Symphony parser (one table per model family) by default; passing
--parser legacythroughrouter.extraArgskeeps the per-model legacy parsers. The choice applies to gRPC and ZMQ workers; an HTTP worker's engine parses its own output. A request'sX-SMG-Parserheader overrides the chosen parser for that request.router.toolCallParserandrouter.reasoningParser, rendered as--tool-call-parserand--reasoning-parser, name the parser or Symphony table for models the registry does not know. Parser Selection covers the two parsers and these flags under each. global.image.tagoverrides the gateway version, which defaults to the chart'sappVersion.- Engine images for
workers[]resolve toghcr.io/smg-project/smg:<tag>by default;workers[].image.registryandimage.repositoryoverride the path. Set each worker'simage.tagto the engine-specific tag (for example1.14.0-vllm-v0.31.0); without it the tag falls back toglobal.image.tagor the chart'sappVersion, a bare gateway version rather than an engine image tag. history.backendselects the conversation-history backend:memory(the default),none(history off),postgres,redis, ororacle. Forpostgresandredis, set the store'surlunderhistory.<backend>, or name a Secret you manage inexistingSecret; fororacle, sethistory.oracle.dsn,user, andpassword, orexistingSecret. The connection URL and credentials reach the router from a Secret through environment variables, never in the pod's arguments. Data Connections covers each backend, including schema migrations.
Verify:
helm test smg -n inference
kubectl -n inference port-forward svc/smg-router 30000:80 &
curl http://localhost:30000/workersThe chart's README, values.yaml, and examples, under deploy/helm/smg in the smg repository, cover engine workers, multiple models, mesh HA, monitoring, and ingress.
Without Helm¶
Start SMG with service discovery:
smg launch \
--service-discovery \
--selector app=sglang-worker \
--service-discovery-namespace inference \
--service-discovery-port 8000 \
--policy cache_awareIn a pod, pass the same flags as container args to the gateway image. SMG needs permission to watch pods in the namespace:
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: smg-discovery
namespace: inference
rules:
- apiGroups: [""]
resources: ["pods"]
verbs: ["get", "list", "watch"]Bind the Role to the gateway pod's service account with a RoleBinding; Service Discovery has the full manifests. Without --service-discovery-namespace, SMG watches pods in all namespaces and needs a ClusterRole instead.
Verify:
kubectl get pods -n inference -l app=sglang-worker
curl http://localhost:30000/workersNavigate by Category¶
Core Setup¶
- Multiple Workers — connect local or external worker endpoints
- gRPC Workers — gateway-side tokenization, parsing, and tool handling
- ZMQ Direct Workers — headless engines with no engine API server in the path
- PD Disaggregation — split prefill and decode paths
- Service Discovery — Kubernetes pod-based worker registration
Operations¶
- Monitoring — Prometheus metrics, tracing, and alerts
- Logging — structured logs and aggregation patterns
- TLS — HTTPS gateway configuration
- Control Plane Auth — secure worker/tokenizer/WASM management endpoints
Reliability and Data¶
- Reliability Controls — concurrency limits, retries, and circuit breakers
- Data Connections — history backend setup for PostgreSQL, Redis, and Oracle
- Tokenization and Parsing APIs — tokenize, detokenize, and parser endpoints
Advanced Features¶
- Load Balancing — policy selection and tuning
- KV Events Cache-Aware Routing — prefix routing driven by engine KV cache events
- Tokenizer Caching — L0/L1 cache setup for gRPC mode
- MCP in Responses API — configure and execute MCP tools through
/v1/responses - External Providers — route to OpenAI, Anthropic, Gemini, and xAI backends
- RL Control Plane — pause, refit, and resume inference engines from an RL training loop
Troubleshooting¶
Gateway starts but can't connect to worker
Symptoms: Gateway logs show connection errors.
Solutions:
- Verify the worker is running:
curl http://localhost:8000/health - Check network connectivity between gateway and worker
- If using Docker, ensure proper network configuration (
--network hostor Docker network)
Request times out
Symptoms: Requests hang or return 504 errors.
Solutions:
- Check worker health:
curl http://localhost:30000/workers - Raise the gateway's per-attempt timeout if long generations exceed it:
--request-timeout-secs(default1800) bounds each attempt at an HTTP worker or over a ZMQ link; gRPC workers have no deadline unless--grpc-request-timeout-secssets one - Check worker logs for errors
Model not found error
Symptoms: a 404 response with error code model_not_found and the message No worker available for model '<model>'.
Solutions:
- The
modelfield in requests should match the model loaded on the worker - Check available models:
curl http://localhost:30000/v1/models