High Availability¶
Run several SMG routers as one cluster. With --enable-mesh, routers form a peer-to-peer mesh: they track each other's membership with a SWIM-style gossip protocol, replicate the workers each router registers, and share cache-aware routing updates, so every router routes with the cluster's workers and cache affinity.
The mesh does not steer client traffic. Put the routers behind a load balancer or a Kubernetes Service; that is what moves clients to the remaining routers when one fails.
Overview¶
Fault Tolerance¶
When a router fails, your load balancer sends its clients to the others, which keep the cache-aware routing state the failed router shared. Workers that only the failed router registered are removed shortly after it is marked DOWN, so register each worker on more than one router (Worker Registry Sync).
Scalability¶
Add routers without restarting the others. A new router joins through a running peer or Kubernetes router discovery and picks up the cluster's workers.
State Synchronization¶
Workers registered on any router and cache-aware routing tree updates reach every peer. Rate limits, sticky sessions, and other policy state stay local to each router.
Rolling Updates¶
Replace routers one at a time: each announces its leave to the mesh and drains in-flight requests on shutdown. With Kubernetes router discovery, a replaced pod rejoins as soon as it is Ready.
Mesh Architecture¶
clients
│
load balancer or Kubernetes Service
┌────────────────┼────────────────┐
▼ ▼ ▼
┌──────────┐ ┌──────────┐ ┌──────────┐
│ router A │◀───▶│ router B │◀───▶│ router C │ mesh: gRPC on --mesh-port
└────┬─────┘ └────┬─────┘ └────┬─────┘
└────────────────┼────────────────┘
▼
workersEvery router is equal: there is no leader and no quorum. Each pair of routers (A and C included) keeps one sync stream over the mesh port, and state converges without coordination.
Gossip Membership¶
SWIM-style membership and failure detection.
- One probe per second to a random peer
- Indirect probes through up to three peers
- Status changes broadcast to live peers
Sync Streams¶
One bidirectional gRPC stream per pair of routers.
- Opened by the router whose name sorts first
- Each side sends pending updates every second
- Kept alive by a heartbeat every 15 seconds
- Closed after 60 seconds without a frame
CRDT Worker State¶
Conflict-free replicated keys for worker state.
- No locks or coordination
- Resent until each peer acknowledges
- Relayed through intermediate peers
Cache-Tree Updates¶
Best-effort broadcasts for cache-aware routing.
- Batched per model every gossip round
- Unknown prefixes trigger a repair from a peer
- Dropped updates are not resent
Configuration¶
Quick Start¶
Start the first router, then point the next one at it with --mesh-peer-urls:
# Router 1 (bootstrap)
smg launch --enable-mesh --mesh-advertise-host 10.0.0.1 --mesh-port 39527 \
--worker-urls http://10.0.1.1:8000 http://10.0.1.2:8000
# Router 2, joining router 1
smg launch --enable-mesh --mesh-advertise-host 10.0.0.2 --mesh-port 39527 \
--mesh-peer-urls 10.0.0.1:39527Router 2 has no workers of its own; it imports router 1's workers through the mesh.
Imported workers last only while the router that registered them stays in the cluster — if router 1 goes down for good, router 2 removes them after a short grace (see Worker Registry Sync). To keep workers served through a router failure, register them on every router, as under Basic Configuration.
Command Line Options¶
| Flag | Default | Description |
|---|---|---|
--enable-mesh |
false |
Start the mesh listener and join the cluster |
--mesh-server-name |
Mesh_ plus 4 random alphanumeric characters |
Node name, unique within the cluster. Must be non-empty and must not contain :. With router discovery, set it to the pod name |
--mesh-host |
0.0.0.0 |
Bind address for the mesh listener (an IP address) |
--mesh-advertise-host |
--mesh-host |
IP address that peers dial. Required when --mesh-host is an unspecified address such as 0.0.0.0 |
--mesh-port |
39527 |
Mesh port, used for both binding and advertising. 0 is rejected |
--mesh-peer-urls |
(none) | Bootstrap peers as IP:port, space- or comma-separated. While the membership table holds no other member, the entries are dialed in turn until one answers |
--mesh-tls-ca-cert |
(none) | CA certificate (PEM) that mesh peer certificates must chain to. Set together with --mesh-tls-cert and --mesh-tls-key to enable mesh mTLS |
--mesh-tls-cert |
(none) | This node's certificate (PEM), with the node's IP as a SAN |
--mesh-tls-key |
(none) | This node's private key (PKCS#8 PEM) |
--router-selector |
(none) | Label selector for Kubernetes router discovery. Takes effect only when Kubernetes worker discovery is selected, with --service-discovery or --discovery-provider kubernetes |
Mesh mTLS¶
--mesh-tls-ca-cert, --mesh-tls-cert, and --mesh-tls-key encrypt the mesh port and authenticate both ends of every mesh connection. The three flags come together: a router started with only one or two of them refuses to start, and the error names the missing flags.
smg launch --enable-mesh --mesh-advertise-host 10.0.0.1 \
--mesh-tls-ca-cert /etc/smg/mesh/ca.pem \
--mesh-tls-cert /etc/smg/mesh/node.pem \
--mesh-tls-key /etc/smg/mesh/node-key.pemWith the flags set, the mesh listener serves TLS with the node's certificate and accepts only peers that present a certificate chaining to the CA. Every dial to a peer presents this node's certificate and verifies the peer's against the CA.
Issue every router a certificate from the same CA that carries the node's IP — the address its peers dial, so its --mesh-advertise-host — as an IP SAN. A dial verifies the peer's certificate against the address it dials, and mesh addresses are always IPs (see the note above), so a certificate with only a hostname never matches. An IPv6 node's SAN is the unbracketed address: fd00::7, not [fd00::7].
The private key must be PKCS#8 PEM. Each of the three files must exist and be readable at startup, or the router fails to start with an error naming the flag whose file it cannot read; the router also warns when the key file is readable or writable by its group or by others. The mesh listener reads the certificate and key once at startup and does not reload them; to rotate certificates, restart the router.
Configure every router in the cluster the same way: a router with the flags serves and dials only TLS, a router without them only plaintext, so a mixed cluster's routers cannot reach each other.
Python Entrypoint¶
The Python CLI (smg launch) and the Docker image's entrypoint (python3 -m smg.launch_router) accept the same mesh flags with the same defaults and address checks, so container args can pass them directly:
smg launch --enable-mesh --mesh-host 0.0.0.0 --mesh-advertise-host 10.0.0.11 --mesh-port 39527smg serve takes router flags with a --router- prefix, for example --router-enable-mesh and --router-mesh-advertise-host.
Basic Configuration¶
Node 1 (Bootstrap)
smg launch --enable-mesh \
--mesh-server-name node1 \
--mesh-host 0.0.0.0 \
--mesh-advertise-host 10.0.0.11 \
--mesh-port 39527 \
--worker-urls http://10.0.1.1:8000 http://10.0.1.2:8000Node 2 (Join)
smg launch --enable-mesh \
--mesh-server-name node2 \
--mesh-host 0.0.0.0 \
--mesh-advertise-host 10.0.0.12 \
--mesh-port 39527 \
--mesh-peer-urls 10.0.0.11:39527 \
--worker-urls http://10.0.1.1:8000 http://10.0.1.2:8000Node 3 (Join)
smg launch --enable-mesh \
--mesh-server-name node3 \
--mesh-host 0.0.0.0 \
--mesh-advertise-host 10.0.0.13 \
--mesh-port 39527 \
--mesh-peer-urls 10.0.0.11:39527 \
--worker-urls http://10.0.1.1:8000 http://10.0.1.2:8000Gossip Protocol¶
Membership and Failure Detection¶
- Probe: Every second, each router pings one random peer that is not
DOWNorLEAVING. The ping carries the sender's membership table, and the receiver adopts any entry with a higher version. - Indirect probe: If the ping fails, the router asks up to three other
ALIVEpeers to ping the target (ping-req). - Suspect, then down: If no probe reaches the target, it moves from
ALIVEtoSUSPECTED, or fromSUSPECTEDtoDOWN, and the new status is broadcast to everyALIVEpeer. - Sync stream: After a successful direct probe of an
ALIVEpeer, the router whose name sorts first opens the pair's sync stream if none is open. A probe that only a relay answered opens none: the pair's own link has just failed.
Node Status States¶
| Status | Description |
|---|---|
ALIVE |
Reachable. Every router starts in this state |
SUSPECTED |
A probe failed: neither the direct ping nor any indirect ping reached the router. Still probed. Router discovery also sets it for a known pod that is not Ready, unless that router is already DOWN |
DOWN |
A probe failed again while SUSPECTED, or the router's pod was deleted or is terminating. Peers stop probing it |
LEAVING |
The router announced a graceful leave, as every router does when its graceful shutdown begins. Peers stop probing it |
INIT |
Defined in the protocol but not used |
Failure Detection Timing¶
| Phase | Value |
|---|---|
| Gossip round (probe, send, and batching cadence) | 1 second |
| Indirect probe fan-out | Up to 3 ALIVE peers |
| Probe connect / request timeout | 5 seconds / 10 seconds |
| Redial backoff after a failed probe of the same peer | 1 second, doubling up to 60 seconds |
| Sync stream heartbeat (sent by the dialing router, answered by the accepting one) | 15 seconds |
| Sync stream idle timeout (no frame received) | 60 seconds |
These values are fixed; there are no flags to tune them.
State Synchronization¶
Synchronized State Types¶
Worker Registry¶
Workers each router registers, shared with every peer.
- URL, model, health, and load
- Worker spec, without its API key
- ZMQ workers (
ipc://andtcp://) excluded
Routing Trees¶
Cache-aware routing state shared across routers.
- HTTP string trees and gRPC token trees
- Tree inserts, sent as per-model deltas
- Repair from a peer for unknown prefixes
Membership¶
The cluster's node table.
- Node names, addresses, and status
- Carried on every gossip ping
Stays Local¶
Each router keeps its own:
- Rate limits and admission queues
- Sticky-session (
manualpolicy) assignments - Health checks and circuit-breaker state
- Routing policy and its settings
Because each router enforces rate limits on its own, the cluster-wide limit is roughly the per-router limit times the number of routers. Sticky-session assignments are also per router, so the same routing key can map to different workers on different routers.
Worker Registry Sync¶
Each router publishes the workers it registered itself, whether from --worker-urls, the worker API, or service discovery, whenever one is added, changes status, or is removed. Peers import them into their own registries:
- Imports are health-checked locally. A peer rebuilds the worker from its published spec and probes it itself. The owner's health flag sets the import's initial state and afterwards only nudges it: healthy promotes a pending or not-ready import to ready, and unhealthy demotes a ready one. When the owner replaces a worker in place — a new model or spec at the same URL — peers rebuild their imports from the republished spec.
- Workers are matched by URL. A router that registered a URL itself keeps its own worker and ignores peers' copies. Give a worker the same URL on every router, and never a loopback address: a peer that imports
http://127.0.0.1:8000routes to its own host. - Only the owner removes a worker. Removing a worker on the router that registered it removes it everywhere. Deleting an imported worker through another router's API lasts at most until that router's next reconcile pass (every 30 seconds) imports it again.
- A removed worker is not re-imported from peers. Once a worker is removed on the router that registered it, that router stops importing the endpoint from the mesh: a peer's copy — a twin registration of the same URL still published by another router, or a lagging peer's stale state — is held back instead of imported again. For a worker registered by service discovery, the hold is tied to the removed instance: a different instance published at the same address (for example, a new pod reusing the IP and port) imports as usual, but a peer copy that names no instance is held back. A removed worker that had no instance identity — one you registered yourself — holds back every peer copy of its URL. Registering a worker at that URL on that router re-admits the endpoint. The first copy held back per endpoint is logged, and the hold lives in memory — up to 4096 removed endpoints, oldest dropped first — so a router restart clears it.
- A departed router's workers are removed. From the moment the cluster state shows a router
DOWNorLEAVING, peers import nothing new from it, so a router that joins afterwards never picks up its workers. Once the router has been marked gone for 20 seconds, peers also remove the workers they already imported from it at their next reconcile pass (every 30 seconds). A shorter blip removes nothing, and a router that returns has its workers imported again. Workers a peer registered itself — the same URL given to several routers — are its own, not imports, and stay. - API keys stay local. The published spec leaves out the worker's
api_key.
Cache-Aware State Sync¶
The cache_aware policy keeps an approximate prefix tree per model: a string tree for HTTP requests and a token tree for gRPC requests. With the mesh on, routing decisions made through these trees are shared:
- Publish: After choosing a worker, the router records it in its own tree and queues a delta: a hash of the request's prefix path plus the worker URL. Once per gossip round, each model's queued deltas go to every connected peer as one batch. Picks made by the KV-pressure fallback land in the router's own tree but are not broadcast.
- Apply: A peer that already knows that prefix path adds the worker to it in its own tree.
- Repair: A peer that does not know the path, such as a router that just started, asks a random
ALIVEpeer for its whole tree for that model and tree type and replays it. The tree arrives in pages of up to about 2 MiB; a repair that makes no progress for 5 seconds is retried, preferably with another peer, up to 3 times. Unknown prefixes for the same model and tree type that arrive meanwhile are folded into the repair in flight.
The default policy, per-model cache-aware policies, and the prefill, decode, and encode policies of disaggregated mode all take part, with no flag beyond --enable-mesh.
What consistency to expect:
- Approximate. Deltas and repairs move the trees toward each other, but they are not guaranteed to match: a dropped delta is not resent, and a repair starts only when a later delta names a path the peer does not know. Each router still evicts its own tree on its own schedule (
--eviction-interval,--max-tree-size); evictions are not shared. - Best effort. A delta batch that a peer's stream cannot accept is dropped, not resent. The peer catches up through repair the next time it sees a prefix it does not know.
- Survives a router failure. Routing decisions that a failed router already shared stay in its peers' trees, so traffic that moves to them keeps that cache affinity.
Not synchronized: event-driven cache-aware routing (KV events), which each router builds from the workers' KV event streams, and the hash placement index (--cache-index hash).
How State Is Replicated¶
| Key prefix | Channel | Delivery | Carries |
|---|---|---|---|
worker: |
CRDT | Last-writer-wins by Lamport timestamp, then replica. Resent until each peer acknowledges; relayed through peers | One key per worker, written only by the router that owns it |
rl: |
CRDT | Epoch-max-wins | Rate-limit counter shards. Registered, but nothing writes to it |
td: |
Stream, broadcast | Best effort, once per gossip round, to directly connected peers | Cache-tree deltas, one entry per model per round |
tree:req:, tree:page: |
Stream, targeted | Retried after 5 seconds without progress, up to 3 times | Cache-tree repair requests and pages |
Worker-state merges order updates with Lamport clocks rather than wall-clock time, so clock skew between routers does not decide which update wins.
Deployment Patterns¶
Static Peers¶
For VMs and bare metal. Give each router a unique name, its own IP as --mesh-advertise-host, and another router's address in --mesh-peer-urls.
- Start routers in any order; the peer is dialed until it joins
- Restart an isolated router under a new name
Kubernetes Router Discovery¶
Routers find each other by pod label, with no bootstrap peer to configure.
- Pods join when Ready and are marked
DOWNwhen deleted - Restarted pods rejoin under their pod name
Cluster Size¶
Any number of routers works. There is no leader election or quorum, so odd counts are not required, and each router keeps serving with the state it has. Every pair of routers keeps one sync stream, so N routers hold N×(N−1)/2 streams and each router sends its updates to N−1 peers.
Kubernetes Deployment¶
StatefulSet Configuration¶
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: smg
namespace: inference
spec:
serviceName: smg-mesh
replicas: 3
selector:
matchLabels:
app: smg
template:
metadata:
labels:
app: smg
annotations:
sglang.ai/mesh-port: "39527"
spec:
serviceAccountName: smg
containers:
- name: smg
image: ghcr.io/smg-project/smg:latest
args:
- --service-discovery
- --service-discovery-namespace=inference
- --selector=app=sglang-worker
- --service-discovery-port=8000
- --router-selector=app=smg
- --enable-mesh
- --mesh-server-name=$(POD_NAME)
- --mesh-host=0.0.0.0
- --mesh-advertise-host=$(POD_IP)
- --mesh-port=39527
env:
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: POD_IP
valueFrom:
fieldRef:
fieldPath: status.podIP
ports:
- containerPort: 30000
name: http
- containerPort: 39527
name: meshEach router discovers the sglang-worker pods itself and finds its peers through the app=smg label. The stable pod names of a StatefulSet double as mesh names, and the smg service account needs the same Pod permissions as worker discovery.
Headless Service¶
apiVersion: v1
kind: Service
metadata:
name: smg-mesh
namespace: inference
spec:
clusterIP: None
selector:
app: smg
ports:
- port: 39527
name: meshThis is the StatefulSet's governing Service. Router discovery dials Pod IPs, so the mesh does not use these DNS names; expose the HTTP port to clients through a regular Service or load balancer.
Kubernetes Router Discovery¶
Router discovery watches Pods that match --router-selector and writes them straight into the mesh membership table, replacing --mesh-peer-urls. It runs as its own task alongside worker discovery.
smg launch --enable-mesh \
--mesh-server-name "$POD_NAME" \
--mesh-advertise-host "$POD_IP" \
--service-discovery \
--service-discovery-namespace inference \
--selector app=sglang-worker \
--router-selector app=smg| Setting | Requirement |
|---|---|
--enable-mesh |
Required. Without it the router logs Router selector configured but mesh is not enabled and skips router discovery |
--service-discovery or --discovery-provider kubernetes |
One of the two is required; both select the same Kubernetes provider. Router discovery is configured through service discovery, so --router-selector alone does nothing, and the file, slurm, and consul providers do not run it. Selecting the provider also starts worker discovery; set --selector for your worker pods |
--router-selector |
Required. Space-separated key=value labels, all of which a Pod must carry. Without a selector, router discovery does not start |
--service-discovery-namespace |
Namespace watched for router Pods, shared with worker discovery. Unset watches all namespaces |
--mesh-server-name |
Must equal the pod name, because discovery registers each peer under its pod name |
sglang.ai/mesh-port annotation |
Mesh port to dial for that Pod. If it is missing or invalid (invalid values log a warning), the discovering router dials its own --mesh-port |
| RBAC | get, list, and watch on Pods, as for worker discovery |
How Pods map to node status:
- Running and Ready:
ALIVEat the Pod IP and mesh port - Not Ready (already known and not
DOWN):SUSPECTED - Deleted or terminating:
DOWN
Monitoring¶
Mesh Metrics¶
Mesh metrics are served with the other gateway metrics on the Prometheus port (default 29000).
| Metric | Type | Labels | Description |
|---|---|---|---|
router_mesh_peer_connections |
Gauge | peer |
Open sync streams accepted from peer: normally 1 while the stream is open and 0 after it closes. A redial that overlaps the old stream's close briefly counts both |
router_mesh_peer_reconnects_total |
Counter | peer |
Sync streams from peer into this router that ended |
router_mesh_sync_round_duration_seconds |
Histogram | peer |
Time to queue one round of updates on the stream to peer |
Each sync stream is opened by the router whose name sorts first and counted by the router that accepts it, under the name the dialer's first frame carries, so a router reports router_mesh_peer_connections only for peers whose names sort before its own, and router_mesh_sync_round_duration_seconds only for peers whose names sort after it. Scrape every router and sum: a fully connected mesh of N routers has N×(N−1)/2 streams. After you remove a router for good, its series stays at 0 on the routers that accepted its streams until they restart.
Alerting Rules¶
groups:
- name: smg-mesh
rules:
# A fully connected mesh of N routers has N*(N-1)/2 sync streams.
# 3 routers -> 3 streams. Scrape every router.
- alert: SMGMeshStreamsMissing
expr: (count(router_mesh_peer_connections > 0) or vector(0)) < 3
for: 2m
labels:
severity: warning
annotations:
summary: "SMG mesh has fewer sync streams than expected"
- alert: SMGMeshStreamDown
expr: router_mesh_peer_connections == 0
for: 2m
labels:
severity: warning
annotations:
summary: "Mesh sync stream from router {{ $labels.peer }} is down"Best Practices¶
No Quorum Needed¶
Any router count works. There is no leader or quorum, so odd counts are not required.
Availability Zones¶
Spread routers across zones so that a zone outage leaves routers holding the shared cache state, and register each worker on routers in more than one zone to keep it served.
Protect the Mesh Port¶
Enable mesh mTLS so only routers with a certificate from your CA can join, and expose the port only to other routers.
Monitoring¶
Alert when count(router_mesh_peer_connections > 0) drops below N×(N−1)/2 for N routers.
Troubleshooting¶
Common Issues¶
| Symptom | Cause | Solution |
|---|---|---|
Startup fails with Invalid mesh peer URL (Python CLI) or Invalid value for field 'mesh_peer_urls' (Rust binary) |
A peer address is a hostname | Pass IP:port; on Kubernetes, use router discovery |
Startup fails with mesh advertise address cannot be unspecified |
--mesh-host is 0.0.0.0 and --mesh-advertise-host is unset |
Set --mesh-advertise-host to the node's IP |
Startup fails with mesh port cannot be 0 |
--mesh-port 0 |
Use a fixed port |
A router stays alone in the mesh, logging No peer address available to connect once per round at debug level |
No --mesh-peer-urls is set and no peer has joined its membership table (for example, router discovery found none) |
Set --mesh-peer-urls to another router's IP:port, or configure router discovery |
A joining router logs Error connecting to peer init_peer with growing retry delays |
Its --mesh-peer-urls peers are not reachable yet. The entries are named init_peer, init_peer_1, … by position until an answer names them |
The dials are retried with backoff and the routers join once a peer is up; if none ever connects, check the addresses, ports, and network path |
| A router stays out of the mesh after a restart or partition | Its peers hold it as DOWN or LEAVING and no longer probe it |
Restart it under a new name, or use router discovery; see Node Status States |
Routers appear under both their pod name and another name in the membership log lines (Member ... is ...) |
--mesh-server-name differs from the pod name under router discovery |
Set --mesh-server-name=$(POD_NAME) |
Log shows Router selector configured but mesh is not enabled |
--router-selector without --enable-mesh |
Add --enable-mesh |
No Router node discovery enabled log line |
--router-selector without Kubernetes worker discovery (the file, slurm, and consul providers do not run router discovery) |
Add --service-discovery or --discovery-provider kubernetes |
| A router Pod never joins | Pod not Running and Ready, labels do not match, or RBAC is missing | Check readiness, labels, and the Role |
| ZMQ workers are missing on peers | ZMQ workers (ipc:// and tcp://) are never published to the mesh |
Expected: each router keeps its own ZMQ workers |
Debug Logging¶
RUST_LOG=warn,smg=info,smg_mesh=debug,smg::mesh=debug \
smg launch --enable-mesh ...Gossip, sync streams, and CRDT merges log under smg_mesh; the worker and cache-tree sync adapters and router discovery under smg::mesh. RUST_LOG replaces the filter built from --log-level, so keep a base level in it.
Verify Cluster Health¶
# Membership: every status change logs one INFO line,
# "Member <name> is <Status> at version <n> (<how>)"
kubectl -n inference logs smg-0 | grep 'Member ' | tail -n 5
# Worker sync: every router lists the workers its peers registered
curl -s http://<router-ip>:30000/workers | jq -r '.workers[] | "\(.url) healthy=\(.is_healthy)"'At --log-level trace, each router also logs its whole node table once per gossip round as Round <n> Status:{...}.
# Sync streams across all routers: N*(N-1)/2 when fully connected
count(router_mesh_peer_connections > 0)What's Next?¶
Service Discovery¶
Discover workers in Kubernetes and set up the Role that router discovery shares.