High Availability

Run several SMG routers as one cluster. With --enable-mesh, routers form a peer-to-peer mesh: they track each other's membership with a SWIM-style gossip protocol, replicate the workers each router registers, and share cache-aware routing updates, so every router routes with the cluster's workers and cache affinity.

The mesh does not steer client traffic. Put the routers behind a load balancer or a Kubernetes Service; that is what moves clients to the remaining routers when one fails.


Overview

Fault Tolerance

When a router fails, your load balancer sends its clients to the others, which keep the cache-aware routing state the failed router shared. Workers that only the failed router registered are removed shortly after it is marked DOWN, so register each worker on more than one router (Worker Registry Sync).

Scalability

Add routers without restarting the others. A new router joins through a running peer or Kubernetes router discovery and picks up the cluster's workers.

State Synchronization

Workers registered on any router and cache-aware routing tree updates reach every peer. Rate limits, sticky sessions, and other policy state stay local to each router.

Rolling Updates

Replace routers one at a time: each announces its leave to the mesh and drains in-flight requests on shutdown. With Kubernetes router discovery, a replaced pod rejoins as soon as it is Ready.


Mesh Architecture

                      clients
                         │
         load balancer or Kubernetes Service
        ┌────────────────┼────────────────┐
        ▼                ▼                ▼
  ┌──────────┐     ┌──────────┐     ┌──────────┐
  │ router A │◀───▶│ router B │◀───▶│ router C │   mesh: gRPC on --mesh-port
  └────┬─────┘     └────┬─────┘     └────┬─────┘
       └────────────────┼────────────────┘
                        ▼
                     workers

Every router is equal: there is no leader and no quorum. Each pair of routers (A and C included) keeps one sync stream over the mesh port, and state converges without coordination.

Gossip Membership

SWIM-style membership and failure detection.

  • One probe per second to a random peer
  • Indirect probes through up to three peers
  • Status changes broadcast to live peers

Sync Streams

One bidirectional gRPC stream per pair of routers.

  • Opened by the router whose name sorts first
  • Each side sends pending updates every second
  • Kept alive by a heartbeat every 15 seconds
  • Closed after 60 seconds without a frame

CRDT Worker State

Conflict-free replicated keys for worker state.

  • No locks or coordination
  • Resent until each peer acknowledges
  • Relayed through intermediate peers

Cache-Tree Updates

Best-effort broadcasts for cache-aware routing.

  • Batched per model every gossip round
  • Unknown prefixes trigger a repair from a peer
  • Dropped updates are not resent

Configuration

Quick Start

Start the first router, then point the next one at it with --mesh-peer-urls:

# Router 1 (bootstrap)
smg launch --enable-mesh --mesh-advertise-host 10.0.0.1 --mesh-port 39527 \
    --worker-urls http://10.0.1.1:8000 http://10.0.1.2:8000

# Router 2, joining router 1
smg launch --enable-mesh --mesh-advertise-host 10.0.0.2 --mesh-port 39527 \
    --mesh-peer-urls 10.0.0.1:39527

Router 2 has no workers of its own; it imports router 1's workers through the mesh.

Imported workers last only while the router that registered them stays in the cluster — if router 1 goes down for good, router 2 removes them after a short grace (see Worker Registry Sync). To keep workers served through a router failure, register them on every router, as under Basic Configuration.

Command Line Options

Flag Default Description
--enable-mesh false Start the mesh listener and join the cluster
--mesh-server-name Mesh_ plus 4 random alphanumeric characters Node name, unique within the cluster. Must be non-empty and must not contain :. With router discovery, set it to the pod name
--mesh-host 0.0.0.0 Bind address for the mesh listener (an IP address)
--mesh-advertise-host --mesh-host IP address that peers dial. Required when --mesh-host is an unspecified address such as 0.0.0.0
--mesh-port 39527 Mesh port, used for both binding and advertising. 0 is rejected
--mesh-peer-urls (none) Bootstrap peers as IP:port, space- or comma-separated. While the membership table holds no other member, the entries are dialed in turn until one answers
--mesh-tls-ca-cert (none) CA certificate (PEM) that mesh peer certificates must chain to. Set together with --mesh-tls-cert and --mesh-tls-key to enable mesh mTLS
--mesh-tls-cert (none) This node's certificate (PEM), with the node's IP as a SAN
--mesh-tls-key (none) This node's private key (PKCS#8 PEM)
--router-selector (none) Label selector for Kubernetes router discovery. Takes effect only when Kubernetes worker discovery is selected, with --service-discovery or --discovery-provider kubernetes

Mesh mTLS

--mesh-tls-ca-cert, --mesh-tls-cert, and --mesh-tls-key encrypt the mesh port and authenticate both ends of every mesh connection. The three flags come together: a router started with only one or two of them refuses to start, and the error names the missing flags.

smg launch --enable-mesh --mesh-advertise-host 10.0.0.1 \
    --mesh-tls-ca-cert /etc/smg/mesh/ca.pem \
    --mesh-tls-cert /etc/smg/mesh/node.pem \
    --mesh-tls-key /etc/smg/mesh/node-key.pem

With the flags set, the mesh listener serves TLS with the node's certificate and accepts only peers that present a certificate chaining to the CA. Every dial to a peer presents this node's certificate and verifies the peer's against the CA.

Issue every router a certificate from the same CA that carries the node's IP — the address its peers dial, so its --mesh-advertise-host — as an IP SAN. A dial verifies the peer's certificate against the address it dials, and mesh addresses are always IPs (see the note above), so a certificate with only a hostname never matches. An IPv6 node's SAN is the unbracketed address: fd00::7, not [fd00::7].

The private key must be PKCS#8 PEM. Each of the three files must exist and be readable at startup, or the router fails to start with an error naming the flag whose file it cannot read; the router also warns when the key file is readable or writable by its group or by others. The mesh listener reads the certificate and key once at startup and does not reload them; to rotate certificates, restart the router.

Configure every router in the cluster the same way: a router with the flags serves and dials only TLS, a router without them only plaintext, so a mixed cluster's routers cannot reach each other.

Python Entrypoint

The Python CLI (smg launch) and the Docker image's entrypoint (python3 -m smg.launch_router) accept the same mesh flags with the same defaults and address checks, so container args can pass them directly:

smg launch --enable-mesh --mesh-host 0.0.0.0 --mesh-advertise-host 10.0.0.11 --mesh-port 39527

smg serve takes router flags with a --router- prefix, for example --router-enable-mesh and --router-mesh-advertise-host.

Basic Configuration

Node 1 (Bootstrap)

smg launch --enable-mesh \
--mesh-server-name node1 \
--mesh-host 0.0.0.0 \
--mesh-advertise-host 10.0.0.11 \
--mesh-port 39527 \
--worker-urls http://10.0.1.1:8000 http://10.0.1.2:8000

Node 2 (Join)

smg launch --enable-mesh \
--mesh-server-name node2 \
--mesh-host 0.0.0.0 \
--mesh-advertise-host 10.0.0.12 \
--mesh-port 39527 \
--mesh-peer-urls 10.0.0.11:39527 \
--worker-urls http://10.0.1.1:8000 http://10.0.1.2:8000

Node 3 (Join)

smg launch --enable-mesh \
--mesh-server-name node3 \
--mesh-host 0.0.0.0 \
--mesh-advertise-host 10.0.0.13 \
--mesh-port 39527 \
--mesh-peer-urls 10.0.0.11:39527 \
--worker-urls http://10.0.1.1:8000 http://10.0.1.2:8000

Gossip Protocol

Membership and Failure Detection

  1. Probe: Every second, each router pings one random peer that is not DOWN or LEAVING. The ping carries the sender's membership table, and the receiver adopts any entry with a higher version.
  2. Indirect probe: If the ping fails, the router asks up to three other ALIVE peers to ping the target (ping-req).
  3. Suspect, then down: If no probe reaches the target, it moves from ALIVE to SUSPECTED, or from SUSPECTED to DOWN, and the new status is broadcast to every ALIVE peer.
  4. Sync stream: After a successful direct probe of an ALIVE peer, the router whose name sorts first opens the pair's sync stream if none is open. A probe that only a relay answered opens none: the pair's own link has just failed.

Node Status States

Status Description
ALIVE Reachable. Every router starts in this state
SUSPECTED A probe failed: neither the direct ping nor any indirect ping reached the router. Still probed. Router discovery also sets it for a known pod that is not Ready, unless that router is already DOWN
DOWN A probe failed again while SUSPECTED, or the router's pod was deleted or is terminating. Peers stop probing it
LEAVING The router announced a graceful leave, as every router does when its graceful shutdown begins. Peers stop probing it
INIT Defined in the protocol but not used

Failure Detection Timing

Phase Value
Gossip round (probe, send, and batching cadence) 1 second
Indirect probe fan-out Up to 3 ALIVE peers
Probe connect / request timeout 5 seconds / 10 seconds
Redial backoff after a failed probe of the same peer 1 second, doubling up to 60 seconds
Sync stream heartbeat (sent by the dialing router, answered by the accepting one) 15 seconds
Sync stream idle timeout (no frame received) 60 seconds

These values are fixed; there are no flags to tune them.


State Synchronization

Synchronized State Types

Worker Registry

Workers each router registers, shared with every peer.

  • URL, model, health, and load
  • Worker spec, without its API key
  • ZMQ workers (ipc:// and tcp://) excluded

Routing Trees

Cache-aware routing state shared across routers.

  • HTTP string trees and gRPC token trees
  • Tree inserts, sent as per-model deltas
  • Repair from a peer for unknown prefixes

Membership

The cluster's node table.

  • Node names, addresses, and status
  • Carried on every gossip ping

Stays Local

Each router keeps its own:

  • Rate limits and admission queues
  • Sticky-session (manual policy) assignments
  • Health checks and circuit-breaker state
  • Routing policy and its settings

Because each router enforces rate limits on its own, the cluster-wide limit is roughly the per-router limit times the number of routers. Sticky-session assignments are also per router, so the same routing key can map to different workers on different routers.

Worker Registry Sync

Each router publishes the workers it registered itself, whether from --worker-urls, the worker API, or service discovery, whenever one is added, changes status, or is removed. Peers import them into their own registries:

  • Imports are health-checked locally. A peer rebuilds the worker from its published spec and probes it itself. The owner's health flag sets the import's initial state and afterwards only nudges it: healthy promotes a pending or not-ready import to ready, and unhealthy demotes a ready one. When the owner replaces a worker in place — a new model or spec at the same URL — peers rebuild their imports from the republished spec.
  • Workers are matched by URL. A router that registered a URL itself keeps its own worker and ignores peers' copies. Give a worker the same URL on every router, and never a loopback address: a peer that imports http://127.0.0.1:8000 routes to its own host.
  • Only the owner removes a worker. Removing a worker on the router that registered it removes it everywhere. Deleting an imported worker through another router's API lasts at most until that router's next reconcile pass (every 30 seconds) imports it again.
  • A removed worker is not re-imported from peers. Once a worker is removed on the router that registered it, that router stops importing the endpoint from the mesh: a peer's copy — a twin registration of the same URL still published by another router, or a lagging peer's stale state — is held back instead of imported again. For a worker registered by service discovery, the hold is tied to the removed instance: a different instance published at the same address (for example, a new pod reusing the IP and port) imports as usual, but a peer copy that names no instance is held back. A removed worker that had no instance identity — one you registered yourself — holds back every peer copy of its URL. Registering a worker at that URL on that router re-admits the endpoint. The first copy held back per endpoint is logged, and the hold lives in memory — up to 4096 removed endpoints, oldest dropped first — so a router restart clears it.
  • A departed router's workers are removed. From the moment the cluster state shows a router DOWN or LEAVING, peers import nothing new from it, so a router that joins afterwards never picks up its workers. Once the router has been marked gone for 20 seconds, peers also remove the workers they already imported from it at their next reconcile pass (every 30 seconds). A shorter blip removes nothing, and a router that returns has its workers imported again. Workers a peer registered itself — the same URL given to several routers — are its own, not imports, and stay.
  • API keys stay local. The published spec leaves out the worker's api_key.

Cache-Aware State Sync

The cache_aware policy keeps an approximate prefix tree per model: a string tree for HTTP requests and a token tree for gRPC requests. With the mesh on, routing decisions made through these trees are shared:

  1. Publish: After choosing a worker, the router records it in its own tree and queues a delta: a hash of the request's prefix path plus the worker URL. Once per gossip round, each model's queued deltas go to every connected peer as one batch. Picks made by the KV-pressure fallback land in the router's own tree but are not broadcast.
  2. Apply: A peer that already knows that prefix path adds the worker to it in its own tree.
  3. Repair: A peer that does not know the path, such as a router that just started, asks a random ALIVE peer for its whole tree for that model and tree type and replays it. The tree arrives in pages of up to about 2 MiB; a repair that makes no progress for 5 seconds is retried, preferably with another peer, up to 3 times. Unknown prefixes for the same model and tree type that arrive meanwhile are folded into the repair in flight.

The default policy, per-model cache-aware policies, and the prefill, decode, and encode policies of disaggregated mode all take part, with no flag beyond --enable-mesh.

What consistency to expect:

  • Approximate. Deltas and repairs move the trees toward each other, but they are not guaranteed to match: a dropped delta is not resent, and a repair starts only when a later delta names a path the peer does not know. Each router still evicts its own tree on its own schedule (--eviction-interval, --max-tree-size); evictions are not shared.
  • Best effort. A delta batch that a peer's stream cannot accept is dropped, not resent. The peer catches up through repair the next time it sees a prefix it does not know.
  • Survives a router failure. Routing decisions that a failed router already shared stay in its peers' trees, so traffic that moves to them keeps that cache affinity.

Not synchronized: event-driven cache-aware routing (KV events), which each router builds from the workers' KV event streams, and the hash placement index (--cache-index hash).

How State Is Replicated

Key prefix Channel Delivery Carries
worker: CRDT Last-writer-wins by Lamport timestamp, then replica. Resent until each peer acknowledges; relayed through peers One key per worker, written only by the router that owns it
rl: CRDT Epoch-max-wins Rate-limit counter shards. Registered, but nothing writes to it
td: Stream, broadcast Best effort, once per gossip round, to directly connected peers Cache-tree deltas, one entry per model per round
tree:req:, tree:page: Stream, targeted Retried after 5 seconds without progress, up to 3 times Cache-tree repair requests and pages

Worker-state merges order updates with Lamport clocks rather than wall-clock time, so clock skew between routers does not decide which update wins.


Deployment Patterns

Static Peers

For VMs and bare metal. Give each router a unique name, its own IP as --mesh-advertise-host, and another router's address in --mesh-peer-urls.

  • Start routers in any order; the peer is dialed until it joins
  • Restart an isolated router under a new name

Basic Configuration →

Kubernetes Router Discovery

Routers find each other by pod label, with no bootstrap peer to configure.

  • Pods join when Ready and are marked DOWN when deleted
  • Restarted pods rejoin under their pod name

Kubernetes Deployment →

Cluster Size

Any number of routers works. There is no leader election or quorum, so odd counts are not required, and each router keeps serving with the state it has. Every pair of routers keeps one sync stream, so N routers hold N×(N−1)/2 streams and each router sends its updates to N−1 peers.


Kubernetes Deployment

StatefulSet Configuration

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: smg
  namespace: inference
spec:
  serviceName: smg-mesh
  replicas: 3
  selector:
    matchLabels:
      app: smg
  template:
    metadata:
      labels:
        app: smg
      annotations:
        sglang.ai/mesh-port: "39527"
    spec:
      serviceAccountName: smg
      containers:
      - name: smg
        image: ghcr.io/smg-project/smg:latest
        args:
        - --service-discovery
        - --service-discovery-namespace=inference
        - --selector=app=sglang-worker
        - --service-discovery-port=8000
        - --router-selector=app=smg
        - --enable-mesh
        - --mesh-server-name=$(POD_NAME)
        - --mesh-host=0.0.0.0
        - --mesh-advertise-host=$(POD_IP)
        - --mesh-port=39527
        env:
        - name: POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: POD_IP
          valueFrom:
            fieldRef:
              fieldPath: status.podIP
        ports:
        - containerPort: 30000
          name: http
        - containerPort: 39527
          name: mesh

Each router discovers the sglang-worker pods itself and finds its peers through the app=smg label. The stable pod names of a StatefulSet double as mesh names, and the smg service account needs the same Pod permissions as worker discovery.

Headless Service

apiVersion: v1
kind: Service
metadata:
  name: smg-mesh
  namespace: inference
spec:
  clusterIP: None
  selector:
    app: smg
  ports:
  - port: 39527
    name: mesh

This is the StatefulSet's governing Service. Router discovery dials Pod IPs, so the mesh does not use these DNS names; expose the HTTP port to clients through a regular Service or load balancer.

Kubernetes Router Discovery

Router discovery watches Pods that match --router-selector and writes them straight into the mesh membership table, replacing --mesh-peer-urls. It runs as its own task alongside worker discovery.

smg launch --enable-mesh \
    --mesh-server-name "$POD_NAME" \
    --mesh-advertise-host "$POD_IP" \
    --service-discovery \
    --service-discovery-namespace inference \
    --selector app=sglang-worker \
    --router-selector app=smg
Setting Requirement
--enable-mesh Required. Without it the router logs Router selector configured but mesh is not enabled and skips router discovery
--service-discovery or --discovery-provider kubernetes One of the two is required; both select the same Kubernetes provider. Router discovery is configured through service discovery, so --router-selector alone does nothing, and the file, slurm, and consul providers do not run it. Selecting the provider also starts worker discovery; set --selector for your worker pods
--router-selector Required. Space-separated key=value labels, all of which a Pod must carry. Without a selector, router discovery does not start
--service-discovery-namespace Namespace watched for router Pods, shared with worker discovery. Unset watches all namespaces
--mesh-server-name Must equal the pod name, because discovery registers each peer under its pod name
sglang.ai/mesh-port annotation Mesh port to dial for that Pod. If it is missing or invalid (invalid values log a warning), the discovering router dials its own --mesh-port
RBAC get, list, and watch on Pods, as for worker discovery

How Pods map to node status:

  • Running and Ready: ALIVE at the Pod IP and mesh port
  • Not Ready (already known and not DOWN): SUSPECTED
  • Deleted or terminating: DOWN

Monitoring

Mesh Metrics

Mesh metrics are served with the other gateway metrics on the Prometheus port (default 29000).

Metric Type Labels Description
router_mesh_peer_connections Gauge peer Open sync streams accepted from peer: normally 1 while the stream is open and 0 after it closes. A redial that overlaps the old stream's close briefly counts both
router_mesh_peer_reconnects_total Counter peer Sync streams from peer into this router that ended
router_mesh_sync_round_duration_seconds Histogram peer Time to queue one round of updates on the stream to peer

Each sync stream is opened by the router whose name sorts first and counted by the router that accepts it, under the name the dialer's first frame carries, so a router reports router_mesh_peer_connections only for peers whose names sort before its own, and router_mesh_sync_round_duration_seconds only for peers whose names sort after it. Scrape every router and sum: a fully connected mesh of N routers has N×(N−1)/2 streams. After you remove a router for good, its series stays at 0 on the routers that accepted its streams until they restart.

Alerting Rules

groups:
- name: smg-mesh
  rules:
  # A fully connected mesh of N routers has N*(N-1)/2 sync streams.
  # 3 routers -> 3 streams. Scrape every router.
  - alert: SMGMeshStreamsMissing
    expr: (count(router_mesh_peer_connections > 0) or vector(0)) < 3
    for: 2m
    labels:
      severity: warning
    annotations:
      summary: "SMG mesh has fewer sync streams than expected"

  - alert: SMGMeshStreamDown
    expr: router_mesh_peer_connections == 0
    for: 2m
    labels:
      severity: warning
    annotations:
      summary: "Mesh sync stream from router {{ $labels.peer }} is down"

Best Practices

No Quorum Needed

Any router count works. There is no leader or quorum, so odd counts are not required.

Availability Zones

Spread routers across zones so that a zone outage leaves routers holding the shared cache state, and register each worker on routers in more than one zone to keep it served.

Protect the Mesh Port

Enable mesh mTLS so only routers with a certificate from your CA can join, and expose the port only to other routers.

Monitoring

Alert when count(router_mesh_peer_connections > 0) drops below N×(N−1)/2 for N routers.


Troubleshooting

Common Issues

Symptom Cause Solution
Startup fails with Invalid mesh peer URL (Python CLI) or Invalid value for field 'mesh_peer_urls' (Rust binary) A peer address is a hostname Pass IP:port; on Kubernetes, use router discovery
Startup fails with mesh advertise address cannot be unspecified --mesh-host is 0.0.0.0 and --mesh-advertise-host is unset Set --mesh-advertise-host to the node's IP
Startup fails with mesh port cannot be 0 --mesh-port 0 Use a fixed port
A router stays alone in the mesh, logging No peer address available to connect once per round at debug level No --mesh-peer-urls is set and no peer has joined its membership table (for example, router discovery found none) Set --mesh-peer-urls to another router's IP:port, or configure router discovery
A joining router logs Error connecting to peer init_peer with growing retry delays Its --mesh-peer-urls peers are not reachable yet. The entries are named init_peer, init_peer_1, … by position until an answer names them The dials are retried with backoff and the routers join once a peer is up; if none ever connects, check the addresses, ports, and network path
A router stays out of the mesh after a restart or partition Its peers hold it as DOWN or LEAVING and no longer probe it Restart it under a new name, or use router discovery; see Node Status States
Routers appear under both their pod name and another name in the membership log lines (Member ... is ...) --mesh-server-name differs from the pod name under router discovery Set --mesh-server-name=$(POD_NAME)
Log shows Router selector configured but mesh is not enabled --router-selector without --enable-mesh Add --enable-mesh
No Router node discovery enabled log line --router-selector without Kubernetes worker discovery (the file, slurm, and consul providers do not run router discovery) Add --service-discovery or --discovery-provider kubernetes
A router Pod never joins Pod not Running and Ready, labels do not match, or RBAC is missing Check readiness, labels, and the Role
ZMQ workers are missing on peers ZMQ workers (ipc:// and tcp://) are never published to the mesh Expected: each router keeps its own ZMQ workers

Debug Logging

RUST_LOG=warn,smg=info,smg_mesh=debug,smg::mesh=debug \
    smg launch --enable-mesh ...

Gossip, sync streams, and CRDT merges log under smg_mesh; the worker and cache-tree sync adapters and router discovery under smg::mesh. RUST_LOG replaces the filter built from --log-level, so keep a base level in it.

Verify Cluster Health

# Membership: every status change logs one INFO line,
# "Member <name> is <Status> at version <n> (<how>)"
kubectl -n inference logs smg-0 | grep 'Member ' | tail -n 5

# Worker sync: every router lists the workers its peers registered
curl -s http://<router-ip>:30000/workers | jq -r '.workers[] | "\(.url) healthy=\(.is_healthy)"'

At --log-level trace, each router also logs its whole node table once per gossip round as Round <n> Status:{...}.

# Sync streams across all routers: N*(N-1)/2 when fully connected
count(router_mesh_peer_connections > 0)

What's Next?

Service Discovery

Discover workers in Kubernetes and set up the Role that router discovery shares.

Service Discovery →

Cache-Aware Routing

How the prefix trees that the mesh shares drive routing.

Cache-Aware Routing →

Graceful Shutdown

Allow in-flight requests to complete during shutdown.

Graceful Shutdown →

Metrics Reference

Gateway metrics to scrape alongside the mesh metrics.

Metrics Reference →