Sticky Sessions and Routing Keys

A multi-turn conversation runs fastest when each turn lands on the worker that served the turn before it, because that worker still holds the conversation's prefix in its KV cache. Sticky sessions give SMG that property: each request carries a routing key, taken from the request body's rid or from a header, and SMG pins the key to one worker for as long as the key stays in use.

SMG provides stickiness in two forms:

  • The manual policy pins every keyed request in its own key-to-worker map.
  • The routing-key override (--routing-key-override, alias --sticky-sessions) adds the same pinning on top of any other policy. Keyed requests stick; requests without a key, and by default the first request for each new key, are still placed by the configured policy, such as cache_aware.

Overview

Conversation Lineage

The key comes from the body rid with turn and retry suffixes stripped, so conv_t1, conv_t2, and conv_t2_r1 all pin as conv.

Routing-Key Headers

SMG reads the first valid value from an ordered list of headers, x-smg-routing-key by default; the body rid is the fallback when no header carries one.

Any Policy

With the override, keyed requests stick on every policy, and by default the configured policy still chooses where a new key starts.

Bounded and Observable

Idle pins expire, a key's requests beyond two in flight are placed again instead of going straight to its pinned worker, and inference responses name the worker that served them.


What a Routing Key Guarantees

A routing key is a promise with bounds. While the key's pinned worker is available, a request that carries the key goes to it, up to the override's in-flight threshold described below; the moment the worker is not available, the request is placed elsewhere, with one bound: when every worker the request could use is vetoed at once, the gateway places it on the least-loaded of them (the pin included) rather than refuse it, and counts the fallback; under --worker-overload-shed an all-overloaded pool is refused with 503 worker_overload_protection_shed instead, while a liveness veto still steers. In full:

  • Same key, same worker. Every request that resolves to the key (the first valid routing-key header, x-smg-routing-key by default, or, under the override, the body rid; see Key Precedence) goes to the key's pinned worker, on every policy, with one exception: consistent_hashing honours a valid X-SMG-Target-Worker header (a healthy worker's index) over the key, as the policy table below states. Under manual and the override, each key remembers a primary worker and one failover candidate (Reuse and Failover); consistent_hashing keeps no map and recomputes the key's place on its ring, which moves about one key in N when a worker joins or leaves.
  • Available means routable. A pinned worker takes the request only while it is healthy (Ready), its circuit breaker is not open, and no veto excludes it. Two vetoes are on by default: overload protection takes a worker out of routing while its last load report shows 8 or more waiting requests or KV-cache usage at or above 0.8 (--worker-overload-waiting-requests, --worker-overload-token-usage; --disable-worker-overload-protection switches it off), and the liveness vetoes exclude a worker whose transport failed and has gone silent, or whose engine has stopped producing. When none of a key's remembered workers is available, the request is placed again: in the override's default delegate mode the new worker becomes the primary and the old one its failover candidate; in the other modes the earlier worker stays first and the key returns to it once it is available (occupied_miss in the metrics). When every candidate is vetoed at once there is no elsewhere: the request goes to the least-loaded worker that is routable but for its veto, overloaded ones before stalled ones, counted in smg_worker_overload_fallback_total or smg_worker_liveness_fallback_total (stage="selection"); under --worker-overload-shed an all-overloaded pool is refused with 503 worker_overload_protection_shed instead, while a liveness veto still steers.
  • The spill is bounded. Under the override, once a key has 2 of its requests in flight on its pinned worker, the next concurrent request for the key is placed again (cap_respill) rather than queued behind the pin, so one busy conversation cannot stack every turn on one worker (In-Flight Threshold). The threshold triggers a new placement; it is not a cap on the worker: the placement may choose the pinned worker again when the policy prefers it, for example because it holds the conversation's prefix, so more than 2 of a key's requests can run there.
  • A failure does not stay put — outside PD mode. A request that fails on the pinned worker with a retryable answer is retried on another worker while an available one remains; only when nothing else is available does the retry return to the worker that just failed. In PD mode, a retry does not skip the workers it already tried: it selects its prefill/decode pair from all available workers again, so a keyed retry can return to the pinned legs even while other workers are free (Retries).
  • Pins expire. Under manual and the override, when eviction is enabled (--max-idle-secs and --eviction-interval both above zero), a key idle for longer than --max-idle-secs (default 14400, four hours) is dropped by the next eviction sweep, and its next request starts over as a new key; 0 on either flag disables eviction (Idle Eviction). consistent_hashing keeps no pins to evict.

What a routing key does not guarantee:

  • One pin across gateway replicas. Each replica keeps its own map (Scoping). Send a conversation to one replica, or use consistent_hashing, which computes the same placement on every replica that sees the same workers.
  • A warm cache. The pin keeps a conversation on the worker that served it; whether that worker still holds the conversation's prefix is the engine's cache policy. For requests without a key, cache-aware routing follows the prefix instead.
  • Order between concurrent turns. Two requests for one key in flight at once are served in whichever order the worker schedules them, and, under the override, a third may run elsewhere (the bounded spill above).

A minimal configuration, and what each flag changes:

# Keyed requests stick on top of cache-aware placement; a new key starts where
# cache-aware puts it, and the default overload veto and the two-in-flight spill stay on
smg launch --worker-urls http://w1:8000 http://w2:8000 http://w3:8000 \
  --policy cache_aware --routing-key-override
What you set What you get
--routing-key-override alone Keys from x-smg-routing-key or, when no header carries a valid value, the body rid (lineage stripped, at most 128 bytes); the first request of a key is placed by the configured policy, then pinned
--routing-key-headers a b The first of a, b carrying a valid value (non-empty UTF-8, at most 128 bytes) is the key; under the override, a body rid that yields a key (at most 128 bytes after suffix removal) is the fallback when none does
--disable-worker-overload-protection A pinned worker stays the pin however deep its queue; the two-in-flight spill still applies
--max-idle-secs N A key idle for N seconds is dropped by the next eviction sweep, which runs every --eviction-interval seconds; 0 on either flag disables eviction

Deriving the Routing Key

Key Precedence

For each request, SMG uses the first of these that yields a key:

  1. Routing-key headers: the first header named in --routing-key-headers that carries a valid value (non-empty UTF-8, at most 128 bytes); the override's pin key is that value with its lineage suffixes stripped (manual and consistent_hashing place on the raw value, see the warning below). manual and consistent_hashing also try X-SMG-Routing-Key when no configured header carries a valid value; the override map reads the configured names only.
  2. Body rid (override enabled only): the rid with its lineage suffixes stripped, when the result is at most 128 bytes.
  3. No key: the configured policy places the request, and nothing is pinned.

A valid routing-key header wins even when the body carries a rid: an explicit key outranks request-ID lineage, so the client or proxy that sets the header chooses the pin, and a proxy must not stamp a per-request value into a routing-key header. manual and consistent_hashing apply this order themselves, with the same validity rules and the extra X-SMG-Routing-Key step.

Body rid Lineage

rid is an SGLang-style request ID accepted in the body of chat completions, completions, messages, /generate, embeddings, classify, and rerank requests (for rerank, the first ID of a list). The Responses API has no rid, so Responses requests can pin only through a header. SMG reads rid for routing only when --routing-key-override is enabled.

SMG removes one trailing retry suffix, then one trailing turn suffix:

<base>[_t<digits>][_r<digits>]  ->  <base>
  • A suffix is an underscore, a lowercase t (turn) or r (retry), and one or more ASCII digits, at the very end of the rid.
  • The retry suffix is removed first, so it has to come last: conv_t2_r1 becomes conv, while conv_r1_t2 becomes conv_r1.
  • Each suffix is removed at most once.
  • If nothing would be left, the whole rid is the key.
  • If the resulting key is longer than 128 bytes, SMG ignores the rid; a request whose headers carried no valid key is then placed without a key.
rid Routing key
conv conv
conv_t2 conv
conv_t2_r1 conv
conv_r1 conv
conv_t1_t2 conv_t1
conv_r1_t2 conv_r1
conv_t, conv_tx1, conv_T2 Unchanged
under_scored_id under_scored_id
_t1 _t1

Stripping only computes the key; the request keeps its full rid. On gRPC workers outside PD mode, SMG sends the rid as the engine request ID, so the suffixes let every turn and retry carry a distinct ID while sharing one key.

Routing-Key Headers

--routing-key-headers takes an ordered list of header names; the default is x-smg-routing-key. Each name must be a valid HTTP header name, or SMG refuses to start, and names match case-insensitively. For each request, SMG walks the list and takes the first header whose value is:

  • non-empty,
  • valid UTF-8, and
  • at most 128 bytes.

A value that fails these checks is skipped without an error, and SMG tries the next name. With the override enabled, header keys get the same lineage stripping as a rid, so a proxy that forwards conv_t2 in a header pins the same entry as a body rid of conv_t2.

# Read an upstream proxy's header first, and keep the SMG header as a fallback
smg launch --worker-urls http://w1:8000 http://w2:8000 \
  --routing-key-override \
  --routing-key-headers x-routing-key x-smg-routing-key

Configuration

Enable the override on top of any policy, here cache_aware:

smg launch --worker-urls http://w1:8000 http://w2:8000 http://w3:8000 \
  --policy cache_aware \
  --routing-key-override

Then give each turn of a conversation a rid that shares a base:

curl http://localhost:30000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "meta-llama/Llama-3.1-8B-Instruct",
    "rid": "chat-7f3a_t1",
    "messages": [{"role": "user", "content": "Plan a three-day trip to Kyoto."}]
  }'

The next turn sends "rid": "chat-7f3a_t2", and a retry of that turn sends "rid": "chat-7f3a_t2_r1". All three pin under the key chat-7f3a. A client that cannot set rid sends -H "x-smg-routing-key: chat-7f3a" instead.

Flag Default Description
--routing-key-override Off Pin keyed requests on any policy. Alias: --sticky-sessions
--routing-key-headers x-smg-routing-key Ordered header names to read the key from; the first valid value wins
--assignment-mode delegate for the override, random for --policy manual How a new key picks its worker: random, min_load, min_group, or delegate
--max-idle-secs 14400 Seconds a pin can go unused before it is evicted. Alias: --sticky-key-idle-secs
--eviction-interval 120 (Python launcher: 60) Seconds between eviction sweeps. Also sets the cache-aware tree eviction interval

--assignment-mode, --max-idle-secs, and --eviction-interval apply to both the manual policy and the override. See the Configuration Reference for every routing option.

Manual Policy or Override?

Aspect --policy manual --routing-key-override
First worker for a new key --assignment-mode (default random) The configured policy (delegate, the default)
Requests without a key Placed by --assignment-mode, not pinned Placed by the configured policy, unchanged
Key sources The first valid header of --routing-key-headers, then X-SMG-Routing-Key, then the body rid when the override is also enabled (lineage stripped, at most 128 bytes) --routing-key-headers, then the body rid (lineage stripped, at most 128 bytes)
Lineage stripping of header keys No Yes
Pins scoped by model No (by PD leg only) Yes (by model and PD leg)
Per-key in-flight threshold No Yes

Choose the override to keep another policy, usually cache_aware, in charge of where conversations start. Choose manual when placement should depend only on the assignment mode.


How the Override Composes with Each Policy

Configured policy With --routing-key-override, a keyed request is Without the override, a routing key is
cache_aware, round_robin, random, power_of_two, least_load, bucket, passthrough Pinned in the override's sticky map; under delegate, the policy chooses a new key's first worker Ignored
prefix_hash Pinned in the override's sticky map; under delegate, the policy chooses a new key's first worker Hashed onto the ring in place of the prompt's token IDs or text, which prefix_hash otherwise hashes
manual Pinned in the manual policy's own map, from the first valid routing-key header (--routing-key-headers, then X-SMG-Routing-Key) and the body rid as the fallback (at most 128 bytes after stripping) Pinned from the first valid routing-key header
consistent_hashing Hashed onto the ring, from the routing-key header first and the body rid as the fallback (at most 128 bytes after stripping); X-SMG-Target-Worker still wins Hashed onto the ring

Requests without a key go to the configured policy unchanged. Under delegate assignment, the override's default, the first request of a conversation is placed exactly as it would be without the override (by prefix match under cache_aware, for example), and later turns follow it. Pinning keeps the worker's cached prefix reachable on every turn; it does not make the engine keep that cache any longer.

In PD mode, the prefill and decode legs are pinned independently.


Assignment Modes and Eviction

Assignment Modes

--assignment-mode decides where a key goes the first time SMG sees it, and where it moves when none of its remembered workers is available.

Mode A new key goes to
random A random available worker
min_load The available worker with the fewest in-flight requests
min_group The available worker with the fewest active routing keys (distinct keys with requests in flight on it)
delegate The worker the configured policy selects. --policy manual has nothing to delegate to, so there it behaves like min_load

min_load and min_group break ties at random, and each gateway replica counts only the requests it dispatched. The default is random for --policy manual and delegate for the override. One flag sets both, so leave --assignment-mode unset to keep delegation under the override.

Reuse and Failover

Each key remembers up to two workers: a primary and one failover candidate. A request goes to the first remembered worker that is available: healthy, with its circuit breaker not open, and not vetoed by overload protection. Adding workers never moves a key.

When no remembered worker is available, SMG places the request again and records the result:

  • Under the override in delegate mode, the configured policy picks the worker, which becomes the new primary; the previous primary is kept as the failover candidate.
  • In the other modes, and in every mode under --policy manual, the assignment mode picks the worker and adds it behind the existing candidate, dropping the older one when two are already remembered. The earlier worker stays first, so the key returns to it as soon as it is available again.

In-Flight Threshold

Under the override, when a key already has 2 requests in flight on its pinned worker, SMG places the next concurrent request for that key again: through the configured policy in delegate mode, otherwise through the assignment mode. The request goes wherever that placement lands. The threshold triggers a new placement; it is not a hard limit. The placement can pick the pinned worker again, for example when cache_aware finds the conversation's prefix there, so a key can have more than 2 requests in flight on one worker. The pin moves only if the placement picks a different worker that has fewer than 2 of the key's requests in flight, so fleet-wide pressure cannot walk a conversation away from the worker that holds its prefix. Other traffic on the worker never triggers a new placement, and each gateway replica counts its own requests. The manual policy has no such threshold.

Idle Eviction

Every --eviction-interval seconds (default 120, or 60 under the Python launcher), SMG removes pins that have not been used for --max-idle-secs (default 14400, four hours). Each request that reuses a pin refreshes it. A key whose pin was evicted is placed like a new key on its next request.

The map has no size limit: it holds one entry per key used within the idle window (per model and PD leg under the override), and each entry remembers at most two workers. Setting --max-idle-secs to 0 disables eviction, and the map then grows without bound. --eviction-interval 0 also disables it, but SMG refuses to start with that value when the policy is cache_aware, the default.


Scoping

  • By model (override): a key used with several models keeps an independent pin for each, so a conversation that switches models cannot evict its own pins. The manual policy's map is not scoped by model.
  • By PD leg: prefill and decode pins are independent, under both manual and the override.
  • By gateway replica: pins live in each replica's memory. They are not shared through mesh HA, and a restart clears them. With several replicas, send each conversation to the same replica upstream, or use consistent_hashing, which computes the same placement on every replica that sees the same workers as available.
  • Self-hosted workers only: requests to external providers and to the Realtime API are placed by SMG's least-load selector and are never pinned.

Request Body Streaming

SMG can stream a large request body straight to an HTTP worker instead of buffering it, when nothing in the gateway needs to read the body; see Request Streaming. Sticky routing changes that decision:

  • Override enabled: SMG buffers every request body so it can read the rid, even when the request carries routing hints. These requests are counted in smg_router_request_body_path_total{path="buffered", reason="routing_key_override"}.
  • Override disabled: a valid x-smg-routing-tokens header lets cache_aware and bucket place the request without its body, so a large body can stream. A routing-key header does not lift that requirement.

The hint header format is in the Request Headers reference.


Observability

Routed Worker Header

Inference responses from HTTP workers carry x-smg-routed-worker-id: the URL of the worker that served the request, as registered with the gateway, including the @<rank> suffix for data-parallel workers. For a prefill/decode pair it names the decode worker. Compare it across turns to confirm that a conversation sticks.

Responses served by gRPC and ZMQ workers carry the header too, on streaming and non-streaming requests alike, with the same decode-worker rule. The value keeps the worker's registered scheme: grpc:// for a gRPC worker, ipc:// or tcp:// for a ZMQ worker. From these workers, only successful responses carry the header; error responses and Responses API responses do not.

curl -s -D - -o /dev/null http://localhost:30000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "x-smg-routing-key: user-1234" \
  -d '{"model": "meta-llama/Llama-3.1-8B-Instruct", "messages": [{"role": "user", "content": "Hi"}]}' \
  | grep -i x-smg-routed-worker-id
# x-smg-routed-worker-id: http://w2:8000

Metrics

Metric Type Labels Description
smg_manual_policy_branch_total Counter branch Sticky decisions by outcome: occupied_hit (pin reused), occupied_miss (remembered workers unavailable, key placed again), vacant (new key), cap_respill (in-flight threshold reached, request placed again), no_routing_id (manual request without a key), no_healthy_workers
smg_manual_policy_cache_entries Gauge — Entries in the sticky map, updated on each sticky decision
smg_routing_key_source_total Counter source Where the key came from for each keyed request the override routes: rid or header
smg_worker_routing_keys_active Gauge worker Distinct routing keys with requests in flight on each worker; the value min_group balances
smg_consistent_hashing_policy_branch_total Counter branch consistent_hashing outcomes: target_worker_hit, target_worker_miss, routing_key_hit, implicit_key_hit (no routing key; the request's x-forwarded-for or cookie header hashed instead), random_fallback, no_healthy_workers
smg_router_request_body_path_total Counter path, reason reason="routing_key_override" counts bodies buffered because the override is enabled

The manual policy and the override both report through the smg_manual_policy_* metrics. The full list is in the Metrics Reference.

Debug Logs

At --log-level debug, every decision the override makes logs one Sticky routing decision line with the key source (rid or header), the key (prefixed internally with the model and PD leg), the branch, the chosen worker, and the model ID. An eviction sweep that removes pins logs ManualPolicy TTL eviction at info.


Troubleshooting

Turns of one conversation land on different workers

Check these causes in order:

  • The override is off. Without --routing-key-override, SMG ignores the body rid, and every policy except manual, consistent_hashing, and prefix_hash ignores routing-key headers.
  • The rid does not match the suffix grammar. conv-t2, conv_turn2, and conv_T2 are each a key of their own. A high vacant rate in smg_manual_policy_branch_total under steady traffic points here.
  • The key is invalid. A key longer than 128 bytes, or an empty or non-UTF-8 header value, is ignored, on every policy.
  • The header name is not configured. The routing-key headers are the names in --routing-key-headers (x-smg-routing-key by default) on every policy; manual and consistent_hashing also read X-SMG-Routing-Key when none of the configured names carries a valid value.
  • The pinned worker became unavailable. Look for occupied_miss, then check worker health, circuit breakers, and overload vetoes.
  • A key has more than two requests in flight. The extra requests are placed again; look for cap_respill.
  • The pin expired. A key idle for longer than --max-idle-secs starts over.
  • Several gateway replicas. Each replica pins keys independently.
Load is uneven across workers

A pinned key stays on its worker while that worker is available; only new keys and keys placed again see current load. Choose min_load or min_group, or keep delegate with a load-aware policy such as cache_aware.

Large request bodies are always buffered

With the override enabled, SMG buffers every body to read the rid; check smg_router_request_body_path_total{reason="routing_key_override"}. To stream large bodies, disable the override and route with x-smg-routing-tokens instead.

The sticky map keeps growing

Each distinct key (per model and PD leg under the override) keeps an entry until it has been idle for --max-idle-secs. Watch smg_manual_policy_cache_entries, make sure neither --eviction-interval nor --max-idle-secs is 0, and lower --max-idle-secs if conversations are short-lived.


What's Next?

Load Balancing

Every routing policy, including manual and consistent_hashing.

Load Balancing →

Cache-Aware Routing

The policy that usually places a conversation's first turn.

Cache-Aware Routing →

Request Streaming

When SMG buffers a request body and when it streams it.

Request Streaming →

Request Headers

Every header SMG reads or adds, including the routing hints.

Request Headers →

Configuration Reference

All routing policy options.

Configuration Reference →

Metrics Reference

Sticky routing, consistent hashing, and body-path metrics.

Metrics Reference →