Sticky Sessions and Routing Keys¶
A multi-turn conversation runs fastest when each turn lands on the worker that served the turn before it, because that worker still holds the conversation's prefix in its KV cache. Sticky sessions give SMG that property: each request carries a routing key, taken from the request body's rid or from a header, and SMG pins the key to one worker for as long as the key stays in use.
SMG provides stickiness in two forms:
- The
manualpolicy pins every keyed request in its own key-to-worker map. - The routing-key override (
--routing-key-override, alias--sticky-sessions) adds the same pinning on top of any other policy. Keyed requests stick; requests without a key, and by default the first request for each new key, are still placed by the configured policy, such ascache_aware.
Overview¶
Conversation Lineage¶
The key comes from the body rid with turn and retry suffixes stripped, so conv_t1, conv_t2, and conv_t2_r1 all pin as conv.
Routing-Key Headers¶
SMG reads the first valid value from an ordered list of headers, x-smg-routing-key by default; the body rid is the fallback when no header carries one.
Any Policy¶
With the override, keyed requests stick on every policy, and by default the configured policy still chooses where a new key starts.
Bounded and Observable¶
Idle pins expire, a key's requests beyond two in flight are placed again instead of going straight to its pinned worker, and inference responses name the worker that served them.
What a Routing Key Guarantees¶
A routing key is a promise with bounds. While the key's pinned worker is available, a request that carries the key goes to it, up to the override's in-flight threshold described below; the moment the worker is not available, the request is placed elsewhere, with one bound: when every worker the request could use is vetoed at once, the gateway places it on the least-loaded of them (the pin included) rather than refuse it, and counts the fallback; under --worker-overload-shed an all-overloaded pool is refused with 503 worker_overload_protection_shed instead, while a liveness veto still steers. In full:
- Same key, same worker. Every request that resolves to the key (the first valid routing-key header,
x-smg-routing-keyby default, or, under the override, the bodyrid; see Key Precedence) goes to the key's pinned worker, on every policy, with one exception:consistent_hashinghonours a validX-SMG-Target-Workerheader (a healthy worker's index) over the key, as the policy table below states. Undermanualand the override, each key remembers a primary worker and one failover candidate (Reuse and Failover);consistent_hashingkeeps no map and recomputes the key's place on its ring, which moves about one key in N when a worker joins or leaves. - Available means routable. A pinned worker takes the request only while it is healthy (
Ready), its circuit breaker is not open, and no veto excludes it. Two vetoes are on by default: overload protection takes a worker out of routing while its last load report shows8or more waiting requests or KV-cache usage at or above0.8(--worker-overload-waiting-requests,--worker-overload-token-usage;--disable-worker-overload-protectionswitches it off), and the liveness vetoes exclude a worker whose transport failed and has gone silent, or whose engine has stopped producing. Outside prefill/decode mode,consistent_hashingdoes not apply the overload veto to a keyed request (a routing-key header, or the override'srid-derived key, and noX-SMG-Target-Worker): it stays on its ring owner however long the owner's queue, and moves only when the owner is notReady, its circuit breaker is open, or a liveness veto excludes it, and--worker-overload-shedrestores the overload veto. When none of a key's remembered workers is available, the request is placed again: in the override's defaultdelegatemode the new worker becomes the primary and the old one its failover candidate; in the other modes the earlier worker stays first and the key returns to it once it is available (occupied_missin the metrics). When every candidate is vetoed at once there is no elsewhere: the request goes to the least-loaded worker that is routable but for its veto, overloaded ones before stalled ones (a keyedconsistent_hashingrequest ignores the overload veto here too, so it is placed on its ring unless every worker is stalled), counted insmg_worker_overload_fallback_totalorsmg_worker_liveness_fallback_total(stage="selection"); under--worker-overload-shedan all-overloaded pool is refused with503 worker_overload_protection_shedinstead, while a liveness veto still steers. - The spill is bounded. Under the override, once a key has
2of its requests in flight on its pinned worker, the next concurrent request for the key is placed again (cap_respill) rather than queued behind the pin, so one busy conversation cannot stack every turn on one worker (In-Flight Threshold). The threshold triggers a new placement; it is not a cap on the worker: the placement may choose the pinned worker again when the policy prefers it, for example because it holds the conversation's prefix, so more than2of a key's requests can run there. - A failure does not stay put. Outside PD mode, a request that fails on the pinned worker with a retryable answer is retried on another worker while one is available; only when no alternative is left can an attempt return to the worker that just failed, and a worker's own
500is never replayed on it. In PD mode, each retry selects its prefill/decode pair from all available workers again, so an attempt can return to the worker that just failed while alternatives remain; a worker's own500still excludes its pair on gRPC workers, while the HTTP PD router retries a500like any other status (Retries). - Pins expire. Under
manualand the override, when eviction is enabled (--max-idle-secsand--eviction-intervalboth above zero), a key idle for longer than--max-idle-secs(default14400, four hours) is dropped by the next eviction sweep, and its next request starts over as a new key;0on either flag disables eviction (Idle Eviction).consistent_hashingkeeps no pins to evict.
What a routing key does not guarantee:
- One pin across gateway replicas. Each replica keeps its own map (Scoping). Send a conversation to one replica, or use
consistent_hashing, which computes the same placement on every replica that sees the same workers. - A warm cache. The pin keeps a conversation on the worker that served it; whether that worker still holds the conversation's prefix is the engine's cache policy. For requests without a key, cache-aware routing follows the prefix instead.
- Order between concurrent turns. Two requests for one key in flight at once are served in whichever order the worker schedules them, and, under the override, a third may run elsewhere (the bounded spill above).
A minimal configuration, and what each flag changes:
# Keyed requests stick on top of cache-aware placement; a new key starts where
# cache-aware puts it, and the default overload veto and the two-in-flight spill stay on
smg launch --worker-urls http://w1:8000 http://w2:8000 http://w3:8000 \
--policy cache_aware --routing-key-override| What you set | What you get |
|---|---|
--routing-key-override alone |
Keys from x-smg-routing-key or, when no header carries a valid value, the body rid (lineage stripped, at most 128 bytes); the first request of a key is placed by the configured policy, then pinned |
--routing-key-headers a b |
The first of a, b carrying a valid value (non-empty UTF-8, at most 128 bytes) is the key; under the override, a body rid that yields a key (at most 128 bytes after suffix removal) is the fallback when none does |
--disable-worker-overload-protection |
A pinned worker stays the pin however deep its queue; the two-in-flight spill still applies |
--max-idle-secs N |
A key idle for N seconds is dropped by the next eviction sweep, which runs every --eviction-interval seconds; 0 on either flag disables eviction |
Deriving the Routing Key¶
Key Precedence¶
For each request, SMG uses the first of these that yields a key:
- Routing-key headers: the first header named in
--routing-key-headersthat carries a valid value (non-empty UTF-8, at most 128 bytes); under the override, the key is that value with its lineage suffixes stripped, on every policy. Without the override,manualandconsistent_hashingplace on the raw value (see the warning below).manualandconsistent_hashingalso tryX-SMG-Routing-Keywhen no configured header carries a valid value; the override map reads the configured names only. - Body
rid(override enabled only): theridwith its lineage suffixes stripped, when the result is at most 128 bytes. - No key: the configured policy places the request, and nothing is pinned.
A valid routing-key header wins even when the body carries a rid: an explicit key outranks request-ID lineage, so the client or proxy that sets the header chooses the pin, and a proxy must not stamp a per-request value into a routing-key header. manual and consistent_hashing apply this order themselves, with the same validity rules and the extra X-SMG-Routing-Key step.
Body rid Lineage¶
rid is an SGLang-style request ID accepted in the body of chat completions, completions, messages, /generate, embeddings, classify, and rerank requests (for rerank, the first ID of a list). The Responses API has no rid, so Responses requests can pin only through a header. SMG reads rid for routing only when --routing-key-override is enabled.
SMG removes one trailing retry suffix, then one trailing turn suffix:
<base>[_t<digits>][_r<digits>] -> <base>- A suffix is an underscore, a lowercase
t(turn) orr(retry), and one or more ASCII digits, at the very end of therid. - The retry suffix is removed first, so it has to come last:
conv_t2_r1becomesconv, whileconv_r1_t2becomesconv_r1. - Each suffix is removed at most once.
- If nothing would be left, the whole
ridis the key. - If the resulting key is longer than 128 bytes, SMG ignores the
rid; a request whose headers carried no valid key is then placed without a key.
rid |
Routing key |
|---|---|
conv |
conv |
conv_t2 |
conv |
conv_t2_r1 |
conv |
conv_r1 |
conv |
conv_t1_t2 |
conv_t1 |
conv_r1_t2 |
conv_r1 |
conv_t, conv_tx1, conv_T2 |
Unchanged |
under_scored_id |
under_scored_id |
_t1 |
_t1 |
Stripping only computes the key; the request keeps its full rid. On gRPC workers outside PD mode, SMG sends the rid as the engine request ID, so the suffixes let every turn and retry carry a distinct ID while sharing one key.
Routing-Key Headers¶
--routing-key-headers takes an ordered list of header names; the default is x-smg-routing-key. Each name must be a valid HTTP header name, or SMG refuses to start, and names match case-insensitively. For each request, SMG walks the list and takes the first header whose value is:
- non-empty,
- valid UTF-8, and
- at most 128 bytes.
A value that fails these checks is skipped without an error, and SMG tries the next name. With the override enabled, header keys get the same lineage stripping as a rid, so a proxy that forwards conv_t2 in a header pins the same entry as a body rid of conv_t2.
# Read an upstream proxy's header first, and keep the SMG header as a fallback
smg launch --worker-urls http://w1:8000 http://w2:8000 \
--routing-key-override \
--routing-key-headers x-routing-key x-smg-routing-keyConfiguration¶
Enable the override on top of any policy, here cache_aware:
smg launch --worker-urls http://w1:8000 http://w2:8000 http://w3:8000 \
--policy cache_aware \
--routing-key-overrideThen give each turn of a conversation a rid that shares a base:
curl http://localhost:30000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "meta-llama/Llama-3.1-8B-Instruct",
"rid": "chat-7f3a_t1",
"messages": [{"role": "user", "content": "Plan a three-day trip to Kyoto."}]
}'The next turn sends "rid": "chat-7f3a_t2", and a retry of that turn sends "rid": "chat-7f3a_t2_r1". All three pin under the key chat-7f3a. A client that cannot set rid sends -H "x-smg-routing-key: chat-7f3a" instead.
| Flag | Default | Description |
|---|---|---|
--routing-key-override |
Off | Pin keyed requests on any policy. Alias: --sticky-sessions |
--routing-key-headers |
x-smg-routing-key |
Ordered header names to read the key from; the first valid value wins |
--assignment-mode |
delegate for the override, random for --policy manual |
How a new key picks its worker: random, min_load, min_group, or delegate |
--max-idle-secs |
14400 |
Seconds a pin can go unused before it is evicted. Alias: --sticky-key-idle-secs |
--eviction-interval |
120 |
Seconds between eviction sweeps. Also sets the cache-aware tree eviction interval |
--assignment-mode, --max-idle-secs, and --eviction-interval apply to both the manual policy and the override. See the Configuration Reference for every routing option.
Manual Policy or Override?¶
| Aspect | --policy manual |
--routing-key-override |
|---|---|---|
| First worker for a new key | --assignment-mode (default random) |
The configured policy (delegate, the default) |
| Requests without a key | Placed by --assignment-mode, not pinned |
Placed by the configured policy, unchanged |
| Key sources | The first valid header of --routing-key-headers, then X-SMG-Routing-Key, then the body rid when the override is also enabled (lineage stripped, at most 128 bytes) |
--routing-key-headers, then the body rid (lineage stripped, at most 128 bytes) |
| Lineage stripping of header keys | Only with --routing-key-override also set |
Yes |
| Pins scoped by model | No (by PD leg only) | Yes (by model and PD leg) |
| Per-key in-flight threshold | No | Yes |
Choose the override to keep another policy, usually cache_aware, in charge of where conversations start. Choose manual when placement should depend only on the assignment mode.
How the Override Composes with Each Policy¶
| Configured policy | With --routing-key-override, a keyed request is |
Without the override, a routing key is |
|---|---|---|
cache_aware, round_robin, random, power_of_two, least_load, bucket, passthrough |
Pinned in the override's sticky map; under delegate, the policy chooses a new key's first worker |
Ignored |
prefix_hash |
Pinned in the override's sticky map; under delegate, the policy chooses a new key's first worker |
Hashed onto the ring in place of the prompt's token IDs or text, which prefix_hash otherwise hashes |
manual |
Pinned in the manual policy's own map, from the first valid routing-key header (--routing-key-headers, lineage-stripped, then X-SMG-Routing-Key, raw) and the body rid as the fallback (at most 128 bytes after stripping) |
Pinned from the first valid routing-key header |
consistent_hashing |
Hashed onto the ring, from the routing-key header first (configured names lineage-stripped) and the body rid as the fallback (at most 128 bytes after stripping); X-SMG-Target-Worker still wins |
Hashed onto the ring |
Requests without a key go to the configured policy unchanged. Under delegate assignment, the override's default, the first request of a conversation is placed exactly as it would be without the override (by prefix match under cache_aware, for example), and later turns follow it. Pinning keeps the worker's cached prefix reachable on every turn; it does not make the engine keep that cache any longer.
In PD mode, the prefill and decode legs are pinned independently.
Assignment Modes and Eviction¶
Assignment Modes¶
--assignment-mode decides where a key goes the first time SMG sees it, and where it moves when none of its remembered workers is available.
| Mode | A new key goes to |
|---|---|
random |
A random available worker |
min_load |
The available worker with the fewest in-flight requests |
min_group |
The available worker with the fewest active routing keys (distinct keys with requests in flight on it) |
delegate |
The worker the configured policy selects. --policy manual has nothing to delegate to, so there it behaves like min_load |
min_load and min_group break ties at random, and each gateway replica counts only the requests it dispatched. The default is random for --policy manual and delegate for the override. One flag sets both, so leave --assignment-mode unset to keep delegation under the override.
Reuse and Failover¶
Each key remembers up to two workers: a primary and one failover candidate. A request goes to the first remembered worker that is available: healthy, with its circuit breaker not open, and not vetoed by overload protection or the liveness vetoes. Adding workers never moves a key.
When no remembered worker is available, SMG places the request again and records the result:
- Under the override in
delegatemode, the configured policy picks the worker, which becomes the new primary; the previous primary is kept as the failover candidate. - In the other modes, and in every mode under
--policy manual, the assignment mode picks the worker and adds it behind the existing candidate, dropping the older one when two are already remembered. The earlier worker stays first, so the key returns to it as soon as it is available again.
In-Flight Threshold¶
Under the override, when a key already has 2 requests in flight on its pinned worker, SMG places the next concurrent request for that key again: through the configured policy in delegate mode, otherwise through the assignment mode. The request goes wherever that placement lands. The threshold triggers a new placement; it is not a hard limit. The placement can pick the pinned worker again, for example when cache_aware finds the conversation's prefix there, so a key can have more than 2 requests in flight on one worker. The pin moves only if the placement picks a different worker that has fewer than 2 of the key's requests in flight, so fleet-wide pressure cannot walk a conversation away from the worker that holds its prefix. Other traffic on the worker never triggers a new placement, and each gateway replica counts its own requests. The manual policy has no such threshold.
Idle Eviction¶
Every --eviction-interval seconds (default 120), SMG removes pins that have not been used for --max-idle-secs (default 14400, four hours). Each request that reuses a pin refreshes it. A key whose pin was evicted is placed like a new key on its next request.
The map has no size limit: it holds one entry per key used within the idle window (per model and PD leg under the override), and each entry remembers at most two workers. Setting --max-idle-secs to 0 disables eviction, and the map then grows without bound. --eviction-interval 0 also disables it, but SMG refuses to start with that value when the policy is cache_aware, the default.
Scoping¶
- By model (override): a key used with several models keeps an independent pin for each, so a conversation that switches models cannot evict its own pins. The
manualpolicy's map is not scoped by model. - By PD leg: prefill and decode pins are independent, under both
manualand the override. - By gateway replica: pins live in each replica's memory. They are not shared through mesh HA, and a restart clears them. With several replicas, send each conversation to the same replica upstream, or use
consistent_hashing, which computes the same placement on every replica that sees the same workers as available. - Self-hosted workers only: requests to external providers and to the Realtime API are placed by SMG's least-load selector and are never pinned.
Request Body Streaming¶
SMG can stream a large request body straight to an HTTP worker instead of buffering it, when nothing in the gateway needs to read the body; see Request Streaming. Sticky routing changes that decision:
- Override enabled: SMG buffers every request body so it can read the
rid, even when the request carries routing hints. These requests are counted insmg_router_request_body_path_total{path="buffered", reason="routing_key_override"}. - Override disabled: a valid
x-smg-routing-tokensheader letscache_awareandbucketplace the request without its body, so a large body can stream. A routing-key header does not lift that requirement.
The hint header format is in the Request Headers reference.
Observability¶
Routed Worker Header¶
Inference responses from HTTP workers carry x-smg-routed-worker-id: the URL of the worker that served the request, as registered with the gateway (credentials in a worker URL are masked), including the @<rank> suffix for data-parallel workers. For a prefill/decode pair it names the decode worker. Compare it across turns to confirm that a conversation sticks.
Responses served by gRPC and ZMQ workers carry the header too, on streaming and non-streaming requests alike, with the same decode-worker rule. The value keeps the worker's registered scheme: grpc:// for a gRPC worker, ipc:// or tcp:// for a ZMQ worker. From these workers, only successful responses carry the header; error responses and Responses API responses do not.
curl -s -D - -o /dev/null http://localhost:30000/v1/chat/completions \
-H "Content-Type: application/json" \
-H "x-smg-routing-key: user-1234" \
-d '{"model": "meta-llama/Llama-3.1-8B-Instruct", "messages": [{"role": "user", "content": "Hi"}]}' \
| grep -i x-smg-routed-worker-id
# x-smg-routed-worker-id: http://w2:8000Metrics¶
| Metric | Type | Labels | Description |
|---|---|---|---|
smg_manual_policy_branch_total |
Counter | branch |
Sticky decisions by outcome: occupied_hit (pin reused), occupied_miss (remembered workers unavailable, key placed again), vacant (new key), cap_respill (in-flight threshold reached, request placed again), no_routing_id (manual request without a key), no_healthy_workers |
smg_manual_policy_cache_entries |
Gauge | — | Entries in the sticky map, updated on each sticky decision |
smg_routing_key_source_total |
Counter | source |
Where the key came from for each keyed request the override routes: rid or header |
smg_worker_routing_keys_active |
Gauge | worker |
Distinct routing keys with requests in flight on each worker; the value min_group balances |
smg_consistent_hashing_policy_branch_total |
Counter | branch |
consistent_hashing outcomes: target_worker_hit, target_worker_miss, routing_key_hit, implicit_key_hit (no explicit key; hashed on X-Forwarded-For or Cookie), random_fallback, no_healthy_workers |
smg_router_request_body_path_total |
Counter | path, reason |
reason="routing_key_override" counts bodies buffered because the override is enabled |
The manual policy and the override both report through the smg_manual_policy_* metrics. The full list is in the Metrics Reference.
Debug Logs¶
At --log-level debug, every decision the override makes logs one Sticky routing decision line with the key source (rid or header), the key (prefixed internally with the model and PD leg), the branch, the chosen worker, and the model ID. An eviction sweep that removes pins logs ManualPolicy TTL eviction at info.
Troubleshooting¶
Turns of one conversation land on different workers
Check these causes in order:
- The override is off. Without
--routing-key-override, SMG ignores the bodyrid, and every policy exceptmanual,consistent_hashing, andprefix_hashignores routing-key headers. - The
riddoes not match the suffix grammar.conv-t2,conv_turn2, andconv_T2are each a key of their own. A highvacantrate insmg_manual_policy_branch_totalunder steady traffic points here. - The key is invalid. A key longer than 128 bytes, or an empty or non-UTF-8 header value, is ignored, on every policy.
- The header name is not configured. The routing-key headers are the names in
--routing-key-headers(x-smg-routing-keyby default) on every policy;manualandconsistent_hashingalso readX-SMG-Routing-Keywhen none of the configured names carries a valid value. - The pinned worker became unavailable. Look for
occupied_miss, then check worker health, circuit breakers, and overload vetoes (overload vetoes do not moveconsistent_hashingkeys). - A key has more than two requests in flight. The extra requests are placed again; look for
cap_respill. - The pin expired. A key idle for longer than
--max-idle-secsstarts over. - Several gateway replicas. Each replica pins keys independently.
Load is uneven across workers
A pinned key stays on its worker while that worker is available; only new keys and keys placed again see current load. Choose min_load or min_group, or keep delegate with a load-aware policy such as cache_aware.
Large request bodies are always buffered
With the override enabled, SMG buffers every body to read the rid; check smg_router_request_body_path_total{reason="routing_key_override"}. To stream large bodies, disable the override and route with x-smg-routing-tokens instead.
The sticky map keeps growing
Each distinct key (per model and PD leg under the override) keeps an entry until it has been idle for --max-idle-secs. Watch smg_manual_policy_cache_entries, make sure neither --eviction-interval nor --max-idle-secs is 0, and lower --max-idle-secs if conversations are short-lived.
What's Next?¶
Cache-Aware Routing¶
The policy that usually places a conversation's first turn.