KV Events Cache-Aware Routing¶

This guide wires gRPC workers that publish KV cache events to SMG's cache_aware policy, so the gateway routes each request to the worker whose KV cache actually holds the longest prefix of it. SGLang is the main example; vLLM and TokenSpeed workers are set up the same way.

Why event-driven?¶

Without events, cache_aware predicts cache contents from its own routing history. With events, it routes against the blocks each engine reports it has stored and evicted.

The choice is made per model, not per worker. Once any worker of a model has streamed events into the index, every token-bearing request for that model is scored against the event index, and workers that do not stream events can then receive that model's requests only through the load-based fallback. Enable events on every worker of a model.

Event-driven routing also needs the default --cache-index tree: with --cache-index hash, the gateway never consults the event index.

How the pieces fit together¶

client
  │ OpenAI-compatible HTTP
  ▼
SMG gateway: cache_aware policy ◀── PositionalIndexer (one per model)
  │                                        ▲
  │ gRPC requests                          │ KvEventMonitor
  ▼                                        │ (SubscribeKvEvents stream)
worker: smg-grpc-servicer ─────────────────┘
  ▲
  │ ZMQ PUB → SUB, on the worker host (--kv-events-config)
engine scheduler (SGLang, vLLM or TokenSpeed)

1. The engine's scheduler publishes block-stored, block-removed and all-blocks-cleared events on a ZMQ PUB socket configured by --kv-events-config.
2. smg-grpc-servicer, which serves the worker's gRPC API, subscribes to the engine's publishers locally and re-publishes the events as the server-streaming RPC SubscribeKvEvents.
3. SMG's KvEventMonitor opens one subscription per gRPC worker and feeds the events into a per-model PositionalIndexer; cache_aware scores token-bearing requests against that index. If a stream breaks, the monitor reconnects and either resumes or rebuilds the worker's entries (see Recovery after gaps).

Step 1 — Launch the SGLang worker¶

Install the SGLang extra of the servicer, then launch the SGLang server with both --smg-grpc-mode and --kv-events-config:

pip install "smg-grpc-servicer[sglang]"

python -m sglang.launch_server \
  --model-path meta-llama/Llama-3.1-8B-Instruct \
  --host 0.0.0.0 \
  --port 50051 \
  --smg-grpc-mode \
  --page-size 16 \
  --kv-events-config '{"publisher":"zmq","endpoint":"tcp://*:5557","replay_endpoint":"tcp://*:5558","topic":"kv-events"}'

What each flag does:

--kv-events-config field reference¶

All fields and defaults match SGLang's KVEventsConfig (see python/sglang/srt/disaggregation/kv_events.py upstream):

With data parallelism, each rank publishes on endpoint port + dp_rank and serves replay on replay_endpoint port + dp_rank; rank 0 keeps the configured ports. Pick port ranges that do not overlap (5557 and 5558 collide as soon as there are two ranks). The servicer subscribes to every rank's publisher and relays them as one stream, each batch tagged with its rank, so the index sees every rank's cache.

Alternative: launch a vLLM worker¶

vLLM publishes KV cache events on a ZMQ socket; enable them with --kv-events-config and run the worker in SMG gRPC mode:

pip install "smg-grpc-servicer[vllm]"

# --grpc serves vLLM through smg-grpc-servicer instead of the OpenAI HTTP server;
# --kv-events-config turns on KV-event publishing:
vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --grpc \
  --host 0.0.0.0 \
  --port 50051 \
  --kv-events-config '{"enable_kv_cache_events": true, "publisher": "zmq", "endpoint": "tcp://*:5557", "replay_endpoint": "tcp://*:5558", "topic": "kv-events"}'

Event-driven routing needs the worker in SMG gRPC mode (--grpc; pip install "vllm[grpc]" pulls in the same servicer package), because KV events stream over the SubscribeKvEvents RPC. See gRPC Workers for other launch flags.

SMG learns the block size from the BlockStored events themselves; set SMG's --block-size to vLLM's --block-size so the approximate token tree uses the same pages.

Alternative: launch a TokenSpeed worker¶

TokenSpeed's scheduler publishes KV cache events on a ZMQ socket; enable them with --kv-events-config. The TokenSpeed gRPC server is the SMG gRPC entrypoint, so there is no separate --grpc flag:

# TokenSpeed is installed from source (engine + kernel + scheduler); see
# scripts/ci_install_tokenspeed.sh in the smg repository. Install the servicer's extra deps:
pip install "smg-grpc-servicer[tokenspeed]"

# --kv-events-config turns on KV-event publishing in the scheduler:
python -m smg_grpc_servicer.tokenspeed \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --host 0.0.0.0 \
  --port 50051 \
  --kv-events-config '{"enable_kv_cache_events": true, "publisher": "zmq", "endpoint": "tcp://*:5557", "topic": "kv-events"}'

--kv-events-config is parsed by TokenSpeed's KVEventsConfig.from_cli. The TokenSpeed servicer does not replay missed events; a sequence gap costs that worker at most the missed batches, bounded by SMG (see Recovery after gaps).

SMG learns the block size from the BlockStored events; TokenSpeed's cache reuse granularity is --prefix-granularity (default 64 tokens; older builds call it --block-size, which remains an alias), so set SMG's --block-size to the same value.

Step 2 — Launch SMG¶

Point SMG at the gRPC workers and select cache_aware:

smg launch \
  --worker-urls grpc://worker-1:50051 grpc://worker-2:50051 \
  --model-path meta-llama/Llama-3.1-8B-Instruct \
  --policy cache_aware \
  --block-size 16 \
  --host 0.0.0.0 \
  --port 30000

worker-2 is a second worker launched the same way, on another host. On a single host, give each worker its own --port and its own event ports, and leave room for each SGLang worker's sidecar on --port + 1.

The flags that matter for event-driven routing:

--model-path tells the gateway which tokenizer to use, as in any gRPC-worker deployment (gRPC Workers). For the other routing knobs, see Tuning.

Block size alignment¶

The event index cuts each request's token IDs into blocks and looks the blocks up by content hash. If the block size differs from the one the engine used, no block matches, and the model's token requests are routed by load alone.

SMG picks the block size per model, in this order:

1. Learned from events (highest priority): each worker's subscription records the block size of the first stored block it receives; the most recent value wins.
2. --block-size (router-wide default).

The kv_block_size field of a worker spec registered through the admin API (POST /workers) is accepted but not applied in v1.11.0. In practice, keep the engine's page size (SGLang --page-size, vLLM --block-size, TokenSpeed --prefix-granularity) and SMG's --block-size equal, and let SMG correct itself once events arrive. The approximate token tree always uses --block-size. Serve each model with one page size: with mixed sizes, the model's block size is whichever a worker reported last.

Step 3 — Send a request¶

The API surface is unchanged:

curl http://localhost:30000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "meta-llama/Llama-3.1-8B-Instruct",
    "messages": [
      {"role": "user", "content": "Hello, who are you?"}
    ]
  }'

Send the same prompt twice. The first request has no cached prefix and goes to the lowest-expected-wait worker. Once that worker reports the blocks it stored, the second request is scored against them and lands on the same worker, as long as that worker is not far busier than the rest of the fleet.

Verifying event delivery¶

The gateway logs these lines when the path is live (timestamps, targets and source locations omitted).

1. Subscription started. When SMG registers a gRPC worker under cache_aware, KvEventMonitor logs:

INFO Starting KV event subscription worker_url=grpc://worker-1:50051 model_id=meta-llama/Llama-3.1-8B-Instruct
INFO KV event stream connected worker_url=grpc://worker-1:50051 start_seq=0

If you do not see the first line for a worker, the worker is not a gRPC worker, the gateway's --policy is not cache_aware (a policy worker label alone does not start subscriptions), or the worker's model is routed by another policy. If events are disabled on the worker, the servicer rejects the stream and SMG stops trying:

WARN Backend does not implement SubscribeKvEvents, disabling KV event subscription for this worker worker_url=grpc://worker-1:50051

2. Backend block size learned. Once the first stored block arrives, SMG records the backend's block size:

INFO Learned block_size from KV event model_id=meta-llama/Llama-3.1-8B-Instruct block_size=16

3. Routing decision uses the index. With RUST_LOG=info,smg::policies::cache_aware=debug, each routed request logs Event-driven routing: overlap match with the chosen worker and a branch of event_hit (affinity held) or event_spill (the spill gate moved the request), or Event-driven routing: no overlap, expected-wait fallback.

While the model's event index is empty, token requests route on the approximate token tree, so cache affinity keeps working, only less accurately.

Recovery after gaps¶

Every batch on the stream carries a sequence number. KvEventMonitor applies batches in order and skips duplicates. When it sees a gap in the sequence, it reconnects immediately and asks the servicer to resume after the last batch it applied; after a stream error or a clean close, it reconnects with exponential backoff from 100 ms up to 5 s. A subscription call that is not answered within 2 s — a port that accepts but does not serve yet, such as an engine still starting — is abandoned and retried.

The servicer waits up to five seconds for each replay message before declaring a gap unfillable.

On SGLang and vLLM workers, the default Python servicer keeps nothing between subscriptions: the stream's numbering is per call, a resume cursor is answered with OUT_OF_RANGE, and a new subscription carries live events only. After any stream break between SMG and the servicer, SMG therefore drops the worker's index entries and rebuilds them from live events. The rebuilt index is not a snapshot of the engine's cache: affinity to that worker returns as it caches new prefixes.

Workers running a Rust servicer keep more. Its relay follows the engine's publisher for the servicer's whole lifetime: a reconnecting gateway is served the batches after its cursor from history, and a new subscription is answered with either the engine's complete event history or an in-band snapshot of the engine's live blocks, which SMG applies in place of whatever it held for that worker. Cache affinity then survives gateway restarts and long disconnects without waiting for new stores.

SMG logs Sequence gap detected, reconnecting for replay from seq N on a gap, and KV event replay cursor expired; clearing worker state and requesting a current snapshot or KV event subscriber fell behind; clearing worker state and requesting a current snapshot when it drops a worker's entries. A restart of the engine's own publisher (its sequence numbers start over) also clears the worker's entries — the engine's cache is empty again — and the stream continues from the new sequence.

Bounding the index¶

The event index drops a worker's entries when its engine reports evictions or the worker is removed. If an engine stops reporting evictions, entries accumulate. Two optional bounds, both off by default, are checked every 30 seconds:

A pass that prunes entries logs Pruned positional indexer with the counts. Both flags exist only in the Rust smg binary: the Python launcher that pip install smg and the container image use does not accept them.

Tuning¶

--cache-threshold does not apply to event-driven routing: any worker holding at least the request's first full block is a candidate. See Cache-Aware Routing for the full list of knobs.

Monitoring¶

The smg_cache_aware_policy_branch_total and smg_cache_aware_match_ratio metrics count tree-mode decisions only; use the debug log above to inspect event-driven decisions.

Caveats¶

• gRPC only. Events stream over gRPC through smg-grpc-servicer. HTTP workers use the approximate string tree, ZMQ (ipc://) workers use the approximate token tree, and MLX workers do not implement SubscribeKvEvents.
• TokenSpeed: DP rank 0 only. The SGLang and vLLM servicers relay every data-parallel rank's publisher; the TokenSpeed servicer subscribes to rank 0's only, so for data-parallel TokenSpeed workers the index reflects rank 0's cache.
• One block size per model. If workers serving the same model use different page sizes, the policy uses whichever block size a worker reported last. Keep page sizes homogeneous within a model.
• Per-gateway index. Each gateway subscribes to every worker itself. With --enable-mesh, gateways synchronize the approximate trees, not the event index.
• No cache partitions. The event index ignores the cache_salt, extra_key and LoRA fields that partition the approximate trees.
• Unaligned stores are skipped. The servicers drop any BlockStored event whose token count is not its block count times its block size (vLLM can emit these for Mamba models); those blocks never reach the index.
• Replay needs replay_endpoint. The SGLang and vLLM servicers fill gaps only when replay_endpoint is set; without it, a gap costs the worker's index entries. The TokenSpeed servicer never replays, and SMG bounds what a gap costs instead (see Recovery after gaps).

Reference¶

• Policy implementation: model_gateway/src/policies/cache_aware.rs
• Event subscription manager: model_gateway/src/worker/kv_event_monitor.rs (gap settling and resyncs: model_gateway/src/worker/kv_event_recovery.rs)
• KV event proto: crates/grpc_client/proto/common.proto (messages SubscribeKvEventsRequest, KvEventBatch, KvSnapshotChunk, KvCacheEvent, KvBlocksStored, KvBlocksRemoved, KvCacheCleared)
• Engine-neutral relay (per-rank subscriptions, gap replay, normalization): grpc_servicer/smg_grpc_servicer/kv_relay.py; the Rust servicer's relay with history and state snapshots: crates/engine_servicer/src/kv_events.rs
• SGLang servicer: grpc_servicer/smg_grpc_servicer/sglang/servicer.py (SubscribeKvEvents) + per-rank sources grpc_servicer/smg_grpc_servicer/sglang/kv_events.py
• vLLM servicer: grpc_servicer/smg_grpc_servicer/vllm/servicer.py (SubscribeKvEvents) + config resolver grpc_servicer/smg_grpc_servicer/vllm/kv_events.py
• TokenSpeed servicer: grpc_servicer/smg_grpc_servicer/tokenspeed/servicer.py (SubscribeKvEvents, a single-rank bridge over grpc_servicer/smg_grpc_servicer/kv_events.py) + config resolver grpc_servicer/smg_grpc_servicer/tokenspeed/kv_events.py
• SGLang upstream config: python/sglang/srt/disaggregation/kv_events.py (class KVEventsConfig)
• vLLM upstream config: vllm/config/kv_events.py (class KVEventsConfig)
• TokenSpeed upstream config: python/tokenspeed/runtime/pd/kv_events.py (class KVEventsConfig)