KV Events Cache-Aware Routing

This guide wires gRPC workers that publish KV cache events to SMG's cache_aware policy, so the gateway routes each request to the worker whose KV cache actually holds the longest prefix of it. SGLang is the main example; vLLM and TokenSpeed workers are set up the same way.


Why event-driven?

Without events, cache_aware predicts cache contents from its own routing history. With events, it routes against the blocks each engine reports it has stored and evicted.

Mode Index Input Used when
Event-driven Chain index, built from engine events Token IDs The model's event index has data
Approximate token tree TokenTree, built from routing decisions Token IDs The request has token IDs and the model has no event data
Approximate string tree Tree, built from routing decisions Raw text HTTP requests without token IDs

The choice is made per model, not per worker. Once any worker of a model has streamed events into the index, every token-bearing request for that model is scored against the event index, and workers that do not stream events can then receive that model's requests only through the load-based fallback. Keep events on for every worker of a model.

Event-driven routing also needs the default --cache-index tree: with --cache-index hash, the gateway never consults the event index.


How the pieces fit together

client
  │ OpenAI-compatible HTTP
  ▼
SMG gateway: cache_aware policy ◀── chain index (one per model)
  │                                        ▲
  │ gRPC requests                          │ KvEventMonitor
  ▼                                        │ (SubscribeKvEvents stream)
worker: smg-grpc-servicer ─────────────────┘
  ▲
  │ ZMQ PUB → SUB, on the worker host
engine scheduler (SGLang, vLLM or TokenSpeed)
  1. The engine's scheduler publishes block-stored, block-removed and all-blocks-cleared events on a ZMQ PUB socket. The servicer configures the publisher itself when the worker is launched without --kv-events-config; an explicit config replaces that default.
  2. smg-grpc-servicer, which serves the worker's gRPC API, subscribes to the engine's publishers locally and re-publishes the events as the server-streaming RPC SubscribeKvEvents.
  3. SMG's KvEventMonitor opens one subscription per gRPC worker placed by cache_aware and feeds the events into a per-model chain index; cache_aware scores token-bearing requests against that index. If a stream breaks, the monitor reconnects and either resumes or rebuilds the worker's entries (see Recovery after gaps).

A hybrid model — one whose engine keeps sliding-window attention or recurrent-state (Mamba-style) layers in KV cache groups beside the full-attention layers — publishes its KV events per cache group: each group's stores and removals carry the group's index, and its stores the group's kind and window. The prefix index is built from the main-attention group alone; the other groups' events tell the gateway which blocks each group still retains on each worker, and a worker's matched-prefix score is bounded by what its sliding-window and state groups can still serve — the same reuse rule the engine applies to its own cache hit, so a worker whose window has moved past a prefix is no longer preferred for it.

A group the gateway cannot interpret leaves the plain score standing, as does a worker whose group events began only after its first blocks were indexed (until that worker's next cache clear). The default Rust servicer forwards group events; the deprecated Python servicer drops them, so its workers are always scored at the plain overlap. Plain full-attention models publish no cache groups and score as before, and in every case the score stays an upper bound of what the engine actually reuses. See Cache-Aware Routing for the scoring details.


Step 1 — Launch the SGLang worker

A worker launched with --grpc-mode and no --kv-events-config already publishes: the servicer turns SGLang's ZMQ KV-event publisher on by itself, at a port found free on the host rather than SGLang's fixed default (with data parallelism, a block of free ports, one per rank), so workers that share a host do not collide. The start log says so: KV event publishing enabled: no --kv-events-config was given, .... Setting SMG_SGLANG_SERVICER_KV_EVENTS=0 in the worker's environment leaves the publisher off.

That default supplies no replay_endpoint, so a missed batch costs the worker's index entries (see Recovery after gaps). Pass --kv-events-config to pin the endpoint and enable gap replay. An explicit config is kept exactly as given — include "publisher":"zmq", since one that leaves it out publishes nothing.

Install the SGLang extra of the servicer, then launch the SGLang server:

pip install "smg-grpc-servicer[sglang]"

python -m sglang.launch_server \
  --model-path meta-llama/Llama-3.1-8B-Instruct \
  --host 0.0.0.0 \
  --port 50051 \
  --grpc-mode \
  --page-size 16 \
  --kv-events-config '{"publisher":"zmq","endpoint":"tcp://*:5557","replay_endpoint":"tcp://*:5558","topic":"kv-events"}'

What each flag does:

Flag Why
--grpc-mode Serves SMG's gRPC SglangScheduler service from smg-grpc-servicer (needs SGLang 0.5.21 or later) instead of SGLang's HTTP server. Required for SMG to talk to this worker over gRPC.
--page-size 16 The KV cache page size, in tokens. Use the same value for SMG's --block-size (see Block size alignment).
--kv-events-config Optional: without it the servicer publishes at a free port, with no gap replay. An explicit JSON object, parsed by SGLang's KVEventsConfig.from_cli, replaces that default exactly as given: "publisher":"zmq" turns publishing on (leaving it out means the default "null", which publishes nothing), and replay_endpoint lets the servicer fill gaps in the stream from SGLang's replay buffer.

--kv-events-config field reference

All fields and defaults match SGLang's KVEventsConfig (see python/sglang/srt/disaggregation/kv_events.py upstream):

Field Default Notes
publisher "null" Set to "zmq" to publish. An explicit config is kept as given even when it turns publishing off: with any other value the servicer answers SubscribeKvEvents with UNIMPLEMENTED, and SMG stops subscribing to that worker.
endpoint "tcp://*:5557" ZMQ PUB address. The publisher binds wildcard endpoints such as tcp://*:PORT (and ipc:// or inproc:// ones) and connects to any other address, which leaves nothing listening, so keep the wildcard form. The servicer connects to it locally, rewriting * and 0.0.0.0 to 127.0.0.1.
topic "" ZMQ topic. The servicer reads the same config and subscribes to this topic, so any value works.
replay_endpoint null ZMQ ROUTER address that serves recent batches for replay. When set, the servicer fills gaps in each rank's events from it before forwarding later batches; without it, a missed batch costs the worker's index entries (see Recovery after gaps).
buffer_steps 10000 Number of recent batches kept for replay.
hwm 100000 ZMQ high-water mark: once this many messages are queued for a subscriber that is not keeping up, new events are dropped. The servicer uses the same value for its receive queue.
max_queue_size 100000 Queue between the scheduler and SGLang's publisher thread.

With data parallelism, each rank publishes on endpoint port + dp_rank and serves replay on replay_endpoint port + dp_rank; rank 0 keeps the configured ports. Pick port ranges that do not overlap (5557 and 5558 collide as soon as there are two ranks). The servicer subscribes to every rank's publisher and relays them as one stream, each batch tagged with its rank, so the index sees every rank's cache.

Alternative: launch a vLLM worker

vLLM works the same way: in SMG gRPC mode, a worker launched without --kv-events-config already publishes — the servicer turns vLLM's ZMQ publisher on at a port found free on the host, and SMG_VLLM_SERVICER_KV_EVENTS=0 leaves it off. Pass --kv-events-config to pin the endpoint and enable gap replay:

pip install "smg-grpc-servicer[vllm]"

# --grpc serves vLLM through smg-grpc-servicer instead of the OpenAI HTTP server;
# --kv-events-config pins the event endpoint and adds gap replay:
vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --grpc \
  --host 0.0.0.0 \
  --port 50051 \
  --kv-events-config '{"enable_kv_cache_events": true, "publisher": "zmq", "endpoint": "tcp://*:5557", "replay_endpoint": "tcp://*:5558", "topic": "kv-events"}'

Event-driven routing needs the worker in SMG gRPC mode (--grpc; pip install "vllm[grpc]" pulls in the same servicer package), because KV events stream over the SubscribeKvEvents RPC. See gRPC Workers for other launch flags.

Field Why
enable_kv_cache_events: true vLLM's master switch. An explicit config is kept as given, so one without it publishes nothing, and SMG stops subscribing to the worker.
publisher: "zmq" Selects the ZMQ publisher the servicer reads. Recent vLLM versions default to it once events are enabled.
endpoint / topic As for SGLang; keep the wildcard tcp://*:PORT form. With data parallelism the port is endpoint port + dp_rank, and the servicer subscribes to every rank.
replay_endpoint As for SGLang: the servicer fills gaps in each rank's events from it before forwarding later batches; without it, a missed batch costs the worker's index entries (see Recovery after gaps). Each rank serves replay on replay_endpoint port + dp_rank.

SMG learns the block size from the BlockStored events themselves; set SMG's --block-size to vLLM's --block-size so the approximate token tree uses the same pages.

Alternative: launch a TokenSpeed worker

TokenSpeed works the same way, and the TokenSpeed gRPC server is the SMG gRPC entrypoint, so there is no separate --grpc flag. A worker launched without --kv-events-config already publishes — the servicer turns the scheduler's ZMQ publisher on at a port found free on the host, and SMG_TOKENSPEED_SERVICER_KV_EVENTS=0 leaves it off.

An encode worker (--disaggregation-mode encode) gets no automatic publisher: it has no KV cache and is left as given. A hybrid model (Gated DeltaNet, Mamba or KDA layers) keeps its recurrent state in State groups of TokenSpeed's cache, and whether its launch gets the automatic publisher follows the installed engine: an engine whose KV events name their cache group (its KVEvent.BlockStored carries group_idx) publishes the model's history chain beside its checkpoints and is served like any other model. On an engine whose events carry no group, a boundary is published only while every group holds it, so the model's history chain would never reach the router: the servicer leaves the publisher off for such a model and logs why once (KV event publishing left off: ...). An explicit --kv-events-config turns the publisher on anyway.

Pass --kv-events-config to pin the endpoint and enable gap replay:

# TokenSpeed is installed from source (engine + kernel + scheduler); see
# scripts/ci_install_tokenspeed.sh in the smg repository. Install the servicer's extra deps:
pip install "smg-grpc-servicer[tokenspeed]"

# --kv-events-config pins the event endpoint and adds gap replay:
python -m smg_grpc_servicer.tokenspeed \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --host 0.0.0.0 \
  --port 50051 \
  --kv-events-config '{"enable_kv_cache_events": true, "publisher": "zmq", "endpoint": "tcp://*:5557", "replay_endpoint": "tcp://*:5558", "topic": "kv-events"}'
Field Why
enable_kv_cache_events: true TokenSpeed master switch. An explicit config is kept as given, so one without it records no events even if a publisher is set.
publisher: "zmq" Selects the ZMQ publisher the servicer reads. Unset defaults to "zmq" when events are enabled; "null" (or any other value) disables event streaming.
endpoint / topic ZMQ PUB address and topic. Use a bind-style endpoint (tcp://*:PORT): TokenSpeed binds only when the endpoint contains * or :: or starts with ipc:// or inproc://, so a concrete address like tcp://127.0.0.1:PORT makes it connect instead, leaving nothing bound and the stream idle. With attention data parallelism each rank publishes on endpoint port + rank, and the servicer subscribes to every rank, so the index sees every rank's cache.
replay_endpoint As for SGLang and vLLM: the servicer fills gaps in each rank's events from it before forwarding later batches; without it, a missed batch costs the worker's index entries (see Recovery after gaps). Each rank serves replay on replay_endpoint port + rank.

--kv-events-config is parsed by TokenSpeed's KVEventsConfig.from_cli. Gap replay needs an explicit replay_endpoint in the config — no default supplies one.

SMG learns the block size from the BlockStored events; TokenSpeed's cache reuse granularity is --prefix-granularity (default 64 tokens; older builds call it --block-size, which remains an alias), so set SMG's --block-size to the same value.


Step 2 — Launch SMG

Point SMG at the gRPC workers and select cache_aware:

smg launch \
  --worker-urls grpc://worker-1:50051 grpc://worker-2:50051 \
  --model-path meta-llama/Llama-3.1-8B-Instruct \
  --policy cache_aware \
  --block-size 16 \
  --host 0.0.0.0 \
  --port 30000

worker-2 is a second worker launched the same way, on another host. On a single host, give each worker its own --port and its own event ports.

The flags that matter for event-driven routing:

Flag Why
grpc://... worker URLs SMG subscribes to KV events only on gRPC workers. HTTP workers are skipped, and ZMQ (ipc:// or tcp://) workers log a warning and use the approximate token tree.
--policy cache_aware The only policy that reads the event index, and the default policy. A gRPC worker streams KV events when the policy that places it is cache_aware.
--block-size 16 Page size of the approximate token tree, which routes until the event index has data, and the block size the event index assumes until it learns the engine's. Match the engine's page size.

The placing policy is not always the gateway's --policy. A regular worker is placed by its model's policy: --policy, or the policy label of the model's first worker. Prefill and decode workers are placed by --prefill-policy and --decode-policy, each of which defaults to the --policy value. Encode workers are placed by --encode-policy, which does not accept cache_aware, so they never stream KV events. With --policy round_robin --prefill-policy cache_aware, the prefill workers stream their KV events even though no other worker does; with --policy cache_aware --prefill-policy round_robin, they do not, even while the rest of the fleet does.

--model-path tells the gateway which tokenizer to use, as in any gRPC-worker deployment (gRPC Workers). For the other routing knobs, see Tuning.

Block size alignment

The event index cuts each request's token IDs into blocks and looks the blocks up by content hash. If the block size differs from the one the engine used, no block matches, and the model's token requests are routed by load alone.

SMG picks the block size per model, in this order:

  1. Learned from events (highest priority): the first stored block a worker's subscription receives replaces the provisional size, and a larger block raises it; a smaller block never lowers it (hybrid engines publish, beside their full blocks, partial entries at their hash granularity, and a request chunked at that size would never match a full block).
  2. --block-size (router-wide default).

The kv_block_size field of a worker spec registered through the admin API (POST /workers) seeds the model's block size until the events name it. In practice, keep the engine's page size (SGLang --page-size, vLLM --block-size, TokenSpeed --prefix-granularity) and SMG's --block-size equal, and let SMG correct itself once events arrive. The approximate token tree always uses --block-size. Serve each model with one page size: with mixed sizes, requests are chunked at the largest block size any worker's events have stored, so the blocks of workers with smaller pages are never matched.


Step 3 — Send a request

The API surface is unchanged:

curl http://localhost:30000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "meta-llama/Llama-3.1-8B-Instruct",
    "messages": [
      {"role": "user", "content": "Hello, who are you?"}
    ]
  }'

Send the same prompt twice. The first request has no cached prefix and goes to the lowest-expected-wait worker. Once that worker reports the blocks it stored, the second request is scored against them and lands on the same worker, as long as that worker is not far busier than the rest of the fleet.


Verifying event delivery

The gateway logs these lines when the path is live (timestamps, targets and source locations omitted).

1. Subscription started. When SMG registers a gRPC worker placed by cache_aware, KvEventMonitor logs:

INFO Starting KV event subscription worker_url=grpc://worker-1:50051 model_id=meta-llama/Llama-3.1-8B-Instruct kind=Events
INFO KV event stream connected worker_url=grpc://worker-1:50051 start_seq=0

If you do not see the first line with kind=Events for a worker, the worker is not a gRPC worker, or the policy that places it is not cache_aware. The placing policy is the model's (--policy, or the first worker's policy label) for a regular worker, and the role's --prefill-policy, --decode-policy or --encode-policy — not the model's — for a prefill, decode or encode worker. The same line with kind=Loads announces a stream of the worker's load records alone, not a KV-event subscription.

If events are disabled on the worker — an explicit --kv-events-config without a ZMQ publisher, the worker's SMG_<ENGINE>_SERVICER_KV_EVENTS=0, or a TokenSpeed hybrid model on an engine whose KV events carry no cache group — the servicer rejects the stream and SMG stops trying:

WARN Backend does not implement SubscribeKvEvents, disabling KV event subscription for this worker; cache-aware routing sees nothing of its cache worker_url=grpc://worker-1:50051

2. Backend block size learned. Once the first stored block arrives, SMG records the backend's block size (a larger block arriving later raises it; a smaller one does not lower it):

INFO Learned block_size from KV event model_id=meta-llama/Llama-3.1-8B-Instruct block_size=16

3. Routing decision uses the index. With RUST_LOG=info,smg::policies::cache_aware=debug, each routed request logs Event-driven routing with the chosen worker, its overlap_blocks out of the request's request_blocks, and a branch: event_hit (the chosen worker holds part of the prefix), event_spill (other workers hold the prefix but the request left them) or event_miss (no worker holds the first block). A warm-up placement logs Cache miss routed to a warming worker (warmup_slice) or Cache hit diverted to a thin worker (warmup_divert) instead.

While the model's event index is empty, token requests route on the approximate token tree, so cache affinity keeps working, only less accurately.


Recovery after gaps

Every batch on the stream carries a sequence number. KvEventMonitor applies batches in order and skips duplicates. When it sees a gap in the sequence, it reconnects immediately and asks the servicer to resume after the last batch it applied; after a stream error or a clean close, it reconnects with exponential backoff from 100 ms up to 5 s. A subscription call that is not answered within 2 s — a port that accepts but does not serve yet, such as an engine still starting — is abandoned and retried.

With the Python servicer — SGLang, vLLM and TokenSpeed alike — a gap in a rank's events is filled from that rank's replay_endpoint before later batches are forwarded, so SMG receives an unbroken stream and the index stays intact. The servicer waits up to five seconds for each replay message before declaring a gap unfillable. A gap it cannot fill — no replay_endpoint, the missed batches already left the replay buffer, the replay failed, or the engine's publisher restarted — ends the stream with DATA_LOSS: SMG drops that worker's index entries and starts a new subscription.

The Python servicer keeps nothing between subscriptions: a resume cursor is answered with OUT_OF_RANGE, and a new subscription carries live events only. After any stream break between SMG and the servicer, SMG therefore drops the worker's index entries and rebuilds them from live events. The rebuilt index is not a snapshot of the engine's cache: affinity to that worker returns as it caches new prefixes.

Workers running a Rust servicer — the default for vLLM, SGLang and TokenSpeed gRPC workers — keep more. Its relay follows the engine's publishers for the servicer's whole lifetime: a reconnecting gateway is served the batches after its cursor from history, and a new subscription is answered with either the engine's complete event history or an in-band snapshot of the engine's live blocks, which SMG applies in place of whatever it held for that worker. With several data-parallel ranks the relay refuses a resume cursor (the request carries a single cursor, and each rank keeps its own sequence numbers), so the gateway clears and resubscribes from zero and is answered with the history or snapshot instead, losing nothing. Cache affinity then survives gateway restarts and long disconnects without waiting for new stores.

SMG logs Sequence gap detected, reconnecting for replay from seq N on a gap, and KV event replay cursor expired; clearing worker state and requesting a current snapshot or KV event subscriber fell behind; clearing worker state and requesting a current snapshot when it drops a worker's entries. A restart of the engine's own publisher (its sequence numbers start over) also clears the worker's entries — the engine's cache is empty again — and the stream continues from the new sequence.

A clear — an AllBlocksCleared event, a publisher restart or a resync — also resets what the gateway knows about that worker's page size, so an engine restarted with a different page size has its fresh stores indexed at their new width as they come. When the first store after a clear names a narrower block than the worker stored before, the gateway logs a warning (the first KV event store after a clear names a narrower block than the worker's previous record: ...).

The model's learned block size is separate: it only rises and is dropped only when the model's last worker is removed. Requests therefore keep being chunked at the old, larger size, and the restarted worker's narrower blocks, though indexed, are not matched. A model redeployed at a smaller page size matches at the new size only once all its workers have left the gateway and the size is learned again from the new workers' events.


Index size

The event index holds what the engines report: block-removed and all-blocks-cleared events shrink it as each engine evicts, and when a worker is removed, its entries go with it. There is no prune and no size flag to set.

The former --kv-indexer-ttl-secs and --kv-indexer-max-entries flags bounded the positional indexer, which the chain index replaced, and were removed with it: the smg binary and the Python launcher both refuse a launch line that still carries one as an unknown argument. Drop them from your launch lines.


Tuning

Knob Where Effect
--block-size SMG Keep equal to the engine's page size.
--balance-abs-threshold / --balance-rel-threshold SMG Spill gate: a holder whose in-flight request count exceeds both margins over the fleet mean sends the request to another worker.
--overlap-decay SMG De-ranks candidates with a waiting-prefill backlog. Needs workers that report their waiting token-work: the SGLang, vLLM, TokenSpeed and TensorRT-LLM gRPC servicers report an estimate from the requests they forward (the deprecated Python SGLang servicer reports the scheduler's own figure, the deprecated Python TokenSpeed servicer none).
--selection-temperature SMG Spreads picks across candidates with near-equal overlap instead of always taking the best.
--balance-token-usage-threshold / --overload-token-usage-threshold SMG Suspend cache affinity under KV pressure (off by default).
replay_endpoint / buffer_steps --kv-events-config Enable gap replay; raise buffer_steps (SGLang and vLLM only) if gaps go unfilled because the missed batches already left the buffer.
hwm --kv-events-config Raise if events are dropped under bursty load.
SMG_KV_EVENT_HISTORY_BATCHES / SMG_KV_EVENT_HISTORY_BYTES Rust servicer environment How much relayed history a Rust servicer keeps per rank for a reconnecting gateway to resume from; see Engine Servicer Variables.

--cache-threshold does not drop candidates in event-driven routing — any worker holding at least the request's first full block is one (on a hybrid model, after the cache-group bound described in How the pieces fit together) — but an overlap at or below it counts as a miss for the worker warm-up slice. See Cache-Aware Routing for the full list of knobs.


Monitoring

Metric Type Labels Description
smg_kv_event_subscription_failures_total counter worker, reason A subscription task failed: panic, join_error or intern_failed. After panic or intern_failed, that worker's events no longer reach the index.
smg_engine_cache_hit_rate gauge worker, model, dp_rank Engine-reported prefix cache hit rate, to check that routing on events pays off.

smg_cache_aware_policy_branch_total counts event-driven decisions too — event_hit, event_spill and event_miss, with warm-up placements as warmup_slice and warmup_divert; tree-mode decisions keep their own branches — and smg_cache_aware_match_ratio observes every decision: in event mode, the chosen worker's overlap in blocks over the request's blocks, 0 when it holds none.

The stream metrics of the subscriptions (smg_kv_event_*, smg_kv_events_unavailable, smg_kv_block_size_tokens) are listed under KV Event Metrics in the metrics reference.


Caveats

  • gRPC only. Events stream over gRPC through smg-grpc-servicer. HTTP workers use the approximate string tree, ZMQ (ipc:// or tcp://) workers use the approximate token tree, and MLX workers do not implement SubscribeKvEvents.
  • One block size per model. If workers serving the same model use different page sizes, requests are chunked at the largest block size any worker's events have stored — the learned size only rises while the model has workers — so the blocks of workers with smaller pages are never matched. Keep page sizes homogeneous within a model.
  • Per-gateway index. Each gateway subscribes to every worker itself. With --enable-mesh, gateways synchronize the approximate trees, not the event index.
  • extra_key does not partition the event index. The LoRA adapter and cache_salt carry over: the gateway hashes stored blocks and request blocks under a seed derived from them, so a salted or LoRA request matches only its own partition's blocks. extra_key, which partitions the approximate trees, is not part of that seed (see Cache Partitions).
  • Unaligned stores are skipped. The servicers drop any main-attention BlockStored event whose token count is not its block count times its block size; those blocks never reach the index. A hybrid model's sliding-window and state group stores span the step's whole token range by design and are exempt: the Rust servicer forwards them without tokens so the gateway can track each group's retention (see How the pieces fit together); they never enter the prefix index.
  • Replay needs replay_endpoint. The servicers fill gaps only when replay_endpoint is set in --kv-events-config; without it, a gap costs the worker's index entries (see Recovery after gaps).

Reference

  • Policy implementation: model_gateway/src/policies/cache_aware.rs
  • Event subscription manager: model_gateway/src/worker/kv_event_monitor.rs (gap settling and resyncs: model_gateway/src/worker/kv_event_recovery.rs)
  • KV event proto: crates/grpc_client/proto/common.proto (messages SubscribeKvEventsRequest, KvEventBatch, KvSnapshotChunk, KvCacheEvent, KvBlocksStored, KvBlocksRemoved, KvCacheCleared)
  • Engine-neutral relay (per-rank subscriptions, gap replay, normalization): grpc_servicer/smg_grpc_servicer/kv_relay.py; the Rust servicer's relay with history and state snapshots: crates/engine_servicer/src/kv_events.rs
  • SGLang servicer: grpc_servicer/smg_grpc_servicer/sglang/servicer.py (SubscribeKvEvents) + per-rank sources grpc_servicer/smg_grpc_servicer/sglang/kv_events.py
  • vLLM servicer: grpc_servicer/smg_grpc_servicer/vllm/servicer.py (SubscribeKvEvents) + config resolver grpc_servicer/smg_grpc_servicer/vllm/kv_events.py
  • TokenSpeed servicer: grpc_servicer/smg_grpc_servicer/tokenspeed/servicer.py (SubscribeKvEvents) + per-rank sources and config resolver grpc_servicer/smg_grpc_servicer/tokenspeed/kv_events.py
  • SGLang upstream config: python/sglang/srt/disaggregation/kv_events.py (class KVEventsConfig)
  • vLLM upstream config: vllm/config/kv_events.py (class KVEventsConfig)
  • TokenSpeed upstream config: python/tokenspeed/runtime/pd/kv_events.py (class KVEventsConfig)