Cache-Aware Routing¶

Cache-aware routing is SMG's most sophisticated load balancing policy. It maintains a multi-tenant radix tree that mirrors backend KV cache state, enabling perfect cache prediction and optimal worker selection.

Overview¶

Why Cache-Aware Routing?¶

LLM inference workers maintain a KV cache of previously computed attention states. When a new request shares a prefix with a previous request, the worker can skip recomputing that portion—dramatically reducing latency.

The Challenge: If requests with shared prefixes go to different workers, each worker must recompute the same prefix independently—wasting GPU cycles.

Cache-Aware Solution: Route requests to workers that already have the relevant prefix cached, maximizing KV cache hits and minimizing redundant computation.

Multi-Tenant Radix Tree Architecture¶

SMG maintains a multi-tenant radix tree that mirrors the KV cache state on each backend worker. This enables true cache-aware routing with 100% prefix match accuracy.

Why This Matters:

• Perfect Cache Prediction: Since the gateway tree mirrors backend behavior exactly, prefix match calculations are 100% accurate
• Kernel-Aware: Honors page boundaries used by inference kernels (FlashInfer, Mamba, etc.)
• Eviction Parity: Tracks the same eviction policy as SGLang/vLLM/TensorRT-LLM schedulers

Routing Algorithm¶

Two-State Decision Model¶

Cache-aware routing operates in two states based on system load:

Imbalance Detection¶

The system is considered imbalanced when both conditions are met:

(max_load - min_load) > balance_abs_threshold
        AND
max_load > balance_rel_threshold × min_load

Selection Flow¶

Performance Impact¶

Benchmarks¶

In typical conversational workloads:

When Cache-Aware Helps Most¶

Tuning Guidelines¶

Cache Threshold¶

The cache-threshold parameter controls how aggressively routing favors cache affinity:

Balance Thresholds¶

Memory Management¶

The radix tree grows with unique prefixes. Configure based on your memory budget:

Eviction: SMG periodically evicts stale entries using LRU policy. Configure with --eviction-interval (default: 120 seconds).

Monitoring¶

Key Metrics¶

Useful PromQL Queries¶

stddev(smg_worker_requests_active) /
avg(smg_worker_requests_active)

rate(smg_router_ttft_seconds_sum[5m]) /
rate(smg_router_ttft_seconds_count[5m])

Alert Thresholds¶

Multi-Gateway Considerations¶

When running multiple SMG instances behind a load balancer:

• Each instance maintains its own radix tree
• Cache efficiency drops by 10-20% compared to single instance
• Consider session affinity at the load balancer to improve cache hits

Load Balancer Configuration¶

upstream smg {
    hash $http_x_user_id consistent;  # Affinity by user
    server smg-1:30000;
    server smg-2:30000;
}

backend smg
    balance hdr(X-User-ID)
    hash-type consistent
    server smg-1 smg-1:30000 check
    server smg-2 smg-2:30000 check

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/upstream-hash-by: "$http_x_user_id"

Event-Driven Block Size¶

By default, SMG uses the --block-size CLI flag (default: 16) to align the radix tree with backend KV cache page boundaries. However, when workers report KV cache events, the cache-aware policy can learn the actual block size directly from those events.

This is especially useful when different backends use different page sizes—rather than requiring a single global flag, each worker's block size is discovered automatically from its event stream. The learned block size takes precedence over the CLI default, ensuring routing accuracy without manual configuration per backend.

Mesh State Synchronization¶

When running in mesh HA mode (--enable-mesh), cache-aware routing state is synchronized across all router nodes. Each node's radix tree receives updates from the mesh, ensuring that every gateway instance has a consistent view of backend KV cache state. This eliminates the cache efficiency penalty described in Multi-Gateway Considerations and enables accurate cache-aware decisions across the entire cluster.

For mesh setup and configuration details, see High Availability.

Comparison with Other Policies¶

What's Next?¶