Cache-Aware Routing¶

Cache-aware routing is SMG's most sophisticated load balancing policy. It maintains a multi-tenant radix tree that mirrors backend KV cache state, enabling perfect cache prediction and optimal worker selection.

Overview¶

Why Cache-Aware Routing?¶

LLM inference workers maintain a KV cache of previously computed attention states. When a new request shares a prefix with a previous request, the worker can skip recomputing that portion—dramatically reducing latency.

The Challenge: If requests with shared prefixes go to different workers, each worker must recompute the same prefix independently—wasting GPU cycles.

Cache-Aware Solution: Route requests to workers that already have the relevant prefix cached, maximizing KV cache hits and minimizing redundant computation.

Multi-Tenant Radix Tree Architecture¶

SMG maintains a multi-tenant radix tree that mirrors the KV cache state on each backend worker. This enables true cache-aware routing with 100% prefix match accuracy.

Why This Matters:

• Perfect Cache Prediction: Since the gateway tree mirrors backend behavior exactly, prefix match calculations are 100% accurate
• Kernel-Aware: Honors page boundaries used by inference kernels (FlashInfer, Mamba, etc.)
• Eviction Parity: Tracks the same eviction policy as SGLang/vLLM/TensorRT-LLM schedulers

Routing Algorithm¶

Two-State Decision Model¶

Cache-aware routing operates in two states based on system load:

Imbalance Detection¶

The system is considered imbalanced when both conditions are met:

(max_load - min_load) > balance_abs_threshold
        AND
max_load > balance_rel_threshold × min_load

Selection Flow¶

Performance Impact¶

Benchmarks¶

In typical conversational workloads:

When Cache-Aware Helps Most¶

Tuning Guidelines¶

Cache Threshold¶

The cache-threshold parameter controls how aggressively routing favors cache affinity:

Balance Thresholds¶

Memory Management¶

The radix tree grows with unique prefixes. Configure based on your memory budget:

Eviction: SMG periodically evicts stale entries using LRU policy. Configure with --eviction-interval (default: 120 seconds).

Monitoring¶

Key Metrics¶

Useful PromQL Queries¶

rate(smg_router_cache_hits_total[5m]) /
(rate(smg_router_cache_hits_total[5m]) +
 rate(smg_router_cache_misses_total[5m]))

stddev(smg_worker_requests_active) /
avg(smg_worker_requests_active)

Alert Thresholds¶

Multi-Gateway Considerations¶

When running multiple SMG instances behind a load balancer:

• Each instance maintains its own radix tree
• Cache efficiency drops by 10-20% compared to single instance
• Consider session affinity at the load balancer to improve cache hits

Load Balancer Configuration¶

upstream smg {
    hash $http_x_user_id consistent;  # Affinity by user
    server smg-1:30000;
    server smg-2:30000;
}

backend smg
    balance hdr(X-User-ID)
    hash-type consistent
    server smg-1 smg-1:30000 check
    server smg-2 smg-2:30000 check

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/upstream-hash-by: "$http_x_user_id"

Comparison with Other Policies¶

What's Next?¶