Tokenizer Caching¶

SMG provides a two-level tokenizer cache that reduces tokenization overhead for repeated content. It applies only where the gateway tokenizes: requests to gRPC and ZMQ workers, and /v1/tokenize.

How It Works¶

On a multi-turn conversation, L1 avoids re-tokenizing the entire history — only new messages are tokenized.

Enable Caching¶

Both cache levels are disabled by default. Enable them with CLI flags:

L0 Only (Exact Match)¶

Best for workloads that send the same whole prompt many times (batch processing, resent requests):

smg \
  --worker-urls grpc://worker:50051 \
  --model-path meta-llama/Llama-3.1-8B-Instruct \
  --tokenizer-cache-enable-l0 \
  --tokenizer-cache-l0-max-entries 10000

L0 + L1 (Exact + Prefix Match)¶

Best for multi-turn chat applications:

smg \
  --worker-urls grpc://worker:50051 \
  --model-path meta-llama/Llama-3.1-8B-Instruct \
  --tokenizer-cache-enable-l0 \
  --tokenizer-cache-l0-max-entries 20000 \
  --tokenizer-cache-enable-l1 \
  --tokenizer-cache-l1-max-memory 104857600

Configuration Reference¶

L0 Cache¶

Each entry keeps the full input text and its encoding, so its size grows with prompt length.

L1 Cache¶

Memory Planning¶

L0 Sizing¶

L0 caps the number of entries, not their bytes. An entry for a short prompt is small, but roughly 2 MB per entry was observed for large inputs (smg-project/smg#2603). Size --tokenizer-cache-l0-max-entries from the number of distinct whole prompts that repeat in your traffic, and watch process memory as you raise it.

L1 Sizing¶

On a miss, L1 stores an entry for every special-token boundary of the input, holding the token IDs of the whole prefix up to that boundary. On a hit it stores one more, for the prefix up to the input's own deepest boundary, so each turn of a conversation leaves the entry its next turn matches (on main after the v1.11.0 release, smg-project/smg#2869 — earlier versions store nothing on a hit). Each entry is charged the prefix's length in bytes plus 4 bytes per token, so a long multi-turn prompt with many boundaries counts for many times its own length, and hits as well as misses grow the cache toward the memory bound.

Recommended Configurations¶

smg \
  --worker-urls grpc://worker:50051 \
  --model-path meta-llama/Llama-3.1-8B-Instruct \
  --tokenizer-cache-enable-l0 \
  --tokenizer-cache-l0-max-entries 50000

smg \
  --worker-urls grpc://worker:50051 \
  --model-path Qwen/Qwen2.5-7B-Instruct \
  --tokenizer-cache-enable-l0 \
  --tokenizer-cache-l0-max-entries 20000 \
  --tokenizer-cache-enable-l1 \
  --tokenizer-cache-l1-max-memory 104857600

smg \
  --worker-urls grpc://worker:50051 \
  --model-path meta-llama/Llama-3.1-8B-Instruct \
  --tokenizer-cache-enable-l0 \
  --tokenizer-cache-l0-max-entries 5000

Monitor Cache Activity¶

SMG exports cache counters on its Prometheus endpoint (smg-project/smg#2603): smg_tokenizer_cache_lookups_total (labels layer and result), smg_tokenizer_cache_evictions_total, and smg_tokenizer_cache_reused_bytes_total (label layer). layer is l0 or l1; result is hit or miss. Check each layer's hit ratio:

sum by (layer) (rate(smg_tokenizer_cache_lookups_total{result="hit"}[5m]))
/
sum by (layer) (rate(smg_tokenizer_cache_lookups_total[5m]))

A low hit ratio together with a steady eviction rate means the cache is too small for the workload. See the Metrics Reference for the full definitions.

Next Steps¶

• Tokenizer Caching Concepts — Cache architecture, special token boundaries, monitoring metrics, PromQL queries
• gRPC Workers — Enable gateway-level tokenization with gRPC mode
• Load Balancing — Choose a routing policy (cache-aware routing uses tokenizer results)