Tokenizer Caching¶

SMG implements a two-level tokenizer cache that reduces tokenization overhead for repeated content. It applies where the gateway tokenizes prompts itself: requests to gRPC and ZMQ workers, and /v1/tokenize. HTTP workers tokenize on the engine, so the cache doesn't affect their requests.

Overview¶

Why Cache Tokenization?¶

The gateway tokenizes—converts text to token IDs—every request it sends to a gRPC or ZMQ worker. Each tokenization is fast, but the cost grows with prompt length and adds up at scale.

Cache Architecture¶

Each tokenizer SMG loads (from --model-path or --tokenizer-path, or for a gRPC or ZMQ worker's model) gets its own L0 and L1 caches with the configured limits. Tokenizers added through POST /v1/tokenizers are not cached. Decoding is never cached.

Special Token Boundaries (L1)¶

L1 splits inputs right after every special token the tokenizer declares: its BOS, EOS, UNK, SEP, PAD, CLS, and MASK tokens, and every added token marked special. An input without special tokens always misses L1. For example:

Multi-Turn Conversation Example¶

Consider how caching helps a typical chat application:

System: You are a helpful assistant.
User: What is Python?

System: You are a helpful assistant.
User: What is Python?
Assistant: Python is a programming language...
User: How do I install it?

Result: Turn 2 tokenizes only the new messages, not the shared history. The hit also stores one new entry covering Turn 2's prompt up to its own deepest boundary, so Turn 3 matches right after Turn 2's last special token — even when Turn 2's match came from a prefix shared with other conversations, such as a common system prompt.

Configuration¶

Model & Tokenizer Paths¶

--model-path¶

HuggingFace model ID or local path to load the tokenizer from.

Usage:

# HuggingFace model ID (downloads automatically)
smg --model-path meta-llama/Llama-3.1-8B-Instruct ...

# Local path to model directory
smg --model-path /models/llama-3.1-8b-instruct ...

# Local path to tokenizer.json file
smg --model-path /models/llama-3.1-8b-instruct/tokenizer.json ...

When pointing to a local directory, SMG looks for a HuggingFace tokenizer.json, a vocab.json plus merges.txt pair, or a tiktoken file (tiktoken.model or *.tiktoken). When pulling from the HuggingFace Hub, SMG looks for the same files in the downloaded snapshot: tokenizer.json, then a tiktoken file, then the vocab.json plus merges.txt pair. A tokenizer_config.json alone is not a tokenizer.

--tokenizer-path¶

Explicit path to a tokenizer file. Overrides --model-path for tokenizer loading.

When to use:

• When the tokenizer is stored separately from the model
• When using a custom tokenizer with a standard model
• When the model directory structure is non-standard

# Load the tokenizer from its own file (--tokenizer-path wins over --model-path)
smg \
  --model-path meta-llama/Llama-3.1-8B-Instruct \
  --tokenizer-path /custom/tokenizers/llama3-tokenizer.json \
  ...

Chat Templates¶

Chat templates convert structured messages (system, user, assistant roles) into the prompt format expected by specific models. SMG uses Jinja2 templates, the same format used by HuggingFace Transformers.

--chat-template¶

Path to a Jinja2 chat template file.

Template discovery priority:

1. Explicit --chat-template path (highest priority)
2. chat_template.jinja in the model directory
3. A chat_template string in tokenizer_config.json
4. Legacy chat_template.json in the model directory
5. Any other .jinja file in the model directory

A chat_template entry in tokenizer_config.json that is not a single string (such as a named-template list) is not read, and discovery falls through to the file fallbacks.

Template Variables¶

Chat templates use Jinja2 syntax with access to:

Keys in a request's chat_template_kwargs are passed to the template as extra variables.

strftime_now matches the function HuggingFace Transformers provides to chat templates: {{ strftime_now('%Y-%m-%d') }} writes today's date in the server's local time, like Python's datetime.now().strftime(format). Every call in one render writes the same instant. An invalid strftime format fails the render with an error naming strftime_now.

To make a prompt that writes the date reproducible, set SOURCE_DATE_EPOCH in SMG's environment to the desired instant as seconds since the Unix epoch (the reproducible-builds convention). SMG reads it once, at the first render, and every render then writes that instant instead of the clock. A value that is not a whole number of seconds is ignored, and the clock is used.

Template Examples¶

{%- for message in messages %}
<|im_start|>{{ message.role }}
{{ message.content }}<|im_end|>
{% endfor %}
{%- if add_generation_prompt %}
<|im_start|>assistant
{% endif %}

<|begin_of_text|>{% for message in messages %}
<|start_header_id|>{{ message.role }}<|end_header_id|>

{{ message.content }}<|eot_id|>
{% endfor %}
{% if add_generation_prompt %}<|start_header_id|>assistant<|end_header_id|>

{% endif %}

L0 Cache Configuration¶

The L0 cache stores complete tokenization results for exact string matches.

L1 Cache Configuration¶

The L1 cache stores tokenization results at special token boundaries.

Memory Planning¶

L0 Cache Sizing¶

L0 is capped by an entry count and a byte budget. Each entry keeps the full input text and its encoding, so its size grows with the input: small for a short prompt, megabytes for a large one. SMG charges each entry its text bytes, token IDs, and per-entry overhead against --tokenizer-cache-l0-max-memory (default 256 MB) and evicts when a new entry would exceed either limit; an input whose entry alone would exceed a quarter of the byte budget is not cached at all. Size --tokenizer-cache-l0-max-entries from the number of distinct whole prompts that actually repeat in your traffic, and the byte budget from how large those prompts are.

L1 Cache Sizing¶

L1 cache is bounded by total memory:

On a miss, L1 stores one entry per special-token boundary of the input: the token IDs of the whole prefix up to that boundary. On a hit, it stores one more entry — the prefix up to the input's own deepest boundary — when that boundary lies past the match, so steady hit traffic also grows the cache. SMG charges each entry the prefix's length in bytes plus 4 bytes per token, so a long multi-turn prompt with many boundaries counts for many times its own length. When new entries would exceed the budget, SMG evicts approximately least recently used entries (sampling 32 at a time) to make room. The budget is an estimate of cache contents, not a cap on process memory.

Recommended Configurations¶

smg \
  --model-path meta-llama/Llama-3.1-8B-Instruct \
  --tokenizer-cache-enable-l0 \
  --tokenizer-cache-l0-max-entries 50000

smg \
  --model-path Qwen/Qwen2.5-7B-Instruct \
  --tokenizer-cache-enable-l0 \
  --tokenizer-cache-l0-max-entries 20000 \
  --tokenizer-cache-enable-l1 \
  --tokenizer-cache-l1-max-memory 104857600

smg \
  --model-path meta-llama/Llama-3.1-8B-Instruct \
  --tokenizer-cache-enable-l0 \
  --tokenizer-cache-l0-max-entries 5000 \
  --tokenizer-cache-l0-max-memory 67108864

smg \
  --model-path meta-llama/Llama-3.1-8B-Instruct
# Caching is disabled by default

Complete Example¶

Production configuration with tokenizer and caching:

smg \
  --worker-urls grpc://worker1:50051 grpc://worker2:50051 \
  --policy cache_aware \
  --model-path meta-llama/Llama-3.1-70B-Instruct \
  --chat-template /templates/llama3.jinja \
  --tokenizer-cache-enable-l0 \
  --tokenizer-cache-l0-max-entries 25000 \
  --tokenizer-cache-enable-l1 \
  --tokenizer-cache-l1-max-memory 104857600 \
  --host 0.0.0.0 \
  --port 8080

Monitoring & Observability¶

The gateway exports cache activity on its Prometheus /metrics endpoint:

The counters are totals across all tokenizers in the process, with no model label, and a disabled layer reports zeros. An L0 hit never reaches L1, and L1 counts inputs without special-token boundaries as misses, so compute each layer's hit ratio against its own lookups:

sum by (layer) (rate(smg_tokenizer_cache_lookups_total{result="hit"}[5m]))
/
sum by (layer) (rate(smg_tokenizer_cache_lookups_total[5m]))

See the Metrics Reference for details.

Sizing Signals to Watch¶

Use these signals when tuning --tokenizer-cache-l0-max-entries and --tokenizer-cache-l1-max-memory:

• A low L0 hit ratio with a steady L0 eviction rate (rate(smg_tokenizer_cache_evictions_total{layer="l0"}[5m])) means more distinct inputs arrive than L0 holds; raise max-entries — or the L0 byte budget, when the inputs are large — if the workload repeats inputs at all.
• L1 evictions mean the prefix cache is at its memory bound. Multi-turn chat traffic with growing context benefits from a larger L1 budget; see L1 Cache Sizing.
• smg_tokenizer_cache_reused_bytes_total separates many small prefix hits from reuse of large prompts: a high hit ratio with little reused volume saves little tokenization work.
• Each L0 entry keeps the full input text and its encoding, so entry size grows with prompt length: large inputs can take megabytes per entry. Watch smg_allocator_allocated_bytes when raising max-entries or --tokenizer-cache-l0-max-memory.

Integration with Other Caching Layers¶

Tokenizer caching is part of SMG's three-level caching strategy:

What's Next?¶