Tenant Rate Limiting Reference

Precise contract for per-tenant token/request rate limiting: every configuration knob, the YAML policy schema, and the exact response shape. For how it works conceptually — the reserve/settle model, retry handling, n>1 accounting — see Tenant Rate Limiting.

Tenant rate limiting is disabled by default. When enabled, it is enforced on every serving router, each reserving once per logical request before any dispatch:

  • gRPC router: the Chat, Generate, Completion, Messages, Decisions, Embeddings, Classify, Audio Transcriptions, and Responses endpoints (Harmony-mode Chat and Responses included). The gRPC router serves both gRPC- and ZMQ-connected workers (--connection-mode zmq, ipc:// or tcp:// worker URLs); for every covered endpoint except Decisions, enforcement does not depend on the worker transport. A Messages request with max_tokens: 0 is answered by the gateway itself — an empty message reporting the prompt's token count — and takes no reservation.
  • Regular HTTP router: /generate, /v1/chat/completions, /v1/completions, /v1/messages, /v1/responses, /v1/embeddings, and /v1/classify. Its /v1/messages/count_tokens, /v1/decisions, /v1/systemone, and /v1/rerank routes, the audio transcription route, and the realtime routes take no reservation.
  • Disaggregated (PD) HTTP router: /generate, /v1/chat/completions, /v1/completions, /v1/messages, and /v1/responses; its /v1/messages/count_tokens and /v1/rerank routes take no reservation.
  • External-provider routers: /v1/chat/completions, /v1/responses, /v1/messages, and /v1/interactions, reserved on the gateway side before the provider router dispatches — a denied request never reaches the provider. Their realtime routes pass through. The reservation is keyed by the canonical model name, so an alias is charged under its model's budget and per-model rule.

The denial is identical on every router: the 429 below, never retried.

On the gRPC router, a Responses request reserves per iteration: every turn of an MCP tool loop takes a reservation of its own — the estimated input tokens plus a flat 1 from the request counter — and settles against that turn's reported usage, so an N-turn loop debits N requests from requests_per_minute. A request without MCP tools runs a single iteration and is charged like a chat request. On the HTTP routers, a /v1/responses request takes a single reservation.

Decisions is the one gRPC endpoint where the worker transport matters: the gRPC router serves it only on gRPC-connected SGLang workers, and a Decisions request for a model with no such worker is rejected with 501 (X-SMG-Error-Code: decisions_not_supported) before any reservation is made.


Enabling it

smg \
  --worker-urls grpc://w1:9000 grpc://w2:9000 \
  --tenant-rate-limit-enabled \
  --tenant-rate-limit-config /etc/smg/tenant-rate-limit.yaml

CLI flags

Flag Default Description
--tenant-rate-limit-enabled false Master switch. When unset, no rate limiter is constructed and every request skips reservation entirely.
--tenant-rate-limit-config unset Path to the tenant-rate-limit YAML. Required when --tenant-rate-limit-enabled is set.

Both the Rust smg binary and the Python launcher (smg launch from pip, and the container image) accept these flags.


YAML configuration

default_policy:
  tokens_per_minute: 100000
  requests_per_minute: 600

tenants:
  - tenant_key: "auth:team-red"
    tokens_per_minute: 500000
    requests_per_minute: 3000
    model_rules:
      - rule_id: gpt4-cap
        matcher:
          type: exact
          value: gpt-4
        tokens_per_minute: 50000
        requests_per_minute: 300
      - rule_id: legacy-models
        matcher:
          type: prefix
          value: "legacy-"
        tokens_per_minute: 10000
        requests_per_minute: 100

  - tenant_key: "anonymous"
    tokens_per_minute: 5000
    requests_per_minute: 60

A tenant not listed under tenants uses default_policy, with its own buckets: tenants never share a bucket. Tenant keys are the same canonical keys SMG resolves elsewhere in the request path — auth:<id>, header:<id>, ip:<address>, or anonymous — there's no separate tenant-identity system for this feature. A --tenant-api-key team-red:<key> caller resolves to auth:team-red; a caller using the shared --api-key resolves to auth: followed by the hex SHA-256 of that key.

default_policy / tenants[] fields

Field Type Meaning
tenant_key string Canonical tenant key. Must be absent on default_policy and present on every entry under tenants.
tokens_per_minute integer, > 0 Token-bucket capacity and full-minute refill rate for this scope. The bucket starts full, so the first request(s) can burst up to this value immediately.
requests_per_minute integer, > 0 Same shape as tokens_per_minute, but for request count. Each admitted reservation debits exactly 1, regardless of how many tokens it used.
model_rules list, optional Per-model overrides layered on top of this scope's own limits. See below.

model_rules[] fields

Field Type Meaning
rule_id string, [A-Za-z0-9._-]+ Stable identifier, unique within the tenant (or default_policy).
matcher.type exact | prefix How matcher.value is compared against the request's model ID.
matcher.value string The model ID (exact) or model ID prefix (prefix) this rule applies to. No surrounding whitespace.
tokens_per_minute integer, > 0 Independent token bucket for this rule.
requests_per_minute integer, > 0 Independent request bucket for this rule.

At most one model rule applies per request. An exact match wins over any prefix match; the longest matching prefix wins among competing prefixes. Rules never stack with each other — only with the tenant-global limits. When a rule applies, a reservation must be affordable in both the tenant-global scope and the rule's scope, or it's denied.

Validation

Checked once, at load, before the gateway ever serves traffic on this config:

  • default_policy must not set tenant_key; every entry under tenants must.
  • Tenant keys must be non-empty, have no surrounding whitespace, be unique across tenants, and be a canonical serving-path tenant key: auth:<id>, header:<id>, ip:<address>, or exactly anonymous — a bare ID copy-pasted without its prefix is rejected rather than silently never matching.
  • The address in an ip: tenant key must be a valid IP address written in its canonical form. A key whose address doesn't parse (ip:not-an-address) is rejected, and so is a parseable but non-canonical spelling such as an uppercased or expanded IPv6 address: ip:2001:DB8::1 is refused because it could never match ip:2001:db8::1, the compressed lowercase form the gateway resolves a request's address to.
  • A header: tenant key requires --trust-tenant-header, since without it no request resolves to a header: tenant.
  • tokens_per_minute and requests_per_minute must be > 0, on every scope (default_policy, each tenant, each model rule).
  • rule_id must match [A-Za-z0-9._-]+, and be unique within its tenant (or default_policy).
  • matcher.value must be non-empty and have no surrounding whitespace.
  • No two rules within the same tenant may share the same exact value, or the same prefix value (an exact and a prefix rule may share the same literal string — they're different match kinds).
  • Unknown YAML fields anywhere in the document are rejected rather than silently ignored (so a typo like tenant: instead of tenants: fails loudly instead of compiling to an empty override list).

Any validation failure is the same as an unparsable file: the gateway refuses to start.


Response codes

Status Condition X-SMG-Error-Code Extra headers
429 Too Many Requests Reservation denied — the tenant (or matching model rule) doesn't have enough budget right now tenant_rate_limit_exceeded Retry-After: <seconds> — present only for a finite wait

The response body is the gateway's standard JSON error envelope:

{
  "error": {
    "type": "rate_limit_error",
    "code": "tenant_rate_limit_exceeded",
    "message": "Tenant rate limit exceeded for this request",
    "param": null
  }
}

The one exception is /v1/messages, whichever router serves it: there the denial body leaves as the Messages error envelope — {"type": "error", "error": {"type": "rate_limit_error", "message": ...}, "request_id": ...} — which has no code or param field. The status, the X-SMG-Error-Code header, and Retry-After carry over unchanged.


How admission is computed

Each affected scope (tenant-global, and the matching model rule if any) is a continuous-refill bucket with two independent counters:

  • Capacity = the configured tokens_per_minute / requests_per_minute value itself. The bucket starts full.
  • Refill rate = capacity ÷ 60, applied continuously (not in discrete per-minute resets).
  • Reserve debits the estimated input-token count from the token counter and a flat 1 from the request counter, only if both counters can currently afford it. A request is denied if either can't.
  • The estimate is the tokenized input's real count on the gRPC router. The HTTP routers (regular, PD, and external) have no tokenizer: a pre-tokenized request reserves the token ids it carries, and text reserves 1.3 tokens per whitespace-separated word, rounded up — counting input that routing leaves out (a chat history's tool-call arguments, a Messages history's tool inputs and tool-result text, a Responses request's instructions and tool calls with their outputs, an Interactions request's system instruction, and every prompt of a batch), but not images, files, or built-in tool items. An x-smg-routing-tokens hint never sets the estimate.
  • Settle applies a signed delta — (real input tokens + real completion tokens) − estimated tokens — to the token counter only. A response that used more than estimated can push the counter temporarily negative (debt); the request counter is never trued up, since it was always exactly 1.
  • The real counts are the backend's reported usage — prompt tokens alone for embeddings and classify, which generate nothing. On the HTTP routers they are read from the buffered response body in its route's shape: OpenAI's prompt_tokens/completion_tokens (chat, completions), the Responses input_tokens/output_tokens, the Messages usage with its cache reads and writes added back onto input_tokens, /generate's per-entry meta_info counts (each prompt counted once across its n samples), and on /v1/interactions, total_input_tokens plus total_tool_use_tokens as the input and total_output_tokens plus total_thought_tokens as the generation (thinking tokens are generated output).
  • A reservation that never learns real usage keeps the estimate as the charge. On the HTTP routers that is every streamed success (the stream is never parsed), every non-2xx answer, and a buffered success whose usage can't be read whole; on the gRPC router, a failed or abandoned request, or a stream that ends without the backend reporting usage.

Buckets are kept per gateway instance: a tenant that reaches N independent SMG instances can consume up to roughly N times the configured budgets.


Observability

There are currently no dedicated Prometheus metrics for tenant rate-limit decisions — admissions and settlement deltas are not exposed as counters or histograms today. Denials are visible as the 429 responses themselves, which the generic smg_http_responses_total counter records with status_code="429" and error_code="tenant_rate_limit_exceeded".


See also

Tenant Rate Limiting Concept

The reserve/settle model, retry handling, and n>1 accounting.

Tenant Rate Limiting →

Rate Limiting Reference

The concurrency-based limiter this feature complements.

Rate Limiting →

Configuration Reference

All other gateway CLI flags and configuration options.

Configuration →