Tenant Rate Limiting Reference¶
Precise contract for per-tenant token/request rate limiting: every configuration knob, the YAML policy schema, and the exact response shape. For how it works conceptually — the reserve/settle model, retry handling, n>1 accounting — see Tenant Rate Limiting.
Tenant rate limiting is disabled by default and, when enabled, only enforced on the gRPC router's Chat, Generate, Completion, Messages, Decisions, Embeddings, Classify, Audio Transcriptions, and Responses endpoints (Harmony-mode Chat and Responses included). The gRPC router serves both gRPC- and ZMQ-connected workers (--connection-mode zmq, ipc:// or tcp:// worker URLs); for every covered endpoint except Decisions, enforcement does not depend on the worker transport. Tenant rate limiting does not apply to the HTTP/external-provider routers.
A Responses request reserves per iteration: every turn of an MCP tool loop takes a reservation of its own — the estimated input tokens plus a flat 1 from the request counter — and settles against that turn's reported usage, so an N-turn loop debits N requests from requests_per_minute. A request without MCP tools runs a single iteration and is charged like a chat request.
Decisions is the one endpoint where the worker transport matters: the gRPC router serves it only on gRPC-connected SGLang workers, and a Decisions request for a model with no such worker is rejected with 501 (X-SMG-Error-Code: decisions_not_supported) before any reservation is made.
Enabling it¶
smg \
--worker-urls grpc://w1:9000 grpc://w2:9000 \
--tenant-rate-limit-enabled \
--tenant-rate-limit-config /etc/smg/tenant-rate-limit.yamlCLI flags¶
| Flag | Default | Description |
|---|---|---|
--tenant-rate-limit-enabled |
false |
Master switch. When unset, no rate limiter is constructed and every request skips reservation entirely. |
--tenant-rate-limit-config |
unset | Path to the tenant-rate-limit YAML. Required when --tenant-rate-limit-enabled is set. |
Both flags are accepted by the Rust smg binary and the Python launcher (smg launch from pip, and the container image) alike; under smg serve they take the --router- prefix (--router-tenant-rate-limit-enabled, --router-tenant-rate-limit-config).
YAML configuration¶
default_policy:
tokens_per_minute: 100000
requests_per_minute: 600
tenants:
- tenant_key: "auth:team-red"
tokens_per_minute: 500000
requests_per_minute: 3000
model_rules:
- rule_id: gpt4-cap
matcher:
type: exact
value: gpt-4
tokens_per_minute: 50000
requests_per_minute: 300
- rule_id: legacy-models
matcher:
type: prefix
value: "legacy-"
tokens_per_minute: 10000
requests_per_minute: 100
- tenant_key: "anonymous"
tokens_per_minute: 5000
requests_per_minute: 60A tenant not listed under tenants uses default_policy, with its own buckets: tenants never share a bucket. Tenant keys are the same canonical keys SMG resolves elsewhere in the request path — auth:<id>, header:<id>, ip:<address>, or anonymous — there's no separate tenant-identity system for this feature. A --tenant-api-key team-red:<key> caller resolves to auth:team-red; a caller using the shared --api-key resolves to auth: followed by the hex SHA-256 of that key.
default_policy / tenants[] fields¶
| Field | Type | Meaning |
|---|---|---|
tenant_key |
string | Canonical tenant key. Must be absent on default_policy and present on every entry under tenants. |
tokens_per_minute |
integer, > 0 |
Token-bucket capacity and full-minute refill rate for this scope. The bucket starts full, so the first request(s) can burst up to this value immediately. |
requests_per_minute |
integer, > 0 |
Same shape as tokens_per_minute, but for request count. Each admitted reservation debits exactly 1, regardless of how many tokens it used. |
model_rules |
list, optional | Per-model overrides layered on top of this scope's own limits. See below. |
model_rules[] fields¶
| Field | Type | Meaning |
|---|---|---|
rule_id |
string, [A-Za-z0-9._-]+ |
Stable identifier, unique within the tenant (or default_policy). |
matcher.type |
exact | prefix |
How matcher.value is compared against the request's model ID. |
matcher.value |
string | The model ID (exact) or model ID prefix (prefix) this rule applies to. No surrounding whitespace. |
tokens_per_minute |
integer, > 0 |
Independent token bucket for this rule. |
requests_per_minute |
integer, > 0 |
Independent request bucket for this rule. |
At most one model rule applies per request. An exact match wins over any prefix match; the longest matching prefix wins among competing prefixes. Rules never stack with each other — only with the tenant-global limits. When a rule applies, a reservation must be affordable in both the tenant-global scope and the rule's scope, or it's denied.
Validation¶
Checked once, at load, before the gateway ever serves traffic on this config:
default_policymust not settenant_key; every entry undertenantsmust.- Tenant keys must be non-empty, have no surrounding whitespace, be unique across
tenants, and be a canonical serving-path tenant key (auth:- orheader:-prefixed,ip:followed by a valid IP address in its canonical form, or exactlyanonymous) — a bare ID copy-pasted without its prefix, or anip:key whose address doesn't parse, is rejected rather than silently never matching. - A
header:tenant key requires--trust-tenant-header, since without it no request resolves to aheader:tenant. tokens_per_minuteandrequests_per_minutemust be> 0, on every scope (default_policy, each tenant, each model rule).rule_idmust match[A-Za-z0-9._-]+, and be unique within its tenant (ordefault_policy).matcher.valuemust be non-empty and have no surrounding whitespace.- No two rules within the same tenant may share the same
exactvalue, or the sameprefixvalue (anexactand aprefixrule may share the same literal string — they're different match kinds). - Unknown YAML fields anywhere in the document are rejected rather than silently ignored (so a typo like
tenant:instead oftenants:fails loudly instead of compiling to an empty override list).
Any validation failure is the same as an unparsable file: the gateway refuses to start.
Response codes¶
| Status | Condition | X-SMG-Error-Code |
Extra headers |
|---|---|---|---|
| 429 Too Many Requests | Reservation denied — the tenant (or matching model rule) doesn't have enough budget right now | tenant_rate_limit_exceeded |
Retry-After: <seconds> — present only for a finite wait |
The response body is the gateway's standard JSON error envelope:
{
"error": {
"type": "rate_limit_error",
"code": "tenant_rate_limit_exceeded",
"message": "Tenant rate limit exceeded for this request",
"param": null
}
}How admission is computed¶
Each affected scope (tenant-global, and the matching model rule if any) is a continuous-refill bucket with two independent counters:
- Capacity = the configured
tokens_per_minute/requests_per_minutevalue itself. The bucket starts full. - Refill rate = capacity ÷ 60, applied continuously (not in discrete per-minute resets).
- Reserve debits the estimated input-token count from the token counter and a flat
1from the request counter, only if both counters can currently afford it. A request is denied if either can't. - Settle applies a signed delta —
(real input tokens + real completion tokens) − estimated tokens— to the token counter only (an embeddings or classify response has no completion, so its real total is the backend's reported prompt count). A response that used more than estimated can push the counter temporarily negative (debt); the request counter is never trued up, since it was always exactly1.
Observability¶
There are no dedicated Prometheus metrics for tenant rate-limit decisions — admissions and settlement deltas are not exposed as counters or histograms. Denials surface through two generic counters, each with blind spots on the Responses endpoint:
smg_http_responses_totalrecords a denial's429—status_code="429",error_code="tenant_rate_limit_exceeded"— when the denial is the response the request gets: a denied non-streaming request fails with the429even partway through an MCP tool loop, and a streaming request denied before its stream starts gets the429in place of the stream (for streaming Responses, a first-iteration denial). A denial on a later iteration of a streaming MCP tool loop (regular and Harmony mode alike) comes after the stream already went out as200: it ends the stream with an in-stream error event, and this counter records the200, not a429.smg_router_request_errors_totalcounts a denial witherror_type="rate_limit_error"andreason="rate_limited"(the gateway's tenant rate limit) under the denied endpoint's label, on every covered endpoint except Responses. The per-iteration Responses reservations record unevenly: a streaming Responses request outside Harmony mode runs each iteration through the chat pipeline, so its denials — first iteration and mid-loop alike — count underendpoint="chat", not a Responses label; a non-streaming Responses request and a Harmony-mode Responses request (streaming or not) record no sample here at all. A mid-loop denial on a Harmony-mode streaming request therefore appears in neither counter, only as the in-stream error event that ends the stream.
Both counters' full label sets are in the Metrics Reference.
See also¶
Tenant Rate Limiting Concept¶
The reserve/settle model, retry handling, and n>1 accounting.