Tenant Rate Limiting Reference¶

Precise contract for per-tenant token/request rate limiting: every configuration knob, the YAML policy schema, and the exact response shape. For how it works conceptually — the reserve/settle model, retry handling, n>1 accounting — see Tenant Rate Limiting.

Tenant rate limiting is disabled by default and, when enabled, only enforced on the gRPC router's Chat, Generate, Completion, and Messages endpoints (Harmony-mode chat included). It does not apply to the Responses endpoint, embeddings, classify, audio transcriptions, or the HTTP/external-provider routers.

Enabling it¶

smg \
  --worker-urls grpc://w1:9000 grpc://w2:9000 \
  --tenant-rate-limit-enabled \
  --tenant-rate-limit-config /etc/smg/tenant-rate-limit.yaml

CLI flags¶

YAML configuration¶

default_policy:
  tokens_per_minute: 100000
  requests_per_minute: 600

tenants:
  - tenant_key: "auth:team-red"
    tokens_per_minute: 500000
    requests_per_minute: 3000
    model_rules:
      - rule_id: gpt4-cap
        matcher:
          type: exact
          value: gpt-4
        tokens_per_minute: 50000
        requests_per_minute: 300
      - rule_id: legacy-models
        matcher:
          type: prefix
          value: "legacy-"
        tokens_per_minute: 10000
        requests_per_minute: 100

  - tenant_key: "anonymous"
    tokens_per_minute: 5000
    requests_per_minute: 60

A tenant not listed under tenants uses default_policy. Tenant keys are the same canonical keys SMG resolves elsewhere in the request path — auth:<id>, header:<id>, ip:<address>, or anonymous — there's no separate tenant-identity system for this feature.

default_policy / tenants[] fields¶

model_rules[] fields¶

At most one model rule applies per request. An exact match wins over any prefix match; the longest matching prefix wins among competing prefixes. Rules never stack with each other — only with the tenant-global limits. When a rule applies, a reservation must be affordable in both the tenant-global scope and the rule's scope, or it's denied.

Validation¶

Checked once, at load, before the gateway ever serves traffic on this config:

• default_policy must not set tenant_key; every entry under tenants must.
• Tenant keys must be non-empty, have no surrounding whitespace, be unique across tenants, and be a canonical serving-path tenant key (auth:, header:, ip:-prefixed, or exactly anonymous) — a bare ID copy-pasted without its prefix is rejected rather than silently never matching.
• tokens_per_minute and requests_per_minute must be > 0, on every scope (default_policy, each tenant, each model rule).
• rule_id must match [A-Za-z0-9._-]+, and be unique within its tenant (or default_policy).
• matcher.value must be non-empty and have no surrounding whitespace.
• No two rules within the same tenant may share the same exact value, or the same prefix value (an exact and a prefix rule may share the same literal string — they're different match kinds).
• Unknown YAML fields anywhere in the document are rejected rather than silently ignored (so a typo like tenant: instead of tenants: fails loudly instead of compiling to an empty override list).

Any validation failure is the same as an unparsable file: logged at ERROR, gateway starts without a rate limiter.

Response codes¶

The response body is the gateway's standard JSON error envelope:

{
  "error": {
    "type": "Too Many Requests",
    "code": "tenant_rate_limit_exceeded",
    "message": "Tenant rate limit exceeded for this request",
    "param": null
  }
}

How admission is computed¶

Each affected scope (tenant-global, and the matching model rule if any) is a continuous-refill bucket with two independent counters:

• Capacity = the configured tokens_per_minute / requests_per_minute value itself. The bucket starts full.
• Refill rate = capacity ÷ 60, applied continuously (not in discrete per-minute resets).
• Reserve debits the estimated input-token count from the token counter and a flat 1 from the request counter, only if both counters can currently afford it. A request is denied if either can't.
• Settle applies a signed delta — (real input tokens + real completion tokens) − estimated tokens — to the token counter only. A response that used more than estimated can push the counter temporarily negative (debt); the request counter is never trued up, since it was always exactly 1.

Observability¶

There are currently no Prometheus metrics for tenant rate-limit decisions — admissions, denials, and settlement deltas are not exposed as counters or histograms today. Denials are visible only via the 429 responses themselves and standard request logging.

See also¶