Priority Scheduler Reference¶

Precise contract for the priority-aware admission scheduler: the request header clients send, the response codes the gateway returns, and every configuration knob with its exact name and default. For how it works conceptually, see Priority Scheduling.

The scheduler is disabled by default. When off, the gateway uses its legacy concurrency-limit admission path.

Request header: x-smg-priority¶

Clients request a priority class with the x-smg-priority request header.

Tenant clamp¶

The header chooses a class; the tenant's configured maximum class caps it. The effective class is:

effective = min(requested_class, tenant_max_class)

• The clamp only ever moves a request down. The header can never promote a request above the tenant's ceiling.
• A tenant whose max_class is default that sends x-smg-priority: system is admitted as default.
• A clamp (effective class below requested) is counted under smg_scheduler_clamp_total.

A tenant's max_class comes from the per-tenant policy in the YAML config, or from the gateway-wide default (--priority-scheduler-default-max-class) for tenants not listed. See Tenant policy.

Response codes¶

The scheduler surfaces admission and preemption outcomes as HTTP status codes. Each rejection also carries the gateway's standard JSON error body and X-SMG-Error-Code header.

Enabling the scheduler¶

The scheduler is controlled by CLI flags, accepted by both the Rust smg binary and the Python launcher (smg launch from pip install smg, also the container image's entrypoint). Per-class tuning and per-tenant policy live in a separate optional YAML file.

The scheduler cannot be combined with PD prefill admission (--prefill-max-inflight-requests-per-worker); that combination is refused at startup.

smg launch \
  --worker-urls http://w1:8000 http://w2:8000 \
  --priority-scheduler-enabled \
  --priority-scheduler-default-max-class interactive \
  --priority-scheduler-config /etc/smg/priority.yaml

CLI flags¶

While the scheduler is active, the legacy admission flags change meaning: --queue-size is not used (the per-class queue_size and queue_timeout_secs govern queueing), --max-concurrent-requests only sets the fallback capacity while no healthy worker is registered, and a positive --rate-limit-tokens-per-second adds a request-rate check before admission that sheds with 429 scheduler_queue_full (setting --max-concurrent-requests to 0 removes the shared token bucket and this check with it). --queue-timeout-secs no longer bounds admission waits, but on the gRPC router it still caps how long an admitted request may sit ahead of worker selection, counted from its scheduler admission (503 selection_queue_timeout, Retry-After: 2). See Rate Limiting.

YAML configuration¶

The file referenced by --priority-scheduler-config has two top-level maps, both optional. An empty or absent file means "use built-in defaults for every class, no per-tenant overrides."

# Per-class tuning. Any class you omit keeps its built-in default.
classes:
  interactive:
    reserved_floor: 128       # always at least 128 slots
    reserved_per_slot: 0.25   # ...and 25% of capacity once the fleet is large
    queue_size: 256
    queue_timeout_secs: 30
    starvation_threshold_secs: 5
    can_preempt: true
  bulk:
    reserved_floor: 0
    queue_size: 1024
    queue_timeout_secs: 300
    starvation_threshold_secs: 120
    can_preempt: false

# Per-tenant priority ceiling. Tenants not listed use
# --priority-scheduler-default-max-class.
tenant_policies:
  "auth:acme":
    max_class: interactive
  "auth:internal-cron":
    max_class: system

Class keys and max_class values are lowercase: system, interactive, default, bulk. An unknown class name — or any unknown key anywhere in the file — is a parse error that stops the gateway at startup, unlike the lenient request header: a misspelt section fails loudly instead of being silently dropped.

Per-class knobs¶

Each entry under classes accepts the following fields. All are per-class.

Built-in defaults¶

These apply to any class with no YAML override.

Higher classes fail fast (short queues, short timeouts) and reserve capacity — interactive and default reserve a share that grows with the fleet, while system keeps a fixed floor (control-plane traffic is low-volume regardless of fleet size). Lower classes wait patiently (deep queues, long timeouts) and reserve nothing.

Validation¶

At startup the gateway validates, refusing to start on a failure:

• queue_timeout_secs > 0 for every class.
• starvation_threshold_secs > 0 for every class.
• reserved_per_slot finite and ≥ 0 for every class.

Reservations against capacity are handled separately: if the floors + shares sum to more than the fleet's capacity at startup, the scheduler falls back to legacy admission instead of stopping the gateway. At runtime, a capacity shrink scales every class's reservation down proportionally rather than locking the scheduler out.

Tenant policy¶

A tenant's priority ceiling is resolved per request:

1. If the tenant key appears in tenant_policies, its max_class is used.
2. Otherwise the gateway-wide --priority-scheduler-default-max-class applies.

The resolved max_class is the upper bound for the tenant clamp. Tenant keys are the same keys the gateway uses elsewhere for tenancy (for example auth:acme).

Metrics¶

The scheduler exposes these Prometheus metrics (see the Metrics Reference for the full catalog):

See also¶