Reliability Controls

This guide provides command-first setup for request protection and failure handling: concurrency limits, retries, and circuit breakers.


1. Concurrency and Queue Limits

Cap how many requests the gateway runs at once, and how many can wait:

smg launch \
  --worker-urls http://w1:8000 http://w2:8000 \
  --max-concurrent-requests 100 \
  --queue-size 200 \
  --queue-timeout-secs 30
  • Each admitted request holds a permit until its response finishes, streaming included, so at most 100 requests are in flight.
  • Up to 200 more wait in arrival order. A request that finds the queue full gets 429; one that waits longer than 30 seconds gets 503. Both carry Retry-After: 2.
  • Size --max-concurrent-requests from what your workers can run at once; see Sizing Guidelines.

--rate-limit-tokens-per-second adds a sustained admission rate on top of the concurrency cap. Admission then spends rate budget that refills at the configured rate, bursting up to --max-concurrent-requests; a finished request frees its concurrency slot but returns no rate budget, so requests in flight stay capped. Unset or 0, the default, applies no rate limit:

smg launch \
  --worker-urls http://w1:8000 http://w2:8000 \
  --max-concurrent-requests 100 \
  --rate-limit-tokens-per-second 100 \
  --queue-size 200 \
  --queue-timeout-secs 30

For per-worker load thresholds that take a saturated worker out of routing, see Overload Protection.


2. Retries

Retries are on by default. Set the attempt budget and backoff explicitly:

smg launch \
  --worker-urls http://w1:8000 http://w2:8000 \
  --retry-max-retries 5 \
  --retry-initial-backoff-ms 50 \
  --retry-max-backoff-ms 30000 \
  --retry-backoff-multiplier 1.5 \
  --retry-jitter-factor 0.2

--retry-max-retries counts attempts including the first, so 5 allows up to four retries and 1 disables retries.

Disable retries when the client handles them:

smg launch \
  --worker-urls http://w1:8000 http://w2:8000 \
  --disable-retries

3. Circuit Breakers

Protect traffic from repeatedly failing workers:

smg launch \
  --worker-urls http://w1:8000 http://w2:8000 \
  --cb-failure-threshold 10 \
  --cb-success-threshold 3 \
  --cb-timeout-duration-secs 60

A worker's circuit opens after 10 consecutive failed requests (408, 500, 502, 503, 504, or no response). A 429 from a worker is capacity pushback: it can still be retried, but it never opens or closes the circuit. See What Counts as a Failure. --cb-window-duration-secs is accepted but not used by the breaker.

A request that is retried on the same worker charges that worker's breaker once, however many of its attempts fail there. Per-request counting covers HTTP workers in regular mode and gRPC and ZMQ workers; for HTTP workers in PD mode and for external providers, each failed attempt counts, so one request's retries can add several failures to a worker's counter. See State Transitions.

Disable only for controlled testing:

smg launch \
  --worker-urls http://w1:8000 http://w2:8000 \
  --disable-circuit-breaker

Production Baseline

A practical starting profile:

smg launch \
  --worker-urls http://w1:8000 http://w2:8000 http://w3:8000 \
  --max-concurrent-requests 150 \
  --queue-size 300 \
  --queue-timeout-secs 30 \
  --retry-max-retries 3 \
  --retry-initial-backoff-ms 50 \
  --retry-max-backoff-ms 5000 \
  --retry-backoff-multiplier 2.0 \
  --retry-jitter-factor 0.2 \
  --cb-failure-threshold 10 \
  --cb-success-threshold 3 \
  --cb-timeout-duration-secs 60

Verify

curl http://localhost:30000/health
curl http://localhost:30000/workers

# Admission and circuit breaker metrics (Prometheus port, default 29000)
curl -s http://localhost:29000/metrics | grep -E '^smg_(admission|http_rate_limit|worker_cb)'

Admission metrics appear once requests start going through admission control.


Next Steps