Reliability & Resilience¶

SMG provides several mechanisms to ensure high availability and stability.

Core Mechanisms¶

• Circuit Breakers: Automatically detect failing workers and temporarily stop sending traffic to them until they recover.
• Retries & Backoff: Configurable retry logic with exponential backoff to handle transient network issues or worker busy states.
• Rate Limiting: Protect your workers from being overwhelmed by controlling the concurrency and request rate.
• Tenant Rate Limiting: Cap per-tenant LLM token and request consumption per minute, independent of worker concurrency.
• Priority Scheduling: Admit higher-priority traffic first with reserved slots, per-class queues, and TTFT-aware preemption.
• Overload Protection: Exclude workers whose engines report saturation from routing. When every candidate worker is saturated, the request is steered to the least-loaded one by default, or shed immediately with --worker-overload-shed.
• Health Checks: Background probes that take unhealthy workers out of the rotation without waiting for requests to fail, and return them when they recover.
• Graceful Shutdown: Ensure in-flight requests complete before the server stops.