Graceful Shutdown¶

Graceful shutdown allows in-flight requests to complete before the gateway terminates, preventing request failures during deployments and restarts.

Overview¶

Why Graceful Shutdown?¶

Without graceful shutdown:

• Abrupt termination: Active requests are immediately disconnected
• Client errors: In-flight requests return connection errors
• Data loss: Streaming responses may be truncated
• Deployment failures: Rolling updates cause visible errors

With graceful shutdown:

• Request completion: Active requests finish normally
• No client errors: Users don't see deployment-related failures
• Clean streaming: Streaming responses complete before shutdown
• Smooth deployments: Zero-downtime rolling updates

How It Works¶

Shutdown Sequence¶

1. Shutdown signal received (SIGTERM or SIGINT). The mesh-only /ha/shutdown API triggers a separate mesh-level broadcast path and is not part of this signal-driven sequence.
2. Stop accepting new connections — axum_server's handle stops the TCP accept loop and marks the in-flight tracker as draining; new connections are refused at the socket level rather than receiving a 503 response.
3. Drain in-flight requests — existing requests continue processing while the server waits on the in-flight tracker.
4. Grace period timer starts — after --shutdown-grace-period-secs, the drain wait times out and the server forces shutdown with any remaining requests still in-flight.
5. Clean exit — once all requests complete (or the grace period expires), background components (MCP orchestrator, etc.) are cleaned up and the process exits.

Configuration¶

smg \
  --worker-urls http://w1:8000 http://w2:8000 \
  --shutdown-grace-period-secs 180

Parameters¶

Recommended Configurations¶

smg --shutdown-grace-period-secs 10

smg --shutdown-grace-period-secs 180

smg --shutdown-grace-period-secs 600

smg --shutdown-grace-period-secs 30

Triggering Shutdown¶

Via Signal¶

# Find the SMG process
pgrep -f smg

# Send SIGTERM for graceful shutdown
kill -TERM <pid>

# Or SIGINT (Ctrl+C in terminal)
kill -INT <pid>

Via API¶

# Trigger graceful shutdown via HTTP (mesh mode only)
curl -X POST http://gateway:30000/ha/shutdown

The /ha/shutdown endpoint lives on the main gateway port (default 30000) and requires mesh mode (--mesh-* flags). Without mesh enabled the endpoint returns 503 Service Unavailable. The mesh handler broadcasts a LEAVING status to peer nodes and stops the mesh rate-limit task — it does not share the same in-flight drain path used by the signal handler.

Kubernetes Integration¶

Kubernetes sends SIGTERM by default when terminating pods. Configure terminationGracePeriodSeconds to match or exceed your SMG grace period:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: smg
spec:
  template:
    spec:
      terminationGracePeriodSeconds: 210  # SMG grace + buffer
      containers:
        - name: smg
          args:
            - --shutdown-grace-period-secs=180

Sizing the Grace Period¶

Consider these factors when setting the grace period:

Calculation Guidelines¶

grace_period = max(
    avg_request_duration * 3,
    p99_request_duration * 1.5,
    max_streaming_duration
)

Example: If your average request is 30s, p99 is 60s, and max streaming is 120s:

grace_period = max(90, 90, 120) = 120 seconds

Integration with Load Balancers¶

For zero-downtime deployments, coordinate with your load balancer:

Pre-Stop Hook (Kubernetes)¶

Remove the pod from the load balancer before shutdown:

spec:
  containers:
    - name: smg
      lifecycle:
        preStop:
          exec:
            command: ["/bin/sh", "-c", "sleep 5"]

The sleep allows the load balancer to stop sending new traffic before SMG begins its graceful shutdown.

Health Check Coordination¶

SMG's /health (liveness) endpoint always returns 200 OK — it does not switch to an unhealthy response during shutdown. To drain traffic cleanly, remove the pod from the load balancer before SMG begins its own drain (for example, with the Kubernetes preStop hook above, or an external control-plane deregister step).

curl http://gateway:30000/health
# Returns 200 OK both during normal operation and throughout the drain

/readiness can still return 503 Service Unavailable when no healthy workers remain, but it reacts to worker state rather than to the shutdown signal itself.

Monitoring¶

Shutdown Events¶

Watch logs for shutdown-related messages:

# Signal received
INFO Received Ctrl+C, starting graceful shutdown
# or
INFO Received terminate signal, starting graceful shutdown

# Gate — in-flight tracker is marked draining and the accept loop stops
INFO Beginning graceful shutdown: gating new connections in_flight=5

# Drain completes within the grace period
INFO All in-flight requests drained

# Or the grace period expires with requests still running
WARN Drain timed out, forcing shutdown with requests still in-flight remaining=2 timeout_secs=180

# Component teardown
INFO HTTP server stopped. Starting component cleanup...
INFO Cleanup complete. Process exiting.

Metrics During Shutdown¶

Tuning Guidelines¶

What's Next?¶