Health Checks¶

Background health checks continuously monitor worker availability, removing unhealthy workers from the selection pool before they can cause request failures.

Overview¶

Why Health Checks?¶

Without proactive health checks:

• Reactive detection: Failures only discovered when real requests fail
• Wasted requests: Multiple requests may fail before worker is marked unhealthy
• Slower recovery: No way to know when a worker has recovered without trying it

With health checks:

• Proactive detection: Unhealthy workers removed before they cause failures
• Fast recovery: Workers rejoin the pool as soon as they're healthy
• No wasted requests: Real requests only go to verified healthy workers

How It Works¶

SMG sends periodic HTTP requests to each worker's health endpoint:

Worker States¶

The smg_worker_health gauge collapses these to 1 (Ready) and 0 (anything else), so existing dashboards continue to work.

State Transitions¶

Pending → Ready: When consecutive successful probes reach --health-success-threshold.

Pending → Failed: If the worker accumulates 10 × failure_threshold total probes without ever reaching the success threshold (prevents misconfigured URLs from lingering forever).

Ready → NotReady: When consecutive failed probes reach --health-failure-threshold.

NotReady → Ready: When consecutive successful probes reach --health-success-threshold.

NotReady → Failed: When consecutive failures reach 3 × --health-failure-threshold (the liveness threshold — analogous to a Kubernetes liveness probe, tolerating longer outages than the readiness threshold).

Failed is terminal: Successful probes do not recover a Failed worker. A failed worker is removed via --remove-unhealthy-workers or requires manual re-registration.

Configuration¶

smg \
  --worker-urls http://w1:8000 http://w2:8000 \
  --health-check-interval-secs 60 \
  --health-failure-threshold 3 \
  --health-success-threshold 2 \
  --health-check-timeout-secs 5 \
  --health-check-endpoint /health

Parameters¶

Recommended Configurations¶

smg \
  --health-check-interval-secs 10 \
  --health-failure-threshold 2 \
  --health-check-timeout-secs 3

smg \
  --health-check-interval-secs 120 \
  --health-failure-threshold 5 \
  --health-success-threshold 3

smg \
  --health-check-interval-secs 30 \
  --health-failure-threshold 3 \
  --health-success-threshold 2 \
  --health-check-timeout-secs 5

smg --disable-health-check

Worker Health Endpoint¶

SMG expects workers to provide a health endpoint that returns:

• 2xx status code: Worker is healthy
• Any other status or timeout: Worker is unhealthy

Example Health Endpoint (vLLM)¶

vLLM workers expose /health by default:

# vLLM automatically provides /health endpoint
vllm serve meta-llama/Llama-3.1-8B-Instruct --port 8000

Example Health Endpoint (SGLang)¶

SGLang workers expose /health by default:

# SGLang automatically provides /health endpoint
python -m sglang.launch_server --model-path meta-llama/Llama-3.1-8B-Instruct --port 8000

Custom Health Endpoint¶

If your worker uses a different health endpoint:

smg \
  --worker-urls http://worker:8000 \
  --health-check-endpoint /api/health

Interaction with Circuit Breakers¶

Health checks and circuit breakers work together for comprehensive fault detection:

Key differences:

• Health checks: Proactive background monitoring (no request impact)
• Circuit breakers: Reactive detection based on real request failures

Both are recommended for production deployments.

Monitoring¶

Metrics¶

Useful PromQL Queries¶

# Current health status per worker
smg_worker_health

# Count of unhealthy workers
count(smg_worker_health == 0)

# Health check success rate
rate(smg_worker_health_checks_total{result="success"}[5m]) /
rate(smg_worker_health_checks_total[5m])

# Failed checks per minute
rate(smg_worker_health_checks_total{result="failure"}[1m]) * 60

Alert Thresholds¶

Alerting Example¶

groups:
  - name: smg-health-checks
    rules:
      - alert: WorkerUnhealthy
        expr: smg_worker_health == 0
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Worker {{ $labels.worker }} is unhealthy"

      - alert: MajorityUnhealthy
        expr: count(smg_worker_health == 0) > count(smg_worker_health) / 2
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "Majority of workers are unhealthy"

Tuning Guidelines¶

What's Next?¶