Health Checks¶

Background health checks continuously monitor worker availability, removing unhealthy workers from the selection pool before they can cause request failures.

Overview¶

Why Health Checks?¶

Without proactive health checks:

• Reactive detection: Failures only discovered when real requests fail
• Wasted requests: Multiple requests may fail before worker is marked unhealthy
• Slower recovery: No way to know when a worker has recovered without trying it

With health checks:

• Proactive detection: Unhealthy workers removed before they cause failures
• Fast recovery: Workers rejoin the pool as soon as they're healthy
• No wasted requests: Real requests only go to verified healthy workers

How It Works¶

SMG sends periodic HTTP requests to each worker's health endpoint:

Health States¶

State Transitions¶

Healthy → Unhealthy: When consecutive failed health checks reach --health-failure-threshold

Unhealthy → Healthy: When consecutive successful health checks reach --health-success-threshold

Configuration¶

smg \
  --worker-urls http://w1:8000 http://w2:8000 \
  --health-check-interval-secs 60 \
  --health-failure-threshold 3 \
  --health-success-threshold 2 \
  --health-check-timeout-secs 5 \
  --health-check-endpoint /health

Parameters¶

Recommended Configurations¶

smg \
  --health-check-interval-secs 10 \
  --health-failure-threshold 2 \
  --health-check-timeout-secs 3

smg \
  --health-check-interval-secs 120 \
  --health-failure-threshold 5 \
  --health-success-threshold 3

smg \
  --health-check-interval-secs 30 \
  --health-failure-threshold 3 \
  --health-success-threshold 2 \
  --health-check-timeout-secs 5

smg --disable-health-check

Worker Health Endpoint¶

SMG expects workers to provide a health endpoint that returns:

• 2xx status code: Worker is healthy
• Any other status or timeout: Worker is unhealthy

Example Health Endpoint (vLLM)¶

vLLM workers expose /health by default:

# vLLM automatically provides /health endpoint
vllm serve meta-llama/Llama-3.1-8B-Instruct --port 8000

Example Health Endpoint (SGLang)¶

SGLang workers expose /health by default:

# SGLang automatically provides /health endpoint
python -m sglang.launch_server --model-path meta-llama/Llama-3.1-8B-Instruct --port 8000

Custom Health Endpoint¶

If your worker uses a different health endpoint:

smg \
  --worker-urls http://worker:8000 \
  --health-check-endpoint /api/health

Interaction with Circuit Breakers¶

Health checks and circuit breakers work together for comprehensive fault detection:

Key differences:

• Health checks: Proactive background monitoring (no request impact)
• Circuit breakers: Reactive detection based on real request failures

Both are recommended for production deployments.

Monitoring¶

Metrics¶

Useful PromQL Queries¶

# Current health status per worker
smg_worker_health_status

# Count of unhealthy workers
count(smg_worker_health_status == 0)

# Health check success rate
rate(smg_health_check_total{status="success"}[5m]) /
rate(smg_health_check_total[5m])

# Failed checks per minute
rate(smg_health_check_total{status="failure"}[1m]) * 60

Alert Thresholds¶

Alerting Example¶

groups:
  - name: smg-health-checks
    rules:
      - alert: WorkerUnhealthy
        expr: smg_worker_health_status == 0
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Worker {{ $labels.worker_id }} is unhealthy"

      - alert: MajorityUnhealthy
        expr: count(smg_worker_health_status == 0) > count(smg_worker_health_status) / 2
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "Majority of workers are unhealthy"

Tuning Guidelines¶

What's Next?¶