Circuit Breakers¶

Circuit breakers prevent cascade failures by stopping traffic to unhealthy workers. They're essential for maintaining system stability when workers fail.

Overview¶

Why Circuit Breakers?¶

Without circuit breakers, a failing worker can cause:

1. Wasted requests: Requests sent to failing workers timeout
2. Increased latency: Clients wait for timeouts before retry
3. Resource exhaustion: Connections pile up to dead workers
4. Cascade failures: Retry storms overwhelm remaining workers

Circuit breakers fail fast—they detect failing workers and stop sending traffic immediately.

How It Works¶

Each worker has its own circuit breaker with three states:

State Transitions¶

Closed → Open¶

The circuit opens when:

consecutive_failures >= failure_threshold

Within the sliding window (--cb-window-duration-secs).

Open → Half-Open¶

After the circuit has been open for --cb-timeout-duration-secs, it transitions to half-open to test if the worker has recovered.

Half-Open → Closed¶

If --cb-success-threshold consecutive requests succeed, the circuit closes and normal operation resumes.

Half-Open → Open¶

If any request fails during half-open, the circuit immediately reopens.

Configuration¶

smg \
  --worker-urls http://w1:8000 http://w2:8000 \
  --cb-failure-threshold 5 \
  --cb-success-threshold 2 \
  --cb-timeout-duration-secs 30 \
  --cb-window-duration-secs 60

Parameters¶

Configuration Examples¶

smg \
  --cb-failure-threshold 3 \
  --cb-timeout-duration-secs 30

smg \
  --cb-failure-threshold 20 \
  --cb-success-threshold 5 \
  --cb-timeout-duration-secs 120

Tuning Guidelines¶

Example Scenarios¶

Normal Operation¶

Worker Fails¶

Recovery¶

Monitoring¶

Metrics¶

Useful PromQL Queries¶

# Current circuit breaker states
smg_circuit_breaker_state

# Workers with open circuits
count(smg_circuit_breaker_state == 1)

# State transitions rate
rate(smg_circuit_breaker_transitions_total[5m])

# Consecutive failures per worker
smg_circuit_breaker_consecutive_failures

Alert Thresholds¶

Alerting Example¶

groups:
  - name: smg-circuit-breakers
    rules:
      - alert: CircuitBreakerOpen
        expr: smg_circuit_breaker_state == 1
        for: 1m
        labels:
          severity: warning
        annotations:
          summary: "Circuit breaker open for {{ $labels.worker_id }}"

      - alert: AllCircuitsOpen
        expr: count(smg_circuit_breaker_state == 1) == count(smg_circuit_breaker_state)
        for: 30s
        labels:
          severity: critical
        annotations:
          summary: "All worker circuit breakers are open"

Interaction with Other Features¶

Retries¶

When a circuit is open:

• Requests are rejected immediately (no retry to that worker)
• Other workers may be tried if available

When a circuit is half-open:

• Only a limited number of test requests are sent
• Failures don't count against retry budget

Health Checks¶

Circuit breakers and health checks work together:

Disabling Circuit Breakers¶

In some cases, you may want to disable circuit breakers:

smg --worker-urls http://w1:8000 --disable-circuit-breaker

What's Next?¶