Retries¶

SMG implements automatic retries with exponential backoff to handle transient failures gracefully without overwhelming recovering services.

Overview¶

Why Retries?¶

Transient failures are common in distributed systems:

• Network timeouts: Temporary network congestion or packet loss
• Worker overload: Temporary capacity limits (429 responses)
• Intermittent errors: Brief service interruptions during deployments
• Connection issues: Worker restart or network partition

Without retries, every transient failure becomes a client-visible error. With retries, SMG handles these automatically.

Exponential Backoff with Jitter¶

SMG uses exponential backoff with jitter to space out retry attempts:

delay = initial_backoff_ms * (backoff_multiplier ^ attempt)
delay = min(delay, max_backoff_ms)
delay = delay * (1 + random(-jitter_factor, +jitter_factor))

Example Progression¶

With default settings (no jitter for clarity):

Why Jitter?¶

Without jitter, if multiple requests fail simultaneously, they all retry at exactly the same time—potentially overwhelming the recovering service. Jitter spreads out retries randomly to prevent this "thundering herd" problem.

Retryable Status Codes¶

SMG automatically retries requests that fail with these status codes:

Requests with other status codes (e.g., 400 Bad Request, 401 Unauthorized) are not retried because they would likely fail again.

Configuration¶

smg \
  --worker-urls http://w1:8000 http://w2:8000 \
  --retry-max-retries 5 \
  --retry-initial-backoff-ms 50 \
  --retry-max-backoff-ms 30000 \
  --retry-backoff-multiplier 1.5 \
  --retry-jitter-factor 0.2

Parameters¶

Recommended Configurations¶

smg \
  --retry-max-retries 2 \
  --retry-initial-backoff-ms 10 \
  --retry-max-backoff-ms 100

smg \
  --retry-max-retries 3 \
  --retry-initial-backoff-ms 100 \
  --retry-backoff-multiplier 2.0

smg \
  --retry-max-retries 10 \
  --retry-initial-backoff-ms 100 \
  --retry-max-backoff-ms 60000 \
  --retry-backoff-multiplier 2.0

smg --disable-retries

Interaction with Circuit Breakers¶

Retries and circuit breakers work together:

When a circuit is open:

• Requests are rejected immediately (no retry to that worker)
• If other healthy workers exist, the retry goes to them
• If all circuits are open, the request fails

Monitoring¶

Metrics¶

Useful PromQL Queries¶

# Retries per second
rate(smg_retry_attempts_total[5m])

# Retry success rate
rate(smg_retry_attempts_total{status="success"}[5m]) /
rate(smg_retry_attempts_total[5m])

# Average backoff delay
rate(smg_retry_backoff_seconds_sum[5m]) /
rate(smg_retry_backoff_seconds_count[5m])

# 99th percentile backoff
histogram_quantile(0.99, smg_retry_backoff_seconds_bucket)

Alert Thresholds¶

Tuning Guidelines¶

What's Next?¶