Rate Limiting¶

Rate limiting protects workers from being overwhelmed by too many concurrent requests. SMG uses a token bucket algorithm with optional request queuing.

Overview¶

Why Rate Limit?¶

Without rate limiting:

1. Worker overload: Too many concurrent requests degrade performance
2. Memory exhaustion: Workers run out of GPU memory
3. Cascading timeouts: Slow responses cause client timeouts
4. Poor user experience: Some users get fast responses, others wait forever

Rate limiting ensures fair access and predictable performance.

How It Works¶

SMG uses a token bucket algorithm:

Token Bucket¶

• Bucket capacity: Maximum concurrent requests (--max-concurrent-requests)
• Refill rate: Tokens added per second (--rate-limit-tokens-per-second)
• Request cost: Each request consumes one token

Request Queue¶

When no tokens are available, requests can wait in a queue:

• Queue size: Maximum waiting requests (--queue-size)
• Queue timeout: Maximum wait time (--queue-timeout-secs)

Configuration¶

smg \
  --worker-urls http://w1:8000 http://w2:8000 \
  --max-concurrent-requests 100 \
  --rate-limit-tokens-per-second 50 \
  --queue-size 200 \
  --queue-timeout-secs 30

Rate Limit Parameters¶

Timeout Parameters¶

Response Codes¶

429 Response¶

{
  "error": {
    "message": "Rate limit exceeded. Please retry later.",
    "type": "rate_limit_error",
    "code": "rate_limit_exceeded"
  }
}

Includes Retry-After header with recommended wait time.

408 Response¶

{
  "error": {
    "message": "Request timed out waiting in queue.",
    "type": "timeout_error",
    "code": "queue_timeout"
  }
}

Sizing Guidelines¶

Concurrent Requests¶

Base on worker capacity:

max_concurrent_requests = num_workers × requests_per_worker

Queue Size¶

Base on acceptable latency:

queue_size = max_concurrent_requests × queue_depth_factor

Token Refill Rate¶

Base on sustainable throughput:

tokens_per_second = expected_requests_per_second × 1.2

The 1.2 factor provides headroom for bursts.

Example Configurations¶

smg \
  --max-concurrent-requests 50 \
  --queue-size 25 \
  --queue-timeout-secs 5

smg \
  --max-concurrent-requests 200 \
  --queue-size 500 \
  --queue-timeout-secs 60

smg \
  --max-concurrent-requests -1

Monitoring¶

Metrics¶

Useful PromQL Queries¶

# Rate limit decisions per second
rate(smg_http_rate_limit_total[5m])

# By decision type (allowed/rejected)
sum by (decision) (
  rate(smg_http_rate_limit_total[5m])
)

# Queue utilization
smg_queue_depth / smg_queue_size

# 99th percentile queue wait
histogram_quantile(0.99,
  rate(smg_queue_wait_seconds_bucket[5m]))

Alert Thresholds¶

Client-Side Handling¶

Retry Strategy¶

Clients should implement exponential backoff when receiving 429:

import time
import requests

def request_with_retry(url, data, max_retries=5):
    for attempt in range(max_retries):
        response = requests.post(url, json=data)

        if response.status_code == 429:
            retry_after = int(response.headers.get('Retry-After', 2 ** attempt))
            time.sleep(retry_after)
            continue

        return response

    raise Exception("Max retries exceeded")

Adaptive Rate¶

Monitor 429 responses and adjust request rate:

class AdaptiveClient:
    def __init__(self, base_rate=10):
        self.rate = base_rate

    def on_success(self):
        self.rate = min(self.rate * 1.1, 100)  # Increase slowly

    def on_rate_limit(self):
        self.rate = self.rate * 0.5  # Decrease quickly

What's Next?¶