Rate Limiting¶

Rate limiting protects workers from being overwhelmed by too many concurrent requests. SMG uses a token bucket algorithm with optional request queuing.

Overview¶

Why Rate Limit?¶

Without rate limiting:

1. Worker overload: Too many concurrent requests degrade performance
2. Memory exhaustion: Workers run out of GPU memory
3. Cascading timeouts: Slow responses cause client timeouts
4. Poor user experience: Some users get fast responses, others wait forever

Rate limiting ensures fair access and predictable performance.

How It Works¶

SMG uses a token bucket algorithm:

Token Bucket¶

• Bucket capacity: Maximum concurrent requests (--max-concurrent-requests)
• Refill rate: Tokens added per second (--rate-limit-tokens-per-second)
• Request cost: Each request consumes one token

Request Queue¶

When no tokens are available, requests can wait in a queue:

• Queue size: Maximum waiting requests (--queue-size)
• Queue timeout: Maximum wait time (--queue-timeout-secs)

Configuration¶

smg \
  --worker-urls http://w1:8000 http://w2:8000 \
  --max-concurrent-requests 100 \
  --rate-limit-tokens-per-second 50 \
  --queue-size 200 \
  --queue-timeout-secs 30

Rate Limit Parameters¶

Timeout Parameters¶

Response Codes¶

The local rate limiter returns a status-only response with no JSON body (clients should read the HTTP status and X-Request-Id to distinguish cases). SMG does not currently emit a Retry-After header with the response.

When the mesh global rate limit is enabled and exceeded, the 429 response carries a JSON body:

{
  "error": "Rate limit exceeded",
  "current_count": 123,
  "limit": 100
}

Sizing Guidelines¶

Concurrent Requests¶

Base on worker capacity:

max_concurrent_requests = num_workers × requests_per_worker

Queue Size¶

Base on acceptable latency:

queue_size = max_concurrent_requests × queue_depth_factor

Token Refill Rate¶

Base on sustainable throughput:

tokens_per_second = expected_requests_per_second × 1.2

The 1.2 factor provides headroom for bursts.

Example Configurations¶

smg \
  --max-concurrent-requests 50 \
  --queue-size 25 \
  --queue-timeout-secs 5

smg \
  --max-concurrent-requests 200 \
  --queue-size 500 \
  --queue-timeout-secs 60

smg \
  --max-concurrent-requests -1

Monitoring¶

Metrics¶

Useful PromQL Queries¶

# Rate limit decisions per second
rate(smg_http_rate_limit_total[5m])

# By decision type (allowed/rejected)
sum by (result) (
  rate(smg_http_rate_limit_total[5m])
)

# 99th percentile request duration
histogram_quantile(0.99,
  rate(smg_http_request_duration_seconds_bucket[5m]))

Alert Thresholds¶

Client-Side Handling¶

Retry Strategy¶

Clients should implement exponential backoff when receiving 429. SMG does not set a Retry-After header today, so clients must compute their own wait:

import time
import requests

def request_with_retry(url, data, max_retries=5):
    for attempt in range(max_retries):
        response = requests.post(url, json=data)

        if response.status_code == 429:
            # SMG does not emit Retry-After; fall back to exponential backoff.
            time.sleep(2 ** attempt)
            continue

        return response

    raise Exception("Max retries exceeded")

Adaptive Rate¶

Monitor 429 responses and adjust request rate:

class AdaptiveClient:
    def __init__(self, base_rate=10):
        self.rate = base_rate

    def on_success(self):
        self.rate = min(self.rate * 1.1, 100)  # Increase slowly

    def on_rate_limit(self):
        self.rate = self.rate * 0.5  # Decrease quickly

What's Next?¶