Multiple Workers¶

SMG can route across many workers simultaneously — local inference servers, remote cloud APIs, or a mix of both. This guide covers how to add workers and balance traffic across them.

Supported Worker Types¶

SMG connects to workers over HTTP or gRPC, and supports both local inference servers and remote API providers:

Static Workers via CLI¶

Pass multiple URLs to --worker-urls:

smg \
  --worker-urls http://worker1:8000 http://worker2:8000 http://worker3:8000 \
  --policy round_robin \
  --host 0.0.0.0 \
  --port 30000

For gRPC workers, use the grpc:// scheme and provide --model-path so the gateway can load the tokenizer:

smg \
  --worker-urls grpc://worker1:50051 grpc://worker2:50052 \
  --model-path meta-llama/Llama-3.1-8B-Instruct \
  --policy round_robin

See gRPC Workers for details on what gRPC mode enables.

Cloud API Workers¶

Route to cloud providers by setting --backend and passing the provider URL. API keys are either passed by the caller through the Authorization header (BYOK) or stored on the worker record when registering via the admin API. SMG auto-detects the provider (OpenAI, Anthropic, xAI, Gemini) from the model name and applies the correct API transformations.

export OPENAI_API_KEY=sk-...

smg \
  --backend openai \
  --worker-urls https://api.openai.com \
  --host 0.0.0.0 \
  --port 30000

export ANTHROPIC_API_KEY=sk-ant-...

smg \
  --backend openai \
  --worker-urls https://api.anthropic.com \
  --host 0.0.0.0 \
  --port 30000

export XAI_API_KEY=xai-...

smg \
  --backend openai \
  --worker-urls https://api.x.ai \
  --host 0.0.0.0 \
  --port 30000

export GEMINI_API_KEY=...

smg \
  --backend openai \
  --worker-urls https://generativelanguage.googleapis.com \
  --host 0.0.0.0 \
  --port 30000

Dynamic Workers with IGW Mode¶

In Inference Gateway (IGW) mode, SMG starts with no workers and you add or remove them at runtime via the REST API:

smg --enable-igw --host 0.0.0.0 --port 30000

Add a worker¶

curl -X POST http://localhost:30000/workers \
  -H "Content-Type: application/json" \
  -d '{"url": "http://worker1:8000"}'

Response:

{
  "status": "accepted",
  "worker_id": "a1b2c3d4",
  "url": "http://worker1:8000",
  "location": "/workers/a1b2c3d4",
  "message": "Worker addition queued for background processing"
}

List workers¶

curl http://localhost:30000/workers

Remove a worker¶

curl -X DELETE http://localhost:30000/workers/{worker_id}

Worker configuration options¶

The POST /workers endpoint accepts additional fields:

{
  "url": "http://worker:8000",
  "api_key": "optional-key",
  "runtime": "sglang",
  "worker_type": "regular",
  "priority": 50,
  "cost": 1.0,
  "labels": {"region": "us-east"}
}

Realtime-capable workers¶

To route the Realtime API — the WebSocket /v1/realtime endpoint, WebRTC /v1/realtime/calls, and the realtime REST endpoints — through the HTTP router to a local worker, mark that worker with the well-known realtime label. Only workers labeled realtime: "true" receive realtime traffic, so SMG never proxies a realtime connection to a worker that can't serve it.

The worker must itself expose an OpenAI-compatible realtime endpoint. For example, vLLM serving a speech model with the realtime task (such as Qwen/Qwen3-ASR-1.7B) exposes ws://<worker>/v1/realtime for streaming transcription.

curl -X POST http://localhost:30000/workers \
  -H "Content-Type: application/json" \
  -d '{
    "url": "http://asr-worker:8000",
    "runtime": "vllm",
    "labels": {"realtime": "true"}
  }'

The same worker also serves batch transcription via POST /v1/audio/transcriptions, which the HTTP router forwards without requiring the realtime label.

Verify¶

# List connected workers
curl http://localhost:30000/workers

# Check health
curl http://localhost:30000/health

# Send a request
curl http://localhost:30000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "meta-llama/Llama-3.1-8B-Instruct",
    "messages": [{"role": "user", "content": "Hello!"}]
  }'

Next Steps¶

• Monitoring — Track request rates, latency, and worker health
• gRPC Workers — Enable tokenization, chat templates, and tool parsing at the gateway
• PD Disaggregation — Separate prefill and decode onto specialized workers