OpenAI-Compatible API Reference

SMG provides a fully OpenAI-compatible API, allowing you to use existing OpenAI client libraries with your self-hosted inference workers.


Base URL

http://localhost:30000/v1

Authentication

SMG supports optional API key authentication:

curl http://localhost:30000/v1/chat/completions \
  -H "Authorization: Bearer your-api-key" \
  -H "Content-Type: application/json" \
  -d '...'

Enable authentication with --api-key:

smg launch --worker-urls http://worker:8000 --api-key "your-api-key"

For real multi-tenant separation (e.g. per-tenant rate limiting), configure one key per tenant instead with --tenant-api-key tenant_id:key (repeatable). Each key resolves to its own tenant identity; --api-key remains available as a single shared fallback key whose callers all share one identity.

smg launch --worker-urls http://worker:8000 \
  --tenant-api-key team-red:red-secret \
  --tenant-api-key team-blue:blue-secret

Callers authenticate the same way as with --api-key — a Bearer token in the Authorization header — just using their own tenant's key instead of the shared one:

curl http://localhost:30000/v1/chat/completions \
  -H "Authorization: Bearer red-secret" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "meta-llama/Llama-3.1-8B-Instruct",
    "messages": [{"role": "user", "content": "Hello"}]
  }'

This request is attributed to team-red, distinct from a request authenticated with blue-secret. A missing key, or one that doesn't match any configured --api-key or --tenant-api-key value, is rejected with 401 Unauthorized in the standard error envelope with the code invalid_api_key, plus a WWW-Authenticate: Bearer challenge (error="invalid_token" when a key was presented but not recognized). Tenant keys authenticate serving endpoints only — /v1/chat/completions, /v1/completions, /v1/embeddings, etc. They do not grant access to admin/management routes (/workers, /flush_cache, and similar). When control-plane authentication is enabled, those routes require a control-plane credential: the admin role may use every method, the user role only GET and HEAD. Otherwise, they accept the shared --api-key if configured. With neither control-plane auth nor any API keys configured, those routes are open; with only --tenant-api-key configured, every admin request is rejected with 401.

The model routes — GET /v1/models and GET /v1/models/{model} — are public and answer without a key (see List Models and Retrieve Model).


Endpoints

Chat Completions

Create a chat completion.

POST /v1/chat/completions

Request Body

Field Type Required Description
model string Yes Model identifier. A name configured with --model-alias is accepted and resolved to its canonical model
messages array Yes Array of message objects; must not be empty
max_completion_tokens integer No Upper bound on generated completion tokens (at least 1)
max_tokens integer No Deprecated — use max_completion_tokens. Still accepted: it is the cap when max_completion_tokens is unset, and a proxied body reaches the worker with the field as the client wrote it (not rewritten). OpenAI-named models reject the two limits together (see Validation Rules)
temperature number No Sampling temperature (0-2)
top_p number No Nucleus sampling parameter, greater than 0 and at most 1
n integer No Number of completions to generate (1-128)
stream boolean No Enable streaming responses
stream_options object No Streaming options; only allowed with stream: true. See Stream Usage
stop string/array No Up to 4 non-empty stop sequences
presence_penalty number No Presence penalty (-2 to 2)
frequency_penalty number No Frequency penalty (-2 to 2)
logprobs boolean No Return log probabilities of the output tokens
top_logprobs integer No Most likely tokens to return per position (0-20); requires logprobs: true
logit_bias object No Map of token ID to bias
tools array No Function tools the model may call
tool_choice string/object No none, auto, required, {"type": "function", "function": {"name": "..."}}, or {"type": "allowed_tools", "mode": "auto", "tools": [...]} (mode is auto or required). Every value except none and auto requires tools, and a named function must exist in them
parallel_tool_calls boolean No Allow parallel function calls
response_format object No text, json_object, or json_schema. See Structured Output
reasoning_effort string/number No Reasoning effort; a number is converted to its decimal string. See Thinking and Reasoning Effort
thinking object No Typed thinking control (type, effort, keep, clear_thinking). See Thinking and Reasoning Effort
verbosity string No low, medium, or high; any other value is rejected
user string No End-user identifier

Extension Fields

SMG also accepts these SGLang-style extensions on Chat Completions:

Field Type Description
top_k integer Top-k sampling; -1 disables it, otherwise at least 1
min_p number Min-p sampling threshold (0-1)
min_tokens integer Minimum tokens to generate; must not exceed max_completion_tokens
repetition_penalty number Repetition penalty (0-2)
regex, ebnf string Constrain the output. Only one of regex, ebnf, and a json_schema response format may be set, and neither combines with a JSON response_format
stop_token_ids array Token IDs that stop generation
ignore_eos, no_stop_trim boolean Keep generating past EOS; keep the stop sequence in the output
skip_special_tokens boolean Drop special tokens when detokenizing (default true)
continue_final_message boolean Continue the trailing assistant message instead of starting a new one
add_generation_prompt boolean Whether the prompt ends with the assistant's generation header; absent reads true. false renders the conversation without it, leaving the model to continue on its own. Set to true beside continue_final_message it is rejected with 400 add_generation_prompt_conflicts_with_continue_final_message
separate_reasoning, stream_reasoning boolean Return reasoning in reasoning_content and stream it (both default true)
chat_template_kwargs object Extra chat-template variables. On gRPC workers an explicit entry wins over the tool_choice and response_format values SMG passes to the template, but not over the effort: a reasoning_effort entry applies only when the request sets no effort of its own (see Thinking and Reasoning Effort)
lora_path string LoRA adapter to apply
sampling_seed integer Seed for deterministic sampling
return_hidden_states boolean Return the model's hidden states
rid string Request ID forwarded to the backend

Fields SMG does not model are kept and forwarded unchanged to HTTP workers, and so are unknown stream_options keys such as SGLang's step_usage_chunks. The SGLang-native /generate endpoint likewise keeps its nested sampling_params.custom_params object. Typed sampling numbers are forwarded without float widening: "top_p": 0.95 reaches HTTP workers, PD legs, and external providers as 0.95, not 0.949999988079071.

Message Object

Field Type Required Description
role string Yes system, developer, user, assistant, tool, function, or root (MiniMax only; see Provider Profiles)
content string/array Yes Text, or an array of content parts. Optional on system and assistant messages; required on user messages, where an empty string or array is accepted; a plain string or null on function messages (see below)
name string No Participant name
tool_calls array No Tool calls made by an assistant message
tool_call_id string For tool ID of the tool call this tool message answers
reasoning_content string No Reasoning from an earlier assistant turn; reasoning is accepted as an alias
tools array No Kimi only: tools declared on a system or developer message

On a function message (the deprecated predecessor of tool), content is required but nullable, and a null is kept rather than dropped: a body forwarded to an HTTP worker or external provider still carries "content": null, and on gRPC workers the chat template receives it as empty content in the template's own format (an empty string, or an empty array of parts).

Content Parts

type Payload Notes
text text
image_url image_url.url, optional detail Optional image_url.max_long_side_pixel caps the image's long side: a MiniMax-M3 extension that SMG's own media processing applies for any model (see Frame Sampling)
video_url video_url.url Optional video_url.fps and video_url.max_long_side_pixel set the frame sampling rate and frame size: MiniMax-M3 extensions that SMG's own media processing applies for any model (see Frame Sampling)
input_audio input_audio.data (base64), input_audio.format
audio_url audio_url.url
Anything else Forwarded as sent HTTP workers only

A content part whose type SMG does not model, or a known type with a malformed payload, is forwarded unchanged to HTTP workers. gRPC workers reject it with 400 unsupported_content_part, and the gpt-oss (Harmony) path with 400 harmony_build_failed. Which workers accept which media is covered in Multimodal.

Thinking and Reasoning Effort

Two fields control reasoning. reasoning_effort is OpenAI's effort level. thinking is a typed object shared by the Kimi, MiniMax, z.ai, and DeepSeek dialects:

Field Values Meaning
type enabled, disabled, adaptive Thinking toggle. adaptive states no preference; any other value is rejected
effort string Effort level; takes precedence over reasoning_effort
keep string Kimi keep mode; accepted and ignored by SMG's renderers
clear_thinking boolean z.ai: whether earlier turns' reasoning is dropped from the prompt

The effective effort is thinking.effort when present, otherwise reasoning_effort. Leaving thinking out changes nothing: the chat template's own default applies (Kimi K3, for example, thinks by default), subject to reasoning_effort. The exception is the DeepSeek V4 profile, which turns thinking on unless the request turns it off (see Defaults and Rewrites). A thinking object without type expresses no toggle, but its effort still counts.

On gRPC workers SMG renders the chat template itself. The template receives the effective effort as its reasoning_effort variable, and SMG decides whether the model starts in reasoning mode from the first of these that applies:

  1. an explicit thinking toggle in chat_template_kwargs;
  2. for templates with their own effort levels, the effort the template renders (the request's effective effort, else a chat_template_kwargs.reasoning_effort entry): a native level turns thinking on, and none and minimal turn it off. Three exceptions: the DeepSeek-V4 renderer keeps thinking on for every value but none (a model the DeepSeek V4 profile matches never reaches this step: the profile resolves the toggle up front and writes it as the step-1 override, reading a kwargs-only reasoning_effort of none or minimal as off); Kimi-K3 lets only none decide at this step, leaving every other value — minimal and native levels included — to the steps below; and Hunyuan 4 templates, whose reasoning_effort words carry the toggle itself, are read by those words alone: no_think turns thinking off, and every other value — none and minimal included — keeps thinking on;
  3. thinking.type (enabled on, disabled off);
  4. the effective effort, where none and minimal turn thinking off — except on MiniMax models, where no effort value does (MiniMax reads none as the model's default thinking depth and minimal as low).

If none applies, the template default stands. The template and the reasoning parser use the same decision. On gpt-oss (Harmony) models SMG reads reasoning_effort and clamps it to the three levels Harmony knows: none and minimal become low, xhigh and max become high, and an unknown value becomes medium. HTTP workers receive both fields as sent unless a provider profile rewrites them, and provider profiles restrict the accepted values per model family.

Structured Output

response_format accepts:

  • {"type": "text"}
  • {"type": "json_object"}
  • {"type": "json_schema", "json_schema": {"name": "...", "schema": {...}, "strict": true}}

For json_schema, name must not be empty and schema must be a JSON object ({} is allowed). A string, array, boolean, number, or null schema is rejected with 400 json_schema_schema_not_object before the request reaches any worker, on every backend, unless the HTTP router streams the request to the worker unparsed (see Request Streaming).

Example Request

curl http://localhost:30000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "meta-llama/Llama-3.1-8B-Instruct",
    "messages": [
      {"role": "system", "content": "You are a helpful assistant."},
      {"role": "user", "content": "What is the capital of France?"}
    ],
    "max_tokens": 100,
    "temperature": 0.7
  }'

Response

{
  "id": "chatcmpl-abc123",
  "object": "chat.completion",
  "created": 1705312345,
  "model": "meta-llama/Llama-3.1-8B-Instruct",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "The capital of France is Paris."
      },
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 25,
    "completion_tokens": 8,
    "total_tokens": 33
  }
}

Usage

Field Description
prompt_tokens, completion_tokens, total_tokens Token counts. With n > 1 the shared prompt is counted once and the choices' completions are summed
prompt_tokens_details.cached_tokens Prompt tokens served from the worker's prefix cache
completion_tokens_details.reasoning_tokens Reasoning tokens: a chat completion always carries the key (0 without reasoning), while the legacy Completions endpoint includes it only when the count is non-zero. On gRPC workers SGLang reports them, and SMG counts them itself for gpt-oss (Harmony) models
completion_tokens_details.accepted_prediction_tokens Speculative decoding: draft tokens the target model accepted
completion_tokens_details.rejected_prediction_tokens Speculative decoding: draft tokens rejected (drafted minus accepted). Omitted when the drafted count is unknown — see below

On gRPC workers SMG builds usage from the engine's counters; HTTP workers return the engine's own usage object. SMG reads the speculative counters from SGLang, TokenSpeed, vLLM (started with --per-request-spec-decode-metrics), and TensorRT-LLM (once its servicer fills the fields); MLX reports none.

An engine does not always report a usable drafted count next to the tokens it accepted, so the two speculative fields follow different rules. accepted_prediction_tokens carries the accepted count whenever the engine reports speculative activity. rejected_prediction_tokens is drafted minus accepted, reported only when every choice supplied a positive drafted count at least as large as its accepted count. If any choice pairs measured speculation with a zero drafted count, or reports fewer drafted than accepted tokens, the rejection count is unknown and the key is omitted rather than reported as 0.

Without any speculative decoding a chat completion reports both fields as 0; the legacy Completions endpoint includes them only with speculative activity.

A request's streamed and non-streamed answers carry the same usage object. The running snapshots requested with continuous_usage_stats follow the same rules: a choice whose final counters are still outstanding counts as a zero drafted count, so measured speculation on another choice leaves the rejection count unknown.

Streaming Response

With "stream": true, responses are sent as Server-Sent Events:

curl http://localhost:30000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "meta-llama/Llama-3.1-8B-Instruct",
    "messages": [{"role": "user", "content": "Hello"}],
    "stream": true,
    "stream_options": {"include_usage": true}
  }'

Response (abbreviated):

data: {"id":"chatcmpl-abc123","object":"chat.completion.chunk","created":1705312345,"model":"meta-llama/Llama-3.1-8B-Instruct","choices":[{"index":0,"delta":{"role":"assistant"},"finish_reason":null}]}

data: {"id":"chatcmpl-abc123","object":"chat.completion.chunk","created":1705312345,"model":"meta-llama/Llama-3.1-8B-Instruct","choices":[{"index":0,"delta":{"content":"Hello!"},"finish_reason":null}]}

data: {"id":"chatcmpl-abc123","object":"chat.completion.chunk","created":1705312345,"model":"meta-llama/Llama-3.1-8B-Instruct","choices":[{"index":0,"delta":{},"finish_reason":"stop"}]}

data: {"id":"chatcmpl-abc123","object":"chat.completion.chunk","created":1705312345,"model":"meta-llama/Llama-3.1-8B-Instruct","choices":[],"usage":{"prompt_tokens":9,"completion_tokens":2,"total_tokens":11}}

data: [DONE]

An error before the first chunk is an ordinary JSON error response with its own status; when a worker rejects a streaming request, SMG keeps the worker's content type (application/json, not text/event-stream) so SSE clients still see the error body.

If generation fails after a gRPC stream has started, SMG sends data: {"error": {"message": "...", "type": "..."}} followed by data: [DONE]. The type is stream_error when the worker fails the stream: its connection breaks, its server cuts the stream, it answers a server error or a timeout after the first chunk, or the engine itself reports that the request failed. Any other status the engine answers mid-stream keeps the type it would carry in an error body (see Error Format): for example invalid_request_error when the engine rejects the request after its first chunk, or rate_limit_error for capacity pushback. internal_error means the failure was in SMG's own processing.

A quiet open SSE response can contain : ping comments between events. SMG sends them after 15 seconds of silence by default, including while waiting for the first token. SSE clients should ignore comment lines. Configure or disable them with --sse-keepalive-secs.

Stream Usage

stream_options field Default Effect
include_usage false (Kimi, MiniMax: true) Send the whole request's usage in a final chunk with an empty choices array
continuous_usage_stats false gRPC workers, except for gpt-oss (Harmony) models: with include_usage: true, every chunk also carries a running usage snapshot (the prompt plus the tokens generated so far)

On HTTP workers stream_options, including keys SMG does not model, is forwarded to the engine, which produces the usage chunks.

The Kimi and MiniMax profiles turn include_usage on for streaming requests that leave it unset, on every backend; an explicit false is kept.

The DeepSeek V4 profile uses DeepSeek's stream format on gRPC workers instead: aggregate usage rides on the final finish chunk whether or not include_usage is set, and no usage-only chunk follows. With include_usage: true, earlier chunks carry "usage": null. continuous_usage_stats does not apply, and if the backend stream ends before every choice completes, the finish chunk carries no aggregate usage.


Completions

Create a text completion (legacy API).

POST /v1/completions

Request Body

Field Type Required Description
model string Yes Model identifier
prompt string/array Yes A string, an array of strings, an array of token IDs, or an array of token-ID arrays; an array must be non-empty
max_tokens integer No Maximum tokens to generate
temperature number No Sampling temperature (0-2)
top_p number No Nucleus sampling parameter, greater than 0 and at most 1
n integer No Completions per prompt (1-128)
best_of integer No Candidates to generate server-side (0-20); must be greater than n and cannot be combined with stream
logprobs integer No Log probabilities of the most likely tokens (0-20, the bound of chat's top_logprobs; the public API's own limit of 5 is not imposed)
stream boolean No Enable streaming
stream_options object No include_usage; only allowed with stream: true
stop string/array No Up to 4 non-empty stop sequences
echo boolean No Echo prompt in response
suffix string No Text that comes after the completion
presence_penalty, frequency_penalty number No Penalties (-2 to 2)
seed integer No Seed for best-effort deterministic sampling
user string No End-user identifier

Completions takes the same extension fields as Chat Completions except the chat-only ones (continue_final_message, add_generation_prompt, separate_reasoning, stream_reasoning, chat_template_kwargs), plus a json_schema string constraint. On gRPC workers each prompt in an array is its own engine request.

A prompt can arrive already tokenized: an array of token IDs (unsigned 32-bit integers, such as those /v1/tokenize returns), or an array of token-ID arrays for a batch. The first element of an array fixes its shape — a string makes it an array of text prompts, an integer an array of token IDs, an array a batch of token-ID prompts. A mixed array, a negative, fractional, or out-of-range ID, an empty prompt array, and an empty ID list in a batch are all rejected.

On gRPC workers token IDs bypass the gateway's tokenizer and reach the engine exactly as sent. usage.prompt_tokens counts the IDs, and with echo: true each choice opens with the decoded text of its prompt's IDs (special tokens included; an ID the tokenizer has no text for adds none). On that path every ID must be a token of the model — in its tokenizer's vocabulary, added tokens included, or within the embedding table its workers report — otherwise the request is rejected with invalid_prompt_token before any worker is called (see Error Codes). On HTTP workers the prompt is forwarded as sent and the engine applies its own handling.

Example Request

curl http://localhost:30000/v1/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "meta-llama/Llama-3.1-8B-Instruct",
    "prompt": "The quick brown fox",
    "max_tokens": 50
  }'

Response

{
  "id": "cmpl-abc123",
  "object": "text_completion",
  "created": 1705312345,
  "model": "meta-llama/Llama-3.1-8B-Instruct",
  "choices": [
    {
      "text": " jumps over the lazy dog.",
      "index": 0,
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 4,
    "completion_tokens": 7,
    "total_tokens": 11
  }
}

Decisions

Ask predicate (yes/no), choice, and score questions about shared evidence. The model scores each question's possible answers instead of generating text, and every answer carries a probability distribution.

POST /v1/decisions

Requests are always treated as non-streaming: there is no stream field, and SMG returns a single JSON body.

Request Body

Field Type Required Description
model string Yes Model identifier. A name configured with --model-alias is accepted and resolved to its canonical model
input string/array Yes The evidence every question is answered against: a plain string, or an array of user messages (see Decision Input)
questions array Yes Questions to answer; answers come back in the same order (see Decision Questions)
safety_identifier string No Opaque end-user identifier, at most 128 characters

Fields SMG does not model — on the request and on its messages, parts, questions, choices, and levels — are kept as backend extensions: they are forwarded unchanged to non-SGLang HTTP workers and rejected with 400 on SGLang workers (see Decision Workers).

Decision Input

As an array, each input message has:

Field Type Required Description
role string Yes Must be user
type string No message, when present
content string/array Yes Text, or an array of input_text and input_image parts
Part type Payload Limits
input_text text At most 10,485,760 characters per part
input_image image_url, optional detail (low, high, auto, original) Must be an inline base64 image data URL (data:image/png;base64,...); external URLs and file IDs are rejected. At most 128 images per request

model, instructions, name, description, and label values each take at most 1,048,576 characters. A limit violation is rejected with 400 before routing.

Decision Questions

Every question has a type, required instructions, and an optional name. Names need not be unique: each answer repeats its question's name (null when unset), and answers keep question order.

type Extra fields The model estimates
predicate — The probability that the statement in instructions is true
choice choices: array of {"value": ...} with an optional description; value is a string or boolean, and true is distinct from "true" A probability for each choice
score levels: array of {"label": "..."} with an optional description, ordered from lowest to highest A probability for each level

Example Request

curl http://localhost:30000/v1/decisions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen/Qwen3-4B-Instruct-2507",
    "input": "The delivery arrived a day late, but the package was intact and support was friendly.",
    "questions": [
      {"type": "predicate", "instructions": "Did the delivery arrive on time?"},
      {"type": "choice", "name": "sentiment", "instructions": "What is the overall sentiment?",
       "choices": [{"value": "positive"}, {"value": "negative"}]},
      {"type": "score", "instructions": "Rate the customer experience.",
       "levels": [{"label": "poor"}, {"label": "fair"}, {"label": "good"}]}
    ]
  }'

Response

{
  "model": "Qwen/Qwen3-4B-Instruct-2507",
  "answers": [
    {"type": "predicate", "name": null, "probability": 0.11},
    {"type": "choice", "name": "sentiment", "choice": "positive", "confidence": 0.74,
     "probabilities": [
       {"value": "positive", "probability": 0.87},
       {"value": "negative", "probability": 0.13}
     ]},
    {"type": "score", "name": null, "score": 1.62, "confidence": 0.55,
     "probabilities": [
       {"value": 0, "label": "poor", "probability": 0.08},
       {"value": 1, "label": "fair", "probability": 0.22},
       {"value": 2, "label": "good", "probability": 0.70}
     ]}
  ],
  "usage": {
    "input_tokens": 182,
    "output_tokens": 0,
    "total_tokens": 182,
    "input_tokens_details": {"cached_tokens": 0, "cache_write_tokens": 0},
    "output_tokens_details": {"reasoning_tokens": 0}
  }
}

Each answer carries its question's type and name plus:

type Fields
predicate probability (0–1) that the statement is true
choice choice — the value of the most probable choice; confidence (0–1); probabilities — one {value, probability} entry per choice, in request order
score score — the probability-weighted average of the level indices, not a winning level (level i in request order has value: i, so 3 levels give a score between 0 and 2); confidence (0–1); probabilities — one {value, label, probability} entry per level
refusal No further fields: the worker declined to answer the question. SMG's own SGLang scoring always produces one of the three types above

usage reflects scoring work: on SGLang workers (both the HTTP translation and gRPC scoring) the answer labels are scored without generating tokens, so output_tokens and output_tokens_details.reasoning_tokens are 0 and total_tokens equals input_tokens.

On gRPC workers each question is scored as its own prompt over the shared input — input_tokens sums those prompts — and input_tokens_details.cached_tokens reports the engine's prefix-cache hits. The SGLang HTTP translation returns 0 for both cache counters, meaning unreported rather than measured. Non-SGLang HTTP workers return their own usage object.

Decision Workers

What serves the request depends on the worker:

Worker Behavior
HTTP (SGLang runtime) SMG translates the request to SGLang's native System One API (POST /v1/systemone, as served by SGLang 0.5.21) and maps the reply back to the Decisions shape
HTTP (other runtimes) The request is forwarded to the worker's own /v1/decisions, unknown fields included; the worker's response — errors included — is returned as is
gRPC (regular mode, SGLang runtime) SMG renders one prompt per question with the model's chat template and scores the answer labels through the engine, without generation
PD, EPD, ZMQ, and external providers Not implemented (501 decisions_not_supported)

In a mixed fleet, SMG dispatches Decisions only to regular HTTP workers and regular SGLang gRPC workers serving the model.

The SGLang paths (HTTP translation and gRPC scoring) reject these requests with 400 unsupported_decisions_request before dispatch:

  • input_image parts — SGLang Decisions takes text evidence only
  • unknown fields anywhere on the request — SGLang Decisions cannot honor them
  • empty or whitespace-only text evidence or instructions
  • no questions, a choice question outside 1–255 choices, or a score question outside 1–10 levels

gRPC scoring also depends on the model: the internal answer labels SMG scores (yes/no, option letters, level digits — your own choice values and level labels are unrestricted) must each be one distinct token after the chat prompt, and the model's chat template must produce plain text without leaving a reasoning block open. A model that cannot satisfy this answers 400.

Decision Errors

Status code Returned when
400 The failure's code (see Error Format) The body is not valid JSON (invalid_json), does not match the schema (invalid_value for a non-user role or an unknown question type, invalid_type for a numeric choice value, ...), or fails a limit (invalid_value for a length limit, too_many_images, invalid_image_url); param names the offending field when one is known
400 unsupported_decisions_request An SGLang worker was selected and the request needs something SGLang Decisions does not support (see Decision Workers)
404 model_not_found No worker serves the model
501 decisions_not_supported No regular HTTP worker and no regular SGLang gRPC worker serves the model
502 invalid_decisions_response An SGLang worker (HTTP translation or gRPC scoring) reported success, but the result could not be mapped to a valid Decisions response

Backend errors on the SGLang HTTP translation keep their status and OpenAI error body.

For tenant rate limiting on Decisions, see Tenant Rate Limiting.


List Models

List available models.

GET /v1/models

Example Request

curl http://localhost:30000/v1/models

Response

{
  "object": "list",
  "data": [
    {
      "id": "meta-llama/Llama-3.1-8B-Instruct",
      "object": "model",
      "created": 0,
      "owned_by": "self_hosted"
    }
  ]
}

The list covers the models served by self-hosted workers, each once under its canonical ID, followed by one entry per alias (--model-alias names and model-card aliases entries), so every name a client may send is one it can discover. An alias is listed under the model the registry resolves it to and nowhere else: an alias two workers' cards declare appears once, and a name that is also a registered model's ID is never listed as an alias. created is always 0 and owned_by is self_hosted.

When no self-hosted worker serves any model, the endpoint returns 503 in the gateway's JSON error envelope with the code no_models_available.

The endpoint needs no key. If the caller does send a bearer token (or an x-api-key header) that is not one of the gateway's own keys, SMG treats it as the caller's own provider key: it calls /v1/models on the registered external providers of the caller's API family with that token and returns the first non-empty list, falling back to the self-hosted list. A request carrying anthropic-version reaches the Anthropic providers alone; every other request reaches the OpenAI-compatible ones — OpenAI, xAI, custom providers, and external workers that name no provider — so a key presented in one family's API never reaches another family's provider, and a Gemini provider is asked by neither. A gateway key (--api-key or --tenant-api-key) is never forwarded to a provider.

On that provider list, owned_by is the provider name for the recognized provider hosts (openai, anthropic, xai, gemini) and self_hosted for a provider at any other host.

A request carrying an anthropic-version header gets its inventory in the Anthropic Messages shape instead, with that API's paging read from the query: limit (1 to 1000, default 20), one cursor (after_id or before_id, not both), and a lifecycle filter (active, deprecated, or retired; every self-hosted model is active). The page carries has_more, first_id, and last_id, and its data entries type, id, display_name (the model ID when no display name is known), created_at (1970-01-01T00:00:00Z, the counterpart of created: 0), lifecycle, and the model card's context length as max_tokens and max_input_tokens. An out-of-range limit, both cursors at once, an unknown lifecycle, or a cursor naming no entry is a 400 invalid_query_parameter. Errors then come in the Messages envelope (top-level type: "error" and a request_id). The same header selects the shape on Retrieve Model. On the provider fallback, a Messages-shaped caller's upstream is asked for the page its query names and an Anthropic-shaped page is relayed as the upstream served it; an OpenAI-shaped inventory is paged at the gateway in id order.


Retrieve Model

Fetch one model's card.

GET /v1/models/{model}

Like List Models, this is a public route and answers without a key.

{model} takes the whole rest of the path, so an ID with a slash in it works raw or percent-encoded: /v1/models/meta-llama/Llama-3.1-8B-Instruct and /v1/models/meta-llama%2FLlama-3.1-8B-Instruct name the same model. An alias — a name configured with --model-alias, or an aliases entry on a worker's model card (see the Worker Spec) — resolves to its canonical model, and the response carries the canonical ID. The route answers from the self-hosted inventory only; it never asks an external provider.

Example Request

curl http://localhost:30000/v1/models/meta-llama/Llama-3.1-8B-Instruct

Response

{
  "id": "meta-llama/Llama-3.1-8B-Instruct",
  "object": "model",
  "created": 0,
  "owned_by": "self_hosted"
}

With an anthropic-version header, the response is the Anthropic model object (type, id, display_name, created_at, lifecycle, and the card's context length as max_tokens and max_input_tokens) instead.

An unknown ID gets the same 404 model_not_found as every other route, in the OpenAI error envelope — or, under anthropic-version, in the Messages envelope (top-level type: "error", error.type not_found_error, and a request_id), the same refusal /v1/messages gives that model.


Audio Transcriptions

Transcribe an audio file (batch).

POST /v1/audio/transcriptions

The request is multipart/form-data:

Field Required Description
file Yes The audio file; must not be empty
model Yes Model identifier
language No Language of the audio
prompt No Text to guide the transcript
response_format No Output format (default json)
temperature No Sampling temperature (0.0-1.0)
timestamp_granularities[] No word or segment; repeat the field for both
stream No true/false (or 1/0)

A malformed form, a missing or empty file, a missing model, an out-of-range temperature, or an unparseable stream is rejected with 400 in the standard error envelope before routing, with param naming the field when one is known: missing_required_parameter for a missing model or file, invalid_value for an empty file or a bad temperature or stream value, and invalid_multipart for a form that cannot be read. Other form fields are ignored.

curl http://localhost:30000/v1/audio/transcriptions \
  -F file=@audio.wav \
  -F model=Qwen/Qwen3-ASR-1.7B

What serves the request depends on the worker:

Worker Behavior
HTTP The form is forwarded to the worker's /v1/audio/transcriptions; the worker's model and options apply
gRPC (regular mode) SMG runs transcription as its own pipeline endpoint: it turns the audio into a chat turn, renders it with the model's chat template, and returns the transcript. Supported family: Qwen3-ASR
PD, EPD, and external providers Not implemented (501)

On gRPC, a model belongs to the Qwen3-ASR family when its ID, or a serving worker's model ID or model-identity label, contains qwen3-asr or qwen3_asr (ignoring case). Audio input over gRPC is currently accepted by TokenSpeed workers only; other gRPC runtimes answer 400 multimodal_not_supported. The gRPC endpoint is whole-file only:

Request gRPC result
A model outside the supported families 400 audio_transcription_model_not_supported
stream: true 400 streaming_transcription_not_supported
Any timestamp_granularities 400 transcription_timestamps_not_supported
response_format of srt, vtt, or verbose_json 400 unsupported_transcription_response_format
A language outside the model's list (ISO code or English name, such as en or English) 400 unsupported_transcription_language
A prompt over 4096 bytes 400 asr_prompt_too_long

response_format: json returns {"text": "..."} and text returns the plain transcript. Sampling is greedy unless temperature is set, and a given language pins the transcript language.


Realtime API

SMG proxies the OpenAI Realtime API to a realtime-capable worker. Both the OpenAI router (to an upstream provider) and the HTTP router (to a local worker labeled realtime: "true") support it. The worker must speak the OpenAI Realtime protocol — for local workers, for example vLLM serving an ASR model with the realtime task. SMG relays frames as the worker sent them; error frames are the one exception (see Upstream Errors).

Endpoint Transport Purpose
GET /v1/realtime WebSocket Bidirectional realtime session (e.g. live streaming transcription)
POST /v1/realtime/calls WebRTC (SDP) Browser/WebRTC realtime session
POST /v1/realtime/sessions HTTP Create a realtime session
POST /v1/realtime/client_secrets HTTP Mint an ephemeral client secret
POST /v1/realtime/transcription_sessions HTTP Create a realtime transcription session

GET /v1/realtime requires the model query parameter and answers 400 without it. POST /v1/realtime/calls takes the model from the model query parameter for an application/sdp body, or from the session JSON in a multipart/form-data body.

Upstream Errors

A refusal loses the account identifiers in a JSON error's error.message before it reaches the caller: phrases naming an organization (organization org-…) or a project (project proj_…) are removed, as on every provider relay (see Provider Error Responses). Both routers do this, so error frames from local realtime workers are sanitized too.

  • A non-success answer to a realtime REST call or to the /v1/realtime/calls SDP exchange is returned under the upstream's status and content type with its body sanitized this way.
  • On a WebSocket session, error events and conversation.item.input_audio_transcription.failed events are the only frames SMG rewrites, with the same rule; every other frame is relayed as it came, never re-serialized.

WebSocket example

# pip install websockets
import asyncio, websockets

async def main():
    url = "ws://localhost:30000/v1/realtime?model=Qwen/Qwen3-ASR-1.7B"
    headers = {"Authorization": "Bearer your-api-key"}
    async with websockets.connect(url, additional_headers=headers) as ws:
        # Send realtime events (session.update, input_audio_buffer.append, ...)
        # and receive transcription/response events from the worker.
        ...

asyncio.run(main())

Provider Profiles

Some model families come with a vendor contract that differs from the OpenAI baseline. SMG picks a provider profile from the request's model and applies that profile's defaults and validation to Chat Completions requests before routing, on every backend.

Profile Selection

A profile matches when any /-separated segment of the model ID starts with one of its markers, ignoring case, so moonshotai/Kimi-K2-Instruct and /models/kimi-k3 both select Kimi.

Profile Selected by
Kimi kimi*, moonshot*; the K3 rules apply to kimi-k3* and kimi_k3*
MiniMax minimax*, abab*
z.ai (GLM) glm*, zai*, z-ai*; the GLM-5.3 rules apply to glm-5.3*, glm5.3*, glm_5.3*, glm-5-3*, glm5-3*
DeepSeek V4 A segment equal to deepseek-flash, deepseek-v4-flash, deepseek-v4-pro, deepseek-v4.1-flash, or deepseek-v4.1-flash-1017
OpenAI A model none of the above matched that names one of OpenAI's own hosted models: an openai segment (openai/gpt-4.1), a segment starting with gpt- (gpt-oss* is self-hosted and excluded), chatgpt-, codex-, computer-use-preview, text-embedding- or omni-moderation, or the o1/o3/o4 reasoning series (o3, o4-mini)
Baseline (default) Everything else, including older DeepSeek models and THUDM/chatglm3-6b: only the core request validation applies

Names configured with --model-alias are read through the alias table. An alias takes the profile of the served model it resolves to, so the profile is the same whichever of its names a request carries. A served model whose own name selects no vendor contract takes the contract of a vendor-named (Kimi, MiniMax, z.ai, DeepSeek) alias it is served under, series rules included: an opaque alias of a Kimi K3, GLM-5.3, or DeepSeek V4.1 model keeps that series' rules. When two aliases of one served model name different vendors, the first configured wins. An OpenAI model name binds no contract as an alias: with gpt-4o=local-model, a request for gpt-4o is judged as the self-hosted local-model, not under the OpenAI rules. Message-level extension fields that belong to another profile (such as Kimi's message tools sent to a non-Kimi model) are dropped before dispatch, with one warning logged per request.

Profile defaults and validation apply to /v1/chat/completions requests that pass through SMG's request validation. They are not applied to /v1/responses, and a request the HTTP router streams to the worker unparsed skips them (see Request Streaming).

Validation Rules

A violation is rejected with 400 in the validation envelope (see Error Format): error.type is invalid_request_error, error.code is the rule code from the last column, and error.message carries the rule's message. Profile rules judge the request as a whole, so error.param is null:

{
  "error": {
    "message": "invalid thinking.type: only enabled or disabled is allowed for this model",
    "type": "invalid_request_error",
    "param": null,
    "code": "thinking_type_not_supported"
  }
}
Profile Rejected Rule code
All except MiniMax A root message invalid_role
Kimi tools on a user or assistant message tools_role_restricted
Kimi tools on a system or developer message that is not a list of tools tools_malformed
Kimi A non-empty message tools list together with non-empty content on that message tools_content_conflict
Kimi A message tool whose type is not function tool_type_unsupported
Kimi A message tool name not matching [A-Za-z_][A-Za-z0-9_]*, or longer than 256 characters tool_name_invalid
Kimi A message tool name already declared in tools or on another message tool_name_duplicate
Kimi K3 temperature other than 0, 0.6, or 1; top_p other than 0.95; a non-zero presence_penalty or frequency_penalty; n other than 1 temperature_not_allowed, top_p_not_allowed, presence_penalty_not_allowed, frequency_penalty_not_allowed, n_not_allowed
Kimi K3, z.ai, DeepSeek V4 thinking.type: adaptive thinking_type_not_supported
Kimi K3 thinking.effort other than low, high, or max thinking_effort_invalid
MiniMax A tool-call ID used twice in the history tool_call_id_duplicate
MiniMax A tool message whose tool_call_id answers no pending call tool_call_id_mismatch
MiniMax Historical tool-call arguments that are non-empty but not a JSON object tool_call_arguments_invalid_json
MiniMax A tool call with no tool message answering it tool_call_unanswered
z.ai thinking.effort or reasoning_effort outside none, minimal, low, medium, high, xhigh, max (GLM-5.3 also rejects none: the series cannot switch thinking off) reasoning_effort_not_allowed
z.ai GLM-5.3 thinking.type: disabled thinking_disabled_not_supported
z.ai A tool whose type is not function tool_type_not_supported
z.ai A file_url content part (SMG does not fetch files for the model) content_part_not_supported
z.ai (gRPC) max_completion_tokens (or max_tokens) larger than the context window the selected worker advertises context_length_exceeded
DeepSeek V4 reasoning_effort or thinking.effort outside none, minimal, low, medium, high, xhigh, max; V4.1 also takes an integer budget from 1 to 100 reasoning_effort_not_allowed
DeepSeek V4 chat_template_kwargs.thinking or enable_thinking that is neither a boolean nor null invalid_thinking_override
DeepSeek V4 chat_template_kwargs.thinking and enable_thinking that disagree conflicting_thinking_overrides
OpenAI, DeepSeek V4 A tool message with no assistant tool_calls turn right before it; a tool_call_id that turn did not issue; an assistant tool_calls turn whose calls are not each answered by a tool message before the conversation goes on tool_message_without_tool_calls, tool_call_id_not_found, tool_calls_without_tool_messages
OpenAI max_tokens and max_completion_tokens both set invalid_parameter_combination
OpenAI response_format: json_object without the word "json" (any case) in a message's text json_object_requires_json_in_messages
OpenAI A json_schema response format with strict: true whose schema contains an object without "additionalProperties": false invalid_json_schema
OpenAI A tool with function.strict: true whose parameter schema contains an object without "additionalProperties": false invalid_function_parameters

z.ai and DeepSeek V4 check both effort spellings, even the one that thinking.effort shadows.

The z.ai (gRPC) context-window rule needs the selected worker's advertised window, so it is enforced in the gRPC pipeline rather than in request validation: alone in this table, its 400 carries the X-SMG-Error-Code header.

Defaults and Rewrites

Profile Behavior
Kimi Streaming requests get stream_options.include_usage: true unless set. Tools declared on system and developer messages join the request's tools for tool_choice, the tool-choice constraint, and tool-call parsing; a system message that only declares tools may omit content
Kimi K3 Omitted fields get the contract values: temperature 1.0, top_p 0.95, presence_penalty 0, frequency_penalty 0, n 1
MiniMax Streaming requests get stream_options.include_usage: true unless set. root messages are hoisted above all other messages and merged into one leading message that keeps the root role: texts are joined by a blank line in their original order, a parts list keeps its parts, and the merged message carries the first one's name. On gRPC workers, responses are scanned for tool calls even when the request declares no tools
z.ai thinking.clear_thinking becomes the clear_thinking chat-template variable unless chat_template_kwargs sets it. tool_stream is accepted and dropped (SMG streams tool-call deltas anyway). GLM-5.x, GLM-4.7, and GLM-4.6 get temperature 1.0 and top_p 0.95 when omitted
z.ai GLM-5.3 Effort minimal → low, medium → high, and xhigh → max in both reasoning_effort and thinking.effort, matching z.ai's own API; workers and external providers see only the level
DeepSeek V4 Effort minimal → low; on V4, medium and xhigh → high (V4.1 keeps both as native levels, though a request forwarded to an HTTP worker or external provider still sends medium as high, matching DeepSeek's own API). Thinking is on unless turned off (thinking.type: disabled, effort none, or a template override), and the result is written to chat_template_kwargs.thinking when that is unset. Streams use DeepSeek's usage format (see Stream Usage)

Error Responses

Error Format

Errors that SMG generates use this envelope and repeat code in the X-SMG-Error-Code response header:

{
  "error": {
    "type": "invalid_request_error",
    "code": "context_length_exceeded",
    "message": "This model's maximum context length is 2048 tokens. However, your request has 2310 input tokens. Please reduce the length of the input.",
    "param": null
  }
}

type is the public API's vocabulary, which clients switch on: invalid_request_error for a 4xx (the detail is in code; a 403 is permission_error and a 429 rate_limit_error) and server_error for a 5xx. Request parsing and validation failures use the same envelope but carry no X-SMG-Error-Code header. Their code names the failure, and param names the offending field when one is known (for example messages[0].content); it is null for a syntax error or a rule over the whole request:

Failure code
The body is not valid JSON invalid_json
The body does not match the request schema missing_required_parameter, unknown_parameter, invalid_type (a value of the wrong shape, including a non-object body), invalid_value (a value outside an enumeration), or integer_below_min_value (a negative integer where the field cannot be negative)
A field crosses a documented bound The kind of value and the bound crossed: integer_above_max_value, decimal_below_min_value, string_above_max_length, empty_array, ...
A cross-field or provider-profile rule fails The rule's own code: stream_options_requires_stream, tool_choice_requires_tools, mutually_exclusive_parameters, ...
The Content-Type: application/json header is missing, or the body fails to parse for any other reason json_parse_error

A 401 from API-key authentication uses the envelope (code invalid_api_key) with a WWW-Authenticate: Bearer challenge, but carries no X-SMG-Error-Code header either. Errors from HTTP workers are relayed with the worker's status and body. An external provider's error is relayed under its status too, minus the account identifiers in its message (see Provider Error Responses). SMG never forwards a worker's X-SMG-Error-Code header, so that header always means SMG itself produced the error.

Error Codes

Status code Returned when
400 invalid_json The body is not valid JSON
400 missing_required_parameter, unknown_parameter, invalid_type, invalid_value, integer_below_min_value The body does not match the request schema; param names the field when one is known
400 integer_above_max_value, decimal_below_min_value, string_above_max_length, empty_array, ... A field is outside its documented bounds; the code names the kind of value and the bound crossed, param the field
400 The rule's code A cross-field rule (stream_options_requires_stream for stream_options without stream, top_logprobs_requires_logprobs, tool_choice_function_not_found for a tool_choice naming a missing tool, json_schema_schema_not_object for a non-object json_schema, ...) or a provider-profile rule failed
400 json_parse_error The Content-Type: application/json header is missing, or the body failed to parse for a reason outside the set above
400 context_length_exceeded gRPC: the longest prompt is longer than the context window the selected worker advertises (the tighter leg governs in PD, and each prompt of a batched completion counts on its own). Workers that advertise no window are not checked
400 invalid_prompt_token gRPC: a completion's token-ID prompt holds an ID that is not a token of the model. The message names the position and value (prompt[2] = ..., or prompt[1][1] = ... in a batch); an invalid ID that only the decode catches is reported as a failed decode instead. Refused before any worker is called
400 unsupported_content_part gRPC: a content part SMG cannot render
400 audio_output_not_supported, unsupported_output_modality gRPC: return_audio: true or a non-text entry in modalities; gRPC workers produce text only
400 multimodal_not_supported gRPC: the selected engine does not accept one of the request's input modalities
Engine-mapped start_generation_failed gRPC: the engine did not start generation. The status follows the engine's gRPC status, for example invalid argument → 400 (not retried and not counted against the circuit breaker), resource exhausted → 429, unavailable → 503, deadline exceeded → 504, internal → 500
Engine status, or 502 prefill_worker_failed_to_start, decode_worker_failed_to_start PD on either transport: the prefill or decode leg never started. The engine refused it and its status is kept (over gRPC, mapped from the engine's gRPC status as above), or the call to its worker failed below HTTP (502)
Engine status, or 502 prefill_stream_error, decode_stream_error PD on either transport: the leg's stream ended with an error after it began, with the answer to the client still whole. Over gRPC the engine's status is kept (mapped as above); over HTTP the leg's body was cut after its 2xx head (502). A failure after the client's own stream has started ends it with a stream_error event instead (see Streaming Response)
500 pd_rendezvous_failed gRPC PD: the decode leg failed before producing anything, and the prefill's bootstrap port, probed now, no longer accepts connections — the sides could not rendezvous. The message says the port did not accept a connection, with the probe's error; the gateway log names the port's address
401 invalid_api_key Missing or unknown API key. The response adds a WWW-Authenticate: Bearer challenge (error="invalid_token" when a key was presented)
404 model_not_found No worker serves the model
408 request_body_stalled The request body stalled for --stream-body-stall-timeout-secs while being read, whether it is buffered or streamed to the worker
408 request_body_timeout The request body was still incomplete at --request-timeout-secs, whether it is buffered or streamed to the worker
413 request_body_too_large The request body exceeds --max-payload-size, however it arrives: a declared Content-Length over the limit (refused before routing), a buffered or chunked body that crosses it while being read, or a streamed body that grows past it mid-upload
429 admission_queue_full, scheduler_queue_full, tenant_rate_limit_exceeded, pd_prefill_queue_full, pd_prefill_queue_timeout Admission control or rate limiting; see Rate Limiting
500 internal_error and stage-specific codes A failure inside SMG; the code names the failing step
502 upstream_response_too_large A worker's non-streaming response exceeded --max-payload-size
503 no_available_workers Workers serve the model, but every one is unhealthy or has an open circuit breaker, or the routing policy selects none of them (HTTP and gRPC paths)
503 no_models_available GET /v1/models: no self-hosted worker serves any model (see List Models)
503 worker_overload_protection_shed, admission_queue_timeout, selection_queue_timeout, scheduler_queue_timeout, scheduler_preempted Overload and admission sheds; see Rate Limiting
504 prefill_worker_silent, decode_worker_silent gRPC PD: the leg produced nothing within its liveness bound. Retryable, and the retry window closes on that worker alone — its partner stays selectable

On gRPC workers, an engine finish reason of error (outside the OpenAI set) becomes an error response or stream error; clients never receive "finish_reason": "error".


Client Libraries

Python (OpenAI SDK)

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:30000/v1",
    api_key="your-api-key"  # or "not-needed" if auth disabled
)

response = client.chat.completions.create(
    model="meta-llama/Llama-3.1-8B-Instruct",
    messages=[
        {"role": "user", "content": "Hello!"}
    ]
)

print(response.choices[0].message.content)

JavaScript/TypeScript

import OpenAI from 'openai';

const client = new OpenAI({
  baseURL: 'http://localhost:30000/v1',
  apiKey: 'your-api-key'
});

const response = await client.chat.completions.create({
  model: 'meta-llama/Llama-3.1-8B-Instruct',
  messages: [
    { role: 'user', content: 'Hello!' }
  ]
});

console.log(response.choices[0].message.content);

cURL

curl http://localhost:30000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer your-api-key" \
  -d '{
    "model": "meta-llama/Llama-3.1-8B-Instruct",
    "messages": [{"role": "user", "content": "Hello!"}]
  }'

Request Headers

Header names are case-insensitive. SMG reads the following request headers:

Header Purpose Documented in
Content-Type application/json, or multipart/form-data for /v1/audio/transcriptions —
Authorization Bearer <key>; required when --api-key or --tenant-api-key is set Authentication
x-api-key, x-goog-api-key The caller's own key for Anthropic (x-api-key) and Gemini (x-goog-api-key) provider workers. /v1/responses and /v1/interactions prefer it over Authorization, and /v1/messages forwards x-api-key as sent; /v1/chat/completions uses only Authorization External Providers
X-Request-ID Request ID for log correlation. SMG checks x-request-id, x-correlation-id, x-trace-id, and request-id in that order, and generates an ID when none is present. --request-id-headers replaces the list Logging
traceparent, tracestate W3C trace context. With --enable-trace, SMG continues the caller's trace and propagates it to workers Monitoring
X-SMG-Routing-Key Routing key for sticky sessions and the key-based policies (manual, consistent_hashing, prefix_hash). It is the default name in --routing-key-headers Sticky Sessions and Routing Keys
X-SMG-Routing-Tokens Routing hint carrying the prompt's leading token IDs Routing Hint Headers
X-SMG-Target-Worker 0-based index of the worker to use among the model's available workers. Honored only by consistent_hashing; an index with no available worker fails with 503 Load Balancing
X-SMG-Priority Priority class: system, interactive, default, or bulk. Read only when the priority scheduler is enabled Priority Scheduler
X-SMG-Tenant-ID Tenant identity asserted by a trusted upstream, resolved as tenant key header:<value>. Read only with --trust-tenant-header, and only when API-key auth has not already identified the caller. --tenant-header-name renames it Tenant Rate Limiting
X-SMG-MCP enabled makes SMG run the MCP tool loop itself for /v1/messages requests with MCP toolsets sent to an Anthropic provider, instead of forwarding them as-is Messages API
X-SMG-Parser legacy or symphony: which parser reads this request's output, overriding --parser for the one request. Read on Chat Completions, Messages API, and Responses API requests served through the gRPC pipeline (gRPC and ZMQ workers); any other value is rejected with 400 invalid_parser_header gRPC Pipeline
x-smg-cache-trace 1 asks for this request's routing-decision record in the x-smg-cache-trace response header, on requests served through the gRPC pipeline, whatever the SMG_CACHE_TRACE_* variables say Routing Trace Variables
Names set with --storage-context-headers Copied into the storage hook request context Configuration Reference

When SMG proxies a request to an HTTP worker, it forwards only these client headers: Authorization, anthropic-version, anthropic-beta, X-Request-ID, X-Correlation-ID, headers starting with X-Request-ID-, traceparent, tracestate, X-SMG-Routing-Key, and x-override-priority (SGLang's request priority, read by an engine started with SGLANG_ENABLE_REQUEST_HEADER_OVERRIDES=true and not by SMG).

Routing Hint Headers

A client, or a proxy in front of SMG that has already tokenized the prompt, can pass the routing signal in a header so SMG does not need the request body to choose a worker. Hints affect only worker placement. A malformed or over-limit hint is ignored and the request routes as if the header were absent; it never causes an error.

Header Value Limits Effect
X-SMG-Routing-Tokens The prompt's leading token IDs as comma-separated decimal integers from 0 to 4294967295, with no spaces, signs, or empty items At most 512 IDs and 4096 bytes Replaces the token IDs and text SMG would take from the body when choosing a worker. cache_aware matches the hinted prefix against its token tree (its placement index with --cache-index hash) and records it there; prefix_hash hashes it. Read on the regular HTTP path only (not in PD mode or for gRPC workers), and never forwarded to workers
X-SMG-Routing-Key An opaque key Non-empty UTF-8, at most 128 bytes Without --routing-key-override, consistent_hashing and prefix_hash hash the key instead of their other inputs, and manual pins it. With the override, every policy except consistent_hashing pins the key
# X-SMG-Routing-Tokens carries the prompt's first token IDs
curl http://localhost:30000/v1/completions \
  -H "Content-Type: application/json" \
  -H "X-SMG-Routing-Tokens: 128000,791,4062,14198,39935" \
  -d @request.json

With a valid X-SMG-Routing-Tokens hint, the text-routing policies (cache_aware, bucket) no longer need the body to place the request, so a large body can stream to the worker instead of being buffered; see Request Streaming. When --routing-key-override is enabled, SMG always buffers the body to read its rid, and hints do not change that.

Response Headers

Header When SMG sets it Documented in
x-request-id On API responses: the request ID SMG used, whether received or generated Logging
x-smg-routed-worker-id On inference responses from HTTP, gRPC, and ZMQ workers: the registered URL of the worker that served the request, with an @<rank> suffix for data-parallel workers. For a prefill/decode pair it names the decode worker. From gRPC and ZMQ workers, only successful responses carry it, and Responses API responses do not Sticky Sessions and Routing Keys
x-smg-parser On successful Chat Completions, Messages API, and Responses API responses from gRPC and ZMQ workers: the parser that read the output, symphony or legacy. Responses served by the separate gpt-oss (Harmony) pipeline, and the queued object a non-streaming background: true Responses request returns, carry no header gRPC Pipeline
x-smg-cache-trace On successful responses from gRPC and ZMQ workers when the request sent x-smg-cache-trace: 1 or the gateway runs with SMG_CACHE_TRACE_HEADER=1: the dispatch's routing-decision record. Omitted when the record exceeds 2048 bytes or is not plain ASCII Routing Trace Variables
X-SMG-Error-Code On errors the gateway generates in its standard error envelope, with the same value as error.code; request-validation errors do not carry it. SMG drops this header from worker responses, so it always marks a gateway decision —
X-SMG-Preempted true on the 503 returned to a request preempted by the priority scheduler Priority Scheduler
Retry-After On 429 and 503 responses that ask the client to back off: concurrency limits, worker overload, the priority scheduler, tenant rate limits, and the 503 answered when no worker is available (Retry-After: 1) Rate Limiting

Rate Limiting

SMG does not send X-RateLimit-* headers. A request shed by admission control, rate limiting, or overload protection gets 429 or 503 with the standard error envelope, the X-SMG-Error-Code header, and usually a Retry-After header in seconds:

code Status Retry-After Cause
admission_queue_full 429 2 --max-concurrent-requests is reached and the admission queue is full or disabled
admission_queue_timeout 503 2 The request waited in the admission queue longer than --queue-timeout-secs
selection_queue_timeout 503 2 gRPC and ZMQ workers: the request had already waited longer than --queue-timeout-secs inside the gateway when it reached worker selection (the prefill-admission wait of a PD or EPD request included)
scheduler_queue_full 429 2 Priority scheduler: the request's class queue is full
scheduler_queue_timeout 503 2 Priority scheduler: the request waited longer than its class's queue timeout
scheduler_preempted 503 1 Priority scheduler: preempted by higher-priority traffic before its first token; also sets X-SMG-Preempted: true
tenant_rate_limit_exceeded 429 Seconds until the tenant's budget can admit the request; omitted when it never can Tenant rate limiting
pd_prefill_queue_full 429 — PD and EPD: the opt-in prefill admission queue is full
pd_prefill_queue_timeout 429 — PD and EPD: the request waited out --prefill-queue-timeout-secs without a prefill slot
worker_overload_protection_shed 503 --load-monitor-interval (default 10) With --worker-overload-shed (off by default, see Overload Protection): every eligible worker is over its overload threshold, or the selected one crossed it just before dispatch. Also, with or without --worker-overload-shed, on a gRPC PD or EPD dispatch whose decode worker could not admit the request within --pd-admission-wait-secs; HTTP PD dispatches are not gated by the decode admission window

Wait at least Retry-After seconds before retrying. For configuration, see Rate Limiting, Tenant Rate Limiting, Priority Scheduling, and Overload Protection.