OpenAI-Compatible API Reference¶
SMG provides a fully OpenAI-compatible API, allowing you to use existing OpenAI client libraries with your self-hosted inference workers.
Base URL¶
http://localhost:30000/v1Authentication¶
SMG supports optional API key authentication:
curl http://localhost:30000/v1/chat/completions \
-H "Authorization: Bearer your-api-key" \
-H "Content-Type: application/json" \
-d '...'Enable authentication with --api-key:
smg launch --worker-urls http://worker:8000 --api-key "your-api-key"For real multi-tenant separation (e.g. per-tenant rate limiting), configure one key per
tenant instead with --tenant-api-key tenant_id:key (repeatable). Each key resolves to
its own tenant identity; --api-key remains available as a single shared fallback key
whose callers all share one identity.
smg launch --worker-urls http://worker:8000 \
--tenant-api-key team-red:red-secret \
--tenant-api-key team-blue:blue-secretCallers authenticate the same way as with --api-key — a Bearer token in the
Authorization header — just using their own tenant's key instead of the shared one:
curl http://localhost:30000/v1/chat/completions \
-H "Authorization: Bearer red-secret" \
-H "Content-Type: application/json" \
-d '{
"model": "meta-llama/Llama-3.1-8B-Instruct",
"messages": [{"role": "user", "content": "Hello"}]
}'This request is attributed to team-red, distinct from a request authenticated with
blue-secret. A missing key, or one that doesn't match any configured --api-key or
--tenant-api-key value, is rejected with 401 Unauthorized in the standard
error envelope with the code invalid_api_key, plus a
WWW-Authenticate: Bearer challenge (error="invalid_token" when a key was presented
but not recognized). Tenant
keys authenticate serving endpoints only — /v1/chat/completions, /v1/completions,
/v1/embeddings, etc. They do not grant access to admin/management routes (/workers,
/flush_cache, and similar). When control-plane authentication is enabled, those
routes require a control-plane credential: the admin role may use every method,
the user role only GET and HEAD. Otherwise, they
accept the shared --api-key if configured. With neither control-plane auth nor any API keys configured,
those routes are open; with only --tenant-api-key configured, every admin request
is rejected with 401.
The model routes — GET /v1/models and GET /v1/models/{model} — are public and answer
without a key (see List Models and Retrieve Model).
Endpoints¶
Chat Completions¶
Create a chat completion.
POST /v1/chat/completionsRequest Body¶
| Field | Type | Required | Description |
|---|---|---|---|
model |
string | Yes | Model identifier. A name configured with --model-alias is accepted and resolved to its canonical model |
messages |
array | Yes | Array of message objects; must not be empty |
max_completion_tokens |
integer | No | Upper bound on generated completion tokens (at least 1) |
max_tokens |
integer | No | Deprecated — use max_completion_tokens. Still accepted: it is the cap when max_completion_tokens is unset, and a proxied body reaches the worker with the field as the client wrote it (not rewritten). OpenAI-named models reject the two limits together (see Validation Rules) |
temperature |
number | No | Sampling temperature (0-2) |
top_p |
number | No | Nucleus sampling parameter, greater than 0 and at most 1 |
n |
integer | No | Number of completions to generate (1-128) |
stream |
boolean | No | Enable streaming responses |
stream_options |
object | No | Streaming options; only allowed with stream: true. See Stream Usage |
stop |
string/array | No | Up to 4 non-empty stop sequences |
presence_penalty |
number | No | Presence penalty (-2 to 2) |
frequency_penalty |
number | No | Frequency penalty (-2 to 2) |
logprobs |
boolean | No | Return log probabilities of the output tokens |
top_logprobs |
integer | No | Most likely tokens to return per position (0-20); requires logprobs: true |
logit_bias |
object | No | Map of token ID to bias |
tools |
array | No | Function tools the model may call |
tool_choice |
string/object | No | none, auto, required, {"type": "function", "function": {"name": "..."}}, or {"type": "allowed_tools", "mode": "auto", "tools": [...]} (mode is auto or required). Every value except none and auto requires tools, and a named function must exist in them |
parallel_tool_calls |
boolean | No | Allow parallel function calls |
response_format |
object | No | text, json_object, or json_schema. See Structured Output |
reasoning_effort |
string/number | No | Reasoning effort; a number is converted to its decimal string. See Thinking and Reasoning Effort |
thinking |
object | No | Typed thinking control (type, effort, keep, clear_thinking). See Thinking and Reasoning Effort |
verbosity |
string | No | low, medium, or high; any other value is rejected |
user |
string | No | End-user identifier |
Extension Fields¶
SMG also accepts these SGLang-style extensions on Chat Completions:
| Field | Type | Description |
|---|---|---|
top_k |
integer | Top-k sampling; -1 disables it, otherwise at least 1 |
min_p |
number | Min-p sampling threshold (0-1) |
min_tokens |
integer | Minimum tokens to generate; must not exceed max_completion_tokens |
repetition_penalty |
number | Repetition penalty (0-2) |
regex, ebnf |
string | Constrain the output. Only one of regex, ebnf, and a json_schema response format may be set, and neither combines with a JSON response_format |
stop_token_ids |
array | Token IDs that stop generation |
ignore_eos, no_stop_trim |
boolean | Keep generating past EOS; keep the stop sequence in the output |
skip_special_tokens |
boolean | Drop special tokens when detokenizing (default true) |
continue_final_message |
boolean | Continue the trailing assistant message instead of starting a new one |
add_generation_prompt |
boolean | Whether the prompt ends with the assistant's generation header; absent reads true. false renders the conversation without it, leaving the model to continue on its own. Set to true beside continue_final_message it is rejected with 400 add_generation_prompt_conflicts_with_continue_final_message |
separate_reasoning, stream_reasoning |
boolean | Return reasoning in reasoning_content and stream it (both default true) |
chat_template_kwargs |
object | Extra chat-template variables. On gRPC workers an explicit entry wins over the tool_choice and response_format values SMG passes to the template, but not over the effort: a reasoning_effort entry applies only when the request sets no effort of its own (see Thinking and Reasoning Effort) |
lora_path |
string | LoRA adapter to apply |
sampling_seed |
integer | Seed for deterministic sampling |
return_hidden_states |
boolean | Return the model's hidden states |
rid |
string | Request ID forwarded to the backend |
Fields SMG does not model are kept and forwarded unchanged to HTTP workers, and so are
unknown stream_options keys such as SGLang's step_usage_chunks. The SGLang-native
/generate endpoint likewise keeps its nested sampling_params.custom_params object.
Typed sampling numbers are forwarded without float widening: "top_p": 0.95 reaches
HTTP workers, PD legs, and external providers as 0.95, not 0.949999988079071.
Message Object¶
| Field | Type | Required | Description |
|---|---|---|---|
role |
string | Yes | system, developer, user, assistant, tool, function, or root (MiniMax only; see Provider Profiles) |
content |
string/array | Yes | Text, or an array of content parts. Optional on system and assistant messages; required on user messages, where an empty string or array is accepted; a plain string or null on function messages (see below) |
name |
string | No | Participant name |
tool_calls |
array | No | Tool calls made by an assistant message |
tool_call_id |
string | For tool |
ID of the tool call this tool message answers |
reasoning_content |
string | No | Reasoning from an earlier assistant turn; reasoning is accepted as an alias |
tools |
array | No | Kimi only: tools declared on a system or developer message |
On a function message (the deprecated predecessor of tool), content is required but
nullable, and a null is kept rather than dropped: a body forwarded to an HTTP worker or
external provider still carries "content": null, and on gRPC workers the chat template
receives it as empty content in the template's own format (an empty string, or an empty
array of parts).
Content Parts¶
type |
Payload | Notes |
|---|---|---|
text |
text |
|
image_url |
image_url.url, optional detail |
Optional image_url.max_long_side_pixel caps the image's long side: a MiniMax-M3 extension that SMG's own media processing applies for any model (see Frame Sampling) |
video_url |
video_url.url |
Optional video_url.fps and video_url.max_long_side_pixel set the frame sampling rate and frame size: MiniMax-M3 extensions that SMG's own media processing applies for any model (see Frame Sampling) |
input_audio |
input_audio.data (base64), input_audio.format |
|
audio_url |
audio_url.url |
|
| Anything else | Forwarded as sent | HTTP workers only |
A content part whose type SMG does not model, or a known type with a malformed payload,
is forwarded unchanged to HTTP workers. gRPC workers reject it with 400
unsupported_content_part, and the gpt-oss (Harmony) path with 400
harmony_build_failed. Which workers accept which media is covered in
Multimodal.
Thinking and Reasoning Effort¶
Two fields control reasoning. reasoning_effort is OpenAI's effort level. thinking is
a typed object shared by the Kimi, MiniMax, z.ai, and DeepSeek dialects:
| Field | Values | Meaning |
|---|---|---|
type |
enabled, disabled, adaptive |
Thinking toggle. adaptive states no preference; any other value is rejected |
effort |
string | Effort level; takes precedence over reasoning_effort |
keep |
string | Kimi keep mode; accepted and ignored by SMG's renderers |
clear_thinking |
boolean | z.ai: whether earlier turns' reasoning is dropped from the prompt |
The effective effort is thinking.effort when present, otherwise reasoning_effort.
Leaving thinking out changes nothing: the chat template's own default applies (Kimi K3,
for example, thinks by default), subject to reasoning_effort. The exception is the
DeepSeek V4 profile, which turns thinking on unless the request turns it off (see
Defaults and Rewrites). A thinking object without type
expresses no toggle, but its effort still counts.
On gRPC workers SMG renders the chat template itself. The template receives the effective
effort as its reasoning_effort variable, and SMG decides whether the model starts in
reasoning mode from the first of these that applies:
- an explicit thinking toggle in
chat_template_kwargs; - for templates with their own effort levels, the effort the template renders (the request's effective effort, else a
chat_template_kwargs.reasoning_effortentry): a native level turns thinking on, andnoneandminimalturn it off. Three exceptions: the DeepSeek-V4 renderer keeps thinking on for every value butnone(a model the DeepSeek V4 profile matches never reaches this step: the profile resolves the toggle up front and writes it as the step-1 override, reading a kwargs-onlyreasoning_effortofnoneorminimalas off); Kimi-K3 lets onlynonedecide at this step, leaving every other value —minimaland native levels included — to the steps below; and Hunyuan 4 templates, whosereasoning_effortwords carry the toggle itself, are read by those words alone:no_thinkturns thinking off, and every other value —noneandminimalincluded — keeps thinking on; thinking.type(enabledon,disabledoff);- the effective effort, where
noneandminimalturn thinking off — except on MiniMax models, where no effort value does (MiniMax readsnoneas the model's default thinking depth andminimalaslow).
If none applies, the template default stands. The template and the reasoning parser use
the same decision. On gpt-oss (Harmony) models SMG reads reasoning_effort and clamps it to
the three levels Harmony knows: none and minimal become low, xhigh and max
become high, and an unknown value becomes medium. HTTP workers receive both fields as
sent unless a provider profile rewrites them, and provider profiles
restrict the accepted values per model family.
Structured Output¶
response_format accepts:
{"type": "text"}{"type": "json_object"}{"type": "json_schema", "json_schema": {"name": "...", "schema": {...}, "strict": true}}
For json_schema, name must not be empty and schema must be a JSON object ({} is
allowed). A string, array, boolean, number, or null schema is rejected with 400
json_schema_schema_not_object before the request reaches any worker, on every backend,
unless the HTTP router streams the request to the worker unparsed (see
Request Streaming).
Example Request¶
curl http://localhost:30000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "meta-llama/Llama-3.1-8B-Instruct",
"messages": [
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": "What is the capital of France?"}
],
"max_tokens": 100,
"temperature": 0.7
}'Response¶
{
"id": "chatcmpl-abc123",
"object": "chat.completion",
"created": 1705312345,
"model": "meta-llama/Llama-3.1-8B-Instruct",
"choices": [
{
"index": 0,
"message": {
"role": "assistant",
"content": "The capital of France is Paris."
},
"finish_reason": "stop"
}
],
"usage": {
"prompt_tokens": 25,
"completion_tokens": 8,
"total_tokens": 33
}
}Usage¶
| Field | Description |
|---|---|
prompt_tokens, completion_tokens, total_tokens |
Token counts. With n > 1 the shared prompt is counted once and the choices' completions are summed |
prompt_tokens_details.cached_tokens |
Prompt tokens served from the worker's prefix cache |
completion_tokens_details.reasoning_tokens |
Reasoning tokens: a chat completion always carries the key (0 without reasoning), while the legacy Completions endpoint includes it only when the count is non-zero. On gRPC workers SGLang reports them, and SMG counts them itself for gpt-oss (Harmony) models |
completion_tokens_details.accepted_prediction_tokens |
Speculative decoding: draft tokens the target model accepted |
completion_tokens_details.rejected_prediction_tokens |
Speculative decoding: draft tokens rejected (drafted minus accepted). Omitted when the drafted count is unknown — see below |
On gRPC workers SMG builds usage from the engine's counters; HTTP workers return the
engine's own usage object. SMG reads the speculative counters from SGLang, TokenSpeed,
vLLM (started with --per-request-spec-decode-metrics), and TensorRT-LLM (once its
servicer fills the fields); MLX reports none.
An engine does not always report a usable drafted count next to the tokens it accepted,
so the two speculative fields follow different rules. accepted_prediction_tokens
carries the accepted count whenever the engine reports speculative activity.
rejected_prediction_tokens is drafted minus accepted, reported only when every choice
supplied a positive drafted count at least as large as its accepted count. If any choice
pairs measured speculation with a zero drafted count, or reports fewer drafted than
accepted tokens, the rejection count is unknown and the key is omitted rather than
reported as 0.
Without any speculative decoding a chat completion reports both fields as 0; the
legacy Completions endpoint includes them only with speculative activity.
A request's streamed and non-streamed answers carry the same usage object. The running
snapshots requested with continuous_usage_stats follow the same
rules: a choice whose final counters are still outstanding counts as a zero drafted
count, so measured speculation on another choice leaves the rejection count unknown.
Streaming Response¶
With "stream": true, responses are sent as Server-Sent Events:
curl http://localhost:30000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "meta-llama/Llama-3.1-8B-Instruct",
"messages": [{"role": "user", "content": "Hello"}],
"stream": true,
"stream_options": {"include_usage": true}
}'Response (abbreviated):
data: {"id":"chatcmpl-abc123","object":"chat.completion.chunk","created":1705312345,"model":"meta-llama/Llama-3.1-8B-Instruct","choices":[{"index":0,"delta":{"role":"assistant"},"finish_reason":null}]}
data: {"id":"chatcmpl-abc123","object":"chat.completion.chunk","created":1705312345,"model":"meta-llama/Llama-3.1-8B-Instruct","choices":[{"index":0,"delta":{"content":"Hello!"},"finish_reason":null}]}
data: {"id":"chatcmpl-abc123","object":"chat.completion.chunk","created":1705312345,"model":"meta-llama/Llama-3.1-8B-Instruct","choices":[{"index":0,"delta":{},"finish_reason":"stop"}]}
data: {"id":"chatcmpl-abc123","object":"chat.completion.chunk","created":1705312345,"model":"meta-llama/Llama-3.1-8B-Instruct","choices":[],"usage":{"prompt_tokens":9,"completion_tokens":2,"total_tokens":11}}
data: [DONE]An error before the first chunk is an ordinary JSON error response with its own status;
when a worker rejects a streaming request, SMG keeps the worker's content type
(application/json, not text/event-stream) so SSE clients still see the error body.
If generation fails after a gRPC stream has started, SMG sends
data: {"error": {"message": "...", "type": "..."}} followed by data: [DONE]. The
type is stream_error when the worker fails the stream: its connection breaks, its
server cuts the stream, it answers a server error or a timeout after the first chunk,
or the engine itself reports that the request failed. Any other status the engine
answers mid-stream keeps the type it would carry in an error body (see
Error Format): for example invalid_request_error when the engine
rejects the request after its first chunk, or rate_limit_error for capacity pushback.
internal_error means the failure was in SMG's own processing.
A quiet open SSE response can contain : ping comments between events. SMG
sends them after 15 seconds of silence by default, including while waiting for
the first token. SSE clients should ignore comment lines. Configure or disable
them with --sse-keepalive-secs.
Stream Usage¶
stream_options field |
Default | Effect |
|---|---|---|
include_usage |
false (Kimi, MiniMax: true) |
Send the whole request's usage in a final chunk with an empty choices array |
continuous_usage_stats |
false |
gRPC workers, except for gpt-oss (Harmony) models: with include_usage: true, every chunk also carries a running usage snapshot (the prompt plus the tokens generated so far) |
On HTTP workers stream_options, including keys SMG does not model, is forwarded to the
engine, which produces the usage chunks.
The Kimi and MiniMax profiles turn include_usage on for streaming requests that leave
it unset, on every backend; an explicit false is kept.
The DeepSeek V4 profile uses DeepSeek's stream format on gRPC workers instead: aggregate
usage rides on the final finish chunk whether or not include_usage is set, and no
usage-only chunk follows. With include_usage: true, earlier chunks carry
"usage": null. continuous_usage_stats does not apply, and if the backend stream ends
before every choice completes, the finish chunk carries no aggregate usage.
Completions¶
Create a text completion (legacy API).
POST /v1/completionsRequest Body¶
| Field | Type | Required | Description |
|---|---|---|---|
model |
string | Yes | Model identifier |
prompt |
string/array | Yes | A string, an array of strings, an array of token IDs, or an array of token-ID arrays; an array must be non-empty |
max_tokens |
integer | No | Maximum tokens to generate |
temperature |
number | No | Sampling temperature (0-2) |
top_p |
number | No | Nucleus sampling parameter, greater than 0 and at most 1 |
n |
integer | No | Completions per prompt (1-128) |
best_of |
integer | No | Candidates to generate server-side (0-20); must be greater than n and cannot be combined with stream |
logprobs |
integer | No | Log probabilities of the most likely tokens (0-20, the bound of chat's top_logprobs; the public API's own limit of 5 is not imposed) |
stream |
boolean | No | Enable streaming |
stream_options |
object | No | include_usage; only allowed with stream: true |
stop |
string/array | No | Up to 4 non-empty stop sequences |
echo |
boolean | No | Echo prompt in response |
suffix |
string | No | Text that comes after the completion |
presence_penalty, frequency_penalty |
number | No | Penalties (-2 to 2) |
seed |
integer | No | Seed for best-effort deterministic sampling |
user |
string | No | End-user identifier |
Completions takes the same extension fields as Chat Completions except
the chat-only ones (continue_final_message, add_generation_prompt, separate_reasoning,
stream_reasoning, chat_template_kwargs), plus a json_schema string constraint. On gRPC workers each
prompt in an array is its own engine request.
A prompt can arrive already tokenized: an array of token IDs (unsigned 32-bit integers,
such as those /v1/tokenize returns), or an
array of token-ID arrays for a batch. The first element of an array fixes its shape — a string
makes it an array of text prompts, an integer an array of token IDs, an array a batch of
token-ID prompts. A mixed array, a negative, fractional, or out-of-range ID, an empty
prompt array, and an empty ID list in a batch are all rejected.
On gRPC workers token IDs bypass the gateway's tokenizer and reach the engine exactly as
sent. usage.prompt_tokens counts the IDs, and with echo: true each choice opens with
the decoded text of its prompt's IDs (special tokens included; an ID the tokenizer has no
text for adds none). On that path every ID must be a token of the model — in its
tokenizer's vocabulary, added tokens included, or within the embedding table its workers
report — otherwise the request is rejected with invalid_prompt_token before any worker
is called (see Error Codes). On HTTP workers the prompt is forwarded as
sent and the engine applies its own handling.
Example Request¶
curl http://localhost:30000/v1/completions \
-H "Content-Type: application/json" \
-d '{
"model": "meta-llama/Llama-3.1-8B-Instruct",
"prompt": "The quick brown fox",
"max_tokens": 50
}'Response¶
{
"id": "cmpl-abc123",
"object": "text_completion",
"created": 1705312345,
"model": "meta-llama/Llama-3.1-8B-Instruct",
"choices": [
{
"text": " jumps over the lazy dog.",
"index": 0,
"finish_reason": "stop"
}
],
"usage": {
"prompt_tokens": 4,
"completion_tokens": 7,
"total_tokens": 11
}
}Decisions¶
Ask predicate (yes/no), choice, and score questions about shared evidence. The model scores each question's possible answers instead of generating text, and every answer carries a probability distribution.
POST /v1/decisionsRequests are always treated as non-streaming: there is no stream field, and SMG returns
a single JSON body.
Request Body¶
| Field | Type | Required | Description |
|---|---|---|---|
model |
string | Yes | Model identifier. A name configured with --model-alias is accepted and resolved to its canonical model |
input |
string/array | Yes | The evidence every question is answered against: a plain string, or an array of user messages (see Decision Input) |
questions |
array | Yes | Questions to answer; answers come back in the same order (see Decision Questions) |
safety_identifier |
string | No | Opaque end-user identifier, at most 128 characters |
Fields SMG does not model — on the request and on its messages, parts, questions, choices,
and levels — are kept as backend extensions: they are forwarded unchanged to non-SGLang
HTTP workers and rejected with 400 on SGLang workers (see Decision Workers).
Decision Input¶
As an array, each input message has:
| Field | Type | Required | Description |
|---|---|---|---|
role |
string | Yes | Must be user |
type |
string | No | message, when present |
content |
string/array | Yes | Text, or an array of input_text and input_image parts |
Part type |
Payload | Limits |
|---|---|---|
input_text |
text |
At most 10,485,760 characters per part |
input_image |
image_url, optional detail (low, high, auto, original) |
Must be an inline base64 image data URL (data:image/png;base64,...); external URLs and file IDs are rejected. At most 128 images per request |
model, instructions, name, description, and label values each take at most
1,048,576 characters. A limit violation is rejected with 400 before routing.
Decision Questions¶
Every question has a type, required instructions, and an optional name. Names need
not be unique: each answer repeats its question's name (null when unset), and answers
keep question order.
type |
Extra fields | The model estimates |
|---|---|---|
predicate |
— | The probability that the statement in instructions is true |
choice |
choices: array of {"value": ...} with an optional description; value is a string or boolean, and true is distinct from "true" |
A probability for each choice |
score |
levels: array of {"label": "..."} with an optional description, ordered from lowest to highest |
A probability for each level |
Example Request¶
curl http://localhost:30000/v1/decisions \
-H "Content-Type: application/json" \
-d '{
"model": "Qwen/Qwen3-4B-Instruct-2507",
"input": "The delivery arrived a day late, but the package was intact and support was friendly.",
"questions": [
{"type": "predicate", "instructions": "Did the delivery arrive on time?"},
{"type": "choice", "name": "sentiment", "instructions": "What is the overall sentiment?",
"choices": [{"value": "positive"}, {"value": "negative"}]},
{"type": "score", "instructions": "Rate the customer experience.",
"levels": [{"label": "poor"}, {"label": "fair"}, {"label": "good"}]}
]
}'Response¶
{
"model": "Qwen/Qwen3-4B-Instruct-2507",
"answers": [
{"type": "predicate", "name": null, "probability": 0.11},
{"type": "choice", "name": "sentiment", "choice": "positive", "confidence": 0.74,
"probabilities": [
{"value": "positive", "probability": 0.87},
{"value": "negative", "probability": 0.13}
]},
{"type": "score", "name": null, "score": 1.62, "confidence": 0.55,
"probabilities": [
{"value": 0, "label": "poor", "probability": 0.08},
{"value": 1, "label": "fair", "probability": 0.22},
{"value": 2, "label": "good", "probability": 0.70}
]}
],
"usage": {
"input_tokens": 182,
"output_tokens": 0,
"total_tokens": 182,
"input_tokens_details": {"cached_tokens": 0, "cache_write_tokens": 0},
"output_tokens_details": {"reasoning_tokens": 0}
}
}Each answer carries its question's type and name plus:
type |
Fields |
|---|---|
predicate |
probability (0–1) that the statement is true |
choice |
choice — the value of the most probable choice; confidence (0–1); probabilities — one {value, probability} entry per choice, in request order |
score |
score — the probability-weighted average of the level indices, not a winning level (level i in request order has value: i, so 3 levels give a score between 0 and 2); confidence (0–1); probabilities — one {value, label, probability} entry per level |
refusal |
No further fields: the worker declined to answer the question. SMG's own SGLang scoring always produces one of the three types above |
usage reflects scoring work: on SGLang workers (both the HTTP translation and gRPC
scoring) the answer labels are scored without generating tokens, so output_tokens and
output_tokens_details.reasoning_tokens are 0 and total_tokens equals input_tokens.
On gRPC workers each question is scored as its own prompt over the shared input —
input_tokens sums those prompts — and input_tokens_details.cached_tokens reports the
engine's prefix-cache hits. The SGLang HTTP translation returns 0 for both cache
counters, meaning unreported rather than measured. Non-SGLang HTTP workers return their
own usage object.
Decision Workers¶
What serves the request depends on the worker:
| Worker | Behavior |
|---|---|
| HTTP (SGLang runtime) | SMG translates the request to SGLang's native System One API (POST /v1/systemone, as served by SGLang 0.5.21) and maps the reply back to the Decisions shape |
| HTTP (other runtimes) | The request is forwarded to the worker's own /v1/decisions, unknown fields included; the worker's response — errors included — is returned as is |
| gRPC (regular mode, SGLang runtime) | SMG renders one prompt per question with the model's chat template and scores the answer labels through the engine, without generation |
| PD, EPD, ZMQ, and external providers | Not implemented (501 decisions_not_supported) |
In a mixed fleet, SMG dispatches Decisions only to regular HTTP workers and regular SGLang gRPC workers serving the model.
The SGLang paths (HTTP translation and gRPC scoring) reject these requests with 400
unsupported_decisions_request before dispatch:
input_imageparts — SGLang Decisions takes text evidence only- unknown fields anywhere on the request — SGLang Decisions cannot honor them
- empty or whitespace-only text evidence or
instructions - no questions, a
choicequestion outside 1–255 choices, or ascorequestion outside 1–10 levels
gRPC scoring also depends on the model: the internal answer labels SMG scores (yes/no,
option letters, level digits — your own choice values and level labels are unrestricted)
must each be one distinct token after the chat prompt, and the model's chat template must
produce plain text without leaving a reasoning block open. A model that cannot satisfy
this answers 400.
Decision Errors¶
| Status | code |
Returned when |
|---|---|---|
| 400 | The failure's code (see Error Format) | The body is not valid JSON (invalid_json), does not match the schema (invalid_value for a non-user role or an unknown question type, invalid_type for a numeric choice value, ...), or fails a limit (invalid_value for a length limit, too_many_images, invalid_image_url); param names the offending field when one is known |
| 400 | unsupported_decisions_request |
An SGLang worker was selected and the request needs something SGLang Decisions does not support (see Decision Workers) |
| 404 | model_not_found |
No worker serves the model |
| 501 | decisions_not_supported |
No regular HTTP worker and no regular SGLang gRPC worker serves the model |
| 502 | invalid_decisions_response |
An SGLang worker (HTTP translation or gRPC scoring) reported success, but the result could not be mapped to a valid Decisions response |
Backend errors on the SGLang HTTP translation keep their status and OpenAI error body.
For tenant rate limiting on Decisions, see Tenant Rate Limiting.
List Models¶
List available models.
GET /v1/modelsExample Request¶
curl http://localhost:30000/v1/modelsResponse¶
{
"object": "list",
"data": [
{
"id": "meta-llama/Llama-3.1-8B-Instruct",
"object": "model",
"created": 0,
"owned_by": "self_hosted"
}
]
}The list covers the models served by self-hosted workers, each once under its canonical
ID, followed by one entry per alias (--model-alias names and model-card aliases
entries), so every name a client may send is one it can discover. An alias is listed under
the model the registry resolves it to and nowhere else: an alias two workers' cards declare
appears once, and a name that is also a registered model's ID is never listed as an alias.
created is always 0 and owned_by is self_hosted.
When no self-hosted worker serves any model, the endpoint returns 503 in the gateway's
JSON error envelope with the code no_models_available.
The endpoint needs no key. If the caller does send a bearer token (or an x-api-key
header) that is not one of the gateway's own keys, SMG treats it as the caller's own
provider key: it calls /v1/models on the registered external providers of the caller's
API family with that token and returns the first non-empty list, falling back to the
self-hosted list. A request carrying anthropic-version reaches the Anthropic providers
alone; every other request reaches the OpenAI-compatible ones — OpenAI, xAI, custom
providers, and external workers that name no provider — so a key presented in one
family's API never reaches another family's provider, and a Gemini provider is asked by
neither. A gateway key (--api-key or --tenant-api-key) is never forwarded to a
provider.
On that provider list, owned_by is the provider name for the recognized provider hosts
(openai, anthropic, xai, gemini) and self_hosted for a provider at any other
host.
A request carrying an anthropic-version header gets its inventory in the Anthropic
Messages shape instead, with that API's paging read from the query: limit (1 to 1000,
default 20), one cursor (after_id or before_id, not both), and a lifecycle filter
(active, deprecated, or retired; every self-hosted model is active). The page
carries has_more, first_id, and last_id, and its data entries type, id,
display_name (the model ID when no display name is known), created_at
(1970-01-01T00:00:00Z, the counterpart of created: 0), lifecycle, and the model
card's context length as max_tokens and max_input_tokens. An out-of-range limit,
both cursors at once, an unknown lifecycle, or a cursor naming no entry is a 400
invalid_query_parameter. Errors then come in the Messages envelope (top-level
type: "error" and a request_id). The same header selects the shape on
Retrieve Model. On the provider fallback, a Messages-shaped caller's
upstream is asked for the page its query names and an Anthropic-shaped page is relayed
as the upstream served it; an OpenAI-shaped inventory is paged at the gateway in id
order.
Retrieve Model¶
Fetch one model's card.
GET /v1/models/{model}Like List Models, this is a public route and answers without a key.
{model} takes the whole rest of the path, so an ID with a slash in it works raw or
percent-encoded: /v1/models/meta-llama/Llama-3.1-8B-Instruct and
/v1/models/meta-llama%2FLlama-3.1-8B-Instruct name the same model. An alias — a name
configured with --model-alias, or an aliases entry on a worker's model card (see the
Worker Spec) — resolves to its canonical model, and the response
carries the canonical ID. The route answers from the self-hosted inventory only; it never
asks an external provider.
Example Request¶
curl http://localhost:30000/v1/models/meta-llama/Llama-3.1-8B-InstructResponse¶
{
"id": "meta-llama/Llama-3.1-8B-Instruct",
"object": "model",
"created": 0,
"owned_by": "self_hosted"
}With an anthropic-version header, the response is the Anthropic model object (type,
id, display_name, created_at, lifecycle, and the card's context length as
max_tokens and max_input_tokens) instead.
An unknown ID gets the same 404 model_not_found as every other route, in the OpenAI
error envelope — or, under anthropic-version, in the Messages envelope
(top-level type: "error", error.type not_found_error, and a request_id), the same
refusal /v1/messages gives that model.
Audio Transcriptions¶
Transcribe an audio file (batch).
POST /v1/audio/transcriptionsThe request is multipart/form-data:
| Field | Required | Description |
|---|---|---|
file |
Yes | The audio file; must not be empty |
model |
Yes | Model identifier |
language |
No | Language of the audio |
prompt |
No | Text to guide the transcript |
response_format |
No | Output format (default json) |
temperature |
No | Sampling temperature (0.0-1.0) |
timestamp_granularities[] |
No | word or segment; repeat the field for both |
stream |
No | true/false (or 1/0) |
A malformed form, a missing or empty file, a missing model, an out-of-range
temperature, or an unparseable stream is rejected with 400 in the standard
error envelope before routing, with param naming the field when one
is known: missing_required_parameter for a missing model or file, invalid_value
for an empty file or a bad temperature or stream value, and invalid_multipart
for a form that cannot be read. Other form fields are ignored.
curl http://localhost:30000/v1/audio/transcriptions \
-F file=@audio.wav \
-F model=Qwen/Qwen3-ASR-1.7BWhat serves the request depends on the worker:
| Worker | Behavior |
|---|---|
| HTTP | The form is forwarded to the worker's /v1/audio/transcriptions; the worker's model and options apply |
| gRPC (regular mode) | SMG runs transcription as its own pipeline endpoint: it turns the audio into a chat turn, renders it with the model's chat template, and returns the transcript. Supported family: Qwen3-ASR |
| PD, EPD, and external providers | Not implemented (501) |
On gRPC, a model belongs to the Qwen3-ASR family when its ID, or a serving worker's model
ID or model-identity label, contains qwen3-asr or qwen3_asr (ignoring case). Audio
input over gRPC is currently accepted by TokenSpeed workers only; other gRPC runtimes
answer 400 multimodal_not_supported. The gRPC endpoint is whole-file only:
| Request | gRPC result |
|---|---|
| A model outside the supported families | 400 audio_transcription_model_not_supported |
stream: true |
400 streaming_transcription_not_supported |
Any timestamp_granularities |
400 transcription_timestamps_not_supported |
response_format of srt, vtt, or verbose_json |
400 unsupported_transcription_response_format |
A language outside the model's list (ISO code or English name, such as en or English) |
400 unsupported_transcription_language |
A prompt over 4096 bytes |
400 asr_prompt_too_long |
response_format: json returns {"text": "..."} and text returns the plain transcript.
Sampling is greedy unless temperature is set, and a given language pins the transcript
language.
Realtime API¶
SMG proxies the OpenAI Realtime API to a realtime-capable worker. Both the OpenAI router
(to an upstream provider) and the HTTP router (to a local worker labeled
realtime: "true")
support it. The worker must speak the OpenAI Realtime protocol — for local workers, for
example vLLM serving an ASR model with the realtime task. SMG relays frames as the worker
sent them; error frames are the one exception (see Upstream Errors).
| Endpoint | Transport | Purpose |
|---|---|---|
GET /v1/realtime |
WebSocket | Bidirectional realtime session (e.g. live streaming transcription) |
POST /v1/realtime/calls |
WebRTC (SDP) | Browser/WebRTC realtime session |
POST /v1/realtime/sessions |
HTTP | Create a realtime session |
POST /v1/realtime/client_secrets |
HTTP | Mint an ephemeral client secret |
POST /v1/realtime/transcription_sessions |
HTTP | Create a realtime transcription session |
GET /v1/realtime requires the model query parameter and answers 400 without it.
POST /v1/realtime/calls takes the model from the model query parameter for an
application/sdp body, or from the session JSON in a multipart/form-data body.
Upstream Errors¶
A refusal loses the account identifiers in a JSON error's error.message before it
reaches the caller: phrases naming an organization (organization org-…) or a project
(project proj_…) are removed, as on every provider relay (see
Provider Error Responses).
Both routers do this, so error frames from local realtime workers are sanitized too.
- A non-success answer to a realtime REST call or to the
/v1/realtime/callsSDP exchange is returned under the upstream's status and content type with its body sanitized this way. - On a WebSocket session,
errorevents andconversation.item.input_audio_transcription.failedevents are the only frames SMG rewrites, with the same rule; every other frame is relayed as it came, never re-serialized.
WebSocket example¶
# pip install websockets
import asyncio, websockets
async def main():
url = "ws://localhost:30000/v1/realtime?model=Qwen/Qwen3-ASR-1.7B"
headers = {"Authorization": "Bearer your-api-key"}
async with websockets.connect(url, additional_headers=headers) as ws:
# Send realtime events (session.update, input_audio_buffer.append, ...)
# and receive transcription/response events from the worker.
...
asyncio.run(main())Provider Profiles¶
Some model families come with a vendor contract that differs from the OpenAI baseline.
SMG picks a provider profile from the request's model and applies that profile's
defaults and validation to Chat Completions requests before routing, on every backend.
Profile Selection¶
A profile matches when any /-separated segment of the model ID starts with one of its
markers, ignoring case, so moonshotai/Kimi-K2-Instruct and /models/kimi-k3 both select
Kimi.
| Profile | Selected by |
|---|---|
| Kimi | kimi*, moonshot*; the K3 rules apply to kimi-k3* and kimi_k3* |
| MiniMax | minimax*, abab* |
| z.ai (GLM) | glm*, zai*, z-ai*; the GLM-5.3 rules apply to glm-5.3*, glm5.3*, glm_5.3*, glm-5-3*, glm5-3* |
| DeepSeek V4 | A segment equal to deepseek-flash, deepseek-v4-flash, deepseek-v4-pro, deepseek-v4.1-flash, or deepseek-v4.1-flash-1017 |
| OpenAI | A model none of the above matched that names one of OpenAI's own hosted models: an openai segment (openai/gpt-4.1), a segment starting with gpt- (gpt-oss* is self-hosted and excluded), chatgpt-, codex-, computer-use-preview, text-embedding- or omni-moderation, or the o1/o3/o4 reasoning series (o3, o4-mini) |
| Baseline (default) | Everything else, including older DeepSeek models and THUDM/chatglm3-6b: only the core request validation applies |
Names configured with --model-alias are read through the alias table. An alias takes
the profile of the served model it resolves to, so the profile is the same whichever of
its names a request carries. A served model whose own name selects no vendor contract
takes the contract of a vendor-named (Kimi, MiniMax, z.ai, DeepSeek) alias it is served
under, series rules included: an opaque alias of a Kimi K3, GLM-5.3, or DeepSeek V4.1
model keeps that series' rules. When two aliases of one served model name different
vendors, the first configured wins. An OpenAI model name binds no contract as an alias:
with gpt-4o=local-model, a request for gpt-4o is judged as the self-hosted
local-model, not under the OpenAI rules. Message-level extension fields
that belong to another profile (such as Kimi's message tools sent to a non-Kimi model)
are dropped before dispatch, with one warning logged per request.
Profile defaults and validation apply to /v1/chat/completions requests that pass
through SMG's request validation. They are not applied to /v1/responses, and a request
the HTTP router streams to the worker unparsed skips them (see
Request Streaming).
Validation Rules¶
A violation is rejected with 400 in the validation envelope (see
Error Format): error.type is invalid_request_error, error.code is
the rule code from the last column, and error.message carries the rule's message.
Profile rules judge the request as a whole, so error.param is null:
{
"error": {
"message": "invalid thinking.type: only enabled or disabled is allowed for this model",
"type": "invalid_request_error",
"param": null,
"code": "thinking_type_not_supported"
}
}| Profile | Rejected | Rule code |
|---|---|---|
| All except MiniMax | A root message |
invalid_role |
| Kimi | tools on a user or assistant message |
tools_role_restricted |
| Kimi | tools on a system or developer message that is not a list of tools |
tools_malformed |
| Kimi | A non-empty message tools list together with non-empty content on that message |
tools_content_conflict |
| Kimi | A message tool whose type is not function |
tool_type_unsupported |
| Kimi | A message tool name not matching [A-Za-z_][A-Za-z0-9_]*, or longer than 256 characters |
tool_name_invalid |
| Kimi | A message tool name already declared in tools or on another message |
tool_name_duplicate |
| Kimi K3 | temperature other than 0, 0.6, or 1; top_p other than 0.95; a non-zero presence_penalty or frequency_penalty; n other than 1 |
temperature_not_allowed, top_p_not_allowed, presence_penalty_not_allowed, frequency_penalty_not_allowed, n_not_allowed |
| Kimi K3, z.ai, DeepSeek V4 | thinking.type: adaptive |
thinking_type_not_supported |
| Kimi K3 | thinking.effort other than low, high, or max |
thinking_effort_invalid |
| MiniMax | A tool-call ID used twice in the history | tool_call_id_duplicate |
| MiniMax | A tool message whose tool_call_id answers no pending call |
tool_call_id_mismatch |
| MiniMax | Historical tool-call arguments that are non-empty but not a JSON object |
tool_call_arguments_invalid_json |
| MiniMax | A tool call with no tool message answering it |
tool_call_unanswered |
| z.ai | thinking.effort or reasoning_effort outside none, minimal, low, medium, high, xhigh, max (GLM-5.3 also rejects none: the series cannot switch thinking off) |
reasoning_effort_not_allowed |
| z.ai GLM-5.3 | thinking.type: disabled |
thinking_disabled_not_supported |
| z.ai | A tool whose type is not function |
tool_type_not_supported |
| z.ai | A file_url content part (SMG does not fetch files for the model) |
content_part_not_supported |
| z.ai (gRPC) | max_completion_tokens (or max_tokens) larger than the context window the selected worker advertises |
context_length_exceeded |
| DeepSeek V4 | reasoning_effort or thinking.effort outside none, minimal, low, medium, high, xhigh, max; V4.1 also takes an integer budget from 1 to 100 |
reasoning_effort_not_allowed |
| DeepSeek V4 | chat_template_kwargs.thinking or enable_thinking that is neither a boolean nor null |
invalid_thinking_override |
| DeepSeek V4 | chat_template_kwargs.thinking and enable_thinking that disagree |
conflicting_thinking_overrides |
| OpenAI, DeepSeek V4 | A tool message with no assistant tool_calls turn right before it; a tool_call_id that turn did not issue; an assistant tool_calls turn whose calls are not each answered by a tool message before the conversation goes on |
tool_message_without_tool_calls, tool_call_id_not_found, tool_calls_without_tool_messages |
| OpenAI | max_tokens and max_completion_tokens both set |
invalid_parameter_combination |
| OpenAI | response_format: json_object without the word "json" (any case) in a message's text |
json_object_requires_json_in_messages |
| OpenAI | A json_schema response format with strict: true whose schema contains an object without "additionalProperties": false |
invalid_json_schema |
| OpenAI | A tool with function.strict: true whose parameter schema contains an object without "additionalProperties": false |
invalid_function_parameters |
z.ai and DeepSeek V4 check both effort spellings, even the one that thinking.effort
shadows.
The z.ai (gRPC) context-window rule needs the selected worker's advertised window, so it
is enforced in the gRPC pipeline rather than in request validation: alone in this table,
its 400 carries the X-SMG-Error-Code header.
Defaults and Rewrites¶
| Profile | Behavior |
|---|---|
| Kimi | Streaming requests get stream_options.include_usage: true unless set. Tools declared on system and developer messages join the request's tools for tool_choice, the tool-choice constraint, and tool-call parsing; a system message that only declares tools may omit content |
| Kimi K3 | Omitted fields get the contract values: temperature 1.0, top_p 0.95, presence_penalty 0, frequency_penalty 0, n 1 |
| MiniMax | Streaming requests get stream_options.include_usage: true unless set. root messages are hoisted above all other messages and merged into one leading message that keeps the root role: texts are joined by a blank line in their original order, a parts list keeps its parts, and the merged message carries the first one's name. On gRPC workers, responses are scanned for tool calls even when the request declares no tools |
| z.ai | thinking.clear_thinking becomes the clear_thinking chat-template variable unless chat_template_kwargs sets it. tool_stream is accepted and dropped (SMG streams tool-call deltas anyway). GLM-5.x, GLM-4.7, and GLM-4.6 get temperature 1.0 and top_p 0.95 when omitted |
| z.ai GLM-5.3 | Effort minimal → low, medium → high, and xhigh → max in both reasoning_effort and thinking.effort, matching z.ai's own API; workers and external providers see only the level |
| DeepSeek V4 | Effort minimal → low; on V4, medium and xhigh → high (V4.1 keeps both as native levels, though a request forwarded to an HTTP worker or external provider still sends medium as high, matching DeepSeek's own API). Thinking is on unless turned off (thinking.type: disabled, effort none, or a template override), and the result is written to chat_template_kwargs.thinking when that is unset. Streams use DeepSeek's usage format (see Stream Usage) |
Error Responses¶
Error Format¶
Errors that SMG generates use this envelope and repeat code in the X-SMG-Error-Code
response header:
{
"error": {
"type": "invalid_request_error",
"code": "context_length_exceeded",
"message": "This model's maximum context length is 2048 tokens. However, your request has 2310 input tokens. Please reduce the length of the input.",
"param": null
}
}type is the public API's vocabulary, which clients switch on: invalid_request_error for a 4xx
(the detail is in code; a 403 is permission_error and a 429 rate_limit_error) and server_error
for a 5xx. Request parsing and validation failures use the same envelope but carry no
X-SMG-Error-Code header. Their code names the failure, and param names the offending
field when one is known (for example messages[0].content); it is null for a syntax error
or a rule over the whole request:
| Failure | code |
|---|---|
| The body is not valid JSON | invalid_json |
| The body does not match the request schema | missing_required_parameter, unknown_parameter, invalid_type (a value of the wrong shape, including a non-object body), invalid_value (a value outside an enumeration), or integer_below_min_value (a negative integer where the field cannot be negative) |
| A field crosses a documented bound | The kind of value and the bound crossed: integer_above_max_value, decimal_below_min_value, string_above_max_length, empty_array, ... |
| A cross-field or provider-profile rule fails | The rule's own code: stream_options_requires_stream, tool_choice_requires_tools, mutually_exclusive_parameters, ... |
The Content-Type: application/json header is missing, or the body fails to parse for any other reason |
json_parse_error |
A 401 from API-key authentication uses the envelope (code invalid_api_key) with a
WWW-Authenticate: Bearer challenge, but carries no X-SMG-Error-Code header either.
Errors from HTTP workers are
relayed with the worker's status and body. An external provider's error is relayed under
its status too, minus the account identifiers in its message (see
Provider Error Responses).
SMG never forwards a worker's X-SMG-Error-Code header, so that header always means SMG
itself produced the error.
Error Codes¶
| Status | code |
Returned when |
|---|---|---|
| 400 | invalid_json |
The body is not valid JSON |
| 400 | missing_required_parameter, unknown_parameter, invalid_type, invalid_value, integer_below_min_value |
The body does not match the request schema; param names the field when one is known |
| 400 | integer_above_max_value, decimal_below_min_value, string_above_max_length, empty_array, ... |
A field is outside its documented bounds; the code names the kind of value and the bound crossed, param the field |
| 400 | The rule's code | A cross-field rule (stream_options_requires_stream for stream_options without stream, top_logprobs_requires_logprobs, tool_choice_function_not_found for a tool_choice naming a missing tool, json_schema_schema_not_object for a non-object json_schema, ...) or a provider-profile rule failed |
| 400 | json_parse_error |
The Content-Type: application/json header is missing, or the body failed to parse for a reason outside the set above |
| 400 | context_length_exceeded |
gRPC: the longest prompt is longer than the context window the selected worker advertises (the tighter leg governs in PD, and each prompt of a batched completion counts on its own). Workers that advertise no window are not checked |
| 400 | invalid_prompt_token |
gRPC: a completion's token-ID prompt holds an ID that is not a token of the model. The message names the position and value (prompt[2] = ..., or prompt[1][1] = ... in a batch); an invalid ID that only the decode catches is reported as a failed decode instead. Refused before any worker is called |
| 400 | unsupported_content_part |
gRPC: a content part SMG cannot render |
| 400 | audio_output_not_supported, unsupported_output_modality |
gRPC: return_audio: true or a non-text entry in modalities; gRPC workers produce text only |
| 400 | multimodal_not_supported |
gRPC: the selected engine does not accept one of the request's input modalities |
| Engine-mapped | start_generation_failed |
gRPC: the engine did not start generation. The status follows the engine's gRPC status, for example invalid argument → 400 (not retried and not counted against the circuit breaker), resource exhausted → 429, unavailable → 503, deadline exceeded → 504, internal → 500 |
| Engine status, or 502 | prefill_worker_failed_to_start, decode_worker_failed_to_start |
PD on either transport: the prefill or decode leg never started. The engine refused it and its status is kept (over gRPC, mapped from the engine's gRPC status as above), or the call to its worker failed below HTTP (502) |
| Engine status, or 502 | prefill_stream_error, decode_stream_error |
PD on either transport: the leg's stream ended with an error after it began, with the answer to the client still whole. Over gRPC the engine's status is kept (mapped as above); over HTTP the leg's body was cut after its 2xx head (502). A failure after the client's own stream has started ends it with a stream_error event instead (see Streaming Response) |
| 500 | pd_rendezvous_failed |
gRPC PD: the decode leg failed before producing anything, and the prefill's bootstrap port, probed now, no longer accepts connections — the sides could not rendezvous. The message says the port did not accept a connection, with the probe's error; the gateway log names the port's address |
| 401 | invalid_api_key |
Missing or unknown API key. The response adds a WWW-Authenticate: Bearer challenge (error="invalid_token" when a key was presented) |
| 404 | model_not_found |
No worker serves the model |
| 408 | request_body_stalled |
The request body stalled for --stream-body-stall-timeout-secs while being read, whether it is buffered or streamed to the worker |
| 408 | request_body_timeout |
The request body was still incomplete at --request-timeout-secs, whether it is buffered or streamed to the worker |
| 413 | request_body_too_large |
The request body exceeds --max-payload-size, however it arrives: a declared Content-Length over the limit (refused before routing), a buffered or chunked body that crosses it while being read, or a streamed body that grows past it mid-upload |
| 429 | admission_queue_full, scheduler_queue_full, tenant_rate_limit_exceeded, pd_prefill_queue_full, pd_prefill_queue_timeout |
Admission control or rate limiting; see Rate Limiting |
| 500 | internal_error and stage-specific codes |
A failure inside SMG; the code names the failing step |
| 502 | upstream_response_too_large |
A worker's non-streaming response exceeded --max-payload-size |
| 503 | no_available_workers |
Workers serve the model, but every one is unhealthy or has an open circuit breaker, or the routing policy selects none of them (HTTP and gRPC paths) |
| 503 | no_models_available |
GET /v1/models: no self-hosted worker serves any model (see List Models) |
| 503 | worker_overload_protection_shed, admission_queue_timeout, selection_queue_timeout, scheduler_queue_timeout, scheduler_preempted |
Overload and admission sheds; see Rate Limiting |
| 504 | prefill_worker_silent, decode_worker_silent |
gRPC PD: the leg produced nothing within its liveness bound. Retryable, and the retry window closes on that worker alone — its partner stays selectable |
On gRPC workers, an engine finish reason of error (outside the OpenAI set) becomes an
error response or stream error; clients never receive "finish_reason": "error".
Client Libraries¶
Python (OpenAI SDK)¶
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:30000/v1",
api_key="your-api-key" # or "not-needed" if auth disabled
)
response = client.chat.completions.create(
model="meta-llama/Llama-3.1-8B-Instruct",
messages=[
{"role": "user", "content": "Hello!"}
]
)
print(response.choices[0].message.content)JavaScript/TypeScript¶
import OpenAI from 'openai';
const client = new OpenAI({
baseURL: 'http://localhost:30000/v1',
apiKey: 'your-api-key'
});
const response = await client.chat.completions.create({
model: 'meta-llama/Llama-3.1-8B-Instruct',
messages: [
{ role: 'user', content: 'Hello!' }
]
});
console.log(response.choices[0].message.content);cURL¶
curl http://localhost:30000/v1/chat/completions \
-H "Content-Type: application/json" \
-H "Authorization: Bearer your-api-key" \
-d '{
"model": "meta-llama/Llama-3.1-8B-Instruct",
"messages": [{"role": "user", "content": "Hello!"}]
}'Request Headers¶
Header names are case-insensitive. SMG reads the following request headers:
| Header | Purpose | Documented in |
|---|---|---|
Content-Type |
application/json, or multipart/form-data for /v1/audio/transcriptions |
— |
Authorization |
Bearer <key>; required when --api-key or --tenant-api-key is set |
Authentication |
x-api-key, x-goog-api-key |
The caller's own key for Anthropic (x-api-key) and Gemini (x-goog-api-key) provider workers. /v1/responses and /v1/interactions prefer it over Authorization, and /v1/messages forwards x-api-key as sent; /v1/chat/completions uses only Authorization |
External Providers |
X-Request-ID |
Request ID for log correlation. SMG checks x-request-id, x-correlation-id, x-trace-id, and request-id in that order, and generates an ID when none is present. --request-id-headers replaces the list |
Logging |
traceparent, tracestate |
W3C trace context. With --enable-trace, SMG continues the caller's trace and propagates it to workers |
Monitoring |
X-SMG-Routing-Key |
Routing key for sticky sessions and the key-based policies (manual, consistent_hashing, prefix_hash). It is the default name in --routing-key-headers |
Sticky Sessions and Routing Keys |
X-SMG-Routing-Tokens |
Routing hint carrying the prompt's leading token IDs | Routing Hint Headers |
X-SMG-Target-Worker |
0-based index of the worker to use among the model's available workers. Honored only by consistent_hashing; an index with no available worker fails with 503 |
Load Balancing |
X-SMG-Priority |
Priority class: system, interactive, default, or bulk. Read only when the priority scheduler is enabled |
Priority Scheduler |
X-SMG-Tenant-ID |
Tenant identity asserted by a trusted upstream, resolved as tenant key header:<value>. Read only with --trust-tenant-header, and only when API-key auth has not already identified the caller. --tenant-header-name renames it |
Tenant Rate Limiting |
X-SMG-MCP |
enabled makes SMG run the MCP tool loop itself for /v1/messages requests with MCP toolsets sent to an Anthropic provider, instead of forwarding them as-is |
Messages API |
X-SMG-Parser |
legacy or symphony: which parser reads this request's output, overriding --parser for the one request. Read on Chat Completions, Messages API, and Responses API requests served through the gRPC pipeline (gRPC and ZMQ workers); any other value is rejected with 400 invalid_parser_header |
gRPC Pipeline |
x-smg-cache-trace |
1 asks for this request's routing-decision record in the x-smg-cache-trace response header, on requests served through the gRPC pipeline, whatever the SMG_CACHE_TRACE_* variables say |
Routing Trace Variables |
Names set with --storage-context-headers |
Copied into the storage hook request context | Configuration Reference |
When SMG proxies a request to an HTTP worker, it forwards only these client headers: Authorization, anthropic-version, anthropic-beta, X-Request-ID, X-Correlation-ID, headers starting with X-Request-ID-, traceparent, tracestate, X-SMG-Routing-Key, and x-override-priority (SGLang's request priority, read by an engine started with SGLANG_ENABLE_REQUEST_HEADER_OVERRIDES=true and not by SMG).
Routing Hint Headers¶
A client, or a proxy in front of SMG that has already tokenized the prompt, can pass the routing signal in a header so SMG does not need the request body to choose a worker. Hints affect only worker placement. A malformed or over-limit hint is ignored and the request routes as if the header were absent; it never causes an error.
| Header | Value | Limits | Effect |
|---|---|---|---|
X-SMG-Routing-Tokens |
The prompt's leading token IDs as comma-separated decimal integers from 0 to 4294967295, with no spaces, signs, or empty items |
At most 512 IDs and 4096 bytes | Replaces the token IDs and text SMG would take from the body when choosing a worker. cache_aware matches the hinted prefix against its token tree (its placement index with --cache-index hash) and records it there; prefix_hash hashes it. Read on the regular HTTP path only (not in PD mode or for gRPC workers), and never forwarded to workers |
X-SMG-Routing-Key |
An opaque key | Non-empty UTF-8, at most 128 bytes | Without --routing-key-override, consistent_hashing and prefix_hash hash the key instead of their other inputs, and manual pins it. With the override, every policy except consistent_hashing pins the key |
# X-SMG-Routing-Tokens carries the prompt's first token IDs
curl http://localhost:30000/v1/completions \
-H "Content-Type: application/json" \
-H "X-SMG-Routing-Tokens: 128000,791,4062,14198,39935" \
-d @request.jsonWith a valid X-SMG-Routing-Tokens hint, the text-routing policies (cache_aware, bucket) no longer need the body to place the request, so a large body can stream to the worker instead of being buffered; see Request Streaming. When --routing-key-override is enabled, SMG always buffers the body to read its rid, and hints do not change that.
Response Headers¶
| Header | When SMG sets it | Documented in |
|---|---|---|
x-request-id |
On API responses: the request ID SMG used, whether received or generated | Logging |
x-smg-routed-worker-id |
On inference responses from HTTP, gRPC, and ZMQ workers: the registered URL of the worker that served the request, with an @<rank> suffix for data-parallel workers. For a prefill/decode pair it names the decode worker. From gRPC and ZMQ workers, only successful responses carry it, and Responses API responses do not |
Sticky Sessions and Routing Keys |
x-smg-parser |
On successful Chat Completions, Messages API, and Responses API responses from gRPC and ZMQ workers: the parser that read the output, symphony or legacy. Responses served by the separate gpt-oss (Harmony) pipeline, and the queued object a non-streaming background: true Responses request returns, carry no header |
gRPC Pipeline |
x-smg-cache-trace |
On successful responses from gRPC and ZMQ workers when the request sent x-smg-cache-trace: 1 or the gateway runs with SMG_CACHE_TRACE_HEADER=1: the dispatch's routing-decision record. Omitted when the record exceeds 2048 bytes or is not plain ASCII |
Routing Trace Variables |
X-SMG-Error-Code |
On errors the gateway generates in its standard error envelope, with the same value as error.code; request-validation errors do not carry it. SMG drops this header from worker responses, so it always marks a gateway decision |
— |
X-SMG-Preempted |
true on the 503 returned to a request preempted by the priority scheduler |
Priority Scheduler |
Retry-After |
On 429 and 503 responses that ask the client to back off: concurrency limits, worker overload, the priority scheduler, tenant rate limits, and the 503 answered when no worker is available (Retry-After: 1) |
Rate Limiting |
Rate Limiting¶
SMG does not send X-RateLimit-* headers. A request shed by admission control, rate
limiting, or overload protection gets 429 or 503 with the standard
error envelope, the X-SMG-Error-Code header, and usually a
Retry-After header in seconds:
code |
Status | Retry-After |
Cause |
|---|---|---|---|
admission_queue_full |
429 | 2 |
--max-concurrent-requests is reached and the admission queue is full or disabled |
admission_queue_timeout |
503 | 2 |
The request waited in the admission queue longer than --queue-timeout-secs |
selection_queue_timeout |
503 | 2 |
gRPC and ZMQ workers: the request had already waited longer than --queue-timeout-secs inside the gateway when it reached worker selection (the prefill-admission wait of a PD or EPD request included) |
scheduler_queue_full |
429 | 2 |
Priority scheduler: the request's class queue is full |
scheduler_queue_timeout |
503 | 2 |
Priority scheduler: the request waited longer than its class's queue timeout |
scheduler_preempted |
503 | 1 |
Priority scheduler: preempted by higher-priority traffic before its first token; also sets X-SMG-Preempted: true |
tenant_rate_limit_exceeded |
429 | Seconds until the tenant's budget can admit the request; omitted when it never can | Tenant rate limiting |
pd_prefill_queue_full |
429 | — | PD and EPD: the opt-in prefill admission queue is full |
pd_prefill_queue_timeout |
429 | — | PD and EPD: the request waited out --prefill-queue-timeout-secs without a prefill slot |
worker_overload_protection_shed |
503 | --load-monitor-interval (default 10) |
With --worker-overload-shed (off by default, see Overload Protection): every eligible worker is over its overload threshold, or the selected one crossed it just before dispatch. Also, with or without --worker-overload-shed, on a gRPC PD or EPD dispatch whose decode worker could not admit the request within --pd-admission-wait-secs; HTTP PD dispatches are not gated by the decode admission window |
Wait at least Retry-After seconds before retrying. For configuration, see
Rate Limiting,
Tenant Rate Limiting,
Priority Scheduling, and
Overload Protection.