Responses API Reference¶
The Responses API provides an OpenAI-compatible interface for agentic workflows with built-in support for multi-turn conversations, tool execution, and MCP (Model Context Protocol) integration.
Overview¶
Purpose vs Chat Completions API¶
The Responses API differs from the Chat Completions API in several key ways:
| Feature | Chat Completions | Responses API |
|---|---|---|
| Conversation State | Stateless | Server-managed state |
| Tool Execution | Client-side | Server-side with MCP support |
| Multi-turn | Manual | Automatic with previous_response_id |
| Persistence | None | Built-in response/conversation storage |
| Agentic Workflows | Manual orchestration | Built-in tool loop execution |
Agentic Workflow Concepts¶
The Responses API enables agentic workflows where the model can:
- Reason about tasks using optional reasoning parameters
- Plan tool usage with automatic tool selection
- Execute tools via MCP servers or function calling
- Iterate through multiple tool calls in a single request
- Persist conversation history for multi-session workflows
Backend Support¶
How SMG serves /v1/responses depends on the worker behind the model:
| Worker | Behavior |
|---|---|
| gRPC and ZMQ direct workers | SMG runs the request through its chat pipeline, executes MCP tools, and stores responses and conversations. gpt-oss models use SMG's Harmony pipeline |
| gRPC EPD (encode-prefill-decode) | Not served (501) |
| HTTP workers, including PD | Forwarded to the worker's own /v1/responses; storage, chaining, and tools are the worker's |
| External OpenAI-compatible providers | Forwarded upstream; SMG executes MCP tools and stores responses and conversations |
Unless noted otherwise, the behavior on this page is that of gRPC workers and applies
equally to ZMQ direct workers (ipc:// or tcp:// URLs), which SMG serves through the
same pipeline. Stored responses and conversations live in SMG's history backend
(--history-backend, default memory); see
Chat History.
Stored responses and conversations are scoped to the tenant that created them — another tenant's ID is answered exactly as an unknown one; see Tenant Scope.
Base URL¶
http://localhost:30000/v1Create Response¶
Create a new response with optional tool execution and conversation management.
POST /v1/responsesRequest Body¶
| Field | Type | Required | Description |
|---|---|---|---|
model |
string | Yes | Model identifier |
input |
string or array | Yes | Input text or array of input items. Must not be empty; see Input Formats |
instructions |
string | No | System instructions for the model |
max_output_tokens |
integer | No | Maximum tokens to generate (at least 1, as max_tokens on the other APIs; the public API's own floor of 16 is not imposed) |
max_tool_calls |
integer | No | Maximum tool calls SMG executes for this response (at least 1); see Tool Call Limits |
temperature |
number | No | Sampling temperature (0-2), default: 1.0 |
top_p |
number | No | Nucleus sampling parameter, greater than 0 and at most 1 |
stream |
boolean | No | Enable streaming responses |
stream_options |
object | No | Streaming options, such as include_obfuscation |
store |
boolean | No | Store response for later retrieval, default: true |
background |
boolean | No | Return a queued response object at once and run the request in the background; see Background Mode |
tools |
array | No | Available tools: function, namespace, mcp, and built-in tool types |
tool_choice |
string/object | No | auto, none, required, a function ({"type": "function", "name": "...", "namespace": "..."}), allowed_tools, an MCP server ({"type": "mcp", "server_label": "..."}), or a built-in tool type. Anything except none or auto requires a non-empty tools array; with auto and no tools the model answers in text, as if tool_choice were omitted |
parallel_tool_calls |
boolean | No | Allow parallel tool execution, default: true when tools is set |
previous_response_id |
string | No | Continue from a stored response |
conversation |
string or object | No | Conversation ID (conv_...) or {"id": "conv_..."}; mutually exclusive with previous_response_id. The conversation's stored items replay ahead of the request's input, bounded at the newest 100 items (see Conversation Management) |
reasoning |
object | No | Reasoning configuration |
text |
object | No | Text format for structured outputs |
include |
array | No | Extra output, such as reasoning.encrypted_content or message.output_text.logprobs |
top_logprobs |
integer | No | Most likely tokens per position (0-20) |
metadata |
object | No | Custom metadata, echoed in the response; at most 16 pairs, keys up to 64 characters, string values up to 512 characters |
user |
string | No | End-user identifier |
An unknown top-level parameter is rejected with 400 unknown_parameter, with param
naming it: every parameter SMG accepts is declared. Besides the fields above and the
SGLang extensions, the declared set covers the public API's
service_tier, truncation, prompt, prompt_cache_key, prompt_cache_retention,
safety_identifier, and context_management, plus SMG's own request_id and
priority. client_metadata, an object of tracing identifiers sent by clients such as
Codex, is also accepted but not forwarded: SMG leaves it out of the request bodies it
sends to workers and providers.
Input Formats¶
Simple text input:
{
"input": "What is the capital of France?"
}Structured input items:
{
"input": [
{
"type": "message",
"role": "user",
"content": [{"type": "input_text", "text": "Hello!"}]
}
]
}A message's role must be assistant, system, developer, or user; any other role
is rejected with 400 before the request reaches a worker.
Replaying a function call and its result:
{
"input": [
{"type": "message", "role": "user", "content": "What's the weather in Paris?"},
{
"type": "function_call",
"call_id": "call_abc123",
"name": "get_weather",
"arguments": "{\"location\": \"Paris\"}"
},
{
"type": "function_call_output",
"call_id": "call_abc123",
"output": [{"type": "input_text", "text": "18°C and sunny"}]
}
]
}A function_call_output.output is a string or an array of content parts, and an empty
result is allowed. gRPC workers concatenate the text parts in order and reject a result
that contains media; HTTP workers receive the array unchanged. A replayed function_call
keeps its namespace when it has one.
Tool results without a new message:
{
"previous_response_id": "resp_abc123",
"input": [
{
"type": "function_call_output",
"call_id": "call_abc123",
"output": "18°C and sunny"
}
]
}A continuation needs no message items: in the standard agentic loop the client answers a
response's function_call by sending back only the matching function_call_output, with
the earlier turns held by previous_response_id or conversation. A store: false
request that replays the history, as in the previous example, likewise needs no new user
message after the tool result. On gpt-oss (Harmony) models each result's call_id must
match a function_call earlier in the resolved input — the replayed items plus any
history loaded from previous_response_id — and an unmatched result is rejected with
400 harmony_build_failed (No function call found for call_id: ...).
Tool Configuration¶
Function tools:
{
"tools": [
{
"type": "function",
"name": "get_weather",
"description": "Get weather for a location",
"parameters": {
"type": "object",
"properties": {
"location": {"type": "string"}
},
"required": ["location"]
}
}
]
}A function tool's parameters may be omitted or set to null: both declare a function
that takes no arguments, and SMG reads the empty schema {} for it wherever a schema is
needed. A schema you do give must be a JSON object; any other JSON type is rejected
with 400 (invalid_type, naming tools[i].parameters). With strict: true, OpenAI's
own hosted model IDs are additionally held to the public API's schema rules; see
Text Format.
When SMG writes a function tool back out — in the tools echoed on a response object,
or in a request body forwarded to an HTTP worker or external provider — the tool always
carries parameters: null for a tool declared without a schema, otherwise the schema
as you gave it, so an explicit {} stays {}.
Namespaced tools:
{
"tools": [
{
"type": "namespace",
"name": "github",
"description": "GitHub repository tools",
"tools": [
{
"type": "function",
"name": "create_issue",
"description": "Open an issue",
"parameters": {
"type": "object",
"properties": {"title": {"type": "string"}},
"required": ["title"]
}
}
]
}
],
"tool_choice": {"type": "function", "name": "create_issue", "namespace": "github"}
}A namespace groups function and custom tools; namespaces do not nest. A function
tool_choice with namespace must name a function in that namespace. On gRPC workers SMG
presents each member to the model as <namespace>.<name> and maps the calls back, so a
generated function_call item carries the member name and a separate namespace field
in responses, streaming events, and replays. A top-level function whose own name contains
a dot keeps that literal name.
MCP tools:
{
"tools": [
{
"type": "mcp",
"server_url": "http://localhost:8080/mcp",
"server_label": "my-mcp-server",
"server_description": "My MCP server for data access",
"require_approval": "never",
"allowed_tools": ["query_database", "search_files"]
}
]
}server_label is required: it must start with a letter, contain only letters, digits,
-, and _, and be unique within the request (ignoring case). server_url and
connector_id cannot both be set. allowed_tools takes a list of tool names or a
{"read_only": ..., "tool_names": [...]} filter. SMG skips a declared MCP server it
cannot connect to and logs a warning; only when no MCP server for the request could be
connected does the request on gRPC workers fail with 424 connect_mcp_server_failed.
Built-in tools: web_search_preview, code_interpreter, file_search, and
image_generation run through MCP servers configured to back those built-in types; see
MCP. The non-preview web_search tool is accepted
with all of its fields, including external_web_access (an explicit false is kept), and
forwarded to HTTP workers and external providers. The local_shell tool is not supported:
a request that declares it is rejected with 400 (tool_not_supported).
Reasoning Configuration¶
{
"reasoning": {
"effort": "medium",
"summary": "auto"
}
}Effort tiers, lowest to highest: none, minimal, low, medium, high, xhigh,
max. A reasoning object without effort defaults to medium. Any other value is
rejected with 400 invalid_value. summary accepts auto, concise, or detailed.
On gRPC workers the tier reaches the chat pipeline as the same reasoning_effort string,
where none and minimal turn thinking off on most models (for the DeepSeek-V4 and
MiniMax exceptions, see
Thinking and Reasoning Effort). gpt-oss
(Harmony) models know only low, medium, and high, so SMG clamps the outer tiers:
none and minimal become low, and xhigh and max become high.
With include: ["reasoning.encrypted_content"], reasoning items also carry an
encrypted_content blob that a store: false client can send back on its next turn. SMG
fills it with the reasoning text in version-tagged base64: opaque, not encrypted.
Text Format (Structured Outputs)¶
{
"text": {
"format": {
"type": "json_schema",
"name": "user_info",
"schema": {
"type": "object",
"properties": {
"name": {"type": "string"},
"age": {"type": "integer"}
}
},
"strict": true
}
}
}format.type is text, json_object, or json_schema. For json_schema, name must
not be empty and schema must be a JSON object; a violation is rejected with 400
(json_schema_name_empty or json_schema_schema_not_object) before the request reaches
a worker.
For model IDs that name one of OpenAI's own hosted models — gpt-* (not the open-weight
gpt-oss family), chatgpt-*, codex-*, the o1/o3/o4 reasoning series,
computer-use-preview, or any ID behind an openai/ route segment, such as
openai/gpt-4.1 — SMG also enforces the public API's content rules: a json_schema
format or a function tool with strict: true must set additionalProperties: false on
every object node of its schema, and a json_object format requires the word "json"
somewhere in instructions or the input text. Self-hosted models are not held to these
rules: the example above, whose strict schema does not pin additionalProperties, is
accepted for them and passed through to the engine, whose grammar compiler decides what
it can constrain.
Tool Call Limits¶
max_tool_calls caps the tool calls SMG executes itself for one response: MCP tools and
MCP-backed built-in tools. Function tools are returned to the client and do not count.
SMG also enforces an internal cap of 10 executions per response, so a larger
max_tool_calls behaves as 10.
When the model asks for a batch of calls that exceeds the remaining allowance, SMG executes the calls that fit and keeps their results:
- If your
max_tool_callsis the limit, the response ends normally with the executed calls; calls past the cap are dropped. - If the internal cap is the limit (no
max_tool_calls, or a value above 10), the response endsfailedwitherror.codemax_tool_calls_exceeded.
A batch that fits, even exactly, gets another model turn, so the model can answer after its
last permitted call. When a turn mixes MCP and function calls, SMG executes the MCP calls
within the allowance and then returns the response with the function calls for the client
to run. A turn that ends with a length or error finish never dispatches tools, even if a
complete call was parsed before the stop. Streaming and non-streaming responses follow the
same rules.
gpt-oss (Harmony) models check the limit before a batch runs and do not execute a batch
that would exceed it; the response then ends failed with max_tool_calls_exceeded, and
a stream ends with a response.failed terminal event carrying the same error.
On external OpenAI-compatible providers, your max_tool_calls and the internal cap end
the response the same way: it completes with incomplete_details.reason set to
max_tool_calls, and the calls that did not run are not dropped but appear in the
output as failed items, in each call's own format — an mcp_call whose structured
error says the call was not executed, or a built-in call's item type with status
failed (see Output Item Statuses). Streaming closes each
such item with its response.output_item.done event (an mcp_call also gets
response.mcp_call.failed) and ends with a normal response.completed.
Background Mode¶
With "background": true, SMG answers before the response is generated: it stores a
response object with "status": "queued" under a new response ID and returns it at once,
and the request runs in the background behind that ID. The queued object echoes the
request's parameters, carries "background": true, and has no output or usage yet. Poll
GET /v1/responses/{response_id} to follow the stored object through
queued → in_progress → completed, incomplete, failed, or cancelled, or stop
the run with POST /v1/responses/{response_id}/cancel.
curl http://localhost:30000/v1/responses \
-H "Content-Type: application/json" \
-d '{
"model": "meta-llama/Llama-3.1-8B-Instruct",
"input": "Write a detailed research report on AI trends",
"background": true
}'- Streaming. With
"stream": truethe client keeps the stream:response.createdreports the responsequeued, aresponse.queuedevent follows, and the usual events stream as generation proceeds. The work runs behind the stored ID either way, so a client that disconnects early does not stop generation: the response still finishes and is stored. A cancelled background stream ends without a terminal event. - Limits.
--max-background-responses(default 1024) caps how many background responses the gateway runs at once. A request beyond the cap is rejected with429background_responses_limit_reached; retry later or send the request withoutbackground. Setting the flag to0turns background mode off: requests withbackground: truethen run in the foreground like any other request. store: false. An unstored background response still runs in the background and can be polled while it runs; after it reaches a terminal status its result stays retrievable for roughly ten minutes and is then deleted. Nothing of it is written to conversation history.- History backend. The stored response object is the run's state, kept in SMG's
history backend. With
--history-backend nonethequeuedobject is still returned, but no record exists behind it: the response cannot be polled or cancelled.
Example Request¶
curl http://localhost:30000/v1/responses \
-H "Content-Type: application/json" \
-d '{
"model": "meta-llama/Llama-3.1-8B-Instruct",
"input": "Search for the latest news about AI",
"instructions": "Be concise and factual",
"max_output_tokens": 500,
"temperature": 0.7,
"tools": [
{
"type": "mcp",
"server_url": "http://localhost:8080/mcp",
"server_label": "search"
}
],
"tool_choice": "auto"
}'Response¶
Abbreviated:
{
"id": "resp_abc123",
"object": "response",
"created_at": 1705312345,
"status": "completed",
"instructions": "Be concise and factual",
"max_output_tokens": 500,
"model": "meta-llama/Llama-3.1-8B-Instruct",
"output": [
{
"type": "mcp_list_tools",
"id": "mcpl_abc123",
"server_label": "search",
"tools": [
{
"name": "web_search",
"description": "Search the web",
"input_schema": {"type": "object", "properties": {"query": {"type": "string"}}}
}
]
},
{
"type": "mcp_call",
"id": "mcp_abc123",
"status": "completed",
"arguments": "{\"query\": \"latest AI news\"}",
"name": "web_search",
"output": "{\"results\": [...]}",
"server_label": "search"
},
{
"type": "message",
"id": "msg_abc123",
"role": "assistant",
"content": [
{
"type": "output_text",
"text": "Based on my search, here are the latest AI developments...",
"annotations": []
}
],
"status": "completed"
}
],
"parallel_tool_calls": true,
"store": true,
"temperature": 0.7,
"tools": [
{
"type": "mcp",
"server_url": "http://localhost:8080/mcp",
"server_label": "search"
}
],
"usage": {
"input_tokens": 50,
"output_tokens": 150,
"total_tokens": 200,
"input_tokens_details": {"cached_tokens": 0},
"output_tokens_details": {"reasoning_tokens": 0}
},
"metadata": {}
}Output Item Statuses¶
| Item | status |
|---|---|
message |
completed; in_progress when generation failed partway (the partial text is kept) |
function_call |
completed once fully generated; incomplete when its JSON arguments were cut off (by max_output_tokens or anything else that ended the turn mid-call); in_progress when generation failed |
reasoning |
completed |
mcp_call |
completed, or failed with a structured error (see Tool Errors) |
Built-in calls (web_search_call, code_interpreter_call, file_search_call, image_generation_call) |
completed for every call that ran, also on failure; the failure details are in the item content. On external OpenAI-compatible providers, a call not run at the tool-call limit appears as the same item type with status failed |
The response's own status:
status |
Meaning |
|---|---|
completed |
Finished normally, including a stop at your max_tool_calls |
incomplete |
max_output_tokens was reached (incomplete_details.reason is max_output_tokens), or the turn ended leaving a tool call's arguments cut off (incomplete_details is null) |
failed |
Generation failed or the internal tool-call cap was hit; error carries code and message |
queued |
Background response waiting for its run to start |
in_progress |
Background response still generating |
cancelled |
Background response stopped by Cancel Response |
For a failed generation, error keeps the upstream error's code (or type) and
message when the pipeline reported them; otherwise it is server_error with
Upstream generation failed. stream_error marks a failure to read the stream, and
max_tool_calls_exceeded the internal cap.
The response statuses above apply to gpt-oss (Harmony) models too, with one difference:
reaching the tool-call limit — your max_tool_calls included — fails their response,
streamed or not (see Tool Call Limits).
Usage¶
Usage uses the Responses field names: input_tokens, output_tokens, total_tokens,
plus input_tokens_details.cached_tokens and output_tokens_details.reasoning_tokens.
On gRPC workers both details objects are always present, in the response object and in
a streamed terminal event's usage alike, holding zeros when the worker reported no
cached or reasoning counts. With MCP tools, usage is summed over every model turn of the tool loop
(input, output, total, cached, and reasoning tokens) in both streaming and non-streaming
mode; a turn that reports no usage leaves the earlier totals intact. gpt-oss (Harmony)
models report only the last turn's usage.
Streaming Response¶
With "stream": true, responses are sent as Server-Sent Events:
curl http://localhost:30000/v1/responses \
-H "Content-Type: application/json" \
-d '{
"model": "meta-llama/Llama-3.1-8B-Instruct",
"input": "Hello!",
"stream": true
}'Event sequence:
event: response.created
data: {"type":"response.created","sequence_number":0,"response":{"id":"resp_abc123","object":"response","created_at":1705312345,"status":"in_progress","background":false,"model":"meta-llama/Llama-3.1-8B-Instruct","output":[]}}
event: response.in_progress
data: {"type":"response.in_progress","sequence_number":1,"response":{"id":"resp_abc123","object":"response","status":"in_progress"}}
event: response.output_item.added
data: {"type":"response.output_item.added","sequence_number":2,"output_index":0,"item":{"id":"msg_abc123","type":"message","role":"assistant","content":[]}}
event: response.content_part.added
data: {"type":"response.content_part.added","sequence_number":3,"output_index":0,"item_id":"msg_abc123","content_index":0,"part":{"type":"output_text","text":""}}
event: response.output_text.delta
data: {"type":"response.output_text.delta","sequence_number":4,"output_index":0,"item_id":"msg_abc123","content_index":0,"delta":"Hello! How can I help you?"}
event: response.output_text.done
data: {"type":"response.output_text.done","sequence_number":5,"output_index":0,"item_id":"msg_abc123","content_index":0,"text":"Hello! How can I help you?"}
event: response.content_part.done
data: {"type":"response.content_part.done","sequence_number":6,"output_index":0,"item_id":"msg_abc123","content_index":0,"part":{"type":"output_text","text":"Hello! How can I help you?"}}
event: response.output_item.done
data: {"type":"response.output_item.done","sequence_number":7,"output_index":0,"item":{"id":"msg_abc123","type":"message","role":"assistant","content":[{"type":"output_text","text":"Hello! How can I help you?"}],"status":"completed"}}
event: response.completed
data: {"type":"response.completed","sequence_number":8,"response":{"id":"resp_abc123","object":"response","created_at":1705312345,"status":"completed","model":"meta-llama/Llama-3.1-8B-Instruct","output":[...],"usage":{"input_tokens":10,"output_tokens":8,"total_tokens":18,"input_tokens_details":{"cached_tokens":0},"output_tokens_details":{"reasoning_tokens":0}},...}}The stream closes after the terminal event. Unlike Chat Completions, a Responses stream
from gRPC and ZMQ workers has no data: [DONE] sentinel. Streams from HTTP workers are
relayed as the worker sends them. On external providers, a stream served through SMG's
MCP tool loop ends with response.completed followed by a data: [DONE] frame that SMG
appends; a stream the tool loop does not handle is relayed as the upstream sends it.
Paired item events. Every item opens with response.output_item.added and closes with
response.output_item.done, with its content events in between:
| Item | Events between added and done |
|---|---|
reasoning |
response.content_part.added (a reasoning_text part), response.reasoning_text.delta, response.reasoning_text.done, response.content_part.done |
message |
response.content_part.added, response.output_text.delta, response.output_text.done, response.content_part.done |
function_call |
response.function_call_arguments.delta, response.function_call_arguments.done |
custom_tool_call |
response.custom_tool_call_input.delta, response.custom_tool_call_input.done |
mcp_list_tools |
response.mcp_list_tools.in_progress, response.mcp_list_tools.completed |
mcp_call |
response.mcp_call.in_progress, response.mcp_call_arguments.delta, response.mcp_call_arguments.done, then response.mcp_call.completed or response.mcp_call.failed |
| Built-in calls | response.<type>.in_progress, the searching/interpreting/generating event, response.<type>.completed |
Reasoning streams first, as its own item. A streamed function_call item opens with
"status": "in_progress" and closes with its final status.
Terminal events. Exactly one of these ends the stream, unless an internal SMG error cuts it short with an error frame or a cancel ends a background stream with no terminal event:
| Event | response.status |
|---|---|
response.completed |
completed |
response.incomplete |
incomplete; incomplete_details is {"reason": "max_output_tokens"} when max_output_tokens ended the turn, and absent when a cut tool call left it incomplete |
response.failed |
failed, with error: {"code": "...", "message": "..."} |
Before the terminal event, SMG closes every item that is still open, exactly once, even
when generation fails. Partial output stays in the terminal response with its unfinished
status. gpt-oss (Harmony) streams end with the same terminal events,
response.incomplete included; the tool-call limit ends them with response.failed
(see Tool Call Limits), and a pipeline failure ends them with an
error event instead of a terminal event. For every model, a stored response whose
record could not be written ends with response.failed: the code is
storage_unavailable when the history store is down or past its per-call bound, and
store_response_failed when the store itself refused the write (see
Common Errors).
MCP-specific streaming events:
event: response.output_item.added
data: {"type":"response.output_item.added","sequence_number":2,"output_index":0,"item":{"id":"mcpl_abc123","type":"mcp_list_tools","server_label":"search","status":"in_progress","tools":[]}}
event: response.mcp_list_tools.in_progress
data: {"type":"response.mcp_list_tools.in_progress","sequence_number":3,"output_index":0}
event: response.mcp_list_tools.completed
data: {"type":"response.mcp_list_tools.completed","sequence_number":4,"output_index":0,"tools":[...]}
event: response.output_item.done
data: {"type":"response.output_item.done","sequence_number":5,"output_index":0,"item":{"id":"mcpl_abc123","type":"mcp_list_tools","server_label":"search","status":"completed","tools":[...]}}
event: response.output_item.added
data: {"type":"response.output_item.added","sequence_number":6,"output_index":1,"item":{"id":"mcp_abc123","type":"mcp_call","name":"web_search","status":"in_progress","arguments":"","server_label":"search"}}
event: response.mcp_call.in_progress
data: {"type":"response.mcp_call.in_progress","sequence_number":7,"output_index":1,"item_id":"mcp_abc123"}
event: response.mcp_call_arguments.delta
data: {"type":"response.mcp_call_arguments.delta","sequence_number":8,"output_index":1,"item_id":"mcp_abc123","delta":"{\"query\": \"latest AI news\"}"}
event: response.mcp_call_arguments.done
data: {"type":"response.mcp_call_arguments.done","sequence_number":9,"output_index":1,"item_id":"mcp_abc123","arguments":"{\"query\": \"latest AI news\"}"}
event: response.mcp_call.completed
data: {"type":"response.mcp_call.completed","sequence_number":10,"output_index":1,"item_id":"mcp_abc123"}
event: response.output_item.done
data: {"type":"response.output_item.done","sequence_number":11,"output_index":1,"item":{"type":"mcp_call","id":"mcp_abc123","status":"completed","arguments":"{\"query\": \"latest AI news\"}","name":"web_search","output":"...","server_label":"search"}}How an MCP call's arguments arrive depends on the serving path. On gRPC workers, models
other than gpt-oss send them as a single response.mcp_call_arguments.delta (as in the
example above), while gpt-oss (Harmony) models open the call with an empty arguments
delta and stream the argument chunks as further deltas. External providers' streams do
not relay the upstream's own argument deltas: SMG buffers them and, when the upstream
finishes the call's arguments, emits a single response.mcp_call_arguments.delta
carrying the full arguments, followed by response.mcp_call_arguments.done — the same
single-delta shape as models other than gpt-oss on gRPC workers.
The model's own deltas for MCP-bound calls are not shown as function_call events; a
server-executed call appears only through its mcp_call (or built-in) events, while
function tools in the same turn keep their own item events. On gRPC workers, models
other than gpt-oss close a failed mcp_call with response.mcp_call.failed, carrying
an error message string (an SMG extension), instead of response.mcp_call.completed.
gpt-oss (Harmony) models and external providers close a failed call with
response.mcp_call.completed; the failure is reported in the final response's
mcp_call item (see Tool Errors). On external providers,
response.mcp_call.failed marks only a call that was not run at the
tool-call limit.
Tool Errors¶
A failed MCP call does not fail the response. SMG passes the failure back to the model as
the tool result and records an mcp_call item with status: "failed" and a structured
error, following the current OpenAI schema:
{
"type": "mcp_call",
"id": "mcp_abc123",
"status": "failed",
"arguments": "{\"query\": \"latest AI news\"}",
"error": {"type": "mcp_tool_execution_error", "content": "upstream broke"},
"name": "web_search",
"output": "",
"server_label": "search"
}error.type is mcp_protocol_error (with code and message), mcp_tool_execution_error
(with content), or http_error (with code and message). SMG itself produces
mcp_tool_execution_error. An older stored item whose error is a plain string is read
as mcp_tool_execution_error, so replaying it sends the object form.
Get Response¶
Retrieve a previously stored response by ID.
GET /v1/responses/{response_id}Path Parameters¶
| Parameter | Type | Description |
|---|---|---|
response_id |
string | The response ID (e.g., resp_abc123) |
Example Request¶
curl http://localhost:30000/v1/responses/resp_abc123def456Response¶
Returns the stored response object as shown in the Create Response section. SMG reads it
from its own history backend, which the gRPC and external-provider paths fill when store
is true; responses produced by HTTP workers are stored by the worker and are not
retrievable through SMG. An unknown ID — or another tenant's response — returns 404
with code not_found.
A background response reads back as queued or in_progress while
it runs and as its terminal object once it ends; its input items are stored with the
queued record, so List Response Input Items answers from
the first poll. A store: false background response stays retrievable for roughly ten
minutes after it ends.
Cancel Response¶
POST /v1/responses/{response_id}/cancelCancels a background response. Behavior depends on the connection mode:
- gRPC workers: A
queuedorin_progressbackground response is stopped and returned with"status": "cancelled". This also works across gateways that share a history backend: the record is closed ascancelled, and the gateway running the response notices within a few seconds and drops the work. Cancelling an alreadycancelledresponse returns its object again. Acompletedorincompleteresponse returns400response_already_completed, afailedone400response_already_failed, a response not created withbackground: true400cancellation_not_supported, and an unknown ID — or another tenant's response —404response_not_found. - HTTP workers: The request is sent to every registered worker; SMG returns the first successful answer, otherwise the last error. Whether cancellation succeeds depends on backend support.
Path Parameters¶
| Parameter | Type | Description |
|---|---|---|
response_id |
string | The response ID to cancel |
Example Request¶
curl -X POST http://localhost:30000/v1/responses/resp_abc123def456/cancelResponse¶
gRPC workers: Returns the cancelled response object (abbreviated):
{
"id": "resp_abc123def456",
"object": "response",
"created_at": 1705312345,
"completed_at": 1705312349,
"status": "cancelled",
"background": true,
"model": "meta-llama/Llama-3.1-8B-Instruct",
"output": []
}Cancelling a response that was not created with background: true returns:
{
"error": {
"type": "invalid_request_error",
"code": "cancellation_not_supported",
"message": "Only responses created with 'background': true can be cancelled.",
"param": null
}
}HTTP workers: Returns the response object from the backend.
Delete Response¶
Delete a stored response.
DELETE /v1/responses/{response_id}Path Parameters¶
| Parameter | Type | Description |
|---|---|---|
response_id |
string | The response ID to delete |
Example Request¶
curl -X DELETE http://localhost:30000/v1/responses/resp_abc123def456Response¶
{
"id": "resp_abc123def456",
"object": "response.deleted",
"deleted": true
}An unknown ID — or another tenant's response — returns 404 with code not_found.
Deleting a background response that is still queued or
in_progress also stops its run: the work is dropped and no result is stored.
List Response Input Items¶
List the input items that were sent with a response.
GET /v1/responses/{response_id}/input_itemsPath Parameters¶
| Parameter | Type | Description |
|---|---|---|
response_id |
string | The response ID |
Example Request¶
curl http://localhost:30000/v1/responses/resp_abc123def456/input_itemsResponse¶
{
"object": "list",
"data": [
{
"id": "msg_abc123",
"type": "message",
"role": "user",
"content": [{"type": "input_text", "text": "Hello!"}]
}
],
"first_id": "msg_abc123",
"last_id": "msg_abc123",
"has_more": false
}All stored input items are returned in one page (has_more is always false). Items
stored without an ID get a generated msg_ ID.
An unknown ID — or another tenant's response — returns 404 with code not_found.
Conversation Management¶
Conversations provide persistent storage for multi-turn interactions, enabling chat history to be maintained across multiple requests.
When a request names a conversation, the conversation's stored items are replayed into
the model's prompt ahead of the request's own input, in the order they were stored. The
replay is bounded: only the newest 100 stored items are sent. Older items stay stored and
listable but do not reach the model; to keep context from older turns in play, repeat it
in the request's input.
A conversation holding exactly 100 items replays whole; a longer one is cut, and the cut
can land inside a turn. The replayed window then begins at its first turn boundary: the
first user, developer, or system message or, when the window holds no such message,
the first tool call whose output is also in the window, together with the reasoning items
immediately before that call. The tail of the turn the cut split is not replayed, and
neither is a tool output whose call fell before the window, so a cut never leaves the
prompt opening with a tool result whose call is missing.
The conversation endpoints report errors in the gateway envelope
({"error": {"message", "type", "param", "code"}}, see Error Format).
An invalid request answers 400 with code invalid_request; an unknown conversation
or item answers 404 with code not_found. The one exception is
Delete Conversation Item, which answers 200, not 404,
for an item that is not in the conversation.
Three 400 rejections carry their own code and param: under
Create Conversation Items, item_already_in_conversation
for an item the conversation already holds (param is items) and
string_above_max_length for an item id over 64 bytes (param is items[].id);
under Get Conversation Item, invalid_value for a bad
include value (param names the offending entry, such as include[0]).
A conversation is scoped to the tenant that created it: on every route below, and as
the conversation of a new response, another tenant's conversation ID answers 404,
the same as an unknown ID (see
Tenant Scope).
Create Conversation¶
POST /v1/conversationsRequest Body¶
| Field | Type | Required | Description |
|---|---|---|---|
metadata |
object | No | Custom metadata (max 16 properties) |
Example Request¶
curl http://localhost:30000/v1/conversations \
-H "Content-Type: application/json" \
-d '{
"metadata": {
"project": "customer-support",
"user_id": "user_123"
}
}'Response¶
{
"id": "conv_abc123def456",
"object": "conversation",
"created_at": 1705312345,
"metadata": {
"project": "customer-support",
"user_id": "user_123"
}
}Get Conversation¶
GET /v1/conversations/{conversation_id}Example Request¶
curl http://localhost:30000/v1/conversations/conv_abc123def456Response¶
{
"id": "conv_abc123def456",
"object": "conversation",
"created_at": 1705312345,
"metadata": {
"project": "customer-support"
}
}Update Conversation¶
Update conversation metadata. Uses merge semantics — set a key to null to delete it. The
merged metadata may hold at most 16 properties.
POST /v1/conversations/{conversation_id}Request Body¶
| Field | Type | Description |
|---|---|---|
metadata |
object | Metadata to merge (null values delete keys) |
Example Request¶
curl http://localhost:30000/v1/conversations/conv_abc123def456 \
-H "Content-Type: application/json" \
-d '{
"metadata": {
"status": "resolved",
"project": null
}
}'Response¶
Returns the updated conversation object.
Delete Conversation¶
DELETE /v1/conversations/{conversation_id}Example Request¶
curl -X DELETE http://localhost:30000/v1/conversations/conv_abc123def456Response¶
{
"id": "conv_abc123def456",
"object": "conversation.deleted",
"deleted": true
}List Conversation Items¶
GET /v1/conversations/{conversation_id}/itemsQuery Parameters¶
| Parameter | Type | Default | Description |
|---|---|---|---|
limit |
integer | 100 | Maximum items to return |
order |
string | desc |
Sort order: asc or desc |
after |
string | - | Cursor for pagination |
Example Request¶
curl "http://localhost:30000/v1/conversations/conv_abc123/items?limit=20&order=asc"Response¶
{
"object": "list",
"data": [
{
"id": "msg_abc123",
"type": "message",
"role": "user",
"content": [{"type": "input_text", "text": "Hello"}],
"status": "completed",
"created_at": 1705312345
},
{
"id": "msg_def456",
"type": "message",
"role": "assistant",
"content": [{"type": "output_text", "text": "Hi there!"}],
"status": "completed",
"created_at": 1705312346
}
],
"has_more": false,
"first_id": "msg_abc123",
"last_id": "msg_def456"
}has_more is true when the page is full (it holds limit items).
Create Conversation Items¶
Add items to a conversation. Maximum 20 items per request.
POST /v1/conversations/{conversation_id}/itemsRequest Body¶
| Field | Type | Required | Description |
|---|---|---|---|
items |
array | Yes | Array of items to add (max 20) |
Supported Item Types¶
These item types are fully supported:
message- User or assistant messages (requiresrole); an item withouttypeis a messagereasoning- Model reasoning contentmcp_list_tools- MCP tool listingmcp_call- MCP tool invocationitem_reference- Reference to an existing item byid; an unknown ID — or another tenant's item — returns404
These types are stored, but the response adds a warnings entry because SMG does not
implement them yet: function_call, function_call_output, file_search_call,
computer_call, computer_call_output, web_search_call, image_generation_call,
code_interpreter_call, local_shell_call, local_shell_call_output,
mcp_approval_request, mcp_approval_response, custom_tool_call, and
custom_tool_call_output. Any other type is rejected with 400.
An item whose id is already in the conversation is rejected with:
{
"error": {
"message": "Item already in conversation",
"type": "invalid_request_error",
"param": "items",
"code": "item_already_in_conversation"
}
}An item submitted with an explicit id that already names a stored item the caller may
see — its own, or one stored without a tenant — reuses that stored item as-is. An ID that
belongs to another tenant's item is rejected with 400 (Item id '...' cannot be used),
neither serving nor overwriting it (see
Tenant Scope).
A supplied item id — an item's own or an item_reference's — may be at most 64 bytes;
a longer one is rejected with 400 (string_above_max_length on items[].id) before
any item of the request is stored.
Example Request¶
curl http://localhost:30000/v1/conversations/conv_abc123/items \
-H "Content-Type: application/json" \
-d '{
"items": [
{
"type": "message",
"role": "user",
"content": [{"type": "input_text", "text": "What is 2+2?"}]
},
{
"type": "message",
"role": "assistant",
"content": [{"type": "output_text", "text": "2+2 equals 4."}]
}
]
}'Response¶
{
"object": "list",
"data": [
{
"id": "msg_abc123",
"type": "message",
"role": "user",
"content": [{"type": "input_text", "text": "What is 2+2?"}],
"status": "completed"
},
{
"id": "msg_def456",
"type": "message",
"role": "assistant",
"content": [{"type": "output_text", "text": "2+2 equals 4."}],
"status": "completed"
}
],
"first_id": "msg_abc123",
"last_id": "msg_def456",
"has_more": false
}Get Conversation Item¶
GET /v1/conversations/{conversation_id}/items/{item_id}An optional include query parameter (repeatable, also accepted as include[]) must name
includable fields: code_interpreter_call.outputs, computer_call_output.output.image_url,
file_search_call.results, message.input_image.image_url, message.output_text.logprobs,
reasoning.encrypted_content, web_search_call.action.sources, or web_search_call.results.
Any other value is rejected with 400 (invalid_value); a valid one is accepted but does
not change the returned item.
Example Request¶
curl http://localhost:30000/v1/conversations/conv_abc123/items/msg_abc123Response¶
Returns the item object. An item that exists but is not in this conversation returns
404.
Delete Conversation Item¶
Remove an item from a conversation. This performs a soft delete — the item may still exist if referenced by other conversations.
DELETE /v1/conversations/{conversation_id}/items/{item_id}Example Request¶
curl -X DELETE http://localhost:30000/v1/conversations/conv_abc123/items/msg_abc123Response¶
Returns the conversation object. An item that is not in the conversation, including an
unknown item ID, is ignored: the request answers 200, not 404. An unknown
conversation still answers 404.
Examples¶
Simple Agentic Workflow¶
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:30000/v1",
api_key="your-api-key"
)
# Create a response with MCP tools
response = client.responses.create(
model="meta-llama/Llama-3.1-8B-Instruct",
input="Search for the weather in San Francisco and summarize it",
tools=[
{
"type": "mcp",
"server_url": "http://localhost:8080/mcp",
"server_label": "weather-service"
}
],
tool_choice="auto"
)
# The response includes tool calls and final answer
for output in response.output:
if output.type == "mcp_call":
print(f"Tool called: {output.name} ({output.status})")
print(f"Result: {output.output}")
elif output.type == "message":
for content in output.content:
if content.type == "output_text":
print(f"Answer: {content.text}")Multi-turn Conversation with Tools¶
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:30000/v1",
api_key="your-api-key"
)
# Create a conversation
conversation = client.conversations.create(
metadata={"session": "support-123"}
)
# First turn
response1 = client.responses.create(
model="meta-llama/Llama-3.1-8B-Instruct",
input="I need help with my order #12345",
conversation=conversation.id,
tools=[
{
"type": "mcp",
"server_url": "http://localhost:8080/mcp",
"server_label": "order-service"
}
]
)
print(f"First response: {response1.id}")
# Second turn - continues the conversation
response2 = client.responses.create(
model="meta-llama/Llama-3.1-8B-Instruct",
input="Can you also check if there are any discounts available?",
conversation=conversation.id,
tools=[
{
"type": "mcp",
"server_url": "http://localhost:8080/mcp",
"server_label": "order-service"
}
]
)
print(f"Second response: {response2.id}")
# List conversation history (tool items have no role)
items = client.conversations.items.list(conversation.id)
for item in items.data:
print(item.type, getattr(item, "role", None))Streaming Response Handling¶
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:30000/v1",
api_key="your-api-key"
)
# Stream a response
with client.responses.create(
model="meta-llama/Llama-3.1-8B-Instruct",
input="Explain quantum computing",
stream=True
) as stream:
for event in stream:
if event.type == "response.output_text.delta":
print(event.delta, end="", flush=True)
elif event.type == "response.mcp_call.in_progress":
print(f"\n[Calling tool: {event.item_id}]")
elif event.type == "response.completed":
print(f"\n\nTokens used: {event.response.usage.total_tokens}")
elif event.type in ("response.incomplete", "response.failed"):
print(f"\n\nStopped: {event.response.status}")Using Previous Response ID¶
# Alternative to conversations - chain responses directly
response1 = client.responses.create(
model="meta-llama/Llama-3.1-8B-Instruct",
input="What are the main programming paradigms?",
store=True
)
# Continue from previous response
response2 = client.responses.create(
model="meta-llama/Llama-3.1-8B-Instruct",
input="Can you elaborate on functional programming?",
previous_response_id=response1.id,
store=True
)A chained turn does not have to add a message: after a response ends in function_call
items, the next request's input can be only their function_call_output items (see
Input Formats).
Error Responses¶
Error Format¶
Errors that SMG generates use the gateway envelope, with the public API's error type as type
(invalid_request_error for a 4xx, the detail in code; permission_error for a 403;
rate_limit_error for a 429; server_error for a 5xx) and the code repeated in the
X-SMG-Error-Code header:
{
"error": {
"type": "invalid_request_error",
"code": "conversation_not_found",
"message": "Conversation 'conv_abc123' not found. Please create the conversation first using the conversations API.",
"param": null
}
}Request parsing and validation failures use the same envelope without the header. Their
code names the failure: invalid_json for a syntax error, a schema code such as
unknown_parameter or invalid_type for a body that does not match the request type, a
bound code such as integer_above_max_value for a field outside its range, or the
violated rule's own code such as invalid_conversation_id. param names the offending
field when one is known. See Error Responses for the full
vocabulary.
Common Errors¶
| HTTP Status | code |
Description |
|---|---|---|
| 400 | The failure's code (see Error Format) | Malformed request or validation failure: invalid_json, a schema or bound code (unknown_parameter, invalid_type, integer_above_max_value, ...), or the violated rule's code (invalid_conversation_id, mutually_exclusive_parameters, ...) |
| 400 | previous_response_not_found |
previous_response_id is not in SMG's storage, or is another tenant's response |
| 400 | convert_request_failed |
gRPC: an input item the chat pipeline cannot represent, such as media in a function_call_output |
| 400 | response_already_completed, response_already_failed, cancellation_not_supported |
Cancel refused; see Cancel Response |
| 401 | invalid_api_key |
Invalid or missing API key: the error envelope with a WWW-Authenticate challenge, without the X-SMG-Error-Code header |
| 404 | model_not_found |
No worker serves the model |
| 404 | conversation_not_found |
The conversation does not exist, or belongs to another tenant |
| 404 | not_found |
Unknown or another tenant's ID: a response on get, delete, or input items, or a conversation or item on the conversation routes |
| 404 | response_not_found |
Unknown or another tenant's response ID on cancel |
| 424 | connect_mcp_server_failed |
gRPC: no MCP server declared by the request could be connected; an unreachable server is otherwise skipped with a warning |
| 429 | admission_queue_full, scheduler_queue_full |
Admission control; see Rate Limiting |
| 429 | background_responses_limit_reached |
The gateway already runs --max-background-responses background responses; see Background Mode |
| 500 | load_previous_response_chain_failed, check_conversation_failed, storage_error, store_response_failed |
A failure the history store itself answered: a refused statement, a record that cannot be decoded, an invalid chain, or a refused write of a finished response (store_response_failed: the response was generated but nothing of it was stored). Retrying does not help |
| 503 | storage_unavailable |
The history store is down or did not answer within its per-call bound (--postgres-op-timeout-secs / --redis-op-timeout-secs, default 5 seconds); retry once the store answers again. See Store Failures While Serving |
| 503 | no_available_workers |
No healthy workers available |
Validation Errors¶
{
"error": {
"message": "conversation: Invalid 'conversation': 'invalid-id'. Expected an ID that begins with 'conv_'.",
"type": "invalid_request_error",
"param": "conversation",
"code": "invalid_conversation_id"
}
}{
"error": {
"message": "Mutually exclusive parameters. Ensure you are only providing one of: 'previous_response_id' or 'conversation'.",
"type": "invalid_request_error",
"param": null,
"code": "mutually_exclusive_parameters"
}
}SGLang Extensions¶
The Responses API includes additional sampling parameters specific to SGLang:
| Field | Type | Default | Description |
|---|---|---|---|
top_k |
integer | -1 | Top-k sampling (-1 = disabled) |
min_p |
number | 0.0 | Min-p sampling threshold |
repetition_penalty |
number | 1.0 | Repetition penalty (1.0 = disabled) |
frequency_penalty |
number | - | OpenAI-compatible frequency penalty |
presence_penalty |
number | - | OpenAI-compatible presence penalty |
stop |
string/array | - | Stop sequences |
Where they take effect:
- HTTP workers receive them with the request.
- gpt-oss (Harmony) models on gRPC: all six fields are passed to the engine on every
model turn of the tool loop. Where the serving engine cannot match strings, SMG matches
stopsequences itself, so astoptakes effect on every backend except MLX, where a request with stringstopsequences gets a400(see gRPC Workers). - Other models on gRPC: all six fields are passed to the engine on every model turn
of the tool loop; which of them affect sampling is up to the serving engine.
top_k,min_p, andrepetition_penaltysent at their default values (-1,0.0, and1.0respectively) count as unset, the same as omitting them: the model's own sampling defaults apply. A matchedstopsequence ends generation and is trimmed from the output, as on Chat Completions.
On gpt-oss (Harmony) models on gRPC, the first stop match ends the model turn:
everything generated past the match, tool calls included, is dropped, and the matched
sequence stays as the suffix of the output — it is not trimmed, unlike on other models.
A stop can match inside the model's reasoning, ending the turn before any message, so
the response may carry a reasoning item and no message. A turn ended by stop
completes normally: "status": "completed" with no incomplete_details.
Example:
{
"model": "meta-llama/Llama-3.1-8B-Instruct",
"input": "Write a story",
"top_k": 50,
"min_p": 0.05,
"repetition_penalty": 1.1
}