Responses API Reference

The Responses API provides an OpenAI-compatible interface for agentic workflows with built-in support for multi-turn conversations, tool execution, and MCP (Model Context Protocol) integration.


Overview

Purpose vs Chat Completions API

The Responses API differs from the Chat Completions API in several key ways:

Feature Chat Completions Responses API
Conversation State Stateless Server-managed state
Tool Execution Client-side Server-side with MCP support
Multi-turn Manual Automatic with previous_response_id
Persistence None Built-in response/conversation storage
Agentic Workflows Manual orchestration Built-in tool loop execution

Agentic Workflow Concepts

The Responses API enables agentic workflows where the model can:

  1. Reason about tasks using optional reasoning parameters
  2. Plan tool usage with automatic tool selection
  3. Execute tools via MCP servers or function calling
  4. Iterate through multiple tool calls in a single request
  5. Persist conversation history for multi-session workflows

Backend Support

How SMG serves /v1/responses depends on the worker behind the model:

Worker Behavior
gRPC and ZMQ direct workers SMG runs the request through its chat pipeline, executes MCP tools, and stores responses and conversations. gpt-oss models use SMG's Harmony pipeline
gRPC EPD (encode-prefill-decode) Not served (501)
HTTP workers, including PD Forwarded to the worker's own /v1/responses; storage, chaining, and tools are the worker's
External OpenAI-compatible providers Forwarded upstream; SMG executes MCP tools and stores responses and conversations

Unless noted otherwise, the behavior on this page is that of gRPC workers and applies equally to ZMQ direct workers (ipc:// or tcp:// URLs), which SMG serves through the same pipeline. Stored responses and conversations live in SMG's history backend (--history-backend, default memory); see Chat History.

Stored responses and conversations are scoped to the tenant that created them — another tenant's ID is answered exactly as an unknown one; see Tenant Scope.


Base URL

http://localhost:30000/v1

Create Response

Create a new response with optional tool execution and conversation management.

POST /v1/responses

Request Body

Field Type Required Description
model string Yes Model identifier
input string or array Yes Input text or array of input items. Must not be empty; see Input Formats
instructions string No System instructions for the model
max_output_tokens integer No Maximum tokens to generate (at least 1, as max_tokens on the other APIs; the public API's own floor of 16 is not imposed)
max_tool_calls integer No Maximum tool calls SMG executes for this response (at least 1); see Tool Call Limits
temperature number No Sampling temperature (0-2), default: 1.0
top_p number No Nucleus sampling parameter, greater than 0 and at most 1
stream boolean No Enable streaming responses
stream_options object No Streaming options, such as include_obfuscation
store boolean No Store response for later retrieval, default: true
background boolean No Return a queued response object at once and run the request in the background; see Background Mode
tools array No Available tools: function, namespace, mcp, and built-in tool types
tool_choice string/object No auto, none, required, a function ({"type": "function", "name": "...", "namespace": "..."}), allowed_tools, an MCP server ({"type": "mcp", "server_label": "..."}), or a built-in tool type. Anything except none or auto requires a non-empty tools array; with auto and no tools the model answers in text, as if tool_choice were omitted
parallel_tool_calls boolean No Allow parallel tool execution, default: true when tools is set
previous_response_id string No Continue from a stored response
conversation string or object No Conversation ID (conv_...) or {"id": "conv_..."}; mutually exclusive with previous_response_id. The conversation's stored items replay ahead of the request's input, bounded at the newest 100 items (see Conversation Management)
reasoning object No Reasoning configuration
text object No Text format for structured outputs
include array No Extra output, such as reasoning.encrypted_content or message.output_text.logprobs
top_logprobs integer No Most likely tokens per position (0-20)
metadata object No Custom metadata, echoed in the response; at most 16 pairs, keys up to 64 characters, string values up to 512 characters
user string No End-user identifier

An unknown top-level parameter is rejected with 400 unknown_parameter, with param naming it: every parameter SMG accepts is declared. Besides the fields above and the SGLang extensions, the declared set covers the public API's service_tier, truncation, prompt, prompt_cache_key, prompt_cache_retention, safety_identifier, and context_management, plus SMG's own request_id and priority. client_metadata, an object of tracing identifiers sent by clients such as Codex, is also accepted but not forwarded: SMG leaves it out of the request bodies it sends to workers and providers.

Input Formats

Simple text input:

{
  "input": "What is the capital of France?"
}

Structured input items:

{
  "input": [
    {
      "type": "message",
      "role": "user",
      "content": [{"type": "input_text", "text": "Hello!"}]
    }
  ]
}

A message's role must be assistant, system, developer, or user; any other role is rejected with 400 before the request reaches a worker.

Replaying a function call and its result:

{
  "input": [
    {"type": "message", "role": "user", "content": "What's the weather in Paris?"},
    {
      "type": "function_call",
      "call_id": "call_abc123",
      "name": "get_weather",
      "arguments": "{\"location\": \"Paris\"}"
    },
    {
      "type": "function_call_output",
      "call_id": "call_abc123",
      "output": [{"type": "input_text", "text": "18°C and sunny"}]
    }
  ]
}

A function_call_output.output is a string or an array of content parts, and an empty result is allowed. gRPC workers concatenate the text parts in order and reject a result that contains media; HTTP workers receive the array unchanged. A replayed function_call keeps its namespace when it has one.

Tool results without a new message:

{
  "previous_response_id": "resp_abc123",
  "input": [
    {
      "type": "function_call_output",
      "call_id": "call_abc123",
      "output": "18°C and sunny"
    }
  ]
}

A continuation needs no message items: in the standard agentic loop the client answers a response's function_call by sending back only the matching function_call_output, with the earlier turns held by previous_response_id or conversation. A store: false request that replays the history, as in the previous example, likewise needs no new user message after the tool result. On gpt-oss (Harmony) models each result's call_id must match a function_call earlier in the resolved input — the replayed items plus any history loaded from previous_response_id — and an unmatched result is rejected with 400 harmony_build_failed (No function call found for call_id: ...).

Tool Configuration

Function tools:

{
  "tools": [
    {
      "type": "function",
      "name": "get_weather",
      "description": "Get weather for a location",
      "parameters": {
        "type": "object",
        "properties": {
          "location": {"type": "string"}
        },
        "required": ["location"]
      }
    }
  ]
}

A function tool's parameters may be omitted or set to null: both declare a function that takes no arguments, and SMG reads the empty schema {} for it wherever a schema is needed. A schema you do give must be a JSON object; any other JSON type is rejected with 400 (invalid_type, naming tools[i].parameters). With strict: true, OpenAI's own hosted model IDs are additionally held to the public API's schema rules; see Text Format.

When SMG writes a function tool back out — in the tools echoed on a response object, or in a request body forwarded to an HTTP worker or external provider — the tool always carries parameters: null for a tool declared without a schema, otherwise the schema as you gave it, so an explicit {} stays {}.

Namespaced tools:

{
  "tools": [
    {
      "type": "namespace",
      "name": "github",
      "description": "GitHub repository tools",
      "tools": [
        {
          "type": "function",
          "name": "create_issue",
          "description": "Open an issue",
          "parameters": {
            "type": "object",
            "properties": {"title": {"type": "string"}},
            "required": ["title"]
          }
        }
      ]
    }
  ],
  "tool_choice": {"type": "function", "name": "create_issue", "namespace": "github"}
}

A namespace groups function and custom tools; namespaces do not nest. A function tool_choice with namespace must name a function in that namespace. On gRPC workers SMG presents each member to the model as <namespace>.<name> and maps the calls back, so a generated function_call item carries the member name and a separate namespace field in responses, streaming events, and replays. A top-level function whose own name contains a dot keeps that literal name.

MCP tools:

{
  "tools": [
    {
      "type": "mcp",
      "server_url": "http://localhost:8080/mcp",
      "server_label": "my-mcp-server",
      "server_description": "My MCP server for data access",
      "require_approval": "never",
      "allowed_tools": ["query_database", "search_files"]
    }
  ]
}

server_label is required: it must start with a letter, contain only letters, digits, -, and _, and be unique within the request (ignoring case). server_url and connector_id cannot both be set. allowed_tools takes a list of tool names or a {"read_only": ..., "tool_names": [...]} filter. SMG skips a declared MCP server it cannot connect to and logs a warning; only when no MCP server for the request could be connected does the request on gRPC workers fail with 424 connect_mcp_server_failed.

Built-in tools: web_search_preview, code_interpreter, file_search, and image_generation run through MCP servers configured to back those built-in types; see MCP. The non-preview web_search tool is accepted with all of its fields, including external_web_access (an explicit false is kept), and forwarded to HTTP workers and external providers. The local_shell tool is not supported: a request that declares it is rejected with 400 (tool_not_supported).

Reasoning Configuration

{
  "reasoning": {
    "effort": "medium",
    "summary": "auto"
  }
}

Effort tiers, lowest to highest: none, minimal, low, medium, high, xhigh, max. A reasoning object without effort defaults to medium. Any other value is rejected with 400 invalid_value. summary accepts auto, concise, or detailed.

On gRPC workers the tier reaches the chat pipeline as the same reasoning_effort string, where none and minimal turn thinking off on most models (for the DeepSeek-V4 and MiniMax exceptions, see Thinking and Reasoning Effort). gpt-oss (Harmony) models know only low, medium, and high, so SMG clamps the outer tiers: none and minimal become low, and xhigh and max become high.

With include: ["reasoning.encrypted_content"], reasoning items also carry an encrypted_content blob that a store: false client can send back on its next turn. SMG fills it with the reasoning text in version-tagged base64: opaque, not encrypted.

Text Format (Structured Outputs)

{
  "text": {
    "format": {
      "type": "json_schema",
      "name": "user_info",
      "schema": {
        "type": "object",
        "properties": {
          "name": {"type": "string"},
          "age": {"type": "integer"}
        }
      },
      "strict": true
    }
  }
}

format.type is text, json_object, or json_schema. For json_schema, name must not be empty and schema must be a JSON object; a violation is rejected with 400 (json_schema_name_empty or json_schema_schema_not_object) before the request reaches a worker.

For model IDs that name one of OpenAI's own hosted models — gpt-* (not the open-weight gpt-oss family), chatgpt-*, codex-*, the o1/o3/o4 reasoning series, computer-use-preview, or any ID behind an openai/ route segment, such as openai/gpt-4.1 — SMG also enforces the public API's content rules: a json_schema format or a function tool with strict: true must set additionalProperties: false on every object node of its schema, and a json_object format requires the word "json" somewhere in instructions or the input text. Self-hosted models are not held to these rules: the example above, whose strict schema does not pin additionalProperties, is accepted for them and passed through to the engine, whose grammar compiler decides what it can constrain.

Tool Call Limits

max_tool_calls caps the tool calls SMG executes itself for one response: MCP tools and MCP-backed built-in tools. Function tools are returned to the client and do not count. SMG also enforces an internal cap of 10 executions per response, so a larger max_tool_calls behaves as 10.

When the model asks for a batch of calls that exceeds the remaining allowance, SMG executes the calls that fit and keeps their results:

  • If your max_tool_calls is the limit, the response ends normally with the executed calls; calls past the cap are dropped.
  • If the internal cap is the limit (no max_tool_calls, or a value above 10), the response ends failed with error.code max_tool_calls_exceeded.

A batch that fits, even exactly, gets another model turn, so the model can answer after its last permitted call. When a turn mixes MCP and function calls, SMG executes the MCP calls within the allowance and then returns the response with the function calls for the client to run. A turn that ends with a length or error finish never dispatches tools, even if a complete call was parsed before the stop. Streaming and non-streaming responses follow the same rules.

gpt-oss (Harmony) models check the limit before a batch runs and do not execute a batch that would exceed it; the response then ends failed with max_tool_calls_exceeded, and a stream ends with a response.failed terminal event carrying the same error.

On external OpenAI-compatible providers, your max_tool_calls and the internal cap end the response the same way: it completes with incomplete_details.reason set to max_tool_calls, and the calls that did not run are not dropped but appear in the output as failed items, in each call's own format — an mcp_call whose structured error says the call was not executed, or a built-in call's item type with status failed (see Output Item Statuses). Streaming closes each such item with its response.output_item.done event (an mcp_call also gets response.mcp_call.failed) and ends with a normal response.completed.

Background Mode

With "background": true, SMG answers before the response is generated: it stores a response object with "status": "queued" under a new response ID and returns it at once, and the request runs in the background behind that ID. The queued object echoes the request's parameters, carries "background": true, and has no output or usage yet. Poll GET /v1/responses/{response_id} to follow the stored object through queued → in_progress → completed, incomplete, failed, or cancelled, or stop the run with POST /v1/responses/{response_id}/cancel.

curl http://localhost:30000/v1/responses \
  -H "Content-Type: application/json" \
  -d '{
    "model": "meta-llama/Llama-3.1-8B-Instruct",
    "input": "Write a detailed research report on AI trends",
    "background": true
  }'
  • Streaming. With "stream": true the client keeps the stream: response.created reports the response queued, a response.queued event follows, and the usual events stream as generation proceeds. The work runs behind the stored ID either way, so a client that disconnects early does not stop generation: the response still finishes and is stored. A cancelled background stream ends without a terminal event.
  • Limits. --max-background-responses (default 1024) caps how many background responses the gateway runs at once. A request beyond the cap is rejected with 429 background_responses_limit_reached; retry later or send the request without background. Setting the flag to 0 turns background mode off: requests with background: true then run in the foreground like any other request.
  • store: false. An unstored background response still runs in the background and can be polled while it runs; after it reaches a terminal status its result stays retrievable for roughly ten minutes and is then deleted. Nothing of it is written to conversation history.
  • History backend. The stored response object is the run's state, kept in SMG's history backend. With --history-backend none the queued object is still returned, but no record exists behind it: the response cannot be polled or cancelled.

Example Request

curl http://localhost:30000/v1/responses \
  -H "Content-Type: application/json" \
  -d '{
    "model": "meta-llama/Llama-3.1-8B-Instruct",
    "input": "Search for the latest news about AI",
    "instructions": "Be concise and factual",
    "max_output_tokens": 500,
    "temperature": 0.7,
    "tools": [
      {
        "type": "mcp",
        "server_url": "http://localhost:8080/mcp",
        "server_label": "search"
      }
    ],
    "tool_choice": "auto"
  }'

Response

Abbreviated:

{
  "id": "resp_abc123",
  "object": "response",
  "created_at": 1705312345,
  "status": "completed",
  "instructions": "Be concise and factual",
  "max_output_tokens": 500,
  "model": "meta-llama/Llama-3.1-8B-Instruct",
  "output": [
    {
      "type": "mcp_list_tools",
      "id": "mcpl_abc123",
      "server_label": "search",
      "tools": [
        {
          "name": "web_search",
          "description": "Search the web",
          "input_schema": {"type": "object", "properties": {"query": {"type": "string"}}}
        }
      ]
    },
    {
      "type": "mcp_call",
      "id": "mcp_abc123",
      "status": "completed",
      "arguments": "{\"query\": \"latest AI news\"}",
      "name": "web_search",
      "output": "{\"results\": [...]}",
      "server_label": "search"
    },
    {
      "type": "message",
      "id": "msg_abc123",
      "role": "assistant",
      "content": [
        {
          "type": "output_text",
          "text": "Based on my search, here are the latest AI developments...",
          "annotations": []
        }
      ],
      "status": "completed"
    }
  ],
  "parallel_tool_calls": true,
  "store": true,
  "temperature": 0.7,
  "tools": [
    {
      "type": "mcp",
      "server_url": "http://localhost:8080/mcp",
      "server_label": "search"
    }
  ],
  "usage": {
    "input_tokens": 50,
    "output_tokens": 150,
    "total_tokens": 200,
    "input_tokens_details": {"cached_tokens": 0},
    "output_tokens_details": {"reasoning_tokens": 0}
  },
  "metadata": {}
}

Output Item Statuses

Item status
message completed; in_progress when generation failed partway (the partial text is kept)
function_call completed once fully generated; incomplete when its JSON arguments were cut off (by max_output_tokens or anything else that ended the turn mid-call); in_progress when generation failed
reasoning completed
mcp_call completed, or failed with a structured error (see Tool Errors)
Built-in calls (web_search_call, code_interpreter_call, file_search_call, image_generation_call) completed for every call that ran, also on failure; the failure details are in the item content. On external OpenAI-compatible providers, a call not run at the tool-call limit appears as the same item type with status failed

The response's own status:

status Meaning
completed Finished normally, including a stop at your max_tool_calls
incomplete max_output_tokens was reached (incomplete_details.reason is max_output_tokens), or the turn ended leaving a tool call's arguments cut off (incomplete_details is null)
failed Generation failed or the internal tool-call cap was hit; error carries code and message
queued Background response waiting for its run to start
in_progress Background response still generating
cancelled Background response stopped by Cancel Response

For a failed generation, error keeps the upstream error's code (or type) and message when the pipeline reported them; otherwise it is server_error with Upstream generation failed. stream_error marks a failure to read the stream, and max_tool_calls_exceeded the internal cap.

The response statuses above apply to gpt-oss (Harmony) models too, with one difference: reaching the tool-call limit — your max_tool_calls included — fails their response, streamed or not (see Tool Call Limits).

Usage

Usage uses the Responses field names: input_tokens, output_tokens, total_tokens, plus input_tokens_details.cached_tokens and output_tokens_details.reasoning_tokens. On gRPC workers both details objects are always present, in the response object and in a streamed terminal event's usage alike, holding zeros when the worker reported no cached or reasoning counts. With MCP tools, usage is summed over every model turn of the tool loop (input, output, total, cached, and reasoning tokens) in both streaming and non-streaming mode; a turn that reports no usage leaves the earlier totals intact. gpt-oss (Harmony) models report only the last turn's usage.

Streaming Response

With "stream": true, responses are sent as Server-Sent Events:

curl http://localhost:30000/v1/responses \
  -H "Content-Type: application/json" \
  -d '{
    "model": "meta-llama/Llama-3.1-8B-Instruct",
    "input": "Hello!",
    "stream": true
  }'

Event sequence:

event: response.created
data: {"type":"response.created","sequence_number":0,"response":{"id":"resp_abc123","object":"response","created_at":1705312345,"status":"in_progress","background":false,"model":"meta-llama/Llama-3.1-8B-Instruct","output":[]}}

event: response.in_progress
data: {"type":"response.in_progress","sequence_number":1,"response":{"id":"resp_abc123","object":"response","status":"in_progress"}}

event: response.output_item.added
data: {"type":"response.output_item.added","sequence_number":2,"output_index":0,"item":{"id":"msg_abc123","type":"message","role":"assistant","content":[]}}

event: response.content_part.added
data: {"type":"response.content_part.added","sequence_number":3,"output_index":0,"item_id":"msg_abc123","content_index":0,"part":{"type":"output_text","text":""}}

event: response.output_text.delta
data: {"type":"response.output_text.delta","sequence_number":4,"output_index":0,"item_id":"msg_abc123","content_index":0,"delta":"Hello! How can I help you?"}

event: response.output_text.done
data: {"type":"response.output_text.done","sequence_number":5,"output_index":0,"item_id":"msg_abc123","content_index":0,"text":"Hello! How can I help you?"}

event: response.content_part.done
data: {"type":"response.content_part.done","sequence_number":6,"output_index":0,"item_id":"msg_abc123","content_index":0,"part":{"type":"output_text","text":"Hello! How can I help you?"}}

event: response.output_item.done
data: {"type":"response.output_item.done","sequence_number":7,"output_index":0,"item":{"id":"msg_abc123","type":"message","role":"assistant","content":[{"type":"output_text","text":"Hello! How can I help you?"}],"status":"completed"}}

event: response.completed
data: {"type":"response.completed","sequence_number":8,"response":{"id":"resp_abc123","object":"response","created_at":1705312345,"status":"completed","model":"meta-llama/Llama-3.1-8B-Instruct","output":[...],"usage":{"input_tokens":10,"output_tokens":8,"total_tokens":18,"input_tokens_details":{"cached_tokens":0},"output_tokens_details":{"reasoning_tokens":0}},...}}

The stream closes after the terminal event. Unlike Chat Completions, a Responses stream from gRPC and ZMQ workers has no data: [DONE] sentinel. Streams from HTTP workers are relayed as the worker sends them. On external providers, a stream served through SMG's MCP tool loop ends with response.completed followed by a data: [DONE] frame that SMG appends; a stream the tool loop does not handle is relayed as the upstream sends it.

Paired item events. Every item opens with response.output_item.added and closes with response.output_item.done, with its content events in between:

Item Events between added and done
reasoning response.content_part.added (a reasoning_text part), response.reasoning_text.delta, response.reasoning_text.done, response.content_part.done
message response.content_part.added, response.output_text.delta, response.output_text.done, response.content_part.done
function_call response.function_call_arguments.delta, response.function_call_arguments.done
custom_tool_call response.custom_tool_call_input.delta, response.custom_tool_call_input.done
mcp_list_tools response.mcp_list_tools.in_progress, response.mcp_list_tools.completed
mcp_call response.mcp_call.in_progress, response.mcp_call_arguments.delta, response.mcp_call_arguments.done, then response.mcp_call.completed or response.mcp_call.failed
Built-in calls response.<type>.in_progress, the searching/interpreting/generating event, response.<type>.completed

Reasoning streams first, as its own item. A streamed function_call item opens with "status": "in_progress" and closes with its final status.

Terminal events. Exactly one of these ends the stream, unless an internal SMG error cuts it short with an error frame or a cancel ends a background stream with no terminal event:

Event response.status
response.completed completed
response.incomplete incomplete; incomplete_details is {"reason": "max_output_tokens"} when max_output_tokens ended the turn, and absent when a cut tool call left it incomplete
response.failed failed, with error: {"code": "...", "message": "..."}

Before the terminal event, SMG closes every item that is still open, exactly once, even when generation fails. Partial output stays in the terminal response with its unfinished status. gpt-oss (Harmony) streams end with the same terminal events, response.incomplete included; the tool-call limit ends them with response.failed (see Tool Call Limits), and a pipeline failure ends them with an error event instead of a terminal event. For every model, a stored response whose record could not be written ends with response.failed: the code is storage_unavailable when the history store is down or past its per-call bound, and store_response_failed when the store itself refused the write (see Common Errors).

MCP-specific streaming events:

event: response.output_item.added
data: {"type":"response.output_item.added","sequence_number":2,"output_index":0,"item":{"id":"mcpl_abc123","type":"mcp_list_tools","server_label":"search","status":"in_progress","tools":[]}}

event: response.mcp_list_tools.in_progress
data: {"type":"response.mcp_list_tools.in_progress","sequence_number":3,"output_index":0}

event: response.mcp_list_tools.completed
data: {"type":"response.mcp_list_tools.completed","sequence_number":4,"output_index":0,"tools":[...]}

event: response.output_item.done
data: {"type":"response.output_item.done","sequence_number":5,"output_index":0,"item":{"id":"mcpl_abc123","type":"mcp_list_tools","server_label":"search","status":"completed","tools":[...]}}

event: response.output_item.added
data: {"type":"response.output_item.added","sequence_number":6,"output_index":1,"item":{"id":"mcp_abc123","type":"mcp_call","name":"web_search","status":"in_progress","arguments":"","server_label":"search"}}

event: response.mcp_call.in_progress
data: {"type":"response.mcp_call.in_progress","sequence_number":7,"output_index":1,"item_id":"mcp_abc123"}

event: response.mcp_call_arguments.delta
data: {"type":"response.mcp_call_arguments.delta","sequence_number":8,"output_index":1,"item_id":"mcp_abc123","delta":"{\"query\": \"latest AI news\"}"}

event: response.mcp_call_arguments.done
data: {"type":"response.mcp_call_arguments.done","sequence_number":9,"output_index":1,"item_id":"mcp_abc123","arguments":"{\"query\": \"latest AI news\"}"}

event: response.mcp_call.completed
data: {"type":"response.mcp_call.completed","sequence_number":10,"output_index":1,"item_id":"mcp_abc123"}

event: response.output_item.done
data: {"type":"response.output_item.done","sequence_number":11,"output_index":1,"item":{"type":"mcp_call","id":"mcp_abc123","status":"completed","arguments":"{\"query\": \"latest AI news\"}","name":"web_search","output":"...","server_label":"search"}}

How an MCP call's arguments arrive depends on the serving path. On gRPC workers, models other than gpt-oss send them as a single response.mcp_call_arguments.delta (as in the example above), while gpt-oss (Harmony) models open the call with an empty arguments delta and stream the argument chunks as further deltas. External providers' streams do not relay the upstream's own argument deltas: SMG buffers them and, when the upstream finishes the call's arguments, emits a single response.mcp_call_arguments.delta carrying the full arguments, followed by response.mcp_call_arguments.done — the same single-delta shape as models other than gpt-oss on gRPC workers. The model's own deltas for MCP-bound calls are not shown as function_call events; a server-executed call appears only through its mcp_call (or built-in) events, while function tools in the same turn keep their own item events. On gRPC workers, models other than gpt-oss close a failed mcp_call with response.mcp_call.failed, carrying an error message string (an SMG extension), instead of response.mcp_call.completed. gpt-oss (Harmony) models and external providers close a failed call with response.mcp_call.completed; the failure is reported in the final response's mcp_call item (see Tool Errors). On external providers, response.mcp_call.failed marks only a call that was not run at the tool-call limit.

Tool Errors

A failed MCP call does not fail the response. SMG passes the failure back to the model as the tool result and records an mcp_call item with status: "failed" and a structured error, following the current OpenAI schema:

{
  "type": "mcp_call",
  "id": "mcp_abc123",
  "status": "failed",
  "arguments": "{\"query\": \"latest AI news\"}",
  "error": {"type": "mcp_tool_execution_error", "content": "upstream broke"},
  "name": "web_search",
  "output": "",
  "server_label": "search"
}

error.type is mcp_protocol_error (with code and message), mcp_tool_execution_error (with content), or http_error (with code and message). SMG itself produces mcp_tool_execution_error. An older stored item whose error is a plain string is read as mcp_tool_execution_error, so replaying it sends the object form.


Get Response

Retrieve a previously stored response by ID.

GET /v1/responses/{response_id}

Path Parameters

Parameter Type Description
response_id string The response ID (e.g., resp_abc123)

Example Request

curl http://localhost:30000/v1/responses/resp_abc123def456

Response

Returns the stored response object as shown in the Create Response section. SMG reads it from its own history backend, which the gRPC and external-provider paths fill when store is true; responses produced by HTTP workers are stored by the worker and are not retrievable through SMG. An unknown ID — or another tenant's response — returns 404 with code not_found.

A background response reads back as queued or in_progress while it runs and as its terminal object once it ends; its input items are stored with the queued record, so List Response Input Items answers from the first poll. A store: false background response stays retrievable for roughly ten minutes after it ends.


Cancel Response

POST /v1/responses/{response_id}/cancel

Cancels a background response. Behavior depends on the connection mode:

  • gRPC workers: A queued or in_progress background response is stopped and returned with "status": "cancelled". This also works across gateways that share a history backend: the record is closed as cancelled, and the gateway running the response notices within a few seconds and drops the work. Cancelling an already cancelled response returns its object again. A completed or incomplete response returns 400 response_already_completed, a failed one 400 response_already_failed, a response not created with background: true 400 cancellation_not_supported, and an unknown ID — or another tenant's response — 404 response_not_found.
  • HTTP workers: The request is sent to every registered worker; SMG returns the first successful answer, otherwise the last error. Whether cancellation succeeds depends on backend support.

Path Parameters

Parameter Type Description
response_id string The response ID to cancel

Example Request

curl -X POST http://localhost:30000/v1/responses/resp_abc123def456/cancel

Response

gRPC workers: Returns the cancelled response object (abbreviated):

{
  "id": "resp_abc123def456",
  "object": "response",
  "created_at": 1705312345,
  "completed_at": 1705312349,
  "status": "cancelled",
  "background": true,
  "model": "meta-llama/Llama-3.1-8B-Instruct",
  "output": []
}

Cancelling a response that was not created with background: true returns:

{
  "error": {
    "type": "invalid_request_error",
    "code": "cancellation_not_supported",
    "message": "Only responses created with 'background': true can be cancelled.",
    "param": null
  }
}

HTTP workers: Returns the response object from the backend.


Delete Response

Delete a stored response.

DELETE /v1/responses/{response_id}

Path Parameters

Parameter Type Description
response_id string The response ID to delete

Example Request

curl -X DELETE http://localhost:30000/v1/responses/resp_abc123def456

Response

{
  "id": "resp_abc123def456",
  "object": "response.deleted",
  "deleted": true
}

An unknown ID — or another tenant's response — returns 404 with code not_found.

Deleting a background response that is still queued or in_progress also stops its run: the work is dropped and no result is stored.


List Response Input Items

List the input items that were sent with a response.

GET /v1/responses/{response_id}/input_items

Path Parameters

Parameter Type Description
response_id string The response ID

Example Request

curl http://localhost:30000/v1/responses/resp_abc123def456/input_items

Response

{
  "object": "list",
  "data": [
    {
      "id": "msg_abc123",
      "type": "message",
      "role": "user",
      "content": [{"type": "input_text", "text": "Hello!"}]
    }
  ],
  "first_id": "msg_abc123",
  "last_id": "msg_abc123",
  "has_more": false
}

All stored input items are returned in one page (has_more is always false). Items stored without an ID get a generated msg_ ID.

An unknown ID — or another tenant's response — returns 404 with code not_found.


Conversation Management

Conversations provide persistent storage for multi-turn interactions, enabling chat history to be maintained across multiple requests.

When a request names a conversation, the conversation's stored items are replayed into the model's prompt ahead of the request's own input, in the order they were stored. The replay is bounded: only the newest 100 stored items are sent. Older items stay stored and listable but do not reach the model; to keep context from older turns in play, repeat it in the request's input.

A conversation holding exactly 100 items replays whole; a longer one is cut, and the cut can land inside a turn. The replayed window then begins at its first turn boundary: the first user, developer, or system message or, when the window holds no such message, the first tool call whose output is also in the window, together with the reasoning items immediately before that call. The tail of the turn the cut split is not replayed, and neither is a tool output whose call fell before the window, so a cut never leaves the prompt opening with a tool result whose call is missing.

The conversation endpoints report errors in the gateway envelope ({"error": {"message", "type", "param", "code"}}, see Error Format). An invalid request answers 400 with code invalid_request; an unknown conversation or item answers 404 with code not_found. The one exception is Delete Conversation Item, which answers 200, not 404, for an item that is not in the conversation.

Three 400 rejections carry their own code and param: under Create Conversation Items, item_already_in_conversation for an item the conversation already holds (param is items) and string_above_max_length for an item id over 64 bytes (param is items[].id); under Get Conversation Item, invalid_value for a bad include value (param names the offending entry, such as include[0]).

A conversation is scoped to the tenant that created it: on every route below, and as the conversation of a new response, another tenant's conversation ID answers 404, the same as an unknown ID (see Tenant Scope).

Create Conversation

POST /v1/conversations

Request Body

Field Type Required Description
metadata object No Custom metadata (max 16 properties)

Example Request

curl http://localhost:30000/v1/conversations \
  -H "Content-Type: application/json" \
  -d '{
    "metadata": {
      "project": "customer-support",
      "user_id": "user_123"
    }
  }'

Response

{
  "id": "conv_abc123def456",
  "object": "conversation",
  "created_at": 1705312345,
  "metadata": {
    "project": "customer-support",
    "user_id": "user_123"
  }
}

Get Conversation

GET /v1/conversations/{conversation_id}

Example Request

curl http://localhost:30000/v1/conversations/conv_abc123def456

Response

{
  "id": "conv_abc123def456",
  "object": "conversation",
  "created_at": 1705312345,
  "metadata": {
    "project": "customer-support"
  }
}

Update Conversation

Update conversation metadata. Uses merge semantics — set a key to null to delete it. The merged metadata may hold at most 16 properties.

POST /v1/conversations/{conversation_id}

Request Body

Field Type Description
metadata object Metadata to merge (null values delete keys)

Example Request

curl http://localhost:30000/v1/conversations/conv_abc123def456 \
  -H "Content-Type: application/json" \
  -d '{
    "metadata": {
      "status": "resolved",
      "project": null
    }
  }'

Response

Returns the updated conversation object.


Delete Conversation

DELETE /v1/conversations/{conversation_id}

Example Request

curl -X DELETE http://localhost:30000/v1/conversations/conv_abc123def456

Response

{
  "id": "conv_abc123def456",
  "object": "conversation.deleted",
  "deleted": true
}

List Conversation Items

GET /v1/conversations/{conversation_id}/items

Query Parameters

Parameter Type Default Description
limit integer 100 Maximum items to return
order string desc Sort order: asc or desc
after string - Cursor for pagination

Example Request

curl "http://localhost:30000/v1/conversations/conv_abc123/items?limit=20&order=asc"

Response

{
  "object": "list",
  "data": [
    {
      "id": "msg_abc123",
      "type": "message",
      "role": "user",
      "content": [{"type": "input_text", "text": "Hello"}],
      "status": "completed",
      "created_at": 1705312345
    },
    {
      "id": "msg_def456",
      "type": "message",
      "role": "assistant",
      "content": [{"type": "output_text", "text": "Hi there!"}],
      "status": "completed",
      "created_at": 1705312346
    }
  ],
  "has_more": false,
  "first_id": "msg_abc123",
  "last_id": "msg_def456"
}

has_more is true when the page is full (it holds limit items).


Create Conversation Items

Add items to a conversation. Maximum 20 items per request.

POST /v1/conversations/{conversation_id}/items

Request Body

Field Type Required Description
items array Yes Array of items to add (max 20)

Supported Item Types

These item types are fully supported:

  • message - User or assistant messages (requires role); an item without type is a message
  • reasoning - Model reasoning content
  • mcp_list_tools - MCP tool listing
  • mcp_call - MCP tool invocation
  • item_reference - Reference to an existing item by id; an unknown ID — or another tenant's item — returns 404

These types are stored, but the response adds a warnings entry because SMG does not implement them yet: function_call, function_call_output, file_search_call, computer_call, computer_call_output, web_search_call, image_generation_call, code_interpreter_call, local_shell_call, local_shell_call_output, mcp_approval_request, mcp_approval_response, custom_tool_call, and custom_tool_call_output. Any other type is rejected with 400.

An item whose id is already in the conversation is rejected with:

{
  "error": {
    "message": "Item already in conversation",
    "type": "invalid_request_error",
    "param": "items",
    "code": "item_already_in_conversation"
  }
}

An item submitted with an explicit id that already names a stored item the caller may see — its own, or one stored without a tenant — reuses that stored item as-is. An ID that belongs to another tenant's item is rejected with 400 (Item id '...' cannot be used), neither serving nor overwriting it (see Tenant Scope).

A supplied item id — an item's own or an item_reference's — may be at most 64 bytes; a longer one is rejected with 400 (string_above_max_length on items[].id) before any item of the request is stored.

Example Request

curl http://localhost:30000/v1/conversations/conv_abc123/items \
  -H "Content-Type: application/json" \
  -d '{
    "items": [
      {
        "type": "message",
        "role": "user",
        "content": [{"type": "input_text", "text": "What is 2+2?"}]
      },
      {
        "type": "message",
        "role": "assistant",
        "content": [{"type": "output_text", "text": "2+2 equals 4."}]
      }
    ]
  }'

Response

{
  "object": "list",
  "data": [
    {
      "id": "msg_abc123",
      "type": "message",
      "role": "user",
      "content": [{"type": "input_text", "text": "What is 2+2?"}],
      "status": "completed"
    },
    {
      "id": "msg_def456",
      "type": "message",
      "role": "assistant",
      "content": [{"type": "output_text", "text": "2+2 equals 4."}],
      "status": "completed"
    }
  ],
  "first_id": "msg_abc123",
  "last_id": "msg_def456",
  "has_more": false
}

Get Conversation Item

GET /v1/conversations/{conversation_id}/items/{item_id}

An optional include query parameter (repeatable, also accepted as include[]) must name includable fields: code_interpreter_call.outputs, computer_call_output.output.image_url, file_search_call.results, message.input_image.image_url, message.output_text.logprobs, reasoning.encrypted_content, web_search_call.action.sources, or web_search_call.results. Any other value is rejected with 400 (invalid_value); a valid one is accepted but does not change the returned item.

Example Request

curl http://localhost:30000/v1/conversations/conv_abc123/items/msg_abc123

Response

Returns the item object. An item that exists but is not in this conversation returns 404.


Delete Conversation Item

Remove an item from a conversation. This performs a soft delete — the item may still exist if referenced by other conversations.

DELETE /v1/conversations/{conversation_id}/items/{item_id}

Example Request

curl -X DELETE http://localhost:30000/v1/conversations/conv_abc123/items/msg_abc123

Response

Returns the conversation object. An item that is not in the conversation, including an unknown item ID, is ignored: the request answers 200, not 404. An unknown conversation still answers 404.


Examples

Simple Agentic Workflow

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:30000/v1",
    api_key="your-api-key"
)

# Create a response with MCP tools
response = client.responses.create(
    model="meta-llama/Llama-3.1-8B-Instruct",
    input="Search for the weather in San Francisco and summarize it",
    tools=[
        {
            "type": "mcp",
            "server_url": "http://localhost:8080/mcp",
            "server_label": "weather-service"
        }
    ],
    tool_choice="auto"
)

# The response includes tool calls and final answer
for output in response.output:
    if output.type == "mcp_call":
        print(f"Tool called: {output.name} ({output.status})")
        print(f"Result: {output.output}")
    elif output.type == "message":
        for content in output.content:
            if content.type == "output_text":
                print(f"Answer: {content.text}")

Multi-turn Conversation with Tools

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:30000/v1",
    api_key="your-api-key"
)

# Create a conversation
conversation = client.conversations.create(
    metadata={"session": "support-123"}
)

# First turn
response1 = client.responses.create(
    model="meta-llama/Llama-3.1-8B-Instruct",
    input="I need help with my order #12345",
    conversation=conversation.id,
    tools=[
        {
            "type": "mcp",
            "server_url": "http://localhost:8080/mcp",
            "server_label": "order-service"
        }
    ]
)
print(f"First response: {response1.id}")

# Second turn - continues the conversation
response2 = client.responses.create(
    model="meta-llama/Llama-3.1-8B-Instruct",
    input="Can you also check if there are any discounts available?",
    conversation=conversation.id,
    tools=[
        {
            "type": "mcp",
            "server_url": "http://localhost:8080/mcp",
            "server_label": "order-service"
        }
    ]
)
print(f"Second response: {response2.id}")

# List conversation history (tool items have no role)
items = client.conversations.items.list(conversation.id)
for item in items.data:
    print(item.type, getattr(item, "role", None))

Streaming Response Handling

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:30000/v1",
    api_key="your-api-key"
)

# Stream a response
with client.responses.create(
    model="meta-llama/Llama-3.1-8B-Instruct",
    input="Explain quantum computing",
    stream=True
) as stream:
    for event in stream:
        if event.type == "response.output_text.delta":
            print(event.delta, end="", flush=True)
        elif event.type == "response.mcp_call.in_progress":
            print(f"\n[Calling tool: {event.item_id}]")
        elif event.type == "response.completed":
            print(f"\n\nTokens used: {event.response.usage.total_tokens}")
        elif event.type in ("response.incomplete", "response.failed"):
            print(f"\n\nStopped: {event.response.status}")

Using Previous Response ID

# Alternative to conversations - chain responses directly
response1 = client.responses.create(
    model="meta-llama/Llama-3.1-8B-Instruct",
    input="What are the main programming paradigms?",
    store=True
)

# Continue from previous response
response2 = client.responses.create(
    model="meta-llama/Llama-3.1-8B-Instruct",
    input="Can you elaborate on functional programming?",
    previous_response_id=response1.id,
    store=True
)

A chained turn does not have to add a message: after a response ends in function_call items, the next request's input can be only their function_call_output items (see Input Formats).


Error Responses

Error Format

Errors that SMG generates use the gateway envelope, with the public API's error type as type (invalid_request_error for a 4xx, the detail in code; permission_error for a 403; rate_limit_error for a 429; server_error for a 5xx) and the code repeated in the X-SMG-Error-Code header:

{
  "error": {
    "type": "invalid_request_error",
    "code": "conversation_not_found",
    "message": "Conversation 'conv_abc123' not found. Please create the conversation first using the conversations API.",
    "param": null
  }
}

Request parsing and validation failures use the same envelope without the header. Their code names the failure: invalid_json for a syntax error, a schema code such as unknown_parameter or invalid_type for a body that does not match the request type, a bound code such as integer_above_max_value for a field outside its range, or the violated rule's own code such as invalid_conversation_id. param names the offending field when one is known. See Error Responses for the full vocabulary.

Common Errors

HTTP Status code Description
400 The failure's code (see Error Format) Malformed request or validation failure: invalid_json, a schema or bound code (unknown_parameter, invalid_type, integer_above_max_value, ...), or the violated rule's code (invalid_conversation_id, mutually_exclusive_parameters, ...)
400 previous_response_not_found previous_response_id is not in SMG's storage, or is another tenant's response
400 convert_request_failed gRPC: an input item the chat pipeline cannot represent, such as media in a function_call_output
400 response_already_completed, response_already_failed, cancellation_not_supported Cancel refused; see Cancel Response
401 invalid_api_key Invalid or missing API key: the error envelope with a WWW-Authenticate challenge, without the X-SMG-Error-Code header
404 model_not_found No worker serves the model
404 conversation_not_found The conversation does not exist, or belongs to another tenant
404 not_found Unknown or another tenant's ID: a response on get, delete, or input items, or a conversation or item on the conversation routes
404 response_not_found Unknown or another tenant's response ID on cancel
424 connect_mcp_server_failed gRPC: no MCP server declared by the request could be connected; an unreachable server is otherwise skipped with a warning
429 admission_queue_full, scheduler_queue_full Admission control; see Rate Limiting
429 background_responses_limit_reached The gateway already runs --max-background-responses background responses; see Background Mode
500 load_previous_response_chain_failed, check_conversation_failed, storage_error, store_response_failed A failure the history store itself answered: a refused statement, a record that cannot be decoded, an invalid chain, or a refused write of a finished response (store_response_failed: the response was generated but nothing of it was stored). Retrying does not help
503 storage_unavailable The history store is down or did not answer within its per-call bound (--postgres-op-timeout-secs / --redis-op-timeout-secs, default 5 seconds); retry once the store answers again. See Store Failures While Serving
503 no_available_workers No healthy workers available

Validation Errors

{
  "error": {
    "message": "conversation: Invalid 'conversation': 'invalid-id'. Expected an ID that begins with 'conv_'.",
    "type": "invalid_request_error",
    "param": "conversation",
    "code": "invalid_conversation_id"
  }
}
{
  "error": {
    "message": "Mutually exclusive parameters. Ensure you are only providing one of: 'previous_response_id' or 'conversation'.",
    "type": "invalid_request_error",
    "param": null,
    "code": "mutually_exclusive_parameters"
  }
}

SGLang Extensions

The Responses API includes additional sampling parameters specific to SGLang:

Field Type Default Description
top_k integer -1 Top-k sampling (-1 = disabled)
min_p number 0.0 Min-p sampling threshold
repetition_penalty number 1.0 Repetition penalty (1.0 = disabled)
frequency_penalty number - OpenAI-compatible frequency penalty
presence_penalty number - OpenAI-compatible presence penalty
stop string/array - Stop sequences

Where they take effect:

  • HTTP workers receive them with the request.
  • gpt-oss (Harmony) models on gRPC: all six fields are passed to the engine on every model turn of the tool loop. Where the serving engine cannot match strings, SMG matches stop sequences itself, so a stop takes effect on every backend except MLX, where a request with string stop sequences gets a 400 (see gRPC Workers).
  • Other models on gRPC: all six fields are passed to the engine on every model turn of the tool loop; which of them affect sampling is up to the serving engine. top_k, min_p, and repetition_penalty sent at their default values (-1, 0.0, and 1.0 respectively) count as unset, the same as omitting them: the model's own sampling defaults apply. A matched stop sequence ends generation and is trimmed from the output, as on Chat Completions.

On gpt-oss (Harmony) models on gRPC, the first stop match ends the model turn: everything generated past the match, tool calls included, is dropped, and the matched sequence stays as the suffix of the output — it is not trimmed, unlike on other models. A stop can match inside the model's reasoning, ending the turn before any message, so the response may carry a reasoning item and no message. A turn ended by stop completes normally: "status": "completed" with no incomplete_details.

Example:

{
  "model": "meta-llama/Llama-3.1-8B-Instruct",
  "input": "Write a story",
  "top_k": 50,
  "min_p": 0.05,
  "repetition_penalty": 1.1
}