Tokenization and Parsing APIs

SMG exposes utility endpoints for tokenization, detokenization, function-call parsing, and reasoning separation. They use the gateway's own tokenizers and the same legacy parser registries as the gRPC pipeline. You can use them to see exactly how SMG tokenizes a prompt or splits a model's raw output.

Method Path Purpose Auth
POST /v1/tokenize Convert text to token IDs Serving API key
POST /v1/detokenize Convert token IDs to text Serving API key
POST /parse/function_call Extract tool calls from model output Control plane
POST /parse/reasoning Separate reasoning from the final answer Control plane

Tokenize

POST /v1/tokenize

Field Type Description
model string Name of a loaded tokenizer (required)
prompt string or array of strings Text to tokenize (required)

model is the name a tokenizer was loaded under: a worker's model ID, the --model-path or --tokenizer-path value the gateway started with, or the name given to POST /v1/tokenizers. A tokenizer ID also works, and an empty string or "unknown" picks the loaded tokenizer whose name sorts first. The endpoint doesn't add special tokens such as BOS.

Single input:

curl http://localhost:30000/v1/tokenize \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen/Qwen3-4B-Instruct-2507",
    "prompt": "Hello world"
  }'
{"tokens": [9707, 1879], "count": 2, "char_count": 11}

Batch input:

curl http://localhost:30000/v1/tokenize \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen/Qwen3-4B-Instruct-2507",
    "prompt": ["Hello", "World"]
  }'
{"tokens": [[9707], [10134]], "count": [1, 1], "char_count": [5, 5]}
Response field Description
tokens Token IDs; one array per input for a batch
count Number of tokens; an array for a batch
char_count Number of characters in the input; an array for a batch

Detokenize

POST /v1/detokenize

Field Type Description
model string Name of a loaded tokenizer (required)
tokens array of IDs, or array of arrays for a batch Token IDs to decode (required)
skip_special_tokens boolean Drop special tokens from the output. Default true
curl http://localhost:30000/v1/detokenize \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen/Qwen3-4B-Instruct-2507",
    "tokens": [9707, 1879],
    "skip_special_tokens": true
  }'
{"text": "Hello world"}

A batch request returns text as an array of strings. For tiktoken-based tokenizers such as Kimi's, skip_special_tokens drops only the tokens marked "special": true in tokenizer_config.json. Control markers without that flag stay in the text.

Errors

Errors use the body {"error": {"message": "...", "type": "..."}}:

Status type Cause
400 tokenizer_not_found No tokenizer is loaded under that name (the message lists the loaded names), or none is loaded at all
500 tokenization_error / detokenization_error The tokenizer failed on the input

Parse Function Calls

POST /parse/function_call

Runs one tool-call parser over complete model output, the same way the gateway parses a non-streaming response under --parser legacy.

The serving pipeline reads with Symphony tables by default, but the --parser flag and the X-SMG-Parser header change only that pipeline. This endpoint always runs the legacy parser named in tool_call_parser, never a Symphony table; a name that only a table answers to is rejected as unknown.

Field Type Description
text string Raw model output (required)
tool_call_parser string Registered parser name (required)
tools array Tool definitions in OpenAI format (required; may be empty). Schema-aware parsers use them to convert argument types

tool_call_parser accepts the 26 registered names: passthrough, json, gemma4, mistral, qwen, qwen_xml, qwen_coder, nemotron, pythonic, llama, deepseek, deepseek31, deepseek32, deepseek_v4, deepseek_v41, glm45_moe, glm47_moe, step3, sarashina, kimik2, kimi_k3, inkling, minimax_m2, minimax_m3, hy_v4, cohere. The gRPC Pipeline reference shows each parser's format.

If control-plane auth is configured, include an admin bearer token.

curl http://localhost:30000/parse/function_call \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer ${ADMIN_TOKEN}" \
  -d '{
    "text": "{\"name\":\"get_weather\",\"arguments\":{\"city\":\"SF\"}}",
    "tool_call_parser": "json",
    "tools": [
      {
        "type": "function",
        "function": {
          "name": "get_weather",
          "description": "Get weather",
          "parameters": {
            "type": "object",
            "properties": {
              "city": {"type": "string"}
            },
            "required": ["city"]
          }
        }
      }
    ]
  }'
{
  "remaining_text": "",
  "tool_calls": [
    {"function": {"name": "get_weather", "arguments": "{\"city\":\"SF\"}"}}
  ],
  "success": true
}

remaining_text is the output with the tool calls taken out, and each arguments value is a JSON string. When the text holds no tool call, tool_calls is empty and remaining_text is the whole text. An unknown parser name returns 400 with {"error": "Unknown tool parser: <name>", "success": false}, and a parser failure returns 400 with a Failed to parse function calls message.


Parse Reasoning

POST /parse/reasoning

Runs one reasoning parser over complete model output. Like /parse/function_call, it always runs the legacy parser named in reasoning_parser, never a Symphony table; the --parser flag (symphony by default) and the X-SMG-Parser header change only the serving pipeline.

Field Type Description
text string Raw model output (required)
reasoning_parser string Registered parser name (required)

reasoning_parser accepts the 21 registered names: base, passthrough, deepseek_r1, deepseek_v31, deepseek_v4, deepseek_v41, qwen3, qwen3_thinking, gemma4, glm45, hy_v4, step3, kimi, kimi_k25, kimi_thinking, kimi_k3, minimax, minimax_m3, cohere_cmd, nano_v3, inkling.

If control-plane auth is configured, include an admin bearer token.

curl http://localhost:30000/parse/reasoning \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer ${ADMIN_TOKEN}" \
  -d '{
    "text": "<think>internal reasoning</think>Final answer",
    "reasoning_parser": "deepseek_r1"
  }'
{"normal_text": "Final answer", "reasoning_text": "internal reasoning", "success": true}

Each request gets a fresh parser in its default state, with no prompt behind it. Parsers that start in reasoning (deepseek_r1, qwen3_thinking, kimi_thinking, step3, minimax) treat text before the end marker as reasoning even without a start marker. kimi_k3 starts in content mode but doesn't need its start marker either: raw Kimi-K3 output carries <|close|>think<|sep|> with the open marker left in the prompt, and the parser still splits there — reasoning before the close, the answer after it. The others need the start marker, such as <think>, in text. The serving pipeline differs: there, a request with thinking on starts the parser in reasoning mode. An unknown parser name returns 400 with {"error": "Unknown reasoning parser: <name>", "success": false}.


Auth Notes

  • /v1/tokenize and /v1/detokenize are serving routes. When --api-key or --tenant-api-key is set, send one of those keys as a bearer token. The same admission control as inference requests applies (concurrency limit or priority scheduler).
  • /parse/function_call and /parse/reasoning are control-plane routes. With control-plane auth configured (--control-plane-api-keys, or --jwt-issuer with --jwt-audience), they need a bearer credential with the admin role. Without it, they accept only the shared --api-key, never a tenant key. If only --tenant-api-key is set, they reject every request with 401, and with no keys configured at all they're open.

Next Steps