Tokenization and Parsing APIs¶
SMG exposes utility endpoints for tokenization, detokenization, function-call parsing, and reasoning separation. They use the gateway's own tokenizers and the same legacy parser registries as the gRPC pipeline. You can use them to see exactly how SMG tokenizes a prompt or splits a model's raw output.
| Method | Path | Purpose | Auth |
|---|---|---|---|
POST |
/v1/tokenize |
Convert text to token IDs | Serving API key |
POST |
/v1/detokenize |
Convert token IDs to text | Serving API key |
POST |
/parse/function_call |
Extract tool calls from model output | Control plane |
POST |
/parse/reasoning |
Separate reasoning from the final answer | Control plane |
Tokenize¶
POST /v1/tokenize
| Field | Type | Description |
|---|---|---|
model |
string | Name of a loaded tokenizer (required) |
prompt |
string or array of strings | Text to tokenize (required) |
model is the name a tokenizer was loaded under: a worker's model ID, the --model-path or --tokenizer-path value the gateway started with, or the name given to POST /v1/tokenizers. A tokenizer ID also works, and an empty string or "unknown" picks the loaded tokenizer whose name sorts first. The endpoint doesn't add special tokens such as BOS.
Single input:
curl http://localhost:30000/v1/tokenize \
-H "Content-Type: application/json" \
-d '{
"model": "Qwen/Qwen3-4B-Instruct-2507",
"prompt": "Hello world"
}'{"tokens": [9707, 1879], "count": 2, "char_count": 11}Batch input:
curl http://localhost:30000/v1/tokenize \
-H "Content-Type: application/json" \
-d '{
"model": "Qwen/Qwen3-4B-Instruct-2507",
"prompt": ["Hello", "World"]
}'{"tokens": [[9707], [10134]], "count": [1, 1], "char_count": [5, 5]}| Response field | Description |
|---|---|
tokens |
Token IDs; one array per input for a batch |
count |
Number of tokens; an array for a batch |
char_count |
Number of characters in the input; an array for a batch |
Detokenize¶
POST /v1/detokenize
| Field | Type | Description |
|---|---|---|
model |
string | Name of a loaded tokenizer (required) |
tokens |
array of IDs, or array of arrays for a batch | Token IDs to decode (required) |
skip_special_tokens |
boolean | Drop special tokens from the output. Default true |
curl http://localhost:30000/v1/detokenize \
-H "Content-Type: application/json" \
-d '{
"model": "Qwen/Qwen3-4B-Instruct-2507",
"tokens": [9707, 1879],
"skip_special_tokens": true
}'{"text": "Hello world"}A batch request returns text as an array of strings. For tiktoken-based tokenizers such as Kimi's, skip_special_tokens drops only the tokens marked "special": true in tokenizer_config.json. Control markers without that flag stay in the text.
Errors¶
Errors use the body {"error": {"message": "...", "type": "..."}}:
| Status | type |
Cause |
|---|---|---|
| 400 | tokenizer_not_found |
No tokenizer is loaded under that name (the message lists the loaded names), or none is loaded at all |
| 500 | tokenization_error / detokenization_error |
The tokenizer failed on the input |
Parse Function Calls¶
POST /parse/function_call
Runs one tool-call parser over complete model output, the same way the gateway parses a non-streaming response under --parser legacy.
The serving pipeline reads with Symphony tables by default, but the --parser flag and the X-SMG-Parser header change only that pipeline. This endpoint always runs the legacy parser named in tool_call_parser, never a Symphony table; a name that only a table answers to is rejected as unknown.
| Field | Type | Description |
|---|---|---|
text |
string | Raw model output (required) |
tool_call_parser |
string | Registered parser name (required) |
tools |
array | Tool definitions in OpenAI format (required; may be empty). Schema-aware parsers use them to convert argument types |
tool_call_parser accepts the 26 registered names: passthrough, json, gemma4, mistral, qwen, qwen_xml, qwen_coder, nemotron, pythonic, llama, deepseek, deepseek31, deepseek32, deepseek_v4, deepseek_v41, glm45_moe, glm47_moe, step3, sarashina, kimik2, kimi_k3, inkling, minimax_m2, minimax_m3, hy_v4, cohere. The gRPC Pipeline reference shows each parser's format.
If control-plane auth is configured, include an admin bearer token.
curl http://localhost:30000/parse/function_call \
-H "Content-Type: application/json" \
-H "Authorization: Bearer ${ADMIN_TOKEN}" \
-d '{
"text": "{\"name\":\"get_weather\",\"arguments\":{\"city\":\"SF\"}}",
"tool_call_parser": "json",
"tools": [
{
"type": "function",
"function": {
"name": "get_weather",
"description": "Get weather",
"parameters": {
"type": "object",
"properties": {
"city": {"type": "string"}
},
"required": ["city"]
}
}
}
]
}'{
"remaining_text": "",
"tool_calls": [
{"function": {"name": "get_weather", "arguments": "{\"city\":\"SF\"}"}}
],
"success": true
}remaining_text is the output with the tool calls taken out, and each arguments value is a JSON string. When the text holds no tool call, tool_calls is empty and remaining_text is the whole text. An unknown parser name returns 400 with {"error": "Unknown tool parser: <name>", "success": false}, and a parser failure returns 400 with a Failed to parse function calls message.
Parse Reasoning¶
POST /parse/reasoning
Runs one reasoning parser over complete model output. Like /parse/function_call, it always runs the legacy parser named in reasoning_parser, never a Symphony table; the --parser flag (symphony by default) and the X-SMG-Parser header change only the serving pipeline.
| Field | Type | Description |
|---|---|---|
text |
string | Raw model output (required) |
reasoning_parser |
string | Registered parser name (required) |
reasoning_parser accepts the 21 registered names: base, passthrough, deepseek_r1, deepseek_v31, deepseek_v4, deepseek_v41, qwen3, qwen3_thinking, gemma4, glm45, hy_v4, step3, kimi, kimi_k25, kimi_thinking, kimi_k3, minimax, minimax_m3, cohere_cmd, nano_v3, inkling.
If control-plane auth is configured, include an admin bearer token.
curl http://localhost:30000/parse/reasoning \
-H "Content-Type: application/json" \
-H "Authorization: Bearer ${ADMIN_TOKEN}" \
-d '{
"text": "<think>internal reasoning</think>Final answer",
"reasoning_parser": "deepseek_r1"
}'{"normal_text": "Final answer", "reasoning_text": "internal reasoning", "success": true}Each request gets a fresh parser in its default state, with no prompt behind it. Parsers that start in reasoning (deepseek_r1, qwen3_thinking, kimi_thinking, step3, minimax) treat text before the end marker as reasoning even without a start marker. kimi_k3 starts in content mode but doesn't need its start marker either: raw Kimi-K3 output carries <|close|>think<|sep|> with the open marker left in the prompt, and the parser still splits there — reasoning before the close, the answer after it. The others need the start marker, such as <think>, in text. The serving pipeline differs: there, a request with thinking on starts the parser in reasoning mode. An unknown parser name returns 400 with {"error": "Unknown reasoning parser: <name>", "success": false}.
Auth Notes¶
/v1/tokenizeand/v1/detokenizeare serving routes. When--api-keyor--tenant-api-keyis set, send one of those keys as a bearer token. The same admission control as inference requests applies (concurrency limit or priority scheduler)./parse/function_calland/parse/reasoningare control-plane routes. With control-plane auth configured (--control-plane-api-keys, or--jwt-issuerwith--jwt-audience), they need a bearer credential with the admin role. Without it, they accept only the shared--api-key, never a tenant key. If only--tenant-api-keyis set, they reject every request with 401, and with no keys configured at all they're open.
Next Steps¶
- gRPC Pipeline — Parser formats, auto-detection, and per-model parser overrides
- Gateway Extensions API — Route groups for tokenizer management and parser utilities
- Admin API Reference — Add, list, and remove tokenizers with
/v1/tokenizers