Model Recipes

These recipes start from a known model family, pick the hardware topology, then set only the parameters that change runtime behavior.

The commands below are templates. Validate exact model IDs, checkpoint formats, and backend choices against the build you deploy.

Inkling

Blog: https://lightseek.org/blog/tokenspeed-inkling.html

## Docker

### nvidia
docker pull lightseekorg/tokenspeed:latest
### amd
docker pull lightseekorg/tokenspeed-amd:latest

## Launch command

# nvidia
ts serve \
    --model thinkingmachines/Inkling-NVFP4 \
    --attn-tp-size 4 \
    --moe-tp-size 4 \
    --max-model-len 81920 \
    --max-num-seqs 16 \
    --max-prefill-tokens 8192 \
    --chunked-prefill-size 8192 \
    --gpu-memory-utilization 0.95 \
    --disable-cuda-graph-padding \
    --trust-remote-code \
    --attention-backend fa4 \
    --moe-backend flashinfer_trtllm \
    --enable-prefix-caching \
    --disable-kvstore \
    --block-size 128 \
    --enable-cache-report \
    --speculative-algorithm MTP \
    --speculative-num-steps 3 \
    --speculative-eagle-topk 1 \
    --speculative-num-draft-tokens 4

# amd
ts serve \
    --model lightseekorg/Inkling-MXFP4 \
    --attn-tp-size 4 \
    --moe-tp-size 4 \
    --max-model-len 81920 \
    --max-num-seqs 16 \
    --max-prefill-tokens 8192 \
    --chunked-prefill-size 8192 \
    --gpu-memory-utilization 0.95 \
    --disable-cuda-graph-padding \
    --trust-remote-code \
    --enable-prefix-caching \
    --disable-kvstore \
    --block-size 128 \
    --enable-cache-report \
    --speculative-algorithm MTP \
    --speculative-num-steps 3 \
    --speculative-eagle-topk 1 \
    --speculative-num-draft-tokens 4

MiniMax M3

MiniMax M3 uses 128-token MSA blocks. TokenSpeed configures its dense and sparse attention layers automatically; select the dense backend with --attention-backend and run with --disable-kvstore.

tokenspeed serve nvidia/MiniMax-M3-NVFP4 \
    --tensor-parallel-size 4 \
    --max-model-len 81920 \
    --max-num-seqs 16 \
    --max-prefill-tokens 8192 \
    --chunked-prefill-size 8192 \
    --gpu-memory-utilization 0.95 \
    --disable-cuda-graph-padding \
    --attention-backend trtllm \
    --kv-cache-dtype fp8 \
    --moe-backend flashinfer_trtllm \
    --speculative-algorithm EAGLE3 \
    --speculative-draft-model-path Inferact/MiniMax-M3-EAGLE3 \
    --speculative-num-steps 3 \
    --speculative-eagle-topk 1 \
    --speculative-num-draft-tokens 4 \
    --disable-kvstore \
    --block-size 128 \
    --trust-remote-code \
    --host 0.0.0.0 \
    --port 8000

Kimi K2.5 / K2.6

Kimi-style MoE launches usually need remote code, long context, reasoning and tool parsers, and explicit MLA/MoE backends.

tokenspeed serve nvidia/Kimi-K2.5-NVFP4 \
  --served-model-name kimi-k2.5 \
  --trust-remote-code \
  --max-model-len 262144 \
  --kv-cache-dtype fp8 \
  --quantization nvfp4 \
  --tensor-parallel-size 4 \
  --enable-expert-parallel \
  --chunked-prefill-size 8192 \
  --max-num-seqs 256 \
  --attention-backend trtllm_mla \
  --moe-backend flashinfer_trtllm \
  --reasoning-parser kimi_k25 \
  --tool-call-parser kimik2 \
  --host 0.0.0.0 \
  --port 8000

For K2.6, keep the same parameter shape and change the checkpoint and parser only if the model card requires a different value.

To enable a compatible DFlash draft model, keep the target launch shape and add the draft model path plus DFlash speculative decoding options:

tokenspeed serve nvidia/Kimi-K2.6-NVFP4 \
  --served-model-name kimi-k2.6 \
  --trust-remote-code \
  --max-model-len 262144 \
  --kv-cache-dtype fp8 \
  --quantization nvfp4 \
  --tensor-parallel-size 4 \
  --enable-expert-parallel \
  --chunked-prefill-size 8192 \
  --max-num-seqs 256 \
  --attention-backend tokenspeed_mla \
  --moe-backend flashinfer_trtllm \
  --reasoning-parser kimi_k25 \
  --tool-call-parser kimik2 \
  --speculative-algorithm DFLASH \
  --speculative-draft-model-path /path/to/kimi-k2.6-dflash \
  --speculative-num-draft-tokens 8 \
  --speculative-num-steps 7 \
  --drafter-attention-backend fa4 \
  --host 0.0.0.0 \
  --port 8000

Known limitation: native TokenSpeed DFlash currently uses full-history draft attention. It does not yet expose an equivalent of SGLang's --speculative-dflash-draft-window-size; add such a flag before relying on bounded draft attention for long-context deployments.

Kimi K3

Kimi-K3 combines a MoonViT vision encoder with a hybrid KDA (linear-attention) / NoPE-MLA (full-attention) decoder and a DeepSeek-V3-style latent MoE. The KDA layers currently use flash-linear-attention kernels on NVIDIA, so install it first:

pip install flash-linear-attention

Notes:

• K3 uses the grouped paged-cache scheduler and KDA state groups.
• KDA dispatch is vendor-neutral at the runtime boundary. The kernel registry selects the existing FLA-derived NVIDIA implementation or the native AMD implementation, including each backend's preferred recurrent-state layout. The runtime does not transpose or reinterpret that state.
• NVIDIA auto-selects --attention-backend tokenspeed_mla for K3 (fp8 KV required). AMD uses the mla backend.
• tokenspeed serve auto-selects the kimi_k3 reasoning and tool-call parsers. Explicit parser flags override these defaults.
• The SMG packages pinned by TokenSpeed resolve moonshotai/Kimi-K3 directly; a flattened local checkpoint and separately staged remote-code cache are no longer required.
• The checkpoint carries no FP8 KV scaling factors. When the target K3 uses its required FP8 LCM cache, TokenSpeed keeps the separate K3 DSpark draft cache in BF16 so context injection and draft attention match the reference precision.
• DSpark proposal blocks use non-causal MLA draft attention. Both the mla and trtllm_mla draft backends preserve every block row during eager execution and CUDA graph capture. When K3's 128-token logical cache pages feed the 64-token TRT-LLM MLA kernel, the backend expands each logical page into its two physical kernel pages before draft attention.
• For Kimi K3, an eight-token verify window uses seven DSpark draft queries. The anchor query directly predicts the first draft through the Markov head; it must not be padded with an eighth, unused mask row.
• Target features are captured from K3's completed-layer prefix stream before the model-level AttnRes mix and final norm, matching the DSpark checkpoint's vLLM training and inference contract.
• Under tensor parallelism, the draft's final row-parallel MLP output is reduced across TP ranks before final_norm and shared target-head sampling.
• The vision encoder has 12 attention heads. For an 8-way text TP deployment, use --mm-encoder-tp-mode data so each rank runs the vision encoder at TP1 on a different whole image.
• The pinned SMG frontend registers Kimi-K3's chat renderer and multimodal processor. Preserve the checkpoint's media_proc_cfg.in_patch_limit=65536; silently falling back to K2.5's 16384-patch default reduces OCR resolution.
• KDA recurrent-state pages register for prefix-cache reuse only when a prefill chunk ends exactly on a logical cache-page boundary. The engine floors --chunked-prefill-size to the plan's page grain automatically (logged as a warning when it adjusts); the page grain is budget-dependent (e.g. 1472 at 32k context, 1536 at 1M), so do not hand-tune the chunk size against a hard-coded page value. Prefix hits are page-granular.

NVIDIA

Serve with expert parallelism (recommended) on 8x B300:

tokenspeed serve moonshotai/Kimi-K3 \
  --served-model-name kimi-k3 \
  --trust-remote-code \
  --max-model-len 32768 \
  --kv-cache-dtype fp8 \
  --tensor-parallel-size 8 \
  --mm-encoder-tp-mode data \
  --ep-size 8 \
  --moe-backend flashinfer_trtllm \
  --gpu-memory-utilization 0.94 \
  --max-num-seqs 32 \
  --disable-kvstore \
  --host 0.0.0.0 \
  --port 8000

Plain TP8 (drop --ep-size 8) works too. The fused MoE path needs a Blackwell GPU (B200/B300); on other NVIDIA platforms use --moe-backend triton.

AMD

The standard AMD path on 8x gfx950 uses the mla backend. For TP8/EP8, automatic MoE selection uses the specialized Gluon SiTU kernels:

tokenspeed serve moonshotai/Kimi-K3 \
  --served-model-name kimi-k3 \
  --trust-remote-code \
  --max-model-len 8192 \
  --kv-cache-dtype fp8 \
  --tensor-parallel-size 8 \
  --mm-encoder-tp-mode data \
  --enable-expert-parallel \
  --attention-backend mla \
  --moe-backend auto \
  --gpu-memory-utilization 0.92 \
  --max-num-seqs 32 \
  --disable-kvstore \
  --host 0.0.0.0 \
  --port 8000

On gfx950, the replicated 7168↔3584 latent projections automatically select among a one-token Triton GEMV, tuned Gluon GEMMs, and the vendor GEMM according to the current token count. At TP8/EP8, eligible one-token decode also combines the routed MXFP4 experts with the shared-expert down projection, then applies their joint reduction before the fused latent up-projection epilogue. Other shapes and unsupported layouts retain the ordinary composed path. The fused sigmoid-bias top-k route supports the full scheduled token count.

GLM5 / GLM5.2

GLM5 launches usually need remote code, long context, expert parallelism, FP8 KV cache, and the TRTLLM MoE backend. GLM5.2 FP8 is available on Hugging Face as zai-org/GLM-5.2-FP8. TokenSpeed defaults the reasoning parser to glm45; pass an explicit parser flag to override it.

tokenspeed serve zai-org/GLM-5.2-FP8 \
  --served-model-name glm-5.2 \
  --trust-remote-code \
  --tensor-parallel-size 8 \
  --enable-expert-parallel \
  --moe-backend flashinfer_trtllm \
  --kv-cache-dtype fp8 \
  --max-model-len 262144 \
  --chunked-prefill-size 8192 \
  --max-num-seqs 128 \
  --host 0.0.0.0 \
  --port 8000

Qwen3 Dense / Qwen3 30B-A3B

Qwen2, dense Qwen3, and Qwen3 MoE checkpoints use different architecture names. For Qwen3 30B-A3B, the Hugging Face config advertises qwen3_moe and Qwen3MoeForCausalLM, so launch it as a MoE model.

tokenspeed serve Qwen/Qwen3-30B-A3B \
  --served-model-name qwen3-30b-a3b \
  --tensor-parallel-size 2 \
  --enable-expert-parallel \
  --moe-backend flashinfer_cutlass \
  --max-model-len 40960 \
  --reasoning-parser qwen3 \
  --host 0.0.0.0 \
  --port 8000

Qwen3.8

Qwen3.8 shares the hybrid linear-attention (GDN) / full-attention layer pattern with Qwen3.5.

Qwen3.8-Max

Qwen3.8-max needs 16 GPUs, so it runs on two 8-GPU nodes. Launch tokenspeed serve on every node with the same command, changing only --node-rank; every node points --dist-init-addr at node 0, which is the only rank that serves the HTTP API. See Parallelism for the multi-node rules.

This family has no parser auto-selection, so set --reasoning-parser and --tool-call-parser explicitly. --speculative-algorithm MTP without --speculative-draft-model-path drafts from the base checkpoint. Set --dist-init-addr to node 0's own address and port throughout (<node0-host>:25000 below).

TP16

One replica across both nodes. --ep-size defaults to 1, so the experts stay tensor-parallel over the full world and all-to-all stays out of the path. Keep --moe-backend auto: the block-scale FP8 deep_gemm experts implement only the DeepEP legs and are unavailable without --all2all-backend deepep.

# node 0 (serves the HTTP API)
tokenspeed serve /path/to/qwen3.8-max-fp8 \
  --served-model-name qwen3.8-max \
  --nnodes 2 --node-rank 0 --nprocs-per-node 8 --world-size 16 \
  --dist-init-addr <node0-host>:25000 \
  --attn-tp-size 16 \
  --moe-backend auto \
  --quantization fp8 --kv-cache-dtype fp8 \
  --attention-backend trtllm \
  --chunked-prefill-size 8192 \
  --gpu-memory-utilization 0.95 --max-num-seqs 128 \
  --disable-kvstore \
  --speculative-algorithm MTP --speculative-num-steps 3 \
  --speculative-eagle-topk 1 --speculative-num-draft-tokens 4 \
  --reasoning-parser qwen3_thinking --tool-call-parser qwen_coder \
  --host 0.0.0.0 --port 8000

# node 1 (same command, --node-rank 1)
tokenspeed serve /path/to/qwen3.8-max-fp8 \
  --served-model-name qwen3.8-max \
  --nnodes 2 --node-rank 1 --nprocs-per-node 8 --world-size 16 \
  --dist-init-addr <node0-host>:25000 \
  --attn-tp-size 16 \
  --moe-backend auto \
  --quantization fp8 --kv-cache-dtype fp8 \
  --attention-backend trtllm \
  --chunked-prefill-size 8192 \
  --gpu-memory-utilization 0.95 --max-num-seqs 128 \
  --disable-kvstore \
  --speculative-algorithm MTP --speculative-num-steps 3 \
  --speculative-eagle-topk 1 --speculative-num-draft-tokens 4 \
  --reasoning-parser qwen3_thinking --tool-call-parser qwen_coder \
  --host 0.0.0.0 --port 8000

TP8 DP2 EP16 (DeepEP)

Two TP8 attention replicas, experts sharded across all 16 ranks, and expert routing on DeepEP dispatch/combine instead of all-gather:

# node 0 (serves the HTTP API)
tokenspeed serve /path/to/qwen3.8-max-fp8 \
  --served-model-name qwen3.8-max \
  --nnodes 2 --node-rank 0 --nprocs-per-node 8 --world-size 16 \
  --dist-init-addr <node0-host>:25000 \
  --attn-tp-size 8 --data-parallel-size 2 --ep-size 16 \
  --moe-backend deep_gemm \
  --all2all-backend deepep --deepep-mode auto \
  --low-latency-max-num-tokens-per-gpu 64 \
  --quantization fp8 --kv-cache-dtype fp8 \
  --attention-backend trtllm \
  --chunked-prefill-size 8192 \
  --gpu-memory-utilization 0.95 --max-num-seqs 128 \
  --disable-kvstore \
  --speculative-algorithm MTP --speculative-num-steps 3 \
  --speculative-eagle-topk 1 --speculative-num-draft-tokens 4 \
  --reasoning-parser qwen3_thinking --tool-call-parser qwen_coder \
  --host 0.0.0.0 --port 8000

# node 1 (same command, --node-rank 1)
tokenspeed serve /path/to/qwen3.8-max-fp8 \
  --served-model-name qwen3.8-max \
  --nnodes 2 --node-rank 1 --nprocs-per-node 8 --world-size 16 \
  --dist-init-addr <node0-host>:25000 \
  --attn-tp-size 8 --data-parallel-size 2 --ep-size 16 \
  --moe-backend deep_gemm \
  --all2all-backend deepep --deepep-mode auto \
  --low-latency-max-num-tokens-per-gpu 64 \
  --quantization fp8 --kv-cache-dtype fp8 \
  --attention-backend trtllm \
  --chunked-prefill-size 8192 \
  --gpu-memory-utilization 0.95 --max-num-seqs 128 \
  --disable-kvstore \
  --speculative-algorithm MTP --speculative-num-steps 3 \
  --speculative-eagle-topk 1 --speculative-num-draft-tokens 4 \
  --reasoning-parser qwen3_thinking --tool-call-parser qwen_coder \
  --host 0.0.0.0 --port 8000

Notes:

• --low-latency-max-num-tokens-per-gpu sizes DeepEP's NVSHMEM heap (roughly 2.0 GB at 64, 8.1 GB at 256), and that heap is claimed after the KV pool is profiled. An oversized value therefore fails late, when the first dispatch runs out of fabric memory. Size it to the real per-rank decode token bound and no lower: a batch above the capacity is rejected, not truncated.
• Internode DeepEP rides NVSHMEM IBGDA. On a RoCE fabric, mirror the NCCL values into NVSHMEM_IB_GID_INDEX, NVSHMEM_IB_TRAFFIC_CLASS, and NVSHMEM_IB_SL, and point NVSHMEM_BOOTSTRAP_UID_SOCK_IFNAME at the same interface as NCCL_SOCKET_IFNAME.

Choosing a layout

• TP16 has the lower TTFT and TPOT at batch 1-2: no dispatch/combine hop, and the single replica owns the whole batch.
• The DeepEP layout pulls ahead from mid batch up, where its expert kernels and the second attention replica both pay off.

Qwen3.8-27B

A dense 27B-class Qwen3.8 FP8 checkpoint on a single GPU, with self-speculative MTP (the draft model path points at the same checkpoint):

tokenspeed serve /path/to/qwen3.8-27b-fp8 \
  --served-model-name qwen3.8-27b \
  --world-size 1 \
  --gpu-memory-utilization 0.9 \
  --attention-backend trtllm \
  --moe-backend flashinfer_trtllm \
  --chunked-prefill-size 8192 \
  --max-model-len 262144 \
  --max-num-seqs 128 \
  --kv-cache-dtype fp8_e4m3 \
  --speculative-algorithm MTP \
  --speculative-draft-model-path /path/to/qwen3.8-27b-fp8 \
  --speculative-num-steps 3 \
  --speculative-eagle-topk 1 \
  --speculative-num-draft-tokens 4 \
  --disable-kvstore \
  --host 0.0.0.0 --port 8000

GPT-OSS 20B / 120B

Small GPT-OSS launches can start simple. Large GPT-OSS launches usually tune tensor parallelism, scheduler token budget, and KV cache dtype.

tokenspeed serve openai/gpt-oss-20b \
  --served-model-name gpt-oss-20b \
  --tensor-parallel-size 1 \
  --max-model-len 131072 \
  --chunked-prefill-size 8192 \
  --reasoning-parser base \
  --host 0.0.0.0 \
  --port 8000

tokenspeed serve openai/gpt-oss-120b \
  --served-model-name gpt-oss-120b \
  --tensor-parallel-size 4 \
  --max-model-len 131072 \
  --kv-cache-dtype fp8 \
  --chunked-prefill-size 8192 \
  --max-num-seqs 256 \
  --reasoning-parser base \
  --host 0.0.0.0 \
  --port 8000

DeepSeek V4-Flash / V4-Pro

DeepSeek V4 needs FP8 KV cache, the DeepGEMM mega_moe experts, and the FP4 indexer cache. tokenspeed serve auto-selects --reasoning-parser deepseek_v31 and --tool-call-parser deepseek_v4, and auto-sets block_size=256 (pass --block-size N with N != 64 to override). Requires tokenspeed-deepgemm>=2.5.0.post20260629 and tokenspeed-flashmla.

V4-Flash — 4× B200 (SM100), data-parallel + expert-parallel:

tokenspeed serve deepseek-ai/DeepSeek-V4-Flash \
  --served-model-name deepseek-v4-flash \
  --trust-remote-code \
  --data-parallel-size 4 \
  --enable-expert-parallel \
  --kv-cache-dtype fp8_e4m3 \
  --moe-backend mega_moe \
  --attention-use-fp4-indexer-cache \
  --max-model-len 80000 \
  --max-total-tokens 163840 \
  --chunked-prefill-size 8192 \
  --enable-mixed-batch \
  --gpu-memory-utilization 0.9 \
  --disable-kvstore \
  --host 0.0.0.0 \
  --port 8000

V4-Pro — 8× B200, tensor-parallel:

tokenspeed serve deepseek-ai/DeepSeek-V4-Pro \
  --served-model-name deepseek-v4-pro \
  --trust-remote-code \
  --tensor-parallel-size 8 \
  --kv-cache-dtype fp8_e4m3 \
  --moe-backend flashinfer_trtllm \
  --attention-use-fp4-indexer-cache \
  --max-model-len 80000 \
  --max-total-tokens 2560000 \
  --chunked-prefill-size 8192 \
  --gpu-memory-utilization 0.9 \
  --disable-kvstore \
  --host 0.0.0.0 \
  --port 8000

For the expert-parallel topology, swap --tensor-parallel-size 8 for --tensor-parallel-size 8 --enable-expert-parallel --dense-tp-size 1 and --moe-backend flashinfer_trtllm for --moe-backend mega_moe.

MTP speculative decoding

Both variants can drive the checkpoint's NextN/MTP draft layers. Keep the launch flags above and add:

--speculative-algorithm MTP \
--speculative-num-steps 3

With --speculative-draft-model-path omitted, V4 uses the same checkpoint as the draft source (DeepseekV4ForCausalLMNextN). MTP runs on the non-overlap scheduler — the runtime disables overlap scheduling automatically when speculative decoding and paged-cache groups are both active — and prefix caching stays on by default. Add --enable-metrics to read Decoded Tok/Iter and the speculative accept rate from the run summary.

Tuning Order

1. Set model ID, trust policy, tokenizer mode, and served model name.
2. Set context length and KV cache dtype.
3. Set tensor, data, and expert parallelism to match the node topology.
4. Set scheduler budgets: --chunked-prefill-size, --max-num-seqs, and only then --max-total-tokens.
5. Set attention, MoE, and sampling backends explicitly for benchmark runs.
6. Add reasoning, tool-call, grammar, or speculative decoding only when the model and workload need them.