Model Recipes

These recipes start from a known model family, pick the hardware topology, then set only the parameters that change runtime behavior.

The commands below are templates. Validate exact model IDs, checkpoint formats, and backend choices against the build you deploy.

Inkling

Blog: https://lightseek.org/blog/tokenspeed-inkling.html

## Docker

### nvidia
docker pull lightseekorg/tokenspeed:latest
### amd
docker pull lightseekorg/tokenspeed-amd:latest

## Launch command

# nvidia
ts serve \
    --model thinkingmachines/Inkling-NVFP4 \
    --attn-tp-size 4 \
    --moe-tp-size 4 \
    --max-model-len 81920 \
    --max-num-seqs 16 \
    --max-prefill-tokens 8192 \
    --chunked-prefill-size 8192 \
    --gpu-memory-utilization 0.95 \
    --disable-cuda-graph-padding \
    --trust-remote-code \
    --attention-backend fa4 \
    --moe-backend flashinfer_trtllm \
    --enable-prefix-caching \
    --disable-kvstore \
    --block-size 128 \
    --speculative-algorithm MTP \
    --speculative-num-steps 3 \
    --speculative-eagle-topk 1 \
    --speculative-num-draft-tokens 4

# amd
ts serve \
    --model lightseekorg/Inkling-MXFP4 \
    --attn-tp-size 4 \
    --moe-tp-size 4 \
    --max-model-len 81920 \
    --max-num-seqs 16 \
    --max-prefill-tokens 8192 \
    --chunked-prefill-size 8192 \
    --gpu-memory-utilization 0.95 \
    --disable-cuda-graph-padding \
    --trust-remote-code \
    --enable-prefix-caching \
    --disable-kvstore \
    --block-size 128 \
    --speculative-algorithm MTP \
    --speculative-num-steps 3 \
    --speculative-eagle-topk 1 \
    --speculative-num-draft-tokens 4

MiniMax M3

MiniMax M3 uses 128-token MSA blocks. TokenSpeed configures its dense and sparse attention layers automatically; select the dense backend with --attention-backend and run with --disable-kvstore.

EAGLE3 draft

tokenspeed serve nvidia/MiniMax-M3-NVFP4 \
    --tensor-parallel-size 4 \
    --max-model-len 81920 \
    --max-num-seqs 16 \
    --max-prefill-tokens 8192 \
    --chunked-prefill-size 8192 \
    --gpu-memory-utilization 0.95 \
    --disable-cuda-graph-padding \
    --attention-backend trtllm \
    --kv-cache-dtype fp8 \
    --moe-backend flashinfer_trtllm \
    --speculative-algorithm EAGLE3 \
    --speculative-draft-model-path Inferact/MiniMax-M3-EAGLE3 \
    --speculative-num-steps 3 \
    --speculative-eagle-topk 1 \
    --speculative-num-draft-tokens 4 \
    --disable-kvstore \
    --block-size 128 \
    --trust-remote-code \
    --host 0.0.0.0 \
    --port 8000

DSpark draft

nvidia/MiniMax-M3-DSpark is a six-layer Qwen3-shaped GQA block drafter with a vanilla Markov head. Keep the target launch shape and swap the speculative options:

tokenspeed serve nvidia/MiniMax-M3-NVFP4 \
    --tensor-parallel-size 4 \
    --max-model-len 262144 \
    --max-num-seqs 16 \
    --max-prefill-tokens 8192 \
    --chunked-prefill-size 8192 \
    --gpu-memory-utilization 0.95 \
    --disable-cuda-graph-padding \
    --attention-backend trtllm \
    --kv-cache-dtype fp8 \
    --moe-backend flashinfer_trtllm \
    --speculative-algorithm DSPARK \
    --speculative-draft-model-path nvidia/MiniMax-M3-DSpark \
    --speculative-num-steps 8 \
    --speculative-eagle-topk 1 \
    --speculative-num-draft-tokens 9 \
    --disable-kvstore \
    --block-size 128 \
    --trust-remote-code \
    --host 0.0.0.0 \
    --port 8000

Notes:

• This checkpoint's block_size is 8, and a DSpark block_size is the drafted token count, so launch it with --speculative-num-steps 8 and --speculative-num-draft-tokens 9: the verify window is one anchor row plus eight draft queries. Both widths are checked against the checkpoint at startup, so a mismatched launch fails fast instead of drafting a wrong-width block. See Speculative Decoding for the DSpark and DFlash conventions.
• --block-size 128 is the target's MSA page size. The draft writes its KV at the target's cache locations and shares the target's page table, so it inherits that page size; do not set a separate draft block size.
• The draft's 1024-token sliding window is an attention mask its own layers apply. It is deliberately not a cache-retention policy, because the draft's pages are the target's pages.
• Target features are captured from the residual stream after each layer in dflash_config.target_layer_ids ([1, 12, 23, 35, 46, 57] of M3's 60 layers) and concatenated in ascending layer order to feed the draft's fc.
• The draft checkpoint stores fp32 master weights. It is loaded in the target's dtype rather than the standalone fp32-to-fp16 default, because the two exchange hidden states and share the target's embedding and LM head.
• Measured on 4x GB300 with the launch above: gsm8k mean_acc 0.9719 versus 0.9704 without speculative decoding (paired disagreement 10 vs 8, McNemar p ~ 0.81 -- within run-to-run noise), so the draft does not move accuracy.

Kimi K2.5 / K2.6

Kimi-style MoE launches usually need remote code, long context, reasoning and tool parsers, and explicit MLA/MoE backends.

tokenspeed serve nvidia/Kimi-K2.5-NVFP4 \
  --served-model-name kimi-k2.5 \
  --trust-remote-code \
  --max-model-len 262144 \
  --kv-cache-dtype fp8 \
  --quantization nvfp4 \
  --tensor-parallel-size 4 \
  --enable-expert-parallel \
  --chunked-prefill-size 8192 \
  --max-num-seqs 256 \
  --attention-backend trtllm_mla \
  --moe-backend flashinfer_trtllm \
  --reasoning-parser kimi_k25 \
  --tool-call-parser kimik2 \
  --host 0.0.0.0 \
  --port 8000

For K2.6, keep the same parameter shape and change the checkpoint and parser only if the model card requires a different value.

To enable a compatible DFlash draft model, keep the target launch shape and add the draft model path plus DFlash speculative decoding options:

tokenspeed serve nvidia/Kimi-K2.6-NVFP4 \
  --served-model-name kimi-k2.6 \
  --trust-remote-code \
  --max-model-len 262144 \
  --kv-cache-dtype fp8 \
  --quantization nvfp4 \
  --tensor-parallel-size 4 \
  --enable-expert-parallel \
  --chunked-prefill-size 8192 \
  --max-num-seqs 256 \
  --attention-backend tokenspeed_mla \
  --moe-backend flashinfer_trtllm \
  --reasoning-parser kimi_k25 \
  --tool-call-parser kimik2 \
  --speculative-algorithm DFLASH \
  --speculative-draft-model-path /path/to/kimi-k2.6-dflash \
  --speculative-num-draft-tokens 8 \
  --speculative-num-steps 7 \
  --drafter-attention-backend fa4 \
  --host 0.0.0.0 \
  --port 8000

Official DFlash2 checkpoints that declare DFlash2DraftModel use the same --speculative-algorithm DFLASH launch. Their grouped dynamic convolutions and candidate selector are enabled automatically from the draft architecture. On GPU each convolution runs as a single fused Triton kernel; the torch node graph stays as the CPU reference. An MLA DFlash2 draft also writes its context KV through one stacked projection plus one fused norm/RoPE/scatter launch, which lets that write overlap the draft forward and accumulate as the target produces each captured layer. The server logs at INFO which of those paths it took, and why when it declined one. Draft proposals greedily follow the selector's transition-conditioned path. Candidate selection takes each vocabulary shard's local top-k and gathers only those, instead of gathering whole logits rows, whenever the head's shards carry no padding or added tokens. A request's temperature, top_k and top_p are applied by the target's verification step, never by the proposal, so the served distribution is the target's whatever the drafter proposed.

Kimi K3

For Hopper MXFP4 serving with pipeline prefill, attention-DP decode, DeepEP, DSpark and decode CUDA graphs, see the K3 Hopper PD configuration and validation guide.

Kimi-K3 combines a MoonViT vision encoder with a hybrid KDA (linear-attention) / NoPE-MLA (full-attention) decoder and a DeepSeek-V3-style latent MoE. The KDA layers currently use flash-linear-attention kernels on NVIDIA, so install it first:

pip install flash-linear-attention

Notes:

• K3 uses the cache-group scheduler and KDA state groups.
• On Blackwell, --dense-gemm-backend trtllm_cutedsl opts the block-FP8 attention projections into TRT-LLM CuTe-DSL: KDA fused QKV/gates and output, and MLA fused QKV-a/gate, Q-b, and output. Omit it (or use auto) to keep the existing backend selection. This shared option applies across models to standard 128x128 block-FP8 dense linears. BF16/NVFP4 linears, MXFP8, per-tensor FP8, specialized grouped projections, and routed experts are unchanged; it does not change tensor-parallel mapping.
• This backend preserves checkpoint FP8 values and their FP32 128x128 block scales; it does not requantize weights or convert scales to E8M0. Activations use the existing 1x128 FP8 quantization path. The kernel uses FP32 accumulation and BF16 output. Validate model accuracy on your workload; preserving the quantization contract does not guarantee bitwise equality across GEMM implementations.
• It requires Blackwell, CuTe-DSL with TVM-FFI support, BF16 outputs, and aligned local weight dimensions. Kernel variants compile during preparation before CUDA-graph capture. Unsupported selected linears fail instead of silently falling back. Faster GEMMs alone do not establish an end-to-end speedup.
• KDA dispatch is vendor-neutral at the runtime boundary. The kernel registry selects the existing FLA-derived NVIDIA implementation or the native AMD implementation, including each backend's preferred recurrent-state layout. The runtime does not transpose or reinterpret that state.
• The NVIDIA default selects tokenspeed_mla (FP8 KV), whose current kernels support Blackwell. On Hopper explicitly select --attention-backend flashmla or mla with --kv-cache-dtype bfloat16. AMD uses the mla backend.
• tokenspeed serve auto-selects the kimi_k3 reasoning and tool-call parsers. Explicit parser flags override these defaults.
• The SMG packages pinned by TokenSpeed resolve moonshotai/Kimi-K3 directly; a flattened local checkpoint and separately staged remote-code cache are no longer required.
• The checkpoint carries no FP8 KV scaling factors. When the target K3 uses an FP8 LCM cache, TokenSpeed keeps the separate K3 DSpark draft cache in BF16 so context injection and draft attention match the reference precision.
• DSpark proposal blocks use non-causal MLA draft attention. Both the mla and trtllm_mla draft backends preserve every block row during eager execution and CUDA graph capture. When K3's 128-token logical cache pages feed the 64-token TRT-LLM MLA kernel, the backend expands each logical page into its two physical kernel pages before draft attention.
• A K3 DFlash2 draft declares sliding_attention layers, so it needs a drafter backend that applies per-layer sliding windows: --drafter-attention-backend mla. Those layers dispatch to the CuteDSL windowed decode on Blackwell, which walks the KV from the window rather than from token zero, and fall back to the portable Triton kernel anywhere its shape gate does not hold. The draft's full-attention layer is unaffected either way.
• For Kimi K3, an eight-token verify window uses seven DSpark draft queries. The anchor query directly predicts the first draft through the Markov head; it must not be padded with an eighth, unused mask row.
• Target features are captured from K3's completed-layer prefix stream before the model-level AttnRes mix and final norm, matching the DSpark checkpoint's vLLM training and inference contract. A draft trained instead against the pre-norm AttnRes mixture declares "aux_hidden_stream": "attn_res" in its config and is served that stream. Feeding a draft the other stream raises nothing and shows up only as a lower acceptance rate, so the choice is logged next to the tap ids at startup.
• A draft whose config sets "fc_norm": true normalizes each target tap on its own before the taps are concatenated and projected, and ships one fc_norm.N.weight per tap. Declaring it without the weights (or shipping the weights without declaring it) fails the load rather than serving an identity-weight norm.
• Under tensor parallelism, the draft's final row-parallel MLP output is reduced across TP ranks before final_norm and shared target-head sampling.
• The vision encoder has 12 attention heads. For an 8-way text TP deployment, use --mm-encoder-tp-mode data so each rank runs the vision encoder at TP1 on a different whole image.
• The pinned SMG frontend registers Kimi-K3's chat renderer and multimodal processor. Preserve the checkpoint's media_proc_cfg.in_patch_limit=65536; silently falling back to K2.5's 16384-patch default reduces OCR resolution.
• KDA recurrent-state pages register for prefix-cache reuse only when a prefill chunk ends exactly on a logical cache-page boundary. The engine floors --chunked-prefill-size to the plan's page grain automatically (logged as a warning when it adjusts); the page grain is budget-dependent (e.g. 1472 at 32k context, 1536 at 1M), so do not hand-tune the chunk size against a hard-coded page value. Prefix hits are page-granular.

NVIDIA

Serve with expert parallelism (recommended) on 8x B300:

tokenspeed serve moonshotai/Kimi-K3 \
  --served-model-name kimi-k3 \
  --trust-remote-code \
  --max-model-len 32768 \
  --kv-cache-dtype fp8 \
  --tensor-parallel-size 8 \
  --mm-encoder-tp-mode data \
  --ep-size 8 \
  --moe-backend flashinfer_trtllm \
  --gpu-memory-utilization 0.94 \
  --max-num-seqs 32 \
  --disable-kvstore \
  --host 0.0.0.0 \
  --port 8000

Plain TP8 (drop --ep-size 8) works too. The fused MoE path needs a Blackwell GPU (B200/B300); on other NVIDIA platforms use --moe-backend triton.

AMD

The standard AMD path on 8x gfx950 uses the mla backend, which selects MLA kernels automatically. Use gluon to require Gluon MLA kernels for both the target and, when speculative decoding is enabled, its MLA drafter. For TP8/EP8, automatic MoE selection uses the specialized Gluon SiTU kernels:

tokenspeed serve moonshotai/Kimi-K3 \
  --served-model-name kimi-k3 \
  --trust-remote-code \
  --max-model-len 8192 \
  --kv-cache-dtype fp8 \
  --tensor-parallel-size 8 \
  --mm-encoder-tp-mode data \
  --enable-expert-parallel \
  --attention-backend mla \
  --moe-backend auto \
  --gpu-memory-utilization 0.92 \
  --max-num-seqs 32 \
  --disable-kvstore \
  --host 0.0.0.0 \
  --port 8000

To force Gluon attention for an Eagle3 launch, replace the attention option above and add the drafter option:

  --attention-backend gluon \
  --drafter-attention-backend gluon

The explicit policy fails fast when the current GPU, dtype, or attention shape has no registered Gluon kernel instead of silently selecting another solution.

On gfx950, the replicated 7168↔3584 latent projections automatically select among a one-token Triton GEMV, tuned Gluon GEMMs, and the vendor GEMM according to the current token count. At TP8/EP8, eligible one-token decode also combines the routed MXFP4 experts with the shared-expert down projection, then applies their joint reduction before the fused latent up-projection epilogue. Other shapes and unsupported layouts retain the ordinary composed path. The fused sigmoid-bias top-k route supports the full scheduled token count.

GLM5 / GLM5.2

GLM5 launches usually need remote code, long context, expert parallelism, FP8 KV cache, and the TRTLLM MoE backend. GLM5.2 FP8 is available on Hugging Face as zai-org/GLM-5.2-FP8. TokenSpeed defaults the reasoning parser to glm45; pass an explicit parser flag to override it. GLM5 DSA prefill planning reuses the host-side request lengths and the packed page/row plan across full-indexer layers; keep the scheduler-provided CPU length mirrors populated when integrating a custom attention backend to avoid device synchronization in this path.

tokenspeed serve zai-org/GLM-5.2-FP8 \
  --served-model-name glm-5.2 \
  --trust-remote-code \
  --tensor-parallel-size 8 \
  --enable-expert-parallel \
  --moe-backend flashinfer_trtllm \
  --kv-cache-dtype fp8 \
  --max-model-len 262144 \
  --chunked-prefill-size 8192 \
  --max-num-seqs 128 \
  --host 0.0.0.0 \
  --port 8000

GLM 5.3

GLM-5.3 follows the GLM-5.2 DSA serving path. Its base checkpoint includes the NextN draft layer, so MTP does not require a separate draft checkpoint.

ts serve zai-org/GLM-5.3 \
  --served-model-name glm-5.3 \
  --trust-remote-code \
  --tensor-parallel-size 8 \
  --enable-expert-parallel \
  --moe-backend flashinfer_trtllm \
  --kv-cache-dtype fp8 \
  --max-model-len 262144 \
  --max-num-seqs 128 \
  --draft-model-path-use-base \
  --speculative-algorithm MTP \
  --speculative-num-steps 3

GLM 5.3 Flash

GLM-5.3-Flash automatically configures its KDA/DSA backends and supports MTP from the base checkpoint.

Install ffmpeg before serving multimodal requests:

apt-get update && apt-get install -y ffmpeg

On MI350X, use tensor parallel size 4 with expert parallelism disabled for the BF16 or block-FP8 checkpoints. The block-FP8 path retains compact expert weights for its decode-specialized Gluon kernel and materializes BF16 expert copies once at load time for prefill.

On platforms without DeepGEMM, four-stream mHC uses a portable Triton path and switches to its tiled prefill projection above 256 tokens.

ts serve zai-org/GLM-5.3-Flash \
  --trust-remote-code \
  --tensor-parallel-size 8 \
  --enable-expert-parallel \
  --moe-backend flashinfer_trtllm \
  --kv-cache-dtype fp8 \
  --draft-model-path-use-base \
  --speculative-algorithm MTP \
  --speculative-num-steps 2 \
  --speculative-num-draft-tokens 3

Qwen3 Dense / Qwen3 30B-A3B

Qwen2, dense Qwen3, and Qwen3 MoE checkpoints use different architecture names. For Qwen3 30B-A3B, the Hugging Face config advertises qwen3_moe and Qwen3MoeForCausalLM, so launch it as a MoE model.

Qwen3-0.6B on Ascend NPU

The Ascend path supports unquantized Qwen3-0.6B on one or more NPUs. It uses the normal TokenSpeed scheduler and paged KV cache: prefill runs eagerly, while fixed-shape decode batches are captured as ACL Graphs. Aggregate serving can schedule prefill and decode work in the same deployment.

The validated environment is CANN 9.0.0, PyTorch 2.9.0, torch_npu 2.9.0.post2, Transformers 5.12.0, Triton 3.2.0, and Triton-Ascend 3.2.1. During validation, upgrading Transformers from 4.51.0 to 5.12.0 also changed huggingface-hub from 0.36.2 to 1.28.0, tokenizers from 0.21.4 to 0.22.2, and hf-xet from 1.5.1 to 1.6.0. It newly installed typer==0.27.1, shellingham==1.5.4, and annotated-doc==0.0.5. The setup also newly installs apache-tvm-ffi==0.1.13 and editable tokenspeed-kernel-npu==0.1.0. These are all Python-environment mutations made during this validation. Use matching PyTorch and torch_npu builds. From the repository root, install the Ascend dependencies, source CANN, and expose the three source packages:

test/ci_system/install_triton_ascend.sh
source /usr/local/Ascend/cann-9.0.0/set_env.sh
export ASCEND_RT_VISIBLE_DEVICES=0
export PYTHONPATH="${PWD}/python:${PWD}/tokenspeed-kernel/python:${PWD}/tokenspeed-kernel-npu/python:${PYTHONPATH:-}"

python -m tokenspeed.cli serve Qwen/Qwen3-0.6B \
  --served-model-name qwen3-0.6b \
  --device npu \
  --dtype bfloat16 \
  --kv-cache-dtype auto \
  --attention-backend mha \
  --sampling-backend greedy \
  --disable-prefill-graph \
  --disable-pdl \
  --max-model-len 4096 \
  --max-num-seqs 4 \
  --max-total-tokens 16384 \
  --chunked-prefill-size 4096 \
  --prefix-granularity 128 \
  --max-cudagraph-capture-size 4 \
  --cudagraph-capture-sizes 1 2 4 \
  --disable-autotune \
  --host 0.0.0.0 \
  --port 31889

The setup script installs Transformers 5.12.0 and Triton-Ascend 3.2.1, installs tokenspeed-kernel-npu from this checkout in editable mode, and compiles and executes both a vector-add kernel and a TokenSpeed KV-cache kernel on the visible NPU. Set TOKENSPEED_CANN_ROOT when CANN is installed outside the default path.

The standard mha runtime backend selects the registered Ascend kernels on an NPU. It requires eager prefill and disabled PDL, so pass --disable-prefill-graph and --disable-pdl explicitly. Do not pass --enforce-eager, because that also disables the decode ACL Graph. Although the graph flags retain their CUDA-oriented names for CLI compatibility, they control ACL Graph capture on an NPU. The command above captures decode batches 1, 2, and 4 and was validated with a 16,384-token KV pool. Increase --max-model-len, --max-num-seqs, --max-total-tokens, and the capture sizes together when scaling the deployment. --disable-autotune shortens bring-up; remove it after validation when startup tuning is desired.

The current Ascend sampling path is validated with greedy decoding. Because --sampling-backend greedy always performs argmax, send temperature=0 and do not expect request-level top_p or top_k to take effect.

Multi-card deployments use tensor parallelism; the world size must satisfy the model's standard divisibility constraints. Context, pipeline, and data parallel sizes must remain 1. Set --world-size N and expose N NPUs, for example ASCEND_RT_VISIBLE_DEVICES=0,1 with --world-size 2.

Verify the OpenAI-compatible endpoint with the served model name rather than the checkpoint path:

curl http://127.0.0.1:31889/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "qwen3-0.6b",
    "messages": [{"role": "user", "content": "你好，请用一句话介绍你自己。"}],
    "temperature": 0,
    "max_tokens": 64,
    "chat_template_kwargs": {"enable_thinking": false}
  }'

tokenspeed serve Qwen/Qwen3-30B-A3B \
  --served-model-name qwen3-30b-a3b \
  --tensor-parallel-size 2 \
  --enable-expert-parallel \
  --moe-backend flashinfer_cutlass \
  --max-model-len 40960 \
  --reasoning-parser qwen3 \
  --host 0.0.0.0 \
  --port 8000

Qwen3.8

Qwen3.8 shares the hybrid linear-attention (GDN) / full-attention layer pattern with Qwen3.5.

Qwen3.8-2.4T-A95B

Qwen3.8-max needs 16 GPUs, so it runs on two 8-GPU nodes. Launch tokenspeed serve on every node with the same command, changing only --node-rank; every node points --dist-init-addr at node 0, which is the only rank that serves the HTTP API. See Parallelism for the multi-node rules.

This family has no parser auto-selection, so set --reasoning-parser and --tool-call-parser explicitly. --speculative-algorithm MTP without --speculative-draft-model-path drafts from the base checkpoint. Set --dist-init-addr to node 0's own address and port throughout (<node0-host>:25000 below).

TP16

One replica across both nodes. --ep-size defaults to 1, so the experts stay tensor-parallel over the full world and all-to-all stays out of the path. Keep --moe-backend auto: the block-scale FP8 deep_gemm experts implement only the DeepEP legs and are unavailable without --all2all-backend deepep.

# node 0 (serves the HTTP API)
tokenspeed serve Qwen/Qwen3.8-2.4T-A95B \
  --served-model-name Qwen/Qwen3.8-2.4T-A95B \
  --nnodes 2 --node-rank 0 --nprocs-per-node 8 --world-size 16 \
  --dist-init-addr <node0-host>:25000 \
  --attn-tp-size 16 \
  --moe-backend auto \
  --quantization fp8 --kv-cache-dtype fp8 \
  --attention-backend trtllm \
  --chunked-prefill-size 8192 \
  --gpu-memory-utilization 0.95 --max-num-seqs 128 \
  --speculative-algorithm MTP --speculative-num-steps 3 \
  --speculative-eagle-topk 1 --speculative-num-draft-tokens 4 \
  --reasoning-parser qwen3_thinking --tool-call-parser qwen_coder \
  --host 0.0.0.0 --port 8000

# node 1 (same command, --node-rank 1)
tokenspeed serve Qwen/Qwen3.8-2.4T-A95B \
  --served-model-name Qwen/Qwen3.8-2.4T-A95B \
  --nnodes 2 --node-rank 1 --nprocs-per-node 8 --world-size 16 \
  --dist-init-addr <node0-host>:25000 \
  --attn-tp-size 16 \
  --moe-backend auto \
  --quantization fp8 --kv-cache-dtype fp8 \
  --attention-backend trtllm \
  --chunked-prefill-size 8192 \
  --gpu-memory-utilization 0.95 --max-num-seqs 128 \
  --speculative-algorithm MTP --speculative-num-steps 3 \
  --speculative-eagle-topk 1 --speculative-num-draft-tokens 4 \
  --reasoning-parser qwen3_thinking --tool-call-parser qwen_coder \
  --host 0.0.0.0 --port 8000

TP8 DP2 EP16 (DeepEP)

Two TP8 attention replicas, experts sharded across all 16 ranks, and expert routing on DeepEP dispatch/combine instead of all-gather:

# node 0 (serves the HTTP API)
tokenspeed serve Qwen/Qwen3.8-2.4T-A95B \
  --served-model-name Qwen/Qwen3.8-2.4T-A95B \
  --nnodes 2 --node-rank 0 --nprocs-per-node 8 --world-size 16 \
  --dist-init-addr <node0-host>:25000 \
  --attn-tp-size 8 --data-parallel-size 2 --ep-size 16 \
  --moe-backend deep_gemm \
  --all2all-backend deepep --deepep-mode auto \
  --low-latency-max-num-tokens-per-gpu 64 \
  --quantization fp8 --kv-cache-dtype fp8 \
  --attention-backend trtllm \
  --chunked-prefill-size 8192 \
  --gpu-memory-utilization 0.95 --max-num-seqs 128 \
  --speculative-algorithm MTP --speculative-num-steps 3 \
  --speculative-eagle-topk 1 --speculative-num-draft-tokens 4 \
  --reasoning-parser qwen3_thinking --tool-call-parser qwen_coder \
  --host 0.0.0.0 --port 8000

# node 1 (same command, --node-rank 1)
tokenspeed serve Qwen/Qwen3.8-2.4T-A95B \
  --served-model-name Qwen/Qwen3.8-2.4T-A95B \
  --nnodes 2 --node-rank 1 --nprocs-per-node 8 --world-size 16 \
  --dist-init-addr <node0-host>:25000 \
  --attn-tp-size 8 --data-parallel-size 2 --ep-size 16 \
  --moe-backend deep_gemm \
  --all2all-backend deepep --deepep-mode auto \
  --low-latency-max-num-tokens-per-gpu 64 \
  --quantization fp8 --kv-cache-dtype fp8 \
  --attention-backend trtllm \
  --chunked-prefill-size 8192 \
  --gpu-memory-utilization 0.95 --max-num-seqs 128 \
  --speculative-algorithm MTP --speculative-num-steps 3 \
  --speculative-eagle-topk 1 --speculative-num-draft-tokens 4 \
  --reasoning-parser qwen3_thinking --tool-call-parser qwen_coder \
  --host 0.0.0.0 --port 8000

Notes:

• --low-latency-max-num-tokens-per-gpu sizes DeepEP's NVSHMEM heap (roughly 2.0 GB at 64, 8.1 GB at 256), and that heap is claimed after the KV pool is profiled. An oversized value therefore fails late, when the first dispatch runs out of fabric memory. Size it to the real per-rank decode token bound and no lower: a batch above the capacity is rejected, not truncated.
• Internode DeepEP rides NVSHMEM IBGDA. On a RoCE fabric, mirror the NCCL values into NVSHMEM_IB_GID_INDEX, NVSHMEM_IB_TRAFFIC_CLASS, and NVSHMEM_IB_SL, and point NVSHMEM_BOOTSTRAP_UID_SOCK_IFNAME at the same interface as NCCL_SOCKET_IFNAME.

Choosing a layout

• TP16 has the lower TTFT and TPOT at batch 1-2: no dispatch/combine hop, and the single replica owns the whole batch.
• The DeepEP layout pulls ahead from mid batch up, where its expert kernels and the second attention replica both pay off.

Qwen3.8-27B

A dense 27B-class Qwen3.8 FP8 checkpoint on a single GPU, with self-speculative MTP (the draft model path points at the same checkpoint):

tokenspeed serve Qwen/Qwen3.8-27B-FP8 \
  --served-model-name Qwen/Qwen3.8-27B-FP8 \
  --world-size 1 \
  --gpu-memory-utilization 0.9 \
  --attention-backend trtllm \
  --moe-backend flashinfer_trtllm \
  --chunked-prefill-size 8192 \
  --max-model-len 262144 \
  --max-num-seqs 128 \
  --kv-cache-dtype fp8_e4m3 \
  --speculative-algorithm MTP \
  --speculative-draft-model-path Qwen/Qwen3.8-27B-FP8 \
  --speculative-num-steps 3

Qwen3.8 Flash Next

Qwen3.8-Flash-Next is a multimodal MoE model and an early preview of the Qwen4 architecture, playing for Qwen4 the role Qwen3-Next played for Qwen3.5. It pairs a 125B-parameter main model with 51B of N-gram embeddings (6B activated per token), supports 262,144 tokens of context natively and 1M with YaRN, and upgrades four axes of the hybrid design:

• GDN + QSA hybrid attention
• 4-branch Gated Residual
• N-gram (predictive latent) embeddings
• Muon optimization

In TokenSpeed it runs as a hybrid linear-attention (GDN) / full-attention model with optional predictive latent embeddings (PLE), optional QSA sparse attention, and a one-layer MTP draft. Dense and MoE checkpoints share the same launch command.

QSA selects CuTe DSL sparse attention on B200 (SM100) and B300 (SM103) for BF16 queries with BF16 or FP8 E4M3 KV caches, 256-dimensional heads, 6/12/24 query heads, 1/2/4 KV heads, and a selected-slot width of 2051. This applies to ordinary decode and MTP verification under CUDA Graph. Prefill and mixed queries, including one-token prefill, and other supported NVIDIA shapes and architectures use the FlashInfer FA2 fallback.

The decoder passes ordinary sublayer-output tensors and residual tuples between layers. At adjacent HC boundaries, it explicitly calls the consuming mixer's combine_norm() to fuse residual injection with that mixer's grouped RMSNorm. PLE, deepstack updates and row-gather boundaries combine the residual first; the updated, unnormalized HC state remains available for MTP.

ts serve \
    --model Qwen/Qwen3.8-Flash-Next-FP8 \
    --trust-remote-code \
    --tensor-parallel-size 4 \
    --quantization fp8 \
    --moe-backend flashinfer_trtllm \
    --speculative-algorithm MTP \
    --speculative-num-steps 3

Optional --hf-overrides

All keys are optional and can be combined in a single --hf-overrides JSON object:

--hf-overrides \
  '{"ple_embed_dtype":"float8_e4m3fn","index_share_for_mtp_iteration":true}'

• ple_embed_dtype: "float8_e4m3fn": store the PLE n-gram embedding table in FP8 to save memory. Omit it to store the table in the model's compute dtype.
• ple_offload_embedding: keep the PLE table in pinned host memory when true or GPU memory when false. When omitted, offloading is enabled on NVIDIA CUDA and disabled on other platforms.
• index_share_for_mtp_iteration: true: reuse the QSA top-k selection across MTP steps. Checkpoints that already set text_config.index_share_for_mtp_iteration=true do not need this flag.

GPT-OSS 20B / 120B

Small GPT-OSS launches can start simple. Large GPT-OSS launches usually tune tensor parallelism, scheduler token budget, and KV cache dtype.

tokenspeed serve openai/gpt-oss-20b \
  --served-model-name gpt-oss-20b \
  --tensor-parallel-size 1 \
  --max-model-len 131072 \
  --chunked-prefill-size 8192 \
  --reasoning-parser base \
  --host 0.0.0.0 \
  --port 8000

tokenspeed serve openai/gpt-oss-120b \
  --served-model-name gpt-oss-120b \
  --tensor-parallel-size 4 \
  --max-model-len 131072 \
  --kv-cache-dtype fp8 \
  --chunked-prefill-size 8192 \
  --max-num-seqs 256 \
  --reasoning-parser base \
  --host 0.0.0.0 \
  --port 8000

DeepSeek V4-Flash / V4-Pro

DeepSeek V4 uses FP8 KV cache. tokenspeed serve auto-selects --reasoning-parser deepseek_v31 and --tool-call-parser deepseek_v4, and auto-sets block_size=256 (pass --block-size N with N != 64 to override).

NVIDIA

The NVIDIA recipes below require tokenspeed-deepgemm>=2.5.0.post20260629 and tokenspeed-flashmla.

V4-Flash — 4× B200 (SM100), data-parallel + expert-parallel:

tokenspeed serve deepseek-ai/DeepSeek-V4-Flash \
  --served-model-name deepseek-v4-flash \
  --trust-remote-code \
  --data-parallel-size 4 \
  --enable-expert-parallel \
  --kv-cache-dtype fp8_e4m3 \
  --moe-backend mega_moe \
  --attention-use-fp4-indexer-cache \
  --max-model-len 80000 \
  --max-total-tokens 163840 \
  --chunked-prefill-size 8192 \
  --enable-mixed-batch \
  --gpu-memory-utilization 0.9 \
  --disable-kvstore \
  --host 0.0.0.0 \
  --port 8000

V4-Pro — 8× B200, tensor-parallel:

tokenspeed serve deepseek-ai/DeepSeek-V4-Pro \
  --served-model-name deepseek-v4-pro \
  --trust-remote-code \
  --tensor-parallel-size 8 \
  --kv-cache-dtype fp8_e4m3 \
  --moe-backend flashinfer_trtllm \
  --attention-use-fp4-indexer-cache \
  --max-model-len 80000 \
  --max-total-tokens 2560000 \
  --chunked-prefill-size 8192 \
  --gpu-memory-utilization 0.9 \
  --disable-kvstore \
  --host 0.0.0.0 \
  --port 8000

For the expert-parallel topology, swap --tensor-parallel-size 8 for --tensor-parallel-size 8 --enable-expert-parallel --dense-tp-size 1 and --moe-backend flashinfer_trtllm for --moe-backend mega_moe.

AMD

V4-Flash — 2× MI350-series (gfx950), tensor-parallel + MTP:

tokenspeed serve deepseek-ai/DeepSeek-V4-Flash \
  --trust-remote-code \
  --tensor-parallel-size 2 \
  --kv-cache-dtype fp8_e4m3 \
  --max-model-len 4096 \
  --max-total-tokens 16384 \
  --chunked-prefill-size 8192 \
  --prefill-graph-max-tokens 8192 \
  --gpu-memory-utilization 0.9 \
  --disable-kvstore \
  --speculative-algorithm MTP \
  --speculative-num-steps 3 \
  --host 127.0.0.1 \
  --port 8000

V4-Flash — 1× MI450-series (gfx1250), Triton + decode/prefill graphs, without MTP:

tokenspeed serve deepseek-ai/DeepSeek-V4-Flash \
  --served-model-name deepseek-v4-flash \
  --trust-remote-code \
  --tensor-parallel-size 1 \
  --kv-cache-dtype fp8_e4m3 \
  --moe-backend triton \
  --attention-use-fp4-indexer-cache \
  --max-model-len 4096 \
  --max-total-tokens 8192 \
  --max-num-seqs 4 \
  --chunked-prefill-size 256 \
  --gpu-memory-utilization 0.8 \
  --disable-kvstore \
  --max-cudagraph-capture-size 4 \
  --cudagraph-capture-sizes 1 2 3 4 \
  --prefill-graph-max-tokens 256 \
  --prefill-graph-capture-token-sizes 128 256 \
  --host 127.0.0.1 \
  --port 8000

MTP is not yet validated on MI450.

MTP speculative decoding

Both variants can drive the checkpoint's NextN/MTP draft layers. Keep the launch flags above and add:

--speculative-algorithm MTP \
--speculative-num-steps 3

With --speculative-draft-model-path omitted, V4 uses the same checkpoint as the draft source (DeepseekV4ForCausalLMNextN). MTP runs on the non-overlap scheduler — the runtime disables overlap scheduling automatically when speculative decoding and cache groups are both active — and prefix caching stays on by default. Add --enable-metrics to read Decoded Tok/Iter and the speculative accept rate from the run summary.

DSpark speculative decoding with Prefix Replay

A V4-Flash checkpoint that includes complete DSpark draft weights can use same-checkpoint DSpark decoding. Prefix Replay keeps prefix caching enabled while recomputing the bounded prompt suffix needed to rebuild DSpark's request-persistent state:

tokenspeed serve deepseek-ai/DeepSeek-V4-Flash \
  --served-model-name deepseek-v4-flash \
  --trust-remote-code \
  --tensor-parallel-size 8 \
  --kv-cache-dtype fp8_e4m3 \
  --moe-backend flashinfer_trtllm \
  --attention-use-fp4-indexer-cache \
  --max-model-len 96000 \
  --max-num-seqs 80 \
  --max-total-tokens 2560000 \
  --max-prefill-tokens 8192 \
  --chunked-prefill-size 8192 \
  --enable-mixed-batch \
  --enable-prefix-caching \
  --gpu-memory-utilization 0.90 \
  --disable-kvstore \
  --speculative-config '{"method":"dspark","num_speculative_tokens":5}' \
  --speculative-eagle-topk 1 \
  --max-cudagraph-capture-size 80 \
  --prefill-graph-max-tokens 2048 \
  --enable-metrics \
  --host 0.0.0.0 \
  --port 8000

The replay window comes from the checkpoint's DSpark configuration; there is no user-tuned replay-length flag. Startup fails closed when same-checkpoint DSpark weights are incomplete, the replay capability is missing, KVStore is enabled, or the draft checkpoint contains only MTP/NextN weights. External DSpark checkpoints that do not advertise this capability keep the generic scheduler behavior.

Same-checkpoint DSpark computes its public FP32 base logits straight from the target's BF16 LM-head shard: BF16 products are exact in FP32, so a BF16 GEMM with an FP32 accumulator reproduces the reference's FP32 head math up to summation order without an FP32 copy of the shard. The head is read in place under CUDA Graph replay, so in-place target weight updates need no refresh.

The greedy block sampler runs one Triton kernel per block step on each tensor-parallel rank: it gathers the previous token's BF16 Markov bigram row from a replicated table, adds the bigram bias (a BF16 tensor-core dot against the rank's Markov projection shard, which is sharded like the LM head) to the rank's base logits, and packs the best (logit, token) of every vocabulary tile into one int64 candidate. One all-gather per step shares the candidates; the next step's kernel (or the final resolve) reduces them to the torch.argmax winner over the whole vocabulary, ties broken toward the lowest token id. A five-token block is therefore six kernels and five all-gathers instead of a per-step chain of masked embedding lookups, GEMMs, reductions and two all-gathers.

Engram's per-step inputs follow the same pattern. Before each forward, one pinned upload carries every request's history snapshot, and two launches (fill_ngram_history) seed the per-slot accepted prefixes and write every input row's previous-three tokens and validity mask, blanking the graph padding rows. Inside the forward, engram_hash maps the current token and its three predecessors, applies the per-layer multipliers, rolling XOR and prime buckets, and emits the table rows for all Engram layers in one launch; its output is bit-identical to the eager reference, which remains the CPU path.

The CUDA draft path also preserves the checkpoint's UE8M0-scaled FP8 activation round-trip with a fused tokenspeed-kernel operation. It computes the same per-group power-of-two scale and returns dequantized values in the input dtype; the fusion removes intermediate reduction and elementwise launches but does not change the model's quantization contract.

DSpark attention RMSNorm uses the platform kernel on CUDA while retaining its explicit FP32-accumulating PyTorch expression as the CPU reference. The fused path preserves the existing output dtype and is safe to capture and replay in the target CUDA Graph.

For a two-node TP8 deployment, run one process per node with four local workers and the same command on both nodes. See Multi-Node for explicit topology flags and launcher-derived settings. Before applying production load, confirm that every rank reports a nonzero Prefix Replay window, then check completion, speculative acceptance, and cache-hit metrics with fixed prompts and package/model revisions.

DeepSeek V4.1-Flash

DeepSeek V4.1 (deepseek_v41) is served by its own FlatKV attention backend with a four-group KV cache: the global KV chains, the SWA rows and the compressor tails. The recipe declares the last two replayable: they never enter the prefix cache, and a prefix hit re-feeds the cached prefix's last 128 tokens so the model regenerates them into the request's own pages (SWA bounded replay, docs/design/scheduler.md §1.3). The global KV and index rows those replayed tokens recompute are masked, so the shared rows stay exactly what the first computation produced. The CED decoder (layers 20–39) runs only on each prompt's last 128 positions (one row per chunk that does not complete its prompt), so the prefill row count changes at layer 20. The prefill CUDA graph therefore captures the model in two halves around that layer, which runs eager: encoder graphs per token bucket and decoder graphs per decoder-row bucket, the latter shared by every token bucket and capped at 128 rows per request (--max-num-seqs × 128, see docs/design/unified_path.md). --prefill-graph-capture-sizes sets both ladders; --disable-prefill-graph turns both off. Decode CUDA graphs are unaffected.

tokenspeed serve deepseek-ai/DeepSeek-V4.1-Flash \
  --served-model-name deepseek-v41-flash \
  --trust-remote-code \
  --tensor-parallel-size 8 \
  --enable-expert-parallel \
  --dtype bfloat16 \
  --max-model-len 32768 \
  --max-total-tokens 262144 \
  --max-num-seqs 32 \
  --chunked-prefill-size 8192 \
  --max-cudagraph-capture-size 32 \
  --disable-kvstore \
  --host 0.0.0.0 \
  --port 8000

On Hopper the routed MXFP4 experts run on FlashInfer's CUTLASS mixed-input grouped GEMM (flashinfer_cutlass_mxfp4_w4a16_moe_apply), which auto selects over Marlin. Marlin dequantizes in registers and scales linearly with the token count; the CUTLASS kernel stays weight-bandwidth bound. Measured per MoE layer on one EP8 rank of an H20 (µs, CUDA-graph replay): 502 vs 656 at 96 tokens, 621 vs 1552 at 192, 671 vs 2855 at 576, 4517 vs 7999 at an 8192-token prefill chunk. Two consequences:

• Startup runs FlashInfer's tactic autotuner inside the kernel tuning window (about five minutes for this kernel on H20). --disable-autotune skips it and serves heuristic tactics, which is fine for bring-up.
• --moe-mxfp4-fp8-activation switches to the W4A8 variant (FP8 activations, Humming residual scales): 282/338/380/2195 µs at the same token counts, another 1.8x, at a few percent of relative error on the expert outputs (FP8 activation rounding). Treat it as opt-in and confirm the served model on GSM8K or a similar check before relying on it.

--moe-backend marlin keeps the previous kernel, and stays required for DeepEP all-to-all layouts and for Kimi-K3's SiTU experts, which the CUTLASS epilogue does not implement.

Add --speculative-algorithm DSPARK for same-checkpoint DSpark decoding; the draft seeds its context windows from the decoder's kept rows, and each draft stage attends its 128-row window plus the non-causal proposal block through the same selected_attention workspace kernel as the target's prefill (FlashMLA on sm90+, Triton elsewhere); the fp32 arithmetic remains the CPU reference. A hit re-feeds the groups' whole retention window, which is exactly the 128-token attention window (the compressor-tail group retains its unfinished pair the same way); the scheduler requires --chunked-prefill-size of at least that window plus one prefix page and never leaves a prompt's final chunk shorter than it. The replayed rows attend SWA keys from the replay start only, the truncation the model is trained for; the cached global KV is never recomputed from them. usage.prompt_tokens_details.cached_tokens reports the hit through the end of the replayed window, so it stays a multiple of the prefix granularity. Under prefill/decode disaggregation the prefill node replays on its own prefix hits exactly as above and ships each group's retained tail; the decode node lands the tail and never re-feeds. Even on one machine, let Mooncake pick an RDMA transport rather than forcing the intra-node NVLink one.

GB300 Slurm 1P1D CI

deepseek-v4.1-flash-pd-1p1d-dspark-evalscope-gsm8k-gb300-slurm.yaml runs one TP4 prefill engine and one TP4 decode engine on two four-GPU nodes. Node 0 also hosts the SMG gateway and the Slurm evaluation client, which connects to 127.0.0.1:8000. Both engines use same-checkpoint DSpark, mega_moe with expert parallelism on Blackwell, host-resident Engram tables, and Mooncake transfer after the completed prompt (layerwise-interval=0). Decode prefix caching is disabled; the gateway uses deepseek_v31 reasoning parsing. The 262144-token cache budget limits admission independently of the 32768-token per-request context and 16-sequence cap.

The gate checks GSM8K accuracy of at least 0.90 on 100 samples with EvalScope 1.11.1, greedy decoding, concurrency 8, and up to 30000 generated tokens. It participates in the GB300 Slurm per-commit workflow. To run only this case, select its YAML in Slurm Dispatch, choose cluster gb300, and optionally provide a pull request number.

The launcher uses PD_SLURM=1 to assign one role per node and clears Slurm topology discovery only inside each worker process. Its job-and-step-scoped artifact directory must be shared between nodes. It publishes role readiness atomically and verifies cross-node gRPC health before starting the gateway. Worker logs remain separate as prefill.log, decode.log, and lb.log. Set DISAGGREGATION_IB_DEVICE when an explicit RDMA device selection is needed; the GB300 task selects mlx5_0,mlx5_1,mlx5_2,mlx5_3 to keep transfers on the InfiniBand fabric. Automatic discovery also includes Ethernet RNICs, which can cause incompatible RoCE/InfiniBand endpoint pairings during the RDMA handshake. Without PD_SLURM=1, the same launcher retains the single-node smoke topology.

Tuning Order

1. Set model ID, trust policy, tokenizer mode, and served model name.
2. Set context length and KV cache dtype.
3. Set tensor, data, and expert parallelism to match the node topology.
4. Set scheduler budgets: --chunked-prefill-size, --max-num-seqs, and only then --max-total-tokens.
5. Set attention, MoE, and sampling backends explicitly for benchmark runs.
6. Add reasoning, tool-call, grammar, or speculative decoding only when the model and workload need them.