Qwen3.8-2.4T-A95B is an open-source 2.4T-parameter mixture-of-experts (MoE) model with 95B activated parameters. TokenSpeed provides Day-0 support on NVIDIA (G)B200/(G)B300, including cross-node serving, unified FlatKV prefix caching, and DSpark speculative decoding. Deployment commands are available in the deployment guide.

Architecture

Qwen3.8 follows the hybrid architecture used by the Qwen3.5/3.6 series. It has 92 layers and 95B activated parameters. In the official evaluation results below, Qwen3.8-2.4T-A95B scores higher than Qwen3.7 Max on all 16 reported evaluations.

The model combines three main components:

• Hybrid attention: Each layer uses either full attention (attention) or GDN linear attention (linear_attention). Interleaving the two provides global context modeling while retaining linear-complexity inference in GDN layers.
• Gated Delta Network (GDN): The linear-attention layers combine a state-space model (SSM) with causal convolution (Causal Conv1d), giving O(1) per-step inference complexity.
• Sparse MoE: Tokens are routed across a large expert pool that includes shared experts.

Implementation

FP8 Cross-Node Deployment

The required parallelism depends on weight precision. In FP8, the weights of Qwen3.8-2.4T-A95B exceed the memory of a single (G)B200 or (G)B300 node, so the model needs to run across multiple nodes; in NVFP4, the weights have roughly half the FP8 footprint and fit on a single Blackwell node. For FP8, we recommend a 4-node deployment with Attention TP4+DP4 / MoE EP16 on Grace Blackwell nodes, or a 2-node deployment with Attention TP8+DP2 / MoE EP16 on Blackwell nodes.

TP8 × DP2 × EP16 on Two Nodes

On a 16-GPU deployment, TP8 × DP2 × EP16 keeps high-frequency collectives within each NVLink domain and limits cross-node traffic to sparse MoE dispatch and combine operations.

Node-local attention collectives. With nprocs_per_node=8, each TP8 group maps to one node, so the per-layer attention reduction runs entirely over NVLink. AutoBackend uses plain NCCL for groups that span nodes; node-local groups use the CUDA-IPC custom all-reduce and fused allreduce+residual+RMSNorm kernels. The two DP replicas exchange no attention data: each node runs attention on its half of the batch independently.

Reduced cross-node payload. Because attn_tp (8) != moe_tp_ep (16), the runtime uses token-aware reduce-scatter/all-gather (RSAG). After attention, a node-local reduce-scatter leaves each rank with 1/16 of the global batch; only this deduplicated shard enters the MoE stage. TP16 instead requires two full-batch cross-node all-reduces per layer.

Sparse cross-node MoE traffic. EP16 assigns 1/16 of the experts to each GPU, with no weight replication or TP all-reduce inside the FFN. Cross-node communication is limited to DeepEP's sparse top-k dispatch and combine operations: normal mode for prefill and low_latency mode for decode.

KV-cache capacity under GQA. Qwen3.8 has 4 KV heads. TokenSpeed's QKVParallelLinear replicates KV heads whenever tp_size >= num_kv_heads: under TP16, each KV cache page is stored 4× across ranks, so duplicates consume three quarters of aggregate KV memory. TP8 reduces replication to 2×, and the two DP replicas serve disjoint requests. On the same hardware, this provides roughly 2× the effective KV-cache capacity and concurrent batch size. With only 4 KV heads, scaling attention beyond TP8 replicates KV cache without adding capacity. By contrast, DP is an effective way to use the second node without increasing KV-cache replication.

At fixed input/output lengths of 1024/8192, TP8 × DP2 × EP16 sustains 1.28–1.36× the throughput of TP16 across batch sizes 32–256, with a 1.31× average.

Unified FlatKV for Mamba States

Qwen3.8's GDN layers integrate with TokenSpeed's FlatKV. Instead of a growing KV history, a GDN layer holds a fixed-size recurrent state consisting of one SSM state and one causal-convolution tensor. FlatKV represents this state as one logical cache block, the same allocation unit used for KV pages.

Two-Level Cache Layout

FlatKV separates logical cache management from physical storage. At the logical level, the scheduler allocates, hashes, and prefix-matches fixed-size cache blocks. At the physical level, a shared arena contains equal-sized slabs, each packing one or more logical blocks from a single cache group.

Logical level. Allocation, hashing, and prefix matching operate at granularity P, the size of one logical cache block. The scheduler tracks block demand, cache residency, and references without depending on tensor shapes, dtypes, or byte offsets.

Physical level. An "LCM block" is a fixed-size slab that holds several logical blocks from one cache group. The number of logical blocks per slab is the group's packing factor. Full-attention KV and GDN state use separate groups. The SSM and convolution states are two fields in one state page, not separate groups.

Packing accounts for differences in page size. Under Qwen3.5's validated TP4 FP8 geometry, one GDN state page occupies roughly the same space as 32 full-attention KV pages. Using one page per slab for every group would leave most of the smaller page's slab unused. The runtime therefore derives each group's packing factor from the byte-size ratio, so every group fills its physical slabs at a matched rate with no dead space.

Packing changes only physical placement; prefix granularity remains P. A slab is bound to one cache group on first use and is unbound only after it becomes empty. Full-attention and state pages therefore share an arena without sharing individual slabs.

Prefix Cache Design for the Hybrid Architecture

Prefix reuse must account for both full-attention KV and recurrent state. FlatKV looks up the full-attention page cache first, then the state cache. The reusable prefix is the longest P-aligned boundary shared by every cache group. On a hit, FlatKV adds the cached block reference to the new request's block table and increments its reference count; it does not copy the page. Copy-on-write allows the resumed request to read the checkpoint while writing subsequent state to a new page.

Speculative Decoding with DSpark

TokenSpeed supports DSpark for Qwen3.8-2.4T-A95B. The implementation includes three optimizations.

One CUDA graph per decode step. TokenSpeed captures target verification, accept/bonus processing, draft-KV materialization and its cross-stream fork/join events, draft forward, and draft sampling in one graph per batch size. Each step therefore requires one graph replay. Scratch buffers, including the sampling all-gather workspace, are allocated at max_bs capacity during warmup so every captured graph uses fixed addresses.

KV materialization overlaps verification. The fc projection is split column-wise into one sub-weight per capture layer. When target layer ℓᵢ emits its auxiliary hidden state during verification, an auxiliary-stream fc proj folds that slice into the shared accumulator. After the final capture layer, hidden_norm and the fused KV write run on the same stream while the remaining target layers execute. The draft KV is typically committed before verification returns, removing materialization from the critical path.

Vocab-shard DSpark sampling. DSpark adds a rank-r Markov head to its draft backbone. At position k, a bigram bias conditioned on the token sampled at k−1 corrects the base logits over γ left-to-right steps. The bias is computed inside the same shard-space argmax primitive used for base draft sampling, instead of materializing full [bs, V] logits. Each rank evaluates hₖ · W_shardᵀ + markov_w1[prev] · markov_w2[shard]ᵀ for its local vocabulary slice, with O(bs·r·V/tp) bias cost. The TP reduction remains two 2·bs-scalar (max, id) all-gathers per step and is bit-exact with full-vocabulary argmax. All of the optimizations above carry over to DSpark unchanged; the γ-step loop is the only additional serial cost.

Looking Forward

Engine-level support for Qwen3.8, including cross-node serving that works with FlatKV prefix caching and speculative decoding, is necessary but not the hardest part of serving a model at this scale. The next phase of work is hardening this support for production: sustained throughput under real traffic, correctness under high concurrency, and predictable behavior over long-running deployments. The goal is not just raw performance, but a deployment that is production-ready out of the box.