---
url: /blog/tokenspeed-qwen3-8-flash-next.md
---

Efficient model architectures create new opportunities for inference optimization. They also change where execution time goes. *Qwen Sparse Attention (QSA)* introduces an indexing pipeline. Wider residual streams add projections and memory traffic. When used for speculative decoding, *multi-token prediction (MTP)* can reduce the number of target-model passes, but it adds draft computation and speculative state management.

In this post, we explain how we optimized Qwen3.8-Flash-Next in TokenSpeed for faster inference. Our optimizations span three areas: making sparse selection and attention efficient, fusing computation across residual boundaries, and reducing repeated work during MTP. Programmatic dependent launch overlaps independent preparation between the remaining kernels.

## Model architecture

Qwen3.8-Flash-Next is a multimodal *mixture-of-experts (MoE)* model and an early preview of the Qwen4 architecture. The main model has 125 billion parameters, with 6 billion activated per token. The release also includes 51 billion n-gram embedding parameters and a 4-billion-parameter MTP module. For background, see [Qwen’s official introduction](https://qwen.ai/blog?id=qwen3.8-flash-next). This diagram shows the decoder, residual branches, and draft model.

![Qwen3.8-Flash-Next architecture with 48 decoder layers, including 36 Gated DeltaNet layers and 12 Qwen Sparse Attention layers. Four residual branches bypass each sublayer. A separate panel shows the reused decoder layer for multi-token prediction.](/images/202610/tokenspeed-qwen3-8-flash-next-architecture.png)

*Figure 1. Qwen3.8-Flash-Next model architecture and multi-token prediction.*

Three architectural features shape the inference workload:

* **Hybrid attention.** *Gated DeltaNet (GDN)* summarizes history through recurrent state. QSA selects relevant context at micro-block granularity.
* **Four-branch Gated Residual.** Dynamic gates control how sublayers read from and update four residual branches.
* **N-gram embeddings.** Local token context indexes an embedding table, adding model capacity with little computation.

TokenSpeed implements the model in the `qwen4_exp` model family, including *per-layer embedding (PLE)* and a single-layer draft model for MTP. The draft model proposes tokens for the main model to verify together.

TokenSpeed optimizes these components within individual kernels, across sublayer boundaries, and across a round of speculative decoding.

## Serving benchmarks

The frozen SWE-Smith conversation and AgentX benchmarks use the same deployment of four NVIDIA GB300 GPUs. Decoding is the stage in which the model generates output. The benchmarks measure its speed in tokens per second, where a token is a unit of text that the model processes or generates. A batch groups requests for processing. The charts show per-user generation speed and aggregate output across batch sizes and concurrency levels.

Both benchmarks use NVFP4 model quantization, BF16 activations and *key-value (KV)* caches, *tensor parallelism (TP)* of 4, *expert parallelism (EP)* of 4, FlashInfer TRTLLM MoE, 3 MTP draft steps, and 4 verification tokens. These settings specify the numerical formats, GPU parallelism, and speculative-decoding setup used in the measurements. To accelerate prefill, the processing of input tokens before generation, set `TOKENSPEED_QWEN4_EXP_QSA_MAX_LOGITS_MB=2048` in the server environment when running these benchmarks, especially at larger batch sizes.

### SWE-Smith conversations

This chart shows the trade-off between aggregate throughput and per-user decoding rate across batch sizes.

![SWE-Smith throughput and per-user decoding rate at batch sizes 1, 2, 4, 8, 16, and 32. Larger batches increase aggregate throughput from 488.1 to 1,933.7 tokens per second while reducing the per-user rate from 685.9 to 95.7 tokens per second.](/images/202610/tokenspeed-qwen3-8-flash-next-swe-smith-benchmark.png)

*Figure 2. SWE-Smith aggregate throughput and per-user decoding rate across batch sizes.*

The SWE-Smith benchmark covers batch sizes 1, 2, 4, 8, 16, and 32. As batch size increases from 1 to 32, aggregate output throughput rises from 488.1 tokens/s to 1,933.7 tokens/s, while the per-user decoding rate falls from 685.9 tokens/s to 95.7 tokens/s. The chart reports these as separate metrics. MTP acceptance length, reported in tokens per iteration, remains stable: 3.231 at batch size 1 and an average of 3.234 across the other five batch sizes.

### AgentX benchmark

AgentX measures aggregate output and interactivity at different concurrency levels. Concurrency is the number of requests in progress. Each concurrency level runs for 60 minutes with Qwen3.8-Flash-Next-NVFP4, FP32 *state-space model* states, a BF16 KV cache, an 8,192-token prefill chunk, a 256K-token context limit, and 3 MTP draft steps with a shared indexer. The benchmark uses a synthetic acceptance-length target of 2.32, including the bonus token.

In this chart, labels show the number of concurrent requests. The x-axis shows P90 interactivity in tokens per second per user: the reciprocal of the 90th-percentile per-request *inter-token latency*, excluding *time to first token*. This metric describes the pace of output after generation starts. The y-axis shows aggregate output throughput across all four GPUs.

![AgentX aggregate throughput versus P90 interactivity at concurrency levels 1, 2, 4, 8, 16, and 32. Throughput decreases as interactivity increases. Leader lines connect each measured point to its concurrency and exact values; acceptance length uses a synthetic target of 2.32.](/images/202610/tokenspeed-qwen3-8-flash-next-agentx-benchmark.png)

*Figure 3. AgentX aggregate throughput and P90 interactivity across concurrency levels.*

## Where decoding time goes

At small batch sizes, decoding time depends on more than floating-point computation. Loading weights, preparing metadata, moving intermediate tensors, launching kernels, and waiting for dependencies can also affect latency. A kernel is a function that runs on the GPU. Launching it and moving its inputs and outputs also take time.

Sparse attention adds context selection, and four residual streams add projections and memory traffic. MTP reduces the number of target-model passes needed to generate a response while adding draft computation and speculative state management. TokenSpeed reduces these costs through reuse and fusion. *Programmatic dependent launch (PDL)* overlaps independent preparation between the remaining kernels.

This table summarizes the work around each model component’s main computation.

| Component | Supporting work |
| :---- | :---- |
| QSA | History compression, scoring, top-k selection, and cache-slot mapping |
| Gated Residual | Branch normalization, gate projections, mixing, and residual updates |
| PLE | Context indexing, embedding access, short convolution, and state updates |
| MTP | Draft execution, verification, and accepted-state commit |
| GDN | Convolution, gating, recurrent-state access, and normalization |

These costs suggest this optimization sequence:

1. Remove work whose result is no longer needed.
2. Reuse calculations within their valid lifetime.
3. Fuse neighboring operations to reduce kernel launches and reads and writes of intermediate tensors.
4. Overlap independent preparation while preserving data dependencies.

These sections apply this sequence to attention, residual mixing, and speculative decoding, then examine kernel overlap and embedding-table offloading.

## Sparse attention optimizations

Sparse attention reduces the context processed by the final attention operation. Selecting that context also takes work: history compression, block scoring, top-k selection, and cache-slot mapping all contribute to latency. TokenSpeed therefore optimizes the selection pipeline as well as attention itself.

This diagram shows the four stages of QSA execution.

![QSA pipeline: prepare metadata and compressed history, score and select blocks, map selected blocks to physical KV cache slots, then compute sparse attention.](/images/202610/tokenspeed-qwen3-8-flash-next-qsa-pipeline.png)

*Figure 4. The four stages of QSA execution.*

### Top-k selection over valid blocks

Top-k selection chooses the k highest-scoring blocks. TokenSpeed writes QSA block scores to an FP32 buffer and uses a persistent radix selector to find those blocks. The selector receives each query’s valid block count. When the actual context is shorter than the capacity captured in a CUDA Graph, the selector skips padded columns in the score buffer.

### Compressed-key reuse during verification

Verification queries can share data loads even though they need separate selection results. MTP verification scores consecutive queries against the same compressed history. TokenSpeed groups up to 4 queries from one request and loads each compressed-key tile once for the group. Each query retains its own scores, complete-block frontier, head reduction, and top-k selection. Ragged layouts use 1 query per group.

The scoring tile contains 128 compressed blocks instead of 512. With four 128-dimensional BF16 index heads, a tile shared by 4 queries uses 36,864 bytes of shared memory. The previous tile, which served 1 query, used 132,096 bytes.

| Requests per batch | Previous tile (µs) | Shared-key tile (µs) |
| ----- | ----- | ----- |
| 1 | 6.61 | 2.69 |
| 32 | 157.90 | 31.75 |

The GB300 microbenchmarks in the preceding table measure scoring only. They use 4 queries per request, *programmatic dependent launch (PDL)*, repeated inputs, and effective contexts of about 51,000 to 61,000 tokens. Each CUDA Graph contains 100 calls. Results are medians from 15 replays across three rounds with alternating execution order. Sharing key loads reduces scoring work during verification; reusing selected slots across MTP draft steps, described later, removes scoring from subsequent draft steps.

### Sparse attention pipeline in CuTe DSL

After block selection, attention reads directly from physical KV cache slots. TokenSpeed overlaps these reads with computation.

For supported GB200 and GB300 configurations, TokenSpeed uses a kernel written in the *CuTe domain-specific language (CuTe DSL)*, specialized for the model’s attention geometry. It handles ordinary decoding and MTP verification with BF16 queries and either BF16 or FP8 E4M3 KV caches. This diagram shows how the kernel overlaps data movement and computation.

![CuTe DSL sparse attention adapts query rows, context splits, and operand buffers to available parallelism. Split thread blocks combine partial softmax results; unsplit blocks merge per head. Asynchronous gathers overlap computation, with barriers protecting buffer reuse.](/images/202610/tokenspeed-qwen3-8-flash-next-sparse-attention.png)

*Figure 5. Adaptive sparse attention and asynchronous operand loading in CuTe DSL.*

The launch policy uses the amount of independent query-head output work and the GPU’s *streaming multiprocessor (SM)* count to jointly choose the number of query rows per thread block, splits along the selected context, and operand-buffer slots:

* For small launches, cooperating thread blocks process separate parts of the selected context to expose more parallelism. The blocks combine their partial softmax results through distributed shared memory.
* Larger launches can use fewer splits, down to 1. With 1 split, each thread block processes its selected context locally and merges results with 1 warp per head.
* Sufficiently large launches can assign 2 or 4 query rows to one thread block to reuse shared workspace and tensor-memory allocation. Each query retains its own selected slots and softmax state.

Asynchronous gathers load upcoming key and value tiles while separate warps compute *query-key* and *probability-value* products on earlier tiles. Another warp group computes softmax. Barriers coordinate tile readiness and buffer reuse. The diagram abbreviates *cooperative thread array (CTA)*, *distributed shared memory (DSM)*, *tensor memory (TMEM)*, and *matrix multiply-accumulate (MMA)*.

For the supported selection width of 2,051 slots, the merge processes the final 3 entries, avoiding a mostly empty tensor-core tile. Split clusters combine results through distributed shared memory. Unsplit thread blocks merge these entries within each head’s warp.

### Metadata and cache preparation

TokenSpeed also reduces work around selection and attention:

* The runtime prepares shared metadata once within the forward scope where it remains valid.
* One metadata kernel produces positions, request indices, cache locations, and complete-block counts.
* The compression kernel can also normalize index queries and apply *rotary position embeddings (RoPE)*.
* The cache writer combines FP8 KV quantization with cache scatter.

These changes remove intermediate tensor reads and writes, kernel launches, or repeated calculations.

## Gated Residual fusion

Residual streams carry information between sublayers. Gated Residual expands this stream into four branches and adds learned operations to read and update them. TokenSpeed reduces the resulting overhead by fusing the mixer and, where dependencies allow, combining each residual update with the next normalization. This diagram shows the reads and updates around attention and MoE.

![Four residual branches retain their identity paths around attention and mixture-of-experts sublayers. Grouped normalization and read gates mix the branches. Each sublayer writes an update to all four branches before the next sublayer reads them.](/images/202610/tokenspeed-qwen3-8-flash-next-gated-residual.png)

*Figure 6. Four residual branches around attention and mixture-of-experts sublayers.*

### Residual mixer fusion

The four-branch Gated Residual mixer has 2,560 hidden dimensions per branch and a projection rank of 320. For aligned BF16 or FP16 inputs, one Blackwell CuTe DSL kernel combines the down projection, scaling, *sigmoid linear unit (SiLU)* activation, up projection, sigmoid gates, and final weighted mean in one launch. This diagram shows the fused computation.

![Fused residual mixer: down projection, reduction and scaling, SiLU activation, up projection, sigmoid gates, and branch weighting. The weighted mean produces the mixed output. The 324-row variant also produces four injection logits.](/images/202610/tokenspeed-qwen3-8-flash-next-residual-mixer.png)

*Figure 7. The fused Gated Residual mixer.*

Asynchronous operand loads overlap with tensor-core computation. Thread blocks in a split-K cluster combine partial down-projection results before the up projection consumes the activated result. Token count determines tile size, split-K, and pipeline depth. Wider tiles reuse weights across more tokens, and persistent workers reuse temporary storage across tiles. During warmup, the runtime checks kernel occupancy and limits the number of workers so cooperating clusters can run together.

This table compares the complete fused mixer with the previous portable fallback on a GB300 GPU with 152 SMs (SM103). Measurements use BF16 inputs, 324 projection rows, a scale of 1, PDL, and CUDA Graphs after warmup. Latencies are in microseconds. The speedups apply only to this mixer benchmark.

| Token rows | Portable fallback (µs) | Fused CuTe DSL (µs) | Speedup |
| :---- | :---- | :---- | :---- |
| 32 | 30.61 | 8.38 | 3.65× |
| 128 | 43.02 | 14.03 | 3.07× |

Eligible inputs with at most 256 token rows automatically use the fused CuTe DSL mixer. Larger inputs use the unfused implementation by default.

### Residual update and normalization fusion

Fusing operations across sublayer boundaries can avoid reloading the residual before the next normalization.

The residual update produces the input to the next mixer’s normalization:

R′t,g=Rt,g+2σ(at,g)Ot

Nt,g=GemmaRMSNorm(R′t,g,wg)

In these equations, *R* is the residual stream, *O* is the sublayer output, *a* contains injection logits, and *w* is the normalization weight of the consuming mixer.

Separate kernels write the updated residual *R′* to memory and reload it for normalization. A fused kernel computes both operations in one thread block and stores both outputs, as shown in this diagram.

![Separate residual update and normalization kernels store and reload the updated residual. The fused kernel rounds the residual to its storage type, retains it on chip for normalization, and stores both the updated residual and normalized output.](/images/202610/tokenspeed-qwen3-8-flash-next-residual-normalization.png)

*Figure 8. Separate and fused residual update and normalization.*

Both outputs remain necessary. Later residual updates consume the unnormalized state, and MTP also needs the final residual representation.

Fusion applies only when the updated residual feeds normalization directly. PLE, vision deepstack additions, and required pre-attention or final all-gathers consume the residual before normalization. At those boundaries, the runtime updates the residual before the intervening operation and normalizes it at the consuming mixer.

## Reducing work per output token with MTP

MTP reduces target-model passes by adding draft work. A draft model proposes several tokens for the target model to verify together. The cost per output token depends on the time spent drafting, verifying, and committing state, divided by the number of tokens produced per round.

The approximate cost is:

Time per output token≈Tdraft+Tverify+Tcommitoutput tokens per round

Tcommit is the time required to update persistent model state to reflect the accepted token prefix after verification.

TokenSpeed reduces these costs by reusing draft selections, writing verification state directly to its destination, and batching updates to accepted state.

### QSA selection reuse across draft steps

Qwen3.8-Flash-Next supports sharing QSA selections across MTP draft steps. TokenSpeed uses this capability when the `index_share_for_mtp_iteration` option is enabled in the model configuration. The first draft step selects KV cache slots, and later draft steps reuse those slots within the same drafting round, as shown in this diagram.

![The first MTP draft step selects KV cache slots. Two later steps reuse those slots within the same round while computing sparse attention with their own queries.](/images/202610/tokenspeed-qwen3-8-flash-next-mtp-selection-reuse.png)

*Figure 9. QSA selection reuse within one MTP drafting round.*

Later draft steps skip index-query normalization, RoPE, block scoring, top-k selection, and slot mapping. Key compression and state updates still run. Each step also computes sparse attention with its current query.

### Direct writes to verification buffers

Speculative execution creates temporary states for candidate tokens. After verification, the runtime must retain the state that corresponds to the accepted token prefix. It cannot immediately make every candidate state persistent.

TokenSpeed writes state directly from the producing kernels into rollback or commit storage, avoiding intermediate tensors and copies.

During MTP verification, PLE’s n-gram and short-convolution kernels write to preallocated verification buffers, as does the QSA compression kernel. After verification determines the accepted token counts, one commit kernel uses those counts to update persistent QSA state across layers.

### Accepted-state resolution and commit

For GDN and PLE, a fused page-resolution kernel clamps the accepted length, computes the checkpoint position, and looks up the destination page once per state group. A second kernel maps pages to layers and computes source and destination row indices for batched state updates.

PLE copies the shared n-gram context once, then copies local convolution states across layers in one batched launch. GDN uses the resolved rows to copy convolution state and either copy or replay recurrent state, depending on the configuration. Page resolution, row mapping, and state updates remain separate stages; QSA retains its own cross-layer commit kernel. This separation of stages reduces metadata operations and repeated per-layer index preparation in the commit path.

### PLE postprocessing fusion

During MTP verification, PLE processes multiple candidate tokens and retains state for every possible acceptance point. The short-convolution kernel adds gated values and hidden states, then writes each token’s rollback window directly to verification scratch buffers. Keeping these windows removes the need for a separate, unused final-state output.

### State ownership during CUDA Graph replay

TokenSpeed’s cache subsystem owns the persistent GDN, PLE, and QSA request caches. Each backend owns its transient verification workspace.

After verification, the runner calls a shared commit entry point. Each consumer applies the accepted result to its own state.

CUDA Graph replay updates GPU tensors without rerunning the Python code used during capture. State commit must therefore rely on live acceptance data and stable workspace ownership.

## Kernel preparation overlap with PDL

After reuse and fusion reduce the work, PDL overlaps independent preparation between the remaining kernels. A kernel can be ready to launch before its input data is ready to use. This diagram shows this distinction.

![Without PDL, the consumer starts after the producer finishes. With PDL, independent consumer setup overlaps the producer, followed by a wait before dependent work. Bar lengths are schematic and do not represent measured latency.](/images/202610/tokenspeed-qwen3-8-flash-next-pdl-overlap.png)

*Figure 10. Independent preparation overlap with programmatic dependent launch.*

PDL allows a dependent kernel to become eligible to launch before its producer finishes. The consumer can perform independent setup but must wait before reading data from the producer. For the synchronization requirements, see NVIDIA’s [PDL execution model](https://docs.nvidia.com/cuda/cuda-programming-guide/04-special-topics/programmatic-dependent-launch.html).

TokenSpeed applies this ordering across GDN, Gated Residual, and QSA.

A consumer may prepare descriptors, initialize local pipelines, or load weights whose readiness is explicitly guaranteed. It must wait before accessing activations, cache entries, or metadata produced by the preceding kernel.

### Latency of dependent kernel chains

On GB300, PDL ordering changes reduce latency for all four kernel chains at batch sizes 1 and 16. GDN decoding plus normalization has the largest relative reduction: 8.5% at batch size 16. At batch size 1, QSA logits selection plus attention saves 1.249 µs. Gated Residual normalization, mixing, and combination reduce latency by about 6% at both batch sizes. This table shows the latency before and after the changes, in microseconds.

| Kernel chain | Batch size 1 (µs) | Batch size 16 (µs) |
| :---- | :---- | :---- |
| GDN decoding and normalization | 4.606 → 4.354 | 7.930 → 7.254 |
| GDN verification of 4 tokens and normalization | 4.991 → 4.869 | 14.274 → 13.993 |
| Gated Residual normalization, mixing, and combination | 10.116 → 9.491 | 11.171 → 10.531 |
| QSA logits selection and attention | 40.994 → 39.745 | 63.871 → 63.420 |

Each result is the median of five measurements from repeated CUDA Graph replays on the same idle GPU. GDN uses FP32 state. QSA uses 4,096 compressed blocks, 4 index heads, 6 attention heads, and 2,051 selected slots. The residual benchmark rotates 32 weight banks.

The effect depends on the workload. At batch size 4, GDN verification latency increases from 7.167 µs to 7.250 µs.

## Freeing GPU memory with PLE table offload

PLE table offloading frees GPU memory. PLE’s n-gram table is large, but each step reads only a small set of rows. TokenSpeed stores the table in pinned host memory and gathers the required rows directly into GPU output buffers through *CUDA Unified Virtual Addressing*.

Host memory is the CPU’s memory. Offloading keeps the table there and brings only the needed rows to the GPU. FP8 storage further reduces the table’s host-memory footprint, and the gather kernel dequantizes each row as it loads it. Offloading frees GPU memory for the KV cache and larger batches.

Each rank holds a table shard. With TP, ranks look up shared token IDs and combine partial results. With *data parallelism (DP)*, ranks first gather token IDs, perform the sharded lookup, and retain their own results. This diagram shows the TP and DP lookup paths. TokenSpeed starts the host-memory gather on a separate CUDA stream at the beginning of the preceding decoder layer, overlapping the lookup with layer computation.

![PLE table shards reside in pinned host memory. Tensor-parallel ranks use the same token IDs and all-reduce partial lookups. Data-parallel ranks first gather their different token IDs, then all-reduce and select their local results.](/images/202610/tokenspeed-qwen3-8-flash-next-ple-offload.png)

*Figure 11. Tensor-parallel and data-parallel PLE lookup paths.*

With Qwen3.8-Flash-Next-NVFP4, FP8 PLE storage, and TP4 table sharding, offloading frees about 12.22 GiB of persistent GPU allocations per rank: 11.92 GiB for the table shard and 0.30 GiB for its device-side scale buffer. The full FP8 PLE table accounts for about 38.6% of the checkpoint’s tensor size. The freed memory can support a larger KV cache and can also raise serving concurrency.

## Interactions between optimizations

This table pairs each optimization with the cost it reduces.

| Optimization | Cost reduced |
| :---- | :---- |
| Top-k selection over valid blocks | Selection work on padded blocks |
| Compressed-key reuse during scoring | Repeated key loads across verification queries |
| Adaptive CuTe DSL sparse attention | Selected KV processing, split coordination, and setup for each query |
| Batched fused residual mixer | Intermediate handoffs between projections and repeated weight loads |
| Residual update and normalization fusion | Residual reloads and separate launches |
| QSA selection reuse across MTP draft steps | Repeated selection and index-query preparation |
| Direct writes to verification buffers | Temporary tensors and state copies |
| Fused page resolution for state commit | Repeated checkpoint lookup and row-index preparation |
| PLE convolution fusion | Intermediate outputs and separate additions |
| PDL ordering | Exposed preparation time between dependent kernels |

The optimizations affect one another. A faster selector increases the relative importance of attention. Fusion changes the kernel boundaries available for overlap. MTP changes matrix shapes and the amount of verification state. Isolated speedups therefore cannot simply be multiplied to predict serving performance.

## Optimization summary

Qwen3.8-Flash-Next combines sparse attention, wider residual streams, n-gram embeddings, and MTP. Efficient execution requires coordinating their computation and state. TokenSpeed addresses the cost per output token through efficient context selection, pipelined attention, residual fusion across eligible sublayers, direct writes of speculative state, and overlapping independent kernel preparation. These optimizations target computation, memory traffic, and waiting while preserving the model’s numerical and state semantics.