---
url: /blog/tokenspeed-reflection-beam.md
---

Reflection has released Beam, a mixture-of-experts (MoE) model with 501B total parameters, 23B active parameters per token, and a 1M-token context window.

TokenSpeed partnered with Reflection to support Beam on NVIDIA GB200 from day 0. We recommend the NVFP4 checkpoint on two GB200 compute trays. This configuration combines fused routing and normalization kernels, TensorRT-LLM NVFP4 MoE kernels, and a DFlash draft model for speculative decoding. The BF16 and FP8 checkpoints use the same launch command.

To deploy Beam, see the [TokenSpeed deployment guide](https://lightseek.org/tokenspeed/recipes/models#beam).

## Highlights

* **Lower weight memory.** On 8 GB200 GPUs, NVFP4 weights use 36.5 GiB per GPU, compared with 119.1 GiB for BF16. This leaves more memory for the key-value (KV) cache.
* **More KV cache capacity.** Splitting Beam's 43 sliding-window layers into smaller cache groups reduces unused space in full-attention pages. With the drafter enabled and the same KV memory pool, the new layout holds 3.07× as many tokens as the original single-group layout.
* **Fused routing and normalization.** Beam's FP32 expert selection runs in one kernel. For batches of up to 2,048 tokens, each of its 104 sandwich-norm boundaries uses one multi-node NVLink (MNNVL) all-reduce launch.
* **Faster MoE layers.** The default TensorRT-LLM NVFP4 MoE kernels use Beam's FP32 routes for its 4-bit experts. At 8 to 64 tokens per GPU, a MoE layer runs 2.2 to 2.9× as fast as BF16.
* **Faster speculative decoding.** A 1.93B-parameter DFlash drafter proposes 15 tokens at once for verification in the same CUDA graph replay. On our agentic workload, this increases single-user decode speed to 1.71× the speed without the drafter.
* **Template-based output parsing.** The smg gateway compiles Beam's response template into a streaming parser, without a handwritten parser for the model.
* **Full context with the drafter enabled.** TokenSpeed serves the full 1,048,576-token context. Beam retrieves a hidden passkey from prompts of up to 1,045K tokens.

## End-to-end performance preview

All results in this section use 8 GB200 GPUs across two compute trays, with tensor parallelism across all 8 GPUs (TP8). We report per-user decode speed, time to first token (TTFT), and total output throughput.

The [agentic coding workload](#agentic-coding-workload) starts with about 50K tokens of context and includes multiple turns with tool calls. We compare runs with and without the [DFlash drafter](#speculative-decoding-with-dflash), using the default medium reasoning effort. Because the checkpoint specifies no sampling defaults, requests use temperature 1.0 and top-p 0.95.

We calculate per-user decode speed by dividing the number of tokens after the first token by the elapsed time between the first and last tokens. This measures the decode interval; total output throughput is reported separately.

### Precision comparison

| Checkpoint | Weights per GPU | KV pool at launch | Decode speed / TTFT per user, 1 user | Decode speed / TTFT per user, 8 users |
| :---- | :---- | :---- | :---- | :---- |
| NVFP4 | 36.5 GiB | 18.8M tokens | 463 tok/s / 0.71 s | 223 tok/s / 0.87 s |
| FP8 | 61.2 GiB | 15.2M tokens | 349 tok/s / 0.97 s | 162 tok/s / 1.29 s |
| BF16 | 119.1 GiB | 6.53M tokens | 352 tok/s / 0.80 s | 153 tok/s / 0.96 s |

Weight memory excludes the drafter's 1.40 GiB per GPU. The KV pool uses the recommended launch configuration's default GPU memory utilization of 0.95, with the drafter enabled and no token cap. Decode speed and TTFT use the agentic coding workload with the drafter enabled.

BF16 is the reference precision. Among the measured configurations, NVFP4 provides the highest per-user decode speed and the largest KV pool.

For a single NVFP4 user, TTFT is 0.085 s on a follow-up turn, compared with 0.711 s on the first turn with its 50K-token prompt.

### Agentic coding workload

We evaluate Beam on multi-turn coding-agent sessions with real tool calls. Each session provides four tools: read a file, search code, apply a patch, and run tests. A session starts with about 50K tokens of repository files and a question. The model calls the tools, and each round of file reads and searches returns about 2K tokens of new code.

After two to four tool rounds, or after a response without a tool call, the user asks a new question about the updated code. Each conversation contains 10 to 15 questions and 14 to 25 requests when tool rounds are included. Every request includes previous replies in full, including reasoning and tool calls. The prefix cache serves 89% to 94% of prompt tokens, and contexts grow to about 100K tokens.

![Per-user decode speed for NVFP4, FP8, and BF16 with the drafter enabled at one and eight concurrent users](/images/202610/tokenspeed-reflection-beam-agentic-performance.png)

*Figure 1. Per-user decode speed for NVFP4, FP8, and BF16 with the drafter enabled on 8 GB200 GPUs with TP8. The agentic workload uses an initial context of approximately 50K tokens, 10 to 15 questions per session, medium reasoning effort, temperature 1.0, and top-p 0.95.*

With NVFP4 and the drafter enabled, a single user decodes at 463 tokens/s. At 8 concurrent users, per-user decode speed is 223 tokens/s, and total output throughput is 1,476 tokens/s. With the drafter enabled, NVFP4 per-user decode speed is 1.32 to 1.45× that of BF16.

At 32 concurrent users, the same 8 GPUs provide total output throughput of 1,813 tokens/s and per-user decode speed of 78 tokens/s.

### Throughput and interactivity

To measure the trade-off between per-user decode speed and total throughput, we sweep concurrency from 1 to 256 with AIPerf. These runs use 1K-token random prompts, 1K-token outputs, and no drafter. Figure 2 shows the endpoints of the sweep, with per-user decode speed and per-GPU output throughput presented separately. The workload differs from the agentic sessions described earlier, so its decode speeds are not the baseline for the DFlash speedup.

![Concurrency-sweep endpoints: 314 tokens per second for one user, and 81 per user with 1,961 per GPU at 256 users](/images/202610/tokenspeed-reflection-beam-throughput-interactivity.png)

*Figure 2. Endpoints from the NVFP4 concurrency sweep on 8 GB200 GPUs with TP8, measured with AIPerf. The workload uses 1K-token random prompts, 1K-token outputs, and no drafter. Per-user decode speed and per-GPU output throughput are separate metrics. The full trade-off curve is not shown.*

A single user decodes at 314 tokens/s. At 256 concurrent users, output throughput is 1,961 tokens/s per GPU, or about 15,700 tokens/s across the same 8 GPUs. Per-user decode speed is 81 tokens/s.

For end-to-end fixed-length requests with the drafter disabled, NVFP4 delivers 1.3 to 1.7× the BF16 output throughput on the same 8 GPUs. We measured this with AIPerf at input/output lengths of 1K/1K, 1K/8K, and 8K/1K tokens and 32 to 128 concurrent requests.

## Model architecture and precision

### Architecture

Beam is a 52-layer decoder with 256 routed experts and 8 active experts per token. The following features affect serving:

* **Dense and sparse layers.** The first 3 layers use a dense multilayer perceptron (MLP). The other 49 use MoE layers with no shared expert.
* **Sigmoid routing with a correction bias.** The router applies a sigmoid to expert scores and adds a learned bias for each expert. The bias affects expert selection only. The router selects the top 4 of 8 expert groups, then the top 8 experts, and scales the routed output by 2.5. Reflection describes this approach as auxiliary-loss-free routing.
* **Local and global attention.** Most layers use sliding-window attention with a 512-token window. Every sixth layer and the last layer use full attention, for a total of 43 sliding-window layers and 9 full-attention layers. Beam uses grouped-query attention (GQA) with 48 query heads and 8 KV heads.
* **Rotary position embeddings (RoPE).** Only the sliding-window layers use rotary position embeddings. The full-attention layers use no positional encoding.
* **Query and key normalization with scalable softmax.** Beam applies root mean square (RMS) normalization to queries and keys for each head. In full-attention layers, scalable softmax (SSMax) also scales queries by a slowly growing function of position to maintain attention selectivity over long distances.
* **Gated attention output.** A sigmoid gate computed from the attention input scales each output channel before the output projection.
* **SandwichNorm and depth-scaled residuals.** Each attention and MLP block has normalization before and after it. Beam scales the residual addition by depth.
* **Logit soft-capping.** Beam soft-caps logits at 30.

Only 9 of the 52 layers require the full KV history. The other 43 use the last 512 tokens.

### NVFP4 and FP8 checkpoints

Reflection provides NVFP4 and FP8 checkpoints alongside the BF16 reference checkpoint.

The NVFP4 checkpoint quantizes only expert weights. Each weight is a 4-bit floating-point value (E2M1). Every 16 weights share an FP8 (E4M3) scale, and each expert has an additional FP32 scale.

TokenSpeed quantizes expert inputs in the same format at runtime. The expert general matrix multiplications (GEMMs) therefore run as FP4 × FP4 on Blackwell Tensor Cores.

Attention, the three dense layers, the embeddings, and the output head remain in BF16. The router remains in FP32. These components are more sensitive to rounding and are not quantized.

On disk, the checkpoints occupy 294 GB for NVFP4, 1,003 GB for BF16, and 503 GB for FP8.

TokenSpeed runs the FP8 checkpoint natively and keeps its weights in FP8 in GPU memory. Its experts use TensorRT-LLM's per-channel FP8 MoE kernel, and its attention and dense layers use FP8 GEMMs.

### Accuracy

FP8 and NVFP4 scores are within 1.7 percentage points of the BF16 reference on the following evaluations. Reflection provided these scores.

| Evaluation | Reference BF16 | FP8 | NVFP4 |
| :---- | :---- | :---- | :---- |
| SWEBench Verified | 81.6% | 80.2% | 80.7% |
| Terminal Bench v2.1 | 80.1% | 78.9% | 78.42% |
| AIME 2026 | 97.7% | 98.0% | 97.5% |
| GPQA Diamond | 90.6% | 90.3% | 89.3% |
| MCP Atlas | 80.8% | 81.5% | 80.5% |
| LongBench v2 | 67.5% | 67.6% | 67.2% |

## Deployment on GB200

### Hardware and memory

We serve Beam on 8 GB200 GPUs in two compute trays, with 4 GPUs per tray. The NVL72 NVLink fabric connects the trays through MNNVL. By default, TokenSpeed uses tensor parallelism across all 8 GPUs (TP8) for both attention and experts.

With NVFP4, each GPU holds 36.5 GiB of model weights: 33.0 GiB for experts and 3.5 GiB for attention, the dense layers, the embeddings, and the output head. The drafter adds 1.40 GiB per GPU.

With the drafter enabled, the KV cache uses 7.0 KiB per token on each GPU. A full 1,048,576-token request therefore requires about 7.0 GiB per GPU, compared with 4.5 GiB with the drafter disabled.

At the recommended launch configuration's default GPU memory utilization of 0.95, NVFP4 with the drafter enabled and no token cap allocates a KV pool of 18.8 million tokens. At the same memory-utilization setting, this is 2.88× the BF16 KV pool capacity. The raw capacity is sufficient for 17 full 1M-token sessions.

### Full context support

TokenSpeed supports the full 1,048,576-token context declared by Beam's checkpoint. To test retrieval at long context lengths, we place a five-digit passkey in the middle of filler text and ask the model to return it. Beam retrieves the correct passkey from prompts of 377K to 1,045K tokens with NVFP4, FP8, and BF16, both with and without the drafter.

With the drafter enabled and an empty cache, a 1,045,524-token NVFP4 prompt reaches its first token in 25.96 s on the same 8 GPUs. This includes 25.82 s of prefill.

We also compared the full context limit with a limit of 262,144 tokens. With 1,024-token prompts and outputs and the drafter disabled, NVFP4 output throughput stays within 0.7% of the lower-limit configuration at 1, 64, and 128 concurrent requests.

### Mixed prefill and decoding

Our recommended launch uses `--enable-mixed-batch --prefill-graph-max-tokens 8192`. These options let a prompt chunk run in the same step as active decodes, reducing the pauses caused by long prompts. They also increase the maximum step size for CUDA graph replay from 2,048 to 8,192 tokens.

We measured these options with the drafter enabled, using the same launch without the two flags as the baseline. While 4 NVFP4 requests were decoding, we submitted simultaneous uncached 150K-token prompts. With 8 incoming prompts, the longest output pause decreased from 21.4 s to 0.135 s. With 32 incoming prompts, it decreased from 83.1 s to 0.149 s.

On fixed-length AIPerf runs at 64 and 128 concurrent requests, we tested input/output lengths of 1K/1K, 1K/8K, and 8K/1K tokens. Output throughput relative to the baseline was 1.02 to 1.27× for NVFP4, 1.03 to 1.35× for FP8, and 0.95 to 1.12× for BF16.

On multi-turn agentic traffic with 8 users, per-user decode speed increased by 34% for NVFP4, 11% for FP8, and 4% for BF16. Time to the first token of a conversation decreased by a factor of 3.0 to 3.6.

These options use 2.0 to 2.5 GiB more CUDA graph memory per GPU. The KV pool is 2% smaller for NVFP4 and FP8 and 5% smaller for BF16. Startup takes 15 to 30 s longer.

### KV cache groups for sliding-window attention

Beam's 43 sliding-window layers retain only the last 512 tokens. Its 9 full-attention layers and the drafter's 5 layers retain the full context.

TokenSpeed stores KV data in pages of 64 tokens. Each page has one slot for each layer in the largest cache group and belongs to one group at a time.

The initial implementation placed all 43 sliding-window layers in one group, so every page had 43 slots. TokenSpeed releases sliding-window pages when their tokens leave the window. A long request therefore occupies mostly full-attention pages, which used only 9 of their 43 slots, or 14 with the drafter.

We split the sliding-window layers into 5 groups of at most 9 layers. Each page now has 14 slots with the drafter or 9 without it. The window, attention masks, and outputs remain unchanged.

![KV page layouts before and after splitting sliding-window layers, with per-token memory falling from 21.5 to 7.0 KiB](/images/202610/tokenspeed-reflection-beam-kv-cache-groups.png)

*Figure 3. KV page layout on one GPU under TP8 with the drafter enabled, before and after splitting Beam's sliding-window layers into five cache groups. Both configurations use a 21.50 GiB pool per GPU.*

With the drafter enabled, KV memory per token decreases from 21.5 KiB to 7.0 KiB per GPU. For the same 21.50 GiB pool per GPU, capacity increases to 3.07× the capacity of the original single-group layout with the drafter enabled and 4.78× with it disabled.

Under a 25-minute NVFP4 workload of long and chat requests with the drafter enabled, peak page usage decreased from 99% to 38% of the pool.

We also measured output throughput with AIPerf, using the same 21.50 GiB pool per GPU, 8,192-token outputs, and the drafter disabled. At 64 concurrent requests, throughput was comparable: 6,822 versus 6,733 tokens/s. At 128 concurrent requests, where the original layout exhausted the pool, throughput increased from 6,013 to 8,017 tokens/s.

### Tensor parallelism and expert parallelism

TokenSpeed uses TP8 for both attention and experts by default. Each GPU holds a 1/8 slice of every expert, and each attention and MLP block ends with an all-reduce over NVLink.

TP8 distributes expert computation evenly across GPUs. It also fuses the all-reduce with the sandwich norms, as described in [MNNVL all-reduce and SandwichNorm fusion](#mnnvl-all-reduce-and-sandwichnorm-fusion).

With expert parallelism across 8 GPUs (EP8), each GPU holds 32 complete experts and processes the tokens routed to those experts. Within one NVLink domain, the same all-reduce sums the MoE output, so EP8 requires no all-to-all communication.

EP8 gives each GPU fewer, larger expert GEMMs, which can improve performance at large batch sizes. However, the load can be uneven across GPUs, and EP8 runs normalization separately from the all-reduce.

![Reported TP8 throughput advantages and near-parity range relative to EP8 at selected concurrency levels](/images/202610/tokenspeed-reflection-beam-tp8-ep8-throughput.png)

*Figure 4. TP8 throughput relative to EP8 on two GB200 trays with NVFP4 and the drafter disabled, measured with AIPerf. The 8,192-token-output runs at 128 concurrent requests are within 2% of each other.*

On two trays, TP8 provides 8% to 16% higher output throughput than EP8 at 32 and 64 concurrent requests. At 128 concurrent requests, the advantage decreases to 4% to 7% with 1,024-token outputs. With 8,192-token outputs, the two configurations are within 2% of each other. These results support TP8 as the default.

## Kernel optimizations

Beam requires FP32 routing with a correction bias, sandwich norms at every attention and MLP boundary, and additional attention and output operations. TokenSpeed uses fused kernels for these model-specific operations.

The expert, attention, and all-reduce operations use native TensorRT-LLM kernels. The following sections describe the fused operations and the NVFP4 MoE kernel.

### Fused FP32 routing

Beam's correction bias affects expert selection, so small rounding errors can change which experts the router selects.

TokenSpeed keeps the entire routing computation in FP32. The router multiplies BF16 activations by FP32 weights and returns FP32 scores. It uses a matrix-vector multiplication (GEMV) kernel tuned for each batch-size range.

The router selects from three FP32 kernels based on batch size:

* For up to 16 tokens, one thread block per expert reads the expert's weights once for all tokens.
* For larger batches of up to 96 tokens, the GEMV runs on FP32 CUDA cores.
* For batches larger than 96 tokens, including prefill, the GEMV runs on Tensor Cores using an exact three-way BF16 decomposition of the FP32 weights.

For NVFP4 and FP8 experts, a single Triton kernel then computes the sigmoid, bias addition, group top-k selection, and final routing weights in FP32. It also incorporates Beam's routed scale of 2.5 into the routing weights. For BF16 experts, Beam's FP32 routing runs within the TensorRT-LLM MoE launch.

![One-token FP32 routing latency of 58.1 microseconds for stock PyTorch and 7.5 microseconds for fused TokenSpeed kernels](/images/202610/tokenspeed-reflection-beam-fp32-routing.png)

*Figure 5. Full routing latency at one token for one Beam MoE layer on one GB200 GPU, including the router GEMV, top-k selection, and routing weights.*

At one token, the router GEMV and fused kernel together take 7.5 µs per layer. The same FP32 computation takes 58.1 µs with standard PyTorch operations, giving a 7.7× speedup. Across 1 to 64 tokens, the speedup ranges from 5.0 to 8.5×.

Compared with TokenSpeed's generic BF16 routing path, the FP32 path takes 1.4 µs less at one token and 5.0 µs more at 64 tokens.

### MNNVL all-reduce and SandwichNorm fusion

Beam applies normalization after attention, before the MLP, and after the MLP. Without fusion, each tensor-parallel boundary would require separate launches for an all-reduce, normalization, residual addition, and another normalization.

TokenSpeed fuses the all-reduce, post-norm, depth-scaled residual addition, and next pre-norm into one MNNVL kernel. For batches of up to 2,048 tokens, each of the 104 boundaries in a forward pass requires one launch.

Because NVFP4 expert GEMMs take less time, these boundaries account for a larger share of each decode step. For a single user with the drafter disabled, the 104 fused boundaries account for about 18% of GPU kernel time in an NVFP4 decode step.

![One-token all-reduce boundary latency of 8.2 microseconds with separate kernels and 6.2 microseconds with fusion](/images/202610/tokenspeed-reflection-beam-sandwichnorm-all-reduce.png)

*Figure 6. Latency for one all-reduce boundary across 8 GB200 GPUs on two trays, at one token and a hidden size of 5,120.*

At one token, the fused kernel takes 6.2 µs per boundary, compared with 8.2 µs for the all-reduce and separate normalization kernels. The fused kernel is faster at every measured batch size up to 2,048 tokens, with speedups of 1.10 to 1.34×.

### Attention and output fusions

TokenSpeed also fuses the following attention and output operations. Each runs in one kernel:

* **Output gating.** One kernel adds a per-head bias, applies a sigmoid, and scales the attention output. It performs all three operations in FP32 and rounds once.
* **Query and key normalization with RoPE.** On the 43 sliding-window layers, one kernel normalizes queries and keys, rounds them to BF16, and applies RoPE in FP32. This preserves the operation order in Beam's reference code.
* **SSMax scaling.** On the 9 full-attention layers, the query and key normalization kernel also computes each token's SSMax scale from its position. This requires no additional kernel launch.
* **Logit soft-capping.** One kernel reads and writes the logits once, replacing four elementwise kernels.

Together, these kernels take 211 µs, or about 5% of GPU kernel time in a single-user NVFP4 decode step with the drafter disabled.

For the FP8 checkpoint, TokenSpeed also fuses per-token input quantization into kernels adjacent to the attention and dense GEMMs during decoding. On 1,024-token random prompts with the drafter disabled, these fusions reduce single-user FP8 time per output token from 6.95 ms to 4.52 ms on average. The median decreases from 7.03 ms to 4.64 ms. We measured these results with AIPerf.

### TensorRT-LLM NVFP4 MoE

On GB200, TokenSpeed uses TensorRT-LLM's NVFP4 MoE kernel through FlashInfer by default.

Beam computes routes in FP32 and passes the selected experts and their FP32 routing weights to the kernel. In one call, the kernel runs both expert GEMMs, fuses SwiGLU into the first GEMM, and computes the weighted sum of expert outputs for each token.

The kernel accepts NVFP4 inputs. Input quantization does not depend on routing, so TokenSpeed runs it on a separate stream alongside the router within the same CUDA graph.

We compared this kernel with FlashInfer's CUTLASS-based NVFP4 MoE kernel and TensorRT-LLM's BF16 MoE kernel, using the same weights converted to BF16 for the BF16 comparison. All three kernels run one real Beam MoE layer on one GPU with the per-GPU shapes used by TP8 and EP8.

![NVFP4 MoE speedup over BF16 at one token, 8 to 64 tokens, and 8,192 tokens, excluding the invalid EP8 one-token sample](/images/202610/tokenspeed-reflection-beam-nvfp4-moe-kernels.png)

*Figure 7. TensorRT-LLM NVFP4 MoE speedups over BF16 for one real Beam MoE layer on one GB200 GPU. Ranges summarize TP8 and EP8 shapes. The EP8 one-token sample is excluded because it did not activate a local expert.*

At 8 to 64 tokens per GPU, the NVFP4 kernel achieves a 2.2 to 2.9× speedup over BF16 because it reads about a quarter of the weight bytes. At one token, the speedup is 1.35× for the TP8 shape. At 8,192 tokens, FP4 Tensor Cores provide a 1.6 to 2.5× speedup. Across 1 to 64 tokens, the kernel is also 1.2 to 2.2× as fast as the CUTLASS NVFP4 MoE kernel.

The BF16 checkpoint uses TensorRT-LLM's BF16 MoE kernel. The FP8 checkpoint uses TensorRT-LLM's per-channel FP8 MoE kernel. TokenSpeed retains its Triton MoE kernel as a portable fallback for other GPUs and layouts.

## Speculative decoding with DFlash

Reflection provides a DFlash drafter with Beam. This 5-layer, 1.93B-parameter model proposes a block of 15 tokens in one forward pass.

The drafter reads hidden states from five layers distributed through the target model. It shares the target's embedding and output head, adding 1.40 GiB per GPU.

Each decode step uses one CUDA graph replay:

1. The target verifies a 16-token block: the last accepted token and 15 draft tokens.
2. The target retains the longest matching prefix.
3. The drafter writes its KV cache and proposes the next block within the same graph.

Both models use TensorRT-LLM attention kernels and share a KV cache pool. Like Beam's rotary table, the drafter's rotary table covers the full 1,048,576-token context and the additional positions that speculative decoding writes beyond the context boundary.

Under TP8, every GPU holds the full embedding output, and the drafter sums its inputs across GPUs. TokenSpeed passes the embedding from one GPU only. Passing it from all eight would count the embedding eight times and reduce the acceptance rate without an explicit error.

![Per-user decode speed with the DFlash drafter is 1.71 times the baseline for one user and 1.51 times for eight users](/images/202610/tokenspeed-reflection-beam-speculative-decoding.png)

*Figure 8. Per-user decode speedup with the DFlash drafter on the agentic workload, using NVFP4 on 8 GB200 GPUs with TP8. Each concurrency level uses its own run with the drafter disabled as the 1.0× baseline. Results are shown for 1 and 8 users only.*

With the drafter enabled, NVFP4 per-user decode speed is 1.71× the baseline for a single user and 1.51× for 8 concurrent users. Each verification step accepts about 3.4 tokens on average. Acceptance depends on the text and sampling settings: these runs use temperature 1.0, whereas the GSM8K evaluation uses greedy decoding.

On GSM8K, with 1,319 problems and greedy decoding, NVFP4 with the drafter scores 97.2% and accepts about 6.8 tokens per verification step. With the drafter enabled, BF16 scores 97.2% and FP8 scores 97.0%.

## Output parsing with the smg gateway

Beam generates reasoning before its final answer and uses a model-specific format for tool calls.

When the smg gateway loads Beam's tokenizer, it reads the model's response template and compiles it into a streaming parser. Models with templates that smg supports require no additional gateway parser code.

The parser separates reasoning, final text, and tool calls in both streaming and non-streaming responses. It supports the Chat Completions and Messages APIs.

Automatically selected and forced tool calls, JSON schema outputs, and turns without tools are all grammar-constrained.

Beam's chat template requires the reasoning from each previous assistant turn. If you use a multi-turn client, return each turn's `reasoning_content` unchanged in subsequent requests.

We tested the gateway under load with about 4,660 streamed requests per checkpoint. Most requests ran while a full GSM8K evaluation shared the server. For the same prompts, streaming and non-streaming responses returned the same numbers.

## Future work

The following improvements are planned after the day 0 release:

* **Direct FP8 output from all-reduce.** An MNNVL all-reduce implementation that writes FP8 directly for subsequent GEMMs removes the remaining quantization launches at small batch sizes. This implementation is built and tested but is not part of the day 0 release.
* **Routing within the MoE kernel.** Beam's FP32 routing for BF16 experts already runs within the TensorRT-LLM MoE launch. We plan to add support for Beam's FP32 correction bias to the NVFP4 and FP8 MoE kernels so that their routing can run within the MoE launch too.
* **Automatic cache groups.** Beam's configuration explicitly defines the sliding-window layer groups. We plan to make the cache planner split large groups automatically for every hybrid model.

## Acknowledgments

We thank the Reflection team for early access to Beam, its NVFP4 checkpoint, and its drafter, and for their help with model details. We also thank NVIDIA DevTech and the open-source communities around TokenSpeed and SMG.