---
url: /blog/eagle-4.md
description: >-
  EAGLE-4 training architecture, TokenSpeed serving optimizations, and
  evaluation results.
---

EAGLE-4 is the latest generation of the EAGLE family of speculative decoding methods. TorchSpec trains EAGLE-4 drafters, and TokenSpeed serves them with the full EAGLE-4 feature set. TokenSpeed supports the dynamic trees and chains of EAGLE-Harness™, sampled trees at temperatures greater than 0, and the cost-measurement toolchain.

Speculation is lossless: greedy decoding returns the target model's choices, and tree rejection sampling preserves the target model's distribution for sampled outputs. The [floating-point evaluation](#lossless-speculation-and-floating-point-behavior) describes the rounding differences introduced by specific optimizations.

On multi-turn SPEED-Bench, TokenSpeed serves Qwen3.8-27B in bfloat16 (BF16) on four NVIDIA B300 GPUs with four-way tensor parallelism (TP4). For a single request at temperature 1, it decodes at 750 tokens/s: 2.85× the speed of decoding without speculation and 1.28× the speed with the model's own multi-token prediction (MTP) drafter. On an agentic coding workload with sampling, EAGLE-4 provides about a 3.0× speedup over decoding without speculation for a single user. The evaluation also covers greedy decoding and long contexts.

## EAGLE-4 overview

In each round, a small drafter proposes a tree of tokens. The target model verifies the tree in one forward pass and accepts the longest path that agrees with the target, plus one bonus token. EAGLE drafters read the target model's hidden states, which lets a single layer draft accurately.

EAGLE-4 retains the normalized, single-layer drafter from EAGLE-3.1. The drafter reads three separately normalized target hidden states together with the token embedding, then feeds its normalized output back as the next step's input. EAGLE-4 adds the following features:

* EAGLE-Harness compares the expected gain from another draft depth with costs profiled once per hardware setup. Once per round, it also selects how many drafted nodes to verify. A deployment uses a dynamic tree at low concurrency or a dynamic chain at high concurrency. Within the selected mode, EAGLE-Harness reshapes the draft each round.

* EAGLE Efficient LM Head™ uses a rank-128 projection to score the full 248K-token vocabulary. The drafter's full language model (LM) head then scores only the best 128 candidates.

* Sliding-window attention limits the drafter to the most recent 4,096 positions. Draft key-value (KV) entries are appended without eviction, so the drafter's attention cost stays flat as the context grows.

![EAGLE-4 architecture showing the target model, single-layer drafter, draft tree, and EAGLE-Harness feedback paths.](/images/202610/eagle-4-overview.png)

*Figure 1. EAGLE-4 overview. The drafter reads three target hidden states and drafts with one sliding-window layer and the EAGLE Efficient LM Head. After each draft depth, EAGLE-Harness compares the expected gain from another depth with its measured cost. When drafting stops, it selects how many nodes the target verifies (blue). The tree is schematic.*

## Training with TorchSpec

TorchSpec supports EAGLE-4 training at launch (day 0). EAGLE-4 introduces the following architectural changes:

* Fc-norm and post-norm balance the magnitude of hidden states passed from the target model to the drafter, preventing attention drift.

* The `eh_proj` projection and pre-attention normalization map the concatenated embedding and target hidden states from twice the hidden-state width to a single hidden-state width.

* The decoder layer optionally supports no positional encoding and sliding-window attention.

* A mixture-of-experts (MoE) architecture can replace the dense multilayer perceptron (MLP), with an additional normalization after the dense layer.

* A low-rank approximation selects the top 128 tokens. The original LM head then performs the full-precision calculation for this reduced vocabulary.

![Input fusion, decoder-layer, and LM-head paths with optional-component boundaries.](/images/202610/eagle-4-training-architecture.png)

*Figure 2. EAGLE-4 training architecture, including input fusion, the decoder layer, and the LM head. The diagram identifies components inherited from EAGLE-3, additions in EAGLE-4, and optional components.*

TorchSpec also supports continued training from an EAGLE-3 checkpoint. This lets you warm-start the model without training for hundreds of GPU hours to match the baseline. Figure 3 compares training metrics for dense warm-start and MoE configurations.

![Three training plots for Qwen3.8-27B EAGLE-4 drafts on open-perfectblend, comparing dense warm-start and MoE configurations over optimizer steps.](/images/202610/eagle-4-training-metrics.png)

*Figure 3. Held-out simulated acceptance length, held-out loss, and MoE expert load imbalance (maximum-to-mean ratio) during training on open-perfectblend.*

The acceptance-length and loss plots track draft quality during training. The expert-load plot measures load imbalance: at each optimizer step, it reports the maximum expert load divided by the mean load across experts.

Figure 4 compares the share of draft GPU time spent in the LM head for chain and tree drafting.

![LM-head GPU-time shares: EAGLE-3, 27% for chain and 32% for tree; EAGLE-4 Efficient LM Head, 19% and 18%; EAGLE-4 full LM head, 66% and 67%.](/images/202610/eagle-4-lm-head-gpu-time.png)

*Figure 4. Share of draft GPU time spent in the LM head for chain and tree drafting. The comparison covers EAGLE-3 with a 32K-token pruned vocabulary and EAGLE-4 with either the rank-128 Efficient LM Head or the full 248K-token LM head.*

For EAGLE-4, the Efficient LM Head reduces this share from 66% to 19% for chains and from 67% to 18% for trees, compared with the full LM head.

## Serving with TokenSpeed

A dynamic tree's shape depends on values available only on the GPU during a round. Making these decisions on the host requires a device-to-host round trip every other draft depth and a CUDA graph for each draft width. TokenSpeed uses the following design to avoid these costs.

### Decisions on the device

At each depth, the graph forks a side stream that runs the EAGLE-Harness decision kernel. The kernel is bit-for-bit identical to the EAGLE-4 policy. It writes the next depth's width, a stop flag, and a go flag to GPU memory. The host never reads a score.

### One CUDA graph per depth

Each padded batch size has one decode graph, one CUDA graph per draft depth, and one finalize graph. The decode graph includes target verification and draft depth 0. Per-round values, including widths, pool offsets, work-list sizes, and live rows, are GPU scalars, so the same graphs serve every round.

### Rounds chained on the GPU

The host launches each round once, enqueuing the decode graph and the first draft depth back to back. Each depth graph ends with a one-thread kernel that reads the go flag written by its decision kernel. This one-thread kernel uses CUDA device graph launch to tail-launch either the next depth's graph or the finalize graph directly from the GPU.

The host never waits within a round. The forward stream depends only on the decode graph. This dependency lets output copies, KV and Gated DeltaNet (GDN) state commits, and input staging for the next round run alongside drafting.

### Paged KV cache with tree compaction

After verification, TokenSpeed compacts the accepted path to the front of the KV window. This preserves support for paged KV caches, prefix caching, and long contexts.

The device policy is the default on NVIDIA GPUs (`--speculative-harness-policy device`). Every EAGLE-4 result in this article uses it. Compared with the optimized host pipeline, the device policy's direct speed improvement ranges from no change to 1.5%. Its structural benefit is simpler graph management: the GPU chains graphs for individual depths without host involvement within a round.

The following optimizations account for most of TokenSpeed's single-request performance:

* Low-latency general matrix multiplication (GEMM) kernels use programmatic dependent launch for small verification and draft operations. These build on the TGV kernel and L2 prefetching in the EAGLE-4 reference implementation.

* The round pipeline keeps host work and preparation for the next forward pass off the critical path.

* GEMM epilogues publish directly into the all-reduce operation.

* Dedicated GDN kernels handle tree verification and state commits.

![Host and GPU timeline for one EAGLE-4 round, showing the stream layout, graph sequence, timing labels, and tail-launch connections.](/images/202610/eagle-4-gpu-timeline.png)

*Figure 5. One EAGLE-4 round for one request with TP4 in a release build. Median timings are 3.83 ms for the decode graph, about 123 µs per draft-depth graph, and 20 µs from the end of a round to the next decode graph. Each depth graph launches the next graph on the GPU. The host launches the round once; output copies, state commits, and next-round input staging overlap with drafting. The diagram shows five depth graphs, the most common count. The decode graph is shortened, and the side streams are schematic.*

## High-throughput optimizations

At high concurrency, a chain verifies only a few tokens per request. A fixed 16-row verification window therefore wastes most of the target forward pass. The following changes improve throughput:

* A width table sets the verification width `W` for each forward pass in the single decode path. Starting at 64 concurrent requests, `W` is 4; starting at 384, `W` is 3. At 256 concurrent requests, per-forward width selection and the width table together increase throughput from 7.2K to 15.6K tokens/s (Figure 6). Any chain drafter can use this mechanism, although only EAGLE-Harness currently does so.

* At 512 concurrent requests, no-wait launches for rounds that use the width table reduce GPU idle time from about 28% to about 3%.

* At 256–512 concurrent requests, a tensor-core kernel for GDN chain verification and an opt-in accumulation mode reduce the combined time for GDN verification and state commits by 15–17%. These changes improve end-to-end throughput by 3–5%.

* Argmax verification uses all-gather to exchange a few kilobytes instead of the full logits. At 512 concurrent requests, a prefill delayer in the C++ scheduler improves throughput by 22%.

![Five cumulative optimization stages at 64, 256, and 512 concurrent requests. Final throughputs are 12.8K, 27.4K, and 31.9K tokens/s, respectively.](/images/202610/eagle-4-throughput.png)

*Figure 6. Cumulative throughput improvements for EAGLE-4 on TokenSpeed at high concurrency. The label `n/a` indicates that a configuration was unsupported or unmeasured at that stage. Stages 1–4 are single runs. Stage 5 is the mean of two runs, with ±3% variation between runs.*

## Evaluation

All methods run on TokenSpeed with the same node, model, clients, and serving stack: four NVIDIA B300 GPUs, TP4, and Qwen3.8-27B in BF16 with thinking enabled. Only one server runs at a time.

At temperature 1, sampling uses top-p of 1 with no top-k limit. EAGLE-4 verifies a sampled tree through tree rejection sampling. MTP uses the model's own MTP layer as a static chain with draft-probability rejection sampling. Both preserve the target model's distribution.

Most optimizations for high concurrency apply to every method evaluated here, including the GDN kernels, argmax verification, prefill delayer, and collective tuning. The width table and no-wait launches are EAGLE-Harness features.

### Single request

The SPEED-Bench evaluation uses every tenth conversation in each category of the qualitative split: 88 conversations and 117 turns, with at most 4,096 tokens per turn. EAGLE-4 uses the dynamic-tree configuration 12/4/16.

The following table compares sampled and greedy decoding for a single request:

| Decoding mode | Decode speed (tokens/s) | Acceptance length | Speedup over decoding without speculation | Speedup over MTP |
| --- | --- | --- | --- | --- |
| Sampling (temperature 1) | 750 | About 3.6 | 2.85× | 1.28× |
| Greedy decoding | 985 | About 4.75 | 3.56× | 1.37× |

MTP uses its fastest chain length for this workload: five draft tokens, with configuration 5/1/6. The 7/1/8 chain used in the EAGLE-4 paper is about 5% slower on TokenSpeed.

### Agentic coding

The agentic workload replays SWE-smith multi-turn sessions. The first turn has about 50K tokens of context. Each subsequent turn adds about 800 tokens, and each turn produces 500 output tokens. The prefix-cache hit rate is roughly 90%.

With sampling at temperature 1, EAGLE-4 provides a 3.04× speedup over decoding without speculation for a single user. With greedy decoding, the speedup over decoding without speculation is 5.21× at one concurrent request and 1.94× at 16 concurrent requests (Figure 7). Over this range, EAGLE-4 generates 4.1× to 1.9× as many tokens per GPU.

![Per-user decode-speed comparison for EAGLE-4 and decoding without speculation at 1, 2, 4, 8, and 16 concurrent requests. Speedup labels are 5.21×, 4.21×, 3.33×, 2.56×, and 1.94×.](/images/202610/eagle-4-decode-speed.png)

*Figure 7. Per-user decode speed on the agentic workload with greedy decoding (temperature 0), from 1 to 16 concurrent requests. Labels show the speedup over decoding without speculation. EAGLE-4 uses a dynamic tree for up to 4 concurrent requests and a dynamic chain from 8 concurrent requests. EAGLE-4 values are the mean of two runs.*

### Long contexts

The target's decode kernel reads prefix attention once per round. Only the draft tree's own keys pass through a small cascade kernel, while the drafter's attention remains within a 4,096-token window.

The following results use 1,024 output tokens with prefix caching disabled. Time to first token (TTFT) includes full prefill.

| Context length (tokens) | Decode speed (tokens/s) | Speedup over decoding without speculation | Full-prefill TTFT (s) |
| --- | --- | --- | --- |
| 2K | 1,029 | 3.73× | 0.07 |
| 32K | 776 | 3.08× | 0.51 |
| 128K | 784 | 3.36× | 2.45 |
| 250K | 745 | 3.42× | 5.83 |

TTFT is essentially the same with and without speculation: at 250K tokens, it is 5.83 s with EAGLE-4 and 5.79 s without speculation, even though EAGLE-4 also prefills the drafter.

### Lossless speculation and floating-point behavior

Three optimizations change floating-point rounding without changing the algorithm:

* The replicated draft embedding, which changes the drafter's input by at most one unit in the last place (ulp)
* The tensor-core GDN chain-verification kernel
* The opt-in GDN accumulation mode

Regression tests check every other optimization for bit-for-bit equality with the previous build.

Checks against a 32-bit floating-point (FP32) reference covered every divergence from the two GDN changes. All divergences were near-ties, with no systematic bias. Acceptance checks also confirmed that all three optimizations left acceptance unchanged.

At temperatures greater than 0, tree rejection sampling preserves the target model's distribution. Stopping based on tokens drawn in the current round would bias the output. Each sampled round therefore drafts to a depth fixed before its draws: the depth at which the previous round's cost criterion would have stopped.

## Looking ahead

Three optimizations and extensions remain in progress:

* Narrowing tree verification to the number of nodes selected by EAGLE-Harness
* Collecting cost measurements for larger batches and longer contexts
* Adding sampled chains for sampling at high concurrency

All three changes preserve lossless speculation.

Together, TorchSpec and TokenSpeed support EAGLE-4 from drafter training through inference. The evaluations in this article use sampled trees for sampling and dynamic trees or chains for greedy decoding. On the tested Qwen3.8-27B workloads, using BF16 and four NVIDIA B300 GPUs (TP4), the current implementation improves decode speed and throughput with the optimizations described above.