Blog

TorchSpec trains EAGLE-4 drafters, and TokenSpeed serves them with lossless speculative decoding. This post covers the training architecture, dynamic drafting, GPU execution optimizations, and Qwen3.8-27B evaluation results.

Day 0 support for Beam on eight GB200 GPUs with NVFP4 MoE kernels, sliding-window KV cache groups, and DFlash 2 speculative decoding.

How TokenSpeed MLA 2 adapts its attention pipeline and native query-row packing to Kimi K3 long-context decoding on NVIDIA Vera Rubin.

TokenSpeed optimizes Qwen3.8-Flash-Next by improving sparse attention, fusing residual operations, and reducing repeated work during multi-token prediction.

TokenSpeed delivers 216 tokens/s per user, 1.4× ATOM’s decode speed on 8× MI355X GPUs.

How TokenSpeed optimizes Kimi K3 on GB300 for long-context, multi-turn agentic serving through communication-aware LatentMoE sharding, KDA state replay, and shape-aware kernel fusion.

TorchSpec's Kimi K3 draft collection brings together EAGLE-3, DFlash2, and DSpark models with efficient distributed training, pipeline-parallel vLLM integration, anchored EAGLE training, offline workflows, and reproducible builds.

SMG, an engine-agnostic frontend and router, and TokenSpeed, a speed-of-light LLM inference engine, joined the PyTorch Ecosystem.

TokenSpeed adds Day-0 support for the 2.4T-parameter Qwen3.8-2.4T-A95B on NVIDIA Blackwell, including cross-node serving, unified FlatKV prefix caching, and DSpark speculative decoding.

LightSeek Foundation argues that engine-layer performance in LLM inference is commoditizing, and traces where value shifts next — into serving operations and platforms, then toward capital, GPU capacity, and data centers.

TokenSpeed enables Day 0 support for Moonshot AI's 2.8T-parameter Kimi K3 on NVIDIA Blackwell and AMD CDNA4 within one week, with a unified flat KV cache, disaggregated serving, and specialized CuteDSL, TensorRT-LLM, and Gluon kernels.

LightSeek Foundation joined the signatories of this open letter supporting open-weight models — models anyone can download, inspect, modify, and run — as a foundation for competition, transparency, and broad access to AI.

TokenSpeed brings native NVFP4 and MXFP4 serving to Thinking Machines Lab’s 975B-parameter open-source MoE model Inkling across NVIDIA and AMD accelerators.

TokenSpeed-kernel is a standalone open-source subsystem for LLM inference kernels, using layered APIs and registry-based dispatch to separate runtime logic from hardware-specific backend implementations.

The TokenSpeed inference engine achieved a record-breaking 580 tps running the Qwen3.5-397B-A17B model on GPUs. This extreme performance for agentic workloads is driven by systematic elimination of memory copies, advanced kernel fusions, and fully overlapped CPU-GPU execution, keeping the GPU saturated at all times. On the functionality side, TokenSpeed also supports hybrid prefix caching and unified Prefill-Decode state transfers to handle complex agentic serving scenarios.

EAGLE 3.1 improves speculative decoding robustness with FC normalization and post-norm hidden-state feedback, with training support in TorchSpec and deployment support in vLLM.

Slides for the PyCon US 2026 talk on building high-performance LLM inference in pure Python with PyTorch custom ops.

LightSeek Foundation shares the TokenSpeed project ethos and how external contributions fit into the project.

TokenSpeed is a speed-of-light LLM inference engine designed from first principles for agentic workloads, with a compiler-backed modeling mechanism for parallelism, a high performance scheduler, a safe KV resource reuse restriction, a pluggable layered kernel system that supports heterogeneous accelerators, and SMG integration.

SMG moves CPU-bound serving work such as tokenization, detokenization, reasoning parsing, tool orchestration, multimodal preprocessing, and routing into a Rust gateway layer, keeping GPU inference engines focused on tensor execution while avoiding Python GIL bottlenecks.

We’re excited to announce TorchSpec, a torch-native framework for scalable speculative decoding training. TorchSpec streams hidden states directly from inference engines to training workers via Mooncake, eliminating the need to materialize massive tensors on disk or co-locate training with the target model. This design enables fully disaggregated pipelines where inference and training scale independently.

This post is jointly published with the PyTorch Foundation and is also available on the official PyTorch Blog.