Blog

TokenSpeed is a speed-of-light LLM inference engine designed from first principles for agentic workloads, with a compiler-backed modeling mechanism for parallelism, a high performance scheduler, a safe KV resource reuse restriction, a pluggable layered kernel system that supports heterogeneous accelerators, and SMG integration.

We’re excited to announce TorchSpec, a torch-native framework for scalable speculative decoding training. TorchSpec streams hidden states directly from inference engines to training workers via Mooncake, eliminating the need to materialize massive tensors on disk or co-locate training with the target model. This design enables fully disaggregated pipelines where inference and training scale independently.