Thinking Machines Lab (TML) has released Inkling 2 Small (thinkingmachines/Inkling-2-Small), the successor to its Inkling model. Inkling 2 Small is smaller and sparser than Inkling: it activates about 19B of its 594B parameters per token, adds an attention-residual path that lets every layer read a learned mix of earlier blocks, runs a 384-expert MoE in every layer, and ships with a separately trained speculative draft.
Working closely with TML, TokenSpeed enabled Day 0 support for Inkling 2 Small on NVIDIA Blackwell and AMD Instinct MI350X, with text and image input, TML's DSPARK speculative decoding, and JSON outputs that respect Inkling 2 Small's message framing. On 4× B300 replaying AgentX agentic coding sessions, a single user gets 492 tokens/s of decode, with the same output distribution as the target model alone, at any temperature. It is available to try out today; see the TokenSpeed deployment guide to get started.
This post walks through what changed from Inkling, how TokenSpeed serves Inkling 2 Small, and early end-to-end numbers. We continue to optimize every layer of the stack—pushing tokens toward the speed of light.
What's new in Inkling 2 Small
| Inkling | Inkling Small | Inkling 2 Small | |
|---|---|---|---|
| Size | 975B total / 41B active; 66 layers, hidden 6,144 | 276B total / 12B active; 42 layers, hidden 4,096 | 594B total / ~19B active; 48 layers, hidden 4,608 |
| Attention heads | 64 query; 8 KV (full), 16 KV (sliding window) | 32 query; 8 KV everywhere | 48 query; 8 KV everywhere |
| Layer pattern | 5 sliding-window layers (window 512), then 1 full | 5 sliding-window layers (window 512), then 1 full | Same pattern, window 1,024 |
| Relative attention | Relative logits, no RoPE | Relative logits, no RoPE | Same, with L2-clipped queries, keys and relative head |
| Attention output | Output projection | Output projection | Sigmoid gate × attention sink, per head, then the projection |
| Residual path | Pre-norm residual | Pre-norm residual | Attention residuals: 8 blocks of 6 layers; each sublayer reads a learned mix of earlier blocks |
| Short convolutions | Width 4, static taps | Width 4, static taps | Same; input-dependent K/V taps on full-attention layers |
| MoE | 256 experts, top-6, 2 shared; first 2 layers dense | 256 experts, top-6, 2 shared; first 2 layers dense | 384 experts, top-8, no shared; every layer; capped SwiGLU |
| Inputs | Text, image, audio | Text, image, audio | Text and image (new ViT encoder) |
| Speculative draft | 8 built-in MTP layers | 8 built-in MTP layers | Separate 5-layer DSPARK draft |
| Checkpoint | BF16, with NVFP4 and MXFP4 variants | BF16, with an NVFP4 variant | One native checkpoint: NVFP4 experts in layers 0–39, MXFP8 in 40–47, the rest BF16 |
Two of these changes shape how Inkling 2 Small is served. With about 19B active parameters and an MoE in every layer, single-user decode is bound by kernel launches and dependency waits rather than math. And the attention-residual depth mix runs twice per layer, so it has to be cheap. The rest carries over from Inkling (see our Inkling Day-0 post).
Figure 1. Inkling 2 Small’s 48 layers form 8 attention-residual blocks of five sliding-window layers and one full-attention layer. Bottom: the sublayers of every layer, with what’s new in Inkling 2 Small in blue.
Serving Inkling 2 Small on the TokenSpeed stack
Inkling 2 Small's model code follows TML's reference implementation and is vendor-neutral, and images for the new ViT encoder are preprocessed in TokenSpeed's Rust gateway, matching TML's processor byte for byte. Most kernels are portable Triton shared by both vendors, with vendor-specific kernels only where they matter most, such as the expert GEMMs, attention and collectives.
On NVIDIA Blackwell, a speculative round is a long chain of short kernels, so we worked on both the math and the gaps between kernels: expert GEMMs run directly on block-scaled tensor cores, kernels overlap under PDL with side work on side streams, the whole round replays as one CUDA graph, and relative attention in the full-attention layers runs on FlashAttention-4.
On AMD Instinct MI350X, we serve the checkpoint exactly as TML ships it, with Gluon kernels for the expert and projection GEMMs.
KV cache layout
Inkling 2 Small reuses the flat KV cache recipe we built for Inkling, with no new state types: full-attention history in 128-token pages, five sliding-window groups that keep only the last 1,024 tokens, and convolution state per request. The attention-residual mix and the relative logits are recomputed in every forward pass, and the DSPARK draft simply adds five ordinary attention layers to the full-history group.
Figure 2. Inkling 2 Small’s per-rank cache at TP4. Only the full-attention history grows with the context; everything else is a fixed amount per request.
Structured outputs that respect the message framing
Inkling 2 Small frames every message with protocol tokens, so a byte-level JSON grammar could let a protocol token's bytes into a JSON string, where the gateway would read them as framing. TokenSpeed masks the reserved token IDs whenever generation is inside a JSON string, across rollback and the draft. The model can still reason first and then answer with valid JSON. To enable JSON output, see the deployment guide.
End-to-end performance and accuracy
NVIDIA B300 on agentic coding sessions
We replay AgentX agentic coding sessions on 4× B300 (TP4 + EP4) with and without DSPARK, using the deployment guide's command, for 20 minutes per concurrency point. The sessions come from the 256K-context AgentX traces: recorded coding-agent conversations of about 72 turns each, plus sub-agents, replayed as live multi-turn traffic. Prompts average 94k tokens, 94% of them prefix-cache hits, and replies average 860 tokens. Concurrency counts active sessions, not requests in flight, since sessions pause between model calls. Per-user decode is the median across requests.
Figure 3. Output throughput per GPU against per-user decode at P90 for Inkling 2 Small on 4× B300, with and without DSPARK, at concurrency 1 to 32. Ninety percent of requests decode at least as fast as the P90 value. Labels give the concurrency; higher is better on both axes.
DSPARK makes per-user decode 81% faster than the target model alone at concurrency 1 (492 versus 272 tokens per second) and 65% faster at concurrency 8 (323 versus 195 tokens per second). At every concurrency from 1 to 32, it also delivers 6–20% more output per GPU.
With DSPARK at temperature 1 and max reasoning effort, Inkling 2 Small on B300 scores 87/90 on AIME26 across three seeds and 93.4% (185/198) on GPQA-Diamond.
Looking forward
Next, we are bringing PD disaggregation to Inkling 2 Small, along with more parallel layouts and platforms. Stay tuned.
Acknowledgments
We thank Thinking Machines Lab for the partnership, early access to Inkling 2 Small and their reference implementation, and NVIDIA and AMD for their support.