Scheduler: admission, retraction and restore

The C++ scheduler (tokenspeed-scheduler/) decides, once per round, what each engine does next. This document covers one axis of that: at what granularity KV capacity is admitted, what happens when admission fails, and how a retracted request comes back — per engine role.

Companion documents: event-loop.md (the control/data plane split that consumes these plans), cache-concepts.md (the vocabulary below — prefix granularity, cache groups, LCM blocks).

1. Admission is per chunk

An optional --chunk-prefill-kv-size also bounds the KV read by a local prefill batch. The runtime converts its token capacity to max_kv_page_num using prefix_granularity. Admission rounds each request's cached-prefix-plus-current-chunk length up separately, then sums those page counts; shared prefixes still count for every request. A candidate may use only the remaining pages. When configured, the page-aligned token capacity must be strictly greater than max_req_input_len; startup rejects an undersized budget instead of reducing the prompt limit. Decode and remote PD landing do not spend this local-prefill budget. Omitting the limit preserves the existing policy.

The scheduler schedules a prompt's prefill in chunks bounded by max_scheduled_tokens (--chunked-prefill-size). It admits capacity for the chunk being scheduled, never for the whole prompt: schedulePrefill / schedulePrefillFirstChunk build one GroupDemand per cache group sized by this chunk's tokens, and the coordinator either grants the pages or the request stays put. Alongside the demands, one RequestProgress per request carries what it computed since its previous admission — the prefix pages completed and its computed-token count — which the coordinator publishes and reclaims inside the same Admit (AdvanceRequestProgress in scheduler/operations/progress.h is the one place that hashes those pages and builds it; see cache-concepts for why publication rides with admission).

Decode-role remote admission covers the complete unmatched prompt in one landing window. Local prefill chunk limits and junction cuts do not shorten it: the peer transfers the full prompt's KV past the hit and its prompt-end state (unless this node already holds that exact checkpoint, §3.2). The window must reach PrefillSize() so the first decode consumes its reserved input width. A truncated remote window would leave that reserve at zero and delay later cache growth even though the device has committed the full prompt.

L3 is fetched before admission, never by it. A request waiting for QUERY or GET is Prefetching and absent from forward candidates. Submitted keeps its ordinary admission meaning, including mixed-mode token reservations and retraction demand. The wait carries a Bootstrapping or Submitted continuation so PD bootstrap can overlap it. A completed read's bounded Ready hold moves with that continuation until admission acquires its Host sources. StorageTransferManager owns query/GET/PUT tickets, Host references and the bounded backup queue. The request holds no Device pages or request-pool slot while preparing. Runtime workers report object and retirement facts; the C++ group matcher chooses the reusable boundary and ordinary admission acquires the resulting Host entries. A soft deadline permits recomputation without retiring native I/O. Failed GET rows and detached GET jobs cannot publish data. An attached, retired GET may publish its successful rows even when other rows failed; ordinary group matching determines the reusable boundary. Buffers still owned by native I/O remain held. State groups use the same checkpoint matcher and fixed two-slot layout as local hits. Retraction remains an exact image restore, never an L3 re-prefill. The D role probes Device history and receives missing data from its peer. See asynchronous L3 for lifetime and admission limits.

A first admission with no free request-pool row skips prefix hashing and matching. This is a slot constraint, not a KV capacity failure: it does not trigger retraction. Asynchronous L3 preparation runs before forward scheduling and can still fetch Host data while rows are full, because it owns no row. Resident prefill chunks and restore admissions keep their existing paths.

What the probe may claim. Before the first chunk, matchPrefixAtAdmission probes the prefix cache for the prompt's leading pages and for a published tail inside the page after them (CacheCoordinator::ProbePromptPrefix, see cache-concepts): the hit boundary T is a token count, and extend_prefix_lens carries it as is. The probe is bounded in tokens, and the bound is the minimum of two rules: the configured replay tail (prefix_replay_tokens, at least the final prompt token, which is always recomputed to produce logits) and the request's own RequestSpec::max_cached_prefix_tokens (default INT32_MAX, no bound); a hit may end exactly at the bound. The per-request bound exists for prompt (input) logprobs: a request that returns them from position s needs logits for every position at or after s, and a cached position has none, so the runtime admits it with the bound set to s and the positions >= s are recomputed as ordinary prefill input whatever the cache holds. The bound limits the probe itself, not a later trim, so excluded hit pages are never claimed and the recomputed suffix lands on private pages. Only a Submitted prompt is probed: a retracted request is never re-probed; it comes back by restoring its image (§4), so no position whose logits were produced is ever matched against another request's pages or recomputed. The decode role of a disaggregated deployment never computes prompt rows (the prefill node returns the logprobs), so the runtime leaves its bound at the default.

The input's full-page chain hashes and their expansion into per-group cache keys are prepared once per request. A failed first admission reuses those immutable identities and probes the current Device and Host contents again; it never reuses a previous residency result. The keys are shared with the pending probe, including optional checkpoint retries. The pre-admission state owns the request's reference and releases it when the first admission succeeds. The pending match views the request-owned hashes. Mutable execution progress copies the reused prefix only after admission succeeds. KV-event descriptors are also registered only after first admission succeeds, before any new cache publication, so capacity-blocked retries do not rebuild them.

An eviction write-back releases its Device entries before its Host segment becomes visible. If recorded or in-flight segment keys would extend a submitted prompt's hit under the ordinary matching rules, first admission waits for the copy ACK and probes again. The request holds no pages or request slot while waiting and is not a capacity blocker; other prompts and decodes can run. Unrelated copies, keys without a complete usable checkpoint, and keys beyond the request's cache bound do not delay admission. The pending keys are only evidence that waiting may help: no load or hit accounting uses them before publication. The coordinator maintains their membership from eviction recording through segment settlement, including evictions by earlier admissions in the same plan, before the transfer manager drains them. Repeated records of one key keep it pending until all settle. A segment refused Host placement settles at the drain, so its keys cannot keep a later plan waiting. Each probe uses fresh cache residency with this maintained key set; it does not rebuild the set or reuse a previous reachability result.

Decode consumes an existing reserve in place when no page or pending State checkpoint became complete and no sliding-window block needs reclaiming; such a step needs no admission. Any other step publishes and admits on progress staged in place (see "resources and publication progress land when an admission succeeds" in §2), so a failed admission still leaves the request unchanged. Previously published checkpoint holders are released after the in-place step, as they are after a successful admission.

Two adjustments ride on top of the raw chunk size. Both are pure token arithmetic kept out of the planner: how a chunk is cut lives in scheduler/operations/prefill_chunk.h (PrefillChunkTokens is the one entry both prefill paths call), what each group demands for it in scheduler/operations/group_demands.h.

Junction cut. A chunk is never shortened to a page boundary: a prompt end publishes its partial page as a tail (§1.2), so a chunk ending mid-page costs nothing. The one cut is the junction of a hybrid hit: when the KV groups' hit T reaches past the longest state checkpoint T_s (AdmissionResult::junction_tokens, kept in CacheProgress::junction_tokens until a chunk passes it), the first chunk after T_s ends exactly at T, so the natural checkpoint it leaves coincides with KV already cached and the next prompt sharing that prefix resumes at T in every group. The next chunk's admission publishes that checkpoint (PublicationPoint::kStateCheckpoint): the state under the junction's key, but not the KV there, which is cached already, so the owner keeps writing its partial block and needs no copy-in (cache-concepts.md, copy-in). Every other chunk end publishes only its completed full pages.

Reserve. What an admission holds beyond the chunk it computes is stated once per round (PrefillReserve: decode width, whether the round completes the prompt, prompt headroom, the sliding-window growth recycling cannot fund) and turned into each group's page demand by ReservePrefillDemands — the only writer of GroupDemand::reserve_tokens. It picks the rule by the group's kind, never by call site:

• Full-history groups hold the decode slot — the chunk that completes the prompt reserves decode_input_tokens, so the first decode step is guaranteed a slot; intermediate chunks reserve nothing because they are not about to decode — raised on a decoding role's first chunk to the rest of the prompt plus the admission headroom (§4), so a partially prefetched request is never stranded.
• Sliding-window groups recycle slid-out pages, so they hold the decode slot, raised on a decoding role's first chunk, while another request holds pages, only to the growth recycling cannot fund, capped by the window's lookback (window - 1 tokens) and the prompt headroom. That growth is the larger of two terms: a prefix hit (a shared page frees nothing when it slides out), and the rest of the prompt when a local chunk leaves some behind (the next chunk reads this one's lookback before any of it can slide out). The hit term is 0 in a replayable group, whose pages over the hit are private (§1.3), and once the prompt's own tokens after the hit reach window - 1 + block_granularity: every hit page then slides out before the first decode. A remote landing has no later local chunk, so it prepays only the hit. Any other page holder raises the reserve, not only one sharing the hit: the prepay deliberately covers hit pages that may become shared after the admission. With no other request holding pages the raise is 0 and the group holds only the decode slot: hit pages are then held by the prefix index alone and recycle like the request's own, and the single-request bound (§1.4) counts no prepay, so a prepay could leave a request the bound accepts inadmissible on its own. Growth left unfunded by pages taken after the admission is left to the victim policy's empty-round retraction (§2). Broadcasting the headroom to them once kept a 54K-token DeepSeek-V4 prompt waiting on a pool that had room for it.
• State groups reserve nothing: their table is two slots whatever the prompt length, and a decode step writes the live slot in place (§1.2).

The decode slot is reserved on every role, the P role included. A P node never decodes locally, but with speculation configured the forward that completes a prompt runs the drafter once and writes its candidate block into the decode_input_tokens slots behind the prompt — the same window a decoding role verifies into — before plan.remote_decode ships the candidates. Without that reserve those rows have no page and fall to the dummy slot, and the block attention that reads them back proposes garbage. What P does not reserve is decode growth: no admission headroom, no overlap protection (§3.1). The capacity model (§1.4) states the same split.

1.1 Head-of-line: an incomplete prefill holds the queue

holdsHeadOfLine breaks the candidate loop after scheduling a chunk of a prefill that is still incomplete. Nothing behind it is scheduled that round (on the P role, prefill_chunk_yield_tokens lets whole prompts complete beside it; see the end of this section). A resident chunk that fails admission ends the prefill phases too: admitting anything behind it would strand it.

The reason is that per-chunk admission gives an in-progress prefill no claim on the capacity it still needs. Let a newcomer take pages while a prompt is half prefilled, and the half-prefilled one may never assemble its remaining chunks — it holds pages, makes no progress, and eventually has to be retracted, throwing away work already done.

The cost is one round of queue latency; the alternative risks a retraction cycle. One full chunk already saturates the GPU, so interleaving a newcomer into the same round buys no throughput to offset that risk.

Holding is a property of the incomplete, decided at plan-build time. A chunk that reaches the end of its prompt moves the request to PrefillDone inside the same plan build (SchedulePrefillEvent picks the successor state by whether the window reaches PrefillSize()), so a prompt scheduled in full never holds the line. A round can therefore carry several prefills that complete this round plus at most one truncated one — and the truncated one is necessarily last, because holdsHeadOfLine seals the phase the moment it appears; a chunk that gives way (below) is admitted before the whole prompts it carries, but its row moves behind theirs. The runtime relies on this order (unified_path.md, "Prefill requests without generated outputs"). With 6K of budget left and prompts of 2K / 1K / 10K waiting, the round schedules the 2K and the 1K in full and the first 3K chunk of the 10K prompt, then stops. The same rule makes the finishing round of a chunked prefill cheap: its final chunk releases the line within the round, so the prompts queued behind it start in that round, not the next.

Decodes are not part of the line. In mixed mode the decode batch is built before the prefill phases and takes its token budget first (§3.3), so a round is all decodes + the completing prefills + at most one truncated prefill. Outside mixed mode, prefill and decode never share a round at all — decodes get the round only when no prefill scheduled, or ahead of the prefills while a prefill/decode interval is owed (§3.3) — so head-of-line only ever orders prefills against each other, never a decode behind a prefill.

A chunk that cannot finish may give way (P role, opt-in). With prefill_chunk_yield_tokens N (--prefill-chunk-yield-tokens), a chunk that cannot finish its prompt this round — a resident's, or a new prompt's first — shares its forward with waiting bootstrapped prompts that fit whole. Once per round, wholePromptsAfter probes up to eight Submitted candidates (for a new prompt's chunk, only those behind it) the way admission would. A candidate qualifies when its first chunk could complete it: no eviction store pending on its prefix, no junction cut, no in-forward checkpoint spanned from inside a page, and — beside a resident chunk on a model with State groups — no page shared with what that chunk's admission is about to publish (its probe would be stale). What it takes is counted past its hit, replay included: a 100k-token prompt with 1k uncached tokens takes 1k. carriedPrompts keeps them in FIFO order while they fit N, the batch, the free request slots and, with max_kv_page_num, the page budget. The chunk then ends on the last page (with aligned State checkpoints, segment boundary) at least their total before the end of its budget (YieldingChunkBudget). Once it is admitted and keeps the head of line, carryWholePrompts admits each carried prompt only if its first chunk completes it on the budget left; the first capacity failure ends the fill. None of them holds the head of line, so the round is still completing prefills + at most one truncated prefill, and the truncated one still runs last: carryWholePrompts then moves its row behind the carried prompts', so the completing prefills form a prefix of the batch.

The chunk gives way only while such prompts wait, and only what they take: fewer than N plus one page. N is at most half of max_scheduled_tokens (B), so with R = B - N - page + 1 the long prompt advances by at least max(R, page) tokens in every round it starts with the whole budget (a resident always does; a first chunk may find part of it spent by prompts admitted ahead of it), less a hit's replay window, and finishes U uncached tokens within 1 + ceil((U - B) / R) such rounds. A final chunk never gives way. A resident chunk whose admission fails still blocks everything behind it, and the chunk that gave way is admitted first again next round. Off (the default), the plans are the ones built without the option. The fused role does not offer it: there, retraction picks the largest incomplete prefill first, and a carried prompt could make the yielding request that victim.

New prompts retain submission order by default.--enable-short-prefill-first opts in to ordering by prompt tokens minus currently reusable Device or Host prefixes, plus any tokens replayed to regenerate replayable groups; equal work preserves submission order. Resident chunks retain priority. The policy applies to P and fused local prefill; D's remote admission and decode ordering are unchanged.

Only the oldest waiting prompt counts successful later first admissions. After eight, it receives selection priority; once it starts or leaves, the next oldest gets its own allowance. Priority is reevaluated after each admission, including within one batch. Failed admissions and resident chunks do not count. When chunk yielding is also enabled, carried first admissions count toward the same allowance; the bounded carry scan retains FIFO order. On the fused role, a first admission granted after capacity retraction also counts; a resident chunk granted capacity does not. Priority cannot bypass resource or transfer dependencies, and an inadmissible protected prompt does not block other admissible requests.

Both policies share the same admission and execution path. Ranking does not reserve pages or replace the admission-time cache check. With the option disabled, there are no extra prefix probes for ranking or bypass counts. When the new-prompt phase runs with the option enabled, ranking probes every eligible submitted candidate, without a fixed scan window. This finds short work beyond the front of a deep queue, but adds prefix-index lookup cost on each such round. Request hashes and keys are reused; cache matches are not cached across rounds because publication, eviction and transfers can change them.

1.2 State groups: two slots, natural checkpoints

A State group (a recurrent layer's state, AttnKind::kState) keeps no history: one block holds the whole state after some prefix, so the group has no token span (block_granularity = 0) and its table is two slots whatever the prompt length. Slot 0 is the input checkpoint a forward reads, slot 1 the output it writes. A prefill row reads slot 0 and writes slot 1; a decode row reads slot 0 when it holds a block and slot 1 otherwise, and writes slot 1 — the row contract the runtime's block_tables carry. Every admission states what the pair does through the group's StateStep (MakeGroupDemands sets it for all State groups at once):

• kRotate — a prefill chunk, or the first decode of a fused request whose prompt end is a publication point: the previous output becomes the input (slot 1 → slot 0) and a fresh block is acquired for slot 1. The input it displaces is released, which only frees it when nothing published it. On a prompt's first admission the pair is shaped instead: slot 0 is the checkpoint the probe matched or, with none, a fresh zeroed block — the zero state — and slot 1 is fresh. A request therefore holds its two blocks from its first admission on, and every later rotation swaps the released input for the fresh live slot; a pool sized at two blocks per live request (§1.4) back-pressures at admission instead of wedging at the first decode, when the rotation would otherwise need a block the pool no longer has.
• kRelease — every later decode step, and a fused first decode whose prompt end is not one (prefix caching off, or a dead prompt end the in-forward checkpoint skips, below): slot 0 is let go and slot 1 is written in place. Nothing is acquired, so a decode consumes no state capacity (ConsumeReservedTokens skips State groups; the in-place fast path releases the input through ReleaseStateInputs).
• kKeep — a restore and the D role's first decode: the pair is left as it is (the landed or restored slot 1 is the state the decode continues from).
• kLand — the D role's remote landing: slot 0 is the claimed prompt-end checkpoint or null (nothing on this node reads a zero state), slot 1 a fresh block the peer's transfer fills.

Natural checkpoints. A state is published only at a point where its owner stops writing it — a PublicationPoint other than an ordinary chunk end (§1): a fused prompt end (the first decode's admission publishes slot 1 under the prompt's key before rotating it into slot 0), a chunk end that falls on a junction (§1; the KV there is cached already, so only the state is published), and a finish or retraction on every role (kFinal: the P's transferred prompt, the D's and the fused engine's last decoded state), except the dead prompt ends and finishes that the in-forward checkpoint skips (below). An ordinary chunk end or decode step publishes nothing for a State group (CompletedPages::publish_state is false). The key is the key of the exact token count T: the chain hash of the last page when T is a page multiple, otherwise the tail hash hash@T the KV groups use — so one key names a request's progress across every group kind (cache-concepts.md). A published state block is never written again: rotation moves it to slot 0, where it is read only, and the next step releases the request's reference while the cache keeps the block.

For State groups with checkpoint_granularity > 1, only prefill-produced states may become reusable checkpoints. Publication remains in the ordinary scheduling path: a later admission publishes any retained in-forward checkpoint before rotating or reclaiming the owner's blocks. The first fused decode admission may deliver the preceding prefill's prompt-end and interior checkpoints; it does not publish a decode-produced state. Later decode states stay private at finish and retraction even when their positions are aligned. A prefill finish may still publish its endpoint, and retraction still saves the private live state for that request's own restore. The D role does not publish its transferred prompt state in this restricted mode. Unrestricted State groups keep the policy above. Forward result events carry no checkpoint data and do not trigger publication.

In-forward checkpoint (opt-in). By default a prompt that diverges inside the last page of an earlier long prompt, or resends it, resumes from an earlier checkpoint: the prompt-end state cannot serve a hit that must replay the final token. With state_checkpoint_in_forward (--enable-state-checkpoint-in-forward, off by default, requires prefix caching) the forward that spans the last prefix-page boundary before a long prompt's replay tail (PrefillStateCheckpointTokens: a prompt longer than max_scheduled_tokens; the tail is at least one token, prefix_replay_tokens and the largest replayable group's window) also writes the state there. No chunk is cut: the admission of that forward acquires one more block per State group outside the two slots (StateStep::checkpoint, AdmissionResult::state_checkpoint_blocks), the forward's rows carry it as a third entry with the tokens before it (extend_state_checkpoint_lens), and the runtime stores every State group's state at that position into it. The request holds the block in CacheProgress until its next publication point -- the next chunk's or first decode's admission, a finish or a retraction -- which publishes it under the boundary's page hash, like a natural checkpoint, with the sliding window behind it as the endpoint's window, even at a chunk end, never as an unproven boundary that eviction takes first (cache-concepts.md). A boundary a chunk ends on needs none (the chunk end's slot 1 is published there, as at a junction). The scan emits states only whole pages from a forward's start, and the budget a batch's decodes leave rarely ends on a page, so a chunk ending less than a full chunk before the boundary ends on a page when the budget reaches one, unless a junction at or past its end comes first (the chunk after the junction starts there anyway). A chunk that would still span the boundary from inside a page (a hit on a prompt-end tail) ends at the boundary, and so does one whose admission cannot also hold the extra block (a cache with no spare block): the next chunk's admission publishes its live slot there, at the cost of one more round. Multi-turn traffic whose next turn re-renders the end of the previous prompt then reuses the common prefix immediately, and the prompt ends and finishes that such a turn cannot resume at are skipped (dead checkpoints, below).

An active request retains one such aligned input checkpoint and its cached, non-replayable sliding windows across both tiers. Its full-history tables already hold the remaining dependencies. This matters for restricted State: a long decode cannot publish a replacement checkpoint, so releasing the input as soon as it leaves slot 0 can lose the next turn's reusable prefix while its producer is still running. The hold uses ordinary references in the shared cache pools, acquired once when the input checkpoint becomes available; decode does not scan or refresh it each step. Retention requires a nonzero input checkpoint boundary, including when publishing completed pages from a prior chunk. At finish, its entries become recent without counting as a cache hit, then ownership is released. An admission that cannot fit drops its own optional hold before retrying, and retraction drops it before imaging the request. Neither an incomplete checkpoint nor a partial set of sliding-window dependencies is retained.

Dead checkpoints. A next turn that re-renders the end of the previous prompt and its reply resumes at neither that prompt end nor its finish. That is the traffic the in-forward checkpoint serves, so with state_checkpoint_in_forward the scheduler skips both on a model with State groups, KV tails included; without it nothing is skipped. A prompt end T inside the page its pre-tail checkpoint C (PrefillStateCheckpointTokens, the in-forward checkpoint's boundary) starts is dead (promptEndCheckpoint); any other prompt end is published as before. A request that resumed from a tail beyond C keeps its prompt end: it extends an earlier prompt end in place, and its successor likely extends this one the same way. An identical prompt cannot resume at T (it replays the final token) and resumes at C; a longer prompt repeating T's partial page resumes at C too, recomputing less than a page. The fused role's first decode skips a dead prompt end and is then an ordinary decode step: the partial page stays the owner's and the live slot is written in place (kRelease). That decision reads only the prompt and where the request resumed, so the prompt's admissions take it too, before C exists, and reserve no copy-in block for a dead prompt end (§1, cache-concepts.md, copy-in); it stands even if C has left the cache by the first decode. C normally exists by then: the forward spanning it writes it, and a chunk that starts or ends there finds or leaves the state cached. A publication in place at T — the P role's finish (§3.1), or a fused finish no decode ran past — skips T only once C exists: pending from the request's final forward, which the same publication publishes, or cached in every State group, on either tier as the probe takes it (promptEndIsCoveredByCheckpoint). The request's own hold on a cached C is optional (above): once an admission lets it go and evicts C, such a publication publishes T. A fused finish past the prompt end publishes like a chunk end as well, whether or not the prompt has a pre-tail checkpoint — its completed full pages, no tail and no state (finishCheckpointIsDead). A next turn that repeats the previous reply verbatim therefore resumes at C, or at a prompt end the rule kept, and recomputes the reply (without the in-forward checkpoint it would resume at the finish). The skipped entries occupy no cache block and never reach Host. A retraction still publishes its frontier (§2), the D role publishes as before, and a model without State groups keeps its tails: they are resume points there. With aligned State checkpoints (below), which require the in-forward checkpoint, C is the prompt's last segment entrance -- for any prompt longer than one segment, whatever the chunk budget -- so the rule covers a prompt end in the first page past it. It never skips an aligned checkpoint: a dead prompt end is never a page boundary, and a finish past the prompt end holds a decode-produced state, which those groups keep private anyway. Their other unaligned prompt ends still publish a KV tail, although no hit resumes there either.

Short prompt ends. A prompt too short for a pre-tail checkpoint (PrefillStateCheckpointTokens is 0: by default a prompt within one chunk budget) has no C to make its end dead, so the rule above publishes it. With drop_short_prompt_end_checkpoints (on with --enable-state-checkpoint-in-forward, P role only) the P role's finish skips that end too when it lies inside a page: it publishes the full pages alone, like a chunk end (shortPromptEndIsDropped). A re-rendered next turn cannot resume at that end either; a longer prompt that repeats the whole prompt resumes at an earlier checkpoint instead and recomputes the rest of it. A page-aligned end keeps its state: it is a chain boundary a longer prompt resumes from. The decision reads only the prompt, so it holds whatever the cache holds, and nothing before the finish depends on it: the P role reserves no copy-in block for its prompt end and never retracts. Such a finish publishes no end state, and it marks no input superseded (below): the tail it resumed from, if any (a junction's, whose state a junction cut published), may still be the deepest checkpoint a prompt extending it resumes at. Under write-back the pages it leaves, with no checkpoint at or under them, are dead entries an eviction drops instead of writing back. The fused role keeps its short prompt ends, and Validate refuses the flag there: the drop is a P-role policy, chosen from PD measurements. The same case exists at the fused first decode, which publishes a short prompt's end and moves off it into its copy-in blocks; applying the drop there, decided at admission like promptEndCheckpoint, is a possible follow-up.

A finish settles when its last forward has landed. forward::Finish (and the P role's pd::SucceededEvent) moves a page-holding request to Finishing and nothing else; after every batch of events, settleFinishedRequests publishes the frontier of each Finishing request with no forward in flight (kFinal: KV tail and state, unless they are dead, above) and releases its pages (ReleaseEvent). One path for every schedule: a request that finished quiescent settles with its Finish event — the timing every caller always saw — and one whose finish was decided on a landed token while the overlap schedule had already dispatched the next decode settles when that decode's result lands, at the frontier the tokens it reports leave. The live slot is therefore never published while a forward writes it, and a paused loop, which drains its forwards without building plans, settles its finishes all the same. The runtime reports every landed forward row, tracked or not (the control plane's forward commit, ts-control's output), so the count always reaches zero. A Finishing request is never scheduled and never a retraction victim; an abort releases it without publishing.

A finish publishes the state before the stop token. The device ends a request where the control plane does: a verify round keeps its accepted tokens only through the first stop token and within the token limit, so the state commit and the frontier never pass the token the finish is decided on, and the decode the overlap schedule dispatched behind that round runs the request as a padding row and reports no token. Whatever the schedule and however many tokens a round accepts, the finish frontier T is therefore the stop token's position (the last kept token's for a token limit): the state and KV before it, a prefix of the next turn's prompt when its chat template keeps the reply verbatim, whether or not it keeps the stop token. A finish the device cannot decide (a stop string, a grammar, a client abort) reports every token the round kept instead, so the frontier still names the state the device holds (cache-concepts.md, "Recurrent state").

Hits. The probe (§1, "What the probe may claim") resumes a hybrid model at the longest T with a published state checkpoint in every State group and KV coverage in every token-spanned group; the state lands in slot 0 as a claimed cache block (a Host checkpoint is loaded into a fresh slot 0 like any Host hit). The tiers combine under the Host-suffix rule (Cache concepts): the KV is the Device's prefix plus a Host suffix, and each State group's checkpoint comes from whichever tier holds it, the Device first -- so Device KV with a Host checkpoint resumes, loading the checkpoint alone. When the KV groups reach further than the state (kv_tokens > num_common_tokens), the difference is the junction: the first chunk ends there and publishes the checkpoint the cache lacked, so the next prompt that shares the prefix resumes at the KV boundary in every group. For unrestricted State groups no alignment is imposed on a chunk, and apart from the opt-in in-forward checkpoint above no checkpoint is produced inside a forward: a chunk's state output is its own natural checkpoint. The backend contract for the two slots is in Cache concepts.

A superseded input leaves first. A prompt that resumes at a tail checkpoint on the Device -- where its previous turn ended (AdmissionResult::resumed_tail_hash) -- dates it with its access epoch at admission, and its finish dates the end state it publishes with the same epoch. The two would rank level in the Device's eviction order, so block location would decide, and as often as not the end state the next turn resumes at would leave first. The finish that publishes the end state (on the P role, once the transfer completes) therefore marks that input superseded (MarkSupersededStateInput) on every role, under either Host write policy and without a Host tier: its checkpoints and its tail's KV blocks are registered again at access epoch 0, level with the blocks no cache entry holds. It stays cached and matchable -- the same prompt sent again resumes there -- and is the first eviction candidate: an admission or a restore that needs its room evicts it, counted as request reclaim, and one whose later victims make the room alone keeps it. A request that resumes there again unmarks it until its own finish (an earlier request finishing meanwhile leaves it unmarked, even once an older request's publication of the same tail has dated it back; a request's own restore, which dates what it loads back with the request's epoch, is no later hit), and a request that fails before its finish (a transfer that fails) marks nothing, nor does a finish whose end state is the input itself (a D request that claimed its prompt-end checkpoint and leaves before any decode). One at a page boundary -- a junction or an in-forward checkpoint, which other prompts resume at -- is never marked. On the fused role a turn also publishes its prompt end, at its first decode and at the same epoch: that one and the end state still rank level, since which of them the next turn resumes at depends on whether the client sends the reply back as it was generated. A dead finish that publishes no end state (above) marks the input all the same: the prompt end, or the pre-tail checkpoint the next turn resumes at instead, was published or found cached earlier. A short prompt end the opt-in drop skips (above) is the exception: nothing replaces the input, so it stays unmarked.

Superseded checkpoints. With in-forward checkpoints, each turn of a conversation that re-renders the end of the last prompt resumes at the checkpoint the turn before wrote and writes its own, and the prompt ends and finishes that such a turn cannot resume at are skipped (dead checkpoints, above). A long conversation therefore leaves a chain of in-forward checkpoints, and only the newest two serve its next turn (one that diverges inside the last turn resumes at the older). With demote_superseded_state_checkpoints (on with --enable-state-checkpoint-in-forward; it requires in-forward checkpoints and has no effect on the D role or without State groups) a finish that publishes an in-forward checkpoint C(n) past the page boundary it resumed from, itself an in-forward checkpoint C(n-1) whose request resumed from an earlier one C(n-2), demotes C(n-2): its State entries and the sliding-window blocks behind it, less any block C(n-1) or C(n) resumes with, are dated back to access epoch 0 on both tiers and marked demoted (CacheCoordinator::DemoteEntries). The Device and a write-through Host evict them first, ahead of every dated entry and of the aged checkpoints checkpoint retention ranks at its cutoff, which keeps the live conversations' prefixes cached longer. A write-back Host segment plan takes a Host-only checkpoint boundary whose checkpoints are all demoted after dead entries, Host boundaries the Device covers or supersedes, and the Host copies of KV a request holds on the Device, but before the Host copies of unheld Device KV (Cache concepts). The scheduler records which in-forward checkpoint each finish published and where its request resumed (CheckpointLineage, bounded). A shared C(n-2) stays: one another first admission also resumed at, or one an acquisition or publication touched after C(n-1)'s request resumed there (BoundaryTree::Node::latest_state_access_epoch) -- a fork, or a system prompt every new conversation resumes at. The scheduler sees no conversations, so a boundary only this conversation has resumed from so far is demoted even when another request published it before, until the next hit there. A boundary no in-forward checkpoint was recorded at (a junction or a prompt end alone) is never demoted. That holds for its State entries, not for the window blocks it shares with C(n-2): another checkpoint whose own window reaches into C(n-2)'s, such as the prompt end of the turn that wrote C(n-2) when that one is not dead, or a junction just before it, can still need a demoted block. No entry a request or a transfer holds is demoted either; a hit, touch or republication dates an entry forward again and ends its demotion. No new store or eviction write-back copies a demoted entry to the Host: a finish's store skips it, and an eviction under the write-back policy drops it whatever evicted it (counted under that cause and in HostCacheStats::demoted_entries_not_written_back). A Host copy made before the demotion stays there, demoted too, and one an eviction write-back is still carrying lands demoted (TierTransferManager::DemoteInFlightEvictions). Only a demotion marks an entry demoted (PrefixCacheIndex::CachedBlockMetadata::demoted): a demoted session's entries (DemoteSessions) and superseded inputs sit at epoch 0 too, but keep their own rules with the flag on or off.

Aligned state checkpoints. A State group can declare checkpoint_granularity, a multiple of the prefix-hash grain. Its published and matched states must end on that boundary. Intermediate prefill forwards end on whole segments; a remaining prompt fitting the budget runs once. The existing in-forward checkpoint mechanism saves the entrance of its last segment, (prompt_tokens - 1) / grain * grain, including when the whole prompt fits one round. Thus 8192 tokens run in one forward while saving the state at 4096. Prefix reuse stays strictly below the prompt length, so a full 8192 match resumes at most at 4096. Only that extra entrance is retained; aligned prefill endpoints remain reusable through the same admission path. A decode step reaching an aligned position creates no checkpoint. The grain constrains each request's logical position, independently of its packed batch offset. Unrestricted groups preserve their original scheduling and publication.

Capacity. A State group holds at most two blocks per live request on every role — the pair, never a page per token — so ReservePrefillDemands reserves nothing for it and the capacity model (§1.4) charges 2 per request. An in-forward checkpoint's block is a third only between its forward and the next publication point, and only when the pool holds it then (otherwise the chunk is cut at the boundary); the bound charges nothing for it. On the D role the landing writes slot 1 and slot 0 stays null unless this node already holds the prompt-end checkpoint (§3.2).

Retraction and restore. A victim publishes its frontier like a finish, so its slot 1 rides the L2 leg of the image when there is a Host cache and the unpublished slot 0 goes to the snapshot pool (§2). The restore rebuilds the pair in place (StateStep::kKeep), copying the published slot 1 into a fresh block rather than claiming it — the resumed request writes it next — so a resumed request needs no checkpoint it did not already own (§4).

1.3 Bounded replay

A sliding History group can be declared replayable (CacheGroupConfig:: replayable, see Cache concepts): it leaves prefix caching entirely — never matched, published or streamed — and the model regenerates its rows from re-fed prompt tokens. DeepSeek V4.1's SWA rows and compressor tails are the motivating case: caching them persistently costs more than recomputing a bounded window, and the prefix hit should depend on the global KV alone. What a hit re-feeds is the group's whole retention window: retention keeps exactly what the queries after the hit read, and every page the group retains must be regenerated, so there is no second number to declare.

The cache facts live on the CacheCoordinator, next to the specs they derive from: ReplayWindowTokens() (W, the largest sliding_window_tokens over the replayable groups) and ReplayTokens(P) (min(W, P), what a hit at P must re-feed). The scheduling rules live with the other chunk-cutting rules in scheduler/operations/prefill_chunk.h; the forward planner never branches on replay — the two decisions below reach the common path only through PrefillChunkTokens, the one chunk-sizing helper both prefill paths call, which also carries the junction cut (§1):

• After a prefix hit at P, the first chunk re-feeds [P − min(W, P), P) ahead of its new tokens, so the queries at and after P find the whole window regenerated. This is the only place tokens are re-fed.
• No prompt's final chunk is shorter than W (ChunkKeepingFinalWindow): a chunk that would leave 0 < remainder < W is shortened so exactly W remain. A junction (§1) that falls inside that final window yields: no chunk end satisfies both rules, so the chunk passes the junction and the State groups publish their checkpoint at the prompt end instead of the request waiting forever. The model narrows its decoder to the prompt's last window, which must therefore arrive in one forward — as new tokens, never by re-feeding what an earlier chunk of the same request already computed.

The re-fed rows are ordinary forward input — they consume the round's token budget like any other row and input_length counts them — but they are not progress: TokenContainer::Window{begin, size, replay} keeps begin/size as the tokens this chunk computes (replay is non-zero only on a hit's first chunk), and MakePrefillInfo derives the model input [begin − replay, begin + size). The runtime sees the pair extend_prefix_lens[i] = begin − replay and extend_replay_lens[i] = replay on the ForwardBatch; positions [extend_prefix_lens[i], extend_prefix_lens[i] + extend_replay_lens[i]) regenerate the replayable groups only and must not be written into any other group, whose rows already sit in the shared cached pages the hit claimed.

Capacity: a replayable group claims no hit pages, so at first admission its table is empty and CacheCoordinator::Admit itself materializes it as a sparse private suffix from the replay window's first token — slots below stay null holes, exactly as absolute-slot tables require — while closed groups keep the dense demand beyond P the caller stated. The FSM derives the window's replay from the same coordinator (SchedulePrefillFirstChunkEvent), so no event or scheduler operation carries a replay parameter. Later chunks change nothing.

Budget: SchedulerConfig::Validate requires max_scheduled_tokens ≥ W + max(W, P) — a hit chunk re-feeds up to W and must still carry the prompt's final window whole, and keeps at least a prefix page of new tokens when the window is smaller than a page (a junction cut may shorten a chunk to any positive count, so it imposes no minimum of its own) — the first chunk spends the hit window before sizing its new tokens (and waits for a fresher budget when none is left), and in fused mixed mode the decode batch leaves that same amount for a pending local prefill (MinPrefillChunkTokens). Replay is derived at admission only: a retracted request's replayable pages are imaged with every other group's, and the chunk it resumes has replay = 0 by construction, so nothing is re-derived at restore (§4). Replayable groups combine freely with State groups: a junction inside the final window yields to the window rule (above), and nothing else about a State group constrains a chunk.

PD: the decode role caches none of a replayable group's pages, so remote admission allocates its whole retained window, independently of the global prefix hit. full_suffix selects the same complete window for the destination manifest, its validation and the prefill peer's layerwise selection: every page intersecting the last sliding_window_tokens − 1 positions. Ordinary cached windows still reuse their matched pages. The prefill role replays on its own local hits exactly as the fused role does, so every page of the retained tail exists when it sends it. The decode role computes no prompt rows, so SchedulePrefillFirstChunkEvent gives a remote prefill replay = 0 and Admit leaves a demand that already names the landing's sparse suffix alone; the landed tail is what the first decode steps read, with no regeneration anywhere.

1.4 What bounds a single request

The runtime's usable request length is the minimum of the model context and MaxSingleRequestTokens. Input admission leaves room for the first decode/verify window. The generation budget is capped at usable_length - prompt_length; that input reserve is not subtracted again from the remaining output budget.

MaxSingleRequestTokens is a startup bound computed by binary search over CapacityModel::SingleRequestGroupPages (csrc/scheduler/capacity_model.h): the largest prompt whose worst-case working set — dense or sliding history with its decode reserve and overlap-depth protection, the blocks a tail handover holds for one admission, two slots per State group — fits the pool. It is not a live check against currently free capacity; a prompt within the bound can still fail admission right now and waits. The bound counts no sliding-window prepay (§1), and a request with no other request holding pages prepays nothing, so the prepay never makes a request the bound accepts inadmissible on its own. Beside other requests the prepay can ask for up to one window lookback more than the bound counts; such a request waits until their pages are released, at the latest until it is alone, unless later arrivals keep taking the pages. Under Host write-back (§3.1) the Device evicts by whole segments, and a segment holding a block in use cannot leave — the request's own blocks included, such as the pages under the checkpoint it resumed from. So once whole segments cannot fund the admission of a request no other one holds pages beside, the blocks they passed over leave one by one (cache-concepts.md, AdmissionPlanner): alone, there is no release left to wait for. Only the admission's last variant takes them: an optional part of it — an in-forward checkpoint's block (§1.2) — is dropped first and never pays for such an eviction. Beside another page holder no block leaves one by one: the request waits for that holder's release (on the P role, until its PD transfer is acknowledged), where write-through, which evicts block by block, could fund it at once. The block-by-block pass lifts only the segment rule. Under either policy the blocks the admission's own prefix hit keeps never leave, and the tail-handover charge (below) is bucket-blind, so on a KV-sharded pool of a few pages a request alone can still wait for good, when its hit holds the only cache left and its copy of the tail needs a page in a bucket that is full.

A request's expired slots and released State input are different from live pins: Admit releases their table references before evicting those blocks. Once the plan selects their reclaim set, a later segment may absorb that set even though the table still holds its references during planning. Segments rejected before that selection are reconsidered before any block-by-block fallback: a cached rejection is valid only while the selected reclaim blocks remain unchanged. Another retry requires newly selected reclaim blocks, so an unchanged pin cannot cause an endless scan. The merge counts each physical block once, and minimization keeps or restores the combined set whole. This lets a P request rotate past a junction and evict its unpinned canonical KV duplicates in the same admission. Another reader's reference, or a prefix hit this admission keeps, still prevents reclaim.

The CapacityModel is deliberately config-only: it reads every SchedulerConfig field that is known before a pool exists and no total_pages, validating that subset through SchedulerConfig::ValidateCapacityInputs(). That is what lets the Python recipes size a pool from the same model before the arena is allocated (recipes/scheduler_bridge.py builds an unsized config and asks ConcurrentGroupPages(max_total_tokens, max_context_len) for each group's demand at max_batch_size live requests), and then lets the Scheduler bound requests against the pool they sized. The per-request working set — decode_width + overlap_schedule_depth * decode_width protected tokens, the next verify window a speculative decode's draft steps write on the roles that decode (DraftWindowTokens, which scheduleDecode reserves from), the two slots of a State group (kStateSlots), the most blocks a sliding window can need behind a boundary (one more than behind a page-aligned one, SlidingWindowBlocks) — exists in that one file; neither side restates it. The two answers are tied by an invariant the model's tests sweep: for one live request of L tokens, ConcurrentGroupPages(L, L) is never below SingleRequestGroupPages(L) in any group, so a pool sized for the configured concurrency admits every request the bound accepts.

Per group, ConcurrentGroupPages charges: a State group two slots per live request (the working set does not grow with history); a prefix-closed history group ceil(T / g) dense pages plus, per request, ceil((g - 1 + protected) / g) for the unaligned tail and the protected tokens (the draft window included) that may spill past it; a sliding group, per request, ceil((min(W - 1, ctx) + decode_width + protected + g - 1) / g) resident pages, plus one in-flight prefill chunk behind its lookback (or, on the decode role, the landing bound min(dense, lookback + window) per request). With the prefix cache on, every token-spanned group adds per request the blocks a tail handover holds for the length of one admission (tailHandoverBlocks): a hitter copies a tail's blocks out of shared entries that stay pinned as sources until the copies land. A tail is at most P − 1 tokens, so that is ceil((P − 1) / g) blocks on a role that prefills locally and floor((P − 1) / g) on the D role (its partial block is a fresh page, not a copy); a sliding group caps it at its lookback. The fused role, whose first decode publishes the prompt end and keeps writing past it, adds the same count again for the copy-in blocks (copyInReserveBlocks, cache-concepts.md, copy-in): the request holds them beside its tables from the admission that allocates the blocks they serve -- the first, in full-history groups -- until that decode copies the published blocks into them, and the first admission can be a hitter's that still pins its copy sources. A dead prompt end (§1.2) holds none, but whether a prompt end is dead depends on the prompt, so the charge stays. The charges are bucket-blind, so a sharded group whose copy's bucket is full while other buckets have holes can still make such an admission wait.

Speculative decode admission grows from the committed token frontier plus the verify spans still in flight, the span being scheduled and the next span, which the draft steps write; that last span stops at the request's token limit, so it stays inside the admission limit, a request without a declared budget reserves none, and the capacity model charges it per live request. Already reserved slots cover that extent first. A prefill interruption must not charge the same speculative slots again when decode resumes; otherwise rejected draft tokens accumulate in the logical tables beyond the context-length bound.

A State group's working set is the pair, on every role and in every state of the request: a chunk reads slot 0 and writes slot 1, a decode writes slot 1 in place, the D role lands into slot 1, and a restore rebuilds the pair (§1.2). Tests cover small pools that must reject an oversized request instead of accepting a request that can never produce a forward.

2. Retraction: when admission fails

maybeRetractForCapacity fires when no prefill made progress this round (PlanBuild::NoPrefillProgress(): nothing was admitted and no resident prefill advanced a chunk) and an admission failed for capacity. Decode steps do not count as progress — they release no capacity, so a round of pure decode leaves a stalled prefill exactly as stuck as an empty one. Imaging a victim needs a snapshot pool (SchedulerConfig::HasSnapshotPool(), §4) and a blob slot; a round that finds no victim it can image falls back to the last resort below and aborts one instead, so a capacity block never waits on a host budget.

Only a resident the round being built has dispatched no work for is offered as a victim, in both tiers (below). One with this round's forward out cannot quiesce before the retraction runs, and the next round schedules it again before retraction runs, so waiting for it would last as long as its reserve does while every request blocked beside it holds idle pages; nor can its pages free in this round, which is what an abort is for. A request armed by the debug knob (below) sits its round out: marked scheduled with nothing dispatched, it stays offered. When every offered resident still has an outstanding forward or PD transfer — an earlier round's result still landing — capacity retraction waits before ranking or planning images: no possible choice can quiesce in that round. Otherwise the normal rank and Host-fit order applies among the offered residents, including waiting for a better-ranked one whose earlier forward is still landing. Decode release counts are computed once per candidate for each ranking pass, before sorting.

A retracted request loses no work. It is suspended: its block tables are imaged to Host byte-for-byte, its Device pages are released, and the restore (§4) copies the image back into fresh pages and resumes the request in the state it left — mid-prefill at its next chunk, or at its next decode step. The cost of a retraction is the image's bytes (proportional to the pages held) plus the client-visible pause; nothing is recomputed.

Retract-and-grant, in one round. The retraction serves a specific request — the first eligible candidate whose admission failed for capacity (AdmissionFeedback::capacity_blocker). Victims are retracted and the blocked admission is retried in the same plan build, looping (retract → retry → retract) until it fits or the victims run out. The freed capacity therefore reaches the request it was freed for within the round; there is never a free page waiting for whoever asks first next round, which is what previously required a cross-round capacity barrier. The empty-round retraction (below) is the one exception: it offers the freed pages to the round's decodes first. A round with no room left for the grant — its batch full or its token budget spent — retracts nothing: the victim would serve nobody, and its pages would wait for whoever asks first next round. A D-role round full by batch count retracts nothing either, although its remote admission takes no batch slot: the batch already runs as many requests as it can hold, so retracting a resident for the admission would add no throughput and cost an image and a restore.

With the fused prefill gate, a prefill excluded by the current window or watermark is ineligible both as a capacity blocker and as a grant recipient. The blocker-redirect order below filters every prefill alternative through that same live eligibility check. Such a prefill cannot cause a retraction or acquire its freed pages through a retry; decode-driven retraction and readmission restores are unchanged.

Two edges of the loop:

• The victim may BE the blocker (a resident request blocked on its own next page is the preferred victim). It comes back through the restore phase, so the grant is redirected: to the next request whose admission failed for capacity this round (AdmissionFeedback::capacity_blocked, in phase order), else to the first waiting prompt, else to a resident prefill the head-of-line rule (§1.1) kept from trying this round — granting the pages straight back to the victim's own readmission is the loop the grant exists to break. The held-back prefill is who the victim's wait would starve: behind a prefill blocked on its next chunk no other prefill runs, so two prompts mid-prefill holding every page, with a waiting readmission sealing new prompts and no decode left, would wait for a completion nothing can bring. With nobody to redirect to, the victim is not retracted (except in the empty-round retraction below, when a decode is left to take its pages): the retraction would cost its image's copies and its restore and serve no one, so it waits for a completion instead.
• A grant that cannot legally join its round's batch (a fused prefill beside decodes outside mixed mode) still retracts one victim, and the blocker waits for the next round, whose restore phase runs first (§3.3): the victim's own readmission, once its image has landed, can take the freed pages back before the blocker tries. So while decodes keep running, the victim can be retracted and restored as often as every other round, paying its image and its restore each time, and a redirect (above) does not break that loop. It ends when the decodes finish: with no decode in its round, the grant joins that round.

The image has two legs, both stream-ordered. retractVictim first publishes the victim's computed frontier into the Device index exactly as a finish does (AdvanceRequestProgress at PublicationPoint::kFinal + CacheCompletedBlocks: the full pages, the partial page as a tail, and each State group's slot 1; §1.2), then TierTransferManager::SplitRetractionImage classifies every data slot of every table and StartRetractionStores takes the image as the split planned:

• The L2 leg. A slot whose Device block is a published prefix entry becomes an ordinary Host L2 entry, acquired — like every Host copy — in the Device block's bucket (cache-concepts.md, placement). Keys already Host-cached, or carried by a store still in flight that publishes them at its ACK (never an eviction write-back's segment, which may publish late or not at all), are pinned rather than copied again; the rest ride one WriteBackOperation issued with StoreSourceGuard::kStreamOrdered. The image holds a pinned CacheBlockRef on each Host entry until the restore lands, so the planner cannot evict it and ClearCache refuses, but the entries are published prefix cache like any other and later prompts may hit them. The pin follows the publication, not the ticket's block: when the store's ACK finds the key already canonical on Host (an L3 prefetch of the same page by another prompt landed first), PrefixCacheIndex::Register redirects the ticket to that entry, and CompleteWriteBack hands the entries as published to every Retracted request that waited for the op (RetractionImage::FollowPublished), which re-points its slot to the canonical block and lets its own unindexed block return to the pool. An image therefore never pins a Host block the index does not know, and the restore's L2 rows always come from published entries.
• The pool leg. Every other slot — a State group's unpublished slot 0, replayable groups, any block the index did not take, the published slots the Host does not take (below), and with no Host cache the whole image — gets a block of the request-private snapshot pool (snapshot_allocator; never prefix-indexed, never evicted), again in the Device block's bucket. These rows, plus the slot-state blob of the request's runtime row (sampling state, speculative readiness — exported by the runtime into snapshot_slot), ride one SnapshotStoreOperation; the op may carry zero page rows, the blob always.

The split plans the L2 leg before anything is acquired or evicted for it. The slots to copy are grouped by boundary (a content hash, with every group's slot at it), the shallowest first, and the Host takes the leading boundaries that fit whole (CacheCoordinator::NumHostBoundariesThatFit, using the shared eviction planner). All rows fit jointly against shadow capacity before any placement is committed; the retraction (AcquireHostBoundaries) evicts exactly the planned victims and places exactly those rows. The first boundary the Host cannot take whole goes to the pool with every later one, even where the Host has room for some of their rows. Under the write-back policy the plan takes Host room for every boundary as an eviction write-back does, by whole segments first (AcquireHostSegments' plan: whole victims, none above a page an eviction write-back is still bringing, boundaries dropped from the deepest), then block by block, as above, for the boundaries the segment step dropped, against the Host less its victims. Both policies can use all eligible capacity: the segment step changes victim preference, while the shared block planner fits the remaining rows jointly. Placing block by block may take the Host page above an eviction write-back in flight. The write-back still publishes at its ACK — the settlement reads the tail hint and the checkpoint boundaries under its node, not the gap above — and the next dead-entry collection takes it as dead entries under the gap, and may widen the gap: a copy wasted, accepted because the image comes before the cache.

An image does not fit when no blob slot is free (SnapshotSlotAllocator, max_retracted_requests slots) or when the pool cannot hold what the split leaves it (ImageShortfall): kSnapshotPool when it could not even with every published slot on the Host, kHostL2 when it could but the Host takes too few of them. retractVictim then reports the shortfall before anything is acquired or evicted for the image (the publication it did is written back as the victim's progress). The last resort is an abort, in one place.chooseVictim ranks only candidates whose image fits (imageShortfall: a blob slot is free and the split of the image the retraction would take once it has published, projected without mutation, has no shortfall — the probe and the retraction run the same plan of the L2 leg); when none does — and without a pool none ever does — it names the newest retractable resident it is offered (the least work lost among those whose pages can free this round; exempt requests excluded, they finish on their own reserve, except in the empty-round retraction below) and Scheduler::onImageDoesNotFit aborts it with that resident's shortfall: fsm::AbortEvent frees its pages and slot in that very round, the blocked grant proceeds exactly as after a retraction, and the plan records it (ExecutionPlan::aborts, a SchedulerAbort{request_id, AbortReason::kImageDoesNotFit, detail}) so the runtime fails the request toward its client with the shortfall and the knob to raise (--retraction-snapshot-host-gb / --retraction-snapshot-ratio, --retraction-snapshot-max-requests, or for kHostL2 --kvstore-size (or --kvstore-ratio when no size is given), with the Host's size, its pinned entries and whether copies are in flight). The same site aborts a chosen victim whose image its own publication left without room, which the probe does not foresee: a tail that publication drops under the per-prior cap takes its Host entries along; under the write-back policy the segment step ranks Host victims by what the Device holds, which the publication adds to; and under KV parallelism a slot the publication re-points to the Device entry already holding its key rides the snapshot pool in that entry's bucket, not the one the probe counted. The split made after the publication can then fall short. Waiting instead would deadlock once every resident needs a page; charging worst-case host room at admission would cost concurrency on every request to protect against a rare shortfall. The debug knob (below) is not capacity pressure: it probes its victim with imageShortfall before retractVictim publishes anything; a victim that does not fit is refused, logged and left running. A victim refused only after the publication (any of the ways above) is aborted at the same site, never left running: the publication registered blocks it would write again (its partial page as a tail, and a State group's end state, which a decode step writes in place), and a published block is never written again by its owner (the prefix cache depends on it).

The victim's pages are released — and grantable — immediately, before either copy has run. Both legs are stream-ordered: their tickets pin only the Host destinations, and the runtime fences the forward thread's stream on the D2H copies' completion ahead of everything else the plan does to those pages — that stream carries the zeroing, fences the forwards, and gates a granted remote prefill's RDMA trigger (see DeviceHandle.execute and event-loop.md) — so the copies read the old bytes whatever the scheduler does with the pages. The Host write-back policy's eviction stores (§3.1) are the third stream-ordered kind, for the same reason: the pages an admission or a restore evicts fund the admissions and restores of the same plan. These are the only stores that pay for their ordering on the forward's critical path, and they have to: the pages are gone in the same round.

Every other store pins its Device sources until the ack. Boundary publications of a live request and the finish-time flush (write-through only) are issued with StoreSourceGuard::kPinnedUntilAck: the ticket holds a CacheBlockRef on each source, so the block stays cached and unevictable — the admission planner cannot take it, ClearDeviceCache refuses, NumNewlyReleasableLcmBlocks does not count it — until CompleteWriteBack publishes the Host entry and drops the pin. Nothing else needs to know the copy is in flight, so the runtime copies on its own stream and no forward waits. A cached block is never written again by its owner (prefix reuse already depends on that), so the pin alone makes the copy race-free. Both guards share one ticket (StartPendingStores(guard, cause) drains the publication queue; StartRetractionStores builds the image's L2 leg from the victim's tables; StartEvictionStores drains what the plan's admissions and restores evicted), and the op carries source_pinned to the runtime, which branches on the guard and never on the cause. Whatever its guard or cause, the op is a WriteBackOperation, batched in the plan's WriteBackBatch (WriteBackOp in Python) and acknowledged by CompleteWriteBack: "write back" in these names means any copy from the Device to Host L2, under either Host write policy, not only under write_back (§3.1). The HostStoreCause (publication, finish, retraction or eviction) names the drain point; it only selects the Host L2 counters a store's outcomes land in, which no scheduling decision reads. They count deterministic events only (no clock), so the mirrored schedulers on every rank hold the same values.

Per-victim quiescence, not global. A request whose own forward is still out must not be retracted — its result would land on pages it no longer owns, and the image must know the landed tokens — and one whose pages a PD transfer still pins must not be either. Both are checked on the chosen victim; if it is not quiescent — an earlier round's result still landing — retraction waits for it rather than sacrificing a worse-ranked request. A request the round being built has dispatched work for is never offered (§2, above), so this wait ends when that result lands. Two global gates remain. An in-flight load-back: it is writing pages its admission owns, and the victim policy cannot see that write. And an in-flight pinned store: it holds Device capacity the ack returns by itself, so retracting anyone for that capacity would be the thrash of §4 — the blocked admission retries against the released pins next round instead. Stream-ordered stores — both legs of an image, and eviction write-backs — hold nothing and gate nothing, and an in-flight restore gates nothing: its request is Restoring, invisible to chooseVictim.

The forward-out check is a count in fsm::ForwardResources, incremented when a forward is scheduled and cleared when its result lands. It lives in the resource bundle rather than on the states that consume a token, because a forward is out against the pages, and the bundle is what holds the pages. Every page-holding state carries one bundle, and a transition moves it whole to the successor state — so the count, like the pages, cannot be dropped on the way from one state to the next.

The bundle follows one rule: resources and publication progress land when an admission succeeds; a state transition only moves them, never modifies them. The coordinator fills the block tables inside Admit; the scheduler advances the cache progress (prefix-hash chain, pending junction) in place, hands it to that same admission — which publishes the newly completed pages — and keeps the advance only if the admission succeeds (StagedCacheProgress in scheduler/operations/progress.h). A failed attempt truncates the chain back to its length before the attempt, which only appended to it, and restores the other fields from copies. Staging in place instead of on a copy keeps a step's cost independent of the context length: the chain holds one hash string per page. A failed admission therefore leaves both untouched (except the optional retained input checkpoint, which admission may release for capacity), and the retry re-derives the same completed pages and asks for their publication again. Committing progress before admission would record the pages as hashed while never publishing them. The scheduling events carry nothing but the shape of the next state (chunk size, decode reserve). An intermediate prefill chunk produces no token but does write KV, so it reports back with an empty ExtendResult: the arrival is the point, not the payload. Work this engine does not perform — the peer's decode on a P node, the peer's prefill on a D node — is not counted here; those are fenced by the PD transfer ack.

Victim choice (chooseVictim, shared by D and fused). Neither tier loses work, so the key is who frees the most for the least interruption: an incomplete prefill first — no client is streaming it yet, and the mid-prompt prefill is usually the request that blocked on its own next page, so retracting it and granting its pages to a prompt that can finish is the shortest path out of head-of-line — largest first, freeing the most at once; then decode work by most newly releasable LCM blocks and fewest generated tokens — the needed pages with the fewest victims disturb the fewest clients, and among equal frees the client that has streamed least is interrupted. Only offered residents are ranked — none the round being built has dispatched work for (§2) — and among them only those whose image fits, the Host L2 leg planned as the retraction places it (the abort fallback above takes over when none does). Exempt in both tiers: a request whose reserve already covers its whole generation (Request::ReserveCoversGeneration) — retracting it frees exactly what its restore must take back, pure thrash — and it is not aborted either, except in the empty-round retraction below. Excluded by state: Retracted, Restoring, RemotePrefilling.

Empty-round retraction. The exemption trusts the admission reserve to cover all growth, and a sliding-window group's does not always: it prepays only against pages other requests hold at the admission (§1). A page that becomes shared after the admission (a later prefix hit on it, or publish-time dedup of identical prompt pages) frees nothing when it slides out, and a sequence still shorter than the window, or a prompt whose next chunk still reads its lookback, has nothing to recycle yet. When no resident can be retracted under the exemption and the round built nothing — no forward, restore, remote prefill or decode — no completion of it can free capacity, so the choice is repeated with the exemption dropped. A round that runs anything keeps the exemption: that work moves toward completions that free capacity without a retraction. So does a round in which a finished request still waits for its last forward (Finishing, §1.2): its pages come back when that forward lands. The empty-round retraction changes who may be chosen and where the freed pages go first; the quiescence checks, the global gates and the abort of a victim whose image does not fit still apply. The round ran nothing, so none of its decodes got a page, and they are offered the freed pages before the blocker: a generated token survives any later retraction, so each empty-round retraction that lets a decode run is progress no retract/restore cycle can undo. Granted to a prompt instead, the pages can go to one that blocks the same way on its next chunk and becomes the next victim; granted to nobody (the victim was the blocker and no other request wants its pages), the victim can be restored ahead of the decodes and take the same pages back, round after round. A victim that is the blocker, with nobody to redirect to, is therefore retracted here whenever the round has a decode left to offer its pages to — there is no completion to wait for — and otherwise waits as above. Only when no decode fits does the blocker get the grant, by the same rules as for any victim, and the loop may take several victims in one round until a decode or the blocked admission fits.

Stall diagnostic. NextExecutionPlan counts consecutive rounds whose plan carries nothing — no forward, remote stream, cache op or abort — while requests are live and nothing is out (no forward result, PD transfer or tier transfer) that could change the next plan, and warns at 1024 such rounds and every doubling after. It never changes a plan.

Forced retraction (debug_force_retraction_interval, a debug knob; 0 off) exercises the path without pressure: every |interval| plans the oldest Decoding (> 0) or Prefilling (< 0) request is armed — kept out of that round's batch so its outstanding forward lands — and retracted at the first plan where it is quiescent, bypassing the victim policy's exemption but not the image-fit refusal (probed before the publication; a refusal after it aborts the victim, see "does not fit" above). It runs at the start of the D and fused grammars, before the restore phase, so the freed pages are granted by the ordinary phases.

The P role never retracts: buildPrefillWorkerPlan does not call maybeRetractForCapacity (the only two call sites are the D and fused grammars). See §3.1 for why.

3. Per role: explicit phases

Each role's plan builder is a sequence of phases over one stable candidate order: submission order (requests_ is a vector — the FIFO — with a side index by id for lookups). It is identical on every rank because the mirrored schedulers receive identical submission batches, so no sort is needed for determinism, and within a phase older requests win — FIFO is the fairness policy, not an accident of key order. There is no priority ladder: what used to be a rank in a ladder is now the position of a phase in its builder, readable top to bottom. A round schedules each request at most once (PlanBuild::Scheduled), whatever states it moves through while the phases run.

A pass's mutable state is split in two on a layer boundary. PlanBuild — the output plan, the batch under construction, budgets, and the composition flags — is held by the role grammars alone, and every operation enters the batch through one gate, pushOperation, where budget and flag accounting live. The per-request admission layer (admit, schedulePrefill*, scheduleDecode) sees none of that: it receives only the output plan (to record fresh pages to zero) and an AdmissionFeedback (admission_failed, capacity_blocker, the capacity_blocked list the grant may be redirected to), so it can report outcomes but never compose the batch.

3.1 P — prefill worker

Phases: completed prompts out on plan.remote_decode (their pages stay pinned until the transfer finishes, so releasing them outranks feeding more prompt work), then the shared local-prefill phases (scheduleLocalPrefillWork): resident chunks, then new prompts. With prefill_chunk_yield_tokens, a chunk that cannot finish gives way to the whole prompts waiting behind it, which complete in its forward, ahead of its row (§1.1). Each carried prompt's pages are then pinned until its own transfer is acknowledged, like any completed prompt's.

Reserve: the decode slot, nothing else. The completing chunk reserves decode_input_tokens like every role, because the drafter writes the first candidate block there before the remote decode carries it to the peer. The growth reserves stay off: no admission headroom (nothing is ever retracted), no overlap protection (no local decode is ever in flight).

The transfer ACK (pd::SucceededEvent) publishes the prompt like a finish — its full pages, its partial page as a tail and each State group's prompt-end state (§1.2) — and releases request ownership; cached entries remain subject to normal eviction. With the in-forward checkpoint a prompt end it covers is published like a chunk end instead, and with drop_short_prompt_end_checkpoints so is a short prompt's end inside a page (§1.2).

Host L2: write-back. With host_write_policy kWriteBack (--kvstore-write-policy write_back; this role or the fused one (§3.3), no L3, and in the runtime one pipeline stage and no query shard) nothing streams to the Host at publication. Instead what an admission evicts for the cache leaves by whole segments — or, for a request alone that whole segments cannot fund, block by block (§1.4) — and is written back in the same plan, by the drain right after the build (one stream-ordered op, §2); how a segment is recorded, placed, published or dropped is in Cache concepts. Every attention-TP rank of the replica runs the same plan and copies the rows it owns, each Host block in its Device block's bucket (§5, "Every tier transfer pairs blocks of equal residue"), and each round's head hands the scheduler only the ACKs every rank holds (CacheOps::acknowledged, event-loop.md), a rank that owns none of an op's rows included. Nothing depends on an ACK arriving before the next plan is built, so it can land several plans later: at overlap depth 1, or across a rank's idle rounds under attention DP. So the two tiers together hold distinct content up to about the sum of their capacities, less pinned and in-flight blocks, where write-through keeps on the Host a copy of what the Device holds.

Retraction: none. A P node's pressure valve is the transfer itself — pages are pinned until the peer acknowledges, then released wholesale. Retracting a prompt whose KV is mid-transfer would strand the decode side.

While P holds a request, prefix publication uses a copy of its block table. Deduplication may redirect that copy to canonical cache blocks, but the real request keeps the physical source blocks named by queued layerwise transfers. A client abort waits for the transfer retirement event; a failed transfer also waits for outstanding local forward results before freeing its tables. Failed requests waiting for those results receive no further forward work. Successful and failed terminal paths both release the source references, without an extra long-lived copy of the tables.

This preservation is limited to P publications while the request owns its transfer sources (CompletedPages::preserve_source_blocks). Fused and D requests use ordinary canonical deduplication once their publication is safe. A fused owner that keeps writing past its tail retains the existing copy-in rule: register its own tail without taking another owner's canonical block, then copy any tail block the index took into its pre-reserved private block.

The PD pin is a function of the FSM (Scheduler::pdTransferInFlight). On this role every page-holding state is pinned — the peer's decode reads the pages from the first scheduled chunk until the PD ACK finishes the request. On the D role the pin is RemotePrefilling: the peer's prefill is writing the destination pages, and RemotePrefillDone ends it by leaving that state. A fused engine never transfers. Because the pin is the state, no event handler can forget to clear it.

A transfer reads and writes pages through the page IDs its operations exported, so while the pin holds, a pinned table loses no block a transfer may still address. P publication keeps its source references, including State checkpoints, and D publishes its landing table after successful receive. The owner-tail copy-in runs only on the fused role, which never transfers.

A pinned request that ends — an abort, or a failed transfer — does not free its pages: the transfer may still be reading or writing them. It parks whole in PdTransferring, which is pinned on both roles, until the transport reports pd::TransferStoppedEvent once every reader and writer on every rank stopped (or the transfer's success lands first), and every outstanding local forward result has landed. Observing transport retirement before a local result records that fact in PdTransferring; a repeated abort preserves it and cannot restart the transfer wait. The state owns this record along with the retained resources, so leaving it needs no separate abort-marker cleanup. Nothing is scheduled, published or retracted from that state.

Restore: n/a. Nothing on this role is ever Retracted, so the restore phase does not exist in its grammar; Validate refuses a snapshot pool on it.

3.2 D — decode worker

Phases:

0. Forced retraction, if the debug knob is set (§2).
1. The one restore this round may start (§4) — the first of the ranked Retracted requests whose landed image fits, resuming a streaming client, so it takes capacity ahead of fresh work. It rides beside the batch as a cache op: no token budget, no batch slot, no forward.
2. The decode batch (scheduleDecodeBatch) — every PrefillDone first decode and Decoding step; decodes consume no token budget on this role.
3. At most one remote admission — the whole prompt at once (the peer prefills it), riding plan.remote_prefill beside the decode batch: it consumes no token budget and no batch slot, so there is nothing to defer for. Capped at one per round because each reserves an entire prompt's pages; a queue's worth in one round would drain the pool before any KV arrives. Head-of-line (1.1) does not apply — there is no mid-way. A restore that failed for capacity in phase 1 seals this phase: a newcomer taking the pages it waits for would starve it. Remote admission therefore bypasses local prefill chunking and prefix promotion boundaries; the destination tables must cover the entire unmatched suffix and each replayable group's complete retained window advertised by the transfer manifest (§1.3).
4. maybeRetractForCapacity (§2), whose grant also rides beside the batch (a remote admission) or joins it (a blocked decode).

A decode batch, a restore and a remote admission coexist routinely. Nothing on this role is ever Prefilling: a prompt is the peer's work, and a retracted request comes back by restore, never by a local prefill — so no round is ever claimed by an extend forward, and the capacity model charges no prefill peak (§1.4).

What the D probe may claim. The remote admission probes this node's own Device cache (CacheCoordinator::ProbeDecodeDevicePrefix) so the peer transfers only what is missing. For the token-spanned groups it is the ordinary probe: extend_prefix_lens is the hit T, page-aligned or a published tail, and the block containing an unaligned T is left a hole for the peer to fill whole; the tail's full blocks below it, when a group's blocks are smaller than a page, are the hitter's own copies like on every role (pages_to_copy). For the State groups only one checkpoint counts: the one at exactly the prompt end (PrefixProbe::state_at_prompt_end). When every State group holds it, the admission claims it as slot 0 and reports state_prefix_cached on the PrefillOperation (per extend row on the ForwardBatch), and the peer omits the state from the transfer; the first decode then reads slot 0 and writes slot 1 (StateStep::kKeep). A state checkpoint at any shorter T is ignored and claims nothing — this node runs no prefill that could continue from it, so the KV hit is trimmed independently and the peer's prompt-end state lands in slot 1 as usual. On finish (and on the transfer ACK) the D publishes its own frontier like a fused engine: the KV tail and slot 1 (§1.2), which is how the exact prompt-end checkpoint of a repeated prompt comes to exist on this node.

Retraction and restore: victims are chosen by the shared rule in §2 — everything resident is decode work (prompts the peer prefilled, in PrefillDone or Decoding); RemotePrefilling is pinned by the peer's in-flight prefill and never a victim. The image is taken from this node's own pages and nothing crosses the PD wire: the receiver and admission record were already released at RemotePrefillDone, and the bootstrap token lives in the request's token container, so a PrefillDone victim restored to PrefillDone still gets its first decode exactly as before. On every role the first decode after a restore carries its input token explicitly (decode_input_id = LastToken(), the marker fsm::RestoreMarker that RestoreDoneEvent sets and the first ScheduleDecodeEvent consumes): the request sits in a new request-pool slot, and the capture its last forward left belongs to the slot it was retracted from, so the device cannot fill the input itself as it does for an ordinary overlapped decode. A restore runs no forward, so a decode engine whose attention layout cannot run an extend (head TP, --attn-head-tp-size; see docs/serving/parallelism.md) retracts and restores like any other.

3.3 Fused — one engine, everything local

Phases, mixed mode (enable_mixed_prefill_decode): forced retraction (§2) and the one restore this round may start (§4) first on every mode — the restore resumes a streaming client and takes no budget — then decodes: a client is streaming them, and a long prefill chunk must not starve them of token budget; then the shared local-prefill phases (resident chunks, then new prompts) spend what remains, then maybeRetractForCapacity. The decode batch leaves pending_prefill_reserve untouched when a local prefill is pending: the budget a hit chunk needs to re-feed its replay window and still advance (MinPrefillChunkTokens, §1.3; zero without replayable groups). A restore that failed for capacity seals new-prompt admission for the round (new_prompts_sealed), as on the D role.

Phases, non-mixed: after the restore, the prefill phases run first and alone; decodes get the round only when no prefill scheduled. No prefill reserve is needed — scheduling order is the capacity priority.

Prefill/decode interval (prefill_decode_interval N, either mode; 0 is off): a round that schedules a local prefill chunk owes the running decodes N decode rounds. Until they have run, decodes are scheduled first, and a round that scheduled any skips the prefill phases. Every chunk starts a new interval, but no prompt is held back more than N rounds in total. Each Submitted or Prefilling request that a round does not schedule counts the rounds in which the interval skipped the prefill phases while it waited (Request::PrefillDeferredRounds). The count is never reset: a victim retracted in such a round counts the round, and the count survives the restore. While any request has counted N, the scheduler suspends the interval and plans the round as if no decode rounds were owed. So the interval adds at most N decode steps to a prompt's time to first token, whatever its number of chunks and however many other prompts' intervals it waits through. A round that the prompt would have waited anyway, for a request slot, for capacity, or behind an older chunk, counts too.

A suspended round runs without the interval: outside mixed mode, the overdue prompt's remaining chunks, and any older chunk ahead of it in the line (§1.1), run back to back while the decodes stall. A round with no decode to schedule runs prefill work as usual, so the engine never idles for the interval. A deferred prefill tries no admission, so it never blocks on capacity; a deferred resident chunk holds its pages idle and is the first victim (§2) when a decode blocks. A capacity grant that admits a prompt (mixed mode) still runs and starts a new interval. The counters follow the plans alone, so mirrored ranks agree; under attention DP each replica counts its own rounds. The P and D roles refuse the knob: neither runs local prefill and decode on one engine.

Retraction: the shared victim rule (§2): incomplete prefills first, then decode work. The Host cache decides the image's split, not whether there is one: with L2 the published prefix — tail and state checkpoint included — travels as pinned L2 entries and only the unpublished slots go to the snapshot pool; without L2 the whole image goes to the pool. Either way the request resumes where it stopped, and never competes for admission as a newcomer.

Host L2: write-back. With host_write_policy kWriteBack this role writes back as the P role does (§3.1): nothing streams to the Host at a publication — the prompt end at the first decode, each page a decode completes, the finish, a retraction — and what an admission or a restore evicts for the cache leaves by whole segments (an admission of a request alone also block by block, §1.4), copied to the Host in the plan that evicts it. A mixed batch's admissions and decode steps share the one drain after the build. A segment's ACK can also land after a plan that only restores (§4). A retraction image's L2 leg takes the published slots not on the Host already as §2 plans them (whole segments first), and the snapshot pool the rest. The fused role adds three shapes to what the P role sees:

• A decoding request holds the KV pages under its prompt end, so when the prompt ends on a page boundary that checkpoint boundary's segment can leave the Device only after the request finishes.
• A turn that decoded leaves two checkpoint boundaries, its prompt end and its finish; the next turn continues from one of them, and the other, once evicted, is written back like any other cache.
• An eviction in a plan that only decodes is written back on that plan's ordered stream ahead of its forward, unthrottled: the evicted pages are not pinned, the forward reuses them, so the copy cannot wait for a later plan.

4. Suspend and resume

What the scheduler keeps across requests for retraction is one integer (next_retraction_epoch_) and one slot allocator (snapshot_slots_, a SnapshotSlotAllocator handing out blob slots 1..max_retracted_requests); everything else lives on the two FSM states a suspended request moves through and is dropped with them. There is no queue to keep in step with the FSM: a request that finishes or aborts while suspended stops qualifying, and its CacheBlockRefs release what it held.

fsm::Retracted holds no Device pages. It carries the request's token container and cache progress, the RetractionImage — per group an ImageTable (num_blocks, reclaimed_prefix_blocks, available_tokens, and per data slot its slot_index, a pinned ref on the Host block that holds the bytes — an L2 entry with its CacheKey, or a snapshot-pool block with none) — the blob slot, the ResumeShape that names the state it resumes (ResumePrefilling{window, reserve}, ResumePrefillDone{window, reserve} or ResumeDecoding{reserve}), its retraction_epoch and resumes_generation stamp, and image_stores: a StoreWait on every op the image waits for — both legs' own ops and any earlier in-flight store carrying one of its keys. Each wait is a weak reference to the op's in-flight record, which the transfer manager drops at the op's ACK (cache::WriteBackDone, cache::SnapshotDone). The image is landed (ImageLanded()) when every wait has expired; nothing records the ACKs on the request. Until then the request is not a readmission candidate, because a restore started earlier would copy bytes that have not arrived.

Readmission order (rankedReadmissions) is derived, not stored: among this round's candidates whose image landed, victims with generated output first (they resume a generation a client is already reading), then oldest epoch. The flag is Request::HasGeneratedOutput() — token count above the submitted prompt size — rather than "was the victim decoding": a victim taken mid-prefill may still own generated tokens, and its standing survives.

The restore phase (scheduleReadmission, phase 1 of both grammars) tries the ranked readmissions in turn until one restores — one restore per round — scanning at most kMaxRestoreAttemptsPerRound (4) of them, because each failed attempt is an admission-planner pass. So a 60K-token image that does not fit the free Device pages does not hold a 2K image behind it hostage; it does still seal new-prompt admission for the round (any landed image that waited seals, whether or not a later one restored: a newcomer must not take the pages it waits for). A readmission that found no request-pool slot stops the scan without sealing — nothing later would get a slot, and neither would a newcomer.

The restore (scheduleRestore) rebuilds the request on fresh Device pages:

• CacheCoordinator::Restore gives every group a table of identical shape — same num_blocks, same null holes (a sliding group's reclaimed prefix, a State group's empty slot) — with one fresh block per imaged slot in the imaged bucket, then runs the ordinary Admit demand on the rebuilt tables inside the same planner pass: DenseGrowth{0} (StateStep::kKeep for a State group) plus the reserve ReservePrefillDemands derives from the resume shape exactly as for a first chunk — the decode slot when the request resumes decoding or its completed prompt, the rest of the prompt too when it resumes mid-prefill — and, on every resume shape, the escalated admission headroom (Request::AdmissionHeadroom, below): the restore is the admission that re-secures the room the retraction proved too optimistic, whichever state the victim was in. Cache-only blocks are evicted for it as for any admission, and a failed restore leaves nothing allocated. Under the write-back policy the cache victims a restore evicts are written back in the plan that evicts them, as an admission's are (§3.1), ahead of the restore's own copies: the plan's stores run before its loads.
• An L2 slot whose key still has a Device-cached canonical block is claimed instead of copied (the same bytes by construction — the victim's own block, cache-only since it was freed — and protected from the planner's eviction like a prefix hit). Every other slot becomes one row of a single SnapshotRestoreOperation, tagged with its source tier (HostTier::kL2 rows carry their key, kSnapshotPool rows none). Under segment eviction a restore whose claims leave no plan plans again claiming nothing, so every slot is a row (cache-concepts.md, AdmissionPlanner). A copy fills its destination block whole, so — as for a Host hit's load-back — only the reserve pages appended beyond the imaged shape are listed in plan.pages_to_zero. The op also names the blob slot and the request's new request-pool index: the runtime imports the slot-state blob into that row.
• A victim taken before its first decode held copy-in blocks for its prompt end (cache-concepts.md, copy-in), unless the prompt end is dead (§1.2), which the restore decides as the first admission did, from where the victim resumed. They hold no bytes, so the image does not carry them: the retraction releases them with the tables, and the restore reserves them again with the blocks they serve, inside the same planner pass -- for a rebuilt slot of the partial page at the image's frontier in that slot's bucket, for a slot the reserve's growth fills (a victim taken mid-prompt) as a pair with its block. The copy-in fills them whole, so they are not zeroed either.
• The request moves to fsm::Restoring: it holds the rebuilt tables (a ForwardResources bundle, so it occupies capacity like any resident), the image and the op id, is never scheduled, never a victim and never a readmission candidate. Both ends of every row, the blob slot and the new request-pool row stay pinned by the transfer manager until the ACK, so an abort while restoring cannot re-grant a page, slot or row the copy is still writing. A restore takes no token budget and no batch slot; it is a cache op beside the batch, like a remote admission.
• cache::RestoreDone republishes the L2-tier destinations into the Device prefix index (as a prefix load-back does), drops the image — the L2 pins go, the entries stay published and evictable; the pool blocks return — and RestoreDoneEvent moves the request to the state its shape names: a Decoding request decodes next round with no prefill, a PrefillDone one takes its first decode, a Prefilling one schedules its next chunk with replay = 0.

Why a Restoring state at all: the copy is asynchronous and the request must hold its pages while it runs, yet nothing may be scheduled against pages whose bytes are still arriving. Retracted holds no pages by definition, and the schedulable states must stay free of a "but not yet" flag every phase would have to test; a state the phases do not know is the one way to hold pages and be invisible at once.

Finish or abort while suspended drops the state: from Retracted, the pinned Host entries become ordinary evictable entries at once and the pool blocks return when the copies still writing them land (the tickets hold the refs); from Restoring, the rebuilt tables are freed and the restore's ACK only drops its pins — it republishes nothing, because the token descriptors the KV-event feed needs for a publication died with the request. The two non-page resources an image op uses follow the same rule: the blob slot is a shared_ptr<SnapshotSlotIndex> held by the state and by the store (InFlightSnapshotStore) or restore (InFlightSnapshotRestore) ticket that exports into or imports from it, so it returns only when the last of them lets go; the request-pool row a restore imports into is owned by the restore ticket until its ACK (Restoring::resources.req_pool_index is empty meanwhile) and handed to the resumed state by RestoreDoneEvent — or dropped with the ticket when the request is gone. Neither can be re-granted to another request while a copy still writes it. A late ACK for a dropped image is harmless.

KV-event descriptors. With enable_kv_cache_events, every Device publication is a mutation of a boundary that must already have its token descriptor (registerKvEventPrefixPages), and DrainKvEvents drops the descriptor of a boundary with no cached child. A boundary is BlockStored only while every group holds its child -- the State groups included, whose checkpoint exists only at a natural point or the opt-in in-forward checkpoint (§1.2) -- so a hybrid model emits BlockStored for its prompt end and its finish (unless they are dead, §1.2), its junction cuts and the in-forward checkpoint's page boundary, never for another interior chunk end's chain hash; a tail's descriptor carries the rows below T and the residency check counts a tail's blocks from it. A publication registers its newly hashed pages and its tail, and every page below them it registers an entry under (CacheCoordinator::FirstPublishedPage: the deepest page, where a State checkpoint lands when no page is new, and the pages a sliding window behind one of its boundaries reaches back into). Those pages keep their descriptors while the request's own published blocks hold them on the Device, but not once a restore has brought the request back on unpublished copies of them (snapshot-pool rows) and a drain has dropped them. Retraction publishes twice without an admission: retractVictim registers before CacheCompletedBlocks (as the finish-time publication does), and the RestoreDone handler registers the restored request's whole prefix chain, and the tails the image's published slots sit at, before CompleteSnapshotRestore republishes its L2-tier destinations — the victim's pages left the Device when they were granted away, so their descriptors are typically gone by the time the restore lands. No page hash names such a tail (slot 0 of a State group holding the checkpoint of a prompt end inside a page), so the retraction reads its boundary off the descriptor while the victim still holds it and keeps it with the image (Retracted::kv_event_tails).

A readmission that does not fit waits. Its failed restore never triggers retraction (it is never recorded as the capacity blocker): when the restore needs a victim, the two do not fit together, and swapping them — two images and two restores per swap — is pure thrash. The resident request keeps running and its completion frees the space. Unlike escalation-bounded ping-pong this makes the evict-each-other cycle structurally impossible. Nor does a waiting readmission stall anyone else: decodes run regardless, and only new-prompt admission is sealed behind it (a newcomer taking the pages it waits for would starve it) — the remote admission on the D role, the new-prompt tier on the fused role.

Escalating headroom and the exemption — kept, as thrash damping. Both rules predate the snapshot model, when they also kept a recompute-style retraction from losing work without bound. That reason is gone (nothing is recomputed) and is not why they stay. They stay because a retraction is still an expensive interruption — two Host copies, a restore, a client-visible pause — and without them one long request can be imaged and restored on every page boundary for as long as the pool is tight. Being retracted means the previous admission was still too optimistic, so each retraction raises the decode headroom the next admission must secure:

Request::AdmissionHeadroom(safe_steps)
    = min(RemainingNewTokens(), safe_steps * (1 + retraction_count))

with safe_steps = 4096 — note the 1 +: a fresh admission already prepays one window (see schedulePrefillFirstChunk), so for prompts with max_new_tokens <= 4096 the reserve covers the whole generation up front and retraction never touches them, except the empty-round retraction (§2). Capped by the generation budget the request could ever use, so after a couple of retractions it holds enough room to run to completion — at which point ReserveCoversGeneration exempts it from the victim policy and it cannot be retracted again, and is not aborted either (§2): it needs no further page, so it finishes on its own reserve and frees its pages then — except where a sliding-window group's growth outruns that reserve (§1), which the empty-round retraction covers (§2). This is a per-request adaptive backoff: it penalizes only the request whose admission proved over-optimistic, and never makes anyone else wait. The constant is a thrash bound only; no liveness argument rests on it.

The exemption compares the windows the admission secured against the budget that was open at that admission (Request::RemainingNewTokensAtAdmission, stamped by the first chunk's and the restore's scheduling events), never against the current remaining budget. Decode spends the prepaid headroom exactly as fast as it shrinks that budget, so judging the window against today's remainder would count spent headroom as still held: a request that outgrew a partial reserve would look covered the moment its remainder dipped under the window — exactly when it needs a new page — and once every resident request looked covered, retraction would have no victim short of the empty-round retraction (§2), and nothing else could free that page.

There is no retraction without an image. The one case that used to recompute — an L3 prefetch that missed after admission — no longer exists: L3 objects are fetched into Host before admission (§1), so an admission's load-back cannot miss and no forward is ever skipped or retracted for L3.

Weight updates and flushes. Scheduler::RetractedSize() counts the requests suspended with an image (Retracted + Restoring), and every flush (ClearCache, ClearL1Cache, and the CanClearCache probe the replica MIN-reduces before clearing) refuses while it is non-zero: a suspended request continues from its image once restored, so a flush under it — a weight update — would resume old-weight KV under new weights. The pinned Host entries of an L2 leg would refuse the Host clear by themselves, but a snapshot-pool leg (always, with --disable-kvstore) is indexed nowhere and visible through no pin, hence the explicit count. A flush is likewise refused while any image copy is in flight (HasAnyInFlight includes the snapshot ops). The pause drain counts suspended requests as waiting, so a correct runtime never hits either refusal; they are defense in depth. SnapshotPoolFreeBlocks() and HostPoolPinnedBlocks() are the leak checks: with no request suspended both must read empty and zero.

Configuration is explicit on every role: snapshot_allocator.total_pages (1 = the null page alone = no image ever fits, so a capacity block aborts the newest resident instead of imaging one; §2) and max_retracted_requests (0 with no pool, > 0 with one; the runtime's slot-state arena has that many rows plus the null row) are validated together. The P role refuses a snapshot pool. L3 addresses page-cyclic sharded groups through replica-stable object keys (cache concepts). debug_force_retraction_interval (§2) is the only knob that chooses victims outside capacity pressure, and it is refused without a pool. Every diagnostic names the binding field and the server arg behind it (num_snapshot_pages from the pool the runtime resolved -- --retraction-snapshot-host-gb, --retraction-snapshot-ratio, or the derived default of docs/configuration/server.md, "Retraction snapshot pool" -- max_retracted_requests from --retraction-snapshot-max-requests, --debug-force-retraction-interval), because the runtime surfaces the message verbatim at startup. The scheduler takes page counts; how the runtime arrives at them (the Host KVStore-style size-over-ratio rule, the tail-per-request default beside L2 that this model's SingleRequestGroupPages and LcmBlocksNeededFor count) is python/tokenspeed/runtime/cache/l2/sizing.py.

Runtime contract. num_snapshot_pages is explicit (1 without a pool). The runtime executes and counts snapshot and restore ops alongside L2 copies. Query, GET, PUT and cancellation are Host-only variants of CacheOperation. Their StorageDone, StorageTick and StorageDetach events enter the same scheduler head feedback point. Prefetching is a request waiting state; it does not add a second admission or forward path. ExecutionPlan.aborts reports capacity aborts for the runtime to finish toward clients. The runtime and scheduler must use matching contracts; releases follow the sequence in AGENTS.md.

5. Invariants a change must preserve

• Admission never grants pages for tokens beyond the chunk being scheduled, except the decode reserve on the completing chunk (1) and the admission headroom (4) — which full-history groups hold whole and sliding-window groups at most one window lookback of, and only while another request holds pages: the growth recycling cannot fund on a first chunk (1). A request alone prepays nothing, so the prepay never makes a request the single-request bound (1.4) accepts inadmissible on its own. A State group holds two slots and nothing else (1.2). A replayable group's private suffix starts at the replay window, which is inside the forward's input, not beyond it (1.3). A restore grants exactly the imaged shape plus the same reserve a first chunk of its resume shape would (4).
• A replayable group is never matched, published or streamed (1.3); its re-fed rows are forward input that debits the token budget but never advances num_computed_tokens; only a hit's first chunk re-feeds, and no final chunk is shorter than the replay window (ChunkKeepingFinalWindow). Its request-private pages are imaged and restored like any other group's.
• A prefill demand's reserve is decided once per group, by kind, in ReservePrefillDemands (1); no later step rewrites reserve_tokens, and the helper asserts it found none set.
• An incomplete local prefill is not overtaken (1.1). Decodes are never hostage to it: they consume no fresh capacity within their reserve, so they keep running beside a stalled prefill. With prefill_chunk_yield_tokens on the P role, whole prompts may complete in its forward, ahead of its row, within the tokens its chunk gave up. The incomplete prefill is still admitted first, and a resident chunk that fails admission still stops everything behind it.
• Retraction fires only when no prefill progressed and an admission failed (2). The chosen victim must be quiescent — no forward of its own in flight, no PD transfer against its pages (§3.1) — and, in both tiers, is chosen only among the residents the same round has dispatched no work for (an armed debug victim sitting the round out stays offered); an in-flight load-back or an in-flight pinned store defers all retraction; stream-ordered stores and in-flight restores defer nothing. A victim that would serve nobody is not retracted, and a round with no room left for the grant retracts nobody. A victim whose image does not fit is aborted through onImageDoesNotFit and nowhere else; the debug knob aborts only a victim whose image is refused after its publication, and leaves one its probe refuses running with nothing published.
• Freed capacity is granted to the request it was freed for in the same plan build whenever the round's grammar admits the grant, except that the empty-round retraction offers it to the round's decodes first (2); the image copies → page copies (ExecutionPlan::pages_to_copy) → zero → load → forward order on the forward thread's stream is what makes the immediate release safe -- a tail hit's copy source may be evicted and re-granted by a later admission of the same round, as a fresh page or as another copy's destination -- and changing DeviceHandle.execute's ordering breaks it. Within the copies, the arena reads each source byte range before any pair overwrites it (CacheArena.copy_blocks orders waves across all groups by their field byte ranges). This includes a sliding-window block whose LCM parent is re-granted to another group with different packing. Disjoint child ranges remain independent, so parent reuse alone does not form a copy cycle. Every source is read as it was when the plan was built. A copy destination is never zeroed.
• A published block is never written again: a tail's blocks are copied into the hitter's table (or loaded from Host into private pages), an owner that keeps writing past its published prompt end moves to copies of the blocks it published, in the copy-in blocks it holds, and never takes a block another request published into its table (a dead prompt end, 1.2, publishes no tail, and the owner keeps writing its private block), a chunk end cut at a junction publishes no KV block, and a restore copies the imaged tail blocks instead of claiming them (1, cache-concepts.md).
• A request whose admission prepaid its generation never needs a block it did not reserve: the decode slot is reserved on the completing chunk, a State group's rotation swaps a released block for a fresh one, and each copy-in block of the prompt end is reserved with the block it serves -- in full-history groups by the first admission, which prepays the prompt (1, cache-concepts.md). Its admissions and its first decode agree on whether the prompt end is published: a dead one (1.2) holds no copy-in block and its first decode spends none. Without this a pool full of retraction-exempt requests wedges at their first decodes. A sliding-window group's growth can outrun its reserve (1); the empty-round retraction keeps such a pool live (2).
• A store either pins its Device sources until the ack or is stream-ordered; never neither (2). Only the two legs of a retraction image and the Host write-back policy's eviction stores are stream-ordered — their sources are granted away in the same round — and only such ops may be fenced ahead of the plan's page reuse by the runtime. A new store site chooses its guard explicitly (StartPendingStores has no default). Under write-back no store pins its sources.
• What an admission or a restore evicts under write-back is drained in the plan that evicted it (StartEvictionStores, right after the build); no record outlives its plan (by construction: only the build records, and that drain takes every record; ClearDeviceCache, which runs between plans, asserts none is left), and the segments of one plan that depend on each other — a later admission's segment taking pages an earlier one's checkpoint still needs — are merged before the Host places them, so they arrive together or not at all (3.1).
• Every tier transfer pairs blocks of equal residue (bucket): L2 store, L2 prefix load, both image legs and the restore; the transfer manager asserts it when it builds a batch, so under page-cyclic sharding the rank that owns one end owns the other (cache-concepts.md).
• An admission loads Host entries only; L3 objects reach Host through a storage transaction issued before admission (1); its waiting request holds no Device page, no request-pool row and no head of line.
• Only computed tokens are published as a prefix, and exactly those. Request::NumComputedTokens() is the one frontier for prefix hashing and retention on admission and retraction, and the extent an image covers: the scheduled window end while prefilling (an incomplete prefill's whole token count would publish pages never computed), and every token but the last while decoding — feedback ends with the sampled token the next forward computes. It does not subtract the verify width: a decode result lands its accepted tokens, not a fixed number, so any margin is an estimate that lags the real endpoint and delays publication and reclaim behind it. A finish that overtakes a forward still in flight (an overlapped decode) publishes the KV rows below the frontier, which are final, but not the State groups' live slot that forward is writing (PublicationPoint::kFinalKvOnly).
• A Retracted request holds no Device pages; its image is held by that state alone (Host pins and pool refs) and dropped with it. It becomes a readmission candidate only once every store it waits for has landed, at most one restore starts per round (after a bounded scan of the ranked candidates), and a Restoring request is never scheduled, never a victim and never re-probed (4). A readmission that fails admission waits, seals new prompts for the round, and never triggers retraction (4).
• A request whose admission prepaid the generation budget open at that admission is never a victim (2) in a round that builds a forward, restore, remote prefill or remote decode, or waits for a finished request's last forward; with the fresh-admission prepay this bounds such retraction to requests whose max_new_tokens exceeds one safe-step window (or is undeclared). Spending a partial reserve never makes it qualify: the exemption is judged against the budget open at admission, not the current remainder (4). Only the empty-round retraction (2), in a round that builds nothing and waits for no finished request, retracts or aborts an exempt request.

Deferred PD cancellation belongs to the request: it first waits for transfer retirement, then for any outstanding local forward. Repeated cancellation cannot restart a transfer wait after its retirement event has arrived. Destroying the request destroys this state, so reusing a client request ID cannot inherit an old abort.