November 8, 2025 · 7 min read
KV cache as a first-class distributed resource
Treat the KV cache like a database — paged, evictable, sharable across agents — and the rest of the runtime falls into place.
Every shipping LLM serving stack treats the KV cache as a private, per-process side effect. That assumption made sense when the deployment unit was a single completion. It does not survive contact with multi-step agents.
The shift
When an agent runs for ten minutes and forks three sub-agents, each of them needs the same prompt prefix. Re-prefilling that prefix is the single most expensive thing your runtime does. Sharing the cache across processes is no longer an optimization — it is the difference between a viable agent and a bankrupt one.
What "first-class" means
We borrow the database analogy deliberately. A first-class KV cache:
- has an addressable identity (
page_03f1) - has a lifecycle (allocate, hot, warm, cold, evict)
- has policies (LRU + prefix-share)
- has metrics (hit rate, eviction rate)
- has a wire protocol so other processes can attach
In our field guide we map this design space to L3 of the runtime taxonomy. In practice it is the layer with the largest perf gap between prototypes and production.
What we're working on
A reference implementation that lifts the KV cache out of the inference process and into a paged, network-attached resource — addressable from any process in the cluster, with capability-based access control so two tenants can share prefixes without sharing secrets.