A UPPR deep dive into the part of LLM inference that quietly decides how much context you can afford, and how many users you can serve at once.
The problem the KV cache solves
Every time a language model generates a token, it needs to look back at every token that came before it. That look back happens through attention, and attention is expensive to compute. Without any shortcut, generating token number 5,000 would mean redoing that lookback, in full, over all 4,999 previous tokens again, from scratch, for every single new token you generate.
The KV cache is the shortcut that avoids that. To see exactly what it's shortcutting, it helps to open up what attention itself is doing, one level deeper than "the model looks back at previous tokens."
A quick primer on attention
For every token, the model creates three different vectors out of it, using three separate learned transformations of that token's representation:
Query (Q). A representation of what this token is looking for. Think of it as the question this token is asking the rest of the sequence.
Key (K). A representation of what this token has to offer, matched against other tokens' queries. Think of it as a label this token holds up to say "here's what I'm about."
Value (V). The actual content this token contributes, once it's been decided that it's relevant.
Attention works by comparing the current token's query against every other token's key. That comparison is a dot product, a similarity score, and the token whose key lines up best with the query gets treated as most relevant. Those similarity scores get turned into weights that all add up to 1 (a softmax), and the model then builds its output for the current token as a weighted mix of everyone's values, weighted exactly by how relevant their keys turned out to be.
So a query asks a question, keys are what get matched against that question, and values are the actual information that flows through once a match is found. That's also exactly why Q doesn't need caching, but K and V do. Every new token brings its own fresh query, it's only ever comparing itself against the past. But its key and value, once computed, are exactly what every future token will need to compare against and pull information from. Store those once, and every future token can reuse them instead of the whole sequence recomputing keys and values for tokens that already happened.
That's the entire mechanism the KV cache is built around: keep what future tokens will need to look back at, and only ever compute it once per token, no matter how many more tokens get generated afterward. The first time the model processes a token, it computes that token's key and value. Instead of throwing those away, it stores them in memory, and every later token just reuses them instead of recomputing them. Generation stays fast as the conversation grows, instead of getting slower and slower.
That speed comes at a direct cost. Every token you keep in the conversation has its own K and V stored in GPU memory, for every layer of the model, for as long as that conversation stays open. Long context and many simultaneous users both mean more of these entries sitting in memory at once. Understanding how big that gets is the difference between a deployment that comfortably serves a team, and one that runs out of memory the moment someone pastes in a long document.
What actually gets stored
For every token, every layer, and every attention head, the model stores one key vector and one value vector. The size of the whole cache comes down to a straightforward multiplication:
KV cache size = 2 × layers × heads × head dimension × tokens × bytes per number
The leading 2 is there because you're storing both a key and a value for everything. Everything else in that formula is just more tokens, more layers, or more heads all multiplying the total.
A worked example
Take a mid sized model: 32 layers, 8 KV heads, a head dimension of 128, running at 16 bit precision (2 bytes per number), for a single 8,192 token conversation.
KV cache size = 2 × 32 × 8 × 128 × 8,192 × 2 bytes
Working through it: 2 × 32 = 64. 64 × 8 = 512. 512 × 128 = 65,536. 65,536 × 8,192 = 536,870,912. Times 2 bytes = 1,073,741,824 bytes, right around 1GB, for one 8k token conversation.
Now stretch that same model out to a 128k token context, sixteen times longer, and the cache scales linearly with it: roughly 16GB, for one single conversation. Double the number of people using it at once, and you double it again. This is why context length and concurrent users are the two levers that most directly decide whether a deployment fits in the GPU memory you have, often well before the model weights themselves become the limiting factor.
Grouped Query Attention: cutting the head count that matters
The head count in that formula is usually the easiest lever to pull, and modern models already pull it. Older architectures gave every attention head its own K and V, Multi Head Attention. Most current models instead use Grouped Query Attention, GQA, where several query heads share one smaller set of K and V heads.
Going from, say, 32 heads down to 8 shared KV heads cuts the cache by 4x, for a small and usually well worth it trade in quality. Some models push further into Multi Query Attention, MQA, where every query head shares a single KV head. That's why the head count in a memory calculation is often the KV head count, not the total attention head count, and why it's worth checking which one a model actually uses before estimating its footprint.
Quantizing the cache itself
The same idea that shrinks model weights applies just as well to the KV cache. Instead of storing each key and value number at 16 bit, you can store them at 8 bit, FP8, cutting that whole memory formula in half again, on top of whatever GQA already saved.
This is a different quantization target than the model weights. Weight quantization shrinks how big the model is to load. KV cache quantization shrinks how much room is left over for context and concurrent conversations once the model is already loaded. Both matter, but they solve different problems, and a deployment can quantize one, the other, or both.
Why this shows up as a GPU utilization problem
GPU memory is one shared pool, split between three things: the model weights sitting there permanently, the KV cache growing and shrinking as conversations come and go, and a smaller working memory for the computation itself. The weights are fixed once loaded. The KV cache is not, and it's the part that determines how much is actually left over for real work.
This is exactly why inference servers ask you to set a GPU memory utilization limit and a maximum number of concurrent sequences up front, rather than letting the cache grow freely. Push context length or concurrent users too far past what the leftover memory allows, and requests start failing or getting queued, not because the model is too big, but because there's no room left for its memory of the conversation.
On our own DGX Spark setup running Qwen3.6-35B-A3B, this trade off is explicit in the launch configuration. The KV cache is kept in FP8 rather than 16 bit, GPU memory utilization is capped at 65% to leave deliberate headroom, and the model supports up to 262,144 tokens of context, but how much of that is actually usable at once depends entirely on how many of those tokens, across how many concurrent conversations, the leftover memory can hold. Peak VRAM on that setup sits at 79.4GB out of 128GB, most of that gap is exactly the room being reserved for the KV cache to grow into.
The takeaway for anyone sizing a deployment
Model size answers the question of whether a model fits on your hardware at all. The KV cache answers a different question: once it fits, how much context and how many simultaneous users can it actually handle. Two deployments running the identical model can have very different practical capacity, purely based on KV cache settings, GQA versus full attention, and whether the cache itself is quantized.
Getting this right is less about the model card and more about the serving configuration around it, which is exactly the kind of tuning that separates a demo that works once from an infrastructure that holds up under real, concurrent, long context use.
Part of UPPR's ongoing series on local AI infrastructure and the DGX Spark. Questions about sizing an LLM deployment for your organization? Get in touch.