Imagine you open a chat and ask a short question. The assistant answers in a few sentences and the exchange ends. Now imagine a long conversation that continues for many turns, with the user adding details, correcting earlier statements, and asking the assistant to revise its answer. The underlying model has not been retrained and its weights have not changed, yet the system may need much more memory to continue. That extra memory is not a copy of the model. It is working state created for this particular conversation, and it grows as the conversation grows [1].

This distinction matters because people often picture an AI system as a single fixed object: load the model, ask a question, get an answer. In practice, generation also creates temporary data that depends on the exact prompt, the earlier turns, and the tokens already produced. A short exchange may need very little of this state. A long exchange may need a great deal, especially when several conversations are active at the same time. The cache is therefore best understood as a notebook for the current conversation, not as part of the model itself [1].

The practical question is not only whether the model fits on a device. It is also whether the cache for the current sequence fits alongside the model, the input, and any other active sequences. When it does not fit, the system may fail, slow down, move data elsewhere, or refuse to continue. Understanding that difference helps explain why two requests to the same model can behave very differently: one is short and isolated, while the other is long, detailed, and competing for memory with other work [1].

Scope matters here. This explanation uses a full-attention decoder that retains its processed prefix. Other attention patterns exist, and the later attention section explains why that qualification matters [2]. The vLLM reference used for tensor shapes and blocks explicitly labels itself a historical design document; it is useful for understanding the mechanism, but it does not describe the current vLLM implementation [3]. The numerical example below is our own hypothetical calculation, not a product specification or a measured result.

The central inference problem is that model parameters are static during generation, whereas the key-value cache is sequence-dependent state. For a fixed model, the parameter tensors occupy a size determined by architecture and numerical precision. The cache, by contrast, depends on the number of tokens already processed, the number of layers, the number of key-value heads, the head dimension, the batch or sequence count, and the storage precision. A model can therefore fit comfortably in memory while a long sequence still exhausts the remaining budget because its cache has grown beyond the available space [1].

This is why memory planning for autoregressive serving must separate at least two quantities. The first is the resident model footprint, which is present whether or not a particular request is long. The second is the per-sequence cache footprint, which starts small for a short prompt and increases as tokens are appended. In multi-sequence serving, the second quantity is multiplied across concurrent requests, so aggregate cache demand can dominate even when each individual conversation appears modest [1].

The cache also has a different lifetime from the weights. Weights are loaded once and reused across requests. Cache entries are created for a specific sequence, extended as that sequence advances, and discarded or reused when the sequence ends or is rolled back. That transient lifetime is the reason a longer conversation can require more memory without any change to the model file, the tokenizer, or the trained parameters [1].

The accounting below assumes homogeneous full-attention decoder layers and a retained prefix. It separates logical KV payload from allocation overhead and total device memory. Source [3] explicitly warns that its kernel explanation is historical, so its tensor shapes support the conceptual derivation rather than a claim about current vLLM internals. Source [2] supplies the architectural caveat about alternative attention patterns. Check a chosen model and software release before translating these concepts into an implementation.

Autoregressive generation means the model produces text one token at a time. A token may be a word, part of a word, punctuation, or another small unit of text. After producing one token, the model uses it, together with everything before it, to predict the next token. Then it repeats the process. This step-by-step method is simple to describe, but it creates a repeated question at every step: how should the new token relate to all the earlier tokens? [1]

Without any stored state, the model would have to recompute its internal representations of the earlier tokens again and again. Each new prediction depends on the previous tokens, so the same earlier material would be processed repeatedly as the answer grows longer [1]. That repeated work would waste time. It would also make long responses increasingly expensive, because the amount of earlier material to reprocess would keep increasing with every new token.

A cache offers a way to avoid some of that repetition. Instead of rebuilding everything from scratch for each token, the system can keep certain intermediate results from earlier steps and reuse them. The idea is similar to keeping notes while reading a long document. You do not reread every page from the beginning each time you want to answer a question about page three. You keep useful notes and return to them when needed. The KV cache plays an analogous role inside the model [1].

In autoregressive decoding, the model computes a distribution over the next token conditioned on the prefix, then samples or selects a token and appends it to the sequence. At step t, the conditioning set contains t tokens, and at step t+1 it contains t+1 tokens. A naive implementation would recompute key and value projections for the entire prefix at every step, leading to redundant work that grows with sequence length. The KV cache stores the per-token key and value tensors produced by earlier steps so that only the new token needs to be projected and appended [1].

Formally, for each layer l and each token position i, the cache retains a key vector k_{l,i} and a value vector v_{l,i}. During decoding of position t, the query q_{l,t} attends over the stored keys k_{l,1..t} and retrieves a weighted combination of the stored values v_{l,1..t}. The cache therefore converts a repeated full-prefix computation into an incremental update: one new key-value pair per layer per token, plus attention over the accumulated history [1].

This incremental structure also explains why cache size is a function of position rather than a fixed property of the model. The first generated token adds a small amount of state. The hundredth token adds another increment of the same size. The ten-thousandth token adds yet another. The per-token increment may be constant for a given model configuration, but the total after t tokens is the sum of those increments, so it grows linearly with t for a standard full-attention cache [1].

Inside the model, each token is represented by numbers. Processing it produces several useful representations. A query can be thought of as the question the current position asks: which available information is relevant now? A key supplies numbers against which that query is compared. A value supplies numbers that contribute to the attention output according to the weight assigned to that position. These labels are an analogy for numerical operations, not a claim that the cache understands a question or stores a readable answer [3].

Queries and keys determine attention scores; the resulting attention weights determine how values are combined. It would be misleading to say that the cache simply stores the attention scores. A later token brings a different query, so its scores against earlier keys must be worked out for that step. The reusable part is the earlier keys and values themselves. The historical vLLM explanation separates its query-key calculation from the later value calculation, making this distinction visible [3].

A useful analogy is a library index. The key is like a label that helps the system find a relevant item quickly. The value is like the contents that are actually brought back once the item is found. The cache keeps both the labels and the contents for earlier tokens. That is why it is called a key-value cache: it stores the keys used for matching and the values used for retrieval, organized by token position and by layer [1].

At an attention layer, the current hidden representation supplies a query, key and value representation. In the single-query decoding case illustrated by the historical vLLM design, the query is compared with cached keys; attention weights then combine cached values into an output. Cached K and V tensors are therefore distinct from the query-key scores calculated for the current query. Reusing K/V projections removes redundant prefix work, while attention for the new query still needs to be evaluated [1][3].

The logical cache is indexed by layer, KV head and retained token position. The historical kernel signature gives queries a num_heads dimension and K/V caches a separate num_kv_heads dimension. That is a useful warning when estimating storage: obtain the model's actual stored KV-head count rather than automatically substituting its query-head count. The physical key and value layouts in the example differ, but both contain token, head and head-size dimensions. The simplified calculation below counts their elements, not a particular release's byte offsets [3].

It is important not to confuse these vectors with the original text. The cache does not store words as readable strings, and it does not store a summary of facts that the model has learned during training. It stores numerical intermediate states derived from the tokens in the current sequence. Those states are useful for continuing the same generation, but they are not a general knowledge base and they do not replace the model's trained parameters [1].

The cache grows because each new token needs its own key and value entries in every layer. A short prompt creates a small set of entries. As the model generates more tokens, it adds more entries. The process is additive: token one adds a set, token two adds another set of the same size, token three adds another, and so on. After many tokens, the total is much larger than it was at the beginning, even though each individual addition was modest [1].

This growth is easy to underestimate because the per-token addition feels small. One token's worth of keys and values may seem negligible. But a long conversation contains thousands of tokens, and the model may have many layers. The total is the per-token size multiplied by the number of tokens and by the number of layers. That multiplication is what turns a small increment into a large memory requirement over a long sequence [1].

The same logic applies to the prompt itself. A long initial prompt already creates a large cache before generation begins. Generated tokens then extend it further. In a chat, the history of earlier turns is part of that growing sequence, so a conversation that has accumulated many turns carries a larger cache than a fresh conversation with a single short question. The cache therefore records the length of the current sequence in memory, not just its content [1].

There is an important boundary to this rule. Hugging Face's attention guide describes Longformer using small local windows, together with selected tokens that have global access. It also explains that stacking local-attention layers can let information travel beyond one small window [2]. This shows that models can use different patterns of communication between positions. It does not mean that deleting old entries from an ordinary full-attention model is an equivalent memory-saving switch. An attention pattern belongs to the model's design; a cache's allocation policy is a separate question. Check the model's actual rules before assuming every layer must keep the same amount of history.

For a deliberately simplified decoder with L identical full-attention layers, H_kv stored KV heads per layer, head dimension d_h, t retained tokens, and b bytes per stored element, the logical K-plus-V payload is 2 * L * H_kv * d_h * t * b bytes. This is our dimensional derivation from two stored tensors and the token/head dimensions illustrated in the historical kernel, together with the cache guide's description of growing K/V storage [1][3]. It excludes block padding, allocator overhead, auxiliary metadata, temporary buffers and model weights. Under these assumptions, doubling t doubles the payload; it does not necessarily double total GPU allocation.

Do not confuse that linear KV-payload estimate with the size of a sequence-wide attention matrix. The attention guide describes full attention using a square matrix and discusses sparse alternatives: Reformer selects query-key candidates through hashing, while Longformer combines local windows with selected globally connected tokens [2]. A square t-by-t matrix contains t squared entries; the cached K/V tensors instead retain vectors for token positions. This distinction is an accounting inference, not a claim that incremental decoding materializes a whole square matrix at every step. Nor does the attention guide establish a universal bounded KV-cache formula for sparse models. Retention rules, global connections and layer types require model-specific documentation.

For several independent sequences of similar length and without shared prefix storage, the same logical payload estimate is multiplied by the sequence count B. For unequal lengths, replace B times t with the sum of retained token counts. This is an accounting assumption for the worked example, not an assertion that every serving implementation duplicates all prefixes or allocates exactly that many bytes. The useful planning point is to budget runtime state across requests as well as the fixed model footprint [1][3].

Model weights are the trained numbers that define how the model behaves. They are created during training and then loaded for inference. They do not change when a user asks a question, and they are shared by every conversation that uses the same model. If you load a model once, its weights stay in memory while many different prompts are processed. The weights are the model's long-term, learned behavior [1].

Cache state is different. It is created from a particular token sequence and is useful while that sequence, or a compatible shared prefix, is being continued. Two different prompts generally require different state even when they use the same model weights. An application can release a cache it no longer needs or deliberately retain a reusable prefix. The cache is working state for generation, while the weights are the trained foundation underneath it; a conversation ending does not itself prove that an application has immediately freed every cached allocation [1].

This difference explains a common surprise. A model may load successfully and answer a short prompt, yet run out of memory on a long one. The weights fit, but there is not enough remaining room for the longer request and its working state. Releasing an unneeded cache can make room for another request while the model remains loaded [1]. Treat that as an application decision: clearing the visible chat window is not evidence about what a server has retained or how its memory manager behaves.

Weights are parameters learned during training and held fixed during inference. Their memory footprint is determined by the parameter count and the storage dtype, and it is independent of sequence length. Cache tensors, by contrast, are activations retained across decoding steps. They are not parameters, they receive no gradient updates during inference, and their size depends on the runtime sequence length and batch composition [1].

The distinction has operational consequences. Weight memory can be estimated once when a model is loaded. Cache memory must be estimated per request and revised as generation proceeds. A serving system that budgets only for weights will systematically underestimate peak memory for long contexts. Conversely, a system that reserves cache space for the maximum possible sequence length may waste memory when most requests are short. Effective planning therefore treats weights as a fixed cost and cache as a variable cost [1].

For ordinary inference, parameters can be reused while requests maintain their own compatible runtime state. Arbitrary cache reuse across unrelated prefixes would substitute states derived from different input. The cache guide illustrates a narrower case: a common prompt is prefetched, its cache is deep-copied, and each continuation receives its own copy [1]. That example supports compatible-prefix reuse. It does not establish a universal implementation of zero-copy sharing, a security guarantee, or a rule that all caches disappear as soon as an application marks a conversation finished.

The following calculation is entirely hypothetical. It uses invented numbers to show how the pieces multiply together. It is not a measurement of any real model, and it should not be used to predict the speed or memory use of a specific product. The purpose is only to make the growth pattern concrete. Assume, purely for illustration, a model with 24 layers, 8 key-value heads per layer, a head dimension of 128, and 2 bytes per stored number. Assume also a sequence length of 1,000 tokens and a single active sequence [1][3].

Using the formula from the previous section, the cache size is 2 times 24 times 8 times 128 times 1,000 times 2 bytes. That equals 98,304,000 bytes, or about 94 mebibytes in this invented example. The arithmetic is straightforward: each token adds the same amount, and 1,000 tokens add that amount 1,000 times. If the sequence grew to 4,000 tokens under the same invented assumptions, the cache would be about four times larger, because the token count is four times larger [1][3].

Now change only the number of concurrent sequences. If 10 such hypothetical sequences were active at once, each with 1,000 tokens, the aggregate cache would be about 10 times the single-sequence figure. The model weights would not change at all. This illustrates why a serving system can run out of memory from many medium-length conversations even when no single conversation is extremely long. The hypothetical numbers are illustrative, but the multiplication pattern is the real lesson [1][3].

The hypothetical example can be written as a dimensional analysis. Let L = 24 layers, H_kv = 8 heads, d_h = 128 elements per head, t = 1,000 tokens, and b = 2 bytes per element. The per-token, per-layer key-plus-value storage is 2 * H_kv * d_h * b = 2 * 8 * 128 * 2 = 4,096 bytes. Across L layers, one token requires 98,304 bytes. Across t tokens, the total is 98,304 * 1,000 = 98,304,000 bytes, approximately 93.75 MiB. These values are invented for exposition and carry no empirical claim about any deployed model [1][3].

In this invented homogeneous configuration, the result scales linearly with each input factor. Increasing the retained length or adding independent equal-length sequences multiplies the payload. Reducing the actual number of stored KV heads would reduce its coefficient. Treat lower-precision arithmetic carefully: reducing bytes per element reduces the ideal payload, but a real quantized cache can also have metadata and differently stored portions. The cache guide describes quantization as a memory-saving strategy; it does not make this simplified formula a complete allocator or device-memory estimate [1][3].

No throughput conclusion follows from this arithmetic. The example estimates storage only. It does not measure latency, bandwidth, or the cost of moving cache data between devices. A smaller cache does not automatically mean a faster system, and a larger cache does not automatically mean a slower one. Storage size and generation speed are related through memory capacity and data movement, but the relationship depends on hardware, implementation, and workload [1][3].

The default DynamicCache described in Hugging Face Transformers grows dynamically as generation progresses. It starts small and adds space as new tokens arrive. This is flexible because the system does not need to know in advance how long the response will be. A short answer uses little space, and a long answer uses more. The trade-off is that the size is not known ahead of time, which can make some optimizations harder [1].

A fixed-size cache takes the opposite approach. It reserves a specific maximum amount of space up front and generates within that limit. Knowing the size in advance can make the computation more predictable and can help certain compiler optimizations. The trade-off is waste: if the actual sequence is much shorter than the reserved maximum, the unused portion still occupies memory, and tokens beyond the intended length may need to be masked so they do not affect attention [1].

Neither choice is universally better. Dynamic growth is natural when sequence lengths vary widely, because it avoids reserving a large buffer for every request. Fixed allocation can be attractive when many sequences have similar lengths, because the reserved size is likely to be used efficiently. The right choice depends on the workload, the hardware, and whether the system values flexibility or predictability more highly [1].

A dynamic cache appends new key-value tensors as t increases, so its allocated size tracks the current sequence length. This minimizes waste for short sequences but produces a changing tensor shape, which can interfere with compilation strategies that assume a static graph. A static cache pre-allocates tensors for a maximum length T_max, allowing the decoding loop to operate on fixed shapes. Positions beyond the current length are masked, and the unused capacity is wasted for shorter sequences [1].

The waste in a static cache is proportional to T_max minus the actual length, multiplied by the per-token coefficient. If T_max is chosen for the longest expected sequence but most sequences are short, aggregate waste can be substantial. If sequence lengths are tightly clustered near T_max, the waste is small and the benefits of a stable shape may dominate. The choice is therefore a workload-dependent trade-off between allocation efficiency and shape stability [1].

The cache guide also warns that a fixed-size cache can waste attention computation on masked capacity. Stable tensor shapes can help compilation, but those potential benefits do not erase the cost of an oversized maximum. A workload mixing one very long request with many short ones may therefore fit this choice differently from a workload with similar-length requests. These are reasons to measure the actual length distribution and compiled implementation, not grounds for declaring either allocation strategy universally faster [1].

Paging organizes cache data into blocks. The vLLM document in our references illustrates a historical design in which each key or value block holds a fixed number of token positions for a head. It explicitly warns that it no longer describes the code used by vLLM today [3]. Use it as a clear model of the idea: a long sequence can be represented by smaller storage units rather than understood only as one uninterrupted slab. The source's cache blocks are also different from GPU thread blocks, which organize executing threads.

A block has capacity, and its useful contents need not fill that capacity perfectly. For a simple invented packing example, suppose a block can hold 16 token positions and one sequence needs 17. If that sequence alone occupies its blocks, it needs two blocks, leaving some capacity unused in the second. That is an illustration of rounding up to whole blocks, not a measurement of a serving system. The reference defines a fixed token count per block; the packing arithmetic follows from that definition [3].

Blocks do not make the retained keys and values disappear. The computation still needs to locate the right block, head and token offset. The historical document shows those pieces in the address calculation for reading a key [3]. Organizing data this way changes its layout and allocation granularity. How a current server allocates, shares, frees or schedules those blocks depends on its implementation. This explanation promises no particular speed-up or memory-saving percentage.

Source [3] is explicitly historical. Its example represents K and V as separate arrays with num_blocks, num_kv_heads, head-size and block-size dimensions; BLOCK_SIZE denotes token positions per cache block. The key layout subdivides the head dimension for access, while the value layout arranges those dimensions differently. These layouts illustrate why a logical per-token payload and a physical allocation are different descriptions. They should not be copied as a specification of current vLLM internals.

For a simple hypothetical exclusive allocation with block capacity S and t retained positions, a sequence needs ceil(t/S) blocks if it cannot use partial blocks belonging to another sequence. The corresponding capacity is ceil(t/S) times S token slots, leaving between zero and S minus one unused slots in the final block. That is our arithmetic inference from fixed-capacity blocks, not a result measured or guaranteed by the reference [3]. Shared prefixes, pooling, layer-specific layouts and allocator metadata would require additional assumptions.

The historical key-address expression combines a physical block number, a KV-head index and an offset inside the block. This supports the conceptual distinction between token position and physical location: an attention kernel must access the correct stored vectors even when data is organized in blocks [3]. Paging describes storage organization, whereas the attention pattern determines which positions may interact. The local and global connections described in the attention guide are therefore a different design issue [2]. Neither reference alone establishes today's scheduler, sharing policy or performance for a selected deployment.

Offloading changes where cache data is kept. The Hugging Face cache guide describes keeping most layers' caches in CPU memory, bringing the current layer's working cache to the GPU and prefetching the next layer. The processed cache is moved back as computation advances. This can reduce the cache's GPU-memory demand, but transfers still take work, and the guide warns that generation throughput can decrease depending on the model and generation settings [1]. It is a capacity trade-off to assess, not free extra GPU memory.

Quantization changes how cache numbers are represented. The guide describes lower-precision cache storage as another way to reduce memory requirements. It also warns that cache quantization can harm latency when context is short and there is already enough GPU memory [1]. Therefore, saving bytes does not by itself prove that a particular conversation will finish faster. The retrieved guide provides no quality measurement for our hypothetical model, so no numerical accuracy or quality promise follows from this explanation.

Keep the choices separate when comparing systems. Paging concerns block layout, offloading concerns device placement, and quantization concerns numerical representation. That distinction does not mean every library supports every combination. The guide explicitly describes offloading for its DynamicCache and StaticCache classes; verify the chosen model, cache class and software version before assuming that another combination works [1]. For an ordinary full-prefix cache, retaining more token positions still adds logical entries even if their placement or representation changes.

The offloading mechanism in source [1] follows the layer-by-layer forward computation: most layer caches reside in host memory, the active layer's cache is used on device, the next is prefetched, and processed state is sent back. Do not turn that description into a guaranteed peak allocation of exactly one layer's payload. Prefetching, transfers and other working buffers still need consideration. The guide directly notes a possible throughput penalty whose size depends on the model and choices such as context length and beam count; benchmark the intended configuration instead of inferring latency from the storage formula.

Quantization reduces the precision used to represent KV state. In an idealized element-count calculation, a smaller b reduces the payload coefficient, but representation metadata and implementation-specific retained buffers can prevent total allocation from scaling by the same exact ratio. The guide's direct warning is that quantized caching may worsen latency for short contexts with ample device memory [1]. It does not supply an accuracy experiment for the invented dimensions used here. Storage accounting, numerical representation and observed answer quality are distinct questions.

Allocation, placement and representation are separate dimensions of a cache design, not a promise of interchangeable features. Source [1] documents offloading with DynamicCache and StaticCache and describes quantized caching separately. Source [3] illustrates a historical paged layout. Those facts do not establish that an arbitrary paged kernel, quantizer and offloading mechanism compose correctly. Confirm supported combinations in the relevant release, include extra memory buffers in the budget, and test the resulting workload. The simple linear full-prefix estimate remains a conceptual payload model, not a universal peak-device-memory prediction.

Several edge cases deserve attention. A very long prompt can create substantial cache state before the first generated answer token, and multiple active sequences add to demand. Generation choices such as beam count also affect memory and offloading throughput; the cache guide includes a multi-beam example that retries after an out-of-memory error [1]. Do not assume a single short request represents multi-user operation. Conversely, do not assume every candidate duplicates every prefix: that depends on how the implementation manages shared or copied state.

Another edge case is the end of a sequence. When a conversation ends, its cache should be released so that memory can be reused. If caches are retained longer than necessary, memory can fill with stale state from finished sequences. Conversely, if a system needs to resume a conversation, it may need to retain or reconstruct the cache. The decision to keep, discard, or rebuild cache state is part of managing the overall memory budget [1].

A practical checklist can help. First, estimate the cache size for the longest expected sequence, not just the average. Second, account for the number of concurrent sequences. Third, decide whether dynamic growth, fixed allocation, paging, offloading, or quantization fits the workload. Fourth, test with long prompts and multiple simultaneous requests, not only with short examples. Finally, monitor memory use during generation, because the cache grows over time and peak usage may occur near the end of a long response [1].

Out-of-memory can arise while processing a long prompt or later as a full-prefix cache grows during decoding. A request that starts within budget may therefore need more state before it completes. Planning should consider expected maximum retained lengths, independent concurrent sequences, and other resident buffers. The cache guide demonstrates retrying a failed generation with offloading [1]; that is one documented response, not proof that every possible failure can be recovered or that the fallback preserves the original performance characteristics.

Additional edge cases include variable-length batches, where sequences in the same batch reach different lengths at different times, and iterative chat settings, where the cache must be extended across turns or rolled back when a candidate continuation is rejected. Prefix sharing introduces another consideration: shared prefixes can be computed once, but the shared cache must be copied or referenced correctly for each sequence that uses it, and any subsequent divergence must not corrupt the shared state [1].

A disciplined evaluation checklist includes the following steps. Estimate per-sequence cache size from the model configuration and maximum sequence length. Multiply by the expected concurrency to obtain aggregate demand. Compare that demand with the memory remaining after weights and other buffers. Choose an allocation strategy that matches the length distribution. Validate behavior with long prompts, concurrent sequences, and the intended generation mode. Monitor peak memory throughout generation rather than only at startup. Document the assumptions, because cache behavior depends on model version, library version, and configuration [1].