大语言模型内存为何昂贵及如何优化

随叫随到不应该让人感觉像是不断地灭火。Datadog 提供的这份指南分解了高效 SRE 团队如何减少警报疲劳、优化事件响应，并设计不会让工程师精疲力尽的轮班。你将学习如何：

通过将信号与真实用户影响关联来减少警报噪音

通过明确角色和更智能的升级路径提高响应效率

将事件转化为改进系统可靠性的反馈循环

获取指南：https://go.bytebytego.com/Datadog_080426

为什么即使模型和硬件保持不变，发送 10 万词的提示给模型的成本也比发送短提示高得多？

答案的关键部分在于一块称为 KV 缓存的工作内存。

这种内存是在模型生成响应时积累的。它与存储在模型权重中的知识是分开的，并且保存了为输入的每个 token 计算的键和值向量。缓存会随着每个 token 的增加而增长，在长上下文中，它可能占用 GPU 上的大量空间。

例如，对于一个 700 亿参数的模型，在 128,000 个 token 的上下文中，这大约需要 40 GB 的 GPU 内存，这是一个随着每增加一个用户而增长的严重内存负担。

上图提出了一个显而易见的问题，即为什么会存在缓存以及它为什么会以这种方式增长。在本文中，我们将学习 LLM 如何使用内存，它是如何变得昂贵的，以及如何解决这个问题。

免责声明：本文基于来自各种来源的公开资料。参考文献见文末。如果你发现任何不准确之处，请评论指出。

让我们从模型生成一个 token 的工作开始。

为了选择下一个词，它会执行一次注意力步骤，其中最新的 token 会与此前的每个 token 进行比较。这个比较使用两个向量，即键和值，它们只是模型在每一层为该 token 计算的数值总结。如果模型在每一步都为每个先前的 token 重新构建键和值，那么随着输入的增长，每个 token 的工作量就会增加。这种重复是完全浪费的，因为一旦处理了一个 token，这些键和值就保持不变。

KV缓存通过在首次计算时存储这些键和值向量来消除浪费。下一步，模型只计算新令牌的键和值，其余直接从缓存读取。

总体来说，这是一个很好的解决方案。然而，缓存解决了速度问题，同时又制造了新的问题。缓存现在必须在每一步都被读取，这才是成本上升的真正原因。

这里需要理解的一个细节是，缓存存储的是向量，而不是原始文本。它还解释了模型本身有余裕空间时，显得显得困惑的内存错误。

代币生成分为两个阶段，它们以不同方式对硬件施加压力：

第一阶段是预填充，模型一次性读取整个输入。它并行处理所有输入令牌，并在一次内将密钥和值向量构建到缓存中。预填充让GPU的数学单位保持忙碌，所以我们称之为计算限制，意味着芯片完成算术的速度限制。

第二阶段是解码，模型一次输出一个令牌。每个新令牌对整个缓存执行注意步，这意味着模型会在发出下一个令牌之前，先从GPU内存读取所有存储的键和值。它对每个令牌都重复该读。这里的限制在于缓存从内存移动到计算单元的速度，因此我们称解码为内存限制。

长上下文生成的成本更多是对每个代币进行扫除，而不是持有缓存。

更大的缓存意味着每个令牌通过内存总线的数据更多，这直接表现为生成速度更慢且成本更高。这也解释了为什么即使请求在内存中舒适地存在，运行速度也会很慢。

如果成本是追踪每一步读取缓存的量，那么缓存的大小就是接下来要明确理解的。

缓存大小是几个数字的乘积：

二的因子涵盖了密钥和数值。

图层之所以重要，是因为每一层都保留自己的缓存。

键值头设置每层存储多少集。

头维度是每个这些向量的大小。

每个数字的字节数是指存储一个值所占的空间。

Tokens 是上下文长度，每个条目计算一次。

批量大小是一次处理的请求数量。

总之，缓存大小等于 2 乘以层数乘以键值头数乘以头维度乘以每个数字的字节数乘以 tokens 再乘以批量大小。

这里应注意的两件事如下：

缓存随着 token 数量线性增长，因此上下文加倍会使缓存加倍。

它随着批量大小的增加而同样增长，因此同时为更多用户服务，其规模同样增长迅速。

作为一个大致的例子，Llama 3 70B 模型有 80 层，8 个键值头，头维度为 128，并且每个数字存储为 2 字节。在单次请求的上下文为 128,000 tokens 时，这些数字大约乘出来为 40 GB，这就是为什么单个长请求就可以填满一个 80 GB 显卡的大部分空间。

现在让我们看看帮助大型语言模型管理内存的优化技术。每种技术都试图针对特定的数值进行优化。

前两种技术改变了注意力机制本身的构建方式，这意味着模型在训练时就已固定。两者都缩小了每个 token 在缓存中占用的空间。

分组查询注意力针对键值头数量。在标准注意力层中，每个查询头都有自己的键和值头，因此有 64 个查询头的模型会存储 64 套键和值。分组查询注意力允许多个查询头共享一个键值头，这大幅减少了存储集合的数量。例如，Llama 2 和 3 70B 以及 Mistral 7B 共享最多 8 个键值头，与完整多头注意力相比可将缓存大约减少八倍。这也是为什么最近的 70B 模型可以拥有比较老的 7B 模型更小的缓存。

有一种更激进的版本称为多查询注意力，每个查询头共享一个键值头。它在所有头共享方案中节省最多的内存。然而，推到那一步时，质量往往下降，训练变得不稳定，所以大多数设置选择分组共享作为更好的折中方案。

第二种方案保持头的数量不变，而是压缩每个头存储的内容。

多头潜在注意力在DeepSeek模型中引入，它将键和值投影为更小的潜在表示，然后缓存，读取后再展开。节省的费用很大。DeepSeek-V3 每个令牌大约有 70 千字节，而类似的分组查询模型则在 192 到 328 千字节之间。成本在于服务，因为压缩会在每次读取中增加工作量，并且与一些标准注意力实现配合得很尴尬，所以当模型和上下文足够大，缓存流量占主导地位时，压缩通常会带来回报。

如前所述，头脑共享和潜在注意力都需要对架构的控制，因此在选择或训练模型时它们很有帮助。接下来的攻击基于我们已有的模型。

量化是在每个数字的字节之后。

键和值通常以16位存储，量化后将它们四舍五入为更小的格式，如8位或4位。由于每数字节项就在方程中，从16位变为8位缓存减半，4位又减半。吸引力在于这适用于我们已有的模型，完全跳过了再培训。

质量成本取决于我们推到多远。

八位存储通常成本远低于准确率的百分之一，这在大多数工作负载中属于噪声范围。四位存储节省更多，并且在多针检索等要求较高的任务中开始显现可衡量的损失，模型需要从长上下文中提取多个具体事实。

专业方法优于纯四舍五入，因为缓存中的少数数字比其他数字更精确，尽管纯四舍五入已经能捕捉大部分收益。

驱逐通过删除模型不太可能需要的条目来减少令牌数量。

常见做法是保留最近的代币窗口，因为最近的上下文通常最重要，同时保留序列开始时的少数代币。这些开场代币起到了极其重要的作用。无论它们实际说什么，它们都会吸引大量注意力，作为锚点保持模型输出稳定。

驱逐问题的根源是结构性的。

一个标记是否重要取决于一个尚未出现的问题。我们现在丢弃的标记可能恰好是生成后续部分所需要的标记，一旦它消失，模型生成的内容就好像该标记一直不存在。这在检索任务中尤为明显，过于压缩的缓存可以处理普通聊天，却可能错过长文档中埋藏的某个事实。

更精细的方案会评估每个标记的重要性，并尝试预测哪些标记可以安全丢弃，这确实有所帮助，不过核心问题依然存在。

即使缓存的内容固定，服务系统管理内存的方式仍有很大提升空间，有两种技术可以优化。

较老的服务系统为每个请求预留一个大的连续内存块，大小为可能生成的最长输出。大多数请求实际完成时远未使用完这些空间，导致预留空间闲置，并且碎片积累。然而，分页注意力借鉴了操作系统的一个理念：将内存拆分为固定大小的小页，并按需分配。缓存被分割为可以位于内存任何位置的小块，通过查找表跟踪，每个请求映射到其对应的小块。结果是，那些曾因碎片浪费 60% 到 80% 缓存内存的系统，将浪费率降到不到 4%，吞吐量提高了 2 到 3 倍，这完全是由于将相同的数据更紧凑地存放。

第二种技术源于第一种。由于缓存存在可共享的小块，两个以相同文本开始的请求可以指向相同的物理块，同时每个请求持有自己的私有续写。这是前缀缓存的基础，而主要 API 称之为提示缓存的产品化版本。

对于任何重复使用前缀的工作负载，这种方式收益很大，例如每次调用都发送相同数千标记系统提示的代理。OpenAI 和 Anthropic 都报告，缓存命中时的成本和延迟降低了 50% 到 90%，且缓存的标记计费远低于新生成的标记。

然而，需要注意的是，在用户之间共享缓存状态会打开时间侧信道，可能泄露其他人的提示信息，这是一个活跃的关注点，我们在此暂且不讨论。

我们所研究的技术在节省内存的程度上看起来相似，但在所需的回报方面差异很大。

有些接近免费的。例如：

分组查询注意力消耗的质量非常少，已经成为安全的默认选择，这也是几乎每个当前模型都配备它的原因。

On-call shouldn’t feel like constant firefighting. This guide from Datadog breaks down how high-performing SRE teams reduce alert fatigue, streamline incident response, and design rotations that don’t burn engineers out. You’ll learn how to:

Cut alert noise by tying signals to real user impact

Improve response with clear roles and smarter escalation paths

Turn incidents into feedback loops that improve system reliability

Get the guide ：https://go.bytebytego.com/Datadog_080426

Why does sending a model a 100K-word prompt cost so much more than sending it a short one, even when the model and the hardware stay the same?

A key part of the answer lies in a block of working memory called the KV cache.

This memory is built up while the model generates a response. It is separate from the knowledge stored in the model’s weights, and it holds the key and value vectors computed for every token of the input. The cache grows with every token, and in a long context, it can take up significant space on the GPU.

For example, for a 70-billion-parameter model at a context of 128,000 tokens, it comes to roughly 40 gigabytes, a serious amount of GPU memory that grows with every user you add.

The above chart raises an obvious question, which is why a cache exists at all and why it grows the way it does. In this article, we will learn how LLMs use memory, how it gets expensive, and how to fix it.

Disclaimer: This post is based on publicly shared details from various sources. References at the end. Please comment if you notice any inaccuracies.

Let us start with the work a model does to produce one token.

To choose the next word, it runs an attention step, where the newest token compares itself against every token that came before it. This comparison uses two vectors for each earlier token, a key and a value, which are simply the numerical summaries the model computes for that token inside each layer. A model that rebuilt the key and value for every earlier token at every step would watch the work per token climb as the input grows. Such a repetition is pure waste, because those keys and values stay the same once a token has been processed.

The KV cache removes the waste by storing those key and value vectors the first time they are computed. On the next step, the model computes the key and value for only the new token and reads the rest straight from the cache.

Overall, this is a great solution. However, caching fixes the speed problem while creating a new one. The cache now has to be read on every step, and this turns out to be the real source of increasing cost.

One detail worth understanding here is that the cache holds vectors rather than the original text. It also explains out-of-memory errors that look puzzling when the model itself fits with room to spare.

Generation of tokens runs in two phases, and they stress the hardware in different ways:

The first phase is prefill, where the model reads the entire input at once. It processes all of the input tokens in parallel and builds their key and value vectors into the cache in a single pass. Prefill keeps the GPU’s math units busy, so we call it compute-bound, meaning the limit is how fast the chip can do arithmetic.

The second phase is decoding, where the model produces the output one token at a time. Each new token runs an attention step against the whole cache, which means the model reads every stored key and value out of GPU memory before it can emit the next token. It repeats that read for every token it produces. The limit here is how fast the cache can move from memory into the compute units, so we call decoding memory-bound.

The expense of long-context generation comes less from holding the cache and more from sweeping through all of it on every single token.

A larger cache means more data crossing the memory bus per token, which shows up directly as slower and costlier generation. It also explains why a request can run slowly even when it fits in memory comfortably.

If the cost tracks how much of the cache we read each step, then the size of the cache is the next thing to understand clearly.

The cache size is the product of a handful of numbers:

A factor of two covers the key and the value.

Layers count because each layer keeps its own cache.

Key-value heads set how many sets each layer stores.

Head dimension is the size of each of those vectors.

Bytes per number is the space one stored value takes.

Tokens are the context length, with one entry each.

Batch size is the number of requests served at once.

To summarize, the cache size equals 2 times layers times key-value heads times head dimension times bytes per number times tokens times batch size.

Two things that should be noted here are as follows:

The cache grows in a straight line with the token count, so doubling the context doubles the cache

It grows the same way with the batch size, so serving more users at once scales it just as fast.

As an approximate example, a Llama 3 70B model has 80 layers, 8 key-value heads, a head dimension of 128, and stores each number in 2 bytes. At a context of 128,000 tokens for a single request, those numbers multiply out to roughly 40 gigabytes, which is why a single long request can fill most of an 80-gigabyte card on its own.

Let us now look at the optimization techniques that help an LLM manage the memory aspects. Each of these techniques tries to push against a specific number.

The first two techniques change how attention itself is built, which means a model is committed to them during training. Both shrink the footprint of each token in the cache.

Grouped-query attention goes after the key-value head count. In a standard attention layer, every query head carries its own key and value head, so a model with 64 query heads stores 64 sets of keys and values. Grouped-query attention lets several query heads share one key-value head, which drops the number of stored sets sharply. For example, Llama 2 and 3 at 70B and Mistral 7B share down to 8 key-value heads, which cuts the cache by roughly eightfold against full multi-head attention. This is why a recent 70B model can hold a smaller cache than an older 7B one.

There is a more aggressive version called multi-query attention, where every query head shares a single key-value head. It saves the most memory of any head-sharing scheme. However, pushed that far, quality tends to drop, and training grows unstable, so most setups settle on the grouped middle ground as the better trade.

The second attack keeps the heads and compresses what each one stores.

Multi-head latent attention, introduced in the DeepSeek models, projects the keys and values down into a smaller latent representation before caching them, then expands them back when they are read. The savings are large. DeepSeek-V3 holds around 70 kilobytes per token, where comparable grouped-query models sit between 192 and 328. The cost lies in serving, since the compression adds work on every read and pairs awkwardly with some standard attention implementations, so it tends to pay off most once models and contexts grow large enough that cache traffic dominates.

As mentioned, head-sharing and latent attention both require control over the architecture, so they help when we are choosing or training a model. The next attacks work on a model we already have in hand.

Quantization goes after the bytes per number.

The keys and values are usually stored at 16 bits each, and quantization rounds them to a smaller format such as 8 bits or 4 bits. Since the bytes-per-number term sits right there in the equation, moving from 16 bits to 8 halves the whole cache, and going to 4 bits halves it again. The appeal is that this applies to a model we already have and skips retraining entirely.

The quality cost depends on how far we push it.

Eight-bit storage often costs well under a percent of accuracy, which puts it within the noise for most workloads. Four-bit storage saves more and starts to show measurable losses on demanding tasks such as multi-needle retrieval, where the model has to pull several specific facts out of a long context.

Specialized methods beat plain rounding because a few numbers in the cache deserve more precision than the rest, though plain rounding already captures most of the gain.

Eviction goes after the token count by dropping entries that the model is unlikely to need.

The common approach keeps a window of the most recent tokens, since recent context usually matters most, along with a few tokens from the very start of the sequence. Those opening tokens turn out to play an outsized role. They absorb a large share of attention regardless of what they actually say, acting as anchors that keep the model’s output stable.

The trouble with eviction is structural.

Whether a token matters depends on a question that has yet to arrive. A token we drop now can be exactly the one a later part of the generation needs, and once it is gone, the model generates as though that token had been absent the whole time. This shows up on retrieval tasks, where an aggressively trimmed cache handles a casual chat well and then misses a fact buried in the middle of a long document.

More refined schemes score each token’s importance and try to predict which ones are safe to drop, which helps, though the core problem remains.

Even with the contents of the cache fixed, the way a serving system manages memory leaves a lot on the table, and two techniques help.

Older serving systems reserved one large contiguous block per request, sized for the longest output it might produce. Most requests finished well short of that, leaving the reserved space idle, and the fragmentation added up. However, paged attention borrows an idea from operating systems, which break memory into small fixed-size pages and hand them out on demand. The cache gets split into small blocks that can live anywhere in memory, tracked by a lookup table that maps each request to its blocks. The result is that systems that wasted 60 to 80 percent of cache memory to fragmentation dropped that figure below 4 percent, and throughput climbed by two to three times, all from packing the same data more tightly.

The second technique results from the first. Since the cache lives in shareable blocks, two requests that begin with the same text can point at the same physical blocks while each holds its own private continuation. This is the foundation of prefix caching, and the productized version that the major APIs call prompt caching.

The win is large for any workload that repeats a prefix, such as an agent that sends the same multi-thousand-token system prompt on every call. OpenAI and Anthropic both report cost and latency reductions of 50 to 90 percent on cache hits, with cached tokens billed at a fraction of fresh ones.

However, one thing to note is that sharing cached state across users has opened timing side-channels that can leak information about other people’s prompts, an active concern we will leave aside here.

The techniques we’ve looked at look alike in how much memory they save and differ widely in what they ask for in return.

Some are close to free. For example:

Grouped-query attention costs very little quality and has become the safe default, which is why nearly every current model ships with it.