{"@context":"https://schema.org","@type":"NewsArticle","generatedAt":"2026-08-11T09:21:12.743Z","headline":"大语言模型内存为何昂贵及如何优化","description":"大语言模型处理长上下文时成本飙升，根源在于KV cache，它随token数和批处理规模线性增长。以Llama 3 70B为例，128，000 token上下文约需40GB GPU显存。优化手段包括分组查询注意力（GQA）等，在训练阶段压缩每个token的缓存占用。","url":"https://www.aioga.com/news/cmsevj4zx18b3ro2eed5gmzqj/","mainEntityOfPage":"https://www.aioga.com/news/cmsevj4zx18b3ro2eed5gmzqj/","datePublished":"2026-08-04T15:31:00.000Z","dateModified":"2026-08-04T15:31:00.000Z","inLanguage":"zh-CN","publisher":{"@type":"NewsMediaOrganization","name":"Aioga","url":"https://www.aioga.com"},"citation":["https://blog.bytebytego.com/p/why-an-llms-memory-gets-expensive","https://aihot.virxact.com/items/cmsevj4zx18b3ro2eed5gmzqj"],"canonicalUrl":"https://www.aioga.com/news/cmsevj4zx18b3ro2eed5gmzqj/","directAnswer":{"@type":"Answer","text":"Aioga 编辑摘要：大语言模型处理长上下文时成本飙升，根源在于KV cache，它随token数和批处理规模线性增长。 Aioga 将其归入「技巧观点」方向，重点关注它对真实使用和行业竞争的影响。","url":"https://www.aioga.com/news/cmsevj4zx18b3ro2eed5gmzqj/","dateCreated":"2026-08-04T15:31:00.000Z","author":{"@type":"Organization","@id":"https://www.aioga.com/authors/aioga-editorial/#editorial-team","name":"Aioga Editorial Team","url":"https://www.aioga.com/authors/aioga-editorial/"}},"evidence":[{"@type":"CreativeWork","name":"blog.bytebytego.com source article","url":"https://blog.bytebytego.com/p/why-an-llms-memory-gets-expensive","datePublished":"2026-08-04T15:31:00.000Z","provider":{"@type":"Organization","name":"blog.bytebytego.com","url":"https://blog.bytebytego.com/p/why-an-llms-memory-gets-expensive"}},{"@type":"CreativeWork","name":"AIHot archive record","url":"https://aihot.virxact.com/items/cmsevj4zx18b3ro2eed5gmzqj","datePublished":"2026-08-04T15:31:00.000Z","provider":{"@type":"Organization","name":"AIHot","url":"https://aihot.virxact.com/items/cmsevj4zx18b3ro2eed5gmzqj"}}],"aggregationSource":"ByteByteGo（RSS）","originalPublisher":{"name":"blog.bytebytego.com","url":"https://blog.bytebytego.com/p/why-an-llms-memory-gets-expensive"},"geoDeepAnswer":null,"article":{"id":"cmsevj4zx18b3ro2eed5gmzqj","slug":"cmsevj4zx18b3ro2eed5gmzqj","url":"https://www.aioga.com/news/cmsevj4zx18b3ro2eed5gmzqj/","title":"大语言模型内存为何昂贵及如何优化","title_en":"Why An LLM's Memory Gets Expensive and How to Fix It","summary":"大语言模型处理长上下文时成本飙升，根源在于KV cache，它随token数和批处理规模线性增长。以Llama 3 70B为例，128，000 token上下文约需40GB GPU显存。优化手段包括分组查询注意力（GQA）等，在训练阶段压缩每个token的缓存占用。","source":"ByteByteGo（RSS）","sourceUrl":"https://blog.bytebytego.com/p/why-an-llms-memory-gets-expensive","aiHotUrl":"https://aihot.virxact.com/items/cmsevj4zx18b3ro2eed5gmzqj","publishedAt":"2026-08-04T15:31:00.000Z","category":"技巧观点","score":46,"selected":false,"articleBody":["On-call shouldn’t feel like constant firefighting. This guide from Datadog breaks down how high-performing SRE teams reduce alert fatigue, streamline incident response, and design rotations that don’t burn engineers out. You’ll learn how to:","Cut alert noise by tying signals to real user impact","Improve response with clear roles and smarter escalation paths","Turn incidents into feedback loops that improve system reliability","Get the guide ：https://go.bytebytego.com/Datadog_080426","Why does sending a model a 100K-word prompt cost so much more than sending it a short one, even when the model and the hardware stay the same?","A key part of the answer lies in a block of working memory called the KV cache.","This memory is built up while the model generates a response. It is separate from the knowledge stored in the model’s weights, and it holds the key and value vectors computed for every token of the input. The cache grows with every token, and in a long context, it can take up significant space on the GPU.","For example, for a 70-billion-parameter model at a context of 128,000 tokens, it comes to roughly 40 gigabytes, a serious amount of GPU memory that grows with every user you add.","The above chart raises an obvious question, which is why a cache exists at all and why it grows the way it does. In this article, we will learn how LLMs use memory, how it gets expensive, and how to fix it.","Disclaimer: This post is based on publicly shared details from various sources. References at the end. Please comment if you notice any inaccuracies.","Let us start with the work a model does to produce one token.","To choose the next word, it runs an attention step, where the newest token compares itself against every token that came before it. This comparison uses two vectors for each earlier token, a key and a value, which are simply the numerical summaries the model computes for that token inside each layer. A model that rebuilt the key and value for every earlier token at every step would watch the work per token climb as the input grows. Such a repetition is pure waste, because those keys and values stay the same once a token has been processed.","The KV cache removes the waste by storing those key and value vectors the first time they are computed. On the next step, the model computes the key and value for only the new token and reads the rest straight from the cache.","Overall, this is a great solution. However, caching fixes the speed problem while creating a new one. The cache now has to be read on every step, and this turns out to be the real source of increasing cost.","One detail worth understanding here is that the cache holds vectors rather than the original text. It also explains out-of-memory errors that look puzzling when the model itself fits with room to spare.","Generation of tokens runs in two phases, and they stress the hardware in different ways:","The first phase is prefill, where the model reads the entire input at once. It processes all of the input tokens in parallel and builds their key and value vectors into the cache in a single pass. Prefill keeps the GPU’s math units busy, so we call it compute-bound, meaning the limit is how fast the chip can do arithmetic.","The second phase is decoding, where the model produces the output one token at a time. Each new token runs an attention step against the whole cache, which means the model reads every stored key and value out of GPU memory before it can emit the next token. It repeats that read for every token it produces. The limit here is how fast the cache can move from memory into the compute units, so we call decoding memory-bound.","The expense of long-context generation comes less from holding the cache and more from sweeping through all of it on every single token.","A larger cache means more data crossing the memory bus per token, which shows up directly as slower and costlier generation. It also explains why a request can run slowly even when it fits in memory comfortably.","If the cost tracks how much of the cache we read each step, then the size of the cache is the next thing to understand clearly.","The cache size is the product of a handful of numbers:","A factor of two covers the key and the value.","Layers count because each layer keeps its own cache.","Key-value heads set how many sets each layer stores.","Head dimension is the size of each of those vectors.","Bytes per number is the space one stored value takes.","Tokens are the context length, with one entry each.","Batch size is the number of requests served at once.","To summarize, the cache size equals 2 times layers times key-value heads times head dimension times bytes per number times tokens times batch size.","Two things that should be noted here are as follows:","The cache grows in a straight line with the token count, so doubling the context doubles the cache","It grows the same way with the batch size, so serving more users at once scales it just as fast.","As an approximate example, a Llama 3 70B model has 80 layers, 8 key-value heads, a head dimension of 128, and stores each number in 2 bytes. At a context of 128,000 tokens for a single request, those numbers multiply out to roughly 40 gigabytes, which is why a single long request can fill most of an 80-gigabyte card on its own.","Let us now look at the optimization techniques that help an LLM manage the memory aspects. Each of these techniques tries to push against a specific number.","The first two techniques change how attention itself is built, which means a model is committed to them during training. Both shrink the footprint of each token in the cache.","Grouped-query attention goes after the key-value head count. In a standard attention layer, every query head carries its own key and value head, so a model with 64 query heads stores 64 sets of keys and values. Grouped-query attention lets several query heads share one key-value head, which drops the number of stored sets sharply. For example, Llama 2 and 3 at 70B and Mistral 7B share down to 8 key-value heads, which cuts the cache by roughly eightfold against full multi-head attention. This is why a recent 70B model can hold a smaller cache than an older 7B one.","There is a more aggressive version called multi-query attention, where every query head shares a single key-value head. It saves the most memory of any head-sharing scheme. However, pushed that far, quality tends to drop, and training grows unstable, so most setups settle on the grouped middle ground as the better trade.","The second attack keeps the heads and compresses what each one stores.","Multi-head latent attention, introduced in the DeepSeek models, projects the keys and values down into a smaller latent representation before caching them, then expands them back when they are read. The savings are large. DeepSeek-V3 holds around 70 kilobytes per token, where comparable grouped-query models sit between 192 and 328. The cost lies in serving, since the compression adds work on every read and pairs awkwardly with some standard attention implementations, so it tends to pay off most once models and contexts grow large enough that cache traffic dominates.","As mentioned, head-sharing and latent attention both require control over the architecture, so they help when we are choosing or training a model. The next attacks work on a model we already have in hand.","Quantization goes after the bytes per number.","The keys and values are usually stored at 16 bits each, and quantization rounds them to a smaller format such as 8 bits or 4 bits. Since the bytes-per-number term sits right there in the equation, moving from 16 bits to 8 halves the whole cache, and going to 4 bits halves it again. The appeal is that this applies to a model we already have and skips retraining entirely.","The quality cost depends on how far we push it.","Eight-bit storage often costs well under a percent of accuracy, which puts it within the noise for most workloads. Four-bit storage saves more and starts to show measurable losses on demanding tasks such as multi-needle retrieval, where the model has to pull several specific facts out of a long context.","Specialized methods beat plain rounding because a few numbers in the cache deserve more precision than the rest, though plain rounding already captures most of the gain.","Eviction goes after the token count by dropping entries that the model is unlikely to need.","The common approach keeps a window of the most recent tokens, since recent context usually matters most, along with a few tokens from the very start of the sequence. Those opening tokens turn out to play an outsized role. They absorb a large share of attention regardless of what they actually say, acting as anchors that keep the model’s output stable.","The trouble with eviction is structural.","Whether a token matters depends on a question that has yet to arrive. A token we drop now can be exactly the one a later part of the generation needs, and once it is gone, the model generates as though that token had been absent the whole time. This shows up on retrieval tasks, where an aggressively trimmed cache handles a casual chat well and then misses a fact buried in the middle of a long document.","More refined schemes score each token’s importance and try to predict which ones are safe to drop, which helps, though the core problem remains.","Even with the contents of the cache fixed, the way a serving system manages memory leaves a lot on the table, and two techniques help.","Older serving systems reserved one large contiguous block per request, sized for the longest output it might produce. Most requests finished well short of that, leaving the reserved space idle, and the fragmentation added up. However, paged attention borrows an idea from operating systems, which break memory into small fixed-size pages and hand them out on demand. The cache gets split into small blocks that can live anywhere in memory, tracked by a lookup table that maps each request to its blocks. The result is that systems that wasted 60 to 80 percent of cache memory to fragmentation dropped that figure below 4 percent, and throughput climbed by two to three times, all from packing the same data more tightly.","The second technique results from the first. Since the cache lives in shareable blocks, two requests that begin with the same text can point at the same physical blocks while each holds its own private continuation. This is the foundation of prefix caching, and the productized version that the major APIs call prompt caching.","The win is large for any workload that repeats a prefix, such as an agent that sends the same multi-thousand-token system prompt on every call. OpenAI and Anthropic both report cost and latency reductions of 50 to 90 percent on cache hits, with cached tokens billed at a fraction of fresh ones.","However, one thing to note is that sharing cached state across users has opened timing side-channels that can leak information about other people’s prompts, an active concern we will leave aside here.","The techniques we’ve looked at look alike in how much memory they save and differ widely in what they ask for in return.","Some are close to free. For example:","Grouped-query attention costs very little quality and has become the safe default, which is why nearly every current model ships with it."],"articleImages":[],"mediaStatus":"none","articleBodyZh":["随叫随到不应该让人感觉像是不断地灭火。Datadog 提供的这份指南分解了高效 SRE 团队如何减少警报疲劳、优化事件响应，并设计不会让工程师精疲力尽的轮班。你将学习如何：","通过将信号与真实用户影响关联来减少警报噪音","通过明确角色和更智能的升级路径提高响应效率","将事件转化为改进系统可靠性的反馈循环","获取指南：https://go.bytebytego.com/Datadog_080426","为什么即使模型和硬件保持不变，发送 10 万词的提示给模型的成本也比发送短提示高得多？","答案的关键部分在于一块称为 KV 缓存的工作内存。","这种内存是在模型生成响应时积累的。它与存储在模型权重中的知识是分开的，并且保存了为输入的每个 token 计算的键和值向量。缓存会随着每个 token 的增加而增长，在长上下文中，它可能占用 GPU 上的大量空间。","例如，对于一个 700 亿参数的模型，在 128,000 个 token 的上下文中，这大约需要 40 GB 的 GPU 内存，这是一个随着每增加一个用户而增长的严重内存负担。","上图提出了一个显而易见的问题，即为什么会存在缓存以及它为什么会以这种方式增长。在本文中，我们将学习 LLM 如何使用内存，它是如何变得昂贵的，以及如何解决这个问题。","免责声明：本文基于来自各种来源的公开资料。参考文献见文末。如果你发现任何不准确之处，请评论指出。","让我们从模型生成一个 token 的工作开始。","为了选择下一个词，它会执行一次注意力步骤，其中最新的 token 会与此前的每个 token 进行比较。这个比较使用两个向量，即键和值，它们只是模型在每一层为该 token 计算的数值总结。如果模型在每一步都为每个先前的 token 重新构建键和值，那么随着输入的增长，每个 token 的工作量就会增加。这种重复是完全浪费的，因为一旦处理了一个 token，这些键和值就保持不变。","KV缓存通过在首次计算时存储这些键和值向量来消除浪费。下一步，模型只计算新令牌的键和值，其余直接从缓存读取。","总体来说，这是一个很好的解决方案。然而，缓存解决了速度问题，同时又制造了新的问题。缓存现在必须在每一步都被读取，这才是成本上升的真正原因。","这里需要理解的一个细节是，缓存存储的是向量，而不是原始文本。它还解释了模型本身有余裕空间时，显得显得困惑的内存错误。","代币生成分为两个阶段，它们以不同方式对硬件施加压力：","第一阶段是预填充，模型一次性读取整个输入。它并行处理所有输入令牌，并在一次内将密钥和值向量构建到缓存中。预填充让GPU的数学单位保持忙碌，所以我们称之为计算限制，意味着芯片完成算术的速度限制。","第二阶段是解码，模型一次输出一个令牌。每个新令牌对整个缓存执行注意步，这意味着模型会在发出下一个令牌之前，先从GPU内存读取所有存储的键和值。它对每个令牌都重复该读。这里的限制在于缓存从内存移动到计算单元的速度，因此我们称解码为内存限制。","长上下文生成的成本更多是对每个代币进行扫除，而不是持有缓存。","更大的缓存意味着每个令牌通过内存总线的数据更多，这直接表现为生成速度更慢且成本更高。这也解释了为什么即使请求在内存中舒适地存在，运行速度也会很慢。","如果成本是追踪每一步读取缓存的量，那么缓存的大小就是接下来要明确理解的。","缓存大小是几个数字的乘积：","二的因子涵盖了密钥和数值。","图层之所以重要，是因为每一层都保留自己的缓存。","键值头设置每层存储多少集。","头维度是每个这些向量的大小。","每个数字的字节数是指存储一个值所占的空间。","Tokens 是上下文长度，每个条目计算一次。","批量大小是一次处理的请求数量。","总之，缓存大小等于 2 乘以层数乘以键值头数乘以头维度乘以每个数字的字节数乘以 tokens 再乘以批量大小。","这里应注意的两件事如下：","缓存随着 token 数量线性增长，因此上下文加倍会使缓存加倍。","它随着批量大小的增加而同样增长，因此同时为更多用户服务，其规模同样增长迅速。","作为一个大致的例子，Llama 3 70B 模型有 80 层，8 个键值头，头维度为 128，并且每个数字存储为 2 字节。在单次请求的上下文为 128,000 tokens 时，这些数字大约乘出来为 40 GB，这就是为什么单个长请求就可以填满一个 80 GB 显卡的大部分空间。","现在让我们看看帮助大型语言模型管理内存的优化技术。每种技术都试图针对特定的数值进行优化。","前两种技术改变了注意力机制本身的构建方式，这意味着模型在训练时就已固定。两者都缩小了每个 token 在缓存中占用的空间。","分组查询注意力针对键值头数量。在标准注意力层中，每个查询头都有自己的键和值头，因此有 64 个查询头的模型会存储 64 套键和值。分组查询注意力允许多个查询头共享一个键值头，这大幅减少了存储集合的数量。例如，Llama 2 和 3 70B 以及 Mistral 7B 共享最多 8 个键值头，与完整多头注意力相比可将缓存大约减少八倍。这也是为什么最近的 70B 模型可以拥有比较老的 7B 模型更小的缓存。","有一种更激进的版本称为多查询注意力，每个查询头共享一个键值头。它在所有头共享方案中节省最多的内存。然而，推到那一步时，质量往往下降，训练变得不稳定，所以大多数设置选择分组共享作为更好的折中方案。","第二种方案保持头的数量不变，而是压缩每个头存储的内容。","多头潜在注意力在DeepSeek模型中引入，它将键和值投影为更小的潜在表示，然后缓存，读取后再展开。节省的费用很大。DeepSeek-V3 每个令牌大约有 70 千字节，而类似的分组查询模型则在 192 到 328 千字节之间。成本在于服务，因为压缩会在每次读取中增加工作量，并且与一些标准注意力实现配合得很尴尬，所以当模型和上下文足够大，缓存流量占主导地位时，压缩通常会带来回报。","如前所述，头脑共享和潜在注意力都需要对架构的控制，因此在选择或训练模型时它们很有帮助。接下来的攻击基于我们已有的模型。","量化是在每个数字的字节之后。","键和值通常以16位存储，量化后将它们四舍五入为更小的格式，如8位或4位。由于每数字节项就在方程中，从16位变为8位缓存减半，4位又减半。吸引力在于这适用于我们已有的模型，完全跳过了再培训。","质量成本取决于我们推到多远。","八位存储通常成本远低于准确率的百分之一，这在大多数工作负载中属于噪声范围。四位存储节省更多，并且在多针检索等要求较高的任务中开始显现可衡量的损失，模型需要从长上下文中提取多个具体事实。","专业方法优于纯四舍五入，因为缓存中的少数数字比其他数字更精确，尽管纯四舍五入已经能捕捉大部分收益。","驱逐通过删除模型不太可能需要的条目来减少令牌数量。","常见做法是保留最近的代币窗口，因为最近的上下文通常最重要，同时保留序列开始时的少数代币。这些开场代币起到了极其重要的作用。无论它们实际说什么，它们都会吸引大量注意力，作为锚点保持模型输出稳定。","驱逐问题的根源是结构性的。","一个标记是否重要取决于一个尚未出现的问题。我们现在丢弃的标记可能恰好是生成后续部分所需要的标记，一旦它消失，模型生成的内容就好像该标记一直不存在。这在检索任务中尤为明显，过于压缩的缓存可以处理普通聊天，却可能错过长文档中埋藏的某个事实。","更精细的方案会评估每个标记的重要性，并尝试预测哪些标记可以安全丢弃，这确实有所帮助，不过核心问题依然存在。","即使缓存的内容固定，服务系统管理内存的方式仍有很大提升空间，有两种技术可以优化。","较老的服务系统为每个请求预留一个大的连续内存块，大小为可能生成的最长输出。大多数请求实际完成时远未使用完这些空间，导致预留空间闲置，并且碎片积累。然而，分页注意力借鉴了操作系统的一个理念：将内存拆分为固定大小的小页，并按需分配。缓存被分割为可以位于内存任何位置的小块，通过查找表跟踪，每个请求映射到其对应的小块。结果是，那些曾因碎片浪费 60% 到 80% 缓存内存的系统，将浪费率降到不到 4%，吞吐量提高了 2 到 3 倍，这完全是由于将相同的数据更紧凑地存放。","第二种技术源于第一种。由于缓存存在可共享的小块，两个以相同文本开始的请求可以指向相同的物理块，同时每个请求持有自己的私有续写。这是前缀缓存的基础，而主要 API 称之为提示缓存的产品化版本。","对于任何重复使用前缀的工作负载，这种方式收益很大，例如每次调用都发送相同数千标记系统提示的代理。OpenAI 和 Anthropic 都报告，缓存命中时的成本和延迟降低了 50% 到 90%，且缓存的标记计费远低于新生成的标记。","然而，需要注意的是，在用户之间共享缓存状态会打开时间侧信道，可能泄露其他人的提示信息，这是一个活跃的关注点，我们在此暂且不讨论。","我们所研究的技术在节省内存的程度上看起来相似，但在所需的回报方面差异很大。","有些接近免费的。例如：","分组查询注意力消耗的质量非常少，已经成为安全的默认选择，这也是几乎每个当前模型都配备它的原因。"],"translationStatus":"translated","bodyOrigin":"source-page","editorial":{"summary":"Aioga 编辑摘要：大语言模型处理长上下文时成本飙升，根源在于KV cache，它随token数和批处理规模线性增长。 Aioga 将其归入「技巧观点」方向，重点关注它对真实使用和行业竞争的影响。","background":"背景分析：模型与研究类动态需要结合能力边界、开放方式、成本、可用性和真实任务表现判断，单项指标领先不等于已经形成稳定采用。","viewpoint":"Aioga 判断：这条动态更适合作为行业观察信号，当前信息足以建立线索，但不足以推导长期结论。","implications":"影响分析：对相关团队而言，短期应先核对来源、可用范围和实际成本，再判断是否值得接入或跟进。","nextStep":"后续观察：继续观察官方文档、实际可用性、价格变化、开发者反馈和竞品回应。","evidenceRefs":["title","summary","articleBody"],"confidence":"medium","status":"published","aiGenerated":false,"autoApproved":true,"generatedBy":"rule-safe-fallback","generatedAt":"2026-08-11T09:23:27.691Z","sourceHash":"32b950fc0a6b4064","validation":{"passed":true,"mode":"rule-safe-fallback","checks":["schema","length","source-attribution","no-html"]}},"tags":["技巧观点","ByteByteGo（RSS）"],"translations":{"zh-CN":{"title":"大语言模型内存为何昂贵及如何优化","summary":"大语言模型处理长上下文时成本飙升，根源在于KV cache，它随token数和批处理规模线性增长。以Llama 3 70B为例，128，000 token上下文约需40GB GPU显存。优化手段包括分组查询注意力（GQA）等，在训练阶段压缩每个token的缓存占用。","category":"技巧观点","source":"blog.bytebytego.com","aggregationSource":"ByteByteGo（RSS）","pageTitle":"大语言模型内存为何昂贵及如何优化 - Aioga AI资讯","description":"大语言模型处理长上下文时成本飙升，根源在于KV cache，它随token数和批处理规模线性增长。以Llama 3 70B为例，128，000 token上下文约需40GB GPU显存。优化手段包括分组查询注意力（GQA）等，在训练阶段压缩每个token的缓存占用。","url":"https://www.aioga.com/news/cmsevj4zx18b3ro2eed5gmzqj/","articleBody":["随叫随到不应该让人感觉像是不断地灭火。Datadog 提供的这份指南分解了高效 SRE 团队如何减少警报疲劳、优化事件响应，并设计不会让工程师精疲力尽的轮班。你将学习如何：","通过将信号与真实用户影响关联来减少警报噪音","通过明确角色和更智能的升级路径提高响应效率","将事件转化为改进系统可靠性的反馈循环","获取指南：https://go.bytebytego.com/Datadog_080426","为什么即使模型和硬件保持不变，发送 10 万词的提示给模型的成本也比发送短提示高得多？","答案的关键部分在于一块称为 KV 缓存的工作内存。","这种内存是在模型生成响应时积累的。它与存储在模型权重中的知识是分开的，并且保存了为输入的每个 token 计算的键和值向量。缓存会随着每个 token 的增加而增长，在长上下文中，它可能占用 GPU 上的大量空间。","例如，对于一个 700 亿参数的模型，在 128,000 个 token 的上下文中，这大约需要 40 GB 的 GPU 内存，这是一个随着每增加一个用户而增长的严重内存负担。","上图提出了一个显而易见的问题，即为什么会存在缓存以及它为什么会以这种方式增长。在本文中，我们将学习 LLM 如何使用内存，它是如何变得昂贵的，以及如何解决这个问题。","免责声明：本文基于来自各种来源的公开资料。参考文献见文末。如果你发现任何不准确之处，请评论指出。","让我们从模型生成一个 token 的工作开始。","为了选择下一个词，它会执行一次注意力步骤，其中最新的 token 会与此前的每个 token 进行比较。这个比较使用两个向量，即键和值，它们只是模型在每一层为该 token 计算的数值总结。如果模型在每一步都为每个先前的 token 重新构建键和值，那么随着输入的增长，每个 token 的工作量就会增加。这种重复是完全浪费的，因为一旦处理了一个 token，这些键和值就保持不变。","KV缓存通过在首次计算时存储这些键和值向量来消除浪费。下一步，模型只计算新令牌的键和值，其余直接从缓存读取。","总体来说，这是一个很好的解决方案。然而，缓存解决了速度问题，同时又制造了新的问题。缓存现在必须在每一步都被读取，这才是成本上升的真正原因。","这里需要理解的一个细节是，缓存存储的是向量，而不是原始文本。它还解释了模型本身有余裕空间时，显得显得困惑的内存错误。","代币生成分为两个阶段，它们以不同方式对硬件施加压力：","第一阶段是预填充，模型一次性读取整个输入。它并行处理所有输入令牌，并在一次内将密钥和值向量构建到缓存中。预填充让GPU的数学单位保持忙碌，所以我们称之为计算限制，意味着芯片完成算术的速度限制。","第二阶段是解码，模型一次输出一个令牌。每个新令牌对整个缓存执行注意步，这意味着模型会在发出下一个令牌之前，先从GPU内存读取所有存储的键和值。它对每个令牌都重复该读。这里的限制在于缓存从内存移动到计算单元的速度，因此我们称解码为内存限制。","长上下文生成的成本更多是对每个代币进行扫除，而不是持有缓存。","更大的缓存意味着每个令牌通过内存总线的数据更多，这直接表现为生成速度更慢且成本更高。这也解释了为什么即使请求在内存中舒适地存在，运行速度也会很慢。","如果成本是追踪每一步读取缓存的量，那么缓存的大小就是接下来要明确理解的。","缓存大小是几个数字的乘积：","二的因子涵盖了密钥和数值。","图层之所以重要，是因为每一层都保留自己的缓存。","键值头设置每层存储多少集。","头维度是每个这些向量的大小。","每个数字的字节数是指存储一个值所占的空间。","Tokens 是上下文长度，每个条目计算一次。","批量大小是一次处理的请求数量。","总之，缓存大小等于 2 乘以层数乘以键值头数乘以头维度乘以每个数字的字节数乘以 tokens 再乘以批量大小。","这里应注意的两件事如下：","缓存随着 token 数量线性增长，因此上下文加倍会使缓存加倍。","它随着批量大小的增加而同样增长，因此同时为更多用户服务，其规模同样增长迅速。","作为一个大致的例子，Llama 3 70B 模型有 80 层，8 个键值头，头维度为 128，并且每个数字存储为 2 字节。在单次请求的上下文为 128,000 tokens 时，这些数字大约乘出来为 40 GB，这就是为什么单个长请求就可以填满一个 80 GB 显卡的大部分空间。","现在让我们看看帮助大型语言模型管理内存的优化技术。每种技术都试图针对特定的数值进行优化。","前两种技术改变了注意力机制本身的构建方式，这意味着模型在训练时就已固定。两者都缩小了每个 token 在缓存中占用的空间。","分组查询注意力针对键值头数量。在标准注意力层中，每个查询头都有自己的键和值头，因此有 64 个查询头的模型会存储 64 套键和值。分组查询注意力允许多个查询头共享一个键值头，这大幅减少了存储集合的数量。例如，Llama 2 和 3 70B 以及 Mistral 7B 共享最多 8 个键值头，与完整多头注意力相比可将缓存大约减少八倍。这也是为什么最近的 70B 模型可以拥有比较老的 7B 模型更小的缓存。","有一种更激进的版本称为多查询注意力，每个查询头共享一个键值头。它在所有头共享方案中节省最多的内存。然而，推到那一步时，质量往往下降，训练变得不稳定，所以大多数设置选择分组共享作为更好的折中方案。","第二种方案保持头的数量不变，而是压缩每个头存储的内容。","多头潜在注意力在DeepSeek模型中引入，它将键和值投影为更小的潜在表示，然后缓存，读取后再展开。节省的费用很大。DeepSeek-V3 每个令牌大约有 70 千字节，而类似的分组查询模型则在 192 到 328 千字节之间。成本在于服务，因为压缩会在每次读取中增加工作量，并且与一些标准注意力实现配合得很尴尬，所以当模型和上下文足够大，缓存流量占主导地位时，压缩通常会带来回报。","如前所述，头脑共享和潜在注意力都需要对架构的控制，因此在选择或训练模型时它们很有帮助。接下来的攻击基于我们已有的模型。","量化是在每个数字的字节之后。","键和值通常以16位存储，量化后将它们四舍五入为更小的格式，如8位或4位。由于每数字节项就在方程中，从16位变为8位缓存减半，4位又减半。吸引力在于这适用于我们已有的模型，完全跳过了再培训。","质量成本取决于我们推到多远。","八位存储通常成本远低于准确率的百分之一，这在大多数工作负载中属于噪声范围。四位存储节省更多，并且在多针检索等要求较高的任务中开始显现可衡量的损失，模型需要从长上下文中提取多个具体事实。","专业方法优于纯四舍五入，因为缓存中的少数数字比其他数字更精确，尽管纯四舍五入已经能捕捉大部分收益。","驱逐通过删除模型不太可能需要的条目来减少令牌数量。","常见做法是保留最近的代币窗口，因为最近的上下文通常最重要，同时保留序列开始时的少数代币。这些开场代币起到了极其重要的作用。无论它们实际说什么，它们都会吸引大量注意力，作为锚点保持模型输出稳定。","驱逐问题的根源是结构性的。","一个标记是否重要取决于一个尚未出现的问题。我们现在丢弃的标记可能恰好是生成后续部分所需要的标记，一旦它消失，模型生成的内容就好像该标记一直不存在。这在检索任务中尤为明显，过于压缩的缓存可以处理普通聊天，却可能错过长文档中埋藏的某个事实。","更精细的方案会评估每个标记的重要性，并尝试预测哪些标记可以安全丢弃，这确实有所帮助，不过核心问题依然存在。","即使缓存的内容固定，服务系统管理内存的方式仍有很大提升空间，有两种技术可以优化。","较老的服务系统为每个请求预留一个大的连续内存块，大小为可能生成的最长输出。大多数请求实际完成时远未使用完这些空间，导致预留空间闲置，并且碎片积累。然而，分页注意力借鉴了操作系统的一个理念：将内存拆分为固定大小的小页，并按需分配。缓存被分割为可以位于内存任何位置的小块，通过查找表跟踪，每个请求映射到其对应的小块。结果是，那些曾因碎片浪费 60% 到 80% 缓存内存的系统，将浪费率降到不到 4%，吞吐量提高了 2 到 3 倍，这完全是由于将相同的数据更紧凑地存放。","第二种技术源于第一种。由于缓存存在可共享的小块，两个以相同文本开始的请求可以指向相同的物理块，同时每个请求持有自己的私有续写。这是前缀缓存的基础，而主要 API 称之为提示缓存的产品化版本。","对于任何重复使用前缀的工作负载，这种方式收益很大，例如每次调用都发送相同数千标记系统提示的代理。OpenAI 和 Anthropic 都报告，缓存命中时的成本和延迟降低了 50% 到 90%，且缓存的标记计费远低于新生成的标记。","然而，需要注意的是，在用户之间共享缓存状态会打开时间侧信道，可能泄露其他人的提示信息，这是一个活跃的关注点，我们在此暂且不讨论。","我们所研究的技术在节省内存的程度上看起来相似，但在所需的回报方面差异很大。","有些接近免费的。例如：","分组查询注意力消耗的质量非常少，已经成为安全的默认选择，这也是几乎每个当前模型都配备它的原因。"]},"en":{"title":"Why large language model memory is expensive and how to optimize it","summary":"The cost of processing long contexts with large language models skyrockets, rooted in the KV cache, which grows linearly with the number of tokens and batch size. For example, Llama 3 70B requires about 40GB of GPU memory for a 128,000 token context. Optimization methods include grouped-query attention (GQA), which compresses the cache usage of each token during training.","category":"Insights","source":"ByteByteGo（RSS）","aggregationSource":"ByteByteGo（RSS）","pageTitle":"Why large language model memory is expensive and how to optimize it - Aioga AI News","description":"The cost of processing long contexts with large language models skyrockets, rooted in the KV cache, which grows linearly with the number of tokens and batch size. For example, Llam...","url":"https://www.aioga.com/en/news/cmsevj4zx18b3ro2eed5gmzqj/","contentTranslated":true,"sourceHash":"fca74342898045be","translatedAt":"2026-08-04T16:43:01.131Z"},"ja":{"title":"大規模言語モデルのメモリが高価な理由と最適化方法","summary":"大規模言語モデルは長いコンテキストを処理する際にコストが急騰します。その原因は KV キャッシュであり、トークン数とバッチサイズに応じて線形に増加します。例えば Llama 3 70B の場合、128,000 トークンのコンテキストで約 40GB の GPU メモリが必要です。最適化手段にはグループ化クエリアテンション（GQA）などがあり、学習段階で各トークンのキャッシュ使用量を圧縮します。","category":"ヒントと視点","source":"ByteByteGo（RSS）","aggregationSource":"ByteByteGo（RSS）","pageTitle":"大規模言語モデルのメモリが高価な理由と最適化方法 - Aioga AIニュース","description":"大規模言語モデルは長いコンテキストを処理する際にコストが急騰します。その原因は KV キャッシュであり、トークン数とバッチサイズに応じて線形に増加します。例えば Llama 3 70B の場合、128,000 トークンのコンテキストで約 40GB の GPU メモリが必要です。最適化手段にはグループ化クエリアテンション（GQA）などがあり、学習段階で各トーク...","url":"https://www.aioga.com/ja/news/cmsevj4zx18b3ro2eed5gmzqj/","contentTranslated":true,"sourceHash":"fca74342898045be","translatedAt":"2026-08-04T16:43:02.203Z"},"ko":{"title":"대형 언어 모델 메모리가 왜 비싸며 어떻게 최적화할 수 있는가","summary":"대형 언어 모델이 긴 컨텍스트를 처리할 때 비용이 급등하는 원인은 KV 캐시에 있으며, 이는 토큰 수와 배치 크기에 따라 선형 증가한다. Llama 3 70B를 예로 들면, 128,000 토큰 컨텍스트는 약 40GB GPU 메모리를 필요로 한다. 최적화 방법에는 그룹 쿼리 어텐션(GQA) 등이 있으며, 학습 단계에서 각 토큰이 사용하는 캐시를 압축하는 방식이 있다.","category":"인사이트","source":"ByteByteGo（RSS）","aggregationSource":"ByteByteGo（RSS）","pageTitle":"대형 언어 모델 메모리가 왜 비싸며 어떻게 최적화할 수 있는가 - Aioga AI 뉴스","description":"대형 언어 모델이 긴 컨텍스트를 처리할 때 비용이 급등하는 원인은 KV 캐시에 있으며, 이는 토큰 수와 배치 크기에 따라 선형 증가한다. Llama 3 70B를 예로 들면, 128,000 토큰 컨텍스트는 약 40GB GPU 메모리를 필요로 한다. 최적화 방법에는 그룹 쿼리 어텐션(GQA) 등이 있으며, 학습 단계에서 각...","url":"https://www.aioga.com/ko/news/cmsevj4zx18b3ro2eed5gmzqj/","contentTranslated":true,"sourceHash":"fca74342898045be","translatedAt":"2026-08-04T16:43:08.924Z"},"es":{"title":"Por qué la memoria de los grandes modelos de lenguaje es costosa y cómo optimizarla","summary":"El costo de procesar contextos largos con grandes modelos de lenguaje aumenta drásticamente debido a la caché KV, que crece linealmente con el número de tokens y el tamaño del lote. Por ejemplo, Llama 3 70B, con un contexto de 128,000 tokens, requiere aproximadamente 40 GB de memoria GPU. Las técnicas de optimización incluyen la atención de consulta agrupada (GQA) y la compresión de la caché por token durante la fase de entrenamiento.","category":"Ideas","source":"ByteByteGo（RSS）","aggregationSource":"ByteByteGo（RSS）","pageTitle":"Por qué la memoria de los grandes modelos de lenguaje es costosa y cómo optimizarla - Aioga Noticias de IA","description":"El costo de procesar contextos largos con grandes modelos de lenguaje aumenta drásticamente debido a la caché KV, que crece linealmente con el número de tokens y el tamaño del lote...","url":"https://www.aioga.com/es/news/cmsevj4zx18b3ro2eed5gmzqj/","contentTranslated":true,"sourceHash":"fca74342898045be","translatedAt":"2026-08-04T16:43:06.584Z"},"fr":{"title":"Pourquoi la mémoire des grands modèles linguistiques est coûteuse et comment l'optimiser","summary":"Le coût des grands modèles linguistiques explose lorsqu'ils traitent de longs contextes, la cause principale étant le cache KV, qui croît linéairement avec le nombre de tokens et la taille des lots. Par exemple, pour Llama 3 70B, un contexte de 128 000 tokens nécessite environ 40 Go de mémoire GPU. Les méthodes d'optimisation incluent l'attention à requêtes groupées (GQA), qui compressent l'occupation de la mémoire pour chaque token pendant l'entraînement.","category":"Analyses","source":"ByteByteGo（RSS）","aggregationSource":"ByteByteGo（RSS）","pageTitle":"Pourquoi la mémoire des grands modèles linguistiques est coûteuse et comment l'optimiser - Aioga Actualités IA","description":"Le coût des grands modèles linguistiques explose lorsqu'ils traitent de longs contextes, la cause principale étant le cache KV, qui croît linéairement avec le nombre de tokens et l...","url":"https://www.aioga.com/fr/news/cmsevj4zx18b3ro2eed5gmzqj/","contentTranslated":true,"sourceHash":"fca74342898045be","translatedAt":"2026-08-04T16:43:15.528Z"},"de":{"title":"Warum der Speicherbedarf großer Sprachmodelle hoch ist und wie man ihn optimiert","summary":"Die Kosten großer Sprachmodelle steigen bei langer Kontextverarbeitung stark an, der Grund liegt im KV-Cache, der linear mit der Tokenanzahl und der Batchgröße wächst. Am Beispiel von Llama 3 70B erfordert ein Kontext von 128.000 Tokens etwa 40 GB GPU-Speicher. Optimierungsmethoden beinhalten unter anderem Grouped Query Attention (GQA), welche die Speicherbelegung pro Token während des Trainings reduziert.","category":"技巧观点","source":"ByteByteGo（RSS）","aggregationSource":"ByteByteGo（RSS）","pageTitle":"Warum der Speicherbedarf großer Sprachmodelle hoch ist und wie man ihn optimiert - Aioga KI-News","description":"Die Kosten großer Sprachmodelle steigen bei langer Kontextverarbeitung stark an, der Grund liegt im KV-Cache, der linear mit der Tokenanzahl und der Batchgröße wächst. Am Beispiel...","url":"https://www.aioga.com/de/news/cmsevj4zx18b3ro2eed5gmzqj/","contentTranslated":true,"sourceHash":"fca74342898045be","translatedAt":"2026-08-04T16:43:16.039Z"},"pt-BR":{"title":"Por que a memória de grandes modelos de linguagem é cara e como otimizar","summary":"O custo de lidar com contextos longos em grandes modelos de linguagem dispara devido ao cache KV, que cresce linearmente com o número de tokens e o tamanho do lote. Por exemplo, o Llama 3 70B, com contexto de 128.000 tokens, exige cerca de 40GB de memória GPU. Métodos de otimização incluem atenção por grupos (GQA) e compressão do cache por token durante a fase de treinamento.","category":"技巧观点","source":"ByteByteGo（RSS）","aggregationSource":"ByteByteGo（RSS）","pageTitle":"Por que a memória de grandes modelos de linguagem é cara e como otimizar - Aioga Notícias de IA","description":"O custo de lidar com contextos longos em grandes modelos de linguagem dispara devido ao cache KV, que cresce linearmente com o número de tokens e o tamanho do lote. Por exemplo, o...","url":"https://www.aioga.com/pt-BR/news/cmsevj4zx18b3ro2eed5gmzqj/","contentTranslated":true,"sourceHash":"fca74342898045be","translatedAt":"2026-08-04T16:43:20.560Z"},"ru":{"title":"Почему память больших языковых моделей дорогая и как её оптимизировать","summary":"При обработке длинного контекста в больших языковых моделях затраты резко возрастают из-за KV-кэша, который растёт линейно с числом токенов и размером батча. Например, для Llama 3 70B контекст в 128 000 токенов требует около 40 ГБ видеопамяти GPU. Методы оптимизации включают группированное внимание (GQA) и сжатие кэша на токен на этапе обучения.","category":"技巧观点","source":"ByteByteGo（RSS）","aggregationSource":"ByteByteGo（RSS）","pageTitle":"Почему память больших языковых моделей дорогая и как её оптимизировать - Aioga Новости ИИ","description":"При обработке длинного контекста в больших языковых моделях затраты резко возрастают из-за KV-кэша, который растёт линейно с числом токенов и размером батча. Например, для Llama 3...","url":"https://www.aioga.com/ru/news/cmsevj4zx18b3ro2eed5gmzqj/","contentTranslated":true,"sourceHash":"fca74342898045be","translatedAt":"2026-08-04T16:43:21.545Z"},"ar":{"title":"لماذا تعتبر ذاكرة النماذج اللغوية الكبيرة مكلفة وكيفية تحسينها","summary":"ترتفع تكلفة النماذج اللغوية الكبيرة عند معالجة سياقات طويلة، ويعود السبب إلى ذاكرة KV cache، التي تنمو بشكل خطي مع عدد الرموز وحجم الدفعة. على سبيل المثال، نموذج Llama 3 70B يحتاج تقريباً إلى 40GB من ذاكرة GPU لسياق يحتوي على 128،000 رمز. تشمل أساليب التحسين تقسيم انتباه الاستعلامات (GQA) وغيرها، وضغط استخدام الذاكرة لكل رمز خلال مرحلة التدريب.","category":"技巧观点","source":"ByteByteGo（RSS）","aggregationSource":"ByteByteGo（RSS）","pageTitle":"لماذا تعتبر ذاكرة النماذج اللغوية الكبيرة مكلفة وكيفية تحسينها - Aioga أخبار الذكاء الاصطناعي","description":"ترتفع تكلفة النماذج اللغوية الكبيرة عند معالجة سياقات طويلة، ويعود السبب إلى ذاكرة KV cache، التي تنمو بشكل خطي مع عدد الرموز وحجم الدفعة. على سبيل المثال، نموذج Llama 3 70B يحتاج...","url":"https://www.aioga.com/ar/news/cmsevj4zx18b3ro2eed5gmzqj/","contentTranslated":true,"sourceHash":"fca74342898045be","translatedAt":"2026-08-04T16:43:27.479Z"},"hi":{"title":"बड़े भाषा मॉडल की मेमोरी महंगी क्यों है और इसे कैसे अनुकूलित करें","summary":"बड़े भाषा मॉडल लंबा संदर्भ (long context) संसाधित करते समय लागत अचानक बढ़ती है, इसका मुख्य कारण KV cache है, जो token संख्या और बैच आकार के साथ रैखिक रूप से बढ़ता है। उदाहरण के लिए, Llama 3 70B में, 128,000 token संदर्भ लगभग 40GB GPU मेमोरी की आवश्यकता होती है। अनुकूलन के तरीके में ग्रुपेड क्वेरी अटेंशन (GQA) आदि शामिल हैं, जो प्रशिक्षण चरण में प्रत्येक token के कैश उपयोग को संकुचित करते हैं।","category":"技巧观点","source":"ByteByteGo（RSS）","aggregationSource":"ByteByteGo（RSS）","pageTitle":"बड़े भाषा मॉडल की मेमोरी महंगी क्यों है और इसे कैसे अनुकूलित करें - Aioga AI समाचार","description":"बड़े भाषा मॉडल लंबा संदर्भ (long context) संसाधित करते समय लागत अचानक बढ़ती है, इसका मुख्य कारण KV cache है, जो token संख्या और बैच आकार के साथ रैखिक रूप से बढ़ता है। उदाहरण के लिए...","url":"https://www.aioga.com/hi/news/cmsevj4zx18b3ro2eed5gmzqj/","contentTranslated":true,"sourceHash":"fca74342898045be","translatedAt":"2026-08-04T16:43:27.435Z"},"it":{"title":"Perché la memoria dei grandi modelli linguistici è costosa e come ottimizzarla","summary":"Quando i grandi modelli linguistici elaborano contesti lunghi, i costi aumentano vertiginosamente, principalmente a causa della cache KV, che cresce linearmente con il numero di token e la dimensione dei batch. Per esempio, Llama 3 70B richiede circa 40 GB di memoria GPU per un contesto di 128.000 token. Le strategie di ottimizzazione includono meccanismi come Grouped Query Attention (GQA), che comprimono l’occupazione della cache per token durante la fase di addestramento.","category":"技巧观点","source":"ByteByteGo（RSS）","aggregationSource":"ByteByteGo（RSS）","pageTitle":"Perché la memoria dei grandi modelli linguistici è costosa e come ottimizzarla - Aioga Notizie IA","description":"Quando i grandi modelli linguistici elaborano contesti lunghi, i costi aumentano vertiginosamente, principalmente a causa della cache KV, che cresce linearmente con il numero di to...","url":"https://www.aioga.com/it/news/cmsevj4zx18b3ro2eed5gmzqj/","contentTranslated":true,"sourceHash":"fca74342898045be","translatedAt":"2026-08-04T16:43:34.080Z"},"nl":{"title":"Waarom geheugen voor grote taalmodellen duur is en hoe het te optimaliseren","summary":"De kosten van grote taalmodellen rijzen de pan uit bij het verwerken van lange contexten, wat komt door de KV-cache, die lineair toeneemt met het aantal tokens en batchgrootte. Bijvoorbeeld, Llama 3 70B heeft voor een context van 128.000 tokens ongeveer 40GB GPU-geheugen nodig. Optimalisatietechnieken omvatten Grouped Query Attention (GQA) en compressie van het geheugen per token tijdens de trainingsfase.","category":"技巧观点","source":"ByteByteGo（RSS）","aggregationSource":"ByteByteGo（RSS）","pageTitle":"Waarom geheugen voor grote taalmodellen duur is en hoe het te optimaliseren - Aioga AI-nieuws","description":"De kosten van grote taalmodellen rijzen de pan uit bij het verwerken van lange contexten, wat komt door de KV-cache, die lineair toeneemt met het aantal tokens en batchgrootte. Bij...","url":"https://www.aioga.com/nl/news/cmsevj4zx18b3ro2eed5gmzqj/","contentTranslated":true,"sourceHash":"fca74342898045be","translatedAt":"2026-08-04T16:43:32.909Z"},"tr":{"title":"Büyük dil modellerinde hafızanın neden maliyetli olduğu ve nasıl optimize edileceği","summary":"Büyük dil modelleri uzun bağlamları işlerken maliyet hızla yükselir, bunun nedeni KV önbelleğidir; token sayısı ve batch boyutuna lineer olarak büyür. Örneğin Llama 3 70B modeli için 128.000 token bağlam yaklaşık 40GB GPU belleği gerektirir. Optimizasyon yöntemleri arasında grup sorgu dikkat (GQA) gibi yaklaşımlar ve eğitim sırasında her token’in önbellek kullanımını sıkıştırmak yer alır.","category":"技巧观点","source":"ByteByteGo（RSS）","aggregationSource":"ByteByteGo（RSS）","pageTitle":"Büyük dil modellerinde hafızanın neden maliyetli olduğu ve nasıl optimize edileceği - Aioga AI Haberleri","description":"Büyük dil modelleri uzun bağlamları işlerken maliyet hızla yükselir, bunun nedeni KV önbelleğidir; token sayısı ve batch boyutuna lineer olarak büyür. Örneğin Llama 3 70B modeli iç...","url":"https://www.aioga.com/tr/news/cmsevj4zx18b3ro2eed5gmzqj/","contentTranslated":true,"sourceHash":"fca74342898045be","translatedAt":"2026-08-04T16:43:39.034Z"},"vi":{"title":"Tại sao bộ nhớ của các mô hình ngôn ngữ lớn lại đắt đỏ và cách tối ưu hóa","summary":"Khi xử lý ngữ cảnh dài, chi phí của các mô hình ngôn ngữ lớn tăng vọt, nguyên nhân nằm ở KV cache, nó tăng theo số token và quy mô batch. Ví dụ với Llama 3 70B, ngữ cảnh 128.000 token cần khoảng 40GB GPU. Các phương pháp tối ưu bao gồm Grouped Query Attention (GQA) và trong giai đoạn huấn luyện nén bộ nhớ cache cho từng token.","category":"技巧观点","source":"ByteByteGo（RSS）","aggregationSource":"ByteByteGo（RSS）","pageTitle":"Tại sao bộ nhớ của các mô hình ngôn ngữ lớn lại đắt đỏ và cách tối ưu hóa - Tin tức AI Aioga","description":"Khi xử lý ngữ cảnh dài, chi phí của các mô hình ngôn ngữ lớn tăng vọt, nguyên nhân nằm ở KV cache, nó tăng theo số token và quy mô batch. Ví dụ với Llama 3 70B, ngữ cảnh 128.000 to...","url":"https://www.aioga.com/vi/news/cmsevj4zx18b3ro2eed5gmzqj/","contentTranslated":true,"sourceHash":"fca74342898045be","translatedAt":"2026-08-04T16:43:40.673Z"},"id":{"title":"Mengapa Memori Model Bahasa Besar Mahal dan Bagaimana Mengoptimalkannya","summary":"Biaya model bahasa besar melonjak saat menangani konteks panjang, akar masalahnya adalah cache KV, yang meningkat secara linear sesuai jumlah token dan ukuran batch. Sebagai contoh Llama 3 70B, konteks 128.000 token membutuhkan sekitar 40GB memori GPU. Metode optimasi termasuk Grouped Query Attention (GQA) dan kompresi penggunaan cache per token pada tahap pelatihan.","category":"技巧观点","source":"ByteByteGo（RSS）","aggregationSource":"ByteByteGo（RSS）","pageTitle":"Mengapa Memori Model Bahasa Besar Mahal dan Bagaimana Mengoptimalkannya - Berita AI Aioga","description":"Biaya model bahasa besar melonjak saat menangani konteks panjang, akar masalahnya adalah cache KV, yang meningkat secara linear sesuai jumlah token dan ukuran batch. Sebagai contoh...","url":"https://www.aioga.com/id/news/cmsevj4zx18b3ro2eed5gmzqj/","contentTranslated":true,"sourceHash":"fca74342898045be","translatedAt":"2026-08-04T16:43:45.991Z"},"th":{"title":"ทำไมหน่วยความจำของโมเดลภาษาใหญ่ถึงแพงและวิธีการปรับแต่ง","summary":"ค่าใช้จ่ายของโมเดลภาษาใหญ่จะพุ่งสูงเมื่อจัดการบริบทยาว สาเหตุหลักมาจาก KV cache ซึ่งเติบโตตามจำนวนโทเค็นและขนาดการประมวลผลแบบกลุ่ม ตัวอย่างเช่น Llama 3 70B ต้องใช้หน่วยความจำ GPU ประมาณ 40GB สำหรับบริบท 128,000 โทเค็น วิธีการปรับแต่งรวมถึงการใช้ Grouped Query Attention (GQA) และการบีบอัด cache ต่อโทเค็นในระหว่างการฝึกฝน","category":"技巧观点","source":"ByteByteGo（RSS）","aggregationSource":"ByteByteGo（RSS）","pageTitle":"ทำไมหน่วยความจำของโมเดลภาษาใหญ่ถึงแพงและวิธีการปรับแต่ง - ข่าว AI Aioga","description":"ค่าใช้จ่ายของโมเดลภาษาใหญ่จะพุ่งสูงเมื่อจัดการบริบทยาว สาเหตุหลักมาจาก KV cache ซึ่งเติบโตตามจำนวนโทเค็นและขนาดการประมวลผลแบบกลุ่ม ตัวอย่างเช่น Llama 3 70B ต้องใช้หน่วยความจำ GPU ป...","url":"https://www.aioga.com/th/news/cmsevj4zx18b3ro2eed5gmzqj/","contentTranslated":true,"sourceHash":"fca74342898045be","translatedAt":"2026-08-04T16:43:48.512Z"},"pl":{"title":"Dlaczego pamięć dużych modeli językowych jest kosztowna i jak ją optymalizować","summary":"Koszty przetwarzania długich kontekstów w dużych modelach językowych gwałtownie rosną, a przyczyną jest pamięć KV cache, która rośnie liniowo wraz z liczbą tokenów i rozmiarem partii. Na przykład w Llama 3 70B kontekst 128 000 tokenów wymaga około 40 GB pamięci GPU. Metody optymalizacji obejmują grupowane zapytania uwagi (GQA) itp., a podczas treningu zmniejszenie pamięci zajmowanej przez każdy token w cache.","category":"技巧观点","source":"ByteByteGo（RSS）","aggregationSource":"ByteByteGo（RSS）","pageTitle":"Dlaczego pamięć dużych modeli językowych jest kosztowna i jak ją optymalizować - Aioga Wiadomości AI","description":"Koszty przetwarzania długich kontekstów w dużych modelach językowych gwałtownie rosną, a przyczyną jest pamięć KV cache, która rośnie liniowo wraz z liczbą tokenów i rozmiarem part...","url":"https://www.aioga.com/pl/news/cmsevj4zx18b3ro2eed5gmzqj/","contentTranslated":true,"sourceHash":"fca74342898045be","translatedAt":"2026-08-04T16:43:55.724Z"}}}}