{"@context":"https://schema.org","@type":"NewsArticle","generatedAt":"2026-11-02T02:00:36.453Z","headline":"Modal 详解如何以万亿 token 规模服务 Kimi K2.6 编码 Agent 推理","description":"Modal 分享为编码 Agent 提供万亿参数模型 Kimi K2.6 推理服务的优化实践，优化后单副本每用户性能提升 2.8x、副本整体吞吐提升 5.6x，单个服务日处理数千亿 token。","url":"https://www.aioga.com/news/cmueg56ob04iqrovxup3jp0ly/","mainEntityOfPage":"https://www.aioga.com/news/cmueg56ob04iqrovxup3jp0ly/","datePublished":"2026-09-23T00:00:00.000Z","dateModified":"2026-09-23T00:00:00.000Z","inLanguage":"zh-CN","publisher":{"@type":"NewsMediaOrganization","name":"Aioga","url":"https://www.aioga.com"},"citation":["https://modal.com/blog/trillion-tokens-trillion-parameters","https://aihot.news/items/cmueg56ob04iqrovxup3jp0ly"],"canonicalUrl":"https://www.aioga.com/news/cmueg56ob04iqrovxup3jp0ly/","directAnswer":{"@type":"Answer","text":"Modal 官方工程博客介绍了为 Moonshot AI 的 Kimi K2.6 编码 Agent 提供推理服务的优化实践。文章称，优化后单副本每用户性能提升 2.8 倍、单副本跨用户整体吞吐提升 5.6 倍。","url":"https://www.aioga.com/news/cmueg56ob04iqrovxup3jp0ly/","dateCreated":"2026-09-23T00:00:00.000Z","author":{"@type":"Organization","@id":"https://www.aioga.com/authors/aioga-editorial/#editorial-team","name":"Aioga Editorial Team","url":"https://www.aioga.com/authors/aioga-editorial/"}},"evidence":[{"@type":"CreativeWork","name":"modal.com source article","url":"https://modal.com/blog/trillion-tokens-trillion-parameters","datePublished":"2026-09-23T00:00:00.000Z","provider":{"@type":"Organization","name":"modal.com","url":"https://modal.com/blog/trillion-tokens-trillion-parameters"}},{"@type":"CreativeWork","name":"AIHot archive record","url":"https://aihot.news/items/cmueg56ob04iqrovxup3jp0ly","datePublished":"2026-09-23T00:00:00.000Z","provider":{"@type":"Organization","name":"AIHot","url":"https://aihot.news/items/cmueg56ob04iqrovxup3jp0ly"}}],"aggregationSource":"Modal 官方工程博客（RSS）","originalPublisher":{"name":"modal.com","url":"https://modal.com/blog/trillion-tokens-trillion-parameters"},"geoDeepAnswer":null,"article":{"id":"cmueg56ob04iqrovxup3jp0ly","slug":"cmueg56ob04iqrovxup3jp0ly","url":"https://www.aioga.com/news/cmueg56ob04iqrovxup3jp0ly/","title":"Modal 详解如何以万亿 token 规模服务 Kimi K2.6 编码 Agent 推理","title_en":"","summary":"Modal 分享为编码 Agent 提供万亿参数模型 Kimi K2.6 推理服务的优化实践，优化后单副本每用户性能提升 2.8x、副本整体吞吐提升 5.6x，单个服务日处理数千亿 token。","source":"Modal 官方工程博客（RSS）","sourceUrl":"https://modal.com/blog/trillion-tokens-trillion-parameters","aiHotUrl":"https://aihot.news/items/cmueg56ob04iqrovxup3jp0ly","publishedAt":"2026-09-23T00:00:00.000Z","category":"行业动态","score":72,"selected":true,"articleBody":["Software is eating the world ：https://a16z.com/why-software-is-eating-the-world/ , and agents are eating software engineering ：https://martinalderson.com/posts/ai-agents-are-starting-to-eat-saas/ . It is imperative that software engineers develop an understanding not just of the agent software that is now their most important tool but also of how the intelligent core of agent software works, through inference by large generative models of language — if not out of the engineer’s need to understand and control their tools, then at least because inference is poised to consume more computing power and produce more benefit than all other uses of computers.","The central fact about inference services for coding agents is that they must operate at extremely high relative and absolute performance.","By relative performance, we mean large fractions of the peak rate or “speed of light” ：https://modal.com/gpu-glossary/perf/peak-rate of the hardware that it uses. By absolute performance, we mean that the scale of that peak rate and the amount of work done per request is large. Contemporary matrix math accelerators like Tensor Cores ：https://modal.com/gpu-glossary/device-hardware/tensor-core operate at the petaFLOP per second scale. Large generative sequence models with sufficient intelligence to automate software development have trillions of floating point parameters, and each of them must be accessed many times per second, even when serving just a single request.","Due to these requirements, economically viable coding agent inference services are currently only feasible by operating at a scale sufficient to amortize hardware and engineering costs — roughly, at the scale of trillions of input and output tokens.","We’ve done this, and we’d like to share how.","At Modal, we operate a number of such inference services for coding agents at this scale and work with a number of customers who do the same. You can use our services indirectly via inference routing platforms like OpenRouter ：https://openrouter.ai/provider/modal or Vercel AI Gateway ：https://vercel.com/ai-gateway/models/providers/modal or directly through our Shared Endpoints ：https://modal.com/endpoints/ .","In this blog post, we will walk through how we optimized inference performance when serving inferences from Moonshot AI’s Kimi K2.6 ：https://huggingface.co/moonshotai/Kimi-K2.6 model to power coding agents. Though this model is “old” by this field’s standards (literally hundreds of days old!), the fundamentals of sequence modeling, hardware, and scaling change slowly enough that the core story and many of the details match what we have done for more recent models that have superseded K2.6 in intelligence and cost-performance, like Kimi K3 ：https://modal.com/library/moonshot/kimi-k3 .","Our optimizations allowed us to scale per-replica performance of inference replicas by 2.8x per user and 5.6x across users on the replica:","This chart relates the individual user’s experience (decode tokens per second per user, aka interactivity ) on the x-axis with the cost-performance of the overall system on the y-axis (total tokens per minute per GPU, aka token throughput), with the number of concurrent users indicated at each point.","More intuitively, that’s the difference between a ruinously expensive service with the UX on the right below and a price-competitive service with the UX on the left:","We then scaled those single-container replicas into deployments and services. One particular service processed hundreds of billions of tokens a day and trillions in aggregate:","Below, we aim to make this performance engineering legible to a general software engineering audience. By sharing how we, and our customers, are able to operate these services, we hope it enables you to do the same — perhaps by deploying a Dedicated Endpoint ：https://modal.com/docs/guide/endpoints on Modal.","We break this down into two sections: understanding the sequence model that infers the response to each request and understanding workload structure across requests.","Contemporary coding agents are powered by probabilistic generative models of unicode sequences pre-trained mainly via unsupervised masked sequence prediction and post-trained mainly by reinforcement of output software correctness. Like the parser of a compiler, they operate not on raw strings but on tokenized sequences, so we call their inputs and outputs tokens . Because we are, in the end, guessing what output tokens should be, this is called inference . If you prefer deduction, stick to databases and operating systems.","The underlying sequence models these days are hybrid-attention, mixture-of-experts Transformer neural networks . These networks apply computations both per token in the sequence and across tokens in the sequence .","Attention has evolved into a generic term for cross-token computation. Mixture-of-experts refers to the dynamically routed block-sparse matrix multiplication that applies the majority of the per-token computation. These computations iteratively update the network’s internal, or latent , representation.","For insight into what’s going on inside these sequence models, see Anthropic’s “A Mathematical Framework for Transformer Circuits” ：https://transformer-circuits.pub/2021/framework/index.html (2021, but still undefeated).","A single forward pass through such a neural network produces both substantial internal state and a probability distribution over the next token(s) in the sequence for each sequence position. Because we predict ( ”regress” ：https://en.wikipedia.org/wiki/Regression_toward_the_mean ) based on our own outputs (”auto”), this is autoregressive sequence modeling.","To respond to a client request, we generally chain multiple forward passes together like this:","Forward passes are expensive, so we want to amortize this work as much as possible. Much of the work in per-token computation amortizes by batching several sequences together. Much of the work in cross-token computation amortizes by caching the internal state. For historical reasons, this is called the key-value cache (KV cache or just KV ), even though contemporary models like Kimi don’t have distinct keys and values. You can read more about the “napkin math” here in Kipply’s excellent “Transformer Inference Arithmetic” blogpost ：https://kipp.ly/p/transformer-inference-arithmetic (2022, but still undefeated).","When a forward pass processes a request’s input tokens, we call it a prefill , because it is “prefilling” the KV cache. When a forward pass produces a response’s output tokens, we call it a decode , because we are “decoding” the model’s “encoding” of past state into predicted future. What about forward passes that do both? Yeah, we don’t like the terminology either.","Prefill performance is mostly tracked by the latency to complete all prefills for a request, aka time-to-first-token (TTFT). Decode performance is mostly measured by the rate at which output tokens are produced after that, aka output tokens per second (TPS). Both can be measured client-side or server-side, causing no end of confusion.","The particular sequence model covered in this post is Kimi K2.6 ：https://huggingface.co/moonshotai/Kimi-K2.6 by Moonshot AI. This model parametrizes its matrix multiplications with approximately one trillion numbers ( weights in its matrices), the majority of which are stored as four bit integers (INT4).","We serve the model, however, with four bit floating point numbers (FP4) ：https://modal.com/llm-almanac/quant-formats/0x6 . Four bits only gives you sixteen distinct values, so you further need a micro-scaling format ：https://modal.com/llm-almanac/block-quants/mx-fp8-e4m3 to scale individual blocks within tensors independently. We chose the NVFP4 micro-scaling format ：https://modal.com/llm-almanac/block-quants/nvidia-fp4 , which has native hardware support at the petaFLOP/s scale in the Tensor Cores ：https://modal.com/gpu-glossary/device-hardware/tensor-core of Blackwell Streaming Multiprocessor Architecture ：https://modal.com/gpu-glossary/device-hardware/streaming-multiprocessor-architecture GPUs like the B200 and B300. Because we operate a dynamic GPU fleet in a time of constrained compute supply, we prepare our deployment to run on both B200 and B300 GPUs. Results below are all for B200 GPUs; B300s are substantively similar but operate at higher request concurrency because they have more high-bandwidth memory (HBM) ：https://modal.com/gpu-glossary/device-hardware/gpu-ram available for caching.","We chose the SGLang inference engine as our base. We found several opportunities to improve performance by patching the engine. As contributors to the SGLang project, we upstreamed these patches, described and linked in the post below.","When you serve such models on coding agent traffic naïvely, you get bad results.","This chart indicates that throughput and interactivity rapidly collapse above 6 concurrent users. Furthermore, even before that peak, the interactivity is below user expectations and the system is below acceptable efficiency.","So from here, you need to increase interactivity and throughput to deliver better outcomes to users while decreasing your own costs. To do that, you need to understand the sequences in this workload deeper than just “tokens in and tokens out”.","Individual requests for output tokens are created in “sessions”: the user, the generative model, and the tool calls chain together iteratively to construct a tower of input sequences, accumulating context — and value — over time. The iterative process of meaning construction, information discovery, and sense-making strikes us as fundamental to the nature of sequence modeling and sequential action, so we expect this pattern to far outlast “coding agents”.","Concretely, a single session looks something like this:","That is, the input sequence (green) for each turn T is the entire session history up to T (darker green), plus something new (lighter green). This has two key consequences.","First, it means requests inherently have long input sequences relative to their output sequences (pink, above) — there are T-1 past output sequences in the input to turn T, and T is in the dozens. For the core workload we used in optimization and served in production, this ratio was 200:1; requests contain roughly 100k input tokens and produce roughly 500 output tokens. That means the majority of processed tokens will be input tokens (just check the token usage numbers in your coding agent software).","Second, it means the input sequences have high overlap with previously processed input sequences — the ones from turns 1 to T-1. That means that on the way to serving turn T, the tokens in turn 1 are processed T times. This makes caching absolutely critical — we can avoid linearly-scaling recomputation to save effort, but we introduce linearly-scaling state that must be managed and has its own performance characteristics. Navigating this tradeoff is the core engineering problem we’ll tackle in this post.","With this picture of the workload in mind, we turn to optimization.","To optimize performance ：https://modal.com/gpu-glossary/perf/performance-bottleneck , build a working system, identify the bottleneck, then lift it. Repeat as needed until you’ve won.","Though our ultimate goal was to optimize an entire service, we decomposed that problem into two simpler problems: optimize a single replica first, then scale from one to many replicas.","We further split the problem of single replica performance into two sub-problems: first maximize interactivity, then maximize throughput without losing interactivity.","Interactivity primarily impacts request latency. Request latency and throughput interact through concurrency , the number of in-flight requests, by a rearrangement of Little’s Law ：https://modal.com/gpu-glossary/perf/littles-law :","Our key bottlenecks for latency, concurrency, and throughput started in the GPU HBM ：https://modal.com/gpu-glossary/device-hardware/gpu-ram .","Our key bottleneck on latency was HBM bandwidth ：https://modal.com/gpu-glossary/perf/memory-bandwidth during decode. We lifted it by parallelizing matrix multiplication across GPUs (tensor parallelism, TP) and by applying custom DFlash ：https://www.lmsys.org/blog/2026-06-15-next-generation-speculative-decoding-dflash-v2/ speculative decoding ：https://modal.com/blog/spec-is-all-u-need — doing more computation per memory load, even when that computation may not be needed.","That created a bottleneck on concurrency through HBM capacity : how much work can we keep in a cache that loads faster than we could just recompute results. We lifted it by clearing up intermediates in HBM, quantizing intermediates to lower floating point precision ：https://modal.com/llm-almanac/quant-formats/ , and extending the cache hierarchy to CPU RAM with HiCache ：https://www.lmsys.org/blog/2025-09-10-sglang-hicache/ . We used the cache hit rate (CHR) as a targeted metric of improvements to caching. CHRs between one and two 9s are very much feasible for most coding agent workloads.","Increasing interactivity increases the system performance as observed by individual users. We chose to work on this first. We made that choice for several reasons.","First and simplest, we found that coding agent users enjoy and will pay more for tokens that come to them faster, so high interactivity was key to building the service that our and our customers’ users wanted.","Second, interactivity is particularly amenable to improvement by speculative decoding ：https://modal.com/blog/achieve-sota-specdec . Because it is a simple, learning-based technique, its performance benefits scale with compute and data: machine learning’s famous “bitter lesson” ：http://www.incompleteideas.net/IncIdeas/BitterLesson.html , returning in performance engineering for ML systems. And we know how to scale training ：https://modal.com/blog/reinforcement-learning-infrastructure-problem .","This choice to interactivity-maxx had two additional benefits, one operational and the other for throughput, which were especially salient because we operate a dynamic, autoscaling fleet ：https://modal.com/blog/resource-solver of thousands of GPUs ：https://modal.com/blog/gpu-health .","Maximum interactivity replicas are smaller and therefore easier to serve.","Using multiple processors together requires an interconnection network (interconnect) for communication. The lowest latency, highest bandwidth interconnect for Nvidia GPUs is NVLink. NVLink operates across a group of processors in a “domain” of some size.","A single host operating system can support an NVLink domain of up to 8 GPUs. The largest NVLink domains that are generally available comprise 72 accelerators (in a multi-node IMEX domain ：https://docs.nvidia.com/multi-node-nvlink-systems/imex-guide/overview.html ). Using more accelerators would require a slower interconnect (IB/RoCE or, worse, standard Ethernet). That means that for maximum interactivity we should not expect to use more than 72 accelerators per replica — the communication overhead will almost surely dominate any per-request latency wins.","But that doesn’t mean we must use 72 accelerators.","Consider these results from SemiAnalysis’s InferenceX benchmarks for this same NVFP4 Kimi K2.6 model ：https://inferencex.semianalysis.com/inference/kimi-k26 , which show throughput per GPU as a function of interactivity, annotated with GPU count, for a variety of deployments:","The highest interactivity is achieved by a deployment with just eight GPUs per replica. Furthermore, that interactivity is achieved with comparable throughput per GPU, which means that by choosing a smaller domain, we are not obviously forgoing peak throughput cost-performance (subject to our interactivity constraint).","To keep the chart legible, we selected only a small subset of deployments most similar to ours, but the pattern holds across more accelerator types and across more models in the InferenceX benchmarks (explore them here ：https://inferencex.semianalysis.com/inference/kimi-k26 ). Generally, you can achieve the highest interactivity at comparable per-GPU throughput with only four or eight GPUs. You can then achieve the same aggregate throughput by scaling smaller replicas. The core Modal serverless platform ：https://modal.com/blog/truly-serverless-gpus makes this scaling performant and reliable.","This is a huge operational win. Smaller, simpler units make for easier scaling. Eight GPUs can be driven by a single host OS kernel. An NVL72 domain, on the other hand, is comprised of nine such subsystems sharing an address space (yes, you should be shuddering). Availability is constrained and contracts are long and inflexible.","Eight-GPU Blackwell systems, on the other hand, are standard enough to be available via on-demand and spot markets, which makes it much more cost-effective to handle variable load ：https://modal.com/blog/how-to-price-serverless . Replicas with one, two, or four GPUs can furthermore be packed inside of a single physical eight-GPU machine — which already has all the resources required to start another replica (model weights, JIT artifacts).","Of course, as and if the compute supply and user demands change, we will happily revisit this choice.","Increasing interactivity indirectly increases throughput.","By reducing the latency of individual requests, we indirectly improve throughput by freeing up resources for new requests.","Agentic coding workloads are approximately “closed-loop” per session. Sessions are almost always chains — of user-written tokens, of tool call responses, and of model outputs. The next request in the session, therefore, almost always arrives some time after the previous response has finished generating. The session’s next request is therefore latent for some time, outside the inference system — for tool calls, 10s of ms to seconds with a tail of minutes; for user responses, seconds to minutes, with a tail of hours or more.","During that time, other requests can be processed on the same node. When you have sufficient load for the active capacity, there are always requests ready for a node to process. When you have sufficient capacity for the active load, there are always nodes to map requests onto. Both of these are guaranteed by our fast autoscaling system ：https://modal.com/blog/truly-serverless-gpus . We’ll talk more about request routing in the section on scaling to multiple replicas.","Interactivity measures output tokens per second per user. Naïvely, autoregressive sequence models like Transformers produce these tokens sequentially. Amdahl’s heartbreaking Law strikes again."],"articleImages":[{"sourceUrl":"https://modal-cdn.com/charles-frye.jpg","alt":"User avatar","afterParagraph":0,"url":"/media/articles/cmueg56ob04iqrovxup3jp0ly/cfee2cd323c3f31e.jpg"},{"sourceUrl":"https://modal-cdn.com/timothy-feng.jpg","alt":"User avatar","afterParagraph":0,"url":"/media/articles/cmueg56ob04iqrovxup3jp0ly/be9be6c97fa05b1f.jpg"},{"sourceUrl":"https://modal-cdn.com/blog/images/trillion-tokens-trillion-parameters-throughput-interactivity-improvement.webp","alt":"","afterParagraph":7,"url":"/media/articles/cmueg56ob04iqrovxup3jp0ly/07815afe1ff2daf0.webp"}],"mediaStatus":"ok","articleBodyZh":["软件正在吞噬世界：https://a16z.com/why-software-is-eating-the-world/，而智能代理正在吞噬软件工程：https://martinalderson.com/posts/ai-agents-are-starting-to-eat-saas/。软件工程师必须不仅理解目前作为其最重要工具的代理软件，还要理解代理软件的智能核心是如何工作的——通过大型生成语言模型进行推理——如果不是出于工程师需要理解和控制其工具的目的，至少也应当意识到推理有可能消耗比计算机的所有其他用途更多的计算能力，并产生更多的收益。","关于用于编程智能体的推理服务的核心事实是，它们必须在极高的相对和绝对性能下运行。","我们所说的相对性能，是指其所使用硬件的峰值速率或“光速” 的很大比例：https://modal.com/gpu-glossary/perf/peak-rate。绝对性能则是指该峰值速率的规模和每次请求完成的工作量都很大。像 Tensor Core 这样的现代矩阵计算加速器：https://modal.com/gpu-glossary/device-hardware/tensor-core，运行在每秒千万亿次浮点运算（petaFLOP/s）的规模。具有足够智能以自动化软件开发的大型生成序列模型拥有数万亿个浮点参数，而且每次请求即便只处理一个请求，这些参数也必须每秒被访问多次。","由于这些要求，经济上可行的编程智能体推理服务目前只有在达到足够覆盖硬件和工程成本的规模时才可实现——大致上，需要处理数万亿的输入和输出 token 的规模。","我们已经实现了这一点，并希望分享我们的经验。","在 Modal，我们运营着多个这样的编程智能体推理服务，并与多位客户合作，也提供类似服务。您可以通过推理路由平台间接使用我们的服务，如 OpenRouter：https://openrouter.ai/provider/modal 或 Vercel AI Gateway：https://vercel.com/ai-gateway/models/providers/modal，或者直接通过我们的共享终端：https://modal.com/endpoints/。","在这篇博客文章中，我们将介绍如何在从 Moonshot AI 的 Kimi K2.6 模型（链接：https://huggingface.co/moonshotai/Kimi-K2.6）为编码代理提供推理服务时优化推理性能。虽然按照该领域的标准，这个模型是“旧的”（字面意思是已经有数百天历史了！），但序列建模、硬件和扩展的基本原理变化足够慢，因此核心内容和许多细节与我们为更近期、在智能和性价比上超越 K2.6 的模型（如 Kimi K3：https://modal.com/library/moonshot/kimi-k3）所做的工作相符。","我们的优化使得每个推理副本的单用户性能提升了 2.8 倍，而跨用户在副本上的性能提升了 5.6 倍：","此图将每个用户的体验（每用户每秒解码的 token，即交互性）绘制在 x 轴上，而整体系统的性价比（每 GPU 每分钟总 token，即 token 吞吐量）绘制在 y 轴上，并在每个点标出并发用户数。","更直观地说，这就是下图中右侧的高昂代价服务与左侧价格竞争力服务之间的差别：","随后，我们将这些单容器副本扩展成部署和服务。其中一个特定服务每天处理数千亿的 token，总量达到数万亿：","下面，我们希望让一般软件工程师能够理解这种性能工程。通过分享我们以及客户如何运维这些服务，我们希望它也能让你做到同样的事情——也许可以通过在 Modal 上部署专用端点（链接：https://modal.com/docs/guide/endpoints）来实现。","我们将内容分为两个部分：理解用于响应每个请求的序列模型，以及理解跨请求的工作负载结构。","当代的编码代理由概率生成模型驱动，这些模型主要通过无监督的掩码序列预测进行预训练，并主要通过增强输出软件正确性进行后训练。就像编译器的解析器一样，它们并不在原始字符串上操作，而是在标记化的序列上操作，因此我们将它们的输入和输出称为标记。因为我们最终是在猜测输出标记应该是什么，所以这被称为推理。如果你更喜欢演绎，请坚持使用数据库和操作系统。","如今的基础序列模型是混合注意力、专家混合的 Transformer 神经网络。这些网络不仅在序列中的每个标记上进行计算，也在序列中的标记之间进行计算。","注意力已经演变为跨标记计算的通用术语。专家混合是指动态路由的块稀疏矩阵乘法，它应用于大部分的每标记计算。这些计算迭代地更新网络的内部或潜在表示。","想了解这些序列模型内部发生了什么，可以参考 Anthropic 的“Transformer 电路的数学框架”：https://transformer-circuits.pub/2021/framework/index.html（2021 年，但仍未被超越）。","通过这样的神经网络进行一次前向传递，会生成大量的内部状态，以及针对序列中每个位置的下一个标记的概率分布。因为我们是基于自己的输出进行预测（“自回归”）：https://en.wikipedia.org/wiki/Regression_toward_the_mean，这是自回归序列建模。","为了响应客户端请求，我们通常像这样将多个前向传递串联在一起：","前向传递非常昂贵，因此我们希望尽可能摊销这部分工作。在每个 token 的计算中，大部分工作可以通过将多个序列一起批处理来摊销。在跨 token 的计算中，大部分工作可以通过缓存内部状态来摊销。出于历史原因，这被称为键值缓存（KV 缓存或简称 KV），尽管像 Kimi 这样的现代模型并没有明确的键和值。你可以在 Kipply 的优秀博客文章《Transformer推理算术》中了解更多关于“餐巾纸数学”的内容：https://kipp.ly/p/transformer-inference-arithmetic（2022 年的文章，但至今仍未被超越）。","当前向传递处理请求的输入 token 时，我们称之为预填充（prefill），因为它是在“预填充” KV 缓存。当前向传递生成响应的输出 token 时，我们称之为解码（decode），因为我们正在将模型对过去状态的“编码”解码成预测的未来。那么既做预填充又做解码的前向传递呢？嗯，我们也不太喜欢这个术语。","预填充性能主要通过完成一个请求的所有预填充所需的延迟来跟踪，也就是首次 token 时间（TTFT）。解码性能主要通过之后输出 token 的生成速率来衡量，也就是每秒输出 token 数（TPS）。这两者都可以在客户端或服务器端测量，这常常会引起混淆。","本文介绍的特定序列模型是 Kimi K2.6：https://huggingface.co/moonshotai/Kimi-K2.6，由 Moonshot AI 提供。该模型使用大约一万亿个数字（矩阵中的权重）来参数化其矩阵乘法，其中大多数以四位整数（INT4）存储。","然而，我们使用四位浮点数（FP4）来服务模型：https://modal.com/llm-almanac/quant-formats/0x6。四位仅提供十六个不同的值，因此还需要微缩放格式：https://modal.com/llm-almanac/block-quants/mx-fp8-e4m3 来独立缩放张量中的各个块。我们选择了 NVFP4 微缩放格式：https://modal.com/llm-almanac/block-quants/nvidia-fp4，它在 Blackwell 流式多处理器架构的张量核心中具有原生硬件支持，可达到 petaFLOP/s 规模：https://modal.com/gpu-glossary/device-hardware/tensor-core 和 https://modal.com/gpu-glossary/device-hardware/streaming-multiprocessor-architecture，如 B200 和 B300 GPU。由于我们在计算资源紧张时期操作动态 GPU 群组，因此我们准备让部署同时在 B200 和 B300 GPU 上运行。以下结果都是针对 B200 GPU；B300 本质上类似，但由于它们有更多高带宽内存（HBM）可用于缓存，能够实现更高的请求并发：https://modal.com/gpu-glossary/device-hardware/gpu-ram。","我们选择 SGLang 推理引擎作为基础。我们发现通过修补引擎有几个提高性能的机会。作为 SGLang 项目的贡献者，我们将这些补丁上游合并，具体描述和链接见以下文章。","当你天真地在编程代理流量上提供此类模型时，结果会很糟糕。","此图表显示，超过 6 个并发用户时，吞吐量和交互性迅速下降。此外，即使在达到峰值之前，交互性也低于用户预期，系统效率低于可接受水平。","因此，从此开始，你需要提高交互性和吞吐量，以向用户提供更好的结果，同时降低自身成本。为此，你需要比仅仅“输入和输出词元”更深入地理解此工作负载中的序列。","单个输出令牌的请求是在“会话”中创建的：用户、生成模型和工具迭代性地调用链在一起，以构建输入序列的塔，同时随着时间的推移积累上下文——以及价值。意义建构、信息发现和理解过程的迭代性，让我们认为这是序列建模和顺序行动本质的基本特征，因此我们预计这种模式将远远超越“编码代理”。","具体来说，单个会话看起来大致如下：","也就是说，每轮 T 的输入序列（绿色）是到 T 为止的整个会话历史（深绿色），加上一些新的内容（浅绿色）。这有两个关键的后果。","首先，这意味着相对于输出序列（上述粉色），请求本质上具有较长的输入序列——在 T 的输入中包含了 T-1 轮的过去输出序列，并且 T 是几十轮。对于我们在优化中使用并在生产中提供的核心工作负载而言，这个比例是 200:1；请求大约包含 10 万个输入令牌，并产生大约 500 个输出令牌。这意味着处理的大部分令牌都会是输入令牌（只需查看你的编码代理软件中的令牌使用数量即可）。","第二，这意味着输入序列与以前处理的输入序列——从第 1 轮到第 T-1 轮的序列——有高度重叠。这意味着在提供第 T 轮时，第 1 轮的令牌会被处理 T 次。这使得缓存变得至关重要——我们可以避免线性增长的重复计算以节省工作量，但同时会引入必须管理的线性增长状态，这种状态本身也有其性能特征。在这种权衡中导航，是我们在本文中要解决的核心工程问题。","了解了这种工作负载的情况后，我们转向优化。","为了优化性能：https://modal.com/gpu-glossary/perf/performance-bottleneck，构建一个可运行的系统，找出瓶颈，然后解决它。根据需要重复，直到你成功为止。","虽然我们的最终目标是优化整个服务，但我们将该问题分解为两个更简单的问题：先优化单个副本，再从一个副本扩展到多个副本。","我们进一步将单副本性能问题拆分为两个子问题：首先最大化交互性，然后在不损失交互性的情况下最大化吞吐量。","交互性主要影响请求延迟。请求延迟和吞吐量通过并发性（即进行中的请求数量）相互作用，可通过Little定律的重新排列来理解：https://modal.com/gpu-glossary/perf/littles-law","我们在延迟、并发性和吞吐量方面的关键瓶颈开始于GPU HBM：https://modal.com/gpu-glossary/device-hardware/gpu-ram","我们在延迟方面的关键瓶颈是在解码过程中HBM带宽：https://modal.com/gpu-glossary/perf/memory-bandwidth。我们通过跨GPU的矩阵乘法并行化（张量并行，TP）以及应用自定义DFlash：https://www.lmsys.org/blog/2026-06-15-next-generation-speculative-decoding-dflash-v2/ 投机解码：https://modal.com/blog/spec-is-all-u-need ——即在每次内存加载中做更多计算，即使这些计算可能不必要，从而提高了性能。","这在HBM容量上形成了并发性瓶颈：我们能在比重新计算结果更快加载的缓存中保留多少工作。我们通过清理HBM中的中间结果、将中间结果量化为较低浮点精度：https://modal.com/llm-almanac/quant-formats/，并通过HiCache扩展缓存层次结构到CPU RAM：https://www.lmsys.org/blog/2025-09-10-sglang-hicache/ 来解决这一问题。我们使用缓存命中率（CHR）作为改进缓存的目标指标。对于大多数编码代理工作负载，达到一到两个9的CHR是非常可行的。","提高交互性提升了用户感知的系统性能。我们选择先从这一点入手。我们做出这一选择有几个原因。","第一个也是最简单的原因是，我们发现编码代理用户喜欢并愿意为更快到达的token付费，因此高交互性是构建我们和客户用户所需服务的关键。","其次，互动性特别适合通过推测解码进行改进：https://modal.com/blog/achieve-sota-specdec。因为这是一种简单、基于学习的技术，其性能收益会随着计算能力和数据量的增加而扩展：机器学习著名的“苦涩教训”：http://www.incompleteideas.net/IncIdeas/BitterLesson.html，在 ML 系统的性能工程中再度显现。我们也知道如何扩展训练：https://modal.com/blog/reinforcement-learning-infrastructure-problem。","选择最大化互动性还有两个额外的好处，一个是操作性，另一个是吞吐量，这在我们运营一个动态、自适应扩展的数千 GPU 队列时尤其显著：https://modal.com/blog/resource-solver；https://modal.com/blog/gpu-health。","最大互动性副本更小，因此更容易提供服务。","将多个处理器一起使用需要一个用于通信的互连网络（interconnect）。Nvidia GPU 的最低延迟、最高带宽互连是 NVLink。NVLink 在一定规模的“域”内操作一组处理器。","单个主机操作系统可以支持最多 8 个 GPU 的 NVLink 域。通常可用的最大 NVLink 域包括 72 个加速器（在多节点 IMEX 域中：https://docs.nvidia.com/multi-node-nvlink-systems/imex-guide/overview.html ）。使用更多加速器将需要速度较慢的互连（IB/RoCE 或更慢的标准以太网）。这意味着为了最大化互动性，每个副本不应期望使用超过 72 个加速器——通信开销几乎肯定会超过每个请求的延迟提升。","但这并不意味着我们必须使用 72 个加速器。","考虑 SemiAnalysis 的 InferenceX 基准测试对同一 NVFP4 Kimi K2.6 模型的结果：https://inferencex.semianalysis.com/inference/kimi-k26，这些结果展示了多种部署中每个 GPU 的吞吐量随互动性的变化，并标注了 GPU 数量：","最高的交互性是通过每个副本仅部署八块 GPU 实现的。此外，这种交互性是在每块 GPU 拥有可比吞吐量的情况下实现的，这意味着通过选择较小的域，我们显然并没有放弃峰值吞吐量的性价比（在我们的交互性约束下）。","为了保持图表的可读性，我们只选择了与我们最相似的一小部分部署，但这种模式在更多加速器类型和 InferenceX 基准测试中的更多模型中同样适用（可在这里查看：https://inferencex.semianalysis.com/inference/kimi-k26）。通常，你只需四或八块 GPU 就可以在每块 GPU 拥有可比吞吐量的情况下实现最高交互性。然后，你可以通过扩展较小的副本来实现相同的总吞吐量。核心 Modal 无服务器平台（https://modal.com/blog/truly-serverless-gpus）使这种扩展在性能和可靠性方面都能得到保障。","这是巨大的运营优势。较小、更简单的单元更容易进行扩展。八块 GPU 可以由单个主机操作系统内核驱动。而 NVL72 域则由九个这样的子系统组成，共享一个地址空间（是的，你应该对此感到不寒而栗）。可用性受限，合同期限长且不灵活。","而八块 GPU 的 Blackwell 系统则足够标准化，可通过按需和竞价市场获得，这使处理可变负载的成本更具效益（https://modal.com/blog/how-to-price-serverless）。一、二或四块 GPU 的副本还可以打包在单个八块 GPU 的物理机器中——该机器已经拥有启动另一副本所需的所有资源（模型权重、JIT 工件）。","当然，如果计算供应和用户需求发生变化，我们将乐于重新审视这一选择。","增加交互性会间接提高吞吐量。","通过减少单个请求的延迟，我们通过释放资源给新请求，从而间接提高了吞吐量。","代理编码工作负载每个会话大约是“闭环”的。会话几乎总是链条——由用户编写的令牌、工具调用响应和模型输出组成。因此，会话中的下一个请求几乎总是在前一个响应生成完成后的一段时间才到达。因此，会话的下一个请求在一段时间内处于潜变量状态，处于推理系统之外——对于工具调用而言，是数十毫秒到数秒，尾部可能延迟数分钟；对于用户响应而言，是几秒到几分钟，尾部可能延迟数小时或更长时间。","在此期间，可以在同一节点上处理其他请求。当您有足够的负载来利用活动容量时，总有请求准备好供节点处理。当您有足够的容量来应对活动负载时，总有节点可以映射请求。这两者都由我们快速的自动扩展系统保证：https://modal.com/blog/truly-serverless-gpus。我们将在多副本扩展部分详细讨论请求路由。","交互性以每个用户每秒输出的令牌数量来衡量。简单来说，自回归序列模型如Transformer是顺序生成这些令牌的。阿姆达尔令人心碎的定律再次显现。"],"translationStatus":"translated","bodyOrigin":"source-page","editorial":{"summary":"Modal 官方工程博客介绍了为 Moonshot AI 的 Kimi K2.6 编码 Agent 提供推理服务的优化实践。文章称，优化后单副本每用户性能提升 2.8 倍、单副本跨用户整体吞吐提升 5.6 倍。","background":"文章指出，编码 Agent 推理服务需要较高的相对与绝对性能，并称经济可行的服务目前需达到足以摊销硬件和工程成本的规模，约为万亿级输入与输出 token。Modal 表示其运营此类服务，并介绍 Kimi K2.6 的优化。","viewpoint":"Aioga 判断：这篇文章的主要价值在于呈现推理服务的性能与规模约束，以及 Modal 对 Kimi K2.6 的优化结果；这些数据反映的是文中所述服务实践，不应直接视为其他模型或部署环境的表现。","implications":"可能影响：关注编码 Agent 推理服务时，需要区分单用户性能与跨用户吞吐指标；文章给出的提升数据不足以说明具体硬件配置、成本变化或其他模型上的效果，也不代表所有服务都能达到相同结果。","nextStep":"后续观察：可留意 Modal 是否进一步披露该优化实践的硬件配置、成本及测试条件，并关注文中提及的较新模型是否有对应的性能数据；在这些信息明确前，不宜外推 Kimi K2.6 的结果。","evidenceRefs":["title","summary","articleBody","source"],"status":"published","aiGenerated":true,"autoApproved":true,"generatedBy":"aioga-editorial:gpt-5.6-sol","reviewedBy":"aioga-editorial-review:gpt-5.6-sol","generatedAt":"2026-09-23T19:30:49.504Z","sourceHash":"649b7c7bea90a56b","review":{"approved":true,"groundedness":96,"clarity":91,"duplicationRisk":28,"blockingIssues":[],"notes":["“主要价值”属于编辑判断，已明确归入 viewpoint，未冒充来源事实。","“硬件配置、成本变化或测试条件”在摘录中未披露，作为信息边界和后续观察建议表述合理；可在完整原文核对后进一步确认。"]},"validation":{"passed":true,"mode":"ai-auto","revisions":0,"checks":["schema","length","source-attribution","editorial-labels","inference-boundary","low-source-overlap","no-html","independent-ai-review"]}},"tags":["行业动态","Modal 官方工程博客（RSS）"],"translations":{"zh-CN":{"title":"Modal 详解如何以万亿 token 规模服务 Kimi K2.6 编码 Agent 推理","summary":"Modal 分享为编码 Agent 提供万亿参数模型 Kimi K2.6 推理服务的优化实践，优化后单副本每用户性能提升 2.8x、副本整体吞吐提升 5.6x，单个服务日处理数千亿 token。","category":"行业动态","source":"modal.com","aggregationSource":"Modal 官方工程博客（RSS）","pageTitle":"Modal 详解如何以万亿 token 规模服务 Kimi K2.6 编码 Agent 推理 - Aioga AI资讯","description":"Modal 分享为编码 Agent 提供万亿参数模型 Kimi K2.6 推理服务的优化实践，优化后单副本每用户性能提升 2.8x、副本整体吞吐提升 5.6x，单个服务日处理数千亿 token。","url":"https://www.aioga.com/news/cmueg56ob04iqrovxup3jp0ly/","articleBody":["软件正在吞噬世界：https://a16z.com/why-software-is-eating-the-world/，而智能代理正在吞噬软件工程：https://martinalderson.com/posts/ai-agents-are-starting-to-eat-saas/。软件工程师必须不仅理解目前作为其最重要工具的代理软件，还要理解代理软件的智能核心是如何工作的——通过大型生成语言模型进行推理——如果不是出于工程师需要理解和控制其工具的目的，至少也应当意识到推理有可能消耗比计算机的所有其他用途更多的计算能力，并产生更多的收益。","关于用于编程智能体的推理服务的核心事实是，它们必须在极高的相对和绝对性能下运行。","我们所说的相对性能，是指其所使用硬件的峰值速率或“光速” 的很大比例：https://modal.com/gpu-glossary/perf/peak-rate。绝对性能则是指该峰值速率的规模和每次请求完成的工作量都很大。像 Tensor Core 这样的现代矩阵计算加速器：https://modal.com/gpu-glossary/device-hardware/tensor-core，运行在每秒千万亿次浮点运算（petaFLOP/s）的规模。具有足够智能以自动化软件开发的大型生成序列模型拥有数万亿个浮点参数，而且每次请求即便只处理一个请求，这些参数也必须每秒被访问多次。","由于这些要求，经济上可行的编程智能体推理服务目前只有在达到足够覆盖硬件和工程成本的规模时才可实现——大致上，需要处理数万亿的输入和输出 token 的规模。","我们已经实现了这一点，并希望分享我们的经验。","在 Modal，我们运营着多个这样的编程智能体推理服务，并与多位客户合作，也提供类似服务。您可以通过推理路由平台间接使用我们的服务，如 OpenRouter：https://openrouter.ai/provider/modal 或 Vercel AI Gateway：https://vercel.com/ai-gateway/models/providers/modal，或者直接通过我们的共享终端：https://modal.com/endpoints/。","在这篇博客文章中，我们将介绍如何在从 Moonshot AI 的 Kimi K2.6 模型（链接：https://huggingface.co/moonshotai/Kimi-K2.6）为编码代理提供推理服务时优化推理性能。虽然按照该领域的标准，这个模型是“旧的”（字面意思是已经有数百天历史了！），但序列建模、硬件和扩展的基本原理变化足够慢，因此核心内容和许多细节与我们为更近期、在智能和性价比上超越 K2.6 的模型（如 Kimi K3：https://modal.com/library/moonshot/kimi-k3）所做的工作相符。","我们的优化使得每个推理副本的单用户性能提升了 2.8 倍，而跨用户在副本上的性能提升了 5.6 倍：","此图将每个用户的体验（每用户每秒解码的 token，即交互性）绘制在 x 轴上，而整体系统的性价比（每 GPU 每分钟总 token，即 token 吞吐量）绘制在 y 轴上，并在每个点标出并发用户数。","更直观地说，这就是下图中右侧的高昂代价服务与左侧价格竞争力服务之间的差别：","随后，我们将这些单容器副本扩展成部署和服务。其中一个特定服务每天处理数千亿的 token，总量达到数万亿：","下面，我们希望让一般软件工程师能够理解这种性能工程。通过分享我们以及客户如何运维这些服务，我们希望它也能让你做到同样的事情——也许可以通过在 Modal 上部署专用端点（链接：https://modal.com/docs/guide/endpoints）来实现。","我们将内容分为两个部分：理解用于响应每个请求的序列模型，以及理解跨请求的工作负载结构。","当代的编码代理由概率生成模型驱动，这些模型主要通过无监督的掩码序列预测进行预训练，并主要通过增强输出软件正确性进行后训练。就像编译器的解析器一样，它们并不在原始字符串上操作，而是在标记化的序列上操作，因此我们将它们的输入和输出称为标记。因为我们最终是在猜测输出标记应该是什么，所以这被称为推理。如果你更喜欢演绎，请坚持使用数据库和操作系统。","如今的基础序列模型是混合注意力、专家混合的 Transformer 神经网络。这些网络不仅在序列中的每个标记上进行计算，也在序列中的标记之间进行计算。","注意力已经演变为跨标记计算的通用术语。专家混合是指动态路由的块稀疏矩阵乘法，它应用于大部分的每标记计算。这些计算迭代地更新网络的内部或潜在表示。","想了解这些序列模型内部发生了什么，可以参考 Anthropic 的“Transformer 电路的数学框架”：https://transformer-circuits.pub/2021/framework/index.html（2021 年，但仍未被超越）。","通过这样的神经网络进行一次前向传递，会生成大量的内部状态，以及针对序列中每个位置的下一个标记的概率分布。因为我们是基于自己的输出进行预测（“自回归”）：https://en.wikipedia.org/wiki/Regression_toward_the_mean，这是自回归序列建模。","为了响应客户端请求，我们通常像这样将多个前向传递串联在一起：","前向传递非常昂贵，因此我们希望尽可能摊销这部分工作。在每个 token 的计算中，大部分工作可以通过将多个序列一起批处理来摊销。在跨 token 的计算中，大部分工作可以通过缓存内部状态来摊销。出于历史原因，这被称为键值缓存（KV 缓存或简称 KV），尽管像 Kimi 这样的现代模型并没有明确的键和值。你可以在 Kipply 的优秀博客文章《Transformer推理算术》中了解更多关于“餐巾纸数学”的内容：https://kipp.ly/p/transformer-inference-arithmetic（2022 年的文章，但至今仍未被超越）。","当前向传递处理请求的输入 token 时，我们称之为预填充（prefill），因为它是在“预填充” KV 缓存。当前向传递生成响应的输出 token 时，我们称之为解码（decode），因为我们正在将模型对过去状态的“编码”解码成预测的未来。那么既做预填充又做解码的前向传递呢？嗯，我们也不太喜欢这个术语。","预填充性能主要通过完成一个请求的所有预填充所需的延迟来跟踪，也就是首次 token 时间（TTFT）。解码性能主要通过之后输出 token 的生成速率来衡量，也就是每秒输出 token 数（TPS）。这两者都可以在客户端或服务器端测量，这常常会引起混淆。","本文介绍的特定序列模型是 Kimi K2.6：https://huggingface.co/moonshotai/Kimi-K2.6，由 Moonshot AI 提供。该模型使用大约一万亿个数字（矩阵中的权重）来参数化其矩阵乘法，其中大多数以四位整数（INT4）存储。","然而，我们使用四位浮点数（FP4）来服务模型：https://modal.com/llm-almanac/quant-formats/0x6。四位仅提供十六个不同的值，因此还需要微缩放格式：https://modal.com/llm-almanac/block-quants/mx-fp8-e4m3 来独立缩放张量中的各个块。我们选择了 NVFP4 微缩放格式：https://modal.com/llm-almanac/block-quants/nvidia-fp4，它在 Blackwell 流式多处理器架构的张量核心中具有原生硬件支持，可达到 petaFLOP/s 规模：https://modal.com/gpu-glossary/device-hardware/tensor-core 和 https://modal.com/gpu-glossary/device-hardware/streaming-multiprocessor-architecture，如 B200 和 B300 GPU。由于我们在计算资源紧张时期操作动态 GPU 群组，因此我们准备让部署同时在 B200 和 B300 GPU 上运行。以下结果都是针对 B200 GPU；B300 本质上类似，但由于它们有更多高带宽内存（HBM）可用于缓存，能够实现更高的请求并发：https://modal.com/gpu-glossary/device-hardware/gpu-ram。","我们选择 SGLang 推理引擎作为基础。我们发现通过修补引擎有几个提高性能的机会。作为 SGLang 项目的贡献者，我们将这些补丁上游合并，具体描述和链接见以下文章。","当你天真地在编程代理流量上提供此类模型时，结果会很糟糕。","此图表显示，超过 6 个并发用户时，吞吐量和交互性迅速下降。此外，即使在达到峰值之前，交互性也低于用户预期，系统效率低于可接受水平。","因此，从此开始，你需要提高交互性和吞吐量，以向用户提供更好的结果，同时降低自身成本。为此，你需要比仅仅“输入和输出词元”更深入地理解此工作负载中的序列。","单个输出令牌的请求是在“会话”中创建的：用户、生成模型和工具迭代性地调用链在一起，以构建输入序列的塔，同时随着时间的推移积累上下文——以及价值。意义建构、信息发现和理解过程的迭代性，让我们认为这是序列建模和顺序行动本质的基本特征，因此我们预计这种模式将远远超越“编码代理”。","具体来说，单个会话看起来大致如下：","也就是说，每轮 T 的输入序列（绿色）是到 T 为止的整个会话历史（深绿色），加上一些新的内容（浅绿色）。这有两个关键的后果。","首先，这意味着相对于输出序列（上述粉色），请求本质上具有较长的输入序列——在 T 的输入中包含了 T-1 轮的过去输出序列，并且 T 是几十轮。对于我们在优化中使用并在生产中提供的核心工作负载而言，这个比例是 200:1；请求大约包含 10 万个输入令牌，并产生大约 500 个输出令牌。这意味着处理的大部分令牌都会是输入令牌（只需查看你的编码代理软件中的令牌使用数量即可）。","第二，这意味着输入序列与以前处理的输入序列——从第 1 轮到第 T-1 轮的序列——有高度重叠。这意味着在提供第 T 轮时，第 1 轮的令牌会被处理 T 次。这使得缓存变得至关重要——我们可以避免线性增长的重复计算以节省工作量，但同时会引入必须管理的线性增长状态，这种状态本身也有其性能特征。在这种权衡中导航，是我们在本文中要解决的核心工程问题。","了解了这种工作负载的情况后，我们转向优化。","为了优化性能：https://modal.com/gpu-glossary/perf/performance-bottleneck，构建一个可运行的系统，找出瓶颈，然后解决它。根据需要重复，直到你成功为止。","虽然我们的最终目标是优化整个服务，但我们将该问题分解为两个更简单的问题：先优化单个副本，再从一个副本扩展到多个副本。","我们进一步将单副本性能问题拆分为两个子问题：首先最大化交互性，然后在不损失交互性的情况下最大化吞吐量。","交互性主要影响请求延迟。请求延迟和吞吐量通过并发性（即进行中的请求数量）相互作用，可通过Little定律的重新排列来理解：https://modal.com/gpu-glossary/perf/littles-law","我们在延迟、并发性和吞吐量方面的关键瓶颈开始于GPU HBM：https://modal.com/gpu-glossary/device-hardware/gpu-ram","我们在延迟方面的关键瓶颈是在解码过程中HBM带宽：https://modal.com/gpu-glossary/perf/memory-bandwidth。我们通过跨GPU的矩阵乘法并行化（张量并行，TP）以及应用自定义DFlash：https://www.lmsys.org/blog/2026-06-15-next-generation-speculative-decoding-dflash-v2/ 投机解码：https://modal.com/blog/spec-is-all-u-need ——即在每次内存加载中做更多计算，即使这些计算可能不必要，从而提高了性能。","这在HBM容量上形成了并发性瓶颈：我们能在比重新计算结果更快加载的缓存中保留多少工作。我们通过清理HBM中的中间结果、将中间结果量化为较低浮点精度：https://modal.com/llm-almanac/quant-formats/，并通过HiCache扩展缓存层次结构到CPU RAM：https://www.lmsys.org/blog/2025-09-10-sglang-hicache/ 来解决这一问题。我们使用缓存命中率（CHR）作为改进缓存的目标指标。对于大多数编码代理工作负载，达到一到两个9的CHR是非常可行的。","提高交互性提升了用户感知的系统性能。我们选择先从这一点入手。我们做出这一选择有几个原因。","第一个也是最简单的原因是，我们发现编码代理用户喜欢并愿意为更快到达的token付费，因此高交互性是构建我们和客户用户所需服务的关键。","其次，互动性特别适合通过推测解码进行改进：https://modal.com/blog/achieve-sota-specdec。因为这是一种简单、基于学习的技术，其性能收益会随着计算能力和数据量的增加而扩展：机器学习著名的“苦涩教训”：http://www.incompleteideas.net/IncIdeas/BitterLesson.html，在 ML 系统的性能工程中再度显现。我们也知道如何扩展训练：https://modal.com/blog/reinforcement-learning-infrastructure-problem。","选择最大化互动性还有两个额外的好处，一个是操作性，另一个是吞吐量，这在我们运营一个动态、自适应扩展的数千 GPU 队列时尤其显著：https://modal.com/blog/resource-solver；https://modal.com/blog/gpu-health。","最大互动性副本更小，因此更容易提供服务。","将多个处理器一起使用需要一个用于通信的互连网络（interconnect）。Nvidia GPU 的最低延迟、最高带宽互连是 NVLink。NVLink 在一定规模的“域”内操作一组处理器。","单个主机操作系统可以支持最多 8 个 GPU 的 NVLink 域。通常可用的最大 NVLink 域包括 72 个加速器（在多节点 IMEX 域中：https://docs.nvidia.com/multi-node-nvlink-systems/imex-guide/overview.html ）。使用更多加速器将需要速度较慢的互连（IB/RoCE 或更慢的标准以太网）。这意味着为了最大化互动性，每个副本不应期望使用超过 72 个加速器——通信开销几乎肯定会超过每个请求的延迟提升。","但这并不意味着我们必须使用 72 个加速器。","考虑 SemiAnalysis 的 InferenceX 基准测试对同一 NVFP4 Kimi K2.6 模型的结果：https://inferencex.semianalysis.com/inference/kimi-k26，这些结果展示了多种部署中每个 GPU 的吞吐量随互动性的变化，并标注了 GPU 数量：","最高的交互性是通过每个副本仅部署八块 GPU 实现的。此外，这种交互性是在每块 GPU 拥有可比吞吐量的情况下实现的，这意味着通过选择较小的域，我们显然并没有放弃峰值吞吐量的性价比（在我们的交互性约束下）。","为了保持图表的可读性，我们只选择了与我们最相似的一小部分部署，但这种模式在更多加速器类型和 InferenceX 基准测试中的更多模型中同样适用（可在这里查看：https://inferencex.semianalysis.com/inference/kimi-k26）。通常，你只需四或八块 GPU 就可以在每块 GPU 拥有可比吞吐量的情况下实现最高交互性。然后，你可以通过扩展较小的副本来实现相同的总吞吐量。核心 Modal 无服务器平台（https://modal.com/blog/truly-serverless-gpus）使这种扩展在性能和可靠性方面都能得到保障。","这是巨大的运营优势。较小、更简单的单元更容易进行扩展。八块 GPU 可以由单个主机操作系统内核驱动。而 NVL72 域则由九个这样的子系统组成，共享一个地址空间（是的，你应该对此感到不寒而栗）。可用性受限，合同期限长且不灵活。","而八块 GPU 的 Blackwell 系统则足够标准化，可通过按需和竞价市场获得，这使处理可变负载的成本更具效益（https://modal.com/blog/how-to-price-serverless）。一、二或四块 GPU 的副本还可以打包在单个八块 GPU 的物理机器中——该机器已经拥有启动另一副本所需的所有资源（模型权重、JIT 工件）。","当然，如果计算供应和用户需求发生变化，我们将乐于重新审视这一选择。","增加交互性会间接提高吞吐量。","通过减少单个请求的延迟，我们通过释放资源给新请求，从而间接提高了吞吐量。","代理编码工作负载每个会话大约是“闭环”的。会话几乎总是链条——由用户编写的令牌、工具调用响应和模型输出组成。因此，会话中的下一个请求几乎总是在前一个响应生成完成后的一段时间才到达。因此，会话的下一个请求在一段时间内处于潜变量状态，处于推理系统之外——对于工具调用而言，是数十毫秒到数秒，尾部可能延迟数分钟；对于用户响应而言，是几秒到几分钟，尾部可能延迟数小时或更长时间。","在此期间，可以在同一节点上处理其他请求。当您有足够的负载来利用活动容量时，总有请求准备好供节点处理。当您有足够的容量来应对活动负载时，总有节点可以映射请求。这两者都由我们快速的自动扩展系统保证：https://modal.com/blog/truly-serverless-gpus。我们将在多副本扩展部分详细讨论请求路由。","交互性以每个用户每秒输出的令牌数量来衡量。简单来说，自回归序列模型如Transformer是顺序生成这些令牌的。阿姆达尔令人心碎的定律再次显现。"]},"en":{"title":"Detailed explanation of how Modal serves Kimi K2.6 encoding Agent reasoning at the scale of trillions of tokens","summary":"Modal Share provides coding agents with trillion-parameter model Kimi K2.6 inference service optimization practices. After optimization, the performance per user per replica increased 2.8x, the overall throughput per replica increased 5.6x, and a single service processes hundreds of billions of tokens per day.","category":"Industry","source":"Modal 官方工程博客（RSS）","aggregationSource":"Modal 官方工程博客（RSS）","pageTitle":"Detailed explanation of how Modal serves Kimi K2.6 encoding Agent reasoning at the scale of trillions of tokens - Aioga AI News","description":"Modal Share provides coding agents with trillion-parameter model Kimi K2.6 inference service optimization practices. After optimization, the performance per user per replica increa...","url":"https://www.aioga.com/en/news/cmueg56ob04iqrovxup3jp0ly/","contentTranslated":true,"sourceHash":"71aa265e02fac465","translatedAt":"2026-09-23T19:14:22.741Z"},"ja":{"title":"Modal 詳細解説：万兆トークン規模で Kimi K2.6 コーディングエージェント推論をサービスする方法","summary":"Modalは、コーディングエージェントに兆単位のパラメータを持つモデルKimi K2.6の推論サービスを提供する最適化の実践を共有し、最適化後は単一コピーあたりのユーザー性能が2.8倍、コピー全体のスループットが5.6倍向上し、単一サービスで1日に数千億トークンを処理します。","category":"業界動向","source":"Modal 官方工程博客（RSS）","aggregationSource":"Modal 官方工程博客（RSS）","pageTitle":"Modal 詳細解説：万兆トークン規模で Kimi K2.6 コーディングエージェント推論をサービスする方法 - Aioga AIニュース","description":"Modalは、コーディングエージェントに兆単位のパラメータを持つモデルKimi K2.6の推論サービスを提供する最適化の実践を共有し、最適化後は単一コピーあたりのユーザー性能が2.8倍、コピー全体のスループットが5.6倍向上し、単一サービスで1日に数千億トークンを処理します。","url":"https://www.aioga.com/ja/news/cmueg56ob04iqrovxup3jp0ly/","contentTranslated":true,"sourceHash":"71aa265e02fac465","translatedAt":"2026-09-23T19:14:32.789Z"},"ko":{"title":"Modal 상세 설명: 어떻게 수조 토큰 규모로 Kimi K2.6 코딩 에이전트 추론을 제공하는가","summary":"Modal는 코딩 에이전트에 trillion 규모 파라미터 모델 Kimi K2.6 추론 서비스를 제공하는 최적화 실천을 공유했으며, 최적화 후 단일 복제본 당 사용자 성능은 2.8배 향상되고, 복제본 전체 처리량은 5.6배 증가했으며, 단일 서비스는 일일 수천억 토큰을 처리합니다.","category":"업계 동향","source":"Modal 官方工程博客（RSS）","aggregationSource":"Modal 官方工程博客（RSS）","pageTitle":"Modal 상세 설명: 어떻게 수조 토큰 규모로 Kimi K2.6 코딩 에이전트 추론을 제공하는가 - Aioga AI 뉴스","description":"Modal는 코딩 에이전트에 trillion 규모 파라미터 모델 Kimi K2.6 추론 서비스를 제공하는 최적화 실천을 공유했으며, 최적화 후 단일 복제본 당 사용자 성능은 2.8배 향상되고, 복제본 전체 처리량은 5.6배 증가했으며, 단일 서비스는 일일 수천억 토큰을 처리합니다.","url":"https://www.aioga.com/ko/news/cmueg56ob04iqrovxup3jp0ly/","contentTranslated":true,"sourceHash":"71aa265e02fac465","translatedAt":"2026-09-23T19:15:15.983Z"},"es":{"title":"Explicación detallada de Modal sobre cómo servir al Agente Kimi K2.6 con razonamiento de codificación a la escala de billones de tokens","summary":"Modal comparte la práctica de optimización del servicio de inferencia del modelo de billones de parámetros Kimi K2.6 para agentes de codificación. Después de la optimización, el rendimiento por usuario por copia aumenta 2,8 veces, el rendimiento total de la copia aumenta 5,6 veces, y un solo servicio procesa cientos de miles de millones de tokens por día.","category":"Industria","source":"Modal 官方工程博客（RSS）","aggregationSource":"Modal 官方工程博客（RSS）","pageTitle":"Explicación detallada de Modal sobre cómo servir al Agente Kimi K2.6 con razonamiento de codificación a la escala de billones de tokens - Aioga Noticias de IA","description":"Modal comparte la práctica de optimización del servicio de inferencia del modelo de billones de parámetros Kimi K2.6 para agentes de codificación. Después de la optimización, el re...","url":"https://www.aioga.com/es/news/cmueg56ob04iqrovxup3jp0ly/","contentTranslated":true,"sourceHash":"71aa265e02fac465","translatedAt":"2026-09-23T19:15:12.559Z"},"fr":{"title":"Explication détaillée de Modal sur la façon de servir l'agent de raisonnement Kimi K2.6 à l'échelle de trillions de tokens","summary":"Modal a partagé les pratiques d'optimisation du service d'inférence du modèle à trillion de paramètres Kimi K2.6 pour les agents de codage, avec une amélioration des performances par utilisateur de 2,8x pour chaque instance après optimisation, une augmentation globale du débit des instances de 5,6x, et le traitement de centaines de milliards de tokens par jour pour un seul service.","category":"Industrie","source":"Modal 官方工程博客（RSS）","aggregationSource":"Modal 官方工程博客（RSS）","pageTitle":"Explication détaillée de Modal sur la façon de servir l'agent de raisonnement Kimi K2.6 à l'échelle de trillions de tokens - Aioga Actualités IA","description":"Modal a partagé les pratiques d'optimisation du service d'inférence du modèle à trillion de paramètres Kimi K2.6 pour les agents de codage, avec une amélioration des performances p...","url":"https://www.aioga.com/fr/news/cmueg56ob04iqrovxup3jp0ly/","contentTranslated":true,"sourceHash":"71aa265e02fac465","translatedAt":"2026-09-23T19:16:00.236Z"},"de":{"title":"Detaillierte Erklärung, wie man den Kimi K2.6-Codierungs-Agenten auf Billionen-Token-Skala bedient","summary":"Modal teilt die Optimierungspraxis des Kodierungsagenten für das Billionen-Parameter-Modell Kimi K2.6 zur Inferenz. Nach der Optimierung steigt die Leistung pro Benutzer auf einer einzelnen Kopie um das 2,8-fache, die Gesamt-Durchsatzrate der Kopien um das 5,6-fache, und ein einzelner Dienst verarbeitet täglich hunderte Milliarden Token.","category":"行业动态","source":"Modal 官方工程博客（RSS）","aggregationSource":"Modal 官方工程博客（RSS）","pageTitle":"Detaillierte Erklärung, wie man den Kimi K2.6-Codierungs-Agenten auf Billionen-Token-Skala bedient - Aioga KI-News","description":"Modal teilt die Optimierungspraxis des Kodierungsagenten für das Billionen-Parameter-Modell Kimi K2.6 zur Inferenz. Nach der Optimierung steigt die Leistung pro Benutzer auf einer...","url":"https://www.aioga.com/de/news/cmueg56ob04iqrovxup3jp0ly/","contentTranslated":true,"sourceHash":"71aa265e02fac465","translatedAt":"2026-09-23T19:15:57.122Z"},"pt-BR":{"title":"Explicação detalhada do Modal sobre como atender o agente de raciocínio de codificação Kimi K2.6 em escala de trilhões de tokens","summary":"Modal compartilhou a prática de otimização do serviço de inferência do modelo de trilhões de parâmetros Kimi K2.6 para agentes de codificação, após a otimização, o desempenho por usuário de uma única cópia aumentou 2,8 vezes, o throughput total da cópia aumentou 5,6 vezes, e um único serviço processa bilhões de tokens por dia.","category":"行业动态","source":"Modal 官方工程博客（RSS）","aggregationSource":"Modal 官方工程博客（RSS）","pageTitle":"Explicação detalhada do Modal sobre como atender o agente de raciocínio de codificação Kimi K2.6 em escala de trilhões de tokens - Aioga Notícias de IA","description":"Modal compartilhou a prática de otimização do serviço de inferência do modelo de trilhões de parâmetros Kimi K2.6 para agentes de codificação, após a otimização, o desempenho por u...","url":"https://www.aioga.com/pt-BR/news/cmueg56ob04iqrovxup3jp0ly/","contentTranslated":true,"sourceHash":"71aa265e02fac465","translatedAt":"2026-09-23T19:16:44.092Z"},"ru":{"title":"Подробное объяснение модального подхода к обслуживанию Kimi K2.6 кодирующего агента на масштабе триллионов токенов","summary":"Modal делится практикой оптимизации инференс-сервиса триллионной модели параметров Kimi K2.6 для кодирования Агентов. После оптимизации производительность на одного пользователя на одной копии увеличилась в 2,8 раза, общая пропускная способность копии выросла в 5,6 раза, а один сервис обрабатывает триллионы токенов в день.","category":"行业动态","source":"Modal 官方工程博客（RSS）","aggregationSource":"Modal 官方工程博客（RSS）","pageTitle":"Подробное объяснение модального подхода к обслуживанию Kimi K2.6 кодирующего агента на масштабе триллионов токенов - Aioga Новости ИИ","description":"Modal делится практикой оптимизации инференс-сервиса триллионной модели параметров Kimi K2.6 для кодирования Агентов. После оптимизации производительность на одного пользователя на...","url":"https://www.aioga.com/ru/news/cmueg56ob04iqrovxup3jp0ly/","contentTranslated":true,"sourceHash":"71aa265e02fac465","translatedAt":"2026-09-23T19:16:41.336Z"},"ar":{"title":"شرح تفصيلي لكيفية خدمة وكيل الترميز Kimi K2.6 للتفكير على نطاق تريليونات الرموز","summary":"شارك Modal ممارسات تحسين خدمة الاستدلال لنموذج تريليون معلمة Agent Kimi K2.6، بعد التحسين، تحسنت أداء كل نسخة لكل مستخدم بمقدار 2.8x، وارتفع إجمالي معدل نقل النسخ 5.6x، ويتعامل كل خادم بمفرده مع تريليونات الرموز يوميًا.","category":"行业动态","source":"Modal 官方工程博客（RSS）","aggregationSource":"Modal 官方工程博客（RSS）","pageTitle":"شرح تفصيلي لكيفية خدمة وكيل الترميز Kimi K2.6 للتفكير على نطاق تريليونات الرموز - Aioga أخبار الذكاء الاصطناعي","description":"شارك Modal ممارسات تحسين خدمة الاستدلال لنموذج تريليون معلمة Agent Kimi K2.6، بعد التحسين، تحسنت أداء كل نسخة لكل مستخدم بمقدار 2.8x، وارتفع إجمالي معدل نقل النسخ 5.6x، ويتعامل كل...","url":"https://www.aioga.com/ar/news/cmueg56ob04iqrovxup3jp0ly/","contentTranslated":true,"sourceHash":"71aa265e02fac465","translatedAt":"2026-09-23T19:17:29.774Z"},"hi":{"title":"Modal विस्तार से समझाता है कि कैसे ट्रिलियन टोकन स्केल पर Kimi K2.6 कोडिंग एजेंट तर्क सेवा प्रदान करें","summary":"Modal ने कोडिंग एजेंट के लिए ट्रिलियन-पैरामीटर मॉडल Kimi K2.6 अनुरूपन सेवा प्रदान करने के अनुभव को साझा किया, अनुकूलन के बाद प्रत्येक प्रतिलिपि में प्रति उपयोगकर्ता प्रदर्शन 2.8x बढ़ा, प्रतिलिपि का कुल थ्रूपुट 5.6x बढ़ा, और एकल सेवा प्रतिदिन हजारों अरब टोकन संसाधित करती है।","category":"行业动态","source":"Modal 官方工程博客（RSS）","aggregationSource":"Modal 官方工程博客（RSS）","pageTitle":"Modal विस्तार से समझाता है कि कैसे ट्रिलियन टोकन स्केल पर Kimi K2.6 कोडिंग एजेंट तर्क सेवा प्रदान करें - Aioga AI समाचार","description":"Modal ने कोडिंग एजेंट के लिए ट्रिलियन-पैरामीटर मॉडल Kimi K2.6 अनुरूपन सेवा प्रदान करने के अनुभव को साझा किया, अनुकूलन के बाद प्रत्येक प्रतिलिपि में प्रति उपयोगकर्ता प्रदर्शन 2.8x ब...","url":"https://www.aioga.com/hi/news/cmueg56ob04iqrovxup3jp0ly/","contentTranslated":true,"sourceHash":"71aa265e02fac465","translatedAt":"2026-09-23T19:17:29.798Z"},"it":{"title":"Spiegazione dettagliata di Modal su come servire l'agente di ragionamento Kimi K2.6 con una scala di trilioni di token","summary":"Modal condivide le pratiche di ottimizzazione per il servizio di inferenza del modello a trilioni di parametri Kimi K2.6 per l'agente di codifica, con un aumento delle prestazioni per utente per copia di 2,8x, un aumento del throughput complessivo delle copie di 5,6x, e il servizio singolo elabora quotidianamente centinaia di miliardi di token.","category":"行业动态","source":"Modal 官方工程博客（RSS）","aggregationSource":"Modal 官方工程博客（RSS）","pageTitle":"Spiegazione dettagliata di Modal su come servire l'agente di ragionamento Kimi K2.6 con una scala di trilioni di token - Aioga Notizie IA","description":"Modal condivide le pratiche di ottimizzazione per il servizio di inferenza del modello a trilioni di parametri Kimi K2.6 per l'agente di codifica, con un aumento delle prestazioni...","url":"https://www.aioga.com/it/news/cmueg56ob04iqrovxup3jp0ly/","contentTranslated":true,"sourceHash":"71aa265e02fac465","translatedAt":"2026-09-23T19:18:16.164Z"},"nl":{"title":"Modal gedetailleerde uitleg over hoe Kimi K2.6 gecodeerde Agent-redenering te bedienen op schaal van biljoenen token","summary":"Modal deelt de optimalisatiepraktijken van de Kimi K2.6 inferentieservice voor codering agenten met een triljard-parameter model. Na optimalisatie is de prestaties per gebruiker per exemplaar met 2,8x verbeterd, de totale doorvoer per exemplaar met 5,6x verhoogd, en kan een enkele service dagelijks duizenden miljarden tokens verwerken.","category":"行业动态","source":"Modal 官方工程博客（RSS）","aggregationSource":"Modal 官方工程博客（RSS）","pageTitle":"Modal gedetailleerde uitleg over hoe Kimi K2.6 gecodeerde Agent-redenering te bedienen op schaal van biljoenen token - Aioga AI-nieuws","description":"Modal deelt de optimalisatiepraktijken van de Kimi K2.6 inferentieservice voor codering agenten met een triljard-parameter model. Na optimalisatie is de prestaties per gebruiker pe...","url":"https://www.aioga.com/nl/news/cmueg56ob04iqrovxup3jp0ly/","contentTranslated":true,"sourceHash":"71aa265e02fac465","translatedAt":"2026-09-23T19:18:11.958Z"},"tr":{"title":"Modal, Kimi K2.6 Kodlama Ajanı Çıkarımı için Trilyon Token Ölçeğinde Hizmet Vermeyi Ayrıntılı Olarak Açıklıyor","summary":"Modal, paylaşımıyla kodlama Agent için trillion parametreli model Kimi K2.6 çıkarım hizmetinin optimizasyon uygulamalarını sağladı, optimizasyon sonrası tek kopyada kullanıcı başına performans 2.8 kat arttı, kopya genelinde toplam verim 5.6 kat arttı ve tek bir hizmet günlük olarak trilyonlarca token işliyor.","category":"行业动态","source":"Modal 官方工程博客（RSS）","aggregationSource":"Modal 官方工程博客（RSS）","pageTitle":"Modal, Kimi K2.6 Kodlama Ajanı Çıkarımı için Trilyon Token Ölçeğinde Hizmet Vermeyi Ayrıntılı Olarak Açıklıyor - Aioga AI Haberleri","description":"Modal, paylaşımıyla kodlama Agent için trillion parametreli model Kimi K2.6 çıkarım hizmetinin optimizasyon uygulamalarını sağladı, optimizasyon sonrası tek kopyada kullanıcı başın...","url":"https://www.aioga.com/tr/news/cmueg56ob04iqrovxup3jp0ly/","contentTranslated":true,"sourceHash":"71aa265e02fac465","translatedAt":"2026-09-23T19:18:58.562Z"},"vi":{"title":"Giải thích chi tiết Modal cách phục vụ Agent suy luận Kimi K2.6 với quy mô hàng nghìn tỷ token","summary":"Modal chia sẻ thực tiễn tối ưu hóa dịch vụ suy luận mô hình hàng nghìn tỷ tham số Kimi K2.6 dành cho Agent mã hóa, sau khi tối ưu hiệu suất của mỗi bản sao tăng 2,8 lần cho mỗi người dùng, tổng thông lượng của bản sao tăng 5,6 lần, một dịch vụ xử lý hàng nghìn tỷ token mỗi ngày.","category":"行业动态","source":"Modal 官方工程博客（RSS）","aggregationSource":"Modal 官方工程博客（RSS）","pageTitle":"Giải thích chi tiết Modal cách phục vụ Agent suy luận Kimi K2.6 với quy mô hàng nghìn tỷ token - Tin tức AI Aioga","description":"Modal chia sẻ thực tiễn tối ưu hóa dịch vụ suy luận mô hình hàng nghìn tỷ tham số Kimi K2.6 dành cho Agent mã hóa, sau khi tối ưu hiệu suất của mỗi bản sao tăng 2,8 lần cho mỗi ngư...","url":"https://www.aioga.com/vi/news/cmueg56ob04iqrovxup3jp0ly/","contentTranslated":true,"sourceHash":"71aa265e02fac465","translatedAt":"2026-09-23T19:19:02.876Z"},"id":{"title":"Penjelasan mendetail Modal tentang bagaimana melayani Agent Kimi K2.6 dengan skala token triliunan untuk inferensi pengkodean","summary":"Modal berbagi praktik optimasi layanan inferensi untuk model parameter triliun Kimi K2.6 bagi agen pengkodean, setelah optimasi kinerja per pengguna per salinan meningkat 2,8x, throughput keseluruhan salinan meningkat 5,6x, dan setiap layanan memproses ribuan miliar token per hari.","category":"行业动态","source":"Modal 官方工程博客（RSS）","aggregationSource":"Modal 官方工程博客（RSS）","pageTitle":"Penjelasan mendetail Modal tentang bagaimana melayani Agent Kimi K2.6 dengan skala token triliunan untuk inferensi pengkodean - Berita AI Aioga","description":"Modal berbagi praktik optimasi layanan inferensi untuk model parameter triliun Kimi K2.6 bagi agen pengkodean, setelah optimasi kinerja per pengguna per salinan meningkat 2,8x, thr...","url":"https://www.aioga.com/id/news/cmueg56ob04iqrovxup3jp0ly/","contentTranslated":true,"sourceHash":"71aa265e02fac465","translatedAt":"2026-09-23T19:19:43.987Z"},"th":{"title":"การอธิบายเชิงลึกเกี่ยวกับ Modal ว่าจะให้บริการ Kimi K2.6 Encoding Agent ในระดับล้านล้าน token ได้อย่างไร","summary":"Modal แชร์แนวปฏิบัติการปรับแต่งการให้บริการการอนุมาน Kimi K2.6 ซึ่งเป็นโมเดลพารามิเตอร์ล้านล้านสำหรับ Agent หลังการปรับแต่ง ประสิทธิภาพต่อผู้ใช้ต่อสำเนาเพิ่มขึ้น 2.8 เท่า กำลังการประมวลผลรวมต่อสำเนาเพิ่มขึ้น 5.6 เท่า หนึ่งบริการสามารถประมวลผล token หลายร้อยพันล้านต่อวัน","category":"行业动态","source":"Modal 官方工程博客（RSS）","aggregationSource":"Modal 官方工程博客（RSS）","pageTitle":"การอธิบายเชิงลึกเกี่ยวกับ Modal ว่าจะให้บริการ Kimi K2.6 Encoding Agent ในระดับล้านล้าน token ได้อย่างไร - ข่าว AI Aioga","description":"Modal แชร์แนวปฏิบัติการปรับแต่งการให้บริการการอนุมาน Kimi K2.6 ซึ่งเป็นโมเดลพารามิเตอร์ล้านล้านสำหรับ Agent หลังการปรับแต่ง ประสิทธิภาพต่อผู้ใช้ต่อสำเนาเพิ่มขึ้น 2.8 เท่า กำลังการป...","url":"https://www.aioga.com/th/news/cmueg56ob04iqrovxup3jp0ly/","contentTranslated":true,"sourceHash":"71aa265e02fac465","translatedAt":"2026-09-23T19:19:54.811Z"},"pl":{"title":"Szczegółowe wyjaśnienie Modal, jak obsługiwać agenta Kimi K2.6 wnioskowania na skali bilionów tokenów","summary":"Modal udostępnia praktyki optymalizacji usługi inferencyjnej dla agenta Kimi K2.6 z modelem o bilionie parametrów, po optymalizacji wydajność pojedynczej kopii na użytkownika wzrosła 2,8 razy, ogólna przepustowość kopii wzrosła 5,6 razy, a pojedyncza usługa przetwarza dziennie setki miliardów tokenów.","category":"行业动态","source":"Modal 官方工程博客（RSS）","aggregationSource":"Modal 官方工程博客（RSS）","pageTitle":"Szczegółowe wyjaśnienie Modal, jak obsługiwać agenta Kimi K2.6 wnioskowania na skali bilionów tokenów - Aioga Wiadomości AI","description":"Modal udostępnia praktyki optymalizacji usługi inferencyjnej dla agenta Kimi K2.6 z modelem o bilionie parametrów, po optymalizacji wydajność pojedynczej kopii na użytkownika wzros...","url":"https://www.aioga.com/pl/news/cmueg56ob04iqrovxup3jp0ly/","contentTranslated":true,"sourceHash":"71aa265e02fac465","translatedAt":"2026-09-23T19:20:42.302Z"}},"evidenceTier":"verified-news","reviewStatus":"editorial-selected","indexable":true,"editorialCover":""}}