Aleph Alpha 发布开放权重德英双语 MoE 模型 Kolibri

在德国统一日，我们发布了我们的新模型：Kolibri。

Kolibri 是一个英德混合专家变压器模型，总参数量为 780 亿，其中活动参数为 30 亿。它支持最长 100 万 token 的上下文长度。该模型可以在 Hugging Face 下载完整权重：https://huggingface.co/Aleph-Alpha/Kolibri-1，并可在 Apache 2.0 许可证条款下使用。

Kolibri 是我们模型训练持续迭代的成果。我们首先建立了一个模型训练流水线：/en/blog/model-training-as-code/，并通过构建 Kolibri Origin（一款总参数 300 亿，活动参数 30 亿，上下文窗口为 65k token 的模型）进行了验证。Kolibri 也是通过同样的流水线完成的：从数据摄取与整理，到消融实验、预训练和后训练，再到最终评估。它使我们能够运行数百次消融实验和稳定的预训练，而无需在硬件故障或数据连接中断时人工干预。我们持续监控训练指标，并为自定义基准建立了标准化监控。我们在构建和迭代该流水线上投入的时间是一项宝贵的投资。这在 Kolibri 优于 Kolibri Origin 的性能中以及两者发布之间的短时间差距上都可以体现。

Kolibri 是一个专门为受监管领域中的主权关键任务工作构建的语言模型，包括公共管理、工业和航空航天。我们为德语、推理、数学、智能行为及客户在生产中所需的其他功能对 Kolibri 进行了专业化设计。专业化的目的在于优化客户特定用例的性能。通过专业化，客户能够在其 AI 运营中实现上下文化性能，并可以监控其经济影响，从而使 ROI 保持可衡量并随时间增长。

仅仅专业化是不够的。主权同样重要。对我们来说，主权结合了两个维度：模型的构建方式，以及它向客户的转移方式。我们提供完整的供应链完整性，并对每一个决策负责，从数据摄取到预训练和后训练，再到最终的评估。我们提供透明度。客户拥有完全的部署自由和知识产权安全，因此合规性成为模型的固有属性。

阅读我们的技术报告：/downloads/tech-report.pdf 获取完整详情。

我们优化了 Kolibri 在各个领域的性能，同时考虑它们特定的领域语言、法规和流程现实。其小巧高效的体积为我们的客户提供了灵活性，使其能够在本地高效运行，而无需将内部数据发送到第三方推理服务。本节的重点介绍了模型的能力，随后我们将在“我们如何高速构建 Kolibri：#how-we-built-kolibri”一节中进行详细描述。

通过 Kolibri，我们优化了模型能力与部署成本之间的权衡，使用了总计 780 亿参数中的 30 亿活跃参数。Kolibri 在英语和德语的质量与服务成本之间处于帕累托前沿。帕累托前沿是经济学中的一个概念，表示两项目标可以实现的最佳组合，提升其中一个意味着必须牺牲另一项。在相同的服务成本下，没有任何比较模型能提供更高的质量，或在成本更低的情况下提供相同质量。

在数学、编码、基础知识和长上下文任务中，Kolibri 的表现可匹敌活跃参数数量是其四倍的模型，如 Nemotron 3 Super。

公共基准测试无法涵盖专业领域需求，因此我们为对客户重要的垂直行业开发了自己的内部评估套件，如德国公共部门、航空、制造和汽车行业。每个套件都反映了这些行业所需的技能、工作流程和特殊情况，并配套的合成训练环境使我们能够在不使用客户数据的情况下改进 Kolibri 的评估表现。更多内容请参阅“上下文化性能：#contextualized-performance”一节。

基于您的文档提供可靠答案。我们使用弃权数据和 Merlin-Arthur 协议训练 Kolibri。因此，当上下文中没有答案时，它会被训练为回答“我不知道”。我们的客户重视并需求此功能，因此我们持续跟踪和验证弃权准确性。更多细节请参阅“基础：#grounding”一节。

母语为德语和英语的模型。我们开发了一个德英双语的分词器，并专注于在模型训练过程中纳入自然的德语数据：/en/blog/sauerkraut-not-burgers-why-german-llms-need-german-data/，因此在预训练的所有标记中，21.3%是德语。我们很少使用翻译（总共占6%），因为翻译文本往往带有其源语言的文化特征。结果是一个按设计就是双语的模型，而不是一个读过一些德语的英语模型。更多内容可参见德语预训练数据：#german-pre-training-data 和 专用分词器：#specialized-tokenizers。

我们对模型权重保持透明：https://huggingface.co/Aleph-Alpha/Kolibri-1 以及对训练数据的整理：/en/blog/sauerkraut-not-burgers-why-german-llms-need-german-data/，使开发背后的决策可被看见。通过其推理轨迹，可以解释模型如何给出特定答案。通过我们的梅林-亚瑟协议：/en/blog/bounding-hallucinations-merlin-arthur-protocols-for-mutual-information-bounds-in-language-models/，模型建立了可信度基础：当上下文不支持答案时，Kolibri会避免回答。

我们的团队在德国构建了模型，并在德国和芬兰的基础设施上进行训练，遵守欧洲和德国法律，无外国控制。我们拥有从数据整理、预训练和后训练，到在我们的模型工厂中进行优化的整个流程。这种对端到端流程的控制支撑了模型的主权。这种控制也传递给我们的客户，Kolibri的小型化使他们拥有完全的部署自由，并支持可控的推理工作量，以在成本、延迟和答案质量之间进行权衡。

我们的模型工厂是我们对迭代速度的回应，它最大限度地减少了从发现模型错误到训练出更好模型所需的时间。我们将训练流程实现为代码，使团队的经验集中在一个有版本控制的训练方案中，而不是分散的脚本和笔记里。这是Kolibri Origin与Kolibri之间的流程情况。

管道的工作始于一月。经过五个月和数百次消融实验后，Kolibri Origin 于6月11日完成了目标规模的预训练。Kolibri 于9月11日完成。在这两个日期之间的三个月里，我们从30B参数增加到78B，从65k标记的上下文窗口扩展到最多1M，并且从7.5T训练标记增加到20T。为了获得这些20T，管道处理了超过200T的原始数据，对其进行过滤、去重并整理成我们实际训练的数据。我们改变了注意力设计，专家数量增加了三倍，提高了稀疏性，更换了路由算法，改进了训练后的数据，环境任务数量增加了一倍以上，并教会模型在四个不同努力水平上进行推理。两个模型每个标记保持类似数量的活跃参数，但由于我们的效率团队的工作，Kolibri 每个标记的训练速度比 Kolibri Origin 更快。

实现这一点的关键是我们的训练管道，这是将科研级模型开发转化为全自动生产就绪基础设施以设计和训练大型语言模型的努力。我们的管道代码库是共享的：/en/blog/model-training-as-code/。每个提议的代码更改都会触发一次小型端到端模型运行：训练、评估，以便在几分钟内发现是否有问题。我们的运行是 GitHub Actions 工作流，复现一次运行只需检出一个提交。我们使用管道进行主要的训练以及进行数百次消融实验，以做出架构和数据混合的决策。

训练检查点大约每小时生成一次，管道会自动评估它们在英语和德语知识、数学、代码、指令遵循、工具使用、长上下文、安全性、回避幻觉和基础事实等方面的表现。我们每天都能看到能力的出现，而不是等到最后才知道模型结果如何。训练本身的表现比我们预期的还要好。在 Kolibri 进行21天的预训练期间，我们遇到了38次意外中断，大约每10,000 GPU小时一次，原因是硬件故障或连接超时。管道会自动处理这些情况，无需人工干预。集群会自动在不同节点上重新启动训练任务，并从最多250步之前的检查点继续训练。

能够访问整个流水线，并让所有人都可以访问检查点，这意味着任何团队都可以从端到端拥有一项能力，而不仅仅是生产线上的一个环节。一个团队将Kolibri的基础能力训练和评估作为一个整体工作，无缝地整合进整体模型。

然而，达到这一点并非一路顺利。我们曾遇到挫折。由于发现了游离在测试之外的数据打乱错误，我们在处理了几万亿token后停止了Kolibri Origin的预训练，并从头开始重启。我们进行了消融实验，做出了决策，但随后又发现配置中存在错误或缺陷，迫使我们重新运行一些实验。在积累自身经验的同时，我们还跟踪了最先进的架构、最佳实践以及LLM领域的最新研究进展。这些积累的经验提升了我们的流程，并在流水线中建立了防护措施。

观察Kolibri Origin和Kolibri之间的差异，提升是显著的，但这也是较容易的方向：更多的参数、更多的数据、一个成熟且易理解的架构家族，而且仍然是在小规模下。随着我们考虑向更大规模扩展，我们很乐意在基础上迎接挑战。但今年我们建设的最持久的东西并不是流水线，而是一支经过验证的团队，他们能够快速地从原始数据构建、后训练并交付LLM。

Kolibri总共有78B参数，是Kolibri Origin的2.5倍，其中约有3B为活跃参数。在我们的实验中，将模型从32B扩大到123B，性能不断提升。然而，规模越大，训练和服务成本也越高。后者推动了我们的决策：123B在两块H100上只能处理3个长上下文256k-token的用户请求，而78B可以处理18个并发请求，且解码速度快28%。我们使用了384个较小的专家模型，而不是少量大型模型，因为它们在测试中表现更好。我们将同样以效率优先的原则应用于注意力机制。在50层中，只有10层处理完整上下文，其余40层使用紧凑的512-token聚焦窗口。这使得这些层在任何上下文长度下的解码计算和内存都保持有限。这种效率提升既有利于服务训练好的模型，也有利于后训练RL的训练，因为后者依赖大量推理计算。

我们在768个B200 GPU上训练Kolibri：/en/blog/scaling-pre-training-in-practice-a-hierarchical-approach/，分为三个阶段：20T代币，16k序列长度，21天，3.44T中期训练token，64k，200B长上下文适应token，256k。总共近24Ttoken，约是Kolibri Origin消耗的三倍。德语占预训练组合的五分之一以上，约4.3Ttoken，英语约62%，代码约14%。与预训练相比，中期训练数据是一个更具选择性的策划数据集池，权重推理、问题解决、代码和代理数据。对于长上下文，我们没有仅在长文档上训练，因为这往往会削弱之前获得的技能，而是将长文档与前一阶段高质量的中期训练数据交错使用。对于长上下文混合，我们移除了合成长文档，以避免人为膨胀如 RULER 的基准。

我们与 Kolibri Origin 一样，采用了 Muon 进行优化。我们特别关注训练稳定性，使得两个模型都能实现稳健的训练运行，没有任何丢失峰值。在 Kolibri 中，为了在训练过程中专家间的路由，我们引入了精确分位数平衡。分位数平衡是在 Kimi K3 中引入的，由于精确计算被认为传输成本过高，因此通过直方图估计全局分位数;我们证明它可以在固定成本下精确计算，且其精确性提升了负载均衡和模型质量。

训练后分为两个阶段;第一步是监督式微调，教授模型核心推理能力及聊天互动技巧，随后进行大规模强化学习，训练模型跨越多种长视野任务进行推理。我们为这两个步骤构建了流水线，使我们能够对模型行为进行细致控制。

对于SFT，我们生成了总计1740亿个标记的合成数据，对其进行了质量筛选，并与宽松许可的开源数据集的筛选版本结合，以获得高质量的训练组合，总计2680亿个标记。对于强化学习，我们在一组广泛的环境中进行了训练，这些环境包含超过120万个经过精心挑选的任务，涵盖了代码和数学推理、智能任务、指令执行、问答、工具调用等多样化领域。我们优化了内部训练代码库以实现高性能，并使用异步训练，即在使用当前模型在环境中生成训练数据的同时，将模型在已生成的数据上进行训练。

在两个阶段中，我们教会模型以不同努力水平（无、低、中、高）进行推理，这意味着用户可以控制模型在寻找任务解决方案时应投入多少计算资源。这使我们的客户能够在成本和推理速度与最终答案质量之间进行权衡。

根据我们在小型代理模型上的实验发现，使用约20%的德语数据进行训练会产生最佳结果，这意味着我们需要找到4万亿个德语标记，以在20万亿标记的训练规模下训练我们的模型。开放的德语数据集有所帮助，但远远不够：在去重和筛选之后，我们只剩下3900亿德语标记，远未达到目标。正如我们在《Sauerkraut, Not Burgers：/en/blog/sauerkraut-not-burgers-why-german-llms-need-german-data/》中所论述的，德语能力必须来源于高质量的德语文本；过度依赖机器翻译会导致结果不佳，因为存在微妙的翻译错误，并且缺乏真实的德语文化背景。我们通过三种不同的方式填补了标记缺口。

第一个方法是我们自己从 Common Crawl 筛选德语数据。我们建立了一个专门处理德语数据的管道。德语不是英语，德语数据不能像英语数据那样随意过滤：我们必须重新调整针对德语的过滤参数。语言数据管道中的一个典型过滤器是删除包含过多长单词的文档，但德语行政文书的平均单词长度通常超过英语的限制，因此标准设置会悄悄删除公共行政文书的文体。在重新调整后，我们的德语管道为我们提供了 1.3 万亿个独特的德语网络标记。

第二个方法是改写我们已经拥有的德语文档。一个大型语言模型（LLM）以百科条目的风格、问答对话或者文本段落的形式重写原有德语文档，并保持其内容。这可以让模型以多种表述形式学习相同的事实，并在稀缺数据中增加信息量，这与翻译操作不同：源文档是德语，因此主题和文化关联仍然是德语：总理，而不是总统。它不会增加太多新知识，而是为语料库中已有的知识提供新的表述方式。改写产生了约 1 万亿个独特标记，使其成为模型中单一最大的德语数据来源。

第三个方法是翻译，而这我们只在 Kolibri Origin 中使用。将英语翻译成德语在模型、提示和分块选择得当时是可行的，但它存在两个问题。输出仍可能显示翻译腔，即习语的字面翻译：“Drive safe!” 会变成 “Fahre sicher!” 而不是 “Komm gut an!”。更重要的是，文化背景无法翻译。从英语翻译而来的语料库会继承英语网络的地理、人口和机构分布，因此以此训练的模型讲德语时描述的世界仍然看起来像美国。

最终，德语被纳入 Kolibri 作为一个 2.4T 令牌的独特语料库，其中 80% 由我们策划或生成，20% 来自开放数据集，模型在预训练令牌中看到它的比例为 21.3%，大约是在 20T 数据中通过上采样得到的 4.3T。每个德语令牌平均被看到 1.8 次，远低于四轮训练的限制，超过该限制重复不会再带来收益。模型阅读的约 85% 德语是网络文本，既包括原创也包括改写的原创内容。其余部分为策划文档——议会记录、法律文本及其他公开领域的数据——以及那小部分翻译内容。

我们构建了一个双语英语-德语分词器，根据每个模型的预训练数据集进行训练和优化。由于数据中德语占比 21%，它比其他最先进模型对德语的压缩效果更好。这带来了更高效的推理（更少的令牌），降低了成本并缩短了响应时间。我们提出了一种新的训练分词器方法 UniBPE，比现有方法更好地尊重语言形态，尤其是德语的复合结构，同时不牺牲英语令牌的效率。

训练分词器的两种主流方法是 BPE：https://arxiv.org/abs/1508.07909 和 Unigram：https://arxiv.org/abs/1804.10959。我们将两者结合，保留 BPE 的自底向上方法，并使用 Unigram 的训练目标来选择合并哪些词汇进入词表。在我们基于英语/德语数据集训练的 128k 词汇表中，这显著改善了分词效果。

联邦社会法院的

常见的 LLM 训练和评估会奖励猜测：猜测有一定机会得到正确答案，而放弃则没有机会。因此，模型会学会用它所掌握的内容回答，即使其输入信息不足以合理回答。要求模型提供引用通常也会适得其反，原因相同：模型可能仅凭想象生成看似合理的来源，以证明没有依据的答案。

对于受监管的客户来说，知道何时不回答的模型是试点项目和实际部署之间的区别。我们将说“我不知道”视为模型的重要能力，并且（1）开发和跟踪专门的基础信息和防幻觉指标，以及（2）使用和开发专门的训练程序来提高这种弃答能力。

为了衡量幻觉弃答能力，我们使用已有的基准测试以及从客户用例中得出的自定义代理指标。我们还开发了自己的“M/A 基础信息评分”，该评分源于我们的Merlin-Arthur设置，用于表示答案中可证明来自文档的内容的下限。

Kolibri 的幻觉比 Kolibri Origin 少得多：在 AA-Omniscience：https://huggingface.co/datasets/ArtificialAnalysis/AA-Omniscience-Public 条目上，它放弃回答而不是给出错误答案的比例为44%（Origin：15%），在 RGB：https://arxiv.org/abs/2309.01431 上，它更频繁地保持不答（86% 对 74%）且虚假信息更少（87% 对 76%）。在我们自己的 M/A 基础信息评分中：/en/blog/bounding-hallucinations-merlin-arthur-protocols-for-mutual-information-bounds-in-language-models/，该评分是对答案多少来自文档的认证而非估计，Kolibri 达到 0.23，而 Kolibri Origin 及其他一些模型为 0。

大多数公开基准无法反映特定行业的需求。为了衡量上下文化表现——即模型处理特定行业独特工作流程和行业知识的能力——我们需要在客户重视的专业垂直领域（如德国公共部门、法律、硬件、消费电子和汽车）测量其表现。

为此，我们首先构建了领域特定的评估套件，这些套件反映了关键行业所需的技能、工具调用、工作流程和边缘案例，从而捕捉上下文化表现。一旦完成，我们便通过生成高质量的专业合成数据来创建训练环境，以涵盖这些所需技能。为了增强模型在现实任务中的稳健性，我们还对环境进行随机化处理，包括具体细节（工具、配置、装具）等。

这些共同促成了一个迭代引擎，使我们能够加强模型的代理式RAG能力、特定领域的逻辑和能力，并且在从未训练过客户数据的情况下，优化情境化评估。

这些评估工具作为我们流程的一部分自动运行：每一个检查点在到达时都会进行评分。Kolibri相较于其他开源权重模型表现良好。在代理式RAG基准测试Honeypot上，Kolibri的表现超过了所有比较的模型，包括更大的Nemotron 3 Super和Mistral Small 4。在清理过的代理式RAG基准测试MuSiQue上，它仅次于更大规模的Nemotron 3 Super，并且远远领先于其他模型。在五个客户应用中，Kolibri在其中四个表现优于或与最佳模型持平。这些发现表明我们能够处理各种数据和设置，并在不同用例中发挥作用，而其他比较模型对这些细节较为敏感。

这些用于测量情境化性能的评估之所以存在，是因为客户告诉我们他们的应用存在哪些不足。每一个评估都编码了我们从这些对话中学到的内容：哪些文档重要，模型应该获取哪些工具，哪些问题具有挑战性，现有部署系统在哪些方面使用户感到不便。这使得交流在两个方向上都变得具体化。展示失败案例的客户会将其转化为基准测试，每个未来的检查点都将进行测量，我们可以在从未训练客户数据或冒过拟合风险的前提下优化目标。

基准测试使用我们自己的测试框架运行：https://github.com/Aleph-Alpha-Research/eval-framework，并且在适用的情况下，所有模型都使用了各自的最高推理努力等级。

我们的模型在Hugging Face上公开提供：https://huggingface.co/Aleph-Alpha/Kolibri-1，采用Apache 2.0许可协议。

Kolibri 需要提供 Kolibri vLLM 插件的 aleph-alpha-inference 软件包。您可以使用提供的容器镜像 ghcr.io/aleph-alpha/aleph-alpha-inference，也可以从 Aleph-Alpha/aleph-alpha-inference：https://github.com/Aleph-Alpha/aleph-alpha-inference 安装该软件包，该软件包还会安装其支持的 vLLM 版本：

启用推理和工具调用来提供模型服务。

要处理超过262,144个标记的上下文，请添加 --max-model-len 1048576 --hf-overrides '{"max_position_embeddings": 1048576}'。该模型推荐的采样参数为 temperature=1.0、top_p=0.97 和 top_k=128。

请联系我们的团队，我们将很高兴通过企业部署和专业化选项为您提供支持：联系销售：/en/sales/。

海德堡斯派耶街14号 69115

On the Day of German Reunification, we are releasing our new model: Kolibri.

Kolibri is an English-German Mixture-of-Experts Transformer with 78B total parameters, 3B active. It supports context lengths of up to 1M tokens. The model can be downloaded with the full weights on Hugging Face：https://huggingface.co/Aleph-Alpha/Kolibri-1 and used under the Apache 2.0 license terms.

Kolibri is the result of continuous iteration of our model training effort. We first built a model training pipeline：/en/blog/model-training-as-code/ and validated it by building Kolibri Origin, a 30B total, 3B active model with a much shorter 65k token context window. Kolibri ran through the same pipeline: from data ingestion and curation, through ablations, pre-training, and post-training, to the final evals. It enabled running hundreds of ablation experiments and stable pre-training that ran without a person having to step in when hardware failed or a data connection dropped. We continuously monitored training metrics and standardized monitoring for custom benchmarks. The time we put into building and iterating on this pipeline was a valuable investment. We see it in how much better Kolibri is than Kolibri Origin, and in how little time separates their releases.

Kolibri is a specialized language model built for sovereign mission-critical work in regulated areas including public administration, industrials and aerospace. We specialized Kolibri for German, reasoning, math, agentic behavior, and further capabilities our customers need in production. The aim of this specialization was to optimize performance in our customers' specific use cases. Through specialization, customers achieve contextualized performance in their AI operations and they can monitor its economic impact, so that ROI stays measurable and grows over time.

Specialization alone is not enough. Sovereignty is just as important. Sovereignty, for us, combines two dimensions: how we built the model, and how it transfers to our customers. We offer full supply-chain integrity and account for every decision, from data ingestion, through pre- and post-training, to the final evaluations. We provide transparency. Customers have full freedom of deployment and intellectual-property safety, so compliance comes as an inherited property of the model.

Read our tech report：/downloads/tech-report.pdf for full details.

We optimized Kolibri for performance across a wide range of sectors considering their particular domain-specific language, regulatory, and procedural realities. Its small and efficient size provides our customers with flexibility to run it efficiently on-premise, without sending internal data to third-party inference services. The spotlight in this section introduces the model's capabilities, before we describe them in section How we built Kolibri at high velocity：#how-we-built-kolibri.

With Kolibri we optimize the trade-off between model capability and deployment costs, using 3B active parameters out of 78B total. Kolibri sits on the Pareto frontier for quality versus serving cost, for both English and German. The Pareto frontier is a concept from economics, marking the best achievable combinations of two objectives, where improving on one means giving up some of the other. None of the compared models delivers more quality at the same serving cost, or the same quality at lower cost.

Across math, coding, grounding, and long-context tasks, Kolibri matches models with up to four times its active parameter count, such as Nemotron 3 Super.

Public benchmarks fail to capture specialized sector needs, so we developed our own internal evaluation suites for the verticals that matter to our customers, such as the German public sector, aviation, manufacturing and the automotive industry. Each suite mirrors the skills, workflows, and edge cases required in these sectors, and paired synthetic training environments let us improve Kolibri against these evaluations without ever training on customer data. Read more in section Contextualized performance：#contextualized-performance.

Answers that are grounded in your documents. We trained Kolibri with abstention data and with our Merlin-Arthur protocol. As a result, it is trained to say "I don't know" when the answer isn't in the context. Our customers value and request this feature, so we continuously track and validate abstention accuracy. We provide more details in the Grounding：#grounding section.

Native German and English model. We developed a bilingual German/English tokenizer and focused on including organic German data：/en/blog/sauerkraut-not-burgers-why-german-llms-need-german-data/ throughout the training process of the model, so that 21.3% of the pre-training tokens are German. We used translation sparingly (6% overall), since translated text tends to carry the cultural fingerprint of its source language. The result is a model that is bilingual by design, not an English model that has read some German. Read more in sections German pre-training data：#german-pre-training-data and Specialized tokenizers：#specialized-tokenizers.

We are transparent about our model weights：https://huggingface.co/Aleph-Alpha/Kolibri-1 and about curation of our training data：/en/blog/sauerkraut-not-burgers-why-german-llms-need-german-data/, so that decisions behind development are visible. Through its reasoning traces, it becomes explainable how the model came to a particular answer. With our Merlin-Arthur protocol：/en/blog/bounding-hallucinations-merlin-arthur-protocols-for-mutual-information-bounds-in-language-models/ the model grounds trustworthiness : Kolibri refrains from answering when the context doesn't support an answer.

Our teams built the model in Germany, trained it on infrastructure in Germany and Finland, under European and German law, with no foreign control . We own the entire pipeline, from data curation, through pre- and post-training, to optimization in our Model Factory. This control of our end to end pipeline underpins the model's sovereignty. Control also passes on to our customers and Kolibri's small size gives them full freedom of deployment and supports controllable reasoning effort to trade cost and latency against answer quality.

Our Model Factory is our answer to iteration speed, minimizing the time it takes to go from knowing what a model gets wrong to training one that does better. We implemented the training pipeline as code, so that the learnings of our team landed in one versioned training recipe instead of scattered scripts and notes. Here is what that looked like between Kolibri Origin and Kolibri.

Work on the pipeline began in January. Five months and hundreds of ablation runs later, Kolibri Origin finished pre-training at target scale on 11 June. Kolibri finished on 11 September. In the three months between those two dates we went from 30B parameters to 78B, from a 65k-token context window to up to 1M, and from 7.5T training tokens to 20T. To get those 20T, the pipeline processed over 200T tokens of raw data, filtering, deduplicating, and curating it down to what we actually trained on. We changed the attention design, tripled the number of experts, increased sparsity, replaced the routing algorithm, improved our post-training data, more than doubled the number of environment tasks, and taught the model to reason at four different effort levels. Both models kept a similar number of parameters active per token, yet Kolibri trained faster per token than Kolibri Origin thanks to the work of our efficiency team.

What made this possible is our training pipeline, an effort to convert research-grade model development into a fully automated production-ready infrastructure to design and train large language models. Our pipeline codebase was shared by all：/en/blog/model-training-as-code/. Every proposed code change triggers a small end-to-end model run; training, evaluation, to find out if something broke within minutes. Our runs are GitHub Actions workflows, and reproducing one means checking out a commit. We used the pipeline for the main training and the hundreds of ablations that ran to make our architectures and data-mix decisions.

Training checkpoints land roughly every hour and the pipeline automatically evaluates them on English and German knowledge, maths, code, instruction following, tool use, long context, safety, abstention to hallucination and grounding. We can watch capabilities appear every day rather than finding out how the model turned out at the end. The run itself held up better than we expected. Over 21 days of pre-training of Kolibri, we hit 38 unplanned interruptions, roughly one per 10,000 GPU-hours, caused by hardware faults or a connection timing out. Those were automatically handled by the pipeline without manual intervention. The cluster automatically restarted the training job(s) on a different set of nodes, and training picked up from a checkpoint at most 250 steps back.

Having access to the whole pipeline, with checkpoints available to everyone, means that any team can own a capability end to end rather than a stage of an assembly line. One team trained and evaluated the grounding capability of Kolibri as a single piece of work, seamlessly integrating into the overall model.

However, getting here was not a linear walk. We stumbled. We stopped the Kolibri Origin pre-training after a few trillion tokens and restarted it from scratch, as we identified a data-shuffling bug that escaped our tests. We ran ablations, took decisions, only to find a bug or an error in the configuration afterwards, forcing us to rerun some experiments. Alongside our own experience, we tracked state-of-the-art architectures, best practices, and the latest research advances in LLMs. Those accumulated learnings led to improvements in our processes and guardrails in our pipeline.

Looking at the delta between Kolibri Origin and Kolibri, the improvement is notable, but this is also the easy direction: more parameters, more data, a well-understood architecture family, and still at small scale. As we are contemplating scaling up, we are happy to face challenges on our foundation. But the most durable thing we built this year is not the pipeline, it is a team with the proven capability to build, post-train, and ship LLMs from raw data at high velocity.

Kolibri has 78B total parameters, which is 2.5 times more than Kolibri Origin, with ~3B active parameters. In our experiments, increasing the size of our model from 32B to 123B led to ever-improved performance. Yet the larger size came with larger training and serving costs. The latter drove the decision: 123B can handle only 3 long-context 256k-token user queries on two H100s, while 78B handles 18 concurrent requests and decodes 28% faster. We used 384 smaller experts rather than fewer wide ones, as they performed better in our tests. We applied the same efficiency-first reasoning to attention. Out of 50 total layers, only 10 process full context, while the remaining 40 use a tight 512-token focused window. This keeps decode computation and memory bounded in those layers regardless of context length. The efficiency gains benefit both serving the trained model and training during post-training RL, which relies on huge amounts of inference.

We trained Kolibri on 768 B200 GPUs：/en/blog/scaling-pre-training-in-practice-a-hierarchical-approach/ in three stages: 20T tokens of pre-training at a 16k sequence length over 21 days, 3.44T tokens of mid-training at 64k, and 200B tokens of long-context adaptation at 256k. This is nearly 24T tokens in total, roughly three times what Kolibri Origin consumed. German accounts for more than a fifth of the pre-training mix, about 4.3T tokens, against roughly 62% English and 14% code. Compared to pre-training, mid-training data is a much more selective pool of curated datasets weighted toward reasoning, problem-solving, code and agentic data. For long-context, rather than train on long documents alone, which tends to erode the skills acquired earlier, we interleaved long documents with the high-quality mid-training data of the previous stage. For the long-context mix, we removed synthetic long documents to avoid artificially inflating benchmarks such as RULER.

We optimized with Muon, as we did for Kolibri Origin. We put special focus on training stability, resulting in a robust training run without any loss spikes for either model. With Kolibri, for routing between experts during training, we introduce exact quantile balancing. Quantile balancing was introduced in Kimi K3, where the global quantile is estimated from histograms because an exact computation was considered too expensive to communicate; we show that it can be computed exactly at fixed cost independent of batch size, and that the exactness improves both load balance and model quality.

Post-training happened in two stages; the first step is supervised fine-tuning to teach the model core reasoning ability and how to interact in a chat, followed by large-scale reinforcement learning to train the model to reason across a diverse suite of long-horizon tasks. We built a pipeline for both of these steps, which allowed us to exercise fine-grained control over the behavior of the model.

For SFT we generated a total of 174B tokens worth of synthetic data, which we filtered for quality and combined with filtered versions of permissively licensed open-source datasets to obtain a high-quality training mix of 268B tokens. For reinforcement learning we trained on a broad set of environments that contained more than 1.2 million curated tasks across diverse domains such as code & math reasoning, agentic tasks, instruction following, question answering, tool calling, and more. We optimized our in-house training codebase for high-performance and use asynchronous training, where we generate training data on our environments using the current model in parallel to training the model on already generated data.

Across both stages, we taught the model to reason at different effort levels (none, low, medium, high), which means that the user can exercise control over how much compute the model should invest to find a solution to the task at hand. This allows our customers to trade-off cost and inference speed against the quality of the final answer.

From our experiments on small proxy models, we found that training with around 20% German data leads to optimal results, which meant we needed to find 4T German tokens to train our model at a 20T horizon. Open German datasets help, but are far from enough: after deduplication and filtering, we were left with 390B German tokens, well short of our goal. As we argued in Sauerkraut, Not Burgers：/en/blog/sauerkraut-not-burgers-why-german-llms-need-german-data/, German capability has to come from high-quality German texts; relying heavily on machine-translations would lead to poor results due to subtle translation errors and a lack of authentic German cultural context. We closed the token gap in three different ways.

The first was to curate German from Common Crawl ourselves. We built a pipeline specialized for German data. German is not English, and German data cannot be filtered like English data: we had to retune filtering parameters for the German language. One typical filter in a language data pipeline is to remove documents with too many long words, but German administrative prose routinely exceeds the English bound on mean word length, so the standard settings quietly remove the register that public administration writes in. After retuning, our German pipeline gave us 1.3T unique tokens of organic German web.

The second was to rephrase German documents we already had. An LLM rewrites an organic German document in the style of an encyclopedia entry, a Q&A dialogue or a text passage, preserving its content. This teaches the model the same facts in several surface forms and multiplies the information contained in scarce data, and it is a different operation from translation: the source is German, so the subject matter and the cultural affinity stay German: chancellor, not president. It does not add much new knowledge, rather new phrasings of knowledge that was already in the corpus. Rephrasing gave us about 1T unique tokens, making it the single largest source of German in the model.

The third was translation, and this we only used in Kolibri Origin. Translating English into German works when the model, the prompts and the chunking are chosen carefully, but it carries two problems. Output can still show translationese, the literal rendering of idioms: "Drive safe!" becomes "Fahre sicher!" rather than "Komm gut an!". More importantly, cultural context does not translate. A corpus translated from English inherits the geographic, demographic and institutional distribution of the English web, so a model trained on it speaks German about a world that looks American.

In the end German entered Kolibri as a 2.4T-token unique pool, 80% of it curated or generated by us and 20% from open datasets, and the model saw it at 21.3% of pre-training tokens, roughly 4.3T over the 20T run through upsampling. Each German token was seen 1.8 times on average, well inside the four-epoch limit past which repetition stops paying off. Around 85% of the German the model read is web text, either organic or rephrased from organic. The rest is curated documents – parliamentary proceedings, legal texts and other data in the public domain – and that small translated share.

We built a bilingual English-German tokenizer, trained and specialized on each model's pre-training dataset. Because of the 21% German share in our data, it compresses German language better than other SOTA models. This led to more efficient inference (fewer tokens) minimizing costs and shortening response times. We introduce a new way to train tokenizers, UniBPE, that respects the morphology of languages better than existing approaches, especially the compound structure of German, without sacrificing English token efficiency.

The two leading approaches to train tokenizers are BPE：https://arxiv.org/abs/1508.07909 and Unigram：https://arxiv.org/abs/1804.10959. We combine those two, keeping the bottom-up approach of BPE and using the Unigram training objective for selecting which merge to add to the vocabulary. On a 128k vocabulary trained on our English/German dataset, this substantially improves tokenization.

Bundessozialgerichtes Federal Social Court (genitive)

Common LLM training and evaluation rewards guessing: a guess has some chance of landing the correct answer, while abstention has none. Therefore, models learn to answer with whatever they have, even if their input does not provide enough information to warrant a response. Asking a model to provide citations often backfires for the same reason: models can simply hallucinate plausible-looking sources to justify an ungrounded answer.

For a regulated customer, a model that knows to abstain is the difference between a pilot and a deployment. We treat saying "I don't know" as an important model capability and (1) develop and track dedicated grounding and anti-hallucination measures and (2) use and develop dedicated training procedures to improve this abstention capability.

For measuring hallucination abstention, we use established benchmarks alongside our own proxies derived from customer use cases. We've also developed our own "M/A grounding score" which falls out of our Merlin-Arthur setup representing a lower bound on how much of the answer provably came from the document.

Kolibri hallucinates far less than Kolibri Origin: it abstains instead of answering wrong on 44% of AA-Omniscience：https://huggingface.co/datasets/ArtificialAnalysis/AA-Omniscience-Public items (Origin: 15%), and on RGB：https://arxiv.org/abs/2309.01431 it holds back more often (86% vs 74%) and invents fewer falsehoods (87% vs 76%). On our own M/A grounding score：/en/blog/bounding-hallucinations-merlin-arthur-protocols-for-mutual-information-bounds-in-language-models/, which certifies rather than estimates how much of an answer came from the document, Kolibri reaches 0.23 where Kolibri Origin, along some other models, reach 0.

Most public benchmarks fail to capture specialized sector needs. To measure contextualized performance – meaning how well a model handles the unique workflows and domain knowledge of specific industries – we need measurements of performance in specialized verticals which are important to our customers, such as the German public sector, legal, hardware, consumer electronics, and automotive.

For this, we first built domain-specific evaluation suites that mirror the skills, tool-calls, workflows, and edge cases required in key sectors and thus capture contextualized performance. Once this was done, we created training environments by generating high-quality specialized synthetic data capturing those required skills. To increase the real-world robustness of the model on those tasks, we additionally randomized the environments around concrete details (tools, configurations, harnesses).

Together, these contributed to an iterative engine that allowed us to sharpen the model's agentic RAG capabilities, domain-specific logic and capabilities, and to hill-climb the contextualized evaluations without ever training on customer data.

These evaluation suites run automatically as part of our pipeline: every checkpoint is scored on them as it lands. Kolibri compares well against competing open-weight models. On the agentic-RAG benchmark Honeypot, Kolibri outperforms all compared models, including the much larger Nemotron 3 Super and Mistral Small 4. On a cleaned version of the agentic-RAG benchmark MuSiQue, it is second only to the more massive Nemotron 3 Super and well ahead of the rest. On the five customer applications, Kolibri leads or is on par with the best model in four of five. These findings speak to our ability to handle a range of data and harness setups, across use-cases, whereas the compared models are sensitive to these details.

These evaluations for measuring contextualized performance can exist because customers tell us how their applications fall short. Each one encodes what we learned from those conversations: what kind of documents matter, which tools the model should get, which questions are hard, where the deployed system frustrates its users today. That makes the exchange concrete in both directions. A customer who shows us a failure case gets it turned into a benchmark that every future checkpoint is measured against, and we can hill-climb the target without ever training on customer data or risking over-fitting.

Benchmarks were run using our own harnesses：https://github.com/Aleph-Alpha-Research/eval-framework and, where applicable, all models used the highest respective reasoning effort.

Our model is available openly on Hugging Face：https://huggingface.co/Aleph-Alpha/Kolibri-1 under an Apache 2.0 License.

Kolibri requires the aleph-alpha-inference package that provides the Kolibri vLLM plugin. You can either use the provided container image ghcr.io/aleph-alpha/aleph-alpha-inference , or install the package from Aleph-Alpha/aleph-alpha-inference：https://github.com/Aleph-Alpha/aleph-alpha-inference, which also installs the vLLM version it supports:

Serve the model with reasoning and tool-calling enabled.

To serve contexts beyond 262,144 tokens, add --max-model-len 1048576 --hf-overrides '{"max_position_embeddings": 1048576}' . The recommended sampling parameters for the model are temperature=1.0 , top_p=0.97 and top_k=128 .

Contact our team who will be happy to support you through our enterprise deployment and specialization options: contact sales：/en/sales/.

Speyerer Straße 14 69115 Heidelberg