transcribe.cpp 发布：基于 ggml 的跨平台语音转录库，支持 16 个 ASR 模型族

我今天非常兴奋地分享 transcribe.cpp：https://github.com/handy-computer/transcribe.cpp。

transcribe.cpp 是一个基于 ggml 的转录库，支持所有最新的转录模型。handy-computer：https://huggingface.co/handy-computer HF 组织发布的每个模型都经过了数值验证和 WER 测试，以确保与参考实现一致。它在各处都有加速。

我是 Handy：https://handy.computer 的作者和维护者。这个库源于将跨平台语音转文本应用分发给多人时所遇到的痛点。

这是 v0.1.0 版本的库，这意味着仍然有一些我一个人无法发现的问题！请报告它们：https://github.com/handy-computer/transcribe.cpp/issues，让我们一起解决！

让我说一下。我认为用当前的 ASR 推理堆栈分发跨平台应用是糟糕的。

你基本上只有 whisper.cpp 和 ONNX，仅此而已。你可以为 Apple 设备引入 MLX，但这样你就需要支持两个不同的引擎，并将模型移植到每个引擎。我一直是 ONNX 的粉丝，因为它能快速将模型支持集成到 Handy，但仅使用 CPU 时有太多性能被浪费。

目前有一些声称支持很多模型的随机库，但据我所知，它们的作者不明，测试不明。他们给我的问题比答案多。

他们什么时候会停止维护该库？创建者是否考虑过绑定，使你真的能在桌面或移动应用中使用它？这实际上是演示代码吗？他们有进行过基准测试吗？它比 ONNX 快吗？

这就是 transcribe.cpp 产生的原因。作为 Handy 的维护者，我需要一个可以信任的库。我需要能够下载文件并运行推理。我需要知道引擎中的模型推理与参考实现一样好。为了获得最佳性能，推理应该在 GPU 上运行。它应该可以轻松嵌入 Handy，不能是庞大的 PyTorch 库。它必须能在 Mac、Windows 和 Linux 上工作。而 ggml：https://github.com/ggml-org/ggml 似乎是迄今为止最好的选择。它有一个强大的社区，以及优秀的分发机制。

你会得到一个快速且准确的推理引擎，并支持广泛的模型。

我们打算支持尽可能多的最先进转录模型。截至目前，我们支持大多数公开可用的现代转录模型。仍有一些缺失，但它们很快会被添加。

我的主要目标之一是能够在 Vulkan 上运行我想要的任何 ASR 模型。在我看来，这是任何提供本地推理的应用程序的底线。对于我们支持的每个模型，都有在 Fedora 上的 Ryzen 4750U（CPU Vulkan）以及我的 M4 Max 上的相应基准测试。

我还希望确保 transcribe.cpp 中的推理准确，并尽可能接近参考实现。这在很大程度上源于我在使用 Hugging Face 上发现的 .onnx 模型时，对推理准确性存在极大的不确定性。为了确保我们的推理是正确的，我们针对每个模型进行与参考的数值验证。在数值验证之上，我们进行完整的 WER 扫描，以确保无论参考输出什么，我们输出的内容是一致的。这意味着每个模型都经过数千条话语运行，并且非常接近或与参考相同。这些数据的结果发布在 transcribe.cpp 仓库中，以及每个模型在 Hugging Face 上。

transcribe.cpp 在某种程度上可以说是对 whisper.cpp 的直接支持。主要原因是：Handy 使用了 whisper.cpp，而我需要发布一个含 transcribe.cpp 的更新来替代它。我需要保持与非常流行的在 whisper.cpp 中运行的 .bin 文件的兼容性，这些文件随 Handy 发布。transcribe.cpp 可以运行它们。whisper.cpp 中有一些标志和功能我们尚未支持。但我认为，对于绝大多数使用情况，我们的 whisper 实现是可靠的，并且可以替代 whisper.cpp，同时性能大致相当。

一开始我就考虑过语言绑定。虽然这个库是用 C/C++ 写的，但我需要在 Rust 中的绑定。我也知道，为了让我们尽可能广泛地分发本地转录，它至少需要对绑定有良好的第一方支持。我选择了四种我认为足够代表人们使用该库的语言。我也欢迎其他人直接为项目贡献绑定，前提是他们愿意承担维护的责任。

当然，归根结底，很多决策都是由 Handy 驱动的。由于 Handy 很受欢迎，我打算维护这个库，就像我尽力维护 Handy 一样。我打算继续维护开源项目，并在可能的情况下为生态系统做出贡献。

没有 Handy，这个库根本不会存在，因为我不会有尝试支持一堆不同 ASR 模型的问题。我也不会学到人们对 ASR 的各种使用场景。我尽力涵盖了我听到最多的场景。当然，库中还有一些情况目前没有被处理。如果有我遗漏的地方，你可以自由地为库做出贡献！

transcribe.cpp 的目标明确是让本地运行 ASR 更加容易。我们知道，转录在大多数设备上都可以非常准确地运行，而且没有必要将你的语音发送到云服务。一个 RK3566 可以通过 transcribe.cpp 在其低功耗 CPU 上实现比实时更快的模型运行。使用 SOTA 模型的实时以上速度转录只需少数瓦特。这不是希望或梦想，而是事实。

我认为，展望未来，由于不同原因，更多的推理将在本地进行。这使得分发故事成为核心。为了让更多应用程序在本地运行推理，我们需要使推理运行更容易。当然，transcribe.cpp 并不能完全解决这个问题，还有很长的路要走，但我希望这是一个小小的进步。我当然学到了很多。

我非常感谢所有支持这个项目的人。

首先感谢 Mozilla AI：https://www.mozilla.ai，他们的 BiR 项目：https://www.mozilla.ai/company/bir，以及来自 Mozilla AI 的 Davide。这个项目在很大程度上只是我头脑中的一个问题，我带着这个问题去找他们，他们决定支持我解决这个问题。当时 transcribe.cpp 甚至还不是一个具体的想法，我只是探索如何在 Handy 中解决加速分发的问题。所以非常感谢他们，以及他们的支持，帮助这个项目得以实现。

ggml：https://github.com/ggml-org。如果没有 ggml 以及所有贡献者，这个项目是不可能实现的。非常感谢大家所做的工作。我认为 ggml 在帮助使本地推理应用分发变得容易和可能方面确实做了惊人的工作。

Modal：https://modal.com/ 对我也提供了关键帮助。我联系了他们，他们给了我一些积分。这些积分用于进行 WER 测试，确保库在 CUDA 上能够良好运行。能够验证工作的正确性是巨大的帮助。

Blacksmith：https://blacksmith.sh 帮助为 transcribe.cpp 提供了一些 CI/CD 支持。我同样联系了他们，他们立即回应并提供了积分。当然，CI/CD 对于确保发布的一切至少经过一定程度的测试是至关重要的。

Hugging Face：https://huggingface.co，既是本地 AI 社区的支柱，也提供了 handy-computer 组织的私有存储，使我可以随意上传模型。

是的，完全正确。我不认为一个人能够在几个月内使用 ggml 从零编写出如此规模的引擎而不依赖外部帮助。这里的任何文字是用 AI 写的吗？没有。它们都来自我的嘴巴或者手指。

I'm super excited to share transcribe.cpp：https://github.com/handy-computer/transcribe.cpp today.

transcribe.cpp is a ggml based transcription library which supports all the latest transcription models. Every model published under the handy-computer：https://huggingface.co/handy-computer HF org has been numerically validated and WER tested to match the reference implementation. It's accelerated everywhere.

I'm the author and maintainer of Handy：https://handy.computer. This library grew from the pains of distributing a cross-platform speech-to-text application to many people.

This is a v0.1.0 library which means that there are some rough edges which I cannot discover alone! Please report them：https://github.com/handy-computer/transcribe.cpp/issues, and let's fix them together!

Let me say this. I think distributing a cross-platform application with the current ASR inference stack is terrible.

You've basically got whisper.cpp and ONNX. That's it. You could roll MLX in for Apple devices, but now you've to support two different engines and port models to each. I've been a fan of ONNX for getting model support into Handy quickly, but so much performance is left on the table with CPU only.

There are a few random libraries out there which claim to support a lot of models, but they have unknown authors, and unknown testing, as far as I've seen. They leave me with more questions than answers.

When will they stop maintaining this library? Has the creator thought about bindings so you can actually use it in a real desktop or mobile app? Is this effectively demo code? Have they benchmarked it? Is it faster than ONNX?

And this is what led to transcribe.cpp. As Handy's maintainer I needed a library I could trust. Where I could download a file and run inference on it. Where I can know that the inference coming from the model in the engine is as good as the reference implementation. The inference should run on the GPU for the best performance. It should be trivially embeddable in Handy, it cannot be a huge pytorch lib. It must be something that works on Mac, Windows, and Linux. And ggml：https://github.com/ggml-org/ggml seemed like by far the best way forward. It has a strong community, and a great distribution story.

You get a fast and accurate inference engine with wide ranging model support.

We intend to support as many state-of-the-art transcription models as possible. As of today, we support most of the modern transcription models that are publicly available. There are a few missing still, but they will be added soon.

One of my top goals was to run any ASR model I wanted on Vulkan. In my opinion this is the floor for any application shipping local inference. For every model we support, there is a corresponding benchmark run from a Ryzen 4750U (CPU + Vulkan) on Fedora as well as on my M4 Max.

I also wanted to make sure that inference in transcribe.cpp is accurate and as close to the reference implementation as possible. This largely came from a huge degree of uncertainty of inference accuracy when using .onnx models I found on Hugging Face. In order to ensure the inference we do is correct we numerically validate every model versus the reference. On top of numerical validation, we run full WER sweeps to make sure that whatever the reference is outputting, we output the same thing. That means every model has run through thousands of utterances and is very close or same as the reference. And the results of this data are published in the transcribe.cpp repo as well as with each model on Hugging Face.

transcribe.cpp is more or less a drop in support for whisper.cpp. The main reason for this is: Handy used whisper.cpp and I needed to ship an update with transcribe.cpp which would replace it. I needed to keep some compatibility with the very popular .bin files which run in whisper.cpp and shipped with Handy. transcribe.cpp can run them. There are some flags and features in whisper.cpp which we do not support yet. But I think for the vast majority of use cases our whisper implementation is solid and can replace whisper.cpp while having about equal performance.

Language bindings were on my mind to begin with. While this library is written in C/C++, I needed bindings in Rust. And I also knew that in order for us to distribute local transcription as widely as possible, it requires at minimum decent first-party support of bindings. I've chosen 4 languages that I think are fairly representative of where people will use the library. I welcome others to contribute bindings directly to the project as well, assuming that they are willing to take on the maintenance burden of doing so.

And of course, at the end of the day, a lot of the decisions were driven by Handy. As a result of Handy being popular, I intend to maintain this library, just as I've done my best to maintain Handy. I intend to be someone who continues to maintain open source projects and contribute to the ecosystem where I can.

This library never would have existed without Handy because I wouldn't have had the problem of trying to support a bunch of different ASR models. I would have never learned all the use cases that people have for ASR. I've done my best to cover the ones that I hear about the most. Certainly, there are cases in the library that are not currently handled. If there are things that I missed, you are free to contribute to the library!

transcribe.cpp is aimed squarely at making locally run ASR easier. We know that transcription can run extremely accurately on most devices, and there should be no need to send your voice to a cloud service. An RK3566 can run models via transcribe.cpp faster than real time on its anemic CPU. Faster than real time transcription with SOTA models runs in a handful of watts. It's not a hope or a dream, it's a fact.

I think as we look forward to the future, more inference will start happening locally for one reason or the other. This brings the distribution story front and center. In order to have more applications running inference locally, we need to make running inference easier. Certainly transcribe.cpp does not solve this on the whole, and there is a long way to go, but I hope it's a small step forward. I've certainly learned a lot.

I am extremely thankful for all the folks who have supported this project.

First and foremost is to Mozilla AI：https://www.mozilla.ai, their BiR program：https://www.mozilla.ai/company/bir, and Davide from Mozilla AI. This project was largely a problem in my head that I came to them with, and they decided to support me in solving the problem. At the time transcribe.cpp wasn't even a concrete idea, I was just exploring how to solve accelerated distribution in Handy. So a huge thanks to them, their support, and helping to bring this project into existence.

ggml：https://github.com/ggml-org. This project wouldn't be possible without ggml and all of the contributors to it. Thank you all so much for the work you've done. I think ggml really does amazing work in helping to make distributing local inference applications easy and possible.

Modal：https://modal.com/ has also been a critical help for me. I reached out to them, and they gave me credits. These credits are put towards doing the WER testing and ensuring the library works well on CUDA. It is an immense help being able to verify the correctness of the work.

Blacksmith：https://blacksmith.sh helps to power some of the CI/CD for transcribe.cpp. Again I reached out to them and they immediately responded with credits. Of course CI/CD is critical for making sure everything put out has been tested to at least some degree.

Hugging Face：https://huggingface.co both for being a pillar in the local AI community, as well as providing the handy-computer org private storage, so I could upload models at my own will.

Yes absolutely. I don't think it's possible for a single individual to write an engine from scratch of this size using ggml in a handful of months without outside assistance. Were any of the words here written using AI? Nope. They came from my mouth or my fingers.