在 M4 Pro Mac mini 上搭建本地 LLM 环境的完整方案

我在我的 M4 Pro Mac mini 上运行本地 LLM 服务器，配备 48 GB 内存。它处理从我的 Hermes 代理后台到手机上的快速聊天查询的一切。整个设置大约需要 30 分钟。

Hermes 作为 Mac mini 上的代理后台运行，我的 MacBook 运行桌面客户端，手机运行 Telegram。对于非 Hermes 的使用，我在 iOS 上使用 Apollo 进行快速聊天（类似 Claude，适合随问随答的问题），Pi 作为我的编程代理（我已经写过这个设置：https://lws.io/blog/pi-harness-cold-start/），以及在我的 Mac 上使用 Raycast AI 来处理随机的事情。

运行本地的主要原因：云 API 是租用的土地。它们可以修改价格、达到你的使用限制，或者在任何时候在后台更换所提供的模型。我经常用尽两个每月 200 美元的订阅，而且感觉在不同时间得到的东西也不同。有时候一个模型很好，有时候它会在没有通知的情况下退化。

还有 AI 主权。我一直在关注美国政府如何限制各种模型的推广。政府可以在任何时候、出于任何原因进行这种限制，而你无法控制。如果你的工作流程依赖于受到限制的云模型，你就必须停止或临时应对。避免这种情况的唯一方法是拥有自己的计算能力。

Mac mini 一直开着。它放在我的桌子上，除了我需要使用它时，我几乎不会注意到它。

Hermes 也运行在 Mac mini 上，使用同一台机器上的本地模型。我通过 Telegram（在手机上）和 MacBook 上的 Hermes 桌面应用访问我的代理。Hermes 桌面应用充当一个‘外壳’，连接到另一台设备上的 Hermes 后台（本例为 Mac mini）。这意味着我可以在所有设备间共享后台、对话历史和技能集。

重点不是替代基于 API 的模型。而是处理 80% 不需要 GPT-5 或 Claude Opus 的请求。当我确实需要这些模型时，它们已经可用。本地模型只是覆盖更多我日常使用的部分，且是免费的。

本地运行大型模型归结为一个问题：它实际在内存中需要多少 RAM。大多数人只看参数数量就会产生误解，因为稠密模型和专家混合（MoE）模型在消费级硬件上的差别非常大。

以下是读取该标识符的方法：

关键区别在于A3B部分。一个密集的27B模型，每个令牌都被加载在RAM中，始终有270亿个参数。像Qwen3.6-35B-A3B这样的MoE模型，总共有350亿个参数，分布在256位专家身上，但每个令牌实际激活的只有大约30亿个。另外320亿个则放在RAM中，什么都不做。

在我的48GB Mac mini上，Qwen3.6-35B-A3B的4位内存大约占用20GB。剩下的28GB用于上下文Windows、操作系统以及机器上运行的其他所有设备。Gemma-4-E4B大约是2.4GB。体积小，可以保留用于简单任务，而使用完整的20GB版本则显得过于繁琐。

我朋友的MacBook Air总共有16GB。4位密集的27B大约需要14GB。这几乎是机器所有的容量，除了操作系统空间。所以它能用一会儿，然后不行就切换到SSD变得很痛苦。

MoE改变了这一点。我标识符中的35B模型可以装在同一台MacBook上，因为每个令牌实际上只有3B的权重是激活的，这意味着GPU/媒体内存占用更接近6B密度模型所需的容量。350亿个总参数权重都放在统一内存中。

如何检查某个型号是否能在你的硬件上运行：

这是没人提的部分。你可以每隔几周更换一次本地模型，因为有新模型发布。这实际上就是下载和重启。

oMLX管理仪表盘内置了HuggingFace模型浏览器。找到模型，点击下载。在Hermes、Pi、Raycast和Apollo中更改模型，我就完成了。

很多操作也可以通过CLI完成，所以我可以从任何设备SSH连接到Mac mini。

为什么这很重要：本地模型和API模型之间的差距正在迅速缩小。一年前的“一般”质量现在在大多数实际任务中都具有竞争力。编码、推理、工具使用才是关键所在。而OptiQ的4位量化技术出乎意料地保持了高质量。35B-A3B在4位测试中相比BF16（16位浮点，未压缩基线）只损失大约1-2分。这是48GB内存使用而非70GB的合理权衡。

Tailscale 在我所有设备之间创建了一个网格。Mac mini、iPhone、MacBook 都在同一个私有网络中。没有任何东西暴露在公共互联网。

oMLX 服务器监听 8000 端口。任何在 tailnet 中的设备都可以连接。Raycast、Apollo iOS、Hermes 桌面客户端在我的 MacBook 上都连接同一个端点。设备之间没有配置漂移。

oMLX 的 KV 缓存持久化在 tailnet 环境下也很重要。编码代理会在会话中反复回到之前的上下文。oMLX 会将每个区块缓存到 SSD 上，所以当代理返回到之前的前缀时，数据可以从磁盘毫秒级恢复，而不是重新计算。这使得本地设置对于代理工作来说实际上是可行的，这也是 Hermes 存在的地方。

Apple Silicon 上的本地模型不再是一个次要实验。M4 Pro Mac mini 轻松应对，模型对于大多数任务已经足够，而且你可以随时更换模型。你不需要按 token 付费。你也不需要通过第三方端点路由敏感数据。当下周出现更好的模型时，你几乎不费力就可以尝试，成本仅是一些硬盘空间。

我已经预订了一台 128GB 的 M5 Max Mac Studio，将在今年晚些时候交付，但到目前为止，我对这台 M4 Pro Mac mini 的性能非常满意。如果你有其他 Apple Silicon 设备，可以试试——可能需要根据你的规格调整模型，但整体设置是一样的。

I run a local LLM server on my M4 Pro Mac mini with 48 GB of RAM. It handles everything from my Hermes agent backend to quick chat queries on my phone. The whole thing takes about 30 minutes to set up.

Hermes runs as the agent backend on the Mac mini, with my MacBook running the desktop client and my phone running Telegram. For non-Hermes usage I use Apollo on iOS for quick chats (reads like Claude, good for throwaway questions), Pi as my coding agent (I already wrote about that setup：https://lws.io/blog/pi-harness-cold-start/), and Raycast AI on my Mac for random things.

The main reason to run local: cloud APIs are rented land. They can change their pricing, hit your usage limits, or swap the model being served behind the scenes whenever they feel like it. I was regularly maxing out two $200/month subscriptions and it felt like I was getting different things from them at different points. Sometimes a model was fine, sometimes it degraded with no notice.

Then there is AI sovereignty. I have been watching how the US government has limited the rollout of various models. That can happen at any point from any government, for any reason, and you have no control over it. If your workflow depends on a cloud model that gets restricted, you have to stop or scramble. The only way to avoid that is to own your compute.

The Mac mini is always on. It sits on my desk and I barely notice it except when I need it.

Hermes runs on the Mac mini as well, using a local model on the same machine. I access my agent through Telegram (on my phone) and the Hermes desktop app on my MacBook. The Hermes desktop app acts as a ‘shell’ and connects to a Hermes backend on another device (in this case the Mac mini). This means I share a backend, conversation history, and skillset across all my devices.

The point is not to replace API-based models. It is to handle the 80% of requests that do not need GPT-5 or Claude Opus. And when I do need those, they are already available. Local just covers more of my day-to-day for free.

Running a large model locally comes down to one thing: how much RAM it actually needs in memory. Most people look at the parameter count and get the wrong idea, because the difference between dense and mixture-of-experts (MoE) models matters a lot on consumer hardware.

Here is how to read the identifier:

The key difference is the A3B part. A dense 27B model has 27 billion parameters loaded in RAM at all times, for every single token. An MoE model like the Qwen3.6-35B-A3B has 35 billion total parameters spread across 256 experts, but only about 3 billion are actually activated per token. The other 32 billion sit in RAM doing nothing.

On my 48GB Mac mini, the Qwen3.6-35B-A3B in 4-bit takes about 20GB of RAM. That leaves 28GB for context windows, the operating system, and everything else running on the machine. The Gemma-4-E4B is roughly 2.4GB. Small enough to keep around for simple tasks where using the full 20GB model is overkill.

My friend’s MacBook Air had 16GB total. A dense 27B in 4-bit needs roughly 14GB. That is literally everything the machine has, minus room for the OS. So it works for a moment, and then when it does not, it swaps to SSD and becomes painful.

MoE changes this. The 35B model in my identifier would fit on the same MacBook because only 3B of weights are actually active per token, which means the GPU/Media Memory footprint is more like what a 6B dense model would need. The 35 billion total parameter weights all sit in unified memory.

How to check if a model will work on your hardware:

This is the part nobody talks about. You can swap out your local models every few weeks as new ones drop. It is literally a download and a restart.

The oMLX admin dashboard has a built-in HuggingFace model browser. Find a model, click download. Change the model in Hermes, Pi, Raycast, and Apollo, and I am all done.

A lot of this can be done via CLI too, so I can SSH into the Mac mini from any of my devices.

Why this matters: the gap between local models and API models is closing fast. What was “meh” quality a year ago is competitive for most real-world tasks now. Coding, reasoning, tool use are where it matters. And the 4-bit quantization from OptiQ keeps quality surprisingly high. The 35B-A3B at 4-bit only loses about 1-2 points on most benchmarks compared to BF16 (16-bit floating point, the uncompressed baseline). That is an acceptable tradeoff for 48GB of memory usage instead of 70.

Tailscale creates a mesh between all my devices. Mac mini, iPhone, MacBook, all on the same private network. Nothing exposed to the public internet.

The oMLX server listens on port 8000. Any device on the tailnet can connect. Raycast, Apollo iOS, Hermes desktop on my MacBook, they all hit the same endpoint. No configuration drift between devices.

oMLX’s KV cache persistence also matters on the tailnet setup. Coding agents repeatedly circle back through earlier context in a session. oMLX caches each block to SSD, so when the agent returns to a previous prefix, it is restored from disk in milliseconds instead of being recomputed. That makes the local setup actually practical for agent work, which is where Hermes lives.

Local models on Apple Silicon are not a side experiment anymore. The M4 Pro Mac mini handles it without breaking a sweat, the models are good enough for most tasks, and you can swap them out whenever you want. You are not paying per token. You are not routing sensitive data through third-party endpoints. And when a better model drops next week, you can try it with barely any effort for the cost of some hard drive space.

I have already ordered a 128GB M5 Max Mac Studio to be delivered later this year, but I am incredibly happy with the performance of this M4 Pro Mac mini so far. If you have other Apple Silicon devices, try it out - you may need to change the model based on your specs, but the general setup holds.