Perplexity 开源 Rust + Metal 推理引擎 Lily，专为 Apple 硅上的 Qwen3.6-35B-A3B 打造

Perplexity 已开源 Lily：https://github.com/perplexityai/pplx-garden/tree/main/lily，这是 Perplexity Computer 中 Hybrid Compute 背后的本地推理引擎：https://www.perplexity.ai/hub/products/hybrid-compute。它是一个单进程运行时：Rust 层加载检查点并驱动生成循环，一个兼容 OpenAI 的聊天完成 API 流式传输令牌，手写的 Metal 内核执行模型。执行路径中没有 PyTorch 或 MLX。Lily 是有意设计得很窄，仅支持一个模型 Qwen3.6-35B-A3B：https://huggingface.co/Qwen/Qwen3.6-35B-A3B，在一个硬件家族上运行，这种窄化正是其性能优势所在。

它可以部署吗？可以。pplx-garden 仓库中有一个独立演示公开：https://github.com/perplexityai/pplx-garden/tree/main/lily。一个使用 Rust 和 Metal 的推理服务器，通过最小化的兼容 OpenAI 的 HTTP API 提供贪婪文本生成。4-bit 检查点大小为 19.4 GB，因此一台具有 32 GB 或更多统一内存的苹果硅 Mac 是实际的最低要求；Perplexity 的 Hybrid Compute 商用产品列出了 macOS 15+，最低 24 GB，最佳效果为 32 GB。

默认的 Mac 堆栈是 MLX：https://github.com/ml-explore/mlx 以及 MLX-LM：https://github.com/ml-explore/mlx-lm，它已经提供 Qwen 的实现：https://github.com/ml-explore/mlx-lm/blob/main/mlx_lm/models/qwen3_5.py，包含分组专家工作、融合的循环 Metal 内核，以及 GQA 感知注意力。但其操作必须在各架构间保持可复用。Lily 放弃了这一点，将模型结构、执行计划和内核选择集中在一个运行时中。

Qwen3.6-35B-A3B 存储 350 亿个参数，每个令牌激活约 30 亿个。一个路由器对 256 个专家进行评分并选择八个，同时还有一个共享专家处理每个令牌。它还混合了 10 层全注意力层，使用分组查询注意力：https://arxiv.org/pdf/2305.13245（16 个查询头，两个 KV 头）与 30 层门控 DeltaNet：https://arxiv.org/abs/2412.06464。这产生三种模式：不均衡的专家组、对增长的 KV 缓存的注意力，以及固定大小的循环。

检查点使用分组仿射4位量化，每组64个权重共享bfloat16的尺度和偏置，约70 GB的bfloat16权重压缩为19.4 GB。金属4张量运算会消耗bfloat16，因此权重必须先重建。Lily在分组后的GEMM中逐格进行，按住会进入线程组内存，累积到FP32，因此扩展后的数组永远不会到达统一内存。在Perplexity的消融中，该融合在512标记提示下提升了端到端预填充率77.4%。

将路由直方图、前缀扫描、散射和块状映射保持在单一 GPU 命令缓冲区内，通过移除每个 MoE 层的 CPU 同步，在 512 个令牌中增加了 89%。从 16 行瓦片到 32 行并配备四个 simdgroup 在 2K 时增加了 13.2%;寄存器驻留的门控 DeltaNet 扫描增加了 5.6%。专家 GEMM 大约占预填充时间的 90%。长提示以有界块形式运行，因此临时激活不会与权重和缓存争夺内存。

第一批译码几乎没有权重重复用，因此带宽限制了上限。一个记录的步骤启动了795个内核，形成了555个顺序阶段;Lily在并发的Metal通道中记录真实依赖，因此独立内核会重叠。所选令牌直接写入下一步的GPU驻留输入槽，省去每个令牌CPU的往返，四条内核链融合以保持寄存器中的中间节点。

合并缓存读取将密钥带宽从33.8提升至47.9 GB/s，价值带宽从42.0提升至61.8 GB/s。GQA打包、四个查询头共用一个线程组，使每行KV加载一次，在32K时解码提升了23.8%。在32K及以上采用固定块注意力布局，解码提升了7.7%，64K时提升了27.4%，128K时提升了40.2%。

在一台40核、128GB M5 Max的第1批次中，加载相同的4位检查点字节，匹配MLX-LM最快的直接生成路径，跨越10个长度，介于256至128K令牌，Lily平均预填充令牌为4,156个，对比3,388个（1.23x），译码令牌为170.0个，译码令牌为126.4个（1.35倍）。在4K提示和4K上下文下，它达到了5,749.9和186.6个令牌/秒，而4,737.5和140.9个，且在所有记录点（1.12–1.42x预填充，1.31–1.37x译码）都更快。教师强制检查了192个位置，结果发现莉莉的困惑度提高了0.04%，而同样排名第一的代币有96.35%的比例。

需要与我们合作推广您的 GitHub 仓库或 Hugging Face 页面或产品发布或网络研讨会等吗？请联系我们：https://forms.gle/wbash1wF6efRj8G58

Asif Razzaq 是 Marktechpost Media Inc. 的首席执行官。作为一位富有远见的企业家和工程师，Asif 致力于利用人工智能的潜力造福社会。他最近的工作是推出人工智能媒体平台 Marktechpost，该平台以深入报道机器学习和深度学习新闻而著称，既技术性强又易于广大读者理解。该平台每月访问量超过 200 万次，显示出其在观众中的受欢迎程度。

Perplexity has open sourced Lily ,：https://github.com/perplexityai/pplx-garden/tree/main/lily the local inference engine behind Hybrid Compute in Perplexity Computer：https://www.perplexity.ai/hub/products/hybrid-compute. It is a single-process runtime: a Rust layer loads the checkpoint and drives the generation loop, an OpenAI-compatible chat-completions API streams tokens, and hand-written Metal kernels execute the model. Neither PyTorch nor MLX sits in the execution path. Lily is deliberately narrow with one model, Qwen3.6-35B-A3B：https://huggingface.co/Qwen/Qwen3.6-35B-A3B, on one hardware family and that narrowness is the performance argument.

Is it deployable? Yes. A standalone demo is public in the pplx-garden repository：https://github.com/perplexityai/pplx-garden/tree/main/lily. A Rust and Metal inference server offering greedy text generation through a minimal OpenAI-compatible HTTP API. The 4-bit checkpoint is 19.4 GB, so an Apple silicon Mac with 32 GB or more of unified memory is the realistic floor; Perplexity’s shipping Hybrid Compute product lists macOS 15+, 24 GB minimum and 32 GB for best results.

The default Mac stack is MLX：https://github.com/ml-explore/mlx plus MLX-LM：https://github.com/ml-explore/mlx-lm, which already ships a Qwen implementation：https://github.com/ml-explore/mlx-lm/blob/main/mlx_lm/models/qwen3_5.py with grouped expert work, a fused recurrent Metal kernel, and GQA-aware attention. But its operations must stay reusable across architectures. Lily gives that up and puts model structure, execution plans, and kernel selection in one runtime.

Qwen3.6-35B-A3B stores 35B parameters and activates roughly 3B per token. A router scores 256 experts and picks eight, alongside one shared expert that sees every token. It also mixes 10 full-attention layers using grouped-query attention：https://arxiv.org/pdf/2305.13245 (16 query heads, two KV heads) with 30 Gated DeltaNet：https://arxiv.org/abs/2412.06464 layers. That yields three patterns: uneven expert groups, attention over a growing KV cache, and a fixed-size recurrence.

The checkpoint uses groupwise affine 4-bit quantization, every group of 64 weights sharing a bfloat16 scale and bias, about 70 GB of bfloat16 weights compressed to 19.4 GB. Metal 4 tensor operations consume bfloat16, so weights must be reconstructed first. Lily does that one tile at a time inside the grouped GEMM, holding results in threadgroup memory and accumulating in FP32, so the expanded array never reaches unified memory. In Perplexity’s ablation that fusion raised end-to-end prefill 77.4% at a 512-token prompt.

Keeping the routing histogram, prefix scan, scatter and block map inside a single GPU command buffer added 89% at 512 tokens by removing CPU synchronization inside each MoE layer. Moving from 16-row to 32-row tiles with four simdgroups added 13.2% at 2K; a register-resident Gated DeltaNet scan added 5.6% . Expert GEMMs are roughly 90% of prefill time. Long prompts run in bounded chunks so temporary activations do not compete with weights and cache for memory.

Batch-1 decode has almost no weight reuse, so bandwidth sets the ceiling. One recorded step launched 795 kernels forming 555 sequential stages; Lily records real dependencies in a concurrent Metal pass so independent kernels overlap. The selected token is written straight into the next step’s GPU-resident input slot, removing a per-token CPU round trip, and four kernel chains are fused to keep intermediates in registers.

Coalesced cache reads lifted key bandwidth from 33.8 to 47.9 GB/s and value bandwidth from 42.0 to 61.8 GB/s. GQA packing, four query heads sharing one threadgroup so each KV row loads once, improved decode 23.8% at 32K. A fixed-block attention layout at 32K and above improved decode 7.7% at 32K, 27.4% at 64K, and 40.2% at 128K.

On one 40-core, 128 GB M5 Max at batch 1, loading identical 4-bit checkpoint bytes against MLX-LM’s fastest direct-generation path across ten lengths from 256 to 128K tokens, Lily averaged 4,156 prefill tokens/s versus 3,388 (1.23x) and 170.0 decode tokens/s versus 126.4 (1.35x) . At a 4K prompt and 4K context it reached 5,749.9 and 186.6 tokens/s against 4,737.5 and 140.9, and was faster at every recorded point: 1.12–1.42x prefill, 1.31–1.37x decode. A teacher-forced check across 192 positions put Lily’s perplexity 0.04% higher, with the same top-ranked token 96.35% of the time.

Need to partner with us for promoting your GitHub Repo OR Hugging Face Page OR Product Release OR Webinar etc.? Connect with us ：https://forms.gle/wbash1wF6efRj8G58

Asif Razzaq is the CEO of Marktechpost Media Inc.. As a visionary entrepreneur and engineer, Asif is committed to harnessing the potential of Artificial Intelligence for social good. His most recent endeavor is the launch of an Artificial Intelligence Media Platform, Marktechpost, which stands out for its in-depth coverage of machine learning and deep learning news that is both technically sound and easily understandable by a wide audience. The platform boasts of over 2 million monthly views, illustrating its popularity among audiences.