AlloyDB 推出列式引擎加速 HNSW，pgvector 向量搜索 QPS 提升最高 4.9 倍

工作场所人工智能的前门

AlloyDB：https://cloud.google.com/alloydb 是一个完全托管的、与 PostgreSQL 兼容的数据库服务，专为最苛刻的企业工作负载而构建。它结合了开源 PostgreSQL 的优点与谷歌的先进技术，提供极大的可扩展性、高可用性和原生 AI 功能。它既可以作为高性能的关系型存储，也可以作为向量和全文搜索的统一后端，同时作为分析引擎，其速度比标准 PostgreSQL 快 100 倍。

向量搜索是现代 AI 和增强检索生成（RAG）应用的基础。对于使用 AlloyDB 和其他 PostgreSQL 数据库的开发者来说，pgvector：https://github.com/pgvector/pgvector 是一个被广泛采用的扩展，用于存储、索引和查询向量嵌入，而 HNSW（分层可导航小世界）是一种高效的基于图的算法，设计用于多层结构的近似最近邻搜索。通过 AlloyDB 中的列式引擎加速 HNSW：https://docs.cloud.google.com/alloydb/docs/ai/accelerate-with-ce（现正处于预览阶段），在向量搜索中可实现比标准 PostgreSQL HNSW 高出 4 倍的每秒查询量（QPS）。

企业 AI 应用面临速度与准确性之间的持续权衡。在搜索数百万或数十亿个向量时，最大化每秒查询量（QPS）而不牺牲搜索质量（召回率），对于扩大生产工作负载至关重要。PostgreSQL 的 pgvector 扩展提供 HNSW 作为可以加速近似最近邻（ANN）搜索的索引之一。让我们深入探讨 AlloyDB 如何解决速度与准确性的权衡问题。

注意：虽然本文关注的是 HNSW 性能，但值得注意的是 HNSW 只是 AlloyDB 高级向量工具包的一部分。AlloyDB 还提供 ScaNN：https://docs.cloud.google.com/alloydb/docs/ai/create-scann-index ——由 Google Research 超过 14 年的研究支持的前沿索引——为您提供选择最适合工作负载的索引的灵活性。此外，对于需要绝对精度的用例，标准的 k 最近邻（KNN）搜索始终可用于 100% 召回率。请查看我们的“选择向量索引指南”：https://docs.cloud.google.com/alloydb/docs/ai/choose-index-strategy，了解它们的比较情况。

AlloyDB 提供免费试用实例：https://cloud.google.com/alloydb/docs/free-trial-cluster，让您可以使用自己的工作负载体验 AlloyDB。此外，新 Google Cloud 用户可获得 300 美元的免费额度。

AlloyDB 列式引擎：https://docs.cloud.google.com/alloydb/docs/columnar-engine/about 是一个内置的内存缓存，自动以专门的、扫描优化的列式格式存储经常查询的数据。它使 AlloyDB 能够处理高达比标准 PostgreSQL 快 100 倍的重分析查询。此外，它通过将索引存储在内存中、使用向量化内存布局进行快速遍历，并绕过标准 PostgreSQL 缓冲区管理器开销，从而加速 ANN 搜索。

为了了解列式引擎加速 HNSW 的现实性能特征，我们绘制了 GloVe 100 Angular 数据集的标准 QPS 对召回率曲线，通过在超过 100 万条记录中以 100 条限制进行搜索。

运行此基准测试脚本：https://colab.research.google.com/github/GoogleCloudPlatform/python-docs-samples/blob/main/alloydb/notebooks/columnar_engine_accelerated_hnsw_vector_search_benchmark.ipynb 得到以下可视化结果：

注意：这些测量是在 AlloyDB C4A 16vCPU 机器上进行的。由于 HNSW 图构建的固有随机性，结果在不同运行间可能略有差异。

数据揭示了两个变革性益处：

需要注意的是，基线（蓝色线）已经表示索引完全缓存在 PostgreSQL 共享缓冲缓存中。这里显示的性能提升并不是将数据从磁盘移动到内存的结果，而是更高效的内存架构带来的结果。

在标准的 PostgreSQL 架构中，索引操作会使用共享缓冲缓存。即使数据完全驻留在内存中，数据库仍然会产生缓冲管理器的显著开销，缓冲管理器需要处理诸如页面固定与解固定、锁获取、缓冲表查找以及最近最少使用（LRU）管理等操作。

AlloyDB 的列存引擎是内置的内存缓存，以专门的、优化扫描的格式存储数据。

在此版本中，AlloyDB 可以使用列存引擎加速的 HNSW 来：

对于企业级应用，这不仅仅是一个更快的数据库问题——而是成本和质量问题：

降低基础设施成本：以显著更低的计算资源实现相同性能。

更佳的 AI 准确性：以以前仅可用于“草稿”（高速、低精度结果）质量搜索的速度，实现更高的召回率和质量。

无需更改应用程序：因为这是内置于 AlloyDB 中的，因此可以使用相同的标准 pgvector SQL 语法获得这些提升。

注意列存引擎确实会使用内存，但它经过高度压缩并经过精心管理。由于引擎以高效的列存格式存储向量数据，相对于巨大的性能提升，内存占用极小——这对于企业工作负载来说是一个非常有利的权衡。

要在 AlloyDB 中试用列存引擎加速的 HNSW，请访问： https://docs.cloud.google.com/alloydb/docs/ai/accelerate-with-ce，并按照以下步骤操作：

1. 启用列存引擎： https://docs.cloud.google.com/alloydb/docs/ai/accelerate-with-ce 并启用索引缓存

确保在 AlloyDB 实例中将 google_columnar_engine.enabled 和 google_columnar_engine.enable_index_caching 标志都设置为开启状态。

2. 将 HNSW 索引添加到列存引擎

一旦通过 pgvector 创建了 HNSW 索引，执行以下 SQL 命令将其缓存在列存引擎中：

初次使用 AlloyDB？通过 30 天免费试用了解 AlloyDB：https://docs.cloud.google.com/alloydb/docs/free-trial-cluster。

Google Colab 笔记本：https://colab.research.google.com/github/GoogleCloudPlatform/python-docs-samples/blob/main/alloydb/notebooks/columnar_engine_accelerated_hnsw_vector_search_benchmark.ipynb：一个端到端的 Python 脚本，用于导入 GloVe 数据集、创建索引，并绘制召回率与 QPS 曲线。

HNSW 是您用例中合适的向量索引选择吗？查看我们的《AlloyDB AI 中选择向量索引》指南：https://docs.cloud.google.com/alloydb/docs/ai/choose-index-strategy。

作者：Sirish Chandrasekaran • 阅读时间：4 分钟

The front door to AI in the workplace

AlloyDB ：https://cloud.google.com/alloydb is a fully managed, PostgreSQL-compatible database service built for your most demanding enterprise workloads. It combines the best of open source PostgreSQL with Google’s advanced technology, offering massive scalability, high availability, and native AI capabilities. It serves as a performant relational store, a unified backend for vector and full text search, and an analytics engine that is up to 100x faster than standard PostgreSQL.

Vector search is the foundation of modern AI and Retrieval Augmented Generation (RAG) applications. For developers using AlloyDB and other PostgreSQL databases, pgvector ：https://github.com/pgvector/pgvector is a widely adopted extension for storing, indexing, and querying vector embeddings, and HNSW (Hierarchical Navigable Small World) is a highly efficient graph-based algorithm designed for approximate nearest neighbor search across multi-layered structures. With columnar engine accelerated HNSW ：https://docs.cloud.google.com/alloydb/docs/ai/accelerate-with-ce in AlloyDB (now in preview), you can achieve up to 4x higher queries per second (QPS) for vector search compared to standard PostgreSQL HNSW.

Enterprise AI applications face a constant trade-off between speed and accuracy. When searching through millions or billions of vectors, maximizing Queries per Second (QPS) without sacrificing search quality (recall) is critical for scaling production workloads. The PostgreSQL pgvector extension offers HNSW as one of the indexes that can speed up Approximate Nearest Neighbor (ANN) searches. Let’s dive deep into how AlloyDB solves the speed vs. accuracy trade-off.

Note: While this post focuses on HNSW performance, it’s worth noting that HNSW is just one part of AlloyDB’s advanced vector toolkit. AlloyDB also features ScaNN ：https://docs.cloud.google.com/alloydb/docs/ai/create-scann-index —a cutting-edge index backed by over 14 years of Google Research—giving you the flexibility to choose the perfect index for your workload. Additionally, for use cases demanding absolute precision, standard k-nearest neighbor (KNN) search is always available for 100% recall. Check out our Choose a Vector Index Guide ：https://docs.cloud.google.com/alloydb/docs/ai/choose-index-strategy to see how they stack up.

AlloyDB offers a free trial instance ：https://cloud.google.com/alloydb/docs/free-trial-cluster for you to try AlloyDB with your own workload. In addition, new Google Cloud customers get $300 in free credits.

The AlloyDB columnar engine ：https://docs.cloud.google.com/alloydb/docs/columnar-engine/about is a built-in, in-memory cache that automatically stores frequently queried data in a specialized, scan-optimized columnar format. It allows AlloyDB to handle heavy analytical queries up to 100x faster than standard PostgreSQL. Additionally, it accelerates ANN searches by storing the index in memory, using a vectorized memory layout for fast traversals, and bypassing standard PostgreSQL buffer manager overhead.

To understand the real-world performance characteristics of columnar engine Accelerated HNSW, we plotted standard QPS vs Recall curves for the GloVe 100 Angular dataset by searching more than 1M records with a limit of 100.

Running this benchmark script ：https://colab.research.google.com/github/GoogleCloudPlatform/python-docs-samples/blob/main/alloydb/notebooks/columnar_engine_accelerated_hnsw_vector_search_benchmark.ipynb yields the following visualization:

Note: These measurements were taken on an AlloyDB C4A 16vCPU machine. Due to the inherent randomness in HNSW graph building, results may slightly vary across runs.

The data reveals two transformative benefits:

It is important to note that the baseline (blue line) already represents the index being fully cached in the PostgreSQL shared buffer cache. The performance gains shown here are not the result of moving data from disk to RAM, but rather the result of a more efficient memory architecture.

In standard PostgreSQL architectures, index operations utilize the shared buffer cache. Even when data is fully in-memory, the database still incurs significant overhead from the buffer manager, which must handle operations such as page pinning and unpinning, lock acquisition, buffer table lookups, and Least Recently Used (LRU) management.

AlloyDB's columnar engine is a built-in, in-memory cache that stores data in a specialized, scan-optimized format.

With this release, AlloyDB can use columnar engine accelerated HNSW to:

For enterprise-scale applications, this isn't just about a faster database—it's about cost and quality:

Reduced infrastructure costs: Achieve the same performance with significantly lower compute resources.

Better AI accuracy: Reach higher recall and quality at speeds that were previously only possible for "draft" (high-speed, lower-accuracy results) quality search.

No application changes required: Because this is built into AlloyDB, you get these gains using the same standard pgvector SQL syntax.

Note that the columnar engine does utilize memory, but it is highly compressed and meticulously managed. Because the engine stores vector data in an efficient columnar format, the memory footprint is minimal compared to the massive performance gains—making it a highly favorable trade-off for enterprise workloads.

To try out columnar engine accelerated HNSW ：https://docs.cloud.google.com/alloydb/docs/ai/accelerate-with-ce in AlloyDB, follow these steps:

1. Enable the columnar engine ：https://docs.cloud.google.com/alloydb/docs/ai/accelerate-with-ce and index caching

Ensure that both google_columnar_engine.enabled and google_columnar_engine.enable_index_caching flags are set to on for your AlloyDB instance.

2. Add the HNSW Index to columnar engine

Once your HNSW index is created via pgvector , execute the following SQL command to cache it in the columnar engine:

New to AlloyDB? Discover AlloyDB with a 30-day free trial ：https://docs.cloud.google.com/alloydb/docs/free-trial-cluster .

Google Colab Notebook ：https://colab.research.google.com/github/GoogleCloudPlatform/python-docs-samples/blob/main/alloydb/notebooks/columnar_engine_accelerated_hnsw_vector_search_benchmark.ipynb : An end-to-end Python script to ingest the GloVe dataset, create indexes, and plot Recall vs QPS curves.

Is HNSW the right vector index choice for your use case? Check our ‘ Choose a vector index in AlloyDB AI ：https://docs.cloud.google.com/alloydb/docs/ai/choose-index-strategy ’ guide.

By Sirish Chandrasekaran • 4-minute read