LangChain SmithDB 全文搜索：为对象存储设计倒排索引，Agent 追踪中位延迟 400ms

SmithDB 支持对代理追踪执行全文搜索和 JSON 过滤，P50（中位数）延迟为 400 毫秒，即便底层数据由存储在对象存储中的大型、深层嵌套 JSON 文档组成。

全文搜索是成熟的技术领域。Lucene 已有二十年的历史；Tantivy：https://github.com/quickwit-oss/tantivy 和 Quickwit：https://quickwit.io/blog/quickwit-101 已经将搜索和索引移到了对象存储上。然而，在将文本搜索集成到 SmithDB 时，我们决定从基本原理出发处理问题，因为为搜索工作负载索引代理追踪具有独特挑战。

每个 LangSmith 事件将字段 inputs 和 outputs 编码为其总字节的绝大多数。输入和输出的 payload 大小超过 1 MB 很常见，其中一些未压缩的数据可达数百兆字节。这些内容列在数量级上远超身份、时间戳及其他元数据列。

此外，如原 SmithDB 博客文章所述：https://www.langchain.com/blog/introducing-smithdb#agents-present-a-new-data-problem，与代理追踪相关的 payload 随时间持续增大。这是 LLM 上下文窗口变大以及代理运行时间延长的直接结果，导致 LLM 积累了更多上下文。

这些特性颠倒了传统搜索索引的经济学。传统日志引擎索引数十亿个小文档，因此索引相对于每个文档而言很小。我们索引数十亿个庞大文档，其中一个文档可能生成的索引数据比许多小日志行还多。通常：https://aws.amazon.com/blogs/big-data/amazon-opensearch-service-101-how-many-shards-do-i-need/ 日志的源数据:索引比约为 1:1.25。然而对于 LangSmith 中的代理追踪，我们观察到平均比率更接近 1:1.9。由此得出三点结论：

倒排索引阻止了内容查询执行全量 payload 扫描，并且它必须承载重负载、偏斜的半结构化 payload。

SmithDB 将所有持久数据保存在对象存储中，因此计算相对无状态，系统通过增加节点进行扩展，而无需管理本地磁盘。

查询的成本大致与（对对象存储发出的请求次数）×（每个请求读取的字节数）成正比。在对象存储上：

SmithDB 的倒排索引的每一个方面，从存储布局到查询执行，都是在考虑这些限制的情况下设计的。

在深入了解倒排索引的存储布局之前，让我们先回顾索引需要处理的主要查询模式。SmithDB 的查询范围归结为三类谓词，它们在匹配对象和允许的模式语法上有所不同。

每一节后文都会引用此表：当我们说“仅路径查询”时，我们指 json_key，“键值查询”指 json_key_search，而“全文搜索”指 search。

倒排索引是驱动每个搜索库的数据结构，从 Lucene：https://lucene.apache.org/ 到 Tantivy：https://github.com/quickwit-oss/tantivy。它就像教科书后面的索引：查找一个术语然后直接跳到提到它的页面，而不必阅读每一页。SmithDB 基于这个理念，并为其在对象存储中存储的大型 agent-trace 数据专门优化了存储布局。

倒排索引结构基于三个概念：

拿五条按文本建立索引的 trace 举例：

索引为每个术语保留一个条目，指向包含该术语的文档：

每个术语对应一个字典条目：查找该值，读取它的 posting list，你就能准确知道要获取哪些文档。像 search("deep agents") 这样的查询会取 deep ([1, 2, 3, 4]) 和 agents ([0, 1, 2, 3]) 的 posting list 的交集得到 [1, 2, 3]，无需扫描数据内容。

positions 列记录每个文档中该术语出现的 token 偏移，例如 1:[0] 表示文档 1 的位置 0。这就是短语搜索成为可能的原因：search("langsmith engine") 能匹配文档 1，因为 langsmith 在偏移 0 处而 engine 在偏移 1（0 + 1 == 1），但不能匹配文档 4，因为 powers 和 the 居中之间（langsmith 在 1，engine 在 4）。

Tantivy：https://github.com/quickwit-oss/tantivy 是一个出色的搜索索引库，也是 Rust 中 Lucene 风格搜索的显而易见的参考点。我们一开始问自己是否可以直接采用它。我们最终设计的方案深受 Tantivy 的启发，但一些限制使它直接用于我们的用例变得不太合适：

Vortex：https://vortex.dev/ 是 SmithDB 用于对象存储的可扩展列式文件格式。与 Parquet 等固定格式不同，Vortex 允许可插拔的编码和自定义文件布局，这使我们能够根据工作负载调整压缩和 I/O 访问模式，而无需分叉文件格式。

每次读取都会使用统计信息裁剪整个行组，将生存的行过滤为一个掩码，并只投影查询实际需要的列。

Vortex：https://docs.vortex.dev/developer-guide/internals/io 文件中的 I/O 单位是一个段：一个连续的物理字节范围。在对象存储上往返大约需要 100 毫秒，因此查询延迟的主要杠杆是最小化请求次数。Vortex 的 I/O 调度程序会将相邻的段读取合并为一个请求，将 1 MB 间隙内的读取合并成一个请求，最多可达 16 MB 的窗口，因此索引中的顺序访问模式对应非常少的对象存储 GET 操作。

第一个版本几乎是教科书倒排索引的直译。两个列（term_key 用于路径，term_value 用于令牌）使得一个布局可以服务于三种查询形式：路径存在读取 term_key，键控搜索在两列之间交叉 postings，全文搜索只在 term_value 上交叉。Postings 存储为 List cells，位置存储为 List >。

我们依赖 Vortex 的默认设置：term 列使用 FSST 编码，postings 和 positions 使用 bitpacked 编码，以及允许查询时裁剪的分区存储布局。仅位置（用于短语搜索）就比其他每一列大一个数量级，因此我们将索引保存在与核心运行数据分开的文件中。这让我们能够将索引构建和合并与核心写路径解耦。Vortex 的 API 基于行索引和掩码，因此将索引过滤委托给兄弟文件是一种自然的组合方式。

我们的v2布局通过将组织单元从“每行组N个术语”改为基于字节预算的行组，并通过管理每列的字节布局而不是直接依赖Vortex默认值，解决了所有v1问题。本节的其余部分将讲解新的组织单元、其内部内容以及获得字节预算的编码选择。

由于行组是修剪和I/O的单位，我们使用固定的独立字节预算而不是固定行数来确定行组大小。

按字节而不是按术语计数进行大小调整，这解决了v1的第三个问题。术语偏斜使得术语计数成为I/O大小的糟糕代理，因为v1行组中的一个高频术语可能会将其压缩后推到甚至超过500MB。字节预算为每个行组提供了从对象存储获取字节或执行查询时的内存占用的上限。

通过术语列上的分区存储布局，每行组的最小/最大/计数信息允许查询计划器在访问FST之前跳过整个行组。对于针对特定前缀的路径查询，这是最大节省：大多数行组在谓词范围内根本没有内容。

每个行组包含四列（其中三列用于term_key，跳过位置列）：

一次查找包括一次字典遍历、一次偏移表读取和一次字节范围获取。FST将术语解析为序数。序数索引到term_info，term_info给出postings的偏移量，以及（对于短语查询）positions的偏移量。查询直接读取这些字节范围。无需载荷扫描，无需嵌套列表解码，并且由于每列都是自有的分块布局，非短语查询只获取term + term_info + postings，从不打开positions列。

我们在术语字典中使用FST。我们在一个具有2.79M术语出现的代表性行组上，将FST与明显的替代方案（Vortex默认的FSST字符串编码、前缀共享的keep_add和普通zstd）进行比较。优势的大小取决于基数：

在 term_key 上，当几百个 JSON 路径在数百万行中重复时，FST 将整个字典压缩到 3.8 KiB：比原始字节小四个数量级，比 zstd 小约 4 倍。在高基数的 term_value 列上，FST 大约比 zstd 大 1.5 倍，但仍然优于 FSST。关键点在于 zstd 是不透明的：每次查找都需要解压块。而 FST 本身就是索引——精确查找、前缀和范围扫描，以及自动机遍历（LIKE、模糊、正则）都直接在压缩字节上运行，成本为 O(|term|)，无需哈希。

我们还将键控搜索和全文查询形态折叠到每个行组的单个 FST 中，通过将 term_value 条目存储为 {token}\0{flattened_path}。键控搜索变为精确的 FST 查找；全文搜索变为 token\0 的前缀扫描，遍历该 token 出现的每条路径。

我们对每个 term 使用块位打包增量。索引和位置都使用相同的 Tantivy/Lucene 风格的两级编码。编码的形状使每个 term 的控制成为可能，也使合并变得廉价。

每个 term 列表分成固定的 128 元素块，加上一个剩余元素的尾块：

在一个块内，我们存储连续文档 ID 之间的增量，而不是 ID 本身，并将块按最小宽度位打包以适应其最大增量。一条密集、规则的 ID 序列每个 ID 只需几位。尾部的部分块（对于高频 term 定义上较少，对于低频 term 则是整个 posting 列表）回退到 VInt，每个小增量约 1 字节，在长尾条件下渐进降级。

它产生了两个 v1 列表编码所没有的属性：

Tantivy 也利用 FST，但为每个段构建一个 FST 并进行分片分区。我们为每个行组构建一个 FST。行组大小的 FST 足够小，写入器可以流式处理，而无需在内存中持有段级 FST，且区域级修剪在查询时进行任何 FST 工作之前就会跳过大多数行组。权衡在于一次查找可能会触及每个文件的多个 FST，但修剪在实践中使这种成本很少；存活的 FST 足够小，使得遍历成本低。

在第二部分中，我们将探讨如何实现倒排索引的构建与合并，以及如何在读取路径中利用该索引。在这里阅读第二部分：https://www.langchain.com/blog/full-text-search-in-smithdb-constructing-and-querying-our-inverted-index-pt-2

我们正在构建 SmithDB，以解决与代理可观测性相关的系统问题。如果这种基础设施工作听起来有趣，我们正在招聘：https://www.langchain.com/careers#explore-jobs。

LangSmith，我们的代理工程平台，帮助开发者调试每一个代理决策、评估更改，并一键部署。

SmithDB supports full-text search and JSON filtering over agent traces with a median (P50) latency of 400 ms, even though the underlying data consists of large, deeply nested JSON documents stored in object storage.

Full-text search is well-trodden ground. Lucene is two decades old; Tantivy：https://github.com/quickwit-oss/tantivy and Quickwit：https://quickwit.io/blog/quickwit-101 have already pushed search and indexing onto object storage. However, when building text search into SmithDB, we decided to approach the problem from first principles because indexing agent traces for search workloads presents a unique challenge.

Every LangSmith event encodes the fields inputs and outputs as the overwhelming majority of its total bytes. 1 MB+ payload sizes for inputs and outputs are common, with some of these stretching to hundreds of megabytes uncompressed. These content columns dwarf the identity, timestamp, and other metadata columns by orders of magnitude.

Additionally, as mentioned in the original SmithDB blogpost：https://www.langchain.com/blog/introducing-smithdb#agents-present-a-new-data-problem, the payloads associated with agent traces continue to increase in size over time. This is direct result of LLM context window sizes growing larger and agents running for longer time horizons, causing LLMs to accumulate more context.

These characteristics invert the usual economics of a search index. A traditional log engine indexes billions of small documents, so the index is small relative to each document. We index billions of enormous documents, where one document can produce more index data than many small log lines. Typically：https://aws.amazon.com/blogs/big-data/amazon-opensearch-service-101-how-many-shards-do-i-need/ the source:index ratio for logs is about to 1:1.25. However for agent traces in LangSmith, we observed the average to be closer to 1:1.9. Three things follow:

An inverted index is what prevents a content query from performing a full payload scan, and it has to absorb heavy, skewed, semi-structured payloads.

SmithDB keeps all durable data in object storage so compute is relatively stateless and the system scales by adding nodes without having to manage local disks.

The cost of a query is roughly proportional to (requests issued to object storage) × (bytes read per request) . On object storage:

Every aspect of SmithDB’s inverted index, from its storage layout to query execution is designed with these constraints in mind.

Before going deeper into the storage layout for our inverted index, let’s go over the main query patterns the index has to answer. SmithDB's query surface boils down to three predicate families, and they differ in what they match against and what pattern syntax they admit.

Every later section refers back to this table: when we say "path-only query" we mean json_key, "keyed value" means json_key_search , and "full-text" means search .

An inverted index is the data structure powering every search library, from Lucene：https://lucene.apache.org/ to Tantivy：https://github.com/quickwit-oss/tantivy. It is like the index at the back of a textbook: look up a term once and jump straight to the pages that mention it instead of reading every page. SmithDB builds on this idea and specializes the storage layout for the large agent-trace payloads it stores in object storage.

The inverted index structure rests on three concepts:

Take five traces indexed on their text:

The index keeps one entry per term, pointing at the documents that mention it:

Each term is one dictionary entry: look up the value, read its posting list, and you know exactly which documents to fetch. A query like search("deep agents") intersects the posting lists for deep ( [1, 2, 3, 4] ) and agents ( [0, 1, 2, 3] ) to get [1, 2, 3] with no payload scan.

The positions column records, per document, the token offset(s) where the term appears, e.g. 1:[0] means doc 1, position 0. That is what makes phrase search possible: search("langsmith engine") matches doc 1 because langsmith is at offset 0 and engine at offset 1 ( 0 + 1 == 1 ), but not doc 4, where powers and the sit between them ( langsmith at 1, engine at 4).

Tantivy：https://github.com/quickwit-oss/tantivy is an excellent search indexing library and the obvious reference point for Lucene-style search in Rust. We started by asking whether we could adopt it directly. The design we ended up with is heavily inspired by Tantivy, but a few constraints made it an awkward fit for our use-case directly:

Vortex：https://vortex.dev/ is an extensible and columnar file format SmithDB uses for object storage. Unlike fixed formats such as Parquet, Vortex allows pluggable encodings and custom file layouts which lets us tailor compression and I/O access patterns to our workload without forking the file format.

Every read prunes entire row groups using statistics, filters surviving rows down to a mask, and projects only the columns the query actually needs.

The unit of I/O in a Vortex：https://docs.vortex.dev/developer-guide/internals/io file is a segment: a contiguous physical byte range. On object storage a round-trip costs roughly 100 ms, so the primary lever for query latency is minimizing the number of requests. Vortex's I/O scheduler coalesces nearby segment reads into a single request, merging reads within a 1 MB gap into one, up to a 16 MB window, so sequential access patterns in the index map to very few object store GETs.

The first version was a near-literal translation of the textbook inverted index. Two columns ( term_key for paths and term_value for tokens) lets one layout serve all three query shapes: path-existence read term_key , keyed search intersected postings across both columns, and full-text intersected on term_value alone. Postings were stored as List cells, positions as List > .

We leaned on Vortex's defaults: FSST encoding for the term columns, bitpacked encoding for postings and positions, and a zoned storage layout that allowed for pruning at query time. Positions (required for phrase search) alone were an order of magnitude larger than every other column, so we kept the index in a separate file from the core run data. This let us decouple index construction and merge from the core write path. Vortex's APIs work on row indices and masks, so delegating index filtering to a sibling file composed naturally.

Our v2 layout addresses all three v1 problems by changing the unit of organization from "N terms per row group" to a byte-budgeted row group , and by owning the byte layout per column instead of directly relying on Vortex defaults. The rest of this section walks through the new unit of organization, what lives inside it, and the encoding choices that earned the byte budget.

Since a row group is the unit of pruning and I/O, we determine the row group sizes with fixed independent byte budgets instead of a fixed row count.

Sizing in bytes , not term count, is what fixes v1's third problem. Term skew makes term count a poor proxy for IO size, as one high-frequency term in a v1 row group could push it past even 500 MB compressed. The byte budgets give us an upper bound on every row group for the amount of bytes fetched from object store or memory footprint while executing queries.

Per-row-group min/max/count via a zoned storage layout on the term column lets the query planner skip entire row groups before touching the FST. For path queries that target a specific prefix this is the single biggest saving: most row groups simply don't contain anything in the predicate's range.

Each row group carries four columns (three for term_key , which skips positions):

A lookup is one walk through the dictionary, one offset table read, and one byte-range fetch. The FST resolves the term to an ordinal. The ordinal indexes into term_info , which gives an offset into postings and (for phrase queries) an offset into positions . The query reads those byte ranges directly. No payload scan, no nested-list decode, and because each column is its own chunked layout, a non-phrase query fetches just term + term_info + postings and never opens the positions column.

We use FST for the term dictionary. We compared FST against the obvious alternatives (Vortex's default FSST string encoding, prefix-shared keep_add , and plain zstd) on a representative row group with 2.79M term occurrences. The shape of the win depends on cardinality:

On term_key , where a few hundred JSON paths repeat across millions of rows, the FST collapses the entire dictionary to 3.8 KiB: four orders of magnitude smaller than the raw bytes and ~4× smaller than zstd. On the high-cardinality term_value column, FST is ~1.5× larger than zstd but still beats FSST. The crucial point is that zstd is opaque : every lookup requires decompressing the block. The FST is the index itself — exact lookup, prefix and range scans, and automaton walks ( LIKE , fuzzy, regex) all run directly against the compressed bytes with O(|term|) cost and no hashing.

We also fold the keyed-search and full-text query shapes into a single FST per row group by storing term_value entries as {token}\0{flattened_path} . Keyed search becomes exact FST lookup; full-text search becomes a prefix scan on token\0 , walking every path the token appears under.

We use block-bitpacked deltas per term. Postings and positions both use the same Tantivy/Lucene-style two-tier encoding. The shape of the encoding is what makes per-term control possible and what makes merge cheap.

Each per-term list is split into fixed 128-element blocks plus a tail of leftovers:

Within a block we store deltas between successive doc IDs, not the IDs themselves, and bitpack the block to the minimum width that fits its max delta. A dense, regular run of IDs packs down to just a few bits each. The trailing partial block (by definition rare for high-frequency terms, and the entire posting list for low-frequency ones) falls back to VInt , ~1 byte per small delta, degrading gracefully on the long tail.

Two properties fall out of it that the v1 List encoding didn't have:

Tantivy also leverages FSTs, but builds one FST per segment with sharded partitioning. We build one FST per row group. A row-group-sized FST is small enough that the writer streams through it without ever holding a segment-wide FST in memory, and zone-level pruning skips most row groups before any FST work happens at query time. The trade-off is that a single lookup may touch multiple FSTs per file, but pruning makes that cost rare in practice; the surviving FSTs are small enough that the walks are cheap.

In part 2, we’ll explore how we implemented inverted index construction and merging, as well as how we leverage the index in our read path. Read part 2 here.：https://www.langchain.com/blog/full-text-search-in-smithdb-constructing-and-querying-our-inverted-index-pt-2

We’re building SmithDB to solve the systems problems that come with agent observability. If that kind of infrastructure work sounds interesting, we’re hiring ：https://www.langchain.com/careers#explore-jobs .

LangSmith, our agent engineering platform, helps developers debug every agent decision, eval changes, and deploy in one click.