LangChain 与 Airbyte 集成：让数据摄取达到生产级就绪

在过去几个月中，我们的一个重要关注点是使团队能够从原型走向生产。将他们在一小时内开发的应用程序带到可以可靠使用的状态。可以说，LangChain 帮助实现的最大类别的应用程序是基于检索的应用程序（即将大语言模型连接到您自己的数据）。要将基于检索的应用程序从原型推向生产，需要一些条件。

其中一个组成部分是与数据查询相关的所有内容。这就是我们推出 LangSmith 的原因：https://blog.langchain.com/announcing-langsmith/ - 旨在帮助调试和监控大语言模型如何与用户查询以及检索到的文档交互。另一个重要方面是查询算法及其用户体验——这就是为什么我们在推动像 Conversational Retrieval Agents 这样的功能：https://blog.langchain.com/conversational-retrieval-agents/。（如果您对这一部分特别感兴趣，我们将在 8 月 9 日举办关于“高级检索”的网络研讨会：https://www.crowdcast.io/c/kqz7nl8nps42?ref=blog.langchain.com）。第三个——可以说是最重要的部分——是数据获取逻辑本身。将应用程序投入生产时，您希望它连接的数据能够按某种计划可靠且高效地刷新。

我们首次尝试解决这一问题，是与 Airbyte 进行另一种更深度的集成。此前的 Airbyte 集成：https://reference.langchain.com/python/langchain-community/document_loaders/airbyte 展示了如何将他们的一个数据源用作 LangChain 中的文档加载器。此次集成则是相反方向，并在 Airbyte 中添加了一个 LangChain 目标。

想了解更多关于此次集成的信息，可以查看 Airbyte 的发布博客：https://airbyte.com/blog/airbyte-now-supports-vector-databases-powered-by-langchain?ref=blog.langchain.com。我们尽量不重复博客内容，而是讲述为什么我们认为这是一个重要的步骤。

LangChain 自身提供了“数据源”和“目标”——我们拥有数百个文档加载器和 50 多个向量存储/检索器集成。但这远不是相互替代，而是一种互利的集成，为社区带来了诸多好处。

首先，Airbyte 提供了更多数百个数据源：https://docs.airbyte.com/integrations/?ref=blog.langchain.com，一个强大的编排逻辑，以及工具：https://docs.airbyte.com/connector-development/connector-builder-ui/overview?ref=blog.langchain.com 来创建更多的数据源。让我们关注编排逻辑。当你创建一个可以访问你数据索引的聊天机器人时，你不仅仅是将数据索引一次然后就不管它。你希望按照某种计划重新索引，以保持数据的最新状态。这类数据管道正是 Airbyte 擅长且一直在构建的。

其次，数据摄取过程不仅仅是将数据从源移动到目标。还有一些重要的、非平凡且复杂的转换，这是实现有效检索所必需的。最重要的两个是——文本拆分和嵌入。

文本拆分很重要，因为你需要创建数据块以放入向量存储。你希望这些数据块本身在语义上是有意义的——这样在检索时才有意义。这就是为什么它往往比简单地每1000个字符拆分一次文本要复杂一些。LangChain 提供了15种以上不同文本拆分方式的实现，由不同算法驱动，并针对不同文本类型（Markdown 与 Python 代码等）进行优化。为了帮助探索这些不同的文本拆分器提供的功能，我们开源了：https://github.com/langchain-ai/text-split-explorer?ref=blog.langchain.com 并托管了：https://langchain-text-splitter.streamlit.app/?ref=blog.langchain.com 一个便于探索的实验平台。

嵌入对于实现这些数据块的检索非常重要，这通常是通过比较用户查询的嵌入与已摄取文档的嵌入来完成的。有许多不同的嵌入提供商和托管平台——LangChain 提供了对50多种嵌入的集成。

总体而言，我们对这个 LangChain 与 Airbyte 的集成感到非常兴奋。它为摄取任务提供了稳健的编排和调度，同时利用了 LangChain 的转换逻辑和集成。我们也认为，还有更多功能（和集成）可以添加，使数据摄取达到生产级别——在接下来的几周内，敬请关注更多相关更新。

LangSmith，我们的代理工程平台，帮助开发者调试每个代理决策，评估更改，并一键部署。

A big focus of ours over the past few months has been enabling teams to go from prototype to production. To take apps they developed in an hour and get them into a place where they can actually be reliably used. Arguably the biggest category of applications LangChain helps enable is retrieval based applications (where you connect LLMs to your own data). There are a few things that are needed to take retrieval based applications from prototype to production.

One component of that is everything related to the querying of the data. That’s why we launched LangSmith：https://blog.langchain.com/announcing-langsmith/ - to help debug and monitor how LLMs interact with the user query as well as the retrieved documents. Another huge aspect is the querying algorithms and UX around that - which is why we’re pushing on things like Conversational Retrieval Agents：https://blog.langchain.com/conversational-retrieval-agents/. (If you are interested in this part in particular, we’re doing a webinar on “Advanced Retrieval：https://www.crowdcast.io/c/kqz7nl8nps42?ref=blog.langchain.com” on August 9th). A third - and arguably the most important part - is the ingestion logic itself. When taking an application into production, you want the data it’s connecting to be refreshed on some schedule in a reliable and efficient way.

Our first stab at tackling this is another, deeper integration with Airbyte. The previous Airbyte integration：https://reference.langchain.com/python/langchain-community/document_loaders/airbyte showed how to use one of their sources as a Document Loader within LangChain. This integration goes the other direction, and adds a LangChain destination within Airbyte.

To read more about this integration, you can check out Airbyte’s release blog here：https://airbyte.com/blog/airbyte-now-supports-vector-databases-powered-by-langchain?ref=blog.langchain.com. We will try not to repeat too much of that blog, but rather cover why we think this is an important step.

LangChain provides “sources” and “destinations” of our own - we have hundreds of document loaders and 50+ vectorstore/retriever integrations. But far from being replacements for one another, this is rather a mutually beneficial integration that provides a lot of benefits for the community.

First, Airbyte provides hundreds more sources：https://docs.airbyte.com/integrations/?ref=blog.langchain.com, a robust orchestration logic, as well as tooling：https://docs.airbyte.com/connector-development/connector-builder-ui/overview?ref=blog.langchain.com to create more sources. Let’s focus on the orchestration logic. When you create a chatbot that has access to an index of your data, you don’t just want to index your data there once and forget about it. You want to reindex it on some schedule, so that it stays up to date. This type of data pipelines is exactly what Airbyte excels at and has been building.

Second, the ingestion process isn’t only about moving data from a source to a destination. There’s also some important, non-trivial and nuanced transformations that are necessary to enable effective retrieval. Two of the most important - text splitting and embedding.

Splitting text is important because you need to create chunks of data to put in the vectorstore. You want these chunks to be semantically meaningful by themselves - so that they make sense when retrieved. This is why it’s often a bit trickier than just splitting a text every 1000 characters. LangChain provides implementations of 15+ different ways to split text, powered by different algorithms and optimized for different text types (markdown vs Python code, etc). To assist in the exploration of what these different text splitters offer, we've open-source：https://github.com/langchain-ai/text-split-explorer?ref=blog.langchain.com and hosted：https://langchain-text-splitter.streamlit.app/?ref=blog.langchain.com a playground for easy exploration.

Embeddings are important to enable retrieval of those chunks, which is often done by comparing embeddings of a user query to embeddings of ingested documents. There are many different embedding providers and hosting platforms - and LangChain provides integrations with 50+ of them.

Overall, we’re really excited about this LangChain - Airbyte integration. It provides robust orchestration and scheduling for ingestion jobs while leveraging LangChain’s transformation logic and integrations. We also think there’s more features (and integrations) to add to make data ingestion production ready - keep on the lookout for more of those over the next few weeks.

LangSmith, our agent engineering platform, helps developers debug every agent decision, eval changes, and deploy in one click.