NVIDIA 发布 DeepStream 9.1，新增 13 项智能体技能与多视角 3D 追踪

NVIDIA 刚刚发布了 DeepStream 9.1：https://github.com/NVIDIA/DeepStream。此次更新针对视频分析中的一个长期存在的问题。传统上，要在多台摄像机之间跟踪同一个物体，通常需要手动摄像机校准和复杂的计算。DeepStream 9.1 通过两个新增功能解决了这一问题：多视角 3D 跟踪（MV3DT）和自动魔法校准（AutoMagicCalib，AMC）。这两者都作为编程代理的智能技能提供。因此，开发者可以更快地从概念阶段进入运行中的处理管线。

要理解此次更新，先从基础平台开始。DeepStream 是 NVIDIA 提供的用于 AI 视频和图像理解的流分析工具包。它提供了一个基于 GStreamer 的框架，用于在 NVIDIA GPU 上进行多流、多模型推理。处理管线结合了硬件加速的解码和编码、TensorRT 推理、物体跟踪以及消息中间件集成。

在此基础上，9.1 版本增加了五个显著的新功能：

在这些新增功能中，MV3DT 是主要技能，所以需要了解它的工作原理。核心上，MV3DT 将来自多台校准摄像机的检测结果投射到一个共享的 3D 坐标系中。然后，它将不同摄像机视角下对同一物体的观测进行关联。最后，为每个物体分配一个全局一致的对象 ID。

具体来说，数据流分四个阶段运行。对于检测，每个摄像机流运行一个对象检测器。MV3DT 开箱即用支持三种模型：

接下来，对于单目 3D 感知，每台摄像机使用存储在 YAML 校准文件中的 3×4 投影矩阵。该矩阵将二维边界框反投影到假设地面平面的三维世界坐标中。然后，在多视角关联阶段，追踪器通过 MQTT（消息队列遥测传输）共享 tracklet。MQTT 是一种轻量级的发布/订阅消息协议。当两台摄像机观察到同一人时，它会根据三维世界空间中的接近程度匹配 tracklet。

关联完成后，结果以三种形式输出。屏幕显示（OSD）呈现一个带有二维和三维边界框的网格图。俯视图（BEV）渲染一个自上而下的轨迹地图。Kafka 消息传递提供每帧的 protobuf 元数据，包括传感器 ID、对象 ID 以及三维边界框。

MV3DT依赖于校准摄像头，这传统上意味着使用棋盘格和停机时间。相反，AMC通过分析现有视频文件或流中的跟踪对象来校准网络。它会估计每个摄像头的内参（焦距、主点、镜头畸变），还会估计外参（旋转、平移、世界位置）。

在底层，该流程运行五个阶段。这些阶段是每个摄像头的轨迹提取、单视图矫正、多视图小轨迹匹配、束调整和可选的VGGT精炼。VGGT（视觉几何基础转换器）在物体运动有限时提供帮助。AMC作为微服务运行，提供REST API和网页界面。用户只需提供布局图像和几个对齐点。

在定义了MV3DT和AMC之后，交付机制就是技能本身。无需编辑配置文件，你只需用自然语言描述意图。这些技能可与Claude Code、Codex、Cursor及类似代理协作。设置过程简短：

启动代理后，通过单个提示即可运行参考应用：

从这里，MV3DT技能会验证先决条件，拉取容器，并安装Kafka和Mosquitto代理服务。它还会下载模型权重，生成流水线配置，并启动跟踪。值得注意的是，如果校准文件缺失，它会自动触发AMC技能。

作为背景，下表显示了各版本之间的变化。

基于这些能力，功能对应到具体部署：

要查看机制，下面的嵌入式演示展示了一个人在三个摄像头视野间行走的动画。切换查看单摄像头二维跟踪和MV3DT三维融合，即可观察物体ID保持一致。

需要与我们合作来推广您的GitHub仓库或Hugging Face页面或产品发布或网络研讨会等吗？请与我们联系：https://forms.gle/wbash1wF6efRj8G58

Asif Razzaq 是 Marktechpost Media Inc. 的首席执行官。作为一个有远见的企业家和工程师，Asif 致力于利用人工智能的潜力为社会带来积极影响。他最近的努力是推出人工智能媒体平台 Marktechpost，该平台以对机器学习和深度学习新闻的深入报道而著称，这些报道既技术上可靠，又易于广大受众理解。该平台每月浏览量超过 200 万次，显示了其在观众中的受欢迎程度。

构建一个自主事件场地运营商 [完整代码]：https://pxllnk.co/twdn5

谢谢！我们的团队会尽快与您联系

NVIDIA just released DeepStream 9.1：https://github.com/NVIDIA/DeepStream . The update targets a persistent problem in video analytics. Tracking one object across many cameras traditionally requires manual camera calibration and complicated calculations. DeepStream 9.1 addresses this with two additions: Multi-View 3D Tracking (MV3DT) and AutoMagicCalib (AMC). Both ship as agentic skills for coding agents. As a result, developers move from concept to a running pipeline faster.

To understand the update, start with the base platform. DeepStream is NVIDIA’s streaming analytics toolkit for AI-based video and image understanding. It provides a GStreamer-based framework for multi-stream, multi-model inference on NVIDIA GPUs. Pipelines combine hardware-accelerated decoding and encoding, TensorRT inference, object tracking, and message-broker integration.

Building on that base, version 9.1 adds five notable items:

Among those additions, MV3DT is the main skill, so consider how it works. At its core, MV3DT projects detections from multiple calibrated cameras into a shared 3D coordinate system. It then associates observations of the same object across camera views. Finally, it assigns one globally consistent object ID.

Concretely, the data flow runs in four stages. For detection, each camera stream runs an object detector. MV3DT supports three models out of the box:

Next, for monocular 3D perception, each camera uses a 3×4 projection matrix stored in a YAML calibration file. This back-projects 2D bounding boxes into 3D world-space coordinates using a ground-plane assumption. Then, for multi-view association, the tracker shares tracklets using Message Queuing Telemetry Transport (MQTT). MQTT is a lightweight pub/sub messaging protocol. When two cameras observe the same person, it matches tracklets by proximity in 3D world space.

After association, results stream out in three forms. The On-Screen Display (OSD) shows a tiled grid with 2D and 3D bounding boxes. The Bird’s-Eye View (BEV) renders a top-down trajectory map. Kafka messaging delivers per-frame protobuf metadata, including sensor ID, object ID, and 3D bounding box.

MV3DT depends on calibrated cameras, which traditionally means checkerboards and downtime. Instead, AMC calibrates a network by analyzing tracked objects in existing video files or streams. It estimates each camera’s intrinsic parameters (focal length, principal point, lens distortion). It also estimates extrinsic parameters (rotation, translation, world position).

Under the hood, the pipeline runs five stages. These are per-camera trajectory extraction, single-view rectification, multi-view tracklet matching, bundle adjustment, and optional VGGT refinement. VGGT (Visual Geometry Grounded Transformer) helps when object movement is limited. AMC runs as a microservice with REST APIs and a web interface. Users supply only a layout image and a few alignment points.

With MV3DT and AMC defined, the delivery mechanism is the skills themselves. Rather than editing configuration files, you describe intent in natural language. The skills work with Claude Code, Codex, Cursor, and similar agents. Setup is short:

After launching the agent, a single prompt runs the reference app:

From there, the MV3DT skill validates prerequisites, pulls the container, and installs Kafka and Mosquitto broker services. It also downloads model weights, generates the pipeline config, and launches tracking. Notably, if calibration files are missing, it triggers the AMC skills automatically.

For context, the table below shows what changed between releases.

Given these capabilities, the features map to concrete deployments:

To see the mechanism, the embedded demo below animates one person walking between three camera fields of view. Toggle between naive per-camera 2D tracking and MV3DT 3D fusion to watch the object ID stay consistent.

Need to partner with us for promoting your GitHub Repo OR Hugging Face Page OR Product Release OR Webinar etc.? Connect with us ：https://forms.gle/wbash1wF6efRj8G58

Asif Razzaq is the CEO of Marktechpost Media Inc.. As a visionary entrepreneur and engineer, Asif is committed to harnessing the potential of Artificial Intelligence for social good. His most recent endeavor is the launch of an Artificial Intelligence Media Platform, Marktechpost, which stands out for its in-depth coverage of machine learning and deep learning news that is both technically sound and easily understandable by a wide audience. The platform boasts of over 2 million monthly views, illustrating its popularity among audiences.

Build an Agentic Event Venue Operator [Full Codes]：https://pxllnk.co/twdn5

Thanks! Our team will contact you soon