Reimagining Independence： How Meta's AI Models Are Helping the University of Pittsburgh Transform Assistive Robotics

在匹兹堡的中心，一场静悄悄的革命正在进行——它有望重新定义残障人士的独立意义。这场运动的核心是人类工程研究实验室（HERL）：https://www.herl.pitt.edu/，这是匹兹堡大学的一个先锋研究机构，目前正在领导一项获得美国健康先进研究项目署（ARPA-H）资助高达4150万美元的项目：https://arpa-h.gov/。ARPA-H是美国设立的研究资助机构，旨在支持变革性的生物医学和健康突破。在ARPA-H的支持下，HERL团队与创新辅助技术的先驱ATDev：https://assistivetech.dev/，正在建设“为残障人士提供独立性的机器人辅助移动与操作平台（RAMMP）”，融合尖端机器人技术、人工智能和以用户为中心的设计。

对于美国估计约有550万轮椅使用者：https://www.volpe.dot.gov/news/us-dot-volpe-center-human-factors-study-informs-rule-air-travelers-disabilities，保持行动能力意味着个体机会和日常生活参与。当辅助技术无法反映现实世界的复杂性时，它可能削弱自信、独立性和身体安全。统计数据令人深思：每年美国急诊科治疗超过10万起与轮椅相关的伤害：https://www.pittwire.pitt.edu/features-articles/2025/11/04/advanced-research-projects-agency-health-rammp-herl，这些伤害通常是由于绊倒和跌倒造成的。因此，对于更智能、更安全、更适应性强的技术的需求迫在眉睫。

RAMMP：辅助移动的新纪元

由ARPA-H支持的机器人辅助移动与操作平台（RAMMP：https://arpa-h.gov/explore-funding/awards/2771）项目旨在通过整合先进的机器人技术、新型操作系统和数字孪生技术来解决当前机器人移动平台设计中的不足——创建一个虚拟模拟环境，用于安全、可扩展的测试和开发。RAMMP的方法还整合了人工智能，包括使用Meta的几种开源AI视觉模型，如DINO：https://ai.meta.com/research/dinov3/ 和 Segment Anything Model (SAM)：https://ai.meta.com/research/sam3/。

DINO和SAM已经在从医疗成像到野生动物保护的各种项目中提供了帮助。DINO是一种自监督视觉Transformer，擅长从未标注的数据中学习视觉表示，非常适合标注数据稀缺的环境。另一方面，SAM被设计为“分割任何物体”，能够在图像或视频中以最少提示识别并勾画任何对象。Segment Anything的多功能性和准确性使其成为精确对象识别至关重要领域的基础工具。

为现实世界工程设计：在边缘设备上运行DINOv3和SAM

对于依赖辅助设备的人而言，每一秒都至关重要。日常环境的不可预测性，例如孩子冲向人行道、突然出现的路缘或掉落的一串钥匙，都需要立即反应。在设备上直接处理摄像头图像和传感器数据，这被称为边缘计算，使机器人移动平台能够作为响应性工具发挥作用。这些工具必须在用户生活的各种动态环境中保持稳健和一致。

同时，在有限的电池供电硬件上部署像DINOv3和SAM这样强大的AI模型也带来了重大工程挑战。现实世界的部署必须考虑实际因素，包括电池寿命、散热、不可靠的网络连接，以及严格的尺寸和重量要求。

然而，如果最终用户无法使用这些新的机器人系统完成日常生活活动，那么克服这些限制对他们来说毫无意义。这些新方法允许用户以更自然的方式与机器人互动，利用其即时环境作为上下文，从而使工程师和用户都无需设计和操作复杂、耗时的界面。结合图像数据使用自然语言查询用户和机器人环境并提供更直接的指令的能力，直接降低了认知负荷，并减少了用户在执行诸如从桌子上拿起杯子这样简单操作时所需的上下文切换次数。

RAMMP 团队已经将此功能集成到他们的首个原型中。利用基于 DINO 构建的工具来查询机器人的图像传感器，以检测自动门按钮、杯子以及导航辅助用的路缘/地面。随着此功能现在可以进行实际环境测试，工程师正专注于语音和触控输入，使用户能够选择并与其周围的特定物体进行交互。除了确保模型输出的准确性和时间一致性的现有挑战之外，这还带来了确保在用户提示和输入中保持稳健性和可预测性的额外挑战。

DINOv3 作为设备的紧凑高效的“视觉大脑”——一个通用基础，可在其上叠加针对特定任务的轻量级模块以执行诸如物体检测或动作跟踪等操作，从而实现视觉数据的重用并节约能耗。

在机器人系统开发中应用这两种模型时，工程师会对模型进行边缘设备优化，减少内存占用，在适当情况下使用较低精度，并以适合实际环境的格式进行部署。这确保了用户可以获得可靠的实时操作体验。通过在实际分辨率下运行并采用高效批处理，两个模型即使在紧凑的电池供电机器人移动平台和机械臂硬件上，也能保持快速和可靠——有时会为用户所需的速度和稳定性而略微牺牲边界精度和/或特征细节。这种精度与实用性之间的平衡是该项目理念的核心。

“对于辅助机器人来说，性能不仅仅通过基准精度来衡量，而是看系统能否在日常生活的不确定性中可靠运行，”RAMMP项目参谋长Sivashankar Sivakanthan表示。“在设备上运行像DINOv3和SAM这样的模型，使得用户能够信任的实时感知成为可能——无需依赖网络连接，也不影响安全性。”

RAMMP的感知系统建立在RF-DETR之上，这是一个经过DINOv2嵌入微调的轻量级检测模型。训练数据使用SAM进行自动标注，使团队能够快速生成覆盖辅助设备在实际使用中遇到的各种角度、高度、背景和光照条件的高质量标注。数据增强和多视角策略进一步确保了不同视角下的一致性。最终形成的系统智能、具有适应性，并提供更安全、更自信的移动能力。

通过在微调的RF-DETR模型中结合SAM的标注能力和DINOv2的丰富视觉表示，RAMMP实现了实时360度环境感知和自适应物体检测。

面向未来的协作愿景

HERL和ATDev作为RAMMP联盟中的关键合作伙伴共同工作，各自在项目中发挥独特优势。HERL凭借其在生物医学工程和以用户为中心的研究方面的深厚专业知识领导该项目，为下一代辅助移动设备设定了愿景。ATDev则提供坚实的工程视角，将HERL的转化研究应用于现实设备。HERL和ATDev共同展示了学术领导力与技术创新如何融合，创造出既具有开创性又深度贴合用户实际体验的移动解决方案。

除了整合最先进的人工智能技术外，HERL的方法高度协作，设计过程中充分参与轮椅使用者、临床医生和倡导组织。这样的参与式行动方法确保技术解决方案能满足现实需求，而不仅仅是理论需求。RAMMP背后的全国性联盟包括Kinova Robotics、LUCI Mobility、ATDev，以及卡内基梅隆大学、康奈尔大学、东北大学和普渡大学等学术领导者的合作伙伴。

“通过RAMMP，ARPA-H旨在创造一个未来，使行动不便的美国人能够更轻松地独立生活、从事工作并享受休闲活动，同时利用先进的机器人技术，”ARPA-H项目经理Mansoor Khan说道：https://arpa-h.gov/about/people/mansoor-khan。“Meta的视觉模型是一项关键技术，使机器人平台能够‘看见’。这些模型帮助机器人理解场景、导航世界并拾取正确的物体——显著减轻用户的认知负担，提升独立能力。”

除了提升个人行动能力外，RAMMP还旨在在匹兹堡及整个宾夕法尼亚州催生新的劳动力和制造业机会，推动先进的行动解决方案和国内生产。这一愿景将技术进步与经济和社会发展交织在一起。

“RAMMP团队通过ARPA-H正在构建的，是当我们不再接受现状时，美国健康创新的一个缩影，”ARPA-H主任Alicia Jackson博士说道：https://arpa-h.gov/about/people/alicia-jackson。“数百万行动不便的美国人应得到能够满足其需求的技术，而不是需要他们去适应的技术。与像Meta这样的世界级创新者合作，将前沿AI引入辅助机器人，是我们比任何人想象的都更快实现这一目标的方式。”

展望未来，RAMMP团队将继续推进下一代感知模型的整合，包括SAM 3.1和DINOv3，以进一步提升实时环境理解和交互能力。未来的工作将侧重于增强时间一致性、在各种现实环境中提高稳健性，以及与决策和控制系统实现更紧密的整合。通过结合这些进展，RAMMP旨在迈向能够不仅准确感知环境，还能随时间适应不同用户的独特需求、行为和情境的辅助系统。下一代辅助机器人不仅会移动人们——它将推动社会走向一个每个人，无论能力如何，都能充分独立参与的世界。

我们最新的更新已发送至您的邮箱

In the heart of Pittsburgh, a quiet revolution is underway — one that promises to redefine what independence means for people with disabilities. At the center of this movement is the Human Engineering Research Laboratories (HERL) ：https://www.herl.pitt.edu/, a pioneering institute at the University of Pittsburgh, now leading an initiative with up to $41.5 million in funding from the Advanced Research Projects Agency for Health (ARPA-H) ：https://arpa-h.gov/, a US research funding agency established to support transformative biomedical and health breakthroughs. With support from ARPA-H, the HERL team and ATDev ：https://assistivetech.dev/, a pioneer in innovative assistive technology, are building the Robotic Assistive Mobility and Manipulation Platform Providing Independence for People with Disabilities (RAMMP), blending cutting-edge robotics, artificial intelligence, and user-centered design.

For the estimated 5.5 million wheelchair users in the United States ：https://www.volpe.dot.gov/news/us-dot-volpe-center-human-factors-study-informs-rule-air-travelers-disabilities, preservation of mobility ensures individual opportunity and participation in everyday life. When assistive technology does not reflect the complexity of the real world, it can undermine confidence, independence, and physical safety. The statistics are sobering: over 100,000 wheelchair-related injuries are treated in US emergency departments each year ：https://www.pittwire.pitt.edu/features-articles/2025/11/04/advanced-research-projects-agency-health-rammp-herl, often due to trips and falls. As a result, the need for smarter, safer, and more adaptable technology is urgent.

RAMMP: A New Era for Assistive Mobility

The ARPA-H-supported Robotic Assistive Mobility and Manipulation Platform ( RAMMP ：https://arpa-h.gov/explore-funding/awards/2771) project aims to address the shortcomings in current robotic mobility platform design through integration of advanced robotics, novel operating systems, and digital twin technology — creating a virtual simulation environment for safe, scalable testing and development. RAMMP’s approach also integrates artificial intelligence, including the use of several of Meta’s open source AI vision models, including DINO ：https://ai.meta.com/research/dinov3/ and Segment Anything Model (SAM) ：https://ai.meta.com/research/sam3/.

DINO and SAM have already assisted in a variety of projects ranging from those focused on medical imaging to wildlife conservation. DINO, a self-supervised vision transformer, excels at learning visual representations from unlabeled data, making it ideal for environments where annotated datasets are scarce. SAM, on the other hand, is designed to “segment anything” and has the ability to identify and outline any object in an image or video with minimal prompting. Segment Anything’s versatility and accuracy have made it a foundational tool in domains where precise object recognition is paramount.

Engineering for the Real World: Running DINOv3 and SAM on Edge Devices

For people relying on assistive devices, every second counts. The unpredictable nature of everyday environments, such as a child darting across a sidewalk, the sudden appearance of a curb, or a dropped set of keys, requires an immediate reaction. Processing camera images and sensor data directly on the device, known as edge computing, empowers robotic mobility platforms to function as responsive tools. These tools must be robust and consistent across the wide range of dynamic environments in which people live.

At the same time, deploying powerful AI models like DINOv3 and SAM on limited, battery-powered hardware presents significant engineering challenges. Real-world deployments must consider practical factors, including battery life, heat dissipation, unreliable network connectivity, and strict size and weight requirements.

However, overcoming these constraints mean nothing to the end user if they can't perform their activities of daily living using these new robotic systems. These newer methods allow users to interact with the robot more naturally, using their immediate surroundings as context, freeing both engineers and users from having to design and navigate complex, time-consuming interfaces. The ability to use natural language combined with image data to query the user’s and robot’s environment and provide more direct commands directly reduces the cognitive load and amount of context switching required for a user to do something as simple as picking up a cup off a table.

This functionality is already being integrated by the RAMMP team into their first prototype. Leveraging tools built off of DINO to enable querying the robot’s image sensors to detect automatic door buttons, cups, and curbs/ground for navigation assistance. With this functionality now ready for real-world testing, engineers are focusing on voice and touch input, letting users select and interact with specific objects in their surroundings. In addition to the existing challenges of ensuring accuracy and temporal coherence of the model outputs, this provides the additional challenge of ensuring robustness and predictability across user prompts and inputs.

DINOv3 serves as a compact, efficient 'visual brain' for devices — a general-purpose foundation on which task-specific, lightweight modules can be layered for actions such as object detection or movement tracking, enabling reuse of visual data and conserving power.

Applying both models as part of the development of robotics systems, engineers optimize models for edge devices, reducing memory footprint, using lower precision when appropriate, and deploying in formats tailored for real-world conditions. This ensures reliable, real-time operation for users. By running at practical resolutions and with efficient batching, both models stay fast and dependable, even on the compact, battery-powered hardware used in robotic mobility platforms and robotic arms, — sometimes trading a little bit of boundary precision and/or feature detail for the speed and stability needed by users on the go. This balance between precision and practicality is central to the project's philosophy.

“For assistive robotics, performance is not measured by benchmark accuracy alone, but by whether a system can operate reliably in the unpredictability of everyday life," said Sivashankar Sivakanthan, Chief of Staff to the RAMMP project. “Running models like DINOv3 and SAM on-device is what enables real-time perception that users can trust - without relying on connectivity or compromising safety.”

RAMMP's perception system is built on RF-DETR, a lightweight detection model fine-tuned with DINOv2 embeddings. Training data is auto-labeled using SAM, enabling the team to rapidly generate high-quality annotations across the full range of angles, heights, backgrounds, and lighting situations that assistive devices encounter in the real world. Data augmentations and multi-view strategies further enforce consistency across perspectives. The result is a system that is smart, adaptive, and offers safer and more confident mobility.

By combining SAM's labeling power with DINOv2's rich visual representations in a fine-tuned RF-DETR model, RAMMP achieves real-time 360-degree environmental awareness and adaptive object detection.

A Collaborative Vision for the Future

HERL and ATDev are working together as key partners within the RAMMP consortium, each bringing distinct strengths to the project. HERL leads the initiative with its deep expertise in biomedical engineering and user-centered research, setting the vision for next-generation assistive mobility. ATDev brings a robust engineering perspective, taking HERL's translational research and making it function in real-world devices. Together, HERL and ATDev exemplify how academic leadership and technical innovation can combine to create mobility solutions that are both groundbreaking and deeply attuned to the lived experience of users.

In addition to integration of cutting-edge artificial intelligence, HERL’s approach is deeply collaborative, engaging wheelchair users, clinicians, and advocacy groups throughout the design process. This participatory action methodology ensures that technology addresses real-world needs, not just hypothetical ones. The national consortium behind RAMMP includes partners like Kinova Robotics, LUCI Mobility, ATDev, and academic leaders at Carnegie Mellon, Cornell, Northeastern, and Purdue.

“Through RAMMP, ARPA-H aims to create a future where Americans with limited mobility can more easily live independently, pursue work, and enjoy leisure activities by leveraging advanced robotics,” said Mansoor Khan ：https://arpa-h.gov/about/people/mansoor-khan , ARPA-H Program Manager. “Meta’s vision models are a critical technology that allows the robotics platform to ‘see.’ These models help the robot understand the scene, navigate the world, and pick up the right objects — significantly reducing user cognitive burden and increasing independence.”

Beyond enhancing individual mobility, RAMMP aims to catalyze new workforce and manufacturing opportunities in Pittsburgh and across Pennsylvania, fostering advanced mobility solutions and domestic production. This vision weaves technological progress with economic and social advancement.

“What the RAMMP team is building through ARPA-H is a glimpse of what American health innovation looks like when we stop accepting the status quo,” said Alicia Jackson, Ph.D. ：https://arpa-h.gov/about/people/alicia-jackson, ARPA-H Director. “Millions of Americans with limited mobility deserve technology that meets them where they are, not technology that asks them to adapt. Partnering with world-class innovators like Meta to bring frontier AI into assistive robotics is how we make that happen faster than anyone thought possible."

Looking ahead, the RAMMP team will continue to advance the integration of next-generation perception models, including SAM 3.1 and DINOv3, to further improve real-time environmental understanding and interaction. Future work will focus on strengthening temporal consistency, robustness across diverse real-world conditions, and tighter integration with decision-making and control systems. By combining these advances, RAMMP aims to move toward assistive systems that not only perceive their environment accurately, but also adapt over time to the unique needs, behaviors, and contexts of each user. The next generation of assistive robotics will not just move people — it will move society closer to a world where everyone, regardless of ability, can participate fully and independently.

Our latest updates delivered to your inbox