Why Physical AI Needed a Completely New Data Stack - Gradient Dissent: Conversations on AI Recap

Podcast: Gradient Dissent: Conversations on AI

Published: 2025-12-16

Duration: 1 hr 1 min

Summary

This episode explores the advancements in robotics and AI, particularly through the lens of Nico West's company, rerun.ai, which is redefining data logging for robotics applications. The conversation also touches on the evolving landscape of reinforcement learning and the challenges of benchmarking in this field.

What Happened

In this episode, host Lukas Biewald engages in a deep conversation with Nico West, founder of rerun.ai, a company focused on high-performance logging for robotics. They discuss the challenges of visualizing data in robotics applications and how rerun.ai has evolved its data model multiple times to better meet industry needs. West shares insights into the complexities of robotics, including the combination of reinforcement learning with imitation learning, which is gaining traction after years of development hurdles.

West highlights that rerun.ai initially targeted spatial computing and augmented reality but has since broadened its scope to include a variety of applications, particularly in robotics. The discussion reveals the importance of having a robust data stack that can handle multimodal data over time, such as sensor inputs and neural network outputs. Biewald and West also compare their experiences in the robotics domain, noting that while rerun.ai and Weights and Biases share similar foundations, their product focuses have diverged to address specific needs in data pipelines and visualization.

Key Insights

• The evolution of data models is crucial for effective data visualization in robotics.
• Reinforcement learning is increasingly being combined with imitation learning for better robotic manipulation.
• Creating benchmarks in robotics is challenging due to the need for co-designing hardware and software.
• Rerun.ai is expanding beyond augmented reality to serve a diverse range of industries, including finance.

Key Questions Answered

What is rerun.ai and what services do they provide?

Rerun.ai is a company that specializes in high-performance logging for robotics and functions as a system of record for robotics companies. They focus on creating an open-source SDK for logging, modeling, querying, and visualizing multimodal data, which is particularly valuable for complex systems that change over time. Their platform has been tailored to ease the debugging process in robotic systems, making it significantly more efficient for developers.

How is reinforcement learning evolving in robotics?

The episode discusses how reinforcement learning, which has traditionally been used for walking and motion tasks, is now being combined with imitation learning to enhance robotic manipulation. This combination addresses longstanding challenges in the field, enabling robots to learn more effectively from both direct reinforcement and observed behaviors. This evolution signifies a substantial shift in how robots can be trained to perform complex tasks.

What are the challenges of creating benchmarks in robotics?

Creating benchmarks for robotics is complicated due to the need for co-training and co-designing hardware alongside software. The intricacies of real-world problems often make it difficult to establish standardized benchmarks that can accurately measure performance across different systems. The conversation highlights that solutions in robotics are often tailored to specific hardware configurations, complicating the benchmarking process.

Why did rerun.ai initially focus on augmented reality?

Rerun.ai initially targeted augmented reality and spatial computing because these areas presented significant challenges in managing and visualizing complex multimodal data. The company aimed to simplify the debugging of these systems, which involve multiple sensors and data types. As they progressed, they naturally expanded into robotics as their technology proved valuable in a broader range of applications.

What are the key features of rerun.ai's logging product?

Rerun.ai's logging product is designed to handle high-speed logging and visualization of multimodal data, including inputs from various sensors and time-series data. The platform is built to be user-friendly, with a simple API that allows for easy integration and performance tuning. The focus on creating a highly performant system has made it attractive for a variety of applications, from robotics to finance.