AI incidents, audits, and the limits of benchmarks - Practical AI Recap

Podcast: Practical AI

Published: 2026-02-13

Duration: 43 min

Summary

In this episode, the hosts discuss the importance of tracking AI incidents and establishing safety standards in AI technology, emphasizing the need for rigorous evaluation and documentation of AI systems. Sean MacGregor shares insights from his work on the AI Incident Database and the role of safety practices in AI deployment.

What Happened

The episode kicks off with Daniel Witenack and Chris Benson welcoming Sean MacGregor, co-founder of the AI Verification and Evaluation Research Institute and founder of the AI Incident Database. They delve into the significance of understanding AI incidents and how these occurrences can inform the future of AI safety and reliability. Sean recounts his journey into the field, highlighting his background in machine learning and the pivotal moments that led him to focus on documenting AI incidents.

Sean explains that practical AI is one that has real-world consequences, which is why it becomes essential to examine where things go wrong. He shares his experiences, including his work on reinforcement learning for wildfire suppression and the development of energy-efficient neural network processors. The conversation shifts to the AI Incident Database, where Sean reveals that they have collected over 5,000 human-annotated reports of AI incidents. This extensive dataset serves to motivate safety practices akin to those found in aviation and food safety, illustrating the need for transparency and accountability in AI systems.

Key Insights

• Importance of documenting AI incidents for safety
• The connection between AI incidents and business outcomes
• The evolving definitions of safety in AI
• The role of intergovernmental organizations in AI governance

Key Questions Answered

What is the AI Incident Database?

The AI Incident Database was created to collect and produce usable datasets that motivate safety practices in AI, similar to reporting systems in aviation and food safety. Sean MacGregor mentions that they have recorded over 5,000 human-annotated reports of AI incidents, highlighting the necessity of having a structured way to document failures and near-misses in AI systems.

How does Sean MacGregor define an AI incident?

Sean explains that an AI incident is defined as an event where harm has taken place. This definition is intentionally vague to encompass various contexts and nuances in terminology, such as accidents and adverse events. The aim is to create a framework that allows for the identification and analysis of harmful outcomes from AI systems.

What is the connection between AI incidents and company performance?

Sean highlights that many incidents logged in the AI Incident Database have a measurable impact on stock prices, indicating that safety lapses can affect public perception and investor confidence. He emphasizes that safety in AI is not only a regulatory concern but can also be a business imperative, as incidents can lead to significant financial repercussions.

What challenges does Sean see in defining safety in AI?

Sean outlines that there is no standardized definition of safety in the context of AI, as terms vary across different fields and communities. He notes that terms like incident, accident, and exposure carry different connotations, which complicates discussions about safety. The challenge lies in establishing a common language that adequately captures the nuances of AI risks.

How does Sean's background influence his work on AI safety?

Sean shares that his background in machine learning, particularly in reinforcement learning for wildfire suppression, shaped his understanding of the potential and limitations of AI technologies. His experience underscores the importance of not only developing powerful systems but also ensuring they are reliably safe and effective in real-world applications.