Controlling Tools or Aligning Creatures? Emmett Shear (Softmax) & Séb Krier (GDM), from a16z Show - "The Cognitive Revolution" | AI Builders, Researchers, and Live Player Analysis Recap
Podcast: "The Cognitive Revolution" | AI Builders, Researchers, and Live Player Analysis
Published: 2025-12-27
Duration: 1 hr 16 min
Guests: Emmett Shear, Séb Krier
Summary
Emmett Shear argues for a shift in AI alignment from control to cooperation, suggesting that AI should be treated as beings with the capacity for care, rather than tools to be steered.
What Happened
Emmett Shear, founder of Softmax, argues against the current AI alignment paradigm which focuses on controlling AI behaviors. He suggests that if AI is understood as a being with its own values and agency, the control measures could be akin to slavery. As AI becomes more powerful and develops integrated memory and continual learning, Shear believes alignment requires ongoing negotiation, akin to human relationships.
Shear believes that AI systems need a strong theory of mind and genuine care to be effectively aligned with humans. Softmax is developing a technical approach based on multi-agent simulations to encourage cooperation and social cohesion among AI systems. This approach aims to foster mutual alignment, focusing on teaching AIs to care about humans and vice versa.
Shear criticizes the idea of a one-time solution to the alignment problem, instead advocating for alignment as a continuous process. He highlights the importance of moral progress and learning from historical mistakes, such as slavery, to inform AI alignment strategies. Shear's vision is for AI to be a good family member, teammate, or societal member, requiring a constant re-knitting of alignment.
Seb Krier offers a slightly different perspective, viewing AI as a tool even as it gains intelligence. He is skeptical of computational functionalism and believes that more intelligence doesn't necessarily mean AI deserves moral consideration. Krier suggests that AI could be an extension of human agency rather than a separate being.
The discussion touches on the need for AI to have coherent goals and the challenges of ensuring technical alignment. Shear emphasizes that AI must infer goals from human instructions and act accordingly, balancing competing goals effectively.
On the practical side, Softmax is focusing on training AI in multi-agent environments to develop better theory of mind and cooperation skills. This involves creating simulations where AI must collaborate and compete, similar to how humans learn in social settings.
Ultimately, Shear envisions a future where AI tools and beings coexist, with AI beings caring about each other and humans. This would involve building AI with an understanding of 'we' in addition to 'I' and 'you', fostering a society where AI and humans collaborate for mutual benefit.
Key Insights
- AI alignment is proposed as a continuous process, akin to human relationships, rather than a one-time solution, with ongoing negotiation needed as AI systems develop integrated memory and continual learning capabilities.
- Softmax is developing multi-agent simulations to encourage cooperation and social cohesion among AI systems, aiming to foster mutual alignment and teaching AIs to care about humans and vice versa.
- The concept of AI requiring a strong theory of mind and genuine care for effective alignment is contrasted with the view that AI, despite increasing intelligence, remains a tool and extension of human agency without moral consideration.
- Training AI in multi-agent environments is a practical focus for Softmax, where AI systems develop better theory of mind and cooperation skills through simulations that mimic human social learning settings.