When AI Discovers The Next Transformer - Robert Lange (Sakana) - Machine Learning Street Talk (MLST) Recap

Podcast: Machine Learning Street Talk (MLST)

Published: 2026-03-13

Duration: 1 hr 18 min

Guests: Robert Lange

Summary

Robert Lange discusses the potential of AI-driven scientific discovery and innovation. The episode explores how open-ended evolutionary algorithms, such as Shinka Evolve, could revolutionize the field.

What Happened

The episode begins with a discussion on how analogies from evolution can be applied to scientific research, particularly through the use of large language models (LLMs). Robert Lange argues that while LLMs are good at refining solutions, they often require the invention of new problems to drive true innovation. He cites the potential of AI to discover new architectural designs, like a new Transformer, which could be a Rubicon moment for the field.

Lange introduces Shinka Evolve, an evolutionary approach to optimizing language models that aims to improve sample efficiency and computational resource usage. He explains how this method can cut down on costs and evaluation times by using technical innovations within evolutionary search. The process involves generating, refining, and evaluating solutions iteratively, with the code being open source to democratize access.

The conversation shifts to specific applications of Shinka Evolve, including the circle packing problem. Lange describes how this method achieved state-of-the-art results with fewer program evaluations by employing a combination of technical innovations. These innovations allow the system to efficiently explore the program space and find optimal solutions quickly.

Lange also discusses the potential for AI-driven systems to evolve both problem formulations and solutions over time, potentially running for extended periods to collect diverse stepping stones. He emphasizes the importance of co-evolving problems and solutions to drive further innovation, citing examples of recursive problem-solving approaches.

The episode touches on the idea of using AI to automate the design of agentic systems, showcasing how Shinka Evolve can be applied to various tasks beyond just circle packing. Lange envisions a future where AI systems can autonomously execute experiments and propose new ones, with humans acting as shepherds to guide the process.

Lange expresses optimism about the future of AI in scientific discovery, noting that while AI systems have not yet achieved human-level creativity, they can serve as powerful amplifiers of human potential. He suggests that the evolution of AI-human interfaces will be critical in shaping how these technologies are integrated into scientific research.

The episode concludes with a discussion on the importance of maintaining human oversight in AI-driven research, particularly in the context of peer review and ensuring the validity of scientific findings. Lange highlights the need for systems that can efficiently discover new insights while remaining accountable to human standards.

Key Insights

• Large language models, like those potentially discovering the next Transformer, excel at refining existing solutions but stumble when new problems must be created to spark real innovation. Robert Lange argues that AI's future lies in its ability to autonomously invent new challenges rather than just solve them.
• Shinka Evolve employs an evolutionary methodology to optimize language models, drastically reducing costs and evaluation times. By iteratively generating and refining solutions, this open-source approach democratizes access and improves computational resource efficiency.
• Achieving state-of-the-art results in the circle packing problem with fewer evaluations, Shinka Evolve demonstrates how technical innovations allow systems to explore and optimize solutions efficiently. This method emphasizes the importance of combining technical prowess with iterative problem-solving.
• AI systems, according to Lange, have not yet reached human-level creativity but can amplify human potential in scientific discovery. The evolution of AI-human interfaces will be crucial for integrating these technologies into research, ensuring that AI serves as a powerful tool rather than an autonomous entity.

Key Questions Answered

What is Shinka Evolve as discussed on Machine Learning Street Talk?

Shinka Evolve is an evolutionary algorithm framework designed to optimize language models by using sample-efficient techniques and computational innovations. It aims to democratize access to advanced AI tools through open-source availability.

How does Robert Lange view the role of AI in scientific discovery?

Robert Lange believes AI has the potential to amplify human creativity and understanding by driving open-ended scientific discovery. He envisions AI systems evolving problem formulations and solutions over time, with humans guiding the process.

What are the key innovations of Shinka Evolve mentioned in the podcast?

Key innovations include using model ensembling to select the best language model for each task, implementing a bandit-based approach for adaptive prioritization, and employing evolutionary algorithms for efficient exploration of program space.