He Co-Invented the Transformer. Now: Continuous Thought Machines - Llion Jones and Luke Darlow [Sakana AI] - Machine Learning Street Talk (MLST) Recap

Podcast: Machine Learning Street Talk (MLST)

Published: 2025-11-23

Duration: 1 hr 13 min

Guests: Llion Jones, Luke Darlow

Summary

Llion Jones and Luke Darlow discuss their new research on Continuous Thought Machines (CTM), which they believe could advance AI by mimicking human-like reasoning and adaptive computation. The episode explores the limitations of current AI architectures, such as Transformers, and how CTM aims to overcome these by incorporating biologically inspired elements.

What Happened

Llion Jones, co-inventor of the Transformer, has shifted his focus away from Transformers, feeling the field has become oversaturated. He now explores new research directions, including the Continuous Thought Machine (CTM), which is inspired by biological principles and aims to solve problems more akin to human reasoning.

Jones and his colleague, Luke Darlow from Sakana AI, have introduced the Continuous Thought Machine, which was spotlighted at NeurIPS 2025. The CTM incorporates native adaptive computation and higher-level concepts for neurons, allowing it to tackle problems with a more human-like, sequential thought process.

The episode delves into the evolution of AI research, comparing the freedom and innovation seen during the development of Transformers to the current pressure-filled environment. Jones emphasizes the need for research freedom to encourage novel discoveries, highlighting his company's efforts to protect this freedom.

Jones and Darlow discuss the historical progression from recurrent neural networks to Transformers and the current stagnation in AI architecture innovation. They express concern that the industry is stuck in a local minimum, focusing too heavily on scaling existing models rather than exploring fundamentally new architectures.

The Continuous Thought Machine is presented as a potential breakthrough, with its unique approach of treating neurons as individual models and focusing on synchronization between neurons over time. This architecture allows for more sophisticated reasoning and problem-solving capabilities.

The episode also touches on the potential for AI systems to perform open-ended research and design new architectures autonomously. Jones envisions a future where AI models collaborate with human researchers, enhancing the research process rather than replacing human input entirely.

Darlow further explains the technical aspects of the CTM, including its internal thought dimension and novel neuron modeling. He suggests that CTM's approach could significantly improve AI's ability to handle complex, sequential tasks like maze solving or language prediction.

Finally, the guests discuss their vision for the future of AI, emphasizing the importance of adaptive computation and the need to develop architectures that can inherently reason and learn over time. They invite interested researchers to join Sakana AI in Japan, where they can pursue innovative AI research with creative freedom.

Key Insights