Ilya Sutskever — We're moving from the age of scaling to the age of research - Dwarkesh Podcast Recap

Podcast: Dwarkesh Podcast

Published: 2025-11-25

Duration: 1 hr 36 min

Summary

Ilya Sutskever discusses the transition from merely scaling AI models to focusing on research and understanding their economic impact. He highlights the complexities of AI model performance versus their real-world utility.

What Happened

In this episode, Ilya Sutskever reflects on the current state of AI, suggesting that while the investment in AI feels abstract and disconnected from everyday life, its impact will eventually be felt throughout the economy. He expresses a sense of disbelief that the advancements in AI are happening in real time, akin to something out of science fiction. Sutskever notes that the slow integration of AI into the economy might make it seem less significant, yet he believes that the ultimate effects will be profound and transformative.

A significant part of the discussion revolves around the puzzling phenomenon where AI models perform exceptionally well on evaluations while their economic impact seems lagging. Sutskever shares an analogy comparing AI models to students in competitive programming, arguing that over-specializing on one area can hinder generalization to other tasks. He emphasizes that while pre-training offers a wealth of data, it may not necessarily equip models with the versatility needed for varying challenges, which leads to a disconnect between their performance in tests and real-world applications.

Key Insights

• The investment in AI is becoming normalized, but its real economic impact is yet to be fully realized.
• AI models may excel in evaluations but struggle to translate that performance into practical applications.
• Over-specialization in training can limit a model's ability to generalize effectively across different tasks.
• The vast data available during pre-training does not guarantee better generalization than focused reinforcement learning.