Notion’s Token Town: 5 Rebuilds, 100+ Tools, MCP vs CLIs and the Software Factory Future — Simon Last & Sarah Sachs of Notion
Latent Space: The AI Engineer Podcast Podcast Recap
Published:
Duration: 1 hr 17 min
Guests: Simon Last, Sarah Sachs, Zach Tratar
Summary
Simon Last and Sarah Sachs from Notion discuss the company's innovative approach to AI integration, focusing on custom agents and their rebuilds. They explore Notion's strategic adaptations to model capabilities and the creation of a 'software factory' for streamlined development.
What Happened
Notion is heavily invested in becoming the best system of record for enterprise work, utilizing a Minimal Compute Platform (MCP) to enhance their permission model. Their recent launch of custom agents was notably successful, marking a milestone in free trials and conversions. The development of these agents began in 2022, coinciding with their access to GPT-4, but early attempts faced limitations due to model capabilities and context length.
A pivotal moment for Notion came with improvements in model capabilities, particularly around Sonic 3.6 or 3.7. This allowed them to overcome initial hurdles and advance their agent technology. Notion's approach involves a portfolio strategy that balances the maintenance of existing products, the introduction of new features, and the exploration of experimental projects.
The concept of a 'software factory' is central to Notion's vision, aiming to automate workflows in code development and maintenance. This strategy includes not resisting model capabilities but instead preparing for future advancements. Notion's culture promotes rebuilding and iterating on products, encouraging low ego among staff and a readiness to delete and rewrite code.
Internally, Notion fosters innovation through hackathons focused on building agentic tool loops and uses a 'Design Playground' for quick UI prototyping. Their team structure is flexible, allowing engineers to move between projects, with reporting structures forming after product launches. Emphasizing practical demonstrations over written memos, Notion's culture values demos as a means of idea communication.
Notion's AI team comprises around 50 members, with additional partners for packaging and product interfaces. They maintain product quality through a robust evaluation system, noting discrepancies in model quality between vendors. Collaborating with Anthropic and OpenAI, they aim for a 30% pass rate in their evaluation systems.
The role of Model Behavior Engineers at Notion has evolved from data specialists to a blend of data science, project management, and prompt engineering. Their evaluation system functions as an agent harness, allowing agents to download datasets, conduct evaluations, and debug and fix issues autonomously. This reflects Notion's vision of a future where agents perform more tasks and humans focus on observation and maintenance.
Key Insights
- Notion's recent launch of custom agents marked their most successful product release, highlighting the importance of timing and model capability advancements.
- The 'software factory' concept at Notion aims to automate workflows in code development, minimizing human intervention while maintaining essential invariants.
- Model Behavior Engineers at Notion have evolved into roles that combine data science, project management, and prompt engineering, supporting the company's evaluation system.
- Notion's culture emphasizes a low-ego environment with a readiness to rebuild and iterate on products, supporting innovation and adaptability to new model capabilities.