Founder Real Talk

How do we systematically evaluate and improve increasingly autonomous AI systems?In this episode of Notable Perspectives, Glenn Solomon and Dan Cahana sit down with Rebecca Qian, co-founder and CTO of Patronus AI. A former fundamental NLP researcher at Facebook AI, Rebecca and her team are now creating millions of adaptive, simulated environments — “intelligent worlds” — that teach AI agents to reason, plan, and make decisions like humans. Join us as we explore:The Eval Problem: Too few evals on too narrow a slice of reality; benchmark and leaderboard culture has created the wrong incentive.Reality of Simulations: The shift from single-turn classification to agents that reason and decide the way people do. And the stakes if you get it wrong — reward hacking, deception, misalignment.Building the Factory: The insight that general intelligence requires generalizable capabilities — not domain-specific data.Chapters:00:00 — The Eval Problem: Why current AI evaluation methods are fundamentally broken—and what it me