How Evals Made GitHub Copilot Happen
Hosted by John Berryman, Shawn Simister, and Hamel Husain
Mon, May 12, 2025
10:00 PM UTC (30 minutes)
Virtual (Zoom)
Free to join
Go deeper with a course


Mon, May 12, 2025
10:00 PM UTC (30 minutes)
Virtual (Zoom)
Free to join
302 students
Go deeper with a course


What you'll learn
Build more reliable LLM-as-judge systems
Learn the three-part eval taxonomy that drove success
Avoid the "ratchet effect" trap in A/B testing
Why this topic matters
You'll learn from
John Berryman
ML Researcher, Software Engineer, and Author; Worked on GitHub Copilot
John Berryman is the founder of Arcturus Labs. His journey through AI and search technologies includes contributing to GitHub Copilot's early development, where he worked on the team that brought AI-assisted coding from concept to reality. Throughout his career, John has helped build search and recommendation systems that millions use daily – from GitHub's code search infrastructure to Eventbrite's discovery platform and the US Patent Office's next-generation search system. This blend of experience in both foundational search technologies and cutting-edge AI applications gives him unique insight into building practical, powerful LLM applications.
John shares his expertise through two books: Relevant Search, which reveals the art and science of building search applications, and Prompt Engineering for LLMs, which guides developers through the emerging practice of language model application development.
Shawn Simister
Member of Technical Staff, Factory
Shawn Simister is a Member of Technical Staff at Factory. Prior to this role, he served as a Staff Software Engineer at GitHub for nearly three years, where he architected evaluation systems for GitHub Copilot.
Before joining GitHub, Shawn spent over a decade at Google, working as both a Software Engineer and a Developer Programs Engineer.
His earlier career includes operating his own consultancy and co-founding MyShield.com.
Shawn focuses on building AI-powered tools to augment human creativity and problem solving. He brings experience spanning both large tech companies and startups.
Hamel Husain
ML Engineer with 20 years of experience
Hamel is a machine learning engineer with over 20 years of experience. He has worked with innovative companies such as Airbnb and GitHub, which included early LLM research used by OpenAI, for code understanding. He has also led and contributed to numerous popular open-source machine-learning tools. Hamel is currently an independent consultant helping companies build AI products.
Previously at
Learn directly from John Berryman, Shawn Simister, and Hamel Husain
By continuing, you agree to Maven's Terms and Privacy Policy.
By continuing, you agree to Maven's Terms and Privacy Policy.