Evals Your Whole Team Can Run, Not Just the Experts
In this video
What you'll learn
Codify your eval process into a reusable skill
How Lucas packaged judge prompts, judge validation, and a human review interface into skills his teammates run on their own, so the process stopped living in one person's head.
Get non-experts running evals, no ML background needed
How Nova Escola's pedagogy team and other PMs started running real evals inside an AI coding agent, and what made the workflow approachable enough to hand off.
Know what to automate and what stays human
Where Lucas leaned on skills versus where expert judgment had to stay in the loop, like error analysis and labeling, and how to make that call on your own product.
Why this topic matters
Most teams have one person who really gets evals, and everything bottlenecks on them. Lucas took Hamel's open-source eval skills and turned them into something his whole team could run. His team uses evals to optimize the WhatsApp assistant Nova Escola ships to teachers across Brazil. We'll show you exactly how, so you can do it on your own product.
You'll learn from
Hamel Husain
ML Engineer with 20+ years of experience
Hamel Husain is a ML Engineer with 20+ years of experience. He has worked with innovative companies such as Airbnb and GitHub, which included early LLM research used by OpenAI, for code understanding. He has also led and contributed to numerous popular open-source machine-learning tools. Hamel is currently an independent consultant helping companies build AI products.
Lucas Machado Rocha
AI Product Manager at Nova Escola
Lucas Machado Rocha is a product manager at Nova Escola, a Brazilian non-profit that supports public school teachers. He has spent over 15 years connecting product, technology, and social impact. At Nova Escola he builds and evaluates an LLM-powered teaching assistant that reaches teachers over WhatsApp, and he has been teaching evals to other product managers across Brazil.
Go deeper with a course
Featured in Lenny’s List
AI Evals For Engineers & PMs
Hamel Husain and Shreya Shankar
ML Engineer with 20+ years of experience.. ML Systems Researcher Making AI Evaluation Work in Practice
Keep exploring
