The cold-start eval problem: evals for new AI products
Hosted by Shreya Shankar and Hamel Husain
Wed, Jul 15, 2026
7:00 PM UTC (45 minutes)
Virtual (Zoom)
Free to join
Go deeper with a course
Save 25% til Sunday
AI Evals For Engineers & PMs
Hamel Husain and Shreya Shankar
ML Engineer with 20 years of experience.. ML Systems Researcher Making AI Evaluation Work in Practice
Wed, Jul 15, 2026
7:00 PM UTC (45 minutes)
Virtual (Zoom)
Free to join
Go deeper with a course
Save 25% til Sunday
AI Evals For Engineers & PMs
Hamel Husain and Shreya Shankar
ML Engineer with 20 years of experience.. ML Systems Researcher Making AI Evaluation Work in Practice
What you'll learn
Bootstrap evals with zero users
How to build a useful eval set before anyone touches your product. You generate realistic test cases by varying who’s using it and what they’re trying to do, so failures show up early.
Measure every change you ship
Turn error analysis into a lightweight harness that tells you whether a change made your AI better or worse, whether it’s a new tool or a reworded prompt.
Watch it built live, from scratch
We put an eval framework onto a real AI writing app that has no tests today. You’ll see the whole process, close enough to copy on your own product the same week.
Why this topic matters
If you have no users and no data yet it can feel like "looks fine to me" is all you've got. Hamel and Shreya show you how to build evals from day zero, so every change you ship is one you can measure.
You'll learn from
Shreya Shankar
AI researcher; incoming CS Professor at Carnegie Mellon
Shreya builds open-source systems for AI-powered data processing. She is a final-year PhD candidate at UC Berkeley and, in 2027, will join Carnegie Mellon University as an Assistant Professor of Computer Science. Shreya created DocETL, an open-source system for analyzing unstructured text at scale, deployed across journalism, law, medicine, policy, finance, and urban planning. Her research has been published at top computer science venues including VLDB, SIGMOD, and UIST (including a Best Paper award), and has influenced systems at Snowflake, BigQuery, LangChain, and OpenAI. Before her PhD, Shreya worked as a machine learning and data engineer at startups. She holds a BS in Computer Science from Stanford University.
Hamel Husain
ML Engineer with 20 years of experience
Hamel Husain is a ML Engineer with over 20 years of experience. He has worked with innovative companies such as Airbnb and GitHub, which included early LLM research used by OpenAI, for code understanding. He has also led and contributed to numerous popular open-source machine-learning tools. Hamel is currently an independent consultant helping companies build AI products.