Free Lesson

The cold-start eval problem: evals for new AI products

Part of AI Product Engineering

Hosted by Shreya Shankar and Hamel Husain

Wed, Jul 15, 2026

7:00 PM UTC (45 minutes)

Virtual (Zoom)

Free to join

267 students

Invite your network

Go deeper with a course

Featured in Lenny’s List
AI Evals For Engineers & PMs
Hamel Husain and Shreya Shankar
View syllabus

What you'll learn

Bootstrap evals with zero users

How to build a useful eval set before anyone touches your product. You generate realistic test cases by varying who’s using it and what they’re trying to do, so failures show up early.

Measure every change you ship

Turn error analysis into a lightweight harness that tells you whether a change made your AI better or worse, whether it’s a new tool or a reworded prompt.

Watch it built live, from scratch

We put an eval framework onto a real AI writing app that has no tests today. You’ll see the whole process, close enough to copy on your own product the same week.

Why this topic matters

If you have no users and no data yet it can feel like "looks fine to me" is all you've got. Hamel and Shreya show you how to build evals from day zero, so every change you ship is one you can measure.

You'll learn from

Shreya Shankar

AI researcher; incoming CS Professor at Carnegie Mellon

Shreya builds open-source systems for AI-powered data processing. She holds a PhD in Electrical Engineering & Computer Science from UC Berkeley and, in 2027, will join Carnegie Mellon University as an Assistant Professor of Computer Science. Shreya created DocETL, an open-source system for analyzing unstructured text at scale, deployed across journalism, law, medicine, policy, finance, and urban planning. Her research has been published at top computer science venues including VLDB, SIGMOD, and UIST (including a Best Paper award), and has influenced systems at Snowflake, BigQuery, LangChain, and OpenAI. Before her PhD, Shreya worked as a machine learning and data engineer at startups. She holds a BS in Computer Science from Stanford University.

Hamel Husain

ML Engineer with 20+ years of experience

Hamel Husain is a ML Engineer with 20+ years of experience. He has worked with innovative companies such as Airbnb and GitHub, which included early LLM research used by OpenAI, for code understanding. He has also led and contributed to numerous popular open-source machine-learning tools. Hamel is currently an independent consultant helping companies build AI products.
See all products from Hamel Husain & Shreya Shankar

Sign up to join this lesson

By continuing, you agree to Maven's Terms and Privacy Policy.