Free Lesson

Stop Letting Your LLM "Find the Problems"

Hosted by Shreya Shankar and Hamel Husain

Wed, Jun 24, 2026

7:00 PM UTC (45 minutes)

Virtual (Zoom)

Free to join

Invite your network

Go deeper with a course

Featured in Lenny’s List
AI Evals For Engineers & PMs
Hamel Husain and Shreya Shankar
View syllabus

What you'll learn

Why "ask AI to find the bugs" backfires

An AI agent will read all your data and hand back confident summaries. The catch: it has no taste. It can't tell a pattern that matters from one that's just noise. You'll learn to spot that gap, so a tidy summary never passes for a real finding.

Read your data the way an analyst would

Learn open coding: a hands-on method for reading through real examples of your product in use. The skill isn't summarizing what happened. It's asking who the user was and what they needed, which is where the real insight hides.

Put AI to work without letting it mislead you

See how to bring AI into your analysis as an assistant: it does the grunt work, you stay the judge. You'll also learn to push back on what it gives you, so an answer that sounds smart but says nothing never makes it onto your roadmap.

Why this topic matters

Your AI product does something dumb. How do you fix it? Most teams guess, or dump everything into an AI and trust whatever comes back. This session is about the alternative: a hands-on way to find what's actually broken in your product, and to put AI to work on the problem without letting it fool you. You'll leave knowing how to debug your AI for real.

You'll learn from

Shreya Shankar

AI researcher; incoming CS Professor at Carnegie Mellon

Shreya builds open-source systems for AI-powered data processing. She is a final-year PhD candidate at UC Berkeley and, in 2027, will join Carnegie Mellon University as an Assistant Professor of Computer Science. Shreya created DocETL, an open-source system for analyzing unstructured text at scale, deployed across journalism, law, medicine, policy, finance, and urban planning. Her research has been published at top computer science venues including VLDB, SIGMOD, and UIST (including a Best Paper award), and has influenced systems at Snowflake, BigQuery, LangChain, and OpenAI. Before her PhD, Shreya worked as a machine learning and data engineer at startups. She holds a BS in Computer Science from Stanford University.

Hamel Husain

ML Engineer with 20 years of experience

Hamel Husain is a ML Engineer with over 20 years of experience. He has worked with innovative companies such as Airbnb and GitHub, which included early LLM research used by OpenAI, for code understanding. He has also led and contributed to numerous popular open-source machine-learning tools. Hamel is currently an independent consultant helping companies build AI products.

See all products from Hamel Husain & Shreya Shankar

Sign up to join this lesson

By continuing, you agree to Maven's Terms and Privacy Policy.