How OpenAI Customers Use Evals To Build Better AI Products

Lightning Lessons

Practical skills & tools to accelerate your career

How OpenAI Customers Use Evals To Build Better AI Products

Hosted by Jim Blomo and Hamel Husain

Mon, Jun 9, 2025

7:30 PM UTC (30 minutes)

Virtual (Zoom)

Free to join

Learn directly from undefined

By continuing, you agree to Maven's Terms and Privacy Policy.

488 students

By continuing, you agree to Maven's Terms and Privacy Policy.

Invite your network

Go deeper with a course

AI Evals For Engineers & PMs

Hamel Husain and Shreya Shankar

View syllabus

Mon, Jun 9, 2025

7:30 PM UTC (30 minutes)

Virtual (Zoom)

Free to join

488 students

Invite your network

Go deeper with a course

AI Evals For Engineers & PMs

Hamel Husain and Shreya Shankar

View syllabus

What you'll learn

Case studies from OpenAI customers

Explore real-world case studies showing how evals drive iterative improvement, prevent regressions, and reduce risk.

Get a sneak peek of OpenAI's integrated eval products

OpenAI offers eval products. Learn what they are and how to use them.

The hidden risks of skipping evals

Learn what happens when you don’t do evals: amplified bugs and bias, credibility loss, and internal confusion.

Design patterns for eval-driven development

Get practical techniques for incorporating evals into your workflow—from prompt evaluation to regression testing.

Why this topic matters

Evaluation-driven development is the key to accelerating AI product development. This session will cover data-driven methods in driving model performance and user outcomes. He will also share some features within OpenAI’s platform for integrating evals into AI product workflows.

You'll learn from

Jim Blomo

Engineering Leader at OpenAI, working on Evals

Jim Blomo is an engineering leader at OpenAI, working on model customization with evals and fine-tuning. His experience ranges from teaching at UC Berkeley to leading engineering teams at Yelp and Outschool. Jim enjoys exploring the food and outdoors of the Bay Area with his family.

Hamel Husain

ML Engineer with 20 years of experience

Hamel is a machine learning engineer with over 20 years of experience. He has worked with innovative companies such as Airbnb and GitHub, which included early LLM research used by OpenAI, for code understanding. He has also led and contributed to numerous popular open-source machine-learning tools. Hamel is currently an independent consultant helping companies build AI products.

Learn directly from Jim Blomo and Hamel Husain

By continuing, you agree to Maven's Terms and Privacy Policy.

Be the first to know what's new on Maven

Contact support: support@maven.com

Learn

Teach

Maven