Online Evals and Production Monitoring

Lightning Lessons

Practical skills & tools to accelerate your career

Building AI-Native Products

Online Evals and Production Monitoring

Hosted by Jason Liu, Ben Hylak, and Sidhant Bendre

624 students

What you'll learn

AI Monitoring Fundamentals

Identify key metrics and set up effective monitoring systems to track AI product performance in real-time.

Issue Detection and Analysis

Learn techniques to pinpoint problems and segment performance by use case to quickly address critical issues.

Feedback-Driven Improvement

Apply user feedback patterns to prioritize product enhancements that directly increase retention and revenue

Why this topic matters

Mastering AI monitoring prevents costly failures that damage trust. This skill helps you catch issues before users do, drive data-informed improvements, and make strategic decisions that directly impact business outcomes. As AI deployment grows, monitoring expertise positions you to lead technical teams and protect company reputation.

You'll learn from

Jason Liu

Consultant at the intersection of Information Retrieval and AI

Jason has built search and recommendation systems for the past 6 years. He has consulted and advised a dozens startups in the last year to improve their RAG systems. He is the creator of the Instructor Python library.

Ben Hylak

Co-Founder, Raindrop

Sidhant Bendre

Co-Founder of Oleve

Sid Bendre is the co-founder of Oleve, where they are building a portfolio of iconic consumer software across multiple verticals. Oleve has already launched two virally successful AI products, amassing over 500 million views across social media platforms including TikTok, Instagram, and YouTube. These products include Quizard AI (ranked #5 in Education as of April 2025) and Unstuck AI (ranked #3 in Education as of April 2025), with a 3rd product launching soon.

Previously at