AI Evals for Product Managers: From Vibe Check to Proof

5.0 (12)

Madalina Turlea

Founder | CPO | 10 years in Product

Catalina Turlea

2x founder | CTO | 14 years in Tech

Build an AI Feature you can prove works

Built from 1,500+ real AI experiments · 5.0 rating
Taught by a product leader and a former CTO, using the same evaluation process we apply to our own product and customer work.

Your AI feature returns polished answers. But can you prove they are accurate, consistent, and good enough to ship?

Testing two or three examples is not enough. Generic “helpfulness” scores will not catch the failures that matter. And waiting for customer feedback means discovering the problem after trust is already lost.

In this hands-on workshop, you’ll build and evaluate a real AI feature from start to finish.

You’ll run the same prompt across multiple models, review outputs blind, uncover the failures the AI actually makes, turn those failures into targeted evals, improve the prompt, and measure whether the new iterations performs better.

By the end, you will have:

A tested production-style system prompt
A quality baseline across multiple AI models
An annotated set of real AI outputs
A prioritized map of the feature’s failure modes
A repeatable process you can use on your next AI feature

You will perform the evaluation yourself, make product decisions from the results, and leave with a working system you understand.

What you’ll learn

Build the skills to evaluate AI features, improve them with evidence, and make confident ship decisions, without relying on vibe checks.

Turn a product requirement into clear system instructions, inputs, and outputs.
Create realistic test cases that expose weak spots before users do.
Structure prompts for consistent performance across many inputs—not just a good demo.

Review AI outputs systematically instead of rating them “good” or “bad.”
Turn individual observations into specific, recurring failure patterns.
Prioritize the failures that create the greatest risk for users and the business.

Define deterministic or LLM-as-judge evals for your AI feature
Understand how different LLMs perform the same task and which one is the optimal for your use case
Understand cost, accuracy and latency at scale to build a business case for your AI feature

Workshop agenda

Build the first production-ready version
Turn a product requirement into a structured system prompt with clear inputs and outputs.
Pressure-test it across multiple models
Run realistic test cases and compare outputs blind so model reputation does not influence judgment.
Find the failures you didn’t anticipate
Review real outputs, write specific annotations, and group them into recurring failure modes.
Turn failures into targeted evals
Create the smallest useful check for each important failure: deterministic rule, human review, or focused LLM judge.
Improve the feature and prove what changed
Revise the prompt, rerun the experiment, and compare the results against your original baseline.
Take the loop back to your product
Map the method onto your own use case and identify the first experiment to run after the workshop.

Learn directly from Madalina & Catalina

Madalina Turlea

Contact

Product leader with 10+ years experience in FinTech | Founder &CPO Lovelaice

Catalina Turlea

Contact

Founder & CEO @Lovelaice | Co-founder & CTO @ nilo | 14 years in tech

See all products from Madalina

Who this workshop is for

Product managers and product leads responsible for an AI-powered feature
Founders moving an AI product from prototype toward production
Domain experts who need to define what a correct AI answer looks like

What's included

Live sessions

Learn directly from Madalina Turlea & Catalina Turlea in a real-time, interactive format.

Lifetime access

Go back to course content and recordings whenever you need to.

Community of peers

Stay accountable and share insights with like-minded professionals.

Certificate of completion

Share your new skills with your employer or on LinkedIn.

Maven Guarantee

Your purchase is backed by the Maven Guarantee.

Free resources

Testimonials

Thank you so much for this wonderful AI-course. I enjoyed it so much! The material was very understandable with a hands-on-approach. I've learnt a lot!
Reka
Start-up Founder at Fluxion
Each session takes us deeper into the advantages of AI, from experiment evaluation to exploring how different models behave. I enjoy how your passion and clear explanations make prompt engineering, practical tips, and even industry insights so easy to understand. Learning from you both is a pleasure.
Maria
Senior Engineer
This course was eye opening and Catalina and Madalina were wonderful teachers, supporting us with personal examples and taking the time to answer all of our questions. Some of my highlights on the course were understanding LLMs behind the scenes, learning how to better structure and optimize our prompts (and how important experimentation is for that - also thanks to Lovelaice), and of course creating custom metrics and running evaluations, especially with LLM-as-a-Judge. Thank you for creating this course!
Pavlina
Frontend Developer
Even if you’re not sure what kind of AI feature you want to build yet, this course will help you gain the clarity you need. Once you go through the material and understand the concepts, it truly feels like a curtain has been lifted. Catalina and Madalina are amazing teachers. They explain everything in a clear and approachable way, making complex topics easy to understand, even for people without a technical background. The live sessions are truly priceless. Being able to ask questions, see real examples, and learn directly from their experience adds incredible value to the course and makes the learning experience much more impactful. Overall, it’s an excellent starting point for anyone interested in building AI features or simply understanding how modern AI systems work.
Barbara
Senior Developer