Build AI Evals from Scratch for Product Managers
Madalina Turlea
Founder | CPO | 10 years in Product
Catalina Turlea
2x founder | CTO | 14 years in Tech
Build AI Evals from Scratch, Hands-On, for Product Builders
Whether you already have an AI feature live or your first one is on the roadmap, the real challenge is defining what AI quality means for your product.
This workshop helps you answer that question, and more:
How do you write a rubric to evaluate your own AI product?
How do you test AI when you don't have any data yet?
Should you write evals before or after shipping?
Why can't you just let an AI write the evals for your AI product?
How should product, engineering and domain experts work together to write strong evals?
How do you use AI to scale your evals, and write an LLM judge that matches your product experts' judgment?
Evals are a new habit, and habits come from reps. People get good at evals by doing the work themselves. This workshop is three hours of that work, on one real AI product.
Two ways to take this workshop:
Public cohort, for individuals
Private workshop, for your team only. To book one, use this page or email madalina@lovelaice.com.
Combined we bring more than 24 years of product and engineering experience and we teach in the workshop the exact same process we use on our own product and with our customers at Lovelaice.
What you’ll learn
Build the skills to measure how well AI performs in your product, by writing evals and use them to improve your AI product.
Start a golden dataset from scratch, even before you have production data
Collect inputs that represent how real users will use the feature
Add edge cases and adversarial inputs, and use expert annotations to define what a correct answer looks like
Review AI outputs systematically instead of rating them “good” or “bad.”
Turn individual observations into specific, recurring failure patterns.
Prioritize the failures that create the greatest risk for users and the business.
Pick the right eval metrics for your AI feature
Balance free, deterministic checks with LLM-judge metrics, and validate the judge against human grades before you trust it
Read evaluation results and use them to plan the next iteration, weighing quality, cost and latency at scale
Workshop agenda
Start from a real AI product
We hand you a working AI product and its system prompt. You see the full setup: inputs, instructions, expected output.
Build your golden dataset
Collect representative inputs, then add edge cases and adversarial scenarios.
Review and annotate real AI answers
Everyone reviews the AI's answers alone and writes specific annotations.
Zoom out to find the failure patterns
Group annotations into failure patterns for this specific AI. Rank them by how often they happen and how much they hurt. Turn them into an evaluation rubric.
Turn failures into targeted evals
Create the smallest useful check for each important failure: deterministic rule or focused LLM judge.
Benchmark across models
Run the same feature on multiple models. Compare quality, cost and latency side by side.
Read the results and plan the next iteration
Compare the new version against your baseline, decide what to change next, and map the loop onto your own product.
Learn directly from Madalina & Catalina
Madalina Turlea
Product leader with 10+ years experience in FinTech | Founder &CPO Lovelaice
Catalina Turlea
Founder & CEO @Lovelaice | Co-founder & ex-CTO @ nilo | 14 years in tech
Who this workshop is for
- Product managers and leaders who want to learn to build AI into their own product and how to shape it directly towards real product outcomes
- Engineers building AI features who want to learn how to define quality together with their PM and how a shared eval process makes that work.
- Data scientist who want to learn how to support product teams to choose AI evaluation metrics and how to test for real impact.
What's included
Live sessions
Learn directly from Madalina Turlea & Catalina Turlea in a real-time, interactive format.
Lifetime access
Go back to course content and recordings whenever you need to.
Lovelaice platform access
You can see all the prompts and inputs in the Lovelaice app.
Bring your questions
Already have questions about evals in your own product? Bring them to the workshop and we will discuss them live
Community of peers
Stay accountable and share insights with like-minded professionals.
Certificate of completion
Share your new skills with your employer or on LinkedIn.
Maven Guarantee
Your purchase is backed by the Maven Guarantee.
Free resources
Testimonials
Thank you so much for this wonderful AI-course. I enjoyed it so much! The material was very understandable with a hands-on-approach. I've learnt a lot!
Reka
Start-up Founder at FluxionEach session takes us deeper into the advantages of AI, from experiment evaluation to exploring how different models behave. I enjoy how your passion and clear explanations make prompt engineering, practical tips, and even industry insights so easy to understand. Learning from you both is a pleasure.
Maria
Senior Engineer
Frequently asked questions
Maven for Teams
Reimbursement
Get your company to pay
Everything L&D needs: email template, receipts, and certificate of completion.
Get reimbursedTeam discount
Learn with your teammates
Save 20%+ when 2 or more teammates enroll in the same cohort.
Save 20%+ with a teamPrivate cohort
Run a cohort for your org
A dedicated cohort with a custom schedule and curriculum, tailored to your team.
Book a private cohort$899
USD
10am–1pm EST
·2 cohorts