Katerina Zanos

How to Design a Metrics-First Recommender System

Principal Machine Learning Engineer, Disney+/ESPN at The Walt Disney Company

Learn how to go from a vague objective to a concrete metrics stack.

Build a Metrics Stack That Reflects Your Product Goal

Understand how your metrics spec drives core engineering choices in recommendation systems.

Connect Metrics to System Design Decisions

Practice using a metrics spec as a shared language with PMs and leadership when building a recommendation system.

Use Metrics to Drive Roadmap & Product Conversations

Learn proven approaches for quickly improving AI applications.  Build AI that works better than the competition, regardless of the use-case.

ML Engineer with 20 years of experience

Airbnb

GitHub

DataRobot

AlixPartners

Hamel Husain

ML Engineer with 20 years of experience.

ML Systems & Applied AI Evals Researcher

Google

UC Berkeley

Stanford University

Shreya Shankar

ML Systems Researcher Making AI Evaluation Work in Practice

All lectures are recorded, 8x live office hours.

Cohort 3

Cohort 4

🚨 The next cohort is January 2026. Enroll now and get immediate access to our course reader and community. 🚨All students in this course get:<ul><li>🗄️ Lifetime access to all materials!</li><li>♾️ Unlimited access to future cohorts: never worry about timing or missing out on new material. </li><li>🤖 6 months of unlimited access to our new AI Eval Assistant (more info below).</li><li>🧑‍🏫 10+ hours of office hours to maximize the value of live interaction.</li><li>🏫 A Discord community with continuous access to instructors to get unstuck (even after the course!).</li></ul>---Do you catch yourself asking any of the following questions while building AI applications?1. How do I test applications when the outputs require subjective judgements?2. If I change the prompt, how do I know I'm not breaking something else?3. Where should I focus my engineering efforts? Do I need to test everything?4. What if I have no data or customers, where do I start?5. What metrics should I track? What tools should I use?6. Can I automate testing and evaluation? If so, how do I trust it?If so, this course is for you.This is a flipped classroom setting. All lectures are professionally recorded with an emphasis on live office hours and student interaction.

Eliminate the guesswork of building AI applications with data-driven approaches.

How To Collect Data For Evals

Get immediate clarity and direction with Error Analysis

Implement Effective Evaluations

Master Architecture-Specific Eval Strategies

Run Evals In Production

Ensure That Evals Lead To High ROI

Engineers &amp; PMs building AI products who are interested in moving beyond proof-of-concepts.

Those interested in moving beyond vibe-checks to data driven measurements you can trust, even when outputs are stochastic or subjective.

Founders and leaders who are unsure of the failure modes of their AI applications and where to allocate resources.

Learn to make decisions tailored to business constraints, understand when & how to apply AI effectively & build a multi-agent application

AI Founder & Advisor to F500s | Ex-AWS

Amazon Web Services

University of Oxford

Microsoft

Aishwarya Naresh Reganti

Tech Lead | Lecturer | Advisor | Researcher | Speaker | Investor

Applied AI @ OpenAI | Ex-Google

OpenAI

Samsung

Databricks

Kiriti Badam

Member of Technical Staff @ OpenAI | AI Advisor | Ex-Google

Hitachi Cohort

Jan 2026

We teach RAG, evals, agents, MCP, multi-agents, context engineering, and all that jazz. But always as tools to solve a business problem. If you want a checklist of hype items without knowing when or why to use them, please don’t join our course. :) 🚨🚨Notes:<ul><li>We hold a 4.9 rating across more than 400 reviews, making us the #1 rated course on Maven at this scale. So why are students raving about out course? Check out our <a target="_blank" rel="noopener noreferrer nofollow" href="https://testimonial.to/building-agentic-ai-applications-with-a-problem-first-approach/all">wall of love</a>!</li><li>Many of our students have No AI experience walking into the course. To estimate what you can achieve at the end of this course check out our student capstones <a target="_blank" rel="noopener noreferrer nofollow" href="https://bit.ly/capstones_pai">here. </a></li><li>We follow a flipped-classroom format. All lectures are pre-recorded so folks can go at their own pace, but we’ll still meet 3 times a week for office hours and live sessions. Check schedule below for more details</li><li>For questions or bulk requests, reach out to: problemfirst.ai@gmail.com</li></ul>This course is an independent offering and is not affiliated with, endorsed by, or related to the instructors' current or past employers.

Design and build impactful agentic AI systems to solve business problems.

Develop a Problem-First AI Intuition For Enterprise Use Cases

Build an Evals Driven Mindset & Iterative Design

Master Practical Prompt and Context Engineering

Build and Optimize Agentic Pipelines

Apply Everything in a Real-World Capstone

You should have coded at least once in your life. The course includes no-code assignments but some knowledge of code helps!

The only prerequisite

Software/AI Engineers, Strategists, Data Professionals, Solution Architects and Consultants who want to master AI system design

Business Leaders and Product Managers seeking to gain the technical understanding needed to make informed decisions &amp; lead AI initiatives

Entrepreneurs looking to understand common generative AI use cases and learn how to develop and implement AI-powered solutions

Learn a survival-metrics system that turns roadmap anxiety into clear, defensible stop / pivot / invest decisions.

Product leader for tough decisions

SmartRecruiters

Columbia University

Pendo

Product Institute

Adam Thomas

Managed Over 200MM In Product Portfolios over the last 15 years - 2x Founder.

Cohort 6

You know this moment: you’re feeling stuck. Every team is begging for more features, and your job is to nod along and “make it work.” You’re managing by keeping everyone happy instead of making real product bets. You can feel that the strategy is off, but you don’t have a clear way to say what should stop, what should pivot, and what deserves more investment.This course gives you that decision system.In Survival Metrics, you’ll build a small set of stop / pivot / invest metrics that tie revenue, risk, and learning back to your product strategy and the politics in your org. You’ll practice turning vague ideas into crisp invest statements, then use them to run better roadmap conversations with your leadership team.Underneath it all, we’ll work on critical thinking: how to frame problems, challenge assumptions, and cut through noise so you’re no longer an order-taker, but the person others trust to make the hard calls.

Stop being an order-taker; use survival metrics to drive product strategy.

Design survival metrics that tell you when to stop, pivot, or invest.

Turn a messy roadmap into a prioritized portfolio of bets.

Lead roadmap conversations that feel clear, calm, and grounded in data.

Connect metrics to a story executives actually understand and remember.

Handle product politics with clear rules instead of endless negotiation.

Upgrade your critical thinking so you act like a product strategist.

PMs and product leaders who own a roadmap but feel prioritization is chaotic, political, and hard to defend.

PMs stuck in prioritization paralysis: everything is priority 1, nothing gets cut, and the roadmap keeps changing.

Product leaders tired of taking orders who need a clear, defensible way to say stop, pivot, or invest.

Acquire and develop a critical skill for product managers who are leading and contributing to AI products.

December 2025

GenAI Product Builder @ Google, ex-Meta

Marily Nika, Ph.D AI/ML

Senior Data, ML and GenAI Scientist

SHI International

National Association of REALTORS

George Zoto

Senior NLP/AI Scientist

Product Manager AI&ML @ Google

Diego Granados

Sr. ML Product Manager,  ex-Meta

Mark Cramer

Sr. ML Product Manager @ Meta

Feb 2026

“Does it work?”... “Is it good enough?”... “Can we ship it?”...How do you answer these questions for AI products? You’re responsible for “running evals” but what does that mean?How do you choose the right metrics, interpret fuzzy results, and make a confident decision?This course gives you a framework to do just that.<ul><li>Map user value to evaluation (eval) objectives so your metrics aren’t abstract. Define success then translate it into measurable criteria.</li><li>Choose metrics you can actually maintain: capability, safety, UX friction, latency, cost and “does this reduce support tickets or increase activation.”</li><li>Set ship/no-ship thresholds you can defend to leadership.</li><li>Build lightweight workflows that work in real teams: human review where it matters, automation where it lasts, documentation that drives decisions.</li><li>Consider domain constraints (e.g., healthcare safety) and know what to avoid: silent failures, misleading proxy metrics and tests that don’t reflect production.</li><li>Tie everything to ROI: impact vs unit cost, eval coverage vs reliability, and the minimum viable monitoring you need post-launch.</li></ul>Experience AI evals through a case-based approach with a real AI product that we evaluate together.

Eliminate uncertainty in shipping AI features

Make confident ship or hold decisions for AI features

Translate user value into clear evaluation goals

Choose the right metrics for capability, safety, UX, and cost

Set defensible thresholds leadership will trust

Build lightweight evaluation workflows teams can maintain

Navigate domain constraints and avoid common failures

PMs leading AI features, growth, or platform initiatives

PMs who partner with ML teams and want to set evaluation standards

PMs who need to make clear “ship or hold” calls without doing the engineering