🚨 JULY 21st IS IS OUR FINAL LIVE COHORT - (We have to get back to building AI applications!) - Enrolled students get LIFETIME ACCESS to all materials and recordings!🚨 Do you catch yourself asking any of the following questions while building AI applications? 1. How do I test applications when the outputs are stochastic and require subjective judgements? 2. If I change the prompt, how do I know I'm not breaking something else? 3. Where should I focus my engineering efforts? Do I need to test everything? 4. What if I have no data or customers, where do I start? 5. What metrics should I track? What tools should I use? Which models are best? 6. Can I automate testing and evaluation? If so, how do I trust it? If you aren't sure about the answers to these questions, this course is for you. This is a hands-on course for engineers and technical PMs. Ideal for those who are comfortable coding or vibe coding. ---WHAT TO EXPECT This course will provide you with hands-on experience. Get ready to sweat through exercises, code and data! We will meet two times a week for four weeks, with generous office hours (read below for course schedule). We will also hold office hours and host Discord community where you can communicate with us and each other. In return, you will be rewarded with skills that will set you apart from the competition by a wide margin. (see testimonials below). All sessions will be recorded and available to students asynchronously.--- COURSE CONTENT Lesson 1: Fundamentals &amp; Lifecycle LLM Application Evaluation- Why evaluation matters for LLM applications - business impact and risk mitigation- Challenges unique to evaluating LLM outputs - common failure modes and context-dependence- The lifecycle approach from development to production- Basic instrumentation and observability for tracking system behavior- Introduction to error analysis and methods for categorizing failures Lesson 2: Systematic Error Analysis- Bootstrap data through effective synthetic data generation- Annotation strategies and quantitative analysis of qualitative data- Translating error findings into actionable improvements- Avoiding common pitfalls in the analysis process- Practical exercise: Building and iterating on an error tracking system Lesson 3: Implementing Effective Evaluations- Defining metrics using code-based and LLM-judge approaches- Techniques for evaluating individual outputs and overall system performance- Organizing datasets with proper structure for inputs and reference data- Practical exercise: Building an automated evaluation pipeline Lesson 4: Collaborative Evaluation Practices- Designing efficient team-based evaluation workflows- Statistical methods for measuring inter-annotator agreement- Techniques for building consensus on evaluation criteria- Practical exercise: Collaborative alignment in breakout groups Lesson 5: Architecture-Specific Evaluation Strategies- Evaluating RAG systems for retrieval relevance and factual accuracy- Testing multi-step pipelines to identify error propagation- Assessing appropriate tool use and multi-turn conversation quality- Multi-modal evaluation for text, image, and audio interactions- Practical exercise: Creating targeted test suites for different architectures Lesson 6: Production Monitoring &amp; Continuous Evaluation- Implementing traces, spans, and session tracking for observability- Setting up automated evaluation gates in CI/CD pipelines- Methods for consistent comparison across experiments- Implementing safety and quality control guardrails- Practical exercise: Designing an effective monitoring dashboard Lesson 7: Efficient Continuous Human Review Systems- Strategic sampling approaches for maximizing review impact- Optimizing interface design for reviewer productivity- Practical exercise: Implementing a continuous feedback collection system Lesson 8: Cost Optimization- Quantifying value versus expenditure in LLM applications- Intelligent model routing based on query complexity- Practical exercise: Optimizing a real-world application for cost efficiency GUEST SPEAKERS ---1. Teresa Torres (Product Talk) : From Noob to 5 Automated Evals in 4 Weeks (as a PM).2. Jeremy Howard &amp; Johno Whitaker (AnswerAI): SolveIt -&nbsp;The Thinking Developer's Environment (a great environment for evals!)3. Leonard Tang (Haize Labs): Scaling Inference-Time Compute for Better LLM Judges4. Charles Frye (Modal): Taming diffusion QR codes with evals and inference-time scaling5. Udi Menkes (Product Manager, Intuit): A playbook for building AI agents you can trust.6. Skylar Payne: &nbsp;10x Your RAG Evaluation by Avoiding these Pitfalls TOOLS TRACK: Eval vendors will provide optional tutorials---- Harrison Chase: LangSmith- Mikyo King &amp; Isaac Flath: Arize Phoenix- Wayde Gilliam: Braintrust

Eliminate the guesswork of building AI applications with data-driven approaches.

Course syllabus

If your events are recurring and at the same time, it might be easiest to use a single line item to communicate your course schedule to students

Monday & Wednesday & Friday

Feel free to type out dates as your title as a way to communicate information about specific live sessions or other events.

May 7, 2022

Schedule items can also be used to convey commitments outside of specific time slots (like weekly projects or daily office hours).

Weekly projects

Course schedule

Engineers &amp; Technical PMs building AI products who are interested in moving beyond proof-of-concepts.

Those interested in moving beyond vibe-checks to data driven measurements you can trust, even when outputs are stochastic or subjective.

Founders and leaders who are unsure of the failure modes of their AI applications and where to allocate resources.

Who this course is for

What you’ll get out of this course

We've tried all of them so you don't have to.

Acquire the best tools for finding, diagnosing, and prioritizing AI errors.

And how to best leverage data when you do have users.

Learn how to bootstrap with synthetic data for testing before you have users

Data flywheels ensure you have examples to draw from for prompts, tests, and fine-tuning.

Create a data flywheel for your applications that guarantees your AI will improve over time.

How can you really trust LLM-as-a-judge? How should you design them? We will show you how. We will also show you how to refine prompts, generate metadata, and other tasks with the assistance of AI.

Automate parts of your AI evaluation with approaches that allow you to actually trust and rely on them.

We will show you approaches to discover all the ways AI is not performing

Ensure your AI is aligned to your preferences, tastes and judgement.

There are an infinite number of things you can try, tests you can write, and data you can look at. We will show you a data-driven process that helps you prioritize the most important problems so you can avoid wasting time and money.

Avoid common mistakes we've seen across 35+ AI implementations.

We will provide end-to-end exercises, examples and code to make sure you come away with the skills you need. We will NOT just throw a bunch of slides at you!

Hands-On Exercises, Examples and Code

Generous office hours ensures students can ask questions about their specific issues and interests.

Personalized Instruction

Learn proven approaches for quickly improving AI applications.  Build AI that works better than the competition, regardless of the use-case.

Hamel Husain

ML Engineer with 20 years of experience.

Shreya Shankar

ML Systems Researcher Making AI Evaluation Work in Practice

Cohort 1 has $1,000 in modal credits.  This is NOT guaranteed for Cohort 2

Cohort 1

THIS IS OUR FINAL COHORT!!   We have to get back to building AI Applications.

Cohort 2 (Our Final Cohort)

🚨 Enroll in the July cohort and get FREE lifetime access to our AI Build Lab program ($1,500 value), where you'll learn to build and deploy powerful AI agents—no code required. This offer is only for our July students. Enrollment Status (July 5th):July Cohorts: 90% FullFinal Incentive: Use code SUMMER20 for 20% off the current price before enrollment closes. Team Enrollment (10+): Contact us for special pricing and arrangements at support@productfaculty.com Why it matters: AI Product Manager is the new Product Manager In 2025, every product decision will be an AI decision. This course will allow you to master the skills you need to lead in the AI-driven future. 👨‍💻 𝗔𝗕𝗢𝗨𝗧 𝗧𝗛𝗘 𝗜𝗡𝗦𝗧𝗥𝗨𝗖𝗧𝗢𝗥 Miqdad Jaffer, a Product Leader at OpenAI, has 18 years of experience in building products at some of the world's most successful companies. Currently, he leads enterprise products at OpenAI and was previously the Head of AI Products at Shopify. Miqdad is one of the world's top AI executives and an award-winning AI Product educator. His work on AI has been featured in TechCrunch, MIT’s AI Podcast, FastCompany, The Globe and Mail, and Spotify. He has been listed as one of the top 100 people to follow in Artificial Intelligence by Business Insider and has received many other awards and accolades. If you have any questions about the training, please reach out to me support@productfaculty.com. 💡 𝗔𝗕𝗢𝗨𝗧 𝗧𝗛𝗘 𝗖𝗢𝗨𝗥𝗦𝗘This course is not just about learning; it's about doing. It’s an exercise-heavy training that goes beyond gaining "AI speak" to enhance your credibility and confidence to build AI Products and future proof your career.&nbsp;The live online sessions spread across three classes that will equip you with a comprehensive AI Product playbook. This playbook covers everything from AI use-case validation, product design, development, and delivery, to launch strategies, metrics, and growth.&nbsp;Learn from successful AI product features at leading companies like Shopify, Google, Klarna, and Amazon, and apply these insights in a capstone project where you'll build your own AI-powered feature. Topics Covered (Amongst Others):&nbsp;• Taxonomy of Prompting Techniques• Basic AI Concepts and Processes• Training and Model Improvement• Quality, Bias, and Interpretability• Computational Concepts (Deterministic vs. Non-Deterministic)• Qualitative vs programmatic evaluation&nbsp;• Model Customization Framework• Types of AI Models and Applications• Chain of Thought Prompting• Self-Reflection &amp; Self-Consistency• ReAcT Prompting Framework• Retrieval Augmented Generation (RAG)• Fine-Tuning &amp; RLHF• Function Calling &amp; Tool Usage• LLM-Powered Agents• LLM Evaluation &amp; Judge LLMs• AI Safety &amp; Moderation Tools• Adversarial Prompting (Jailbreaking and Prompt Injections)• Common Real-World Use Cases of LLMs• Understand Advanced Prompting Techniques&nbsp;• Zero-shot prompting• Few-shot prompting/in-context learning• Chain-of-Thought (CoT) 💡 COURSE PREREQUISITES﻿This course is NOT for you if:• You work exclusively on non-software products (hardware, pharmaceuticals, deep climate tech, defense tech, etc.)• You're looking for a purely theoretical overview with no hands-on practice• You're looking for a low-effort introduction to AIWhat to expect:• A structured learning path that starts with the basics – no prior coding required• Beginner-friendly guidance through setting up the OpenAI Playground environment• Practical exercises that build your confidence step-by-step• With dedicated effort, you'll achieve results that directly apply to your work 💡 𝗔𝗕𝗢𝗨𝗧 PRODUCT FACULTY&nbsp;Product Faculty is one of the most recognized names in the PM training industry and has been around for over 8 years. At Product Faculty, we’ve trained and mentored over 10K product managers working at some of the leading companies in&nbsp;the world. Product Faculty’s training has some of the highest NPS (Net promoter scores in the industry with most classes having an NPS of 70+ (considered exceptional and reflect outstanding customer satisfaction). Product Faculty has been recognized as the top rated Product Management Bootcamp from the following organizations:&nbsp;&gt; 2023 Course Reports - Best PM Bootcamp&gt; 2022 Course Reports - Best PM Bootcamp&gt; Career Carma - #1 Ranked PM Bootcamp&gt; CourseCompare - Best PM Bootcamp&gt; Switchup - Best Bootcamp Reach out to support@productfaculty.com for any questions, corporate trainings, and group/student discounts.

Acquire the skills & confidently build AI products and future proof your career

Access hours of expert-led content on your schedule. Build foundational knowledge in AI Product Management with in-depth modules designed for flexibility. Perfect for busy professionals looking to upskill at their own pace and apply learnings immediately.

OnDemand Content

Join live sessions to work through real-world AI case studies. Engage with exercises and get your questions answered by expert Miqdad. Gain practical insights that turn theory into action, with direct support tailored to your unique challenges.

Live Case Study and Q&A

Complete a hands-on AI capstone that simulates launching a real AI feature. Tackle ideation to deployment with weekly feedback from experts. Finish with a portfolio-ready project that showcases your skills in real-world AI Product Management.

Capstone Project

Get OnDemand Access to the <a href="https://www.productfaculty.com/" rel="noopener noreferrer" target="_blank">#1 Rated Advanced Product Management</a> training offered by Product Faculty as part of this AI Product Management Certification.

OnDemand Access to Advanced Product Management Course

Timings for each session have been chosen based on what works best for most product people in practice. Don't worry about missing classes, recordings for all modules will be made available for you to watch in your Maven account.

Optimized for very busy schedules of Senior PM, GPMs and VPs

For PMs &amp; Leaders: The #1 rated certificate to master the age of AI. Learn the A-to-Z playbook to ship successful AI products.

For Engineers &amp; Tech Leads: Don't just build. Lead. Get the product strategy to make better tech decisions &amp; fast-track your career.

For Founders, Designers &amp; UX: The playbook to build a winning AI venture or design human-first AI experiences users will love.

Who is this course for

Clear, actionable processes for analyzing AI use cases and developing strategies towards tangible outcomes.

Solve Real Problems with AI

I’ll act as your personal innovation guide as you create functional AI applications, no prior coding experience needed.

Build Real AI Prototypes

Grasp foundational models and large language models (LLMs), and learn to code and train models independently.

Dive Deep Into AI's Technical Side

Transform theory into practice by developing your AI product idea or tackling your company's AI challenge, guided by me.

Hands-on Capstone Project

Discover essential strategies for enhancing decision-making and using data to steer results in your professional and personal life.

Make Better, Data-Driven Decisions

Master the art of managing risks by blending automation with human oversight to maximize value from AI applications.

Balance Automation with Oversight

Utilize in-context learning for efficient prototyping, RAG applications, and more, without getting engineers involved.

Learn Prompt Engineering

Deepen your understanding of AI memory/context and Retrieval Augmented Generation (RAG) to enrich AI interactions.

Master Retrieval Augmented Generation

Gain a comprehensive understanding of crucial AI algorithms, concepts, and techniques to foster organization-wide knowledge.

Boost Data & AI Literacy

Master the art of pitching your AI product ideas so clearly and excitingly that everyone's eager to jump on board.

Make your ideas irresistible: 

This intensive weekend course is tailored for professionals with weekday commitments, complete with recorded sessions for revisiting.

Optimized for Busy Senior PMs

Secure lifelong entry to our expansive alumni network for unparalleled opportunities, job connections, and continuous learning.

Lifetime Access to Product Faculty Community

Equip yourself with cutting-edge AI skills and insights to stay ahead in the evolving tech landscape, ensuring your career remains resilient and growth-oriented.

Future-Proof Your Career

The #1 rated certificate for Product Managers, Engineers, Designers, and Founders looking to master the age of AI

Cohort 5

Miqdad Jaffer

Product Leader at OpenAI, TechCrunch & MIT featured, Top 100 by Business Insider

Cohort 4

Cohort 6

Enrollment Status (updated weekly, as of June 30th):- July Cohort: 80% full- October Cohort: 15% full October is our final cohort for the year and the price will increase to $2,500 starting 1st August 2025 to meet rising demand, and maintain our current student-to-instructor ratio. 🚨🚨Note: This course is an independent offering and is not affiliated with, endorsed by, or related to the instructors' current or past employers. ⛳&nbsp;The only prerequisite: you should have coded at least once in your life. The course includes low-code assignments, and even folks who hadn’t touched code in over 15 years have found it approachable and rewarding. That said, a basic understanding of coding really helps you get the most out of it — and of course, there’s AI to assist you along the way. The course is built for everyone, whether you’re a Product Manager, Architect, Director, C-suite leader, or someone seriously exploring agentic AI. Agentic AI or AI systems capable of operating with some degree of autonomy, is transforming how we interact with technology. In the coming years, most software systems will integrate AI agents to enhance their capabilities. This shift will drive a growing demand for professionals who can move beyond surface-level understanding and apply AI effectively to solve real business challenges while navigating practical constraints. This course focuses on practical AI agent development, covering key agentic design and usage paradigms. Instead of just explaining what these techniques are, we focus on when and how to use them, so you're equipped to make informed, business-driven AI decisions. What You'll Learn All core content is pre-recorded so students can focus on two-way interaction. Lectures are watched asynchronously, and we host four office hours each week for questions and brainstorming Week 1 (Let’s get you to understand what problem-first means)Decode why agentic AI breaks traditional software assumptionsFrame hallucinations, latency, and prompt brittleness through the determinism spectrumOpen vs. closed models: tradeoffs across compliance, latency, and costProblem-first, evaluation-driven design using early datasets and proxy metricsDeconstruct a production-grade use case and redesign it across progressive system versions Week 2 (Prompt engineering is still the core part of agents, but do it smarter with right evals)Break down the evolution from zero-shot prompts to self-optimizing modelsMaster prompting: Decomposition, meta-prompts, algorithmic optimizationAnalyze when to use prompting-only systems based on task, cost, and latencyCompare model-level strategies: reasoning vs. regular, and when each makes senseAdd guardrails and evaluation layers using LLM judges, semantic scoring, and offline tests Week 3 (RAG is not dead, it’s in fact the basis of self-improving agents)Address statelessness via dynamic retrieval and memory-backed context injectionBuild robust RAG pipelines with advanced chunking, embedding selection, and retrieval methodsExplore GraphRAG, Agentic RAG and multimodal RAG and other advanced methods and learn tradeoffsArchitect episodic, semantic, procedural, and working memory layers for self-reflective agent behavior Week 4 (MCP from an enterprise lens and multi-agents + Fine-Tuning)Understand planning autonomy in agents and how dynamic tool use and multi-turn reasoning go beyond static workflowsCompare agent levels and their control dimensions: action, planning, evolution, and physical autonomyExplore MCP (Model Context Protocol) and A2A as emerging agent-tool communication standardsInvestigate critical security challenges in MCP and A2A. Understand how guardrails, tool signing, audit trails improve reliabilityAnalyze coordination patterns in multi-agent systems, including shared memory governance, state sync, AI collusion risks, evaluation, logging, and observabilityExplore fine-tuning levers (SFT, RLHF, PEFT etc.), compare with RAG, and determine when to shift from context injection to model adaptation Week 5–6 (Put it all together in a capstone)Work in groups of 6Take a business problem and design/implement a solutionDemo to 4000+ public attendees including leaders, VCs, and hiring managers Homeworks: You'll supplement your learning by building an agentic search system (Perplexity like) in 3 iterations with the final iteration using agentic RAG, MCP and multi-agents. You can choose between low-code/code routes to complete assignments. ---- ❌ Who This Course Is Not For For Those Who Have Already Deployed Gen AI in Enterprise: This course is designed as an applied foundations course for enterprise AI with only basic Python as a prerequisite and no ML background required. If you’re already familiar with deploying AI systems, you won’t gain much from the core content. However, if you're looking to network and refine best practices, you're welcome to join. Those Seeking Heavy Theoretical Knowledge: This course emphasizes applied learning and practical problem-solving, not deep dives into theoretical topics like transformer architecture, pre/post-training optimization, inference techniques, or alignment. Those Who Have Never Coded Before: While we provide low-code options, this course assumes you have some coding experience. It’s not suitable for those who have never written or worked with even basic code. Individuals Expecting Deep AI Research Focus: While we’ll cover cutting-edge techniques, this course is centered on applying AI to business problems, not research-heavy exploration. Scaling and Ops Enthusiasts: This course does not focus heavily on scaling or operational aspects (i.e., LLMOps). Deployment will be covered at a high level, but not in-depth.

Design and build impactful agentic AI systems to solve business problems.

You should have coded at least once in your life. The course includes low-code assignments, and even folks who hadn

The only prerequisite

We offer generous office hours to support you throughout the course, along with 24-hour help on Slack. Typical weekly schedule:<ul><li>Wednesdays at 10 am PT and Saturdays at 9 am PT: Assignment Support</li><li>Thursdays at 5 pm PT and Saturdays at 8 am PT: Content/Use-case discussions</li></ul>

Live Office Hours

Our weekly community discussion channel covers the latest trends in AI, industry updates, and all the unfiltered tea, open, casual, and freeform.

Chai & AI Sessions

We bring in industry experts to share proven insights and discuss real enterprise challenges

Guest Lectures

Software/AI Engineers, Strategists, Data Professionals, Solution Architects and Consultants who want to master AI system design

Business Leaders and Product Managers seeking to gain the technical understanding needed to make informed decisions &amp; lead AI initiatives

Entrepreneurs looking to understand common generative AI use cases and learn how to develop and implement AI-powered solutions

While most courses stop at teaching tools and frameworks, this course goes further by focusing on solving real-world business problems. You’ll tackle practical constraints like cost, scalability, latency, and performance, learning to design AI solutions tailored to real use cases

Solving Real Enterprise Challenges, Not Just Concepts 

While learning applied AI concepts, we’ll put them into action by building a Perplexity-like AI-powered search system through detailed, hands-on tutorials that demonstrate their practical application (Low code options will be provided)

Apply Concepts to Build an Agentic Search System

Learn how to connect cutting-edge research with real-world applications. For the capstone, you’ll use our curated list of the latest research papers to design and implement solutions for practical business use cases. Some of our capstones have received VC funding too. <a href="https://areganti.notion.site/Capstone-Projects-Problem-First-AI-1e0c91fa32368011b935e00a35d9bceb" rel="noopener noreferrer" target="_blank">Examples</a>

Gain a deep understanding of key challenges in building AI systems, including handling hallucinations, adversarial attacks, security, privacy issues etc., and learn best practices to evaluate AI solutions comprehensively

Understand Challenges and Effective Evaluation
 

The course includes guest lectures from industry experts, AMA sessions, and our Chai &amp; AI discussions, culminating in a final in-person meetup in the Bay Area. You'll have plenty of opportunities to network and become part of our community.

Access to the Problem-First AI Community

Learn to make decisions tailored to business constraints, understand when & how to apply AI effectively & build a multi-agent application

Aishwarya Naresh Reganti

Tech Lead | Lecturer | Advisor | Researcher | Speaker | Investor

Kiriti Badam

Member of Technical Staff @ OpenAI | AI Advisor | Ex-Google

May 2025 Cohort

July 2025 Cohort

If you’re a software engineer or data scientist, chances are you’ve built or seen a proof of concept (POC) for a generative AI app. It’s exciting at first—an impressive demo that showcases the potential of large language models. But here’s the problem: that’s where it usually stops. These POCs often fail to scale into reliable, production-ready applications. The result? Endless iteration cycles, unreliable outputs, and frustration as teams struggle in what we call “POC purgatory.” It doesn’t have to be this way. With the right principles, processes, and workflows, you can build LLM applications that work in production – without overcomplicating pipelines or chasing trends. This course focuses on the iterative development, evaluation, and debugging practices that make AI systems production-ready. We’ll explore tools like RAG, embeddings, and agents, but the emphasis is on building scalable, robust systems – ensuring you leave with the mindset and skills to adapt as the technology evolves. You’ll not only learn how to design, test, and deploy generative AI applications, but also how to iterate and improve rapidly. This isn’t just about building something cool – it’s about creating AI systems that solve real problems, deliver real value, and scale with confidence. What to ExpectThis is a highly interactive course designed to get you results. Across live workshops, discussions, and hands-on projects, you’ll build your skills while solving real-world challenges. •	Live coding sessions to design and refine LLM applications.•	Iteration exercises to explore how small changes can dramatically improve reliability.•	Discussions on user-centric development and integrating feedback loops.•	Expert-led Q&amp;As to help you tackle your toughest challenges. What You’ll Learn •	How to transform POC demos into production-grade applications.•	Strategies for managing non-determinism and optimizing outputs through prompt engineering.•	How to monitor, log, and debug AI systems to ensure reliability and performance.•	Best practices for handling structured outputs and integrating function calling into your applications.•	Building workflows for iterative development and experimentation.•	Practical skills through hands-on app development, including creating a Generative AI app to query PDFs. Bonus Perks All students will receive over $1,500 USD in credits! -&nbsp;$500 in Modal credits&nbsp;for cloud compute – giving you resources to power real AI applications.- $500 in Logfire credits (from the creators of Pydantic)- $300 in Google Cloud and Gemini credits for the first 100 registrants.- 6 months free Hugging Face Pro.- 6 months free Prodigy.- 3 months free Learn Prompting Plus. Guest lectures Learn directly from leading practitioners solving real AI problems in production. These sessions focus on practical workflows, debugging strategies, and building scalable systems – not just theoretical research or cutting-edge multi-agent architectures. Guest speakers in this cohort will include: - Jason Liu &amp; Ivan Leo (Instructor, 567 Labs)- Shreya Shankar (UC Berkeley, ex-Google Brain)- Zach Mueller (Hugging Face)- Paige Bailey (Google DeepMind)- Ines Montani (Explosion AI, spaCy)- Alan Nichol (Rasa)- Nathan Danielsen (Carvana)- Charles Frye (Modal)- John Berryman (Arcturus, ex-Github)- Skylar Payne (AI executive for startups. Ex-Google. Ex-LinkedIn)- Vincent Warmerdarm (marimo)- Samuel Colvin (Pydantic, Logfire) Enrolling will also give you access to all recorded guest lectures from previous cohorts of this course: - Hamel Husain (Ex-Airbnb, GitHub) – Basic Data Literacy for Debugging and Evaluating LLMs&nbsp;- Swyx (Latent Space podcast, smol.ai, ai.engineer): Engineering AI Agents in 2025- Sander Schulhoff (CEO, LearnPrompting.org) – Prompt Engineering in the LLM SDLC&nbsp;- Charles Frye (Developer Advocate, Modal) – Hardware for LLM Developers- Ravin Kumar (DeepMind) – End-to-End LLM Product Development This Course Is For •	Data scientists and machine learning engineers who are sick of unreliable POCs and want to ship reliable LLM applications.•	Software engineers who want to learn how to build generative AI systems and master the LLM software development lifecycle. This Course Is Not For Those •	Looking for an introductory course in AI—some programming knowledge is required.•	Expecting ready-made solutions for AI problems without active participation.•&nbsp;Expecting deep dives into advanced evals, multi-agent systems, or large-scale RAG pipelines. This course emphasizes building and iterating on practical, production-ready systems.•	Unable to commit to hands-on projects, live discussions, and iterative development exercises. Why Take This Course By the end of this course, you’ll: •	Understand the first principles of generative AI development.•	Gain hands-on experience building robust, scalable applications.•	Learn how to troubleshoot, monitor, and optimize AI systems for production.•	Develop workflows to iterate and refine AI applications effectively.•	Leave with the confidence to deploy reliable AI systems at scale.

Master the patterns and processes that underpin successful GenAI app development

Tuesdays and Thursdays

To ensure hands-on practical time, there will be project work to complete throughout the course.

Data scientists, machine learning engineers who are sick &amp; tired of seeing and building prototypes &amp; want to ship reliable LLM applications

Software engineers who want to learn how to build Generative AI systems and learn the LLM software development lifecycle.

You’ve seen impressive generative AI demos—but how do you take them to production? In this course, you’ll learn the exact workflows, tools, and engineering practices needed to turn concepts into reliable, scalable applications that deliver real value.

Move beyond POCs and build production-ready AI systems

Struggling with inconsistent outputs? Through interactive exercises, you’ll explore proven techniques for optimizing prompts and ensuring your AI delivers actionable results in production settings.

Master prompt engineering for reliable results

When your AI application breaks, you’ll know exactly how to fix it. We’ll walk you through real-world debugging sessions, monitoring techniques, and tools to ensure your systems stay reliable at scale.

 Learn to debug and monitor AI systems like a pro

Want your AI to produce precise, usable results? You’ll learn how to handle structured outputs and function calls, making your applications smarter and more user-friendly.

Build smarter applications with structured outputs

Stop wasting time in endless iteration cycles. Through guided projects, you’ll discover how to set up workflows that refine your applications faster and with less frustration.

Accelerate your development process with iterative workflows

Tired of theoretical knowledge? You’ll build a fully functional app that uses text and image models to query PDFs, giving you practical experience solving complex real-world problems.

Develop hands-on expertise by building GenAI and LLM apps

Whether you’re looking to lead AI projects, impress stakeholders, or level up your technical expertise, this course gives you the tools to stand out as an AI/ML practitioner.

Advance your career with practical, proven techniques

These aren’t just skills for demos—they’re skills for delivering scalable, production-ready systems that solve real-world business challenges.

Solve Real Problems at Scale

Leave the course with a project you can add to your portfolio, demonstrate to your boss, or use to showcase your ability to deploy reliable AI systems.

Showcase Tangible Results

Build LLM-powered software reliably & from first principles. Learn the GenAI software development lifecycle: agents, evals, iteration & more

Hugo Bowne-Anderson

Host of Vanishing Gradients podcast; ex-Outerbounds; expert in DS/ML.

Stefan Krawczyk

Agentforce; Ex-Stitch Fix; 13+ Years building / productionizing data, ML, Agents

Cohort 2

Cohort 3

ANNOUNCEMENT: AI PM Interview Bootcamp Starts July 5. Limited-Time Bonus Enroll in the August Agentic AI PM Cohort before July 4, and get FREE access to our AI PM Interview Prep Bootcamp.(Worth $1500)This exclusive bundle prepares you for the full AI PM journey, from cracking interviews to building GenAI products with confidence. 📦 What’s Included:AI PM Interview Bootcamp (Starts July 5)Full August AI PM Cohort Access (Starts August 9)1:1 Career Support + Real-World Projects 🚫 June cohort was sold out in no time 🎓 Only August &amp; October 2025 cohorts remain for 2025🎁 Why Apply Now?The earlier you enroll, the sooner you can start preparing and unlock bonus material, bootcamp access, and coaching support. 📞 Have questions? Meet our team here https://calendly.com/team_maheshyadav/ And we’ll walk you through everything. - Bulk Discount of 30% if you bring 2 or more students with you.- 1:1 With Mahesh (worth $1000)- Includes AI PM Interview Prep Bootcamp ($1500 Worth) We’re are consistently recognized as one of Maven’s top courses and being the only 5-star rated courses in AI. We're the only AI PM certification course that genuinely help and track your success even after cohort completion. We are really proud of 27% of our cohort members that have successfully transitioned into AI PM roles. We achieve these results through:- 50+ written lessons- 10+ live workshops (beyond recorded content)- 20+ live AMA sessions directly with Mahesh- Continued support during interviews and continuous learning via weekly session after cohort Our mission is to make this the only AI course you'll need in 2025 (period). To ensure this, we've added hands-on learning labs and strong technical support that empowers everyone—no coding or technical background required. Here, we will empower you to confidently identify , create, eval and launch production-ready agents using GCP Vertex AI, Langflow, Crew AI, Azure Evaluators SDK, and Azure Co-pilot Studio. We are offering two timings to accommodate your busy schedules. Wednesdays at 5:30 PM PSTSaturdays at 9:00 AM PST Know your instructorMahesh Yadav is your mentor and guide to success in AI product management. With over 20 years of experience and 10+ years building AI-powered products at Google, Amazon, Meta, and Microsoft. Mahesh has 9 patents as primary innovator in AI training and Power management. He has presented at premier conferences such as Nvidia GTC, Microsoft Build, and AWS re-Invent. Mahesh brings firsthand insights from advising fast-growing AI startups and collaborating with senior leaders at Google, Meta, and Amazon, giving you an inside look at what truly drives AI innovation in top tech giants and AI startups in bay area. Deep Expertise: Mahesh led the expansion of PyTorch into PyTorch Distribution and contributed to Meta’s first GPT3-scale training model (OPT) in 2019. His work spans key AI technologies including recommendation systems , PyTorch,&nbsp;Agents for AWS Bedrock, and multi-agent for GCP.Practical Insights: His teaching uses real project examples to deliver actionable strategies you can implement immediately.Impact he left: Over 3000 professionals have upskilled through his courses, many now holding roles at top AI companies Ace AI PM Interviews — Internally or ExternallyThis isn't just a course, it’s your career catalyst.Whether you're aiming for an internal transition or preparing for external AI PM interviews, you’ll gain the exact frameworks and confidence needed to stand out in any interview room.We don't stop at theory. We simulate real-world product challenges, review your interview answers, and walk you through how hiring managers at Google, Meta, and top AI startups actually evaluate candidates.And unlike most cohorts that wrap up after 3-4 live sessions, we guarantee more value than you pay for:20+ live sessions with Mahesh during the cohort and countless more on advance trending topics even after the cohorts for lifetime. You won’t just prepare for interviews, you’ll walk in already looking like an insider. Why Mahesh Is the Right Mentor for You?This course isn't just another program with flashy slides and vague frameworks.Here’s what you get here—and only here:✅ Real AI expertise, not theoryMahesh has done hands-on AI work for over 10 years—spanning infrastructure, frameworks, services, and end-to-end product delivery. He doesn't just teach GenAI—he built it before it was a buzzword.✅ Instructor-led from start to finishNo marketing agencies. No “guest speaker” fluff. Mahesh personally teaches, reviews, and gives feedback in every session. You're not just learning—you’re being mentored.✅ 1:1 mentorship, not just group lecturesEvery student gets at least one personalized 1:1 session with Mahesh, valued at $1000—use it for product ideas, interviews, or career clarity.✅ Clear, simple, powerful teachingWhether you’re an AI beginner or a senior PM, Mahesh breaks down complex concepts so clearly a 5-year-old could follow—yet every AI expert still learns something new.✅ Continued support after the courseYour learning doesn’t end in 4 weeks. Mahesh continues to mentor you via free replays, ongoing labs, community sessions, and exclusive invites to alumni Bootcamps.✅ Join a real movementOver 3,000 professionals have taken Mahesh’s courses. Many now lead AI products at top-tier companies or have transitioned into dream GenAI roles. This isn’t just a course. It’s your entry into an elite, supportive AI PM network. What Makes This Course Different?This course is unlike any other AI Product Management program. It’s built to give you deep, real-world skills—not just surface-level learning. You’ll learn to build production-grade AI agents, create enterprise-ready product requirement documents. What makes this course truly stand out is its comprehensive approach to teaching AI Product Management the way it’s done in MAANG companies.You'll gain a strong understanding of the entire AI ecosystem—how AI players operate, the key infrastructure gaps, the full stack, and where the industry is heading. We focus on developing your specific AI PM chops: discovery, design, deployment, pricing, and GTM strategies tailored for AI.The course demystifies complex technical topics like neural networks, transformers, fine-tuning, multi-agent communication, and evaluation—explained in a way that even a beginner can grasp, yet still packed with depth for experienced learners. Non-technical PMs are fully supported and encouraged to dive in—we provide extensive support hours to help everyone succeed.You’ll go beyond just building; you’ll write detailed PRDs for your own AI idea, receive tailored feedback, and turn your vision into an enterprise-grade working prototype. And because we know learning doesn’t stop after the cohort ends, you’ll get lifetime access to recordings, future labs, and our ongoing community sessions—at no extra cost. What You’ll Learn✅ Spotting high-impact AI use cases in any business✅ Master agentic design (memory, knowledge, reflection, focus)✅ Build multi-agent workflows with tools &amp; data✅ Design and write MAANG-level PRDs✅ Use advanced prompting (CoT, ToT) for real applications✅ Deploy agents using Langflow, CrewAI, OpenAI, Vertex AI, and more✅ Grasp core ML concepts — neural networks, transformers, memory mgmt✅ Learn pricing, GTM, and scaling strategies for AI products✅ Work on real MAANG-level case studies Ready to build your first AI Multi-Agent and become an AI product leader?

Master Agentic AI PM skills, concepts by building PRD/MVP that customer love. 

Recordings will be provided in the student portal if you cannot join live.

5 Saturdays 90 mins sessions with Mahesh

You will have 5 lab sessions to build with Mahesh Teams. It is also our weekly open office hour.

Wed Lab Sessions/Optional Q&A

We want to help you to continue your learning journey even after the cohort ends. We meet weekly on my LinkedIn community. Current and previous cohort members join us to keep learning. Join us for a free event:https://www.linkedin.com/company/mahesh-ai-pm-community

Weekly Community Sessions

Product Managers Traditional PM looking to transition into AI Product Management. This course will help you more with GenAI.

Engineers and Designers:Ready to transition into the PM world? Learn to bridge the gap between technical implementation &amp; product strategy

Founders:If you're looking to turn ideas into MVP &amp; gain the knowledge &amp; confidence to lead your team in building Ai Products

Builders Ready to Upskill into the PM world with real toolsLearn to bridge the gap between technical implementation &amp; product strategy

Are you wondering if this course is for you ?

Your Takeaways From This Cohort

Go from beginner to confident AI Product Manager in just a few weeks. In this course, you’ll follow a hands-on, project-based approach to quickly master both the fundamentals and advanced skills needed to build production-ready AI agents that solve real problems.

Fast-Track Your AI PM Journey

Our alumni are landing promotions, pivoting into AI roles, and stepping into leadership positions at top tech companies. As part of the cohort, you’ll tap into an active network that shares referrals, job tips, and insider guidance to help you break into or level up in the AI spa

Accelerate Your Career in AI
 

If you're a founder or builder, this course is your launchpad. You’ll be equipped to take your idea and turn it into a functional AI MVP tailored to your business needs - right away.

Turn Ideas Into Real Products

You won’t just leave with skills—you’ll leave with connections. Join a community of like-minded professionals, learn from each other, and keep growing long after the course ends.

A Lifelong Community That Grows With You

Build AI powered products, Master Evals, and land your Next AI role - In guidance with MAANG leader who has been there done that many times.

Mahesh Yadav

Through multiple real examples of well-run experiments and real stories at Microsoft, Amazon, and Airbnb, you will see the humbling reality that we are terrible at assessing the values of ideas. Trivial changes can be surprisingly useful, whereas large efforts often fail. Accelerate innovation by building Minimum Viable Products and Features (MVPs) and make the organization evidence-based and humbler, as it adopts and learns to use evidence from the gold standard in science: the controlled experiment. You will understand the challenges in designing and running trustworthy controlled experiments, or A/B tests, including the importance of the Overall Evaluation Criterion (OEC), scaling, pitfalls, and Twyman's law. The 2nd week covers additional topics, some more technical, including cultural challenges, institutional memory, maturity model, observational causal studies, offline evaluations, AI/Machine learning and triggering, scaling, build vs. buy, challenges, and requested topics. The advanced topics course is a follow-on to this: https://bit.ly/AdvancedABFS

Empower your organization to be data-driven and innovative

Three x 2-hour sessions in week 1 on Monday, Tuesday, and Thursday

Week 1: Mon, Tue, Thu

Two x 2-hour sessions in week 2 on Monday and Thursday

Week 2: Mon, Thu

Optional Q&A

Data science managers and scientists will be able to design and interpret the experiment results in a trustworthy manner

Program managers focused on growth, revenue, conversions, and prioritization will understand how to provide the org with robust clear metric

Engineering leaders will be able to make the organizations more data-driven and efficient with fewer severe incidents through A/B tests

Who is this class for?

What you will learn from this 10-hour course

You will hear multiple real memorable stories and examples, many that don't make it to books or articles. These were chosen from over 20 years of experimentation. You'll have the data to show that the poor success rate is documented across multiple organizations; expected it!

Understand and internalize the humbling reality that we are poor at assessing the values of ideas: most ideas fail!

Understand key concepts like causality, hierarchy of evidence, and key organizational tenets required for effective experimentation.

Understand the key advantages and limitations of A/B testing

Designing metrics is hard. There is a hierarchy of metrics and perverse incentives. The most important metrics comprise of the OEC - The Overall Evaluation Criterion. We will look at good and bad examples.

Learn how to design metrics and the Overall Evaluation Criterion

Getting numbers is easy; getting numbers you can trust is hard. You'll understand common pitfalls and how to design reliable and trustworthy tests.

Learn how to designing trustworthy A/B tests

Learn about the cultural challenges, the humbling results (most ideas fail, pivoting, iterating, learning), institutional memory, ideation, prioritization, experimentation platforms

Learn about the cultural challenges

When building AI or machine learning models, using A/B testing and triggering to evaluate the models that were built offline based on historical data

Learn about the relationship to AI and Machine Learning Modeling

When you can't run an A/B test, quasi-experimentation methods and the risks of observational causal studies

Learn about complementary quasi-experimental techniques

What are key challenges and open questions in the field

Learn about existing challenges

If there is something specific you want to cover, there is time allocated for topics voted by the audience to discuss

Learn about YOUR topic of interest

Learn from a world-leading expert how to design and analyze trustworthy A/B tests to evaluate ideas, integrate AI/ML, and grow your business