Hamel Husain & Shreya Shankar

Cohort 1

Cohort 1 has $1,000 in modal credits.  This is NOT guaranteed for Cohort 2

Cohort 2

Instant, on demand access to all lessons.

NOT HAPPENING

All lectures are recorded, 8x live office hours.

Cohort 3

Cohort 1: Debug Your Data

Cohort 4

Cohort 6

Cohort 5

These a set of skills that for AI product evals. They distill what I’ve learned implementing evals at 50+ companies and teaching them to 4,000+ students in our&nbsp;<a href="https://maven.com/parlance-labs/evals" rel="noopener noreferrer" target="_blank">course</a>.

Evals Skills For Claude

12 pages with visual bite-sized takeaways of the most important learnings from our course.

AI Eval Flashcards

This mini-book is a compilation of advanced talks on Retrieval Augmented Generation (RAG) that were part of our AI Evals course. The series features leading researchers who share state-of-the-art techniques for optimizing and evaluating RAG systems in applied settings. This book condenses over 5 hours of expert instruction into annotated presentations that allow you to quickly learnadvanced RAG techniques without watching hours of video.

Beyond Naive RAG: Practical
Advanced Methods

These are the most frequent questions we've received while teaching 2000+ engineers and product managers AI evals.

Frequently Asked Questions (And Answers) About AI Evals

We provide each student with a 150 page book, organized by 11 chapters that supplements their learning. This preview contains the table of contents and the first chapter.

Preview Our Course Reader

Lessons from 30+ AI implementations After&nbsp;<a href="https://parlance-labs.com/" rel="noopener noreferrer" target="_blank">advising 30+ companies</a>&nbsp;on generative AI (and 25 years of industry experience), we're sharing our learnings in this&nbsp;free 8-day email course. This is a free preview of our 2-week intensive <a href="https://maven.com/parlance-labs/gen-ai-mastery-for-busy-executives" rel="noopener noreferrer" target="_blank">course</a> starting December 2nd. What You'll Learn <ol><li>Strategies to drive ROI instead of creating vanity projects</li><li>A curated list of our preferred AI tools and vendors</li><li>Real-world case studies</li><li>The most common hiring mistakes (and how to avoid them)</li></ol> 👉 <a href="https://hamel.ck.page/ai-for-execs" rel="noopener noreferrer" target="_blank">Sign up here</a> 👈, or fill out the email below.

Free Email Course: AI Mastery for Busy Executives

Dan Becker

Chief Generative AI Architect @ Straive

Hamel Husain

Founder @ Parlance Labs

Mastering LLMs For Developers & Data Scientists

Data scientist

Data Scientist 

Full Stack Computational Linguist

Machine Learning Engineer

Principal Research Scientist

Senior Machine Learning Engineer

Learn proven approaches for quickly improving AI applications.  Build AI that works better than the competition, regardless of the use-case.

ML Engineer with 20 years of experience

Airbnb

GitHub

DataRobot

AlixPartners

ML Engineer with 20 years of experience.

ML Systems & Applied AI Evals Researcher

Google

UC Berkeley

Stanford University

Shreya Shankar

ML Systems Researcher Making AI Evaluation Work in Practice

🚨 New: August 2026 cohort's material is completely refreshed to cover advancements in the field 🚨 All students get:<ul><li>♾️ Unlimited access to future cohorts &amp; office hours: never worry about timing or missing new material.</li><li>🗄️ Lifetime access to all materials!</li><li>🤖 6 months of unlimited access to our new AI Eval Assistant (more info below).</li><li>🧑‍🏫 10+ hours of office hours to maximize the value of live interaction.</li><li>🏫 A Discord community with continuous access to instructors to get unstuck (even after the course!).</li></ul>---Do you catch yourself asking any of the following questions while building AI applications?1. How do I test applications when the outputs require subjective judgements?2. If I change the prompt, how do I know I'm not breaking something else?3. Where should I focus my engineering efforts? Do I need to test everything?4. What if I have no data or customers, where do I start?5. What metrics should I track? What tools should I use?6. Can I automate testing and evaluation? If so, how do I trust it?If so, this course is for you.All sessions are live and recorded.

Stop guessing if your AI works. Build the feedback loops that make it better.

How To Collect Data For Evals

Get immediate clarity and direction with Error Analysis

Implement Effective Evaluations

Master Architecture-Specific Eval Strategies

Run Evals In Production

Ensure That Evals Lead To High ROI

Engineers and PMs who ship prompt changes and hope nothing breaks. (You'll learn to measure impact before and after every change.)

Teams still spot-checking AI outputs by hand instead of measuring systematically. (You'll learn how build automated evals you can trust.)

Leaders who don't know where their AI is failing or where to invest resources. You'll learn how to systematically find &amp; prioritize issues.

This course is $20 only because Maven doesn't allow free courses. Build skills to be effective with LLMs--- This started as an LLM fine-tuning course. It organically grew into a learning event with world-class speakers on a broad range of LLM topics. The original fine-tuning course is still here as a series of workshops. But there are now many self-contained talks and office hours from experts on many Generative AI topics. All materials + recordings are available instantly, on demand. There are 11 talks and 4 workshops (and growing) in addition to office hours. THIS IS A PAST COURSE THAT WE HAVE MADE NEARLY FREE ($20), ALL TALKS ARE RECORDED AND ACCESSIBLE THROUGH MAVEN FOR LIFE. WE ARE GIFTING THIS COURSE TO THE COMMUNITY. SEE: https://hamel.dev/blog/posts/course/ There is no active support, office hours, or ability to ask questions from instructors. Conference Talks ------------------------------Jeremy Howard: Co-Founder Answer.AI &amp; Fast.AI - Build Applications For LLMs in PythonSophia Yang: Head of Developer Relations, Mistral AI- Best Practices For Fine Tuning MistralSimon Willison: Creator of Datasette, co-creator of Django, PSF Board Member- Language models on the command-lineJJ Allaire: CEO, Posit (formerly RStudio) &amp; Researcher for the UK AI Safety Institute- Inspect, An OSS framework for LLM evalsWing Lian: Creator of Axolotl library for LLM fine-tuning- Fine-Tuning w/AxolotlMark Saroufim and Jane Xu: PyTorch developers @ Meta- Slaying OOMs with PyTorch FSDP and torchaoJason Liu: Creator of Instructor- Systematically improving RAG applications&nbsp;Paige Bailey: DevRel Lead, GenAI, Google- When to Fine-Tune?Emmanuel Ameisen: Research Engineer, Anthropic- Why Fine-Tuning is DeadHailey Schoelkopf: research scientist, Eleuther AI, maintainer, LM Evaluation Harness- A Deep Dive on LLM EvaluationJohno Whitaker: R&amp;D at AnswerAI- Fine-Tuning Napkin MathJohn Berryman: Author of O'Reilly Book Prompt Engineering for LLMs- Prompt Eng Best PracticesBen Clavié: R&amp;D at AnswerAI- Beyond the Basics of RAGAbhishek Thakur leads AutoTrain at HuggingFace- Train (almost) any llm model using 🤗 AutotrainKyle Corbitt is currently building OpenPipe- From prompt to model: fine-tuning when you've already deployed LLMs in prodAnkur Goyal: CEO and Founder at Braintrust- LLM Eval For Text2SQLFreddy Boulton: Software Engineer at 🤗- Let's Go, Gradio!Jo Bergum: Distinguished Engineer at Vespa- Back to basics for RAG Fine-Tuning Course---------------------------Run an end-to-end LLM fine-tuning project with modern tools and best practices. Four workshops guide you through productionizing LLMs, including evals, fine-tuning and serving. Workshop 1: Determine when (and when not) to fine-tune an LLMWorkshop 2: Train your first fine-tuned LLM with AxolotlWorkshop 3: Set up instrumentation and evaluation to incrementally improve your modelWorkshop 4: Deploy Your Model This is accompanied by 5+ hours of office hours. Lectures explain the why and demonstrate the how for all the key pieces in LLM fine-tuning. Your hands-on-experience in the course project will ensure your ready to apply your new skills in real business scenarios. The Fine-Tuning course has these guest speakers: - Shreya Shankar: LLMOps and LLM Evaluations researcher- Zach Mueller: Lead maintainer of HuggingFace accelerate- Bryan Bischof: Director of AI Engineering at Hex- Charles Frye: AI Engineer at Modal Labs- Eugene Yan: Senior Applied Scientist @ Amazon- Harrison Chase: CEO of LangChain- Travis Addair: Co-Founder &amp; CTO of Predibase- Joe Hoover: Lead ML Engineer at ReplicateFAQ: ------- Q: It says this course already started. Should I still Enroll?A: Yes. Everything is recorded, so you can watch videos for any events that have happened so far, join for live events moving forward, and even learn from talks long after the conference is over. Q: Will there be a future cohort?A: No. We were fortunate to have so many world-class speakers. We don't think this can be replicated, so it is now a one-time-only event with all recordings available. Q: Are you still giving out free compute credits?A: No. Students who enrolled after 5/29/2024 are not eligible for compute credits.

👉 SEE OUR NEW COURSE - AI Evals For Engineers & PMs: https://evals.info 👈

Course syllabus

Interactive weekly workshops where you will learn the tools you will apply in your course project.

Tuesdays

You will build and deploy an LLM as part of the course project. The course project is divided into four weekly project. By the end, you will not only know about fine-tuning, but you will have hands-on experience doing it.

Weekly projects

Course schedule

Data scientists looking to repurpose skills from conventional ML into LLMs and generative AI

Software engineers with Python experience looking to add the newest and most important tools in tech

Programmers who have called LLM APIs that now want to take their skills to the next level by building and deploying fine-tuned LLMs

Who Is It For?

What you’ll get out of this conference

Discord with 1000+ members attending the conference.

Connect With A Large Community Of AI Practitioners

Topics such as RAG, Evals, Inference, Fine-Tuning, are covered.

Learn more about LLMs

We have curated the tools that we like the most. Credits for many of these tools are provided.

Learn about the best tools

This conference used to be a fine-tuning LLMs course. That course is still here, and takes place over the course of 4 workshops.

Learn about fine-tuning in-depth

After advising 30+ companies, interviewing over 600 AI practitioners and saving AI projects that cost millions to fix, we've seen a clear pattern: Executives who successfully implement AI aren't the ones with the biggest budgets or the most advanced technology. They're the ones who know how to: - Bridge the gap between technical teams and business goals- Focus on process before tools- Build the right capabilities at the right time- Create feedback loops that catch problems early They have the best mental models and know how to use them. This program shows you exactly how to build and deploy them. WHAT MAKES THIS COURSE DIFFERENT - Executive-Centric Approach: We've tailored it for those who need actionable insight without technical overload.- Expert Instructors: Sessions led by three industry veterans with decades of hands-on AI experience from Strategy to Implementation across leading technology companies.- Designed for Implementation: Focuses on execution rather than theory. Shows you exactly how to measure results.- Interactive Learning: Engage with peers, think through real problems, and apply what you learn immediately. TOPICS YOU WILL MASTER: 1. Crafting a Robust AI Strategy- Diagnosing organizational challenges and opportunities- Identifying sources of competitive advantage- Communicating AI strategy effectively using "Business on a Page" 2. Mobilizing Your Organization for AI Success- Implementing the "Zone to Win" framework for AI initiatives- Allocating resources effectively for AI projects- Fostering AI adoption across your organization 3. AI Implementation: From Tools to Processes- Taking a process-first approach to AI implementation- Designing effective AI evaluation metrics- Linking AI initiatives to business outcomes 4. Building and Managing High-Performance AI Teams- Developing an AI talent roadmap- Making informed build vs. buy decisions for AI solutions- Managing AI specialists effectively WHERE ELSE CAN I LEARN THIS STUFF:Executives who take this course will learn how to master Generative AI and make the right decisions for themselves and organizations not available anywhere else:- Not in any books- Not in briefings, on podcasts, or in any executive coaching.- It's not even available in our own writing because -- it's too deep, example-driven and extensive to cover fully in a newsletter. WHO SHOULD TAKE THIS COURSE:- Senior decision makers and strategists responsible for steering their organizations through the fast evolving AI landscape.- Directors, Senior Directors, VPs and SVPs aiming to transform their functions and understand how to integrate AI into their business to drive customer value.- C-Suite Executives (CEOs, CTOs, CIOs, CMOs, CPOs) looking to stay ahead of competitors by effectively amplifying their AI IQ and operating leverage but don't have time for extensive technical deep dives.- Entrepreneurs and Founders who need to really understand AI and its sharp points in order to innovate and scale their businesses. Is This You?- You're overwhelmed by AI hype and need to cut through the noise.- You want actionable insights without getting lost in technical jargon.- You're eager to boost productivity—for yourself and your teams.- You're ready to make informed decisions about AI investments, risks, and strategies. What insights could you gain if you were part of this exclusive group? WHO SHOULD NOT TAKE THIS COURSE:- AI practitioners who are already on the implementation frontier- Those seeking to improve their hands-on coding skills or in-depth technical training on AI algorithms.- Individuals looking for a basic overview without practical applications. YOUR HOSTS We bridge the gap between AI expertise and executive leadership—helping you implement what works Hamel Husain | AI Practitioner25+ years building AI products, including at GitHub and Microsoft. Now advises Fortune 500s on AI implementation. Greg Ceccarelli | Executive LeaderFormer CPO at Pluralsight. Data exec at GitHub. Built and scaled AI teams that delivered measurable results. Sam Charrington | Executive AdvisorHost of the TWIML AI podcast, Sam is an industry analyst, speaker and thought leader. From his more than 600 interviews he knows what works and what doesn't.

Play to win with AI 

If your events are recurring and at the same time, it might be easiest to use a single line item to communicate your course schedule to students

Tuesdays & Thursdays

Feel free to type out dates as your title as a way to communicate information about specific live sessions or other events.

May 7, 2022

Schedule items can also be used to convey commitments outside of specific time slots (like weekly projects or daily office hours).

Directors, Senior Directors, VPs and SVPs&nbsp;aiming to transform their functions and understand how to integrate AI into their business.

C-Suite Executives (CEOs, CTOs, CIOs, CMOs, CPOs, etc)&nbsp;looking to stay ahead of competitors by amplifying their AI IQ and operating leverage

Entrepreneurs and Founders who need to really understand AI and its sharp points in order to innovate and scale their businesses.

Who is this course for

What you’ll get out of this course

Developed over two decades of experience, leverage templates to effectively diagnose, communicate, and mobilize your AI strategy across all audiences

Master our exclusive 3 part strategic framework

Empower your organization to identify and mitigate potential AI implementation pitfalls before they become costly mistakes.

Employ a comprehensive AI risk assessment toolkit

Build the right team at the right time to drive your AI initiatives forward effectively.

Design a stage-appropriate AI talent roadmap

Enable your leadership team to uncover hidden biases and quality issues in AI systems that even your technical team might miss.

Gain hands-on data review skills

Get direct help with your specific AI challenges. We'll review your data, debug issues, and refine your strategy in interactive sessions. We will hold office hours daily.

Live Problem-Solving Sessions

Identify quick wins and long-term opportunities tailored to your specific business needs and stage

Conduct a thorough AI readiness assessment

Get exclusive access to our searchable directory of over 8000 AI products. Receive cherry picked recommendations across use cases.

Evaluate and assess the best AI products

Master the mental models required to implement winning AI initiatives and deliver ROI. From strategy to implementation. Learn with experts.

Greg Ceccarelli

AI Product Advisor @ Tola Capital

Sam Charrington

Host @ TWIML AI Podcast

What We Do

Consulting Services

What We Do Together

We use your real data to find exactly where your AI is failing, prioritize fixes, and build a measurement plan.

ML Engineer with 20+ years of experience

Debug Your AI: Private Team Workshop

Find and prioritize your AI's biggest failures in 2 days — using your own data.

ML Engineer with 20+ years experience

Trained 4,000+ AI practitioners from 500+ companies on AI evaluation.

ML Systems Researcher 

This isn't a course — we roll up our sleeves and work on your product with your data.You bring your real traces and interaction logs. Over 2 days, we systematically find where your AI is breaking down, why users are losing trust, and what to fix first. Most teams building AI products are stuck in the same loop: ship a change, hope it helps, manually spot-check a few outputs, repeat. Meanwhile, users hit failures nobody on the team has even seen.In this private workshop, we break that cycle. You bring real traces and interaction logs from your product. Together, we systematically uncover where your AI is breaking down, build a catalog of failure modes specific to your use case, and leave with a prioritized plan to fix what matters most.This methodology has been refined across 4,000+ practitioners from 500+ companies, and is part of what we teach in our <a target="_blank" rel="noopener noreferrer nofollow" href="https://maven.com/parlance-labs/evals">full-length course</a>. It's the highest-ROI activity in AI product development.Each workshop can accommodate a team of up to 6 participants. We can help you identify which team members to bring (trust us, you don't want more than 6).Below is a sample agenda, this can be customized to your needs and schedule.

Walk away knowing exactly where your AI product is failing, why, and what to fix first — with a concrete plan your team can execute immediately.

Find hidden failure patterns in your AI product

Build a prioritized roadmap to fix what matters

Design metrics that prove your fixes work

This workshop is hands-on with your data. You need a deployed product generating real interactions we can analyze together.

A live AI product or feature with real users

We work directly with your production data. You'll receive a prep guide with instructions on what to collect and how to format it.

Access to recent traces or interaction logs

Engineers, PMs, and domain experts who know the product. The best results come from cross-functional teams who can act on findings immediately.

A dedicated team of 2-5 participants

Engineering teams with a live AI product that has real users.

Product managers responsible for AI features who need a data-driven improvement plan instead of guessing which changes will move the needle.

Technical leaders who want to stop relying on manual QA and spot-checks, and instead build a systematic approach to improving AI quality.

Automating Evals With Claude Code + Phoenix

Mikyo King

Head of Open Source. Building the future of AI Observability at Arize AI

Set up the toolchain to give Claude Code direct CLI access to your Phoenix traces, spans, and performance metrics.

Connect Claude Code to Phoenix observability data

Learn important Phoenix CLI tools for retrieving observability data and identifying errors in agent-generated code.

Use CLI commands to fetch traces and debug agents

Practice natural language prompts that enable Claude Code to interpret traces, diagnose issues, and suggest fixes.

Prompt AI to analyze system behavior in real-time

Building Effective AI-Powered Data Pipelines

Transform unstructured documents into structured data by chaining semantic operators and classifying data at scale.

Architect Semantic Data Pipelines

Improve accuracy by decomposing complex operators (e.g., converting a "Map" into "Split-Map-Reduce")

Optimize via Semantic Rewrites

Learn how to route easy queries to cheaper models and hard ones to expensive models without sacrificing quality .

Slash Costs with Task Cascades

You Don't Need a Graph DB

Jo Kristian Bergum

Search & Retrieval Expert, CEO of Hornet.dev

Understand the use cases where dedicated graph DBs provide real value versus when they're unnecessary complexity.

When graph databases actually make sense

Learn how to represent (node, edge, node) relationships using PostgreSQL, MySQL, or your existing db infra.

Implement graph structures in general-purpose databases

Discover why defining nodes and edges is the hard part, not traversal speed, and how to approach knowledge graph design

The real bottleneck: graph modeling, not performance

Explore how search engines, document stores, and SQL databases can handle many graph workloads without specialized tools

Alternatives that might serve you better

Context Rot: When Long Context Fails

Kelly Hong

Retrieval Researcher @ Chroma

Why standard benchmarks are misleading and fail to capture the real-world challenges of long-context applications.

Beyond "Needle in a Haystack"

Discover how topically-related "distractors" and even the logical flow of your context can degrade model performance.

How Distractors & Context Structure Cause Failure

Learn why how you present information to an LLM is just as critical as what you present for achieving reliable result

The Importance of Context Engineering

See how different models like Claude and GPT exhibit unique failure patterns, from abstention to confident hallucination

Model-Specific Failure Modes

RAG Without the Lag: Interactive Debugging For RAG Pipelines

Quentin Romero Lauro

Retrieval & RAG Researcher

PhD Student at Berkeley EECS

Over 70% of critical parameter changes require hours of re-processing the data, killing developer momentum.

The Hidden Debugging Crisis in RAG Development

Learn how chunk size changes force prompt rewrites, query rewriting impacts retrieval, and other cascading effects.

Interdependency Traps That Breaks RAG Systems

Discover why engineers always debug retrieval first, plus real-world patterns from 12 engineers for effective testing.

How Engineers Really Debug RAG Pipelines

Optimize Your Dev Setup For Evals w/ Cursor Rules & MCP

Isaac Flath

AI Engineer & Fullstack Developer

Configure MCPs to pull llms.txt intelligently or data from your eval system to automate data analysis and debugging.

How to use MCP context for AI evaluation frameworks

Customize your dev environment for the specific tool you are using and your preferences.  Isaac will share his recipes.

Cursor rules for Phoenix, Braintrust, and Inspect

Greatly reduce the friction of setting up evals by automating away the tedious bits.

Learn directly from Hamel Husain & Shreya Shankar

Hamel Husain & Shreya Shankar

Courses

AI Evals For Engineers & PMs

Mastering LLMs For Developers & Data Scientists

AI Mastery for Busy Executives

Workshops

Debug Your AI Product: Private Team Workshop

Lightning Lessons