Hamel Husain & Shreya Shankar

Cohort 1

Cohort 1 has $1,000 in modal credits.  This is NOT guaranteed for Cohort 2

Cohort 2

Instant, on demand access to all lessons.

NOT HAPPENING

All lectures are recorded, 8x live office hours.

Cohort 3

Cohort 4

Cohort 1: Debug Your Data

Cohort 6

Cohort 5

These a set of skills that for AI product evals. They distill what I’ve learned implementing evals at 50+ companies and teaching them to 4,000+ students in our&nbsp;<a href="https://maven.com/parlance-labs/evals" rel="noopener noreferrer" target="_blank">course</a>.

Evals Skills For Claude

12 pages with visual bite-sized takeaways of the most important learnings from our course.

AI Eval Flashcards

This mini-book is a compilation of advanced talks on Retrieval Augmented Generation (RAG) that were part of our AI Evals course. The series features leading researchers who share state-of-the-art techniques for optimizing and evaluating RAG systems in applied settings. This book condenses over 5 hours of expert instruction into annotated presentations that allow you to quickly learnadvanced RAG techniques without watching hours of video.

Beyond Naive RAG: Practical
Advanced Methods

These are the most frequent questions we've received while teaching 2000+ engineers and product managers AI evals.

Frequently Asked Questions (And Answers) About AI Evals

We provide each student with a 150 page book, organized by 11 chapters that supplements their learning. This preview contains the table of contents and the first chapter.

Preview Our Course Reader

Lessons from 30+ AI implementations After&nbsp;<a href="https://parlance-labs.com/" rel="noopener noreferrer" target="_blank">advising 30+ companies</a>&nbsp;on generative AI (and 25 years of industry experience), we're sharing our learnings in this&nbsp;free 8-day email course. This is a free preview of our 2-week intensive <a href="https://maven.com/parlance-labs/gen-ai-mastery-for-busy-executives" rel="noopener noreferrer" target="_blank">course</a> starting December 2nd. What You'll Learn <ol><li>Strategies to drive ROI instead of creating vanity projects</li><li>A curated list of our preferred AI tools and vendors</li><li>Real-world case studies</li><li>The most common hiring mistakes (and how to avoid them)</li></ol> 👉 <a href="https://hamel.ck.page/ai-for-execs" rel="noopener noreferrer" target="_blank">Sign up here</a> 👈, or fill out the email below.

Free Email Course: AI Mastery for Busy Executives

Engineers and engineering leaders who build, deploy, and scale software systems.

Data scientists who apply statistical and ML techniques to extract insights from data.

Applying AI tools and techniques to professional workflows — assign alongside more specific AI topics.

Operating and deploying large language models in production — fine-tuning, inference, monitoring, and cost optimization.

Dan Becker

Chief Generative AI Architect @ Straive

Hamel Husain

Founder @ Parlance Labs

Mastering LLMs For Developers & Data Scientists

Market Development Associate

Data scientist

Data Scientist 

Full Stack Computational Linguist

Machine Learning Engineer

Principal Research Scientist

Senior Machine Learning Engineer

Build a real AI agent, find where it breaks, and improve it with evals you can trust, working the full loop hands-on.

ML Engineer with 25 years of experience

Airbnb

GitHub

DataRobot

AlixPartners

ML Engineer with 25 years of experience.

ML Systems & Applied AI Evals Researcher

Google

UC Berkeley

Stanford University

Shreya Shankar

ML Systems Researcher Making AI Evaluation Work in Practice

🏆 The most field-tested evals course available. We've refined this exact material over a year of cohorts with 4,500+ engineers and PMs from teams like OpenAI, Google, Meta, Amazon, and Microsoft, folding their feedback in every time. <a target="_blank" rel="noopener noreferrer nofollow" href="https://testimonial.to/ai-evals-course/all">Read what students say →</a> More than a course: an ongoing system, tools, and community for shipping better AI.<ul><li>🎮 A private Discord community for ongoing support, even after the course.</li><li>🤖 6 months of access to our AI Evals assistant.</li><li>♾️ Lifetime access to all recordings, materials, and future cohorts.</li><li>💬 10+ hours of live office hours to get your questions answered.</li></ul>Do you catch yourself asking any of these while building AI applications?<ol><li>How do I test outputs that need subjective judgment?</li><li>If I change the prompt, how do I know I am not breaking something else?</li><li>Where should I focus my efforts? Do I need to test everything?</li><li>What if I have no data or customers yet? Where do I start?</li><li>What should I measure, and what tools should I use?</li><li>Can I automate evaluation, and how do I trust it?</li></ol>If so, this is for you. All sessions are live and recorded.📝 We've completely refreshed the material for our Sep 2026 cohort. Scroll down to see the full syllabus.

 Build, evaluate, and improve AI agents that work in production.

Build an agent you can measure

Find what's broken with error analysis

Build evaluators you trust

Catch regressions with CI/CD

Red-team for safety before users do

Improve accuracy and cut cost with evidence

Engineers and PMs who ship prompt changes and hope nothing breaks. (You'll learn to measure impact before and after every change.)

Teams still spot-checking AI outputs by hand instead of measuring systematically. (You'll learn how to build automated evals you can trust.)

Leaders who don't know where their AI is failing or where to invest resources. You'll learn how to systematically find &amp; prioritize issues.

This course is $20 only because Maven doesn't allow free courses. Build skills to be effective with LLMs--- This started as an LLM fine-tuning course. It organically grew into a learning event with world-class speakers on a broad range of LLM topics. The original fine-tuning course is still here as a series of workshops. But there are now many self-contained talks and office hours from experts on many Generative AI topics. All materials + recordings are available instantly, on demand. There are 11 talks and 4 workshops (and growing) in addition to office hours. THIS IS A PAST COURSE THAT WE HAVE MADE NEARLY FREE ($20), ALL TALKS ARE RECORDED AND ACCESSIBLE THROUGH MAVEN FOR LIFE. WE ARE GIFTING THIS COURSE TO THE COMMUNITY. SEE: https://hamel.dev/blog/posts/course/ There is no active support, office hours, or ability to ask questions from instructors. Conference Talks ------------------------------Jeremy Howard: Co-Founder Answer.AI &amp; Fast.AI - Build Applications For LLMs in PythonSophia Yang: Head of Developer Relations, Mistral AI- Best Practices For Fine Tuning MistralSimon Willison: Creator of Datasette, co-creator of Django, PSF Board Member- Language models on the command-lineJJ Allaire: CEO, Posit (formerly RStudio) &amp; Researcher for the UK AI Safety Institute- Inspect, An OSS framework for LLM evalsWing Lian: Creator of Axolotl library for LLM fine-tuning- Fine-Tuning w/AxolotlMark Saroufim and Jane Xu: PyTorch developers @ Meta- Slaying OOMs with PyTorch FSDP and torchaoJason Liu: Creator of Instructor- Systematically improving RAG applications&nbsp;Paige Bailey: DevRel Lead, GenAI, Google- When to Fine-Tune?Emmanuel Ameisen: Research Engineer, Anthropic- Why Fine-Tuning is DeadHailey Schoelkopf: research scientist, Eleuther AI, maintainer, LM Evaluation Harness- A Deep Dive on LLM EvaluationJohno Whitaker: R&amp;D at AnswerAI- Fine-Tuning Napkin MathJohn Berryman: Author of O'Reilly Book Prompt Engineering for LLMs- Prompt Eng Best PracticesBen Clavié: R&amp;D at AnswerAI- Beyond the Basics of RAGAbhishek Thakur leads AutoTrain at HuggingFace- Train (almost) any llm model using 🤗 AutotrainKyle Corbitt is currently building OpenPipe- From prompt to model: fine-tuning when you've already deployed LLMs in prodAnkur Goyal: CEO and Founder at Braintrust- LLM Eval For Text2SQLFreddy Boulton: Software Engineer at 🤗- Let's Go, Gradio!Jo Bergum: Distinguished Engineer at Vespa- Back to basics for RAG Fine-Tuning Course---------------------------Run an end-to-end LLM fine-tuning project with modern tools and best practices. Four workshops guide you through productionizing LLMs, including evals, fine-tuning and serving. Workshop 1: Determine when (and when not) to fine-tune an LLMWorkshop 2: Train your first fine-tuned LLM with AxolotlWorkshop 3: Set up instrumentation and evaluation to incrementally improve your modelWorkshop 4: Deploy Your Model This is accompanied by 5+ hours of office hours. Lectures explain the why and demonstrate the how for all the key pieces in LLM fine-tuning. Your hands-on-experience in the course project will ensure your ready to apply your new skills in real business scenarios. The Fine-Tuning course has these guest speakers: - Shreya Shankar: LLMOps and LLM Evaluations researcher- Zach Mueller: Lead maintainer of HuggingFace accelerate- Bryan Bischof: Director of AI Engineering at Hex- Charles Frye: AI Engineer at Modal Labs- Eugene Yan: Senior Applied Scientist @ Amazon- Harrison Chase: CEO of LangChain- Travis Addair: Co-Founder &amp; CTO of Predibase- Joe Hoover: Lead ML Engineer at ReplicateFAQ: ------- Q: It says this course already started. Should I still Enroll?A: Yes. Everything is recorded, so you can watch videos for any events that have happened so far, join for live events moving forward, and even learn from talks long after the conference is over. Q: Will there be a future cohort?A: No. We were fortunate to have so many world-class speakers. We don't think this can be replicated, so it is now a one-time-only event with all recordings available. Q: Are you still giving out free compute credits?A: No. Students who enrolled after 5/29/2024 are not eligible for compute credits.

👉 SEE OUR NEW COURSE - AI Evals For Engineers & PMs: https://evals.info 👈

Course syllabus

Interactive weekly workshops where you will learn the tools you will apply in your course project.

Tuesdays

You will build and deploy an LLM as part of the course project. The course project is divided into four weekly project. By the end, you will not only know about fine-tuning, but you will have hands-on experience doing it.

Weekly projects

Course schedule

Data scientists looking to repurpose skills from conventional ML into LLMs and generative AI

Software engineers with Python experience looking to add the newest and most important tools in tech

Programmers who have called LLM APIs that now want to take their skills to the next level by building and deploying fine-tuned LLMs

Who Is It For?

What you’ll get out of this conference

Discord with 1000+ members attending the conference.

Connect With A Large Community Of AI Practitioners

Topics such as RAG, Evals, Inference, Fine-Tuning, are covered.

Learn more about LLMs

We have curated the tools that we like the most. Credits for many of these tools are provided.

Learn about the best tools

This conference used to be a fine-tuning LLMs course. That course is still here, and takes place over the course of 4 workshops.

Learn about fine-tuning in-depth

Can AI Actually Answer Your Data Questions?

The cold-start eval problem: evals for new AI products

Lucas Machado Rocha

Evals Your Whole Team Can Run, Not Just the Experts

Marek Galovic

Isaac Flath

Scaling Late Interaction to Billions of Documents

Joe Barrow

How to choose an OCR Model

Stop Letting Your LLM "Find the Issues" In Your Data

Materials From The Talk

We will send you a link to the slides along with other resources!

ML Engineer w 25 years of Experience

ML Systems Researcher 

Materials From: Return Of The Data Scientist

ML Engineer with 20 years of experience

Two veterans with 40+ years of combined AI experience.

Greg Ceccarelli

Sam Charrington

Find and prioritize your AI's biggest failures in 2 days — using your own data.

ML Engineer with 20+ years experience

Trained 4,000+ AI practitioners from 500+ companies on AI evaluation.

This isn't a course — we roll up our sleeves and work on your product with your data.You bring your real traces and interaction logs. Over 2 days, we systematically find where your AI is breaking down, why users are losing trust, and what to fix first.Most teams building AI products are stuck in the same loop: ship a change, hope it helps, manually spot-check a few outputs, repeat. Meanwhile, users hit failures nobody on the team has even seen.In this private workshop, we break that cycle. You bring real traces and interaction logs from your product. Together, we systematically uncover where your AI is breaking down, build a catalog of failure modes specific to your use case, and leave with a prioritized plan to fix what matters most.This methodology has been refined across 4,000+ practitioners from 500+ companies, and is part of what we teach in our <a target="_blank" rel="noopener noreferrer nofollow" href="https://maven.com/parlance-labs/evals">full-length course</a>. It's the highest-ROI activity in AI product development.Each workshop can accommodate a team of up to 12 participants. We can help you identify which team members to bring.Below is a sample agenda, this can be customized to your needs and schedule.

Walk away knowing exactly where your AI product is failing, why, and what to fix first — with a concrete plan your team can execute immediately.

Find hidden failure patterns in your AI product

Build a prioritized roadmap to fix what matters

Design metrics that prove your fixes work

This workshop is hands-on with your data. You need a deployed product generating real interactions we can analyze together.

A live AI product or feature with real users

We work directly with your production data. You'll receive a prep guide with instructions on what to collect and how to format it.

Access to recent traces or interaction logs

Engineers, PMs, and domain experts who know the product. The best results come from cross-functional teams who can act on findings immediately.

A dedicated team of 3+ participants

Engineering teams with a live AI product that has real users.

Product managers responsible for AI features who need a data-driven improvement plan instead of guessing which changes will move the needle.

Technical leaders who want to stop relying on manual QA and spot-checks, and instead build a systematic approach to improving AI quality.

Learn directly from Hamel Husain & Shreya Shankar

Hamel Husain & Shreya Shankar

Courses

AI Evals For Engineers & PMs

Mastering LLMs For Developers & Data Scientists

Workshops

Debug Your AI Product: Private Team Workshop

Lightning Lessons