Systematically Improving RAG Applications

Jason Liu

Staff machine learning engineer, currently working as an AI consultant

Instructor

This course is popular

7 people enrolled last week.

Stop building RAG systems that impress in demos but disappoint in production

Transform your retrieval from “good enough” to “mission-critical” in weeks, not months. Most RAG systems stall in prototype purgatory: they demo well, but fail on complex queries—eroding trust and wasting engineering time. The difference isn’t just better tech, but a systematic mindset.

With the RAG Flywheel, you’ll:
✅ Pinpoint failures with synthetic evals
✅ Fine-tune embeddings for 20–40% gains
✅ Collect 5x more user feedback
✅ Segment queries to target high-impact fixes
✅ Build multimodal indices for docs, tables, images
✅ Route queries to the best retriever automatically

Week by week, you move from vague “make it better” to clear metrics, focused improvements, and compounding value. Real-world results include +20% accuracy from re-ranking, +14% with cross-encoders, and $50M revenue boosts from better search.

Join 400+ engineers applying this framework in production. Instructor Jason Liu has built multimodal retrieval and recommendation systems at Facebook, Stitch Fix, and through consulting—experience that shaped this practical, battle-tested approach.

What you’ll learn

Follow a repeatable process to continually evaluate and improve your RAG application

  • Evaluate retrieval quality using precision, recall, and MRR metrics to identify system weaknesses

  • Differentiate between leading metrics (experiments run) and lagging metrics (customer satisfaction) to drive actionable improvements

  • Design synthetic data generation pipelines that enable rapid experimentation without waiting for user data

  • Create comprehensive evaluation datasets using LLMs to generate realistic query-answer pairs

  • Establish baselines using tools like LanceDB to benchmark different retrieval implementations

  • Develop multimodal retrieval systems that handle documents, images, tables, and structured data

  • Synthesize lexical (BM25), semantic (embeddings), and metadata-based search for optimal results

  • Extract structured information from diverse data sources to enable precise filtering

  • Classify queries using domain expertise and few-shot classifiers to improve routing accuracy

Learn directly from Jason

Jason Liu

Jason Liu

Staff machine learning engineer, currently working as an AI consultant

Students from

OpenAI
Anthropic
Microsoft
Google
Meta

Who this course is for

  • A product leader, engineer, or data scientist looking to move beyond ad-hoc RAG prototypes into scalable, production-grade AI solutions.

  • A professional who understands LLM basics but wants a repeatable, data-driven methodology to improve retrieval relevance, latency, and user

  • Eager to create feedback loops that continuously refine and enhance the quality of RAG applications as models, data, and user needs evolve.

Prerequisites

  • Deployed a RAG System

    The goal of this course is not just to share with you a how-to guide, but rather how to systematically improve these architectures.

  • Optional (Python)

    We have over 20 iPython notebooks that you can explore, run code to be more hands-on with the experiments that we plan to run.

What's included

Jason Liu

Live sessions

Learn directly from Jason Liu in a real-time, interactive format.

6 Prerecorded Lectures

Short, focused videos that unpack the full RAG-improvement framework that you can rewatch anytime.

6+ Office Hour Q&As

Open office hours for deep dives, debugging help, and personalized feedback.

12 Hands-On Python Notebooks

Ready-to-run notebooks & walkthrough videos so you can practice every concept instantly.

Lifetime Slack Community

Private Slack for peer reviews, job leads, and ongoing support forever.

Expert Speaker Library

Curated talks from builders running large-scale RAG systems in production.

$2K+ in Cloud & AI Credits

Test vector DBs, LLM APIs, and infra with over $2,000 in partner credits.

Free Future Re-Enrollment

Join any future cohort at no cost and get updated content and live coaching again whenever you need it.

Certificate of completion

Showcase your advanced RAG skills to clients, employers, and your LinkedIn network.

Maven Guarantee

This course is backed by the Maven Guarantee. Students are eligible for a full refund up until the halfway point of the course.

Syllabus

23 live sessions • 41 lessons

Week 1

Sep 16—Sep 21

    Lectures & Tutorials

    3 items

    Office Hours

    • Sep

      16

      Introductions

      Tue 9/163:00 PM—4:00 PM (UTC)
    • Sep

      16

      Office Hour

      Tue 9/165:00 PM—6:00 PM (UTC)
    • Sep

      18

      Optional: Watch Lecture

      Thu 9/1812:00 AM—1:00 AM (UTC)
      Optional
    • Sep

      18

      Office Hour

      Thu 9/186:00 PM—7:00 PM (UTC)

    Guest Speakers

    • Sep

      17

      Context Rot: How Input Length Impacts LLM Performance [Kelly Hong]

      Wed 9/175:00 PM—6:00 PM (UTC)

Week 2

Sep 22—Sep 28

    Lectures & Tutorials

    2 items

    Office Hours

    • Sep

      23

      Office Hour

      Tue 9/231:00 PM—2:00 PM (UTC)
    • Sep

      25

      Optional: Watch Lecture

      Thu 9/2512:00 AM—1:00 AM (UTC)
      Optional
    • Sep

      25

      Office Hour

      Thu 9/256:00 PM—7:00 PM (UTC)

    Guest Speakers

    • Sep

      24

      Cheating at Query Understanding with LLMs [Doug Turnbull]

      Wed 9/246:00 PM—7:00 PM (UTC)

Schedule

Office hours: 1 hour per week

Pre-recorded lectures : 1 hour per week

Optional guest lectures. : 1-2 hours per week

Core sessions

    • Sep 16
      Tue
      3:00 PM—4:00 PM (UTC)
    • Sep 16
      Tue
      5:00 PM—6:00 PM (UTC)
    • Sep 17
      Wed
      5:00 PM—6:00 PM (UTC)
    • Sep 18
      Thu
      6:00 PM—7:00 PM (UTC)
    • Sep 23
      Tue
      1:00 PM—2:00 PM (UTC)
    • Sep 24
      Wed
      6:00 PM—7:00 PM (UTC)

Optional sessions

    • Sep 18
      Thu
      12:00 AM—1:00 AM (UTC)
    • Sep 25
      Thu
      12:00 AM—1:00 AM (UTC)
    • Oct 2
      Thu
      12:00 AM—1:00 AM (UTC)
    • Oct 9
      Thu
      12:00 AM—1:00 AM (UTC)
    • Oct 16
      Thu
      12:00 AM—1:00 AM (UTC)
    • Oct 23
      Thu
      12:00 AM—1:00 AM (UTC)

Success stories

  • As an Applied AI Engineer at Anthropic, I was familiar with all of the standard retrieval methods and RAG papers going into the course, but Jason's frameworks helped me to operationalize what I'd learned and it's had an incredibly positive impact in my work with customers.
    Testimonial author image

    Sam Flamini

    Solutions Engineer at Anthropic
  • Evals really moving us forward again and past the "vibe check" plateau. First iteration alone has highlighted multiple non-obvious failure modes of the system. In combination with customer feedback / bug reports / traces. So satisfying to have good visibility again into where we can get some easy wins.
    Testimonial author image

    Nico Neven

    CTO at Vantager

Frequently asked questions

$1,800

USD

·

2 days left to enroll