Systematically Improving RAG Applications

Jason Liu

Staff machine learning engineer, currently working as an AI consultant

Instructor

Stop building RAG systems that impress in demos but disappoint in production

Transform your retrieval from “good enough” to “mission-critical” in weeks, not months. Most RAG systems stall in prototype purgatory: they demo well, but fail on complex queries—eroding trust and wasting engineering time. The difference isn’t just better tech, but a systematic mindset.

With the RAG Flywheel, you’ll:
✅ Pinpoint failures with synthetic evals
✅ Fine-tune embeddings for 20–40% gains
✅ Collect 5x more user feedback
✅ Segment queries to target high-impact fixes
✅ Build multimodal indices for docs, tables, images
✅ Route queries to the best retriever automatically

Week by week, you move from vague “make it better” to clear metrics, focused improvements, and compounding value. Real-world results include +20% accuracy from re-ranking, +14% with cross-encoders, and $50M revenue boosts from better search.

Join 400+ engineers applying this framework in production. Instructor Jason Liu has built multimodal retrieval and recommendation systems at Facebook, Stitch Fix, and through consulting—experience that shaped this practical, battle-tested approach.

What you’ll learn

Follow a repeatable process to continually evaluate and improve your RAG application

  • Evaluate retrieval quality using precision, recall, and MRR metrics to identify system weaknesses

  • Differentiate between leading metrics (experiments run) and lagging metrics (customer satisfaction) to drive actionable improvements

  • Design synthetic data generation pipelines that enable rapid experimentation without waiting for user data

  • Create comprehensive evaluation datasets using LLMs to generate realistic query-answer pairs

  • Establish baselines using tools like LanceDB to benchmark different retrieval implementations

  • Develop multimodal retrieval systems that handle documents, images, tables, and structured data

  • Synthesize lexical (BM25), semantic (embeddings), and metadata-based search for optimal results

  • Extract structured information from diverse data sources to enable precise filtering

  • Classify queries using domain expertise and few-shot classifiers to improve routing accuracy

Learn directly from Jason

Jason Liu

Jason Liu

Staff machine learning engineer, currently working as an AI consultant

Students from

OpenAI
Anthropic
Microsoft
Google
Meta

Who this course is for

  • A product leader, engineer, or data scientist looking to move beyond ad-hoc RAG prototypes into scalable, production-grade AI solutions.

  • A professional who understands LLM basics but wants a repeatable, data-driven methodology to improve retrieval relevance, latency, and user

  • Eager to create feedback loops that continuously refine and enhance the quality of RAG applications as models, data, and user needs evolve.

Prerequisites

  • Deployed a RAG System

    The goal of this course is not just to share with you a how-to guide, but rather how to systematically improve these architectures.

  • Optional (Python)

    We have over 20 iPython notebooks that you can explore, run code to be more hands-on with the experiments that we plan to run.

What's included

Jason Liu

Live sessions

Learn directly from Jason Liu in a real-time, interactive format.

6 Prerecorded Lectures

Short, focused videos that unpack the full RAG-improvement framework that you can rewatch anytime.

6+ Office Hour Q&As

Open office hours for deep dives, debugging help, and personalized feedback.

12 Hands-On Python Notebooks

Ready-to-run notebooks & walkthrough videos so you can practice every concept instantly.

Lifetime Slack Community

Private Slack for peer reviews, job leads, and ongoing support forever.

Expert Speaker Library

Curated talks from builders running large-scale RAG systems in production.

$2K+ in Cloud & AI Credits

Test vector DBs, LLM APIs, and infra with over $2,000 in partner credits.

Free Future Re-Enrollment

Join any future cohort at no cost and get updated content and live coaching again whenever you need it.

Certificate of completion

Showcase your advanced RAG skills to clients, employers, and your LinkedIn network.

Maven Guarantee

This course is backed by the Maven Guarantee. Students are eligible for a full refund up until the halfway point of the course.

Syllabus

10 live sessions • 41 lessons

Week 1

Nov 17—Nov 23

    Lectures & Tutorials

    5 items

    Office Hours

    • Nov

      19

      Introductions

      Wed 11/194:00 PM—5:00 PM (UTC)
    • Nov

      19

      Office Hour

      Wed 11/196:00 PM—7:00 PM (UTC)
    • Nov

      20

      Optional: Watch Lectures

      Thu 11/201:00 AM—2:00 AM (UTC)
      Optional
    • Nov

      20

      Office Hour

      Thu 11/207:00 PM—8:00 PM (UTC)

Week 2

Nov 24—Nov 30

    Lectures & Tutorials

    3 items

    Office Hours

    • Nov

      24

      Office Hour

      Mon 11/245:00 PM—6:00 PM (UTC)
    • Nov

      26

      Optional: Watch Lectures

      Wed 11/261:00 AM—2:00 AM (UTC)
      Optional
    • Nov

      26

      Office Hour

      Wed 11/267:00 PM—8:00 PM (UTC)

Schedule

Office hours: 1 hour per week

Pre-recorded lectures : 1 hour per week

Optional guest lectures. : 1-2 hours per week

Core sessions

    • Nov 19
      Wed
      4:00 PM—5:00 PM (UTC)
    • Nov 19
      Wed
      6:00 PM—7:00 PM (UTC)
    • Nov 20
      Thu
      7:00 PM—8:00 PM (UTC)
    • Nov 24
      Mon
      5:00 PM—6:00 PM (UTC)
    • Nov 26
      Wed
      7:00 PM—8:00 PM (UTC)
    • Dec 2
      Tue
      5:00 PM—6:00 PM (UTC)
    • Dec 4
      Thu
      7:00 PM—8:00 PM (UTC)

Optional sessions

    • Nov 20
      Thu
      1:00 AM—2:00 AM (UTC)
    • Nov 26
      Wed
      1:00 AM—2:00 AM (UTC)
    • Dec 4
      Thu
      1:00 AM—2:00 AM (UTC)

Success stories

  • As an Applied AI Engineer at Anthropic, I was familiar with all of the standard retrieval methods and RAG papers going into the course, but Jason's frameworks helped me to operationalize what I'd learned and it's had an incredibly positive impact in my work with customers.
    Testimonial author image

    Sam Flamini

    Solutions Engineer at Anthropic
  • Evals really moving us forward again and past the "vibe check" plateau. First iteration alone has highlighted multiple non-obvious failure modes of the system. In combination with customer feedback / bug reports / traces. So satisfying to have good visibility again into where we can get some easy wins.
    Testimonial author image

    Nico Neven

    CTO at Vantager

Frequently asked questions

$2,000

USD

·
Nov 17Dec 5