Systematically Improving RAG Applications

4.7 (29)

·

6 Weeks

·

Cohort-based Course

Follow a repeatable process to continually evaluate and improve your RAG application

This course is popular

14 people enrolled last week.

Instructor Clients

Stitch Fix
Meta
Google

Course overview

Acquire the skills & confidently improving and iterate on RAG applications

Become the RAG Product Leader Your Company Needs


Companies are racing to integrate AI features that truly deliver value. To stay ahead, you must master the art of building Retrieval-Augmented Generation (RAG) products that continuously improve—no luck or guesswork required.


6 Weeks to Transform Your AI Product Sense


This course removes the mystery and turns RAG from a risky experiment into a structured, data-driven practice. You’ll learn how to pinpoint what’s working, diagnose what’s not, and steadily raise the bar on performance and user satisfaction.


What You’ll Learn, Week-by-Week


Week 1: Fundamentals & Synthetic Data

Establish a repeatable system for evaluating and improving RAG

Generate synthetic datasets for fast, meaningful experiments

Master precision, recall, and essential retrieval metrics

Lay the groundwork for a self-reinforcing improvement cycle


Week 2: Segmentation & Analysis

Break down complex query patterns and user behaviors

Spot inventory gaps and capability shortfalls quickly

Use clustering, classification, and dashboards to track progress

Set improvement priorities that deliver maximum impact


Week 3: Structured Data & Multimodality

Extract and leverage metadata for more accurate retrieval

Handle images, documents, and tables without losing quality

Create specialized indexes and blend them effectively

Apply earlier techniques to broaden RAG capabilities


Week 4: Query Routing & Tool Selection

Integrate multiple search methods into a single coherent product

Route queries intelligently using parallel function calls

Turn tool selection into a quantifiable, improvable task

Secure higher success probabilities end-to-end


Week 5: Representations & Fine-Tuning

Move beyond generic embeddings into fine-tuned, domain-specific vectors

Develop re-rankers that drastically improve retrieval quality

Discover how small amounts of targeted data can yield big gains

Shorten the experimentation cycle for continuous improvement


Week 6: Product UX & Feedback Loops

Design user interfaces that collect feedback effortlessly

Use streaming and visual hints to make latency feel negligible

Implement citations, follow-up actions, and chain-of-thought prompting

Add validators and checks to ensure reliability at scale


Why This Matters

RAG has become a core differentiator for AI-driven products. The strategies you learn here will never go out of style. As tools and models evolve, you’ll know how to integrate them, measure their impact, and refine your approach—over and over.


What’s Included

Proven evaluation frameworks that demystify “is it working?”

Techniques to handle more data types, faster and more accurately

Methods for continuous feedback collection and refinement

Insights that adapt as the AI landscape changes


About Your Instructor


Jason Liu is a seasoned AI consultant who has guided ai engineering at Stitch Fix, and high-growth startups from Seed stage to public companies.


He also advises many of today's generative AI startups. Jason specializes in transforming vague AI ambitions into concrete, measurable practices that deliver real business value.


Exclusive Bonuses ($1,500+ Value)

$500 Cohere credits

$200 LanceDB credits + free Lance Cloud access

$500 Modal Labs credits

6 months of Notion AI Plus

3 months of Braintrust access ($250 value)


Who Should Apply

Product teams integrating AI features into real software products

AI engineers moving from prototype to production-grade performance

Anyone committed to developing true “AI product sense”—not just hype


You Only Pay After Acceptance & Full Refund Guarantee

We’re so confident in the course’s value that if you don’t see meaningful improvements in your product’s performance within five weeks of starting, we’ll refund you fully.


Don’t let uncertainty stall your AI progress.


Join a small cohort of dedicated practitioners, learn the proven frameworks, and build the RAG systems your users will love.

This Course Is For You If You Are

01

A product leader, engineer, or data scientist looking to move beyond ad-hoc RAG prototypes into scalable, production-grade AI solutions.

02

A professional who understands LLM basics but wants a repeatable, data-driven methodology to improve retrieval relevance, latency, and user

03

Eager to create feedback loops that continuously refine and enhance the quality of RAG applications as models, data, and user needs evolve.

By the end of this course, participants will be able to

Adopt a Systematic, Data-First Methodology

Implement the Data and Evals Flywheel approach to continuously develop and improve RAG applications—breaking free from guesswork and relying on measurable, iterative enhancements.

Run Fast, Unit-Test-Like Evaluations

Quickly assess your retrieval systems using precision and recall metrics, identify bottlenecks, and confidently validate changes without sinking into endless trial-and-error cycles.

Leverage Synthetic Data for Rapid Iteration

Generate and utilize synthetic data sets to speed up experimentation, enabling you to test new approaches, embeddings, and architectures before committing full resources.

Master Fine-Tuning & Hard Negative Mining

Apply fine-tuning strategies for embedding models to boost search relevance and explore hard negative examples to further sharpen retrieval performance.

Classify Queries & Identify Bottlenecks

Use query classification and segmentation techniques to pinpoint exactly where your RAG system falls short—whether it’s due to limited inventory or insufficient capabilities.

Design Specialized Indices for Multiple Modalities

Create tailored indices for documents, images, tables, SQL databases, and more. Learn when and how to fuse or layer these indices to handle complex retrieval tasks elegantly.

Enhance Retrieval with Summarization & Chunking

Implement synthetic text chunk generation and strategic summarization methods to improve retrieval results, ensuring end-users get clear, concise, and contextually rich answers.

Implement Query Routing & Index Fusion

Develop systems that intelligently route queries to the right indices, tools, or pipelines. Blend and fuse indices effectively to handle nuanced, multi-step queries.

Optimize Both Global & Local Performance

Evaluate the performance of your routing logic and each individual index separately. Gain the nuance to fine-tune global system performance and local retrieval accuracy in tandem.


Integrate Feedback Loops for Continuous Improvement

Design explicit and implicit feedback mechanisms—capturing user signals, automating re-labeling, and applying improvements in real-time to keep your RAG systems on an upward trajectory.

This course includes

25 interactive live sessions

Lifetime access to course materials

54 in-depth lessons

Direct access to instructor

Projects to apply learnings

Guided feedback & reflection

Private community of peers

Course certificate upon completion

Maven Satisfaction Guarantee

This course is backed by Maven’s guarantee. You can receive a full refund within 14 days after the course ends, provided you meet the completion criteria in our refund policy.

Course syllabus

Week 1

Feb 4—Feb 9

    Fundamentals and Synthetic Data Generation

    • Feb

      4

      Office Hour

      Tue 2/42:00 PM—3:00 PM (UTC)
      Optional
    • Feb

      4

      Office Hour

      Tue 2/46:00 PM—7:00 PM (UTC)
      Optional
    • Feb

      5

      Building scalable RAG applications: Evolving workflows for sharing and optimization [Ankur Goyal]

      Wed 2/56:00 PM—7:00 PM (UTC)
    • Feb

      6

      Office Hour

      Thu 2/67:00 PM—8:00 PM (UTC)
      Optional

Week 2

Feb 10—Feb 16

    Segmentation and Analysis

    • Feb

      11

      Office Hour

      Tue 2/112:00 PM—3:00 PM (UTC)
      Optional
    • Feb

      11

      Office Hour

      Tue 2/116:00 PM—7:00 PM (UTC)
      Optional
    • Feb

      12

      Building Houses, Digging Holes, and Searching for Gold [Mukul Surajiwale]

      Wed 2/126:00 PM—7:00 PM (UTC)
    • Feb

      13

      Office Hour

      Thu 2/137:00 PM—8:00 PM (UTC)
      Optional

Week 3

Feb 17—Feb 23

    Structured Extraction and Multimodality

    • Feb

      18

      Office Hour

      Tue 2/182:00 PM—3:00 PM (UTC)
      Optional
    • Feb

      18

      Office Hour

      Tue 2/186:00 PM—7:00 PM (UTC)
      Optional
    • Feb

      19

      Building Document Workflow Agents [Jerry Liu]

      Wed 2/196:00 PM—7:00 PM (UTC)
    • Feb

      20

      Office Hour

      Thu 2/207:00 PM—8:00 PM (UTC)
      Optional

Week 4

Feb 24—Mar 2

    Query Routing and Tool Selection

    • Feb

      25

      Office Hour

      Tue 2/252:00 PM—3:00 PM (UTC)
      Optional
    • Feb

      25

      Office Hour

      Tue 2/256:00 PM—7:00 PM (UTC)
      Optional
    • Feb

      26

      Organizing Your Data for Query Routing [Anton Troynikov]

      Wed 2/266:00 PM—7:00 PM (UTC)
    • Feb

      27

      Office Hour

      Thu 2/277:00 PM—8:00 PM (UTC)
      Optional

Week 5

Mar 3—Mar 9

    Representations and Fine-tuning

    • Mar

      4

      Office Hour

      Tue 3/42:00 PM—3:00 PM (UTC)
      Optional
    • Mar

      4

      Office Hour

      Tue 3/46:00 PM—7:00 PM (UTC)
      Optional
    • Mar

      5

      Guest Speaker

      Wed 3/56:00 PM—7:00 PM (UTC)
    • Mar

      6

      Office Hour

      Thu 3/67:00 PM—8:00 PM (UTC)
      Optional

Week 6

Mar 10—Mar 13

    Product Design and User Experience

    • Mar

      11

      Office Hour

      Tue 3/111:00 PM—2:00 PM (UTC)
      Optional
    • Mar

      11

      Office Hour

      Tue 3/115:00 PM—6:00 PM (UTC)
      Optional
    • Mar

      12

      Vantager: RAG for Financial Reports [Nicolas Neven]

      Wed 3/125:00 PM—6:00 PM (UTC)
    • Mar

      13

      Office Hour

      Thu 3/136:00 PM—7:00 PM (UTC)
      Optional

Post-course

    Mar

    19

    Lexical Love: Rediscovering the Power of Text in RAG [John Berryman]

    Wed 3/195:00 PM—6:00 PM (UTC)

    Product Design

    5 items

    Rejecting work

    3 items

    Intro To The Playbook

    2 items

    RAG Evaluation

    4 items

    Synthetic Data

    1 item

    Identifying Areas of Improvement

    4 items

    Production Monitoring and Analysis

    0 items

    Improving Retrieval

    4 items

    Tables and Non-Text Data

    2 items

    Routing Queries

    4 items

    Representations

    0 items

    Synthetic Text Chunks

    3 items

Bonus

    Cohort 1 Workshops

    6 items

    Cohort 1 Guest Lectures

    7 items

    Cohort 1 Office Hours

    9 items

4.7 (29 ratings)

What students are saying

Meet your instructor

Jason Liu

Jason Liu

Jason has built search and recommendation systems for the past 6 years. He has consulted and advised a dozens startups in the last year to improve their RAG systems. He is the creator of the Instructor Python library.

A pattern of wavy dots

Join an upcoming cohort

Systematically Improving RAG Applications

Cohort 2

$1,650

Dates

Feb 4—Mar 13, 2025

Application Deadline

Feb 4, 2025

Course Schedule Each Week

  • Tuesday: Workshops

    1:00 - 2:00PM ET

    Workshops covering each step of the playbook and helping you build process improvements in your RAG application

  • Wednesday: Office Hours + Breakout Sessions

    1:00 - 2:00PM ET

    The first half hour will be interactive breakout sessions, and the closing half-hour each week is Q&A

  • Thursday: Guest Speakers

    1:00 - 2:00PM ET

    Guest instructors covering key topics in both innovative theory and practical applications in RAG system development.

Learning is better with cohorts

Learning is better with cohorts

Active hands-on learning

This course builds on live workshops and hands-on projects

Interactive and project-based

You’ll be interacting with other learners through breakout rooms and project teams

Learn with a cohort of peers

Join a community of like-minded people who want to learn and grow alongside you

Frequently Asked Questions

Stay in the loop

Sign up to be the first to know about course updates.

A pattern of wavy dots

Join an upcoming cohort

Systematically Improving RAG Applications

Cohort 2

$1,650

Dates

Feb 4—Mar 13, 2025

Application Deadline

Feb 4, 2025

$1,650

4.7 (29)

·

6 Weeks