Systematically Improving RAG Applications

4.8 (51)

·

6 Weeks

·

Cohort-based Course

Follow a repeatable process to continually evaluate and improve your RAG application

Instructor Clients

Stitch Fix
Meta
Google

Course overview

Stop building RAG systems that impress in demos but disappoint in production

Transform your retrieval from "good enough" to "mission-critical" in weeks, not months


Most RAG implementations get stuck in prototype purgatory. They work well for simple cases but fail on complex queries—leading to frustrated users, lost trust, and wasted engineering time. The difference between a prototype and a production-ready system isn't just better technology, it's a fundamentally different mindset.


The RAG Implementation Reality

What you're experiencing right now:


❌ Your RAG demo impressed stakeholders, but real users encounter hallucinations when they need accuracy most

❌ Engineers spend countless hours tweaking prompts with minimal improvement

❌ Colleagues report finding information manually that your system failed to retrieve

❌ You keep making changes but have no way to measure if they're actually helping

❌ Every improvement feels like guesswork instead of systematic progress

❌ You're unsure which 10% of possible enhancements will deliver 90% of the value


What your RAG system could be:

With the RAG Flywheel methodology, you'll build a system that:


✅ Retrieves the right information even for complex, ambiguous queries

✅ Continuously improves with each user interaction

✅ Provides clear metrics to demonstrate ROI to stakeholders

✅ Allows your team to make data-driven decisions about improvements

✅ Adapts to different content types with specialized capabilities

✅ Creates value that compounds over time instead of degrading


What Makes This Course Different

Unlike courses that focus solely on technical implementation, this program gives you the systematic, data-driven approach used by companies to transform prototypes into production systems that deliver real business value:


The Improvement Flywheel: Build synthetic evaluation data that identifies exactly what's failing in your system—even before you have users

Fine-tuning Framework: Create custom embedding models with minimal data (as few as 6,000 examples)

Feedback Acceleration: Design interfaces that collect 5x more high-quality feedback without annoying users

Segmentation System: Analyze user queries to identify which segments need specialized retrievers for 20-40% accuracy gains

Multimodal Architecture: Implement specialized indices for different content types (documents, images, tables)

Query Routing: Create a unified system that intelligently selects the right retriever for each query


The Complete RAG Implementation Framework


Week 1: Evaluation Systems

Build synthetic datasets that pinpoint RAG failures instead of relying on subjective assessments


BEFORE: "We need to make the AI better, but we don't know where to start."

AFTER: "We know exactly which query types are failing and by how much."


Week 2: Fine-tune Embeddings

Customize models for 20-40% accuracy gains with minimal examples


BEFORE: "Generic embeddings don't understand our domain terminology."

AFTER: "Our embedding models understand exactly what 'similar' means in our business context."


Week 3: Feedback Systems

Design interfaces that collect 5x more feedback without annoying users


BEFORE: "Users get frustrated waiting for responses and rarely tell us what's wrong."

AFTER: "Every interaction provides signals that strengthen our system."


Week 4: Query Segmentation

Identify high-impact improvements and prioritize engineering resources


BEFORE: "We don't know which features would deliver the most value."

AFTER: "We have a clear roadmap based on actual usage patterns and economic impact."


Week 5: Specialized Search

Build specialized indices for different content types that improve retrieval


BEFORE: "Our system struggles with anything beyond basic text documents."

AFTER: "We can retrieve information from tables, images, and complex documents with high precision."


Week 6: Query Routing

Implement intelligent routing that selects optimal retrievers automatically


BEFORE: "Different content requires different interfaces, creating a fragmented experience."

AFTER: "Users have a seamless experience while the system intelligently routes to specialized components."


Real-world Impact From Implementation



85% blueprint image recall: Construction company using visual LLM captioning

90% research report retrieval: Through better text preprocessing techniques

$50M revenue increase: Retail company enhancing product search with embedding fine-tuning

+14% accuracy boost: Fine-tuning cross-encoders with minimal examples

+20% response accuracy: Using re-ranking techniques

-30% irrelevant documents: Through improved query segmentation


Join 400+ engineers who've transformed their RAG systems with this methodology



Your Instructor


Jason Liu has built AI systems across diverse domains—from computer vision at the University of Waterloo to content policy at Facebook to recommendation systems at Stitch Fix that boosted revenue by $50 million. His background in managing large-scale data curation, designing multimodal retrieval models, and processing hundreds of millions of recommendations weekly has directly informed his consulting work with companies implementing RAG systems.

This Course Is For You If You Are

01

A product leader, engineer, or data scientist looking to move beyond ad-hoc RAG prototypes into scalable, production-grade AI solutions.

02

A professional who understands LLM basics but wants a repeatable, data-driven methodology to improve retrieval relevance, latency, and user

03

Eager to create feedback loops that continuously refine and enhance the quality of RAG applications as models, data, and user needs evolve.

By the end of this course, participants will be able to

Adopt a Systematic, Data-First Methodology

Implement the Data and Evals Flywheel approach to continuously develop and improve RAG applications—breaking free from guesswork and relying on measurable, iterative enhancements.

Run Fast, Unit-Test-Like Evaluations

Quickly assess your retrieval systems using precision and recall metrics, identify bottlenecks, and confidently validate changes without sinking into endless trial-and-error cycles.

Leverage Synthetic Data for Rapid Iteration

Generate and utilize synthetic data sets to speed up experimentation, enabling you to test new approaches, embeddings, and architectures before committing full resources.

Master Fine-Tuning & Hard Negative Mining

Apply fine-tuning strategies for embedding models to boost search relevance and explore hard negative examples to further sharpen retrieval performance.

Classify Queries & Identify Bottlenecks

Use query classification and segmentation techniques to pinpoint exactly where your RAG system falls short—whether it’s due to limited inventory or insufficient capabilities.

Design Specialized Indices for Multiple Modalities

Create tailored indices for documents, images, tables, SQL databases, and more. Learn when and how to fuse or layer these indices to handle complex retrieval tasks elegantly.

Enhance Retrieval with Summarization & Chunking

Implement synthetic text chunk generation and strategic summarization methods to improve retrieval results, ensuring end-users get clear, concise, and contextually rich answers.

Implement Query Routing & Index Fusion

Develop systems that intelligently route queries to the right indices, tools, or pipelines. Blend and fuse indices effectively to handle nuanced, multi-step queries.

Optimize Both Global & Local Performance

Evaluate the performance of your routing logic and each individual index separately. Gain the nuance to fine-tune global system performance and local retrieval accuracy in tandem.


Integrate Feedback Loops for Continuous Improvement

Design explicit and implicit feedback mechanisms—capturing user signals, automating re-labeling, and applying improvements in real-time to keep your RAG systems on an upward trajectory.

This course includes

26 interactive live sessions

Lifetime access to course materials

25 in-depth lessons

Direct access to instructor

Projects to apply learnings

Guided feedback & reflection

Private community of peers

Course certificate upon completion

Maven Satisfaction Guarantee

This course is backed by Maven’s guarantee. You can receive a full refund within 14 days after the course ends, provided you meet the completion criteria in our refund policy.

Course syllabus

Week 1

May 20—May 25

    Week 1

    • May

      20

      Introductions

      Tue 5/203:00 PM—4:00 PM (UTC)
    • May

      21

      Generative Benchmarking [Kelly Hong]

      Wed 5/215:00 PM—6:00 PM (UTC)
    • May

      22

      Optional: Watch Lecture

      Thu 5/2212:00 AM—1:00 AM (UTC)
      Optional

    Office Hours

    • May

      20

      Office Hour

      Tue 5/201:00 PM—2:00 PM (UTC)
      Optional
    • May

      22

      Office Hour

      Thu 5/226:00 PM—7:00 PM (UTC)
      Optional

Week 2

May 26—Jun 1

    Week 2

    • May

      28

      Guest Lecture [Coming Soon]

      Wed 5/285:00 PM—6:00 PM (UTC)
    • May

      29

      Optional: Watch Lecture

      Thu 5/2912:00 AM—1:00 AM (UTC)
      Optional

    Office Hours

    • May

      27

      Office Hour

      Tue 5/271:00 PM—2:00 PM (UTC)
      Optional
    • May

      29

      Office Hour

      Thu 5/296:00 PM—7:00 PM (UTC)
      Optional

Week 3

Jun 2—Jun 8

    Week 3

    • Jun

      4

      Guest Lecture [Coming Soon]

      Wed 6/45:00 PM—6:00 PM (UTC)
    • Jun

      5

      Optional: Watch Lecture

      Thu 6/512:00 AM—1:00 AM (UTC)
      Optional

    Office Hours

    • Jun

      3

      Office Hour

      Tue 6/31:00 PM—2:00 PM (UTC)
      Optional
    • Jun

      5

      Office Hour

      Thu 6/56:00 PM—7:00 PM (UTC)
      Optional

Week 4

Jun 9—Jun 15

    Week 4

    • Jun

      11

      Guest Lecture [Coming Soon]

      Wed 6/115:00 PM—6:00 PM (UTC)
    • Jun

      12

      Watch Lecture

      Thu 6/1212:00 AM—1:00 AM (UTC)

    Office Hours

    • Jun

      10

      Office Hour

      Tue 6/101:00 PM—2:00 PM (UTC)
      Optional
    • Jun

      12

      Office Hour

      Thu 6/126:00 PM—7:00 PM (UTC)
      Optional

Week 5

Jun 16—Jun 22

    Week 5

    • Jun

      18

      Guest Lecture [Coming Soon!]

      Wed 6/185:00 PM—6:00 PM (UTC)
    • Jun

      19

      Watch Lecture

      Thu 6/1912:00 AM—1:00 AM (UTC)

    Office Hours

    • Jun

      17

      Office Hour

      Tue 6/171:00 PM—2:00 PM (UTC)
    • Jun

      19

      Office Hour

      Thu 6/196:00 PM—7:00 PM (UTC)

Week 6

Jun 23—Jun 26

    Week 6

    • Jun

      25

      Guest Lecture [Coming Soon!]

      Wed 6/255:00 PM—6:00 PM (UTC)
    • Jun

      26

      Conclusions

      Thu 6/268:00 PM—9:00 PM (UTC)
    • Jun

      26

      Watch Lecture

      Thu 6/2612:00 AM—1:00 AM (UTC)

    Office Hours

    • Jun

      24

      Office Hour

      Tue 6/241:00 PM—2:00 PM (UTC)
    • Jun

      26

      Office Hour

      Thu 6/266:00 PM—7:00 PM (UTC)

Bonus

4.8 (51 ratings)

What students are saying

What people are saying

        As an Applied AI Engineer at Anthropic, I was familiar with all of the standard retrieval methods and RAG papers going into the course, but Jason's frameworks helped me to operationalize what I'd learned and it's had an incredibly positive impact in my work with customers.
Sam Flamini

Sam Flamini

Solutions Engineer at Anthropic
        Evals really moving us forward again and past the "vibe check" plateau. First iteration alone has highlighted multiple non-obvious failure modes of the system. In combination with customer feedback / bug reports / traces. So satisfying to have good visibility again into where we can get some easy wins.
Nico Neven

Nico Neven

CTO at Vantager

Meet your instructor

Jason Liu

Jason Liu

Jason has built search and recommendation systems for the past 6 years. He has consulted and advised a dozens startups in the last year to improve their RAG systems. He is the creator of the Instructor Python library.

A pattern of wavy dots

Join an upcoming cohort

Systematically Improving RAG Applications

Cohort 3

$1,800

Dates

May 20—June 26, 2025

Payment Deadline

May 21, 2025
Get reimbursed

Learning is better with cohorts

Learning is better with cohorts

Active hands-on learning

This course builds on live workshops and hands-on projects

Interactive and project-based

You’ll be interacting with other learners through breakout rooms and project teams

Learn with a cohort of peers

Join a community of like-minded people who want to learn and grow alongside you

Frequently Asked Questions

Stay in the loop

Sign up to be the first to know about course updates.

A pattern of wavy dots

Join an upcoming cohort

Systematically Improving RAG Applications

Cohort 3

$1,800

Dates

May 20—June 26, 2025

Payment Deadline

May 21, 2025
Get reimbursed

$1,800

4.8 (51)

·

6 Weeks