Systematically Improving RAG Applications

4.8 (30)

·

6 Weeks

·

Cohort-based Course

Follow a repeatable process to continually evaluate and improve your RAG application

Instructor Clients

Stitch Fix
Meta
Google

Course overview

Acquire the skills & confidently improving and iterate on RAG applications

Become the RAG Product Thinker Your Company Needs


In the rapidly evolving world of AI, companies are in a fierce arms race. They're scrambling to find developers and product leaders who can successfully incorporate AI into their products before competitors do it better.


Are you ready to be that leader?


The Challenge: Navigating the RAG Black Box


You're working on AI-powered applications, but:


* Time and resources are limited

* The best path forward isn't clear

* The inner workings of your product are shrouded in uncertainty


The Solution: Develop the "Product Sense" for AI


In just 6 weeks, learn to:


* Optimize search quality and latency

* Design robust feedback loops for continuous improvement

* Implement data-driven strategies for maximum impact

* Navigate AI product decisions in uncertain environments



Course Breakdown: From Fundamentals to Advanced Techniques


Week 1: Fundamentals and Synthetic Data Generation

* Understand the importance of having a system for evaluating and improving RAG

* Learn to generate synthetic data for fast evaluations

* Master precision and recall metrics for retrieval evaluation

* Distinguish between leading and lagging metrics

* Implement the RAG System Inference Flywheel


Week 2: Segmentation and Analysis

Learn the importance of segmenting queries and users

Differentiate between inventory issues and capability issues

Use clustering and classification to identify query types and segments

Set up dashboards to monitor query distributions and performance over time

Prioritize improvements based on impact, volume, and likelihood of success


Week 3: Structured Extraction and Multimodality

Learn techniques for handling different data types: documents, images, tables

Extract metadata and generate synthetic data/summaries to improve search

Implement approaches for document search, image search, and querying tables

Apply Week 1-2 techniques to evaluate and improve individual indices

Balance trade-offs between generalized and specialized approaches in multimodal RAG


Week 4: Query Routing and Tool Selection

Combine multiple search indices into a cohesive application

Implement parallel function calling and API gateways to route queries

Evaluate tool selection as a classification task

Avoid data leakage when generating examples

Break down probability of success into tool selection and retrieval components


Week 5: Representations and Fine-tuning

Understand limitations of pre-trained embedding models

Learn the importance of fine-tuning embeddings and re-rankers on domain-specific data

Collect relevance data to create triplet training examples

Explore benefits of fine-tuning a single model across multiple tasks

Discover how relatively small amounts of data can lead to significant improvements


Week 6: Product Design and User Experience

Implement techniques for collecting user feedback

Design streaming strategies to improve perceived latency

Create UI components for rendering citations and follow-up actions

Apply prompting techniques like chain-of-thought and monologues

Add validators to improve reliability and response quality

Develop strategies for handling negative examples


Why This Course, Why Now?

RAG is becoming essential for competitive AI integration

Focus is shifting from basic implementation to performance optimization

Core principles of effective RAG systems are crystallizing

Early adopters gain significant market advantages


What You'll Learn

Cold start your evaluation pipeline for retrieval

Understand the limitations of embedding models and how to think about rerankers and fine-tuning

Master retrieval metrics and use them to quickly run experiments

Identify high-impact tasks and prioritize effectively

Make informed tradeoffs and choose relevant metrics

Focus on what matters most in AI product development


Why Your Team Needs This

Align on RAG best practices

Save months of trial and error

Build scalable systems that prevent future rewrites

Foster a data-driven culture of continuous improvement

Bridge gaps between technical and business teams

Gain competitive edge in AI implementation

Justify AI investments to leadership

Learn from real-world case studies across industries


About Your Instructors

Jason Liu is a seasoned machine learning consultant with experience at Stitchfix, and Meta, Stitch Fix, and dozens more companies, ranging from startups to Limitless to Larger enterprises like Hubspot and Zzapier. When companies struggle to make progress, they hire Jason help their AI teams find "the path" forward.


Jason Liu is a machine learning engineer and data scientist with 8 years of experience in building recommendation systems and multi-modal semantic search products at Stitchfix


What You'll Get

A community of other operators and AI product thinkers

Hands-on experience with real-world RAG applications

Strategies for continuous system improvement

Skills applicable to all AI initiatives, not just RAG

Access to instructors' real-world experience and immediately applicable insights

Bonus Offers (Over $1500 in Value!)

$500 Cohere credits

$200 LanceDB credits and free access to Lance Cloud

$500 in Modal Labs credits

6 months free Notion AI Plus

3 months Braintrust access ($250 value)


Who Should Apply?

This course is perfect for you if:

You're working on software products incorporating AI

You've built a RAG prototype and want to move to production

You're looking to develop a "product sense" for AI applications


Won't a lot change by February?

Course focuses on enduring principles, not just current tools

You'll learn to evaluate and integrate new technologies rapidly

Strategies taught focus on ongoing system optimization

By February, you'll be positioned to leverage new developments immediately

Course content will be updated to reflect any significant changes

Skills developed are foundational and will remain relevant


Don't Miss Out!

🚀 Limited Spots Available 🚀


Our small-group cohorts fill up fast. Don't wait to level up your RAG skills.



Remember, you only pay after your application is approved. We're so confident in the value of this course that we offer a full refund if you don't see meaningful improvements in your processes within 5 weeks.


Join a small cohort of other teams shipping real applications and take your RAG skills to the next level. Don't do it alone - be part of a community of professionals facing similar challenges and working towards AI excellence.

This Course Is For You If You Are

01

An Engineering or Product leader looking to improve an existing RAG system MVP

02

Solving problems like poor retrieval, unreliable outputs or unhappy customers with your existing application

03

Ready to lead your team in building a data flywheel so you can leverage feedback

By the end of this course, participants will be able to

Implement a systematic approach to developing and improving RAG applications using the Data and Evals Flywheel methodology.


Design and execute fast, unit test-like evaluations to assess retrieval capabilities, including precision and recall metrics.


Generate and utilize synthetic data for rapid evaluation and iteration of RAG systems.


Apply fine-tuning strategies for embedding models and implement hard negative mining techniques to enhance search relevance.


Classify different types of queries and conduct bottleneck analysis to identify performance limitations in RAG systems.

Differentiate between limited inventory and limited capabilities issues, and develop strategies to address both.


Design and implement specialized indices for various data types, including documents, images, tables, and SQL databases.

Apply synthetic text chunk generation and summarization techniques to improve retrieval performance across different modalities.

Develop efficient query routing systems and implement effective index fusion strategies for complex RAG setups.

Evaluate the performance of both query routing and individual indices separately to optimize overall system performance.

Design and integrate both explicit and implicit feedback mechanisms to drive continuous system improvement.

This course includes

24 interactive live sessions

Lifetime access to course materials

32 in-depth lessons

Direct access to instructor

Projects to apply learnings

Guided feedback & reflection

Private community of peers

Course certificate upon completion

Maven Satisfaction Guarantee

This course is backed by Maven’s guarantee. You can receive a full refund within 14 days after the course ends, provided you meet the completion criteria in our refund policy.

Course syllabus

Week 1

Feb 4—Feb 9

    Fundamentals and Synthetic Data Generation

    0 items

    Feb

    4

    Office Hour

    Tue 2/42:00 PM—3:00 PM (UTC)
    Optional

    Feb

    4

    Office Hour

    Tue 2/46:00 PM—7:00 PM (UTC)
    Optional

    Feb

    5

    Guest Speaker

    Wed 2/56:00 PM—7:00 PM (UTC)

    Feb

    6

    Office Hour

    Thu 2/67:00 PM—8:00 PM (UTC)
    Optional

Week 2

Feb 10—Feb 16

    Segmentation and Analysis

    0 items

    Feb

    11

    Office Hour

    Tue 2/112:00 PM—3:00 PM (UTC)
    Optional

    Feb

    11

    Office Hour

    Tue 2/116:00 PM—7:00 PM (UTC)
    Optional

    Feb

    12

    Guest Speaker

    Wed 2/126:00 PM—7:00 PM (UTC)

    Feb

    13

    Office Hour

    Thu 2/137:00 PM—8:00 PM (UTC)
    Optional

Week 3

Feb 17—Feb 23

    Structured Extraction and Multimodality

    0 items

    Feb

    18

    Office Hour

    Tue 2/182:00 PM—3:00 PM (UTC)
    Optional

    Feb

    18

    Office Hour

    Tue 2/186:00 PM—7:00 PM (UTC)
    Optional

    Feb

    19

    Guest Speaker

    Wed 2/196:00 PM—7:00 PM (UTC)

    Feb

    20

    Office Hour

    Thu 2/207:00 PM—8:00 PM (UTC)
    Optional

Week 4

Feb 24—Mar 2

    Query Routing and Tool Selection

    0 items

    Feb

    25

    Office Hour

    Tue 2/252:00 PM—3:00 PM (UTC)
    Optional

    Feb

    25

    Office Hour

    Tue 2/256:00 PM—7:00 PM (UTC)
    Optional

    Feb

    26

    Guest Speaker

    Wed 2/266:00 PM—7:00 PM (UTC)

    Feb

    27

    Office Hour

    Thu 2/277:00 PM—8:00 PM (UTC)
    Optional

Week 5

Mar 3—Mar 9

    Representations and Fine-tuning

    0 items

    Mar

    4

    Office Hour

    Tue 3/42:00 PM—3:00 PM (UTC)
    Optional

    Mar

    4

    Office Hour

    Tue 3/46:00 PM—7:00 PM (UTC)
    Optional

    Mar

    5

    Guest Speaker

    Wed 3/56:00 PM—7:00 PM (UTC)

    Mar

    6

    Office Hour

    Thu 3/67:00 PM—8:00 PM (UTC)
    Optional

Week 6

Mar 10—Mar 13

    Product Design and User Experience

    0 items

    Mar

    11

    Office Hour

    Tue 3/111:00 PM—2:00 PM (UTC)
    Optional

    Mar

    11

    Office Hour

    Tue 3/115:00 PM—6:00 PM (UTC)
    Optional

    Mar

    12

    Guest Speaker

    Wed 3/125:00 PM—6:00 PM (UTC)

    Mar

    13

    Office Hour

    Thu 3/136:00 PM—7:00 PM (UTC)
    Optional

Post-course

    Product Design

    5 items

    Rejecting work

    3 items

    Intro To The Playbook

    2 items

    RAG Evaluation

    4 items

    Synthetic Data

    1 item

    Identifying Areas of Improvement

    4 items

    Production Monitoring and Analysis

    0 items

    Improving Retrieval

    4 items

    Tables and Non-Text Data

    2 items

    Routing Queries

    4 items

    Representations

    0 items

    Synthetic Text Chunks

    3 items

4.8 (30 ratings)

What students are saying

Meet your instructor

Jason Liu

Jason Liu

Jason has built search and recommendation systems for the past 6 years. He has consulted and advised a dozens startups in the last year to improve their RAG systems. He is the creator of the Instructor Python library.

A pattern of wavy dots

Join an upcoming cohort

Systematically Improving RAG Applications

Cohort 2

$1,650

Dates

Feb 4—Mar 13, 2025

Application Deadline

Feb 1, 2025

Course Schedule Each Week

  • Tuesday: Workshops

    1:00 - 2:00PM ET

    Workshops covering each step of the playbook and helping you build process improvements in your RAG application

  • Wednesday: Office Hours + Breakout Sessions

    1:00 - 2:00PM ET

    The first half hour will be interactive breakout sessions, and the closing half-hour each week is Q&A

  • Thursday: Guest Speakers

    1:00 - 2:00PM ET

    Guest instructors covering key topics in both innovative theory and practical applications in RAG system development.

Learning is better with cohorts

Learning is better with cohorts

Active hands-on learning

This course builds on live workshops and hands-on projects

Interactive and project-based

You’ll be interacting with other learners through breakout rooms and project teams

Learn with a cohort of peers

Join a community of like-minded people who want to learn and grow alongside you

Frequently Asked Questions

Stay in the loop

Sign up to be the first to know about course updates.

A pattern of wavy dots

Join an upcoming cohort

Systematically Improving RAG Applications

Cohort 2

$1,650

Dates

Feb 4—Mar 13, 2025

Application Deadline

Feb 1, 2025

$1,650

4.8 (30)

·

6 Weeks