Applied LLMs

Cohort 1

Cohort 2

Cohort 4

Cohort 3

Follow a repeatable process to continually evaluate and improve your RAG application

Jason Liu

Systematically Improving RAG Applications

Product Leader

Staff Analytics Engineer

Systems Engineer

Senior Data Scientist, Product Analytics

ML Engineer

Python Developer

Product Builder

AI Engineer

Transform your retrieval from "good enough" to "mission-critical" in weeks, not months Most RAG implementations get stuck in prototype purgatory. They work well for simple cases but fail on complex queries—leading to frustrated users, lost trust, and wasted engineering time. The difference between a prototype and a production-ready system isn't just better technology, it's a fundamentally different mindset. The RAG Implementation RealityWhat you're experiencing right now: ❌ Your RAG demo impressed stakeholders, but real users encounter hallucinations when they need accuracy most❌ Engineers spend countless hours tweaking prompts with minimal improvement❌ Colleagues report finding information manually that your system failed to retrieve❌ You keep making changes but have no way to measure if they're actually helping❌ Every improvement feels like guesswork instead of systematic progress❌ You're unsure which 10% of possible enhancements will deliver 90% of the value What your RAG system could be:With the RAG Flywheel methodology, you'll build a system that: ✅ Retrieves the right information even for complex, ambiguous queries✅ Continuously improves with each user interaction✅ Provides clear metrics to demonstrate ROI to stakeholders✅ Allows your team to make data-driven decisions about improvements✅ Adapts to different content types with specialized capabilities✅ Creates value that compounds over time instead of degrading What Makes This Course DifferentUnlike courses that focus solely on technical implementation, this program gives you the systematic, data-driven approach used by companies to transform prototypes into production systems that deliver real business value: ✅ The Improvement Flywheel: Build synthetic evaluation data that identifies exactly what's failing in your system—even before you have users✅ Fine-tuning Framework: Create custom embedding models with minimal data (as few as 6,000 examples)✅ Feedback Acceleration: Design interfaces that collect 5x more high-quality feedback without annoying users✅ Segmentation System: Analyze user queries to identify which segments need specialized retrievers for 20-40% accuracy gains✅ Multimodal Architecture: Implement specialized indices for different content types (documents, images, tables)✅ Query Routing: Create a unified system that intelligently selects the right retriever for each query The Complete RAG Implementation Framework Week 1: Evaluation SystemsBuild synthetic datasets that pinpoint RAG failures instead of relying on subjective assessments BEFORE: "We need to make the AI better, but we don't know where to start."AFTER: "We know exactly which query types are failing and by how much." Week 2: Fine-tune EmbeddingsCustomize models for 20-40% accuracy gains with minimal examples BEFORE: "Generic embeddings don't understand our domain terminology."AFTER: "Our embedding models understand exactly what 'similar' means in our business context." Week 3: Feedback SystemsDesign interfaces that collect 5x more feedback without annoying users BEFORE: "Users get frustrated waiting for responses and rarely tell us what's wrong."AFTER: "Every interaction provides signals that strengthen our system." Week 4: Query SegmentationIdentify high-impact improvements and prioritize engineering resources BEFORE: "We don't know which features would deliver the most value."AFTER: "We have a clear roadmap based on actual usage patterns and economic impact." Week 5: Specialized SearchBuild specialized indices for different content types that improve retrieval BEFORE: "Our system struggles with anything beyond basic text documents."AFTER: "We can retrieve information from tables, images, and complex documents with high precision." Week 6: Query RoutingImplement intelligent routing that selects optimal retrievers automatically BEFORE: "Different content requires different interfaces, creating a fragmented experience."AFTER: "Users have a seamless experience while the system intelligently routes to specialized components." Real-world Impact From Implementation﻿✅ 85% blueprint image recall: Construction company using visual LLM captioning✅ 90% research report retrieval: Through better text preprocessing techniques✅ $50M revenue increase: Retail company enhancing product search with embedding fine-tuning✅ +14% accuracy boost: Fine-tuning cross-encoders with minimal examples✅ +20% response accuracy: Using re-ranking techniques✅ -30% irrelevant documents: Through improved query segmentation Join 400+ engineers who've transformed their RAG systems with this methodology Your Instructor Jason Liu has built AI systems across diverse domains—from computer vision at the University of Waterloo to content policy at Facebook to recommendation systems at Stitch Fix that boosted revenue by $50 million. His background in managing large-scale data curation, designing multimodal retrieval models, and processing hundreds of millions of recommendations weekly has directly informed his consulting work with companies implementing RAG systems.

Stop building RAG systems that impress in demos but disappoint in production

Course syllabus

The goal of this course is not just to share with you a how-to guide, but rather how to systematically improve these architectures.

Deployed a RAG System

We have over 20 iPython notebooks that you can explore, run code to be more hands-on with the experiments that we plan to run.

Optional (Python)

A product leader, engineer, or data scientist looking to move beyond ad-hoc RAG prototypes into scalable, production-grade AI solutions.

A professional who understands LLM basics but wants a repeatable, data-driven methodology to improve retrieval relevance, latency, and user

Eager to create feedback loops that continuously refine and enhance the quality of RAG applications as models, data, and user needs evolve.

This Course Is For You If You Are

By the end of this course, participants will be able to

Implement the Data and Evals Flywheel approach to continuously develop and improve RAG applications—breaking free from guesswork and relying on measurable, iterative enhancements.

Adopt a Systematic, Data-First Methodology

Quickly assess your retrieval systems using precision and recall metrics, identify bottlenecks, and confidently validate changes without sinking into endless trial-and-error cycles.

Run Fast, Unit-Test-Like Evaluations

Generate and utilize synthetic data sets to speed up experimentation, enabling you to test new approaches, embeddings, and architectures before committing full resources.

Leverage Synthetic Data for Rapid Iteration

Apply fine-tuning strategies for embedding models to boost search relevance and explore hard negative examples to further sharpen retrieval performance.

Master Fine-Tuning & Hard Negative Mining

Use query classification and segmentation techniques to pinpoint exactly where your RAG system falls short—whether it’s due to limited inventory or insufficient capabilities.

Classify Queries & Identify Bottlenecks

Create tailored indices for documents, images, tables, SQL databases, and more. Learn when and how to fuse or layer these indices to handle complex retrieval tasks elegantly.

Design Specialized Indices for Multiple Modalities

Implement synthetic text chunk generation and strategic summarization methods to improve retrieval results, ensuring end-users get clear, concise, and contextually rich answers.

Enhance Retrieval with Summarization & Chunking

Develop systems that intelligently route queries to the right indices, tools, or pipelines. Blend and fuse indices effectively to handle nuanced, multi-step queries.

Implement Query Routing & Index Fusion

Evaluate the performance of your routing logic and each individual index separately. Gain the nuance to fine-tune global system performance and local retrieval accuracy in tandem.

Optimize Both Global & Local Performance

Design explicit and implicit feedback mechanisms—capturing user signals, automating re-labeling, and applying improvements in real-time to keep your RAG systems on an upward trajectory.

Integrate Feedback Loops for Continuous Improvement

From Text-RAG to Vision-RAG with Cohere

Consultant at the intersection of Information Retrieval and AI

Nils Reimers

VP AI Search, Cohere.com

Learn to architect complete multimodal RAG pipelines that process both text and visual content effectively.

Build End-to-End Vision RAG Systems

Master techniques to integrate cutting-edge vision models for understanding charts, graphs, and images.

Implement Vision Embeddings & LLMs

Develop strategies to seamlessly combine textual and visual information retrieval for enterprise applications.

Bridge Text-Visual Modality Gaps

Building Durable Agents with Pydantic AI

Samuel Colvin

Founder, Pydantic

Build agents that run for hours with state management and streaming results using durable execution.

Design Long-Running Agent Workflows

Implement production-ready agents using Pydantic AI's temporal integration for job tracking and monitoring.

Integrate Pydantic AI with Temporal

Identify when to use durable vs. chat interfaces and architect agents for real economic value creation.

Move Beyond Chat-Based Agent Limits

Reliable Agents: Intent-Driven Failure Detection

Ben Hylak

Co-Founder, Raindrop

Learn to cluster user intents and analyze failure rates to pinpoint which user goals cause issues.

Identify Intent-Based Tool Call Failure Patterns

Set up automated tracking to detect when tools fail for specific intent categories in production.

Build Monitoring Systems for Agent Reliability

Apply intent-failure insights to improve tool selection and boost user retention metrics.

Use Failure Data to Optimize Agent Performance

Context Rot with ChromaDB

Kelly Hong

Researcher at Chroma

Learn how input length degrades model performance through evaluation of 18 state-of-the-art LLMs

Context Length Effects on LLM Reliability

Understand methods to test and quantify how LLMs handle varying context lengths in practice

Measuring Performance Degradation Patterns

Discover why LLMs don't process all tokens equally and implications for real-world applications

Non-Uniform Context Processing in LLMs

Billion-Scale Vector Search on Object Storage

Simon Hørup Eskildsen

turbopuffer co-founder

Students will learn to design vector search systems using object storage over traditional disk storage.

Object Storage Vector Search Architecture

Students will learn cost-effective strategies for scaling vector search to petabyte-scale datasets affordably.

Scaling Economics for Petabyte Data

Students will analyze case studies from Cursor, Notion, and Linear to apply vector search design principles.

Real-World Implementation Analysis

Lessons from building verticalized AI agents

Chris Lovejoy

Head of Clinical AI at Anterior

Use domain expert feedback to systematically identify and fix AI failures in vertical applications.

Bridge Data and AI with Failure Mode Taxonomy

Build expert-driven knowledge bases and UIs that enable non-technical users to improve model performance.

Choose Prompting vs Finetuning for Domain AI

Create systems handling sensitive industry data while enabling expert loops and knowledge generation.

Design Secure Vertical AI Architecture

How You Catch Production Hallucinations in Real Time

Julia Neagu

Co-founder & CEO, Quotient AI

Learn to implement monitoring systems that catch AI errors as they happen in live production environments

Build real-time hallucination detectors

Understand how to identify and troubleshoot broken agent behavior using real-world debugging techniques

Debug production AI agent failures

Master adding detection systems to existing AI workflows using just a few lines of practical code

Deploy monitoring with minimal code

Rethinking RAG from first principles for agents

Beyang Liu

CTO and co-founder of Sourcegraph

Students will identify key structural differences between agent and chatbot systems regarding context control.

Agent vs. Chatbot Architecture Differences

Students will design simpler retrieval systems optimized for agents over complex ranking engines.

Agentic RAG Implementation Strategies

Students will apply multi-agent composition patterns to overcome context window limitations.

Multi-Agent System Composition Techniques

Scaling Judge-Time Compute for Robust Auto LLM Evaluation

Leonard Tang

Co-Founder & CEO @ Haize Labs

Identify key failure modes in automated LLM evaluation including bias and inconsistent outputs

LLM Judge Reliability Issues

Apply reasoning model training techniques to improve evaluation reliability and accuracy

Judge-Time Compute Scaling

Implement reinforcement learning methods to build more robust automated assessment tools

RL-Powered Evaluation Systems

Why Devin does not use multi agents

Walden Yan

Co-founder & CPO @ Cognition

Students will identify why multi-agent LLM frameworks fail in practice.

Multi-agent framework limitations

Students will apply foundational context engineering techniques to improve single-agent LLM performance and reliability.

Context engineering principles

Students will evaluate when to choose single-agent vs multi-agent approaches for different LLM application scenarios.

Applied LLMs

Applied LLMs

Courses

Lightning Lessons

Reviews