Applied LLMs

Cohort 1

Cohort 2

Cohort 4

Cohort 3

Cohort 5

Follow a repeatable process to continually evaluate and improve your RAG application

Jason Liu

Staff machine learning engineer, currently working as an AI consultant

Systematically Improving RAG Applications

Product Manager

Founder, CTO

Independent Software & AI Consultant 🤓✨

Advanced Analytics Lead 

Co-founder / Chief Scientist / CTO / AI Consultant

Indie Consultant

Founder and CTO

Product manager

Consultant

Independent AI Engineer, Consultant & Angel Investor

OpenAI

Anthropic

Microsoft

Google

Meta

Transform your retrieval from “good enough” to “mission-critical” in weeks, not months. Most RAG systems stall in prototype purgatory: they demo well, but fail on complex queries—eroding trust and wasting engineering time. The difference isn’t just better tech, but a systematic mindset.With the RAG Flywheel, you’ll: ✅ Pinpoint failures with synthetic evals ✅ Fine-tune embeddings for 20–40% gains ✅ Collect 5x more user feedback ✅ Segment queries to target high-impact fixes ✅ Build multimodal indices for docs, tables, images ✅ Route queries to the best retriever automaticallyWeek by week, you move from vague “make it better” to clear metrics, focused improvements, and compounding value. Real-world results include +20% accuracy from re-ranking, +14% with cross-encoders, and $50M revenue boosts from better search.Join 400+ engineers applying this framework in production. Instructor Jason Liu has built multimodal retrieval and recommendation systems at Facebook, Stitch Fix, and through consulting—experience that shaped this practical, battle-tested approach.

Stop building RAG systems that impress in demos but disappoint in production

Analyze and Diagnose RAG System Performance

Construct Data-Driven Improvement Frameworks

Design Specialized Search Systems

Optimize Query Understanding & Routing

The goal of this course is not just to share with you a how-to guide, but rather how to systematically improve these architectures.

Deployed a RAG System

We have over 20 iPython notebooks that you can explore, run code to be more hands-on with the experiments that we plan to run.

Optional (Python)

A product leader, engineer, or data scientist looking to move beyond ad-hoc RAG prototypes into scalable, production-grade AI solutions.

A professional who understands LLM basics but wants a repeatable, data-driven methodology to improve retrieval relevance, latency, and user

Eager to create feedback loops that continuously refine and enhance the quality of RAG applications as models, data, and user needs evolve.

The Hidden Signal in Production AI Logs

Consultant at the intersection of Information Retrieval and AI

Scott Clark

Co-Founder & CEO, Distributional

Learn to augment production logs with statistical metrics, evals, and LLM-as-judge signals

Enrich AI logs with behavioral metrics

Use high-dimensional clustering and drift detection to surface hidden behavioral patterns

Apply clustering and anomaly detection

Create systematic processes to translate behavioral signals into product improvements

Build actionable analysis workflows

Dropbox: Knowledge Graphs,  Prompt Optimizers, and MCPs

Josh Clemm

VP of Engineering, Dropbox

Understand the tradeoffs and decision framework for when indexing makes sense in your RAG system

Deciding whether to index or not

Learn the technical approach and measurable customer impact from building graph-based retrieval systems

Building knowledge graphs for retrieval

Discover how super retrieval tools and codegen consolidate multiple tools into streamlined, functional systems

Solving MCP tool sprawl

Shipping an Agent: Lessons from LangChain’s Own Deployment

Anika Somaia

Software Engineer, LangChain

Learn methods to test complex reasoning chains and prevent regressions before deployment

Evaluate multi-turn LLM agents in production

Identify and resolve memory, context, and prompt drift issues using production tooling

Debug common agent failure modes effectively

Connect evaluation metrics, logs, and user data to drive iterative product development

Build feedback loops for AI system improvement

How to Catch Tool-Call Errors in AI Agents

Julia Neagu

Co-founder & CEO, Quotient AI

Learn to identify when AI agents call wrong tools or pass incorrect parameters in production

Detect silent tool-call failures

Apply practical techniques using evaluators like limbic-tool-use to measure agent reliability

Evaluate tool-use accuracy at scale

Implement real-time analysis methods to debug and enhance how agents use external tools

Trace and improve agent tool interactions

Strategies for building self-improving document processing

Eli Badgio

Co-founder and CTO at Extend

Learn techniques for creating evaluation frameworks that measure end-to-end pipeline accuracy

Building high-quality document processing evals

Discover how to implement agents that automatically optimize document processing workflows

Deploying multimodal retrieval agents

Understand methods for building self-improving systems that enhance their own performance

Automating pipeline iteration and improvement

From Text-RAG to Vision-RAG with Cohere

Nils Reimers

VP AI Search, Cohere.com

Learn to architect complete multimodal RAG pipelines that process both text and visual content effectively.

Build End-to-End Vision RAG Systems

Master techniques to integrate cutting-edge vision models for understanding charts, graphs, and images.

Implement Vision Embeddings & LLMs

Develop strategies to seamlessly combine textual and visual information retrieval for enterprise applications.

Learn directly from Applied LLMs