Applied LLMs

Cohort 1

Cohort 2

Cohort 4

Cohort 3

Cohort 5

Follow a repeatable process to continually evaluate and improve your RAG application

Jason Liu

Staff machine learning engineer, currently working as an AI consultant

Systematically Improving RAG Applications

Founder, CTO

Advanced Analytics Lead 

Co-founder / Chief Scientist / CTO / AI Consultant

Indie Consultant

Founder and CTO

Product manager

Consultant

Freelance AI Engineer

Founder

AI Engineer

OpenAI

Anthropic

Microsoft

Google

Stop building RAG systems that impress in demos but disappoint in production

Analyze and Diagnose RAG System Performance

Construct Data-Driven Improvement Frameworks

Design Specialized Search Systems

Optimize Query Understanding & Routing

The goal of this course is not just to share with you a how-to guide, but rather how to systematically improve these architectures.

Deployed a RAG System

We have over 20 iPython notebooks that you can explore, run code to be more hands-on with the experiments that we plan to run.

Optional (Python)

A product leader, engineer, or data scientist looking to move beyond ad-hoc RAG prototypes into scalable, production-grade AI solutions.

A professional who understands LLM basics but wants a repeatable, data-driven methodology to improve retrieval relevance, latency, and user

Eager to create feedback loops that continuously refine and enhance the quality of RAG applications as models, data, and user needs evolve.

The Hidden Signal in Production AI Logs

Consultant at the intersection of Information Retrieval and AI

Scott Clark

Co-Founder & CEO, Distributional

Learn to augment production logs with statistical metrics, evals, and LLM-as-judge signals

Enrich AI logs with behavioral metrics

Use high-dimensional clustering and drift detection to surface hidden behavioral patterns

Apply clustering and anomaly detection

Create systematic processes to translate behavioral signals into product improvements

Build actionable analysis workflows

Dropbox: Knowledge Graphs,  Prompt Optimizers, and MCPs

Josh Clemm

VP of Engineering, Dropbox

Understand the tradeoffs and decision framework for when indexing makes sense in your RAG system

Deciding whether to index or not

Learn the technical approach and measurable customer impact from building graph-based retrieval systems

Building knowledge graphs for retrieval

Discover how super retrieval tools and codegen consolidate multiple tools into streamlined, functional systems

Solving MCP tool sprawl

Shipping an Agent: Lessons from LangChain’s Own Deployment

Anika Somaia

Software Engineer, LangChain

Learn methods to test complex reasoning chains and prevent regressions before deployment

Evaluate multi-turn LLM agents in production

Identify and resolve memory, context, and prompt drift issues using production tooling

Debug common agent failure modes effectively

Connect evaluation metrics, logs, and user data to drive iterative product development

Build feedback loops for AI system improvement

How to Catch Tool-Call Errors in AI Agents

Julia Neagu

Co-founder & CEO, Quotient AI

Learn to identify when AI agents call wrong tools or pass incorrect parameters in production

Detect silent tool-call failures

Apply practical techniques using evaluators like limbic-tool-use to measure agent reliability

Evaluate tool-use accuracy at scale

Implement real-time analysis methods to debug and enhance how agents use external tools

Trace and improve agent tool interactions

Strategies for building self-improving document processing

Eli Badgio

Co-founder and CTO at Extend

Learn techniques for creating evaluation frameworks that measure end-to-end pipeline accuracy

Building high-quality document processing evals

Discover how to implement agents that automatically optimize document processing workflows

Deploying multimodal retrieval agents

Understand methods for building self-improving systems that enhance their own performance

Automating pipeline iteration and improvement

From Text-RAG to Vision-RAG with Cohere

Nils Reimers

VP AI Search, Cohere.com

Learn to architect complete multimodal RAG pipelines that process both text and visual content effectively.

Build End-to-End Vision RAG Systems

Master techniques to integrate cutting-edge vision models for understanding charts, graphs, and images.

Implement Vision Embeddings & LLMs

Develop strategies to seamlessly combine textual and visual information retrieval for enterprise applications.

Bridge Text-Visual Modality Gaps

Building Durable Agents with Pydantic AI

Samuel Colvin

Founder, Pydantic

Build agents that run for hours with state management and streaming results using durable execution.

Design Long-Running Agent Workflows

Implement production-ready agents using Pydantic AI's temporal integration for job tracking and monitoring.

Integrate Pydantic AI with Temporal

Identify when to use durable vs. chat interfaces and architect agents for real economic value creation.

Move Beyond Chat-Based Agent Limits

Pushing document parsing frontiers with Reducto

Evan Vogelbaum

Machine Learning, Reducto

Yifei Hu

Alvin Ryanputra

Engineering, Reducto

Discover how to combine traditional OCR with vision language models to achieve high accuracy and low latency.

Hybrid OCR-VLM Architecture Design

Learn to build production systems that parse complex multi-page documents without hallucination issues.

Large-Scale Document Processing Pipeline

Master techniques for enabling PDF/Word editing through conversational interfaces and structured extraction.

Natural Language Document Automation

Agents in production from a designer's perspective

David Kim

Design engineer, Braintrust

How design thinking shapes agent development from concept through production deployment and iteration

Shipping agents: a designer's journey

Why these technical systems matter for design decisions and how they enable better agent experiences

Eval infrastructure & observability for designers

Concrete examples of design challenges, wins, and failures when creating a production AI agent

Real lessons from building Loop

Real-World Agents: Case Studies from LlamaIndex

Laurie Voss

VP, Developer Relations at LlamaIndex

Students will understand what LlamaIndex is and its role in building document agents.

LlamaIndex Framework Overview

Students will examine actual applications that use LlamaIndex technologies in practice.

Real-World Application Examples

Students will review the actual results and outcomes achieved by these applications.

Applied LLMs

Applied LLMs

Courses

Lightning Lessons

Reviews