Applied LLMs

Cohort 1

Cohort 2

Cohort 4

Cohort 3

Cohort 5

Follow a repeatable process to continually evaluate and improve your RAG application

Jason Liu

Staff machine learning engineer, currently working as an AI consultant

Systematically Improving RAG Applications

Indie Consultant

Founder and CTO

Product manager

Consultant

Freelance AI Engineer

Founder

Software developer

Product Leader

Staff Analytics Engineer

Systems Engineer

OpenAI

Anthropic

Microsoft

Google

Stop building RAG systems that impress in demos but disappoint in production

Analyze and Diagnose RAG System Performance

Construct Data-Driven Improvement Frameworks

Design Specialized Search Systems

Optimize Query Understanding & Routing

The goal of this course is not just to share with you a how-to guide, but rather how to systematically improve these architectures.

Deployed a RAG System

We have over 20 iPython notebooks that you can explore, run code to be more hands-on with the experiments that we plan to run.

Optional (Python)

A product leader, engineer, or data scientist looking to move beyond ad-hoc RAG prototypes into scalable, production-grade AI solutions.

A professional who understands LLM basics but wants a repeatable, data-driven methodology to improve retrieval relevance, latency, and user

Eager to create feedback loops that continuously refine and enhance the quality of RAG applications as models, data, and user needs evolve.

Shipping an Agent: Lessons from LangChain’s Own Deployment

Consultant at the intersection of Information Retrieval and AI

Anika Somaia

Software Engineer, LangChain

Learn methods to test complex reasoning chains and prevent regressions before deployment

Evaluate multi-turn LLM agents in production

Identify and resolve memory, context, and prompt drift issues using production tooling

Debug common agent failure modes effectively

Connect evaluation metrics, logs, and user data to drive iterative product development

Build feedback loops for AI system improvement

How to Catch Tool-Call Errors in AI Agents

Julia Neagu

Co-founder & CEO, Quotient AI

Learn to identify when AI agents call wrong tools or pass incorrect parameters in production

Detect silent tool-call failures

Apply practical techniques using evaluators like limbic-tool-use to measure agent reliability

Evaluate tool-use accuracy at scale

Implement real-time analysis methods to debug and enhance how agents use external tools

Trace and improve agent tool interactions

Strategies for building self-improving document processing

Eli Badgio

Co-founder and CTO at Extend

Learn techniques for creating evaluation frameworks that measure end-to-end pipeline accuracy

Building high-quality document processing evals

Discover how to implement agents that automatically optimize document processing workflows

Deploying multimodal retrieval agents

Understand methods for building self-improving systems that enhance their own performance

Automating pipeline iteration and improvement

From Text-RAG to Vision-RAG with Cohere

Nils Reimers

VP AI Search, Cohere.com

Learn to architect complete multimodal RAG pipelines that process both text and visual content effectively.

Build End-to-End Vision RAG Systems

Master techniques to integrate cutting-edge vision models for understanding charts, graphs, and images.

Implement Vision Embeddings & LLMs

Develop strategies to seamlessly combine textual and visual information retrieval for enterprise applications.

Bridge Text-Visual Modality Gaps

Building Durable Agents with Pydantic AI

Samuel Colvin

Founder, Pydantic

Build agents that run for hours with state management and streaming results using durable execution.

Design Long-Running Agent Workflows

Implement production-ready agents using Pydantic AI's temporal integration for job tracking and monitoring.

Integrate Pydantic AI with Temporal

Identify when to use durable vs. chat interfaces and architect agents for real economic value creation.

Move Beyond Chat-Based Agent Limits

Pushing document parsing frontiers with Reducto

Evan Vogelbaum

Machine Learning, Reducto

Yifei Hu

Alvin Ryanputra

Engineering, Reducto

Discover how to combine traditional OCR with vision language models to achieve high accuracy and low latency.

Hybrid OCR-VLM Architecture Design

Learn to build production systems that parse complex multi-page documents without hallucination issues.

Large-Scale Document Processing Pipeline

Master techniques for enabling PDF/Word editing through conversational interfaces and structured extraction.

Natural Language Document Automation

Agents in production from a designer's perspective

David Kim

Design engineer, Braintrust

How design thinking shapes agent development from concept through production deployment and iteration

Shipping agents: a designer's journey

Why these technical systems matter for design decisions and how they enable better agent experiences

Eval infrastructure & observability for designers

Concrete examples of design challenges, wins, and failures when creating a production AI agent

Real lessons from building Loop

Real-World Agents: Case Studies from LlamaIndex

Laurie Voss

VP, Developer Relations at LlamaIndex

Students will understand what LlamaIndex is and its role in building document agents.

LlamaIndex Framework Overview

Students will examine actual applications that use LlamaIndex technologies in practice.

Real-World Application Examples

Students will review the actual results and outcomes achieved by these applications.

Outcomes and Results Analysis

Reliable RAG Agents: Intent-Driven Failure Detection

Ben Hylak

Co-Founder, Raindrop

Learn to cluster user intents and analyze failure rates to pinpoint which user goals cause issues.

Identify Intent-Based Tool Call Failure Patterns

Set up automated tracking to detect when tools fail for specific intent categories in production.

Build Monitoring Systems for Agent Reliability

Apply intent-failure insights to improve tool selection and boost user retention metrics.

Use Failure Data to Optimize Agent Performance

MCP Security: The Exploit Playbook (And How to Stop Them)

Vitor Balocco

Co-Founder, Anysource

Recognize tool poisoning, rug-pulls, conversation theft, and cross-server shadowing attack patterns.

Identify MCP Attack Vectors

Apply practical security patterns and validation techniques to protect your MCP servers from common vulnerabilities.

Implement MCP Security Defenses

Use security checklists to evaluate MCP implementations and develop incident response strategies for attacks.

Applied LLMs

Applied LLMs

Courses

Lightning Lessons

Reviews