Applied LLMs

Cohort 1

Cohort 2

Cohort 4

Cohort 3

Cohort 5

Follow a repeatable process to continually evaluate and improve your RAG application

Jason Liu

Staff machine learning engineer, currently working as an AI consultant

Systematically Improving RAG Applications

Product Leader

Staff Analytics Engineer

Systems Engineer

Senior Data Scientist, Product Analytics

ML Engineer

AI Engineer

Product Builder

OpenAI

Anthropic

Microsoft

Google

Stop building RAG systems that impress in demos but disappoint in production

Analyze and Diagnose RAG System Performance

Construct Data-Driven Improvement Frameworks

Design Specialized Search Systems

Optimize Query Understanding & Routing

The goal of this course is not just to share with you a how-to guide, but rather how to systematically improve these architectures.

Deployed a RAG System

We have over 20 iPython notebooks that you can explore, run code to be more hands-on with the experiments that we plan to run.

Optional (Python)

A product leader, engineer, or data scientist looking to move beyond ad-hoc RAG prototypes into scalable, production-grade AI solutions.

A professional who understands LLM basics but wants a repeatable, data-driven methodology to improve retrieval relevance, latency, and user

Eager to create feedback loops that continuously refine and enhance the quality of RAG applications as models, data, and user needs evolve.

Strategies for building self-improving document processing

Consultant at the intersection of Information Retrieval and AI

Eli Badgio

Co-founder and CTO at Extend

Learn techniques for creating evaluation frameworks that measure end-to-end pipeline accuracy

Building high-quality document processing evals

Discover how to implement agents that automatically optimize document processing workflows

Deploying multimodal retrieval agents

Understand methods for building self-improving systems that enhance their own performance

Automating pipeline iteration and improvement

From Text-RAG to Vision-RAG with Cohere

Nils Reimers

VP AI Search, Cohere.com

Learn to architect complete multimodal RAG pipelines that process both text and visual content effectively.

Build End-to-End Vision RAG Systems

Master techniques to integrate cutting-edge vision models for understanding charts, graphs, and images.

Implement Vision Embeddings & LLMs

Develop strategies to seamlessly combine textual and visual information retrieval for enterprise applications.

Bridge Text-Visual Modality Gaps

Building Durable Agents with Pydantic AI

Samuel Colvin

Founder, Pydantic

Build agents that run for hours with state management and streaming results using durable execution.

Design Long-Running Agent Workflows

Implement production-ready agents using Pydantic AI's temporal integration for job tracking and monitoring.

Integrate Pydantic AI with Temporal

Identify when to use durable vs. chat interfaces and architect agents for real economic value creation.

Move Beyond Chat-Based Agent Limits

Pushing document parsing frontiers with Reducto

Evan Vogelbaum

Machine Learning, Reducto

Yifei Hu

Alvin Ryanputra

Engineering, Reducto

Discover how to combine traditional OCR with vision language models to achieve high accuracy and low latency.

Hybrid OCR-VLM Architecture Design

Learn to build production systems that parse complex multi-page documents without hallucination issues.

Large-Scale Document Processing Pipeline

Master techniques for enabling PDF/Word editing through conversational interfaces and structured extraction.

Natural Language Document Automation

Agents in production from a designer's perspective

David Kim

Design engineer, Braintrust

How design thinking shapes agent development from concept through production deployment and iteration

Shipping agents: a designer's journey

Why these technical systems matter for design decisions and how they enable better agent experiences

Eval infrastructure & observability for designers

Concrete examples of design challenges, wins, and failures when creating a production AI agent

Real lessons from building Loop

Real-World Agents: Case Studies from LlamaIndex

Laurie Voss

VP, Developer Relations at LlamaIndex

Students will understand what LlamaIndex is and its role in building document agents.

LlamaIndex Framework Overview

Students will examine actual applications that use LlamaIndex technologies in practice.

Real-World Application Examples

Students will review the actual results and outcomes achieved by these applications.

Outcomes and Results Analysis

Reliable RAG Agents: Intent-Driven Failure Detection

Ben Hylak

Co-Founder, Raindrop

Learn to cluster user intents and analyze failure rates to pinpoint which user goals cause issues.

Identify Intent-Based Tool Call Failure Patterns

Set up automated tracking to detect when tools fail for specific intent categories in production.

Build Monitoring Systems for Agent Reliability

Apply intent-failure insights to improve tool selection and boost user retention metrics.

Use Failure Data to Optimize Agent Performance

MCP Security: The Exploit Playbook (And How to Stop Them)

Vitor Balocco

Co-Founder, Anysource

Recognize tool poisoning, rug-pulls, conversation theft, and cross-server shadowing attack patterns.

Identify MCP Attack Vectors

Apply practical security patterns and validation techniques to protect your MCP servers from common vulnerabilities.

Implement MCP Security Defenses

Use security checklists to evaluate MCP implementations and develop incident response strategies for attacks.

Assess MCP Risk & Response

Cheating at Query Understanding with LLMs

Doug Turnbull

Principal Consultant

Learn core principles for using LLMs to extract entities and classify user search intent at scale.

LLM-Powered Query Understanding Fundamentals

Master proven techniques for deploying LLM query understanding across e-commerce and search platforms.

Practical Implementation Strategies

Discover methods to scale LLM query processing while maintaining accuracy and response times.

Performance Optimization for Production

Improving RAG: Embedding models & representation learning

Daniel Svonava

CEO at Superlinked

Students will learn to build embedding systems integrating databases, rerankers, and query models at scale.

Building Effective Embedding Pipelines

Students will learn frameworks for choosing pre-trained models, when to fine-tune, and evaluating training approaches.

Strategic Model Selection & Training Decisions

Students will gain insights from real-world case studies of deploying embedding systems in e-commerce and enterprise.

Applied LLMs

Applied LLMs

Courses

Lightning Lessons

Reviews