Applied LLMs

Cohort 1

Cohort 2

Cohort 4

Cohort 3

Cohort 5

Follow a repeatable process to continually evaluate and improve your RAG application

Jason Liu

Staff machine learning engineer, currently working as an AI consultant

Systematically Improving RAG Applications

Product Manager

Founder, CTO

Independent Software & AI Consultant 🤓✨

Advanced Analytics Lead 

Co-founder / Chief Scientist / CTO / AI Consultant

Indie Consultant

Founder and CTO

Product manager

Consultant

Independent AI Engineer, Consultant & Angel Investor

OpenAI

Anthropic

Microsoft

Google

Stop building RAG systems that impress in demos but disappoint in production

Analyze and Diagnose RAG System Performance

Construct Data-Driven Improvement Frameworks

Design Specialized Search Systems

Optimize Query Understanding & Routing

The goal of this course is not just to share with you a how-to guide, but rather how to systematically improve these architectures.

Deployed a RAG System

We have over 20 iPython notebooks that you can explore, run code to be more hands-on with the experiments that we plan to run.

Optional (Python)

A product leader, engineer, or data scientist looking to move beyond ad-hoc RAG prototypes into scalable, production-grade AI solutions.

A professional who understands LLM basics but wants a repeatable, data-driven methodology to improve retrieval relevance, latency, and user

Eager to create feedback loops that continuously refine and enhance the quality of RAG applications as models, data, and user needs evolve.

Why Devin does not use multi agents

Consultant at the intersection of Information Retrieval and AI

Walden Yan

Co-founder & CPO @ Cognition

Students will identify why multi-agent LLM frameworks fail in practice.

Multi-agent framework limitations

Students will apply foundational context engineering techniques to improve single-agent LLM performance and reliability.

Context engineering principles

Students will evaluate when to choose single-agent vs multi-agent approaches for different LLM application scenarios.

Alternative architecture strategies

Lessons on retrieval for autonomous coding agents

Nik Pash

Head of AI @ Cline

Learn how retrieval systems create cognitive overhead that reduces agent performance and focus

Why RAG distracts coding agents

Discover proven frameworks for measuring RAG pipeline effectiveness and ROI impact

Metrics-driven RAG evaluation methods

Identify scenarios where direct code generation outperforms retrieval-augmented approaches

When to skip RAG for coding tasks

RAG in the age of agents. SWE-Bench as a case study.

Colin Flaherty

Working on something new

Students will learn how retrieval-agent systems achieve top performance on software engineering benchmarks.

Agent-Augmented RAG for Code Tasks

Students will explore how staged task decomposition improves agent performance on complex programming tasks.

Multi-Stage Reasoning in Code Agents

Students will gain insights into building effective code agents from analyzing top SWE-Bench solutions.

Open-Source Agent Implementation Techniques

Lessons from building verticalized AI agents

Chris Lovejoy

Head of Clinical AI at Anterior

Use domain expert feedback to systematically identify and fix AI failures in vertical applications.

Bridge Data and AI with Failure Mode Taxonomy

Build expert-driven knowledge bases and UIs that enable non-technical users to improve model performance.

Choose Prompting vs Finetuning for Domain AI

Create systems handling sensitive industry data while enabling expert loops and knowledge generation.

Design Secure Vertical AI Architecture

Building Durable Agents with Pydantic AI

Samuel Colvin

Founder, Pydantic

Build agents that run for hours with state management and streaming results using durable execution.

Design Long-Running Agent Workflows

Implement production-ready agents using Pydantic AI's temporal integration for job tracking and monitoring.

Integrate Pydantic AI with Temporal

Identify when to use durable vs. chat interfaces and architect agents for real economic value creation.

Move Beyond Chat-Based Agent Limits

Natural Language Search on Semi-Structured Data

Consultant at the intersection of Informational Retrieval and AI

Daniel Svonava

CEO at Superlinked

Students will learn how to combine encoders to build models that understand both text and metadata.

Encoder Stacking Fundamentals

Students will learn practical deployment techniques for encoder stacking in search and recommendation systems.

Real-World Implementation Strategies

Students will learn when to use encoder stacking over traditional metadata approaches.

Metadata Integration Beyond Basics

Understanding Embedding Performance through Generative Evals

Kelly Hong

Researcher at Chroma

Discover why AI systems need specialized benchmarking beyond traditional testing methods.

AI Evaluation Challenges

Identify the shortcomings of public benchmarks, including clean datasets and potential training data contamination.

Benchmark Limitations

Apply techniques to generate benchmark tests that accurately reflect real-world user queries and production conditions.

Representativeness in Testing

Rethinking RAG from first principles for agents

Beyang Liu

CTO and co-founder of Sourcegraph

Students will identify key structural differences between agent and chatbot systems regarding context control.

Agent vs. Chatbot Architecture Differences

Students will design simpler retrieval systems optimized for agents over complex ranking engines.

Agentic RAG Implementation Strategies

Students will apply multi-agent composition patterns to overcome context window limitations.

Multi-Agent System Composition Techniques

Improving retrievers by Reranking and embedding fine-tuning

Ayush Chaurasia

ML Engineer @ LanceDB

Master reranker architecture and implementation to effectively boost retrieval accuracy beyond embedding models alone.

Reranking Fundamentals

Apply practical criteria for deciding when to fine-tune embeddings vs. implementing rerankers based on use case demands.

Strategic Model Fine-tuning

Evaluate deployment tradeoffs to select optimal reranking and embedding approaches for various hardware environments.

Performance Optimization

Online Evals and Production Monitoring

Ben Hylak

Co-Founder, Raindrop

Sidhant Bendre

Co-Founder of Oleve

Identify key metrics and set up effective monitoring systems to track AI product performance in real-time.

AI Monitoring Fundamentals

Learn techniques to pinpoint problems and segment performance by use case to quickly address critical issues.

Issue Detection and Analysis

Apply user feedback patterns to prioritize product enhancements that directly increase retention and revenue

Applied LLMs

Applied LLMs

Courses

Lightning Lessons

Reviews