Lightning Lessons

Context Rot: When Long Context Fails

Hosted by Kelly Hong and Hamel Husain

Wed, Sep 10, 2025

8:00 PM UTC (30 minutes)

Virtual (Zoom)

Free to join

Invite your network

Go deeper with a course

Featured in Lenny’s List
AI Evals For Engineers & PMs
Hamel Husain and Shreya Shankar
View syllabus

What you'll learn

Beyond "Needle in a Haystack"

Why standard benchmarks are misleading and fail to capture the real-world challenges of long-context applications.

How Distractors & Context Structure Cause Failure

Discover how topically-related "distractors" and even the logical flow of your context can degrade model performance.

The Importance of Context Engineering

Learn why how you present information to an LLM is just as critical as what you present for achieving reliable result

Model-Specific Failure Modes

See how different models like Claude and GPT exhibit unique failure patterns, from abstention to confident hallucination

Why this topic matters

As LLMs boast million-token context windows, it’s assumed long-context is a solved problem. Kelly Hong from Chroma reveals their  groundbreaking research on "Context Rot," showing how performance degrades in non-obvious ways. This session is critical for anyone  building reliable long-context applications and avoiding costly, unexpected failures in production.

You'll learn from

Kelly Hong

Retrieval Researcher @ Chroma

Kelly Hong is a researcher at Chroma, where she explores open questions in retrieval. She studied computer science at UC Berkeley before deciding to take a break from school to go all in on working in this space. Her recent work includes projects like generative benchmarking, driven by the motivation to help developers systematically evaluate and improve their AI systems.


Hamel Husain

ML Engineer with 20 years of experience

Hamel is a machine learning engineer with over 20 years of experience. He has worked with innovative companies such as Airbnb and GitHub, which included early LLM research used by OpenAI, for code understanding. He has also led and contributed to numerous popular open-source machine-learning tools. Hamel is currently an independent consultant helping companies build AI products.

Sign up to join this lesson

By continuing, you agree to Maven's Terms and Privacy Policy.