Free Lesson
Why Your AI Product Feels Slow Even When the Model Is Fast
Part of AI Product Engineering
45 min
Jul 2, 2026 2:00 PM
Virtual (Zoom)
In this video
What you'll learn
Trace where the latency comes from
Follow a request from arrival to first token to last token, and see where the time gets lost: queueing, batching, and prefill versus decode.
Pick the right lever for the bottleneck
Match the fix to the problem: KV cache management, chunked prefill, quantization, or speculative decoding, and the compute-bound versus memory-bound tradeoff each one addresses.
Make it feel faster without changing the model
Use streaming and time-to-first-token to improve perceived speed, and batching and concurrency to improve throughput and cost.
Why this topic matters
A capable model can still feel sluggish in production. The lag usually lives in the inference path: how requests queue and batch, prefill versus decode, the KV cache, and whether you stream tokens. Abi does this for a living. She'll show you where the latency comes from, and patterns that make agentic systems feel faster while costing less. There's a hands-on exercise so you can see it yourself.
You'll learn from
Abi Aryan
Founder of Abide AI; author of O'Reilly's LLMOps
Hamel Husain
ML Engineer with 20+ years of experience