Go Beyond AI Evals: Diagnose and Decide

Hosted by Rajiv Shah

In this video

What you'll learn

Go to the next level of error analysis

Don't look at a metric and ask "how do I make this go up?" Instead, is this a model issue or a framing issue?

Apply diagnostic tests that go beyond standard evals

Learn quick diagnostic tests, based on years of experience doing AI projects. Catch issues earlier.

Decide whether to persist, pivot, or stop

You diagnosed the problem. Now learn to connect your signals to a decision: persist, stop, or pivot

Bonus: The AI Framing Worksheet

A one-page tool you can run on any AI project starting today.

Why this topic matters

Your AI project evals look good, but is that the whole story? JP Morgan Chase hit every accuracy target and nobody used the product. We've all watched the Sora saga unfold. Standard metrics fall short when you're solving the wrong problem or ignoring the signals to stop. After hundreds of AI use cases, I've distilled a process to assess your project's health and know when to change direction.

You'll learn from

Rajiv Shah

AI Engineer, 10+ years, 100k followers

Rajiv Shah is an AI Agentic Engineer at OpenHands.

He's worked hands-on with 100+ AI use cases with over 10 years in AI/ML across enterprise, startup, and research - from ML systems, to RAG pipelines, to coding Agents. The failures taught him more than the successes. They almost always came down to framing, not algorithms.

His career spans 20+ patents, been cited over 1000 times, a PhD from UIUC, and an expert in practical AI.

Today he reaches 100K+ practitioners through talks, videos, and content at AI conferences - known by @rajistics.

Follow Rajiv on Linkedin or Reddit

Previous Lightning Lesson