Featured

The Hidden Signal in Production AI Logs

Hosted by Jason Liu and Scott Clark

Wed, Jan 21, 2026

6:00 PM UTC (1 hour)

Virtual (Zoom)

Free to join

Invite your network

Go deeper with a course

Featured in Lenny’s List
Systematically Improving RAG Applications
Jason Liu
View syllabus

What you'll learn

Enrich AI logs with behavioral metrics

Learn to augment production logs with statistical metrics, evals, and LLM-as-judge signals

Apply clustering and anomaly detection

Use high-dimensional clustering and drift detection to surface hidden behavioral patterns

Build actionable analysis workflows

Create systematic processes to translate behavioral signals into product improvements

Why this topic matters

Production AI fails when teams can't see what's actually happening. This lesson teaches you to decode the black box—transforming messy logs into behavioral insights that reveal why your AI works or breaks. You'll gain the analytical framework to systematically improve AI products, moving beyond surface metrics to understand and fix real user issues at scale.

You'll learn from

Jason Liu

Consultant at the intersection of Information Retrieval and AI

Jason has built search and recommendation systems for the past 6 years. He has consulted and advised a dozens startups in the last year to improve their RAG systems. He is the creator of the Instructor Python library. 

Scott Clark

Co-Founder & CEO, Distributional

Scott Clark is Co-Founder and CEO of Distributional, the first enterprise platform that analyzes hidden behavioral signals from production log data to help you continuously improve your AI products.

The platform uses unsupervised learning to surface subsets of log data corresponding to shifts, clusters, or outliers in AI behavior that a user can investigate and track through custom filters, metrics, and alerts to catch issues and adapt this analysis over time. This empowers AI teams to better understand the behavior of their users and AI applications so that they can fix and improve those applications with confidence.

Scott was previously the Co-Founder and CEO of SigOpt, an enterprise AI optimization platform. After selling SigOpt to Intel in 2020, Scott was VP of AI and HPC engineering within the Supercomputing organization. Prior to SigOpt, Scott worked and performed research at various technology companies, universities, and national labs around the world. Scott holds a PhD in Applied Math and MS in Computer Science from Cornell University and BS degrees in Mathematics, Physics, and Computational Physics from Oregon State University. Scott was named to the Forbes 30 under 30 for enterprise tech in 2016.

worked with

Distributional
Stitch Fix
Meta
University of Waterloo
New York University

Sign up to join this lesson

By continuing, you agree to Maven's Terms and Privacy Policy.