WTF is an NDCG!? - Search Relevance Evaluation

Lightning Lessons

Practical skills & tools to accelerate your career

WTF is an NDCG!? - Search Relevance Evaluation

Hosted by Doug Turnbull

201 students

What you'll learn

Basics of search relevance evaluation

How the best search teams evaluate and measure search relevance

Where practice diverges from theory

When to ignore traditional search metrics and just trust your A/B tests. How to interpret offline metrics.

Pros / cons of different types of labeled relevance data

Do you use clicks? Human labels? LLM as a judge? What are the pros/cons of these approaches?

Why this topic matters

Search is central to RAG and AI these days. To get better at search, know the core metrics teams have historically used to evaluate search.

You'll learn from

Doug Turnbull

Ex Reddit, Ex Shopify Search

Doug Turnbull is an expert in search technology and relevance engineering, currently serving as Principal Engineer at Daydream, where he builds hybrid search systems combining lexical and vector retrieval, and develops LLM-driven quality programs for e-commerce search. Previously, he led machine-learning-driven search initiatives at Reddit, significantly improving search relevance through Learning to Rank methods. Doug also advanced e-commerce search at Shopify and served as CTO at OpenSource Connections. He co-authored the influential book Relevant Search (Manning, 2016) and created popular open-source tools, including Quepid and the Elasticsearch Learning to Rank plugin. He regularly speaks at industry conferences, making search relevance accessible to engineers.

Previously at