Use LLMs as Judges for Search Result Quality

Hosted by René Kriegler and Trey Grainger

Tue, Oct 28, 2025

3:00 PM UTC (1 hour 15 minutes)

Virtual (Zoom)

Free to join

72 students

Invite your network

Go deeper with a course

AI-Powered Search
Doug Turnbull and Trey Grainger
View syllabus

What you'll learn

How can LLMs help us quantify search result quality?

Learn how LLMs as judges compare to human and implicit search evaluation and when to use them.

LLMs need to judge based on rules

Learn why you should not just ask the LLM whether a document is relevant and how you can formulate those rules.

Advanced: Using critique models and personas

Improving LLM judgments further and adapting to user segments by using critique models and personas

Why this topic matters

Search result quality evaluation is at the core of any search improvement work. LLMs as Judges can be a game-changing tool for practitioners. The promise to overcome limitations of human or implicit judgments in terms of scalability, rapid availability and in parts also in terms of what aspects of result quality they cover - provided we get them right.

You'll learn from

René Kriegler

Chief Strategy Officer at OpenSource Connections

René has worked in search for almost two decades, including in projects for some of the top 10 German e-commerce sites. He is co-founder and co-organiser of MICES (Mix-Camp E-commerce Search), an event that brings together the e-commerce search community each year. His technological focus is on open-source search technologies. He created and maintains the Querqy open source library for query rewriting. He believes that good search is not just a result of good technology but also of a team’s understanding of their search users, their experimentation capabilities and providing their users with a good search UX.


René works as Chief Strategy Officer at OpenSource Connections, where he and his team empower the company’s clients on search and AI.

Trey Grainger

Author, "AI-Powered Search", Founder @ Searchkernel

Trey is author of the book AI-Powered Search and is the founder of Searchkernel, a software company building the next generation of AI-powered search. He is an advisor to several startups and adjunct professor of computer science at Furman University. He previously served as CTO of Presearch, a decentralized web search engine, and as chief algorithms officer and SVP of engineering at Lucidworks, an search company whose search technology powers hundreds of the world’s leading organizations.


Trey is an instructor of Maven's AI-Powered Search course.

Built Search At

OpenSource Connections
Searchkernel
CareerBuilder
Lucidworks
Presearch

Sign up to join this lesson

By continuing, you agree to Maven's Terms and Privacy Policy.