SearchArray - let's rethink full-text search

Hosted by Doug Turnbull

155 students

In this video

What you'll learn

Why search engine DSLs continue to confuse users

Every search engine forces you to learn a DSL. Why we should be removing, not adding, to the cognitive load.

How to build your own full-text search

How I built an in-memory full text search system good enough to recreate core search functionality around dataframe

A different perspective on search APIs

Why representing search relevance in a dataframe creates maximum flexibility over a traditional API

Why this topic matters

What if we could treat a search corpus as a pandas/polars dataframe? Then we could write Python instead of complex query DSL. We could add a title + body score with a a literal Python "+" operator. Or enforce "vespa" is in the title by a simple: scores[title.score('vespa') == 0] = 0. Many of us are used to numpy + pandas + pals. I share how I've rethought search to be about pure Python.

You'll learn from

Doug Turnbull

Black belt search ninja space cowboy (and consultant)

Doug leads search teams past the BS to find real opportunity in emerging search technologies. He’s enthusiastic about the evolving landscape, while staying mindful of the gap between marketing and reality. Good search strategy separates promising opportunities from dangerous sand traps. Doug helps teams find a clear, practical path forward.

He led machine-learning-driven search at Reddit and Shopify, served as CTO of OpenSource Connections, and co-authored Relevant Search and AI Powered Search.

Doug has trained and advised teams at the Wikimedia Foundation, Wayfair, and AWS, and created Quepid, SearchArray, and the Elasticsearch Learning to Rank plugin.

Go deeper with a course

Cheat at Search with Agents
Doug Turnbull
View syllabus