Free Lesson

Stop picking embedding models off the MTEB leaderboard

Part of AI Product Engineering

Hosted by Radu Gheorghe and Hamel Husain

157 students

In this video

What you'll learn

Choose an embedder for your data and budget

Weigh the traits a leaderboard skips, including vector quantization, model size, and Matryoshka dimensions, so the model fits your latency and cost.

Why binary vectors work for most cases

See when binary vectors are enough, and how multi-stage ranking recovers the relevance that compression costs you.

Fine-tune the embedder on your own data

Use VespaTune, an open source UI, to fine-tune an embedder on your data instead of trusting a benchmark score.

Why this topic matters

It's tempting to pick an embedding model according to which one tops the MTEB leaderboard. But MTEB doesn't tell you how the model will do on your data, where you'll often need to fine-tune it, or how it performs on speed and quality once you change quantization and dimensions. Radu shows how to juggle these variables so you get the most for your money.

You'll learn from

Radu Gheorghe

Software Engineer, Vespa.ai

Radu has been in the search space for many years, mainly on Elasticsearch, Solr, OpenSearch, and, more recently, Vespa.ai. Helps users with both the relevance and the operations side of retrieval. Enjoys education in all its forms (training, blog posts, books, conferences...) and got the chance to be involved in all of them.

Hamel Husain

ML Engineer with 20+ years of experience

Hamel Husain is a ML Engineer with 20+ years of experience. He has worked with innovative companies such as Airbnb and GitHub, which included early LLM research used by OpenAI, for code understanding. He has also led and contributed to numerous popular open-source machine-learning tools. Hamel is currently an independent consultant helping companies build AI products.

See all products from Hamel Husain & Shreya Shankar

Go deeper with a course

Featured in Lenny’s List
AI Evals For Engineers & PMs
Hamel Husain and Shreya Shankar
View syllabus