Every embedding model vendor ships an MTEB leaderboard screenshot, and almost none of that ranking predicts how the model performs on your documents. MTEB averages across dozens of unrelated tasks and domains; a model that's great at clustering news articles can be mediocre at retrieving your internal API docs. The only benchmark that matters is one built from your own queries and your own corpus.

Build a golden set before you compare anything

Pull 100-300 real user queries — from support tickets, search logs, or a domain expert writing plausible ones — and manually label which document(s) should be retrieved for each. This is tedious and nobody wants to do it, which is exactly why skipping it is the most common mistake in embedding selection. Without it, you're choosing based on vibes and marketing copy.

python
golden_set = [
    {"query": "how do I rotate API keys", "relevant_doc_ids": ["docs/auth/rotation.md"]},
    {"query": "rate limit for webhook endpoint", "relevant_doc_ids": ["docs/webhooks.md#limits"]},
]

def recall_at_k(model, golden_set, k=5):
    hits = 0
    for item in golden_set:
        results = model.search(item["query"], top_k=k)
        if any(r.id in item["relevant_doc_ids"] for r in results):
            hits += 1
    return hits / len(golden_set)

Dimensions that actually differentiate models

  • Domain fit: general-purpose embeddings underperform on code, legal, or medical text compared to domain-tuned variants. If your corpus is specialized, test at least one domain model even if its general benchmark scores look worse.
  • Context length: some embedding models truncate silently past a token limit, which means long chunks get scored on their first paragraph only. Check the truncation behavior, not just the advertised max length.
  • Dimensionality vs. storage cost: a 3072-dim embedding isn't automatically better than a 768-dim one for your task, and it triples your vector storage and index build time. Many newer models support Matryoshka-style truncation — you can cut the vector to 256 dims and lose surprisingly little recall, which is worth testing before committing to the full size.
  • Asymmetric vs. symmetric retrieval: search queries and documents are different in structure (short question vs. long passage). Models trained for asymmetric retrieval (separate query/passage encoders or instruction prefixes) usually beat symmetric ones for search use cases.

Hosted API vs. self-hosted

Factor Hosted API Self-hosted
Latency Network round-trip per batch Local, lower and more predictable
Cost at scale Per-token, adds up on re-embedding Fixed infra cost, better at high volume
Model control Vendor decides updates/deprecation You pin the exact checkpoint
Ops burden None GPU capacity planning, batching, monitoring

If you re-embed your corpus frequently — because documents update often or you're iterating on chunking — the API cost compounds fast. Teams doing more than a few million embeddings a month usually find self-hosting a mid-sized open model pays for itself within a quarter.

Re-run the eval on every model swap

Embedding models are not drop-in replacements for each other, even within the same vendor's lineup. A new version can shift the vector space enough that your reranker's calibration or your similarity thresholds stop making sense. Treat swapping the embedding model as a migration: re-embed the full corpus, re-run the golden-set eval, and compare recall@k side by side before cutting traffic over. Never run two embedding models against the same index — cosine similarity across incompatible vector spaces is meaningless, and you'll get retrieval results that look plausible but are actually noise.