The Promise and Limits of Pure Vector Search

Semantic search powered by text embeddings has revolutionized information retrieval. By converting queries and documents into high-dimensional vectors, embedding models capture semantic meaning in ways that traditional keyword search never could. A query for "car" can match documents about "automobiles" and "vehicles" without explicit synonyms. Contextual nuances are preserved, and conceptual similarity drives relevance.

Yet as document collections scale from thousands to millions of items, a troubling pattern emerges: pure embedding-based search begins to fail in subtle but critical ways. The very qualities that make embeddings powerful—their ability to find semantic similarity across broad concepts—become liabilities when precision matters.

The Fundamental Problem: Everything is "Somewhat Similar"

Vector embeddings operate in a continuous space where every document has some degree of similarity to every query. In small collections, this works beautifully. The most relevant documents cluster tightly around the query vector, and irrelevant ones sit far away.

But in large document sets, the distribution changes dramatically:

1. The Curse of Dimensionality

2. Semantic Drift and Topic Bleed

Embeddings excel at capturing broad semantic categories but struggle with specificity. A search for "Python programming exceptions" might return documents about "error handling in Java," "debugging techniques," and "software testing best practices"—all semantically related but not what the user wanted. The model cannot distinguish between "related topics" and "exact matches."

3. The Named Entity Problem

Embeddings are notoriously poor at handling proper nouns, product names, technical identifiers, and domain-specific terminology. A query for "Apache Kafka" might match documents about "Apache Spark," "message queues," or "stream processing frameworks." The word "Kafka" gets diluted into its semantic neighborhood, losing its identity as a specific technology.

4. Multi-Field Embedding Collapse

Even when documents are indexed with embeddings from multiple fields (title, description, content), the fundamental issue persists. Averaging or concatenating field embeddings creates a "semantic soup" where precise terms from high-priority fields get averaged away by bulk content. A document with "machine learning" in the title and 10,000 words about database administration might rank equally with a focused machine learning tutorial.

Real-World Failure Modes

Consider these common scenarios where pure vector search breaks down:

The Hybrid Solution: Semantic Understanding Meets Lexical Precision

The answer is not to abandon embeddings but to combine them with traditional search techniques in a complementary architecture. This is the approach taken by production systems like SANDI-Solr, which implements a sophisticated hybrid search strategy:

The SANDI-Solr Approach

SANDI-Solr exemplifies this hybrid philosophy by integrating:

The result is a search system that combines the semantic understanding of embeddings with the precision of traditional search, delivering both recall and relevance at scale.

Conclusion: Embeddings are Necessary but Not Sufficient

Pure embedding-based semantic search represents a powerful tool in the information retrieval toolkit, but it is not a complete solution for large-scale search systems. The limitations become apparent at scale: loss of precision, difficulty with named entities, and semantic drift.

The future of search is hybrid: leveraging embeddings for semantic understanding while using traditional IR techniques, entity extraction, and lexical filtering to maintain precision. By combining the strengths of both approaches, systems can deliver the "best of both worlds"—finding documents that are semantically relevant while respecting the specific, precise requirements embedded in user queries.

For production search applications, especially those dealing with technical content, product catalogs, or large knowledge bases, the question is not whether to use embeddings, but how to integrate them intelligently with proven traditional search techniques.

Research: Theoretical Limitations of Embedding-Based Retrieval

Abstract Summary

Vector embeddings have been tasked with an ever-increasing set of retrieval tasks over the years, with a nascent rise in using them for reasoning, instruction-following, coding, and more. These new benchmarks push embeddings to work for any query and any notion of relevance that could be given. While prior works have pointed out theoretical limitations of vector embeddings, there is a common assumption that these difficulties are exclusively due to unrealistic queries, and those that are not can be overcome with better training data and larger models.

Theoretical Framework

The research connects known results in learning theory, showing that the number of top-𝑘 subsets of documents capable of being returned as the result of some query is limited by the dimension of the embedding. The study empirically shows that this holds true even if we restrict to 𝑘 = 2, and directly optimize on the test set with free parameterized embeddings.

The LIMIT Dataset

The researchers created a realistic dataset called LIMIT that stress tests models based on these theoretical results. State-of-the-art models fail on this dataset despite the simple nature of the task.

Implications

The research shows the limits of embedding models under the existing single vector paradigm and calls for future research to develop methods that can resolve this fundamental limitation. The results imply that the community should consider how instruction-based retrieval will impact retrievers, as there will be combinations of top-𝑘 documents that current models cannot represent.

Reference: arXiv:2508.21038v1 - "On the Theoretical Limitations of Embedding-Based Retrieval"
Full paper available at: https://arxiv.org/abs/2508.21038