BEIR Benchmark Results: TREC-COVID and NFCorpus
Updated system — November 2026
Both datasets are part of the BEIR benchmark [1], which tests zero-shot retrieval: the models are not trained on the test collections. Both are biomedical, but they differ in almost every property that matters for search evaluation.
TREC-COVID [2] is built from CORD-19 (COVID-19 Open Research Dataset), a collection of scientific articles about COVID-19 and related coronaviruses; BEIR uses the title and abstract of each article. Its 50 topics were written during the pandemic and are phrased as full questions, for example "what is the origin of COVID-19". Relevance was judged by assessors with biomedical expertise on a graded scale, over pools of documents returned by the systems that took part in the original TREC challenge.
NFCorpus (NutritionFacts Corpus) [3] is built from NutritionFacts.org, a health and nutrition website, and the medical articles it cites. Queries are short and written in everyday language, while the documents are scientific abstracts in medical terminology. Relevance is graded and is derived from the links between the website pages and the cited articles.
| Property | TREC-COVID | NFCorpus |
|---|---|---|
| Source | CORD-19 article titles and abstracts | NutritionFacts.org and cited medical articles |
| Domain | Biomedical — epidemiology, treatment, transmission | Biomedical — nutrition, diet, disease prevention |
| Corpus size | 171,332 documents | 3,633 documents |
| Test queries | 50 | 323 |
| Query style | Full questions, about 11 words on average | Short everyday phrases, about 3 words on average |
| Relevant documents per query | About 494 on average | About 38 on average, varying widely by query |
| Relevance labels | Graded, judged by experts over pooled results | Graded, derived from links to cited articles |
| Main difficulty | Ranking the best documents first in a large corpus | Vocabulary gap between everyday queries and medical text |
Both benchmarks were run on SANDI Solr ver.0.6.4-0040 with the same models and settings. Each benchmark was run twice, without and with the reranker.
| Component | Setting |
|---|---|
| Index | Apache Solr 9.8.1 with SolrCloud (ZooKeeper) |
| Embedding model | Qwen3-Embedding-0.6B [4], not quantized (BF16), native 1024 dimensions |
| Vector search | top_k = 100 |
| NLP query analysis | Extraction of entities and important words from the query |
| Synonym expansion | None |
| Precision setting | Low |
| Reranker | Qwen3-Reranker-4B [4], GGUF Q4 quantization, top 30 candidates (when used) |
NLP is used at query time to extract the entities and the important words of the query, which focus the text-matching part of the search. No synonyms are added, so the system works on the original wording of the query. The embedding model runs without quantization (BF16) at its native 1024 dimensions; the reranker is 4-bit quantized (GGUF Q4), which keeps its memory and compute low.
| Metric | No reranker | With reranker — top 30 | Change |
|---|---|---|---|
| NDCG@5 | 0.8981 | 0.9259 | +0.0278 |
| NDCG@10 | 0.8823 | 0.9084 | +0.0261 |
| Precision@10 | 0.9260 | 0.9500 | +0.0240 |
| MRR@10 | 0.9800 | 0.9800 | 0.0000 |
| Recall@10 | 0.0241 | 0.0245 | +0.0004 |
| Recall@100 | 0.1749 | 0.1749 | 0.0000 |
| MAP@10 | 0.0230 | 0.0237 | +0.0007 |
| Metric | No reranker | With reranker — top 30 | Change |
|---|---|---|---|
| NDCG@5 | 0.4136 | 0.4426 | +0.0290 |
| NDCG@10 | 0.3696 | 0.4036 | +0.0340 |
| Precision@10 | 0.2650 | 0.2920 | +0.0270 |
| MRR@10 | 0.5787 | 0.6183 | +0.0396 |
| Recall@10 | 0.1736 | 0.1939 | +0.0203 |
| Recall@100 | 0.3120 | 0.3120 | 0.0000 |
| MAP@10 | 0.1429 | 0.1587 | +0.0158 |
Without any reranker the system reaches NDCG@10 = 0.8823 and Precision@10 = 0.9260: on average 9.3 of the first 10 results are relevant, in a corpus of 171,332 documents. MRR@10 = 0.9800 means that the first result is relevant for at least 48 of the 50 queries.
Reranking the top 30 candidates with Qwen3-Reranker-4B raises NDCG@5 by 2.8 points to 0.9259, NDCG@10 by 2.6 points to 0.9084 and Precision@10 by 2.4 points to 0.9500. MRR@10 stays at 0.9800.
Recall@100 is 0.1749 in both runs. The reranker only reorders the first 30 results, so the set of the first 100 does not change.
Recall@10 = 0.0245 and MAP@10 = 0.0237 look small, but with about 500 relevant documents per query ten results cannot cover more than about 2% of them. They do not indicate a weakness of the system; NDCG@10, Precision@10 and MRR@10 are the metrics that describe quality on this dataset.
On NFCorpus the same system reaches NDCG@10 = 0.3696 without a reranker and NDCG@10 = 0.4036 with it. With the reranker, MRR@10 = 0.6183 means that the first relevant result is typically in the first or the second position, and Precision@10 = 0.2920 is about 3 relevant results in the first 10, on a dataset where many queries have few relevant documents to find.
Recall is informative here: with the reranker, 19.4% of all relevant documents are in the first 10 results and 31.2% in the first 100. Unlike TREC-COVID, MAP@10 = 0.1587 reflects real ranking quality, because the relevant set is small enough for ten results to cover a meaningful part of it.
On NFCorpus the reranker raises NDCG@5 by 2.9 points (0.4136 to 0.4426), NDCG@10 by 3.4 points, MAP@10 by 1.6 points and Recall@10 from 0.1736 to 0.1939. The largest gain is in MRR@10, by 4.0 points from 0.5787 to 0.6183: the first relevant result moves up. Recall@100 is 0.3120 in both runs, for the same reason as on TREC-COVID.
NDCG@10 of 0.9084 on TREC-COVID and 0.4036 on NFCorpus come from the same models and the same settings. The difference is the test, not the system: TREC-COVID rewards putting some of many relevant documents first, with queries that share the vocabulary of the documents; NFCorpus asks to find a few relevant documents from short everyday queries that often share few words with them. Each score should be read against other systems on the same dataset (Section 5).
Both benchmarks were run with no synonyms. The vocabulary bridging on NFCorpus and the results on TREC-COVID come from the embedding model, the NLP extraction of entities and important words, and the reranker.
The same benchmarks were also run with Qwen3-Embedding-4B (GGUF Q4, truncated to 1024 dimensions) on an earlier version of SANDI Solr. The larger model gives better results on NFCorpus: with the reranker it reaches NDCG@10 = 0.4104, against 0.4036 with the 0.6B model. It is more expensive to run: it is slower and requires more memory.
The tables place the new results next to NDCG@10 figures published for other systems on the same datasets.
| System | NDCG@10 | Setup and source |
|---|---|---|
| BGE-M3 (dense) | 0.5472 | Retrieval only, MTEB results [15] |
| BM25 | 0.6559 | Retrieval only, BEIR paper [1] |
| gte-large | 0.7022 | Retrieval only, MTEB results [15] |
| SPLADE-v2 | 0.7057 | Retrieval only [5] |
| BGE-large-en-v1.5 | 0.7470 | Retrieval only [7], MTEB results [15] |
| jina-embeddings-v5-text-nano | 0.7760 | Retrieval only, MTEB results [15] |
| jina-embeddings-v3 | 0.7781 | Retrieval only [13] |
| jina-embeddings-v5-text-small | 0.7849 | Retrieval only, MTEB results [15] |
| ColBERT v2 | 0.7854 | Retrieval only [6] |
| KaLM-Embedding-V2 (mini-instruct-v2) | 0.7927 | Retrieval only, MTEB results [15] |
| monoT5-3B | 0.8071 | Reranks BM25 top 100 [16] |
| Cohere Rerank-v2 | 0.8181 | Reranks BM25 top 100 [16] |
| Hybrid RRF of SPLADE and BGE-base | 0.8282 | Rank fusion, no reranker [10] |
| KaLM-Embedding-V2.5 (mini-instruct-v2.5) | 0.8298 | Retrieval only, MTEB results [15] |
| ListT5-3B | 0.8470 | Reranks BM25 top 100 [16] |
| ResRank (sliding window) | 0.8500 | Reranks BM25 top 100 [11] |
| mxbai-rerank-large | 0.8533 | Reranks BM25 top 100 [16] |
| RankQwen3-8B | 0.8537 | Reranks BM25 top 100 [12] |
| gte-Qwen2-1.5B-instruct | 0.8538 | Retrieval only, MTEB results [15] |
| jina-reranker-v3 | 0.8659 | Reranks jina-embeddings-v3 top 100 [13] |
| Qwen3-Reranker-4B | 0.8708 | Reranks jina-embeddings-v3 top 100 [13] |
| SANDI Solr — no reranker | 0.8823 | Retrieval only, Qwen3-Embedding-0.6B BF16, 1024 dimensions, ver.0.6.4-0040 |
| Qwen3-Embedding-0.6B | 0.8864 | Retrieval only [14] |
| Qwen3-Reranker-4B | 0.8875 | Reranks Qwen3-Embedding-0.6B top 100 [14] |
| DeBERTa-v3 cross-encoder | 0.8920 | Reranks SPLADE-v3 top 200, full documents [9] |
| Qwen3-Reranker-8B | 0.8966 | Reranks Qwen3-Embedding-0.6B top 100 [14] |
| Querit-Reranker-4B | 0.8982 | Reranks Qwen3-Embedding-0.6B top 100 [14] |
| SANDI Solr — with Qwen3-Reranker-4B Q4 | 0.9084 | Reranks Qwen3-Embedding-0.6B BF16, 1024 dimensions, top 30 candidates, ver.0.6.4-0040 |
| System | NDCG@10 | Setup and source |
|---|---|---|
| BGE-M3 (dense) | 0.3141 | Retrieval only, MTEB results [15] |
| BM25 | 0.3255 | Retrieval only, BEIR paper [1] |
| SPLADE-v2 | 0.3382 | Retrieval only [5] |
| ColBERT v2 | 0.3480 | Retrieval only [6] |
| KaLM-Embedding-V2 (mini-instruct-v2) | 0.3517 | Retrieval only, MTEB results [15] |
| Qwen3-Embedding-0.6B | 0.3634 | Retrieval only [14] |
| Cohere Rerank-v2 | 0.3636 | Reranks BM25 top 100 [16] |
| jina-embeddings-v3 | 0.3665 | Retrieval only [13] |
| SANDI Solr — no reranker | 0.3696 | Retrieval only, Qwen3-Embedding-0.6B BF16, 1024 dimensions, ver.0.6.4-0040 |
| mxbai-rerank-large | 0.3708 | Reranks BM25 top 100 [16] |
| KaLM-Embedding-V2.5 (mini-instruct-v2.5) | 0.3711 | Retrieval only, MTEB results [15] |
| ListT5-3B | 0.3770 | Reranks BM25 top 100 [16] |
| BGE-large-en-v1.5 | 0.3806 | Retrieval only [7], MTEB results [15] |
| gte-large | 0.3817 | Retrieval only, MTEB results [15] |
| jina-embeddings-v5-text-nano | 0.3869 | Retrieval only, MTEB results [15] |
| jina-reranker-v3 | 0.3892 | Reranks jina-embeddings-v3 top 100 [13] |
| monoT5-3B | 0.3897 | Reranks BM25 top 100 [16] |
| gte-Qwen2-1.5B-instruct | 0.3934 | Retrieval only, MTEB results [15] |
| jina-embeddings-v5-text-small | 0.3981 | Retrieval only, MTEB results [15] |
| ResRank (sliding window) | 0.3994 | Reranks BM25 top 100 [11] |
| RankQwen3-8B | 0.4005 | Reranks BM25 top 100 [12] |
| SANDI Solr — with Qwen3-Reranker-4B Q4 | 0.4036 | Reranks Qwen3-Embedding-0.6B BF16, 1024 dimensions, top 30 candidates, ver.0.6.4-0040 |
| Qwen3-Reranker-4B | 0.4135 | Reranks Qwen3-Embedding-0.6B top 100 [14] |
| Qwen3-Reranker-4B | 0.4156 | Reranks jina-embeddings-v3 top 100 [13] |
| Qwen3-Reranker-8B | 0.4279 | Reranks Qwen3-Embedding-0.6B top 100 [14] |
| Querit-Reranker-4B | 0.4335 | Reranks Qwen3-Embedding-0.6B top 100 [14] |
Most of the published results use uncompressed models at full precision, at a higher cost in GPU memory and compute and with slower indexing and search.
SANDI Solr is a compromise that trades some of this precision for search speed. Embeddings are stored as BYTE vectors (1 KB per 1024-dimension vector, about 1 GB per million chunks), which reduces index size and speeds up vector scoring. GGUF compressed models allow the whole system, including the Qwen3.8-27B LLM (used for query analysis and answers, not in these benchmark runs), to run on a single NVIDIA RTX 3090.
Reranking 100 candidates, as in most of the published reranker results, is costly: a cross-encoder runs the model once per query-candidate pair, and the cost grows linearly with the number of candidates. On an NVIDIA A100 with 100 TREC-COVID candidates, Qwen3-Reranker-4B takes 1.39 s per query, Qwen3-Reranker-8B 2.32 s and the listwise RankQwen3-8B 16.93 s [12]. GPT-4 reranking of 50 documents can take up to a minute [9] at about $0.60 per query [8].
SANDI Solr reranks the top 30 candidates, a fraction of the cost of reranking 100 full documents.
Each corpus was indexed with document titles and texts, and every test query was sent to the SANDI Solr search API. Results were scored against the standard BEIR test relevance judgments with the ranx library, using graded relevance for NDCG.
Install libraries: pip install ranx requests
Run Python code for TREC-COVID:
https://softcorporation.com/products/sandi/evaluate-beir-trec-covid.txt
Run Python code for NFCorpus:
https://softcorporation.com/products/sandi/evaluate-beir-nfcorpus.txt
The updated SANDI Solr system was tested on two BEIR benchmarks with one set of settings: top_k = 100, NLP extraction of entities and important words from the query, no synonyms, Low precision. With Qwen3-Embedding-0.6B (BF16, 1024 dimensions), ver.0.6.4-0040, on TREC-COVID it reaches NDCG@10 = 0.8823 and MRR@10 = 0.9800 with no reranker, and NDCG@10 = 0.9084 and Precision@10 = 0.9500 with Qwen3-Reranker-4B quantized Q4 over the top 30 candidates. On NFCorpus it reaches NDCG@10 = 0.3696 with no reranker, and NDCG@10 = 0.4036, MRR@10 = 0.6183 and Recall@100 = 0.3120 with the reranker.
The two benchmarks test different abilities. TREC-COVID, with a large corpus, 50 long queries and hundreds of relevant documents per query, measures how well the best documents are ranked first; its recall and MAP figures are low by construction. NFCorpus, with a small corpus, 323 short queries and few relevant documents per query, measures how well the system bridges the gap between everyday wording and medical text; its recall and MAP figures are meaningful, and its ceiling for top-10 metrics is lower. Reading each result within its own dataset, SANDI Solr is strong on both.