SANDI Solr

BEIR Benchmark Results: TREC-COVID and NFCorpus

Updated system — November 2026

Abstract

SANDI Solr — a search system built on Apache Solr that combines dense vector search, text matching, NLP query analysis and an optional cross-encoder reranker — has been updated since the earlier benchmark reports. This document presents the new results of the updated system on two BEIR benchmarks, TREC-COVID and NFCorpus, run with the same models and settings in SANDI Solr ver.0.6.4-0040: Qwen3-Embedding-0.6B (not quantized, BF16, native 1024 dimensions), top_k = 100, NLP extraction of entities and important words from the query, no synonym expansion, Low precision. Each benchmark was run without a reranker and with Qwen3-Reranker-4B (GGUF Q4) over the top 30 candidates. On TREC-COVID the system reaches NDCG@10 = 0.8823 with no reranker and NDCG@10 = 0.9084 with the reranker. On NFCorpus it reaches NDCG@10 = 0.3696 with no reranker and NDCG@10 = 0.4036 with the reranker.

1Datasets: TREC-COVID and NFCorpus

Both datasets are part of the BEIR benchmark [1], which tests zero-shot retrieval: the models are not trained on the test collections. Both are biomedical, but they differ in almost every property that matters for search evaluation.

TREC-COVID

TREC-COVID [2] is built from CORD-19 (COVID-19 Open Research Dataset), a collection of scientific articles about COVID-19 and related coronaviruses; BEIR uses the title and abstract of each article. Its 50 topics were written during the pandemic and are phrased as full questions, for example "what is the origin of COVID-19". Relevance was judged by assessors with biomedical expertise on a graded scale, over pools of documents returned by the systems that took part in the original TREC challenge.

NFCorpus

NFCorpus (NutritionFacts Corpus) [3] is built from NutritionFacts.org, a health and nutrition website, and the medical articles it cites. Queries are short and written in everyday language, while the documents are scientific abstracts in medical terminology. Relevance is graded and is derived from the links between the website pages and the cited articles.

PropertyTREC-COVIDNFCorpus
SourceCORD-19 article titles and abstractsNutritionFacts.org and cited medical articles
DomainBiomedical — epidemiology, treatment, transmissionBiomedical — nutrition, diet, disease prevention
Corpus size171,332 documents3,633 documents
Test queries50323
Query styleFull questions, about 11 words on averageShort everyday phrases, about 3 words on average
Relevant documents per queryAbout 494 on averageAbout 38 on average, varying widely by query
Relevance labelsGraded, judged by experts over pooled resultsGraded, derived from links to cited articles
Main difficultyRanking the best documents first in a large corpusVocabulary gap between everyday queries and medical text
Corpus sizes, query counts, average query lengths and average numbers of relevant documents are the BEIR statistics [1].

How the two differ for search evaluation

2System Configuration

Both benchmarks were run on SANDI Solr ver.0.6.4-0040 with the same models and settings. Each benchmark was run twice, without and with the reranker.

Without reranker

Query
→
NLP: Entities and Important Words
→
Vector + Text Search (top_k = 100)
→
Final Results

With reranker

Query
→
NLP: Entities and Important Words
→
Vector + Text Search (top_k = 100)
→
Top 30 Candidates
→
Qwen3-Reranker-4B
→
Final Results
ComponentSetting
IndexApache Solr 9.8.1 with SolrCloud (ZooKeeper)
Embedding modelQwen3-Embedding-0.6B [4], not quantized (BF16), native 1024 dimensions
Vector searchtop_k = 100
NLP query analysisExtraction of entities and important words from the query
Synonym expansionNone
Precision settingLow
RerankerQwen3-Reranker-4B [4], GGUF Q4 quantization, top 30 candidates (when used)

NLP is used at query time to extract the entities and the important words of the query, which focus the text-matching part of the search. No synonyms are added, so the system works on the original wording of the query. The embedding model runs without quantization (BF16) at its native 1024 dimensions; the reranker is 4-bit quantized (GGUF Q4), which keeps its memory and compute low.

3Results

TREC-COVID with Qwen3-Embedding-0.6B (BF16), ver.0.6.4-0040

Metric No reranker With reranker — top 30 Change
NDCG@50.89810.9259+0.0278
NDCG@100.88230.9084+0.0261
Precision@100.92600.9500+0.0240
MRR@100.98000.98000.0000
Recall@100.02410.0245+0.0004
Recall@1000.17490.17490.0000
MAP@100.02300.0237+0.0007

NFCorpus with Qwen3-Embedding-0.6B (BF16), ver.0.6.4-0040

Metric No reranker With reranker — top 30 Change
NDCG@50.41360.4426+0.0290
NDCG@100.36960.4036+0.0340
Precision@100.26500.2920+0.0270
MRR@100.57870.6183+0.0396
Recall@100.17360.1939+0.0203
Recall@1000.31200.31200.0000
MAP@100.14290.1587+0.0158
Highlighted values mark the better of the two runs in each table. Recall@100 is the same with and without the reranker because the reranker only reorders the first 30 results.

4Findings

TREC-COVID: retrieval alone is already strong

Without any reranker the system reaches NDCG@10 = 0.8823 and Precision@10 = 0.9260: on average 9.3 of the first 10 results are relevant, in a corpus of 171,332 documents. MRR@10 = 0.9800 means that the first result is relevant for at least 48 of the 50 queries.

TREC-COVID: the reranker improves the whole first page

Reranking the top 30 candidates with Qwen3-Reranker-4B raises NDCG@5 by 2.8 points to 0.9259, NDCG@10 by 2.6 points to 0.9084 and Precision@10 by 2.4 points to 0.9500. MRR@10 stays at 0.9800.

Recall@100 is 0.1749 in both runs. The reranker only reorders the first 30 results, so the set of the first 100 does not change.

TREC-COVID: low recall and MAP are properties of the dataset

Recall@10 = 0.0245 and MAP@10 = 0.0237 look small, but with about 500 relevant documents per query ten results cannot cover more than about 2% of them. They do not indicate a weakness of the system; NDCG@10, Precision@10 and MRR@10 are the metrics that describe quality on this dataset.

NFCorpus: lower numbers on a harder matching problem

On NFCorpus the same system reaches NDCG@10 = 0.3696 without a reranker and NDCG@10 = 0.4036 with it. With the reranker, MRR@10 = 0.6183 means that the first relevant result is typically in the first or the second position, and Precision@10 = 0.2920 is about 3 relevant results in the first 10, on a dataset where many queries have few relevant documents to find.

Recall is informative here: with the reranker, 19.4% of all relevant documents are in the first 10 results and 31.2% in the first 100. Unlike TREC-COVID, MAP@10 = 0.1587 reflects real ranking quality, because the relevant set is small enough for ten results to cover a meaningful part of it.

NFCorpus: the reranker raises every ranking metric

On NFCorpus the reranker raises NDCG@5 by 2.9 points (0.4136 to 0.4426), NDCG@10 by 3.4 points, MAP@10 by 1.6 points and Recall@10 from 0.1736 to 0.1939. The largest gain is in MRR@10, by 4.0 points from 0.5787 to 0.6183: the first relevant result moves up. Recall@100 is 0.3120 in both runs, for the same reason as on TREC-COVID.

Scores on the two datasets are not comparable

NDCG@10 of 0.9084 on TREC-COVID and 0.4036 on NFCorpus come from the same models and the same settings. The difference is the test, not the system: TREC-COVID rewards putting some of many relevant documents first, with queries that share the vocabulary of the documents; NFCorpus asks to find a few relevant documents from short everyday queries that often share few words with them. Each score should be read against other systems on the same dataset (Section 5).

No synonym expansion was needed

Both benchmarks were run with no synonyms. The vocabulary bridging on NFCorpus and the results on TREC-COVID come from the embedding model, the NLP extraction of entities and important words, and the reranker.

Larger embedding model

The same benchmarks were also run with Qwen3-Embedding-4B (GGUF Q4, truncated to 1024 dimensions) on an earlier version of SANDI Solr. The larger model gives better results on NFCorpus: with the reranker it reaches NDCG@10 = 0.4104, against 0.4036 with the 0.6B model. It is more expensive to run: it is slower and requires more memory.

5Published Reference Points

The tables place the new results next to NDCG@10 figures published for other systems on the same datasets.

TREC-COVID

SystemNDCG@10Setup and source
BGE-M3 (dense)0.5472Retrieval only, MTEB results [15]
BM250.6559Retrieval only, BEIR paper [1]
gte-large0.7022Retrieval only, MTEB results [15]
SPLADE-v20.7057Retrieval only [5]
BGE-large-en-v1.50.7470Retrieval only [7], MTEB results [15]
jina-embeddings-v5-text-nano0.7760Retrieval only, MTEB results [15]
jina-embeddings-v30.7781Retrieval only [13]
jina-embeddings-v5-text-small0.7849Retrieval only, MTEB results [15]
ColBERT v20.7854Retrieval only [6]
KaLM-Embedding-V2 (mini-instruct-v2)0.7927Retrieval only, MTEB results [15]
monoT5-3B0.8071Reranks BM25 top 100 [16]
Cohere Rerank-v20.8181Reranks BM25 top 100 [16]
Hybrid RRF of SPLADE and BGE-base0.8282Rank fusion, no reranker [10]
KaLM-Embedding-V2.5 (mini-instruct-v2.5)0.8298Retrieval only, MTEB results [15]
ListT5-3B0.8470Reranks BM25 top 100 [16]
ResRank (sliding window)0.8500Reranks BM25 top 100 [11]
mxbai-rerank-large0.8533Reranks BM25 top 100 [16]
RankQwen3-8B0.8537Reranks BM25 top 100 [12]
gte-Qwen2-1.5B-instruct0.8538Retrieval only, MTEB results [15]
jina-reranker-v30.8659Reranks jina-embeddings-v3 top 100 [13]
Qwen3-Reranker-4B0.8708Reranks jina-embeddings-v3 top 100 [13]
SANDI Solr — no reranker0.8823Retrieval only, Qwen3-Embedding-0.6B BF16, 1024 dimensions, ver.0.6.4-0040
Qwen3-Embedding-0.6B0.8864Retrieval only [14]
Qwen3-Reranker-4B0.8875Reranks Qwen3-Embedding-0.6B top 100 [14]
DeBERTa-v3 cross-encoder0.8920Reranks SPLADE-v3 top 200, full documents [9]
Qwen3-Reranker-8B0.8966Reranks Qwen3-Embedding-0.6B top 100 [14]
Querit-Reranker-4B0.8982Reranks Qwen3-Embedding-0.6B top 100 [14]
SANDI Solr — with Qwen3-Reranker-4B Q40.9084Reranks Qwen3-Embedding-0.6B BF16, 1024 dimensions, top 30 candidates, ver.0.6.4-0040

NFCorpus

SystemNDCG@10Setup and source
BGE-M3 (dense)0.3141Retrieval only, MTEB results [15]
BM250.3255Retrieval only, BEIR paper [1]
SPLADE-v20.3382Retrieval only [5]
ColBERT v20.3480Retrieval only [6]
KaLM-Embedding-V2 (mini-instruct-v2)0.3517Retrieval only, MTEB results [15]
Qwen3-Embedding-0.6B0.3634Retrieval only [14]
Cohere Rerank-v20.3636Reranks BM25 top 100 [16]
jina-embeddings-v30.3665Retrieval only [13]
SANDI Solr — no reranker0.3696Retrieval only, Qwen3-Embedding-0.6B BF16, 1024 dimensions, ver.0.6.4-0040
mxbai-rerank-large0.3708Reranks BM25 top 100 [16]
KaLM-Embedding-V2.5 (mini-instruct-v2.5)0.3711Retrieval only, MTEB results [15]
ListT5-3B0.3770Reranks BM25 top 100 [16]
BGE-large-en-v1.50.3806Retrieval only [7], MTEB results [15]
gte-large0.3817Retrieval only, MTEB results [15]
jina-embeddings-v5-text-nano0.3869Retrieval only, MTEB results [15]
jina-reranker-v30.3892Reranks jina-embeddings-v3 top 100 [13]
monoT5-3B0.3897Reranks BM25 top 100 [16]
gte-Qwen2-1.5B-instruct0.3934Retrieval only, MTEB results [15]
jina-embeddings-v5-text-small0.3981Retrieval only, MTEB results [15]
ResRank (sliding window)0.3994Reranks BM25 top 100 [11]
RankQwen3-8B0.4005Reranks BM25 top 100 [12]
SANDI Solr — with Qwen3-Reranker-4B Q40.4036Reranks Qwen3-Embedding-0.6B BF16, 1024 dimensions, top 30 candidates, ver.0.6.4-0040
Qwen3-Reranker-4B0.4135Reranks Qwen3-Embedding-0.6B top 100 [14]
Qwen3-Reranker-4B0.4156Reranks jina-embeddings-v3 top 100 [13]
Qwen3-Reranker-8B0.4279Reranks Qwen3-Embedding-0.6B top 100 [14]
Querit-Reranker-4B0.4335Reranks Qwen3-Embedding-0.6B top 100 [14]
Reference figures are taken from the cited papers and from the MTEB results repository. The systems selected for comparison use models of up to 8B parameters and embeddings of at most 2048 dimensions, twice the 4B parameters of the SANDI Solr reranker and the 1024 dimensions of its embeddings. MTEB scores are submitted by the model authors, and not every listed model is zero-shot on these datasets. The lists are not exhaustive.

6Notes on the Comparison

Most of the published results use uncompressed models at full precision, at a higher cost in GPU memory and compute and with slower indexing and search.

SANDI Solr is a compromise that trades some of this precision for search speed. Embeddings are stored as BYTE vectors (1 KB per 1024-dimension vector, about 1 GB per million chunks), which reduces index size and speeds up vector scoring. GGUF compressed models allow the whole system, including the Qwen3.8-27B LLM (used for query analysis and answers, not in these benchmark runs), to run on a single NVIDIA RTX 3090.

Reranking 100 candidates, as in most of the published reranker results, is costly: a cross-encoder runs the model once per query-candidate pair, and the cost grows linearly with the number of candidates. On an NVIDIA A100 with 100 TREC-COVID candidates, Qwen3-Reranker-4B takes 1.39 s per query, Qwen3-Reranker-8B 2.32 s and the listwise RankQwen3-8B 16.93 s [12]. GPT-4 reranking of 50 documents can take up to a minute [9] at about $0.60 per query [8].

SANDI Solr reranks the top 30 candidates, a fraction of the cost of reranking 100 full documents.

7Evaluation Method

Each corpus was indexed with document titles and texts, and every test query was sent to the SANDI Solr search API. Results were scored against the standard BEIR test relevance judgments with the ranx library, using graded relevance for NDCG.

Code to reproduce test results

Install libraries: pip install ranx requests
Run Python code for TREC-COVID: https://softcorporation.com/products/sandi/evaluate-beir-trec-covid.txt
Run Python code for NFCorpus: https://softcorporation.com/products/sandi/evaluate-beir-nfcorpus.txt

8Conclusion

The updated SANDI Solr system was tested on two BEIR benchmarks with one set of settings: top_k = 100, NLP extraction of entities and important words from the query, no synonyms, Low precision. With Qwen3-Embedding-0.6B (BF16, 1024 dimensions), ver.0.6.4-0040, on TREC-COVID it reaches NDCG@10 = 0.8823 and MRR@10 = 0.9800 with no reranker, and NDCG@10 = 0.9084 and Precision@10 = 0.9500 with Qwen3-Reranker-4B quantized Q4 over the top 30 candidates. On NFCorpus it reaches NDCG@10 = 0.3696 with no reranker, and NDCG@10 = 0.4036, MRR@10 = 0.6183 and Recall@100 = 0.3120 with the reranker.

The two benchmarks test different abilities. TREC-COVID, with a large corpus, 50 long queries and hundreds of relevant documents per query, measures how well the best documents are ranked first; its recall and MAP figures are low by construction. NFCorpus, with a small corpus, 323 short queries and few relevant documents per query, measures how well the system bridges the gap between everyday wording and medical text; its recall and MAP figures are meaningful, and its ceiling for top-10 metrics is lower. Reading each result within its own dataset, SANDI Solr is strong on both.

Apache Solr 9.8.1 Qwen3-Embedding-0.6B Qwen3-Reranker-4B GGUF Q4 NLP Query Processing BEIR Benchmark TREC-COVID NDCG@10: 0.9084 NFCorpus NDCG@10: 0.4036

9References

  1. Thakur N., et al. (2021) BEIR: A Heterogeneous Benchmark for Zero-shot Evaluation of Information Retrieval Models. NeurIPS 2021 Datasets and Benchmarks Track. arXiv:2104.08663. https://arxiv.org/abs/2104.08663
  2. Voorhees E., et al. (2020) TREC-COVID: Constructing a Pandemic Information Retrieval Test Collection. SIGIR Forum. arXiv:2005.04474. https://arxiv.org/abs/2005.04474
  3. Boteva V., et al. (2016) A Full-Text Learning to Rank Dataset for Medical Information Retrieval. ECIR. https://www.cl.uni-heidelberg.de/statnlpgroup/nfcorpus/
  4. Zhang Y., et al. (2025) Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Models. Tongyi Lab, Alibaba Group. arXiv:2506.05176. https://arxiv.org/abs/2506.05176
  5. Formal T., et al. (2021) SPLADE: Sparse Lexical and Expansion Model for First Stage Ranking. SIGIR. arXiv:2107.05720. https://arxiv.org/abs/2107.05720
  6. Santhanam K., et al. (2022) ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interaction. NAACL. arXiv:2112.01488. https://arxiv.org/abs/2112.01488
  7. Xiao S., et al. (2023) C-Pack: Packaged Resources To Advance General Chinese Embedding. SIGIR. arXiv:2309.07597. https://arxiv.org/abs/2309.07597
  8. Sun W., et al. (2023) Is ChatGPT Good at Search? Investigating Large Language Models as Re-Ranking Agents. EMNLP. arXiv:2304.09542. https://arxiv.org/abs/2304.09542
  9. Déjean H., Clinchant S., Formal T. (2024) A Thorough Comparison of Cross-Encoders and LLMs for Reranking SPLADE. Naver Labs Europe. arXiv:2403.10407. https://arxiv.org/abs/2403.10407
  10. Prajapati H. K., Atilla D. G., Sac B. (2026) Hybrid Retrieval for COVID-19 Literature: Comparing Rank Fusion and Projection Fusion with Diversity Reranking. arXiv:2604.13728. https://arxiv.org/abs/2604.13728
  11. Ke X., Zhang S., et al. (2026) ResRank: Unifying Retrieval and Listwise Reranking via End-to-End Joint Training with Residual Passage Compression. Alibaba. arXiv:2604.22180. https://arxiv.org/abs/2604.22180
  12. Liu Q., Zhang Y., Li M., Long D., Xie P., Mao J. (2025) E2Rank: Your Text Embedding can Also be an Effective and Efficient Listwise Reranker. arXiv:2510.22733. https://arxiv.org/abs/2510.22733
  13. Wang F., Li Y., Xiao H. (2025) jina-reranker-v3: Last but Not Late Interaction for Listwise Document Reranking. Jina AI. arXiv:2509.25085. https://arxiv.org/abs/2509.25085
  14. Zhong Y., Yang J., et al. (2026) Querit-Reranker: Training Compact Multilingual Rerankers via Efficient Label-Free Distribution Adaptation. arXiv:2606.19037. https://arxiv.org/abs/2606.19037
  15. MTEB official results repository, TRECCOVID and NFCorpus task results (accessed 14 October 2026). https://github.com/embeddings-benchmark/results
  16. Abdallah A., Piryani B., Mozafari J., Ali M., Jatowt A. (2025) How Good are LLM-based Rerankers? An Empirical Analysis of State-of-the-Art Reranking Models. Findings of EMNLP 2025. arXiv:2508.16757. https://arxiv.org/abs/2508.16757