A research benchmark for RAG over rare-disease literature
RareRAG Bench measures how reliably retrieval and RAG systems find, combine, and cite sparse scientific evidence for rare-disease questions. It is a benchmark, not a product — every question is answered only in terms of measured retrieval/generation behavior on a frozen corpus, never in terms of clinical truth.
What this is — and isn’t
- A frozen-corpus IR/RAG benchmark: retrieval, reranking, generation, and citation-grading measured against real gold evidence.
- An honest measurement instrument — every reported difference is bootstrap-tested, and non-significant results are reported as such, not hidden.
- A first-class abstention benchmark: “insufficient evidence” is a valid, gradeable answer, not a failure mode.
- A medical device, diagnostic tool, or patient-care system.
- A source of personalized diagnosis or treatment advice — no question here asks “what does this patient have.”
- A statement about clinical validity — results describe IR/RAG behavior on one frozen corpus and task set only.
A null result that motivated its own fix
At v0 scale (20 diseases, 18 gold-bearing tasks), every retrieval comparison this project ran — hybrid vs. dense, hybrid vs. BM25, reranking vs. pre-rerank hybrid — produced descriptively promising differences that did not survive a paired bootstrap significance check. Every 95% CI touched zero. Rather than report those numbers as findings, the project treated this as a scale problem and expanded the benchmark to 35 diseases and 45 tasks specifically to address it (Phase 11).
At v1 scale, two of those comparisons crossed into real, CI-backed significance:
Hybrid vs. dense on Recall@10 (diff -0.036, CI [-0.114, 0.038]) and every nDCG/MRR comparison remain not distinguishable from noise even at n=35 — expanding the task set was not a guarantee of significance across the board, only for two of the tested comparisons. See the full leaderboard for every comparison with its actual CI.
Central research questions
Every question is answerable only in terms of measured retrieval/generation behavior — “the system got the right answer” is not a valid framing; “the system retrieved/cited evidence that a grader judged supportive” is.