RareRAG Bench: A Small, Honest Benchmark for Evidence-Grounded RAG Under Rare-Disease Evidence Scarcity

Status: v1 technical report. This is a working paper describing a project-scale (not publication-scale) benchmark, written to accompany the rare-rag-bench repository. Every number below traces to a row in research/claims_registry.md and a real artifact in results/. Where a finding is preliminary or not statistically supported, that is stated explicitly rather than omitted.

Abstract

Retrieval-augmented generation (RAG) systems are increasingly proposed for biomedical question answering, but most existing benchmarks evaluate them on well-studied conditions with abundant literature. Rare diseases present a harder and more consequential case: evidence is sparse, scattered across heterogeneous sources, and a system that cannot find it should say so rather than answer anyway. We introduce RareRAG Bench, a small benchmark (35 rare diseases, 832 frozen PubMed abstracts, 45 hand-curated tasks) that evaluates retrieval, hybrid and reranked retrieval, and full RAG generation with claim-level citation grading and calibrated abstention, explicitly stratified by a corpus-derived rarity proxy. Our first benchmark pass (20 diseases, 23 tasks) produced an honest null result: no retrieval-method comparison survived a bootstrap significance check. Rather than overclaim from that result, we expanded the benchmark (35 diseases, 45 tasks) specifically to test whether more data — not a different method — was the fix. It was: two comparisons that were statistically indistinguishable from noise at the smaller scale (hybrid vs. BM25, and reranking vs. no reranking, both on Recall@10) became real, CI-backed findings at the larger scale. We also find that a RAG system's self-reported per-claim confidence is a usable abstention signal (risk- coverage accuracy rises monotonically from 0.868 to 1.000 as the confidence threshold tightens), while its ability to detect fine-grained evidence absence — "this disease has some relevant literature, but not about this specific product" — is markedly weaker than its ability to detect topic-level absence. We report these findings, our benchmark construction methodology, and our limitations in full, including a methodological finding about LLM generation non-determinism that we believe is broadly relevant to any small-n LLM-based benchmark.

1. Introduction

The central research problem, unchanged from project inception: how reliably can modern retrieval and RAG systems locate, combine, and cite sparse evidence for rare-disease questions, particularly when relevant evidence is distributed across heterogeneous sources? We treat "insufficient evidence in the indexed corpus" as a legitimate, gradeable answer, not a failure mode to be engineered away.

This paper reports on a project built at genuinely small scale — tens of diseases, tens of tasks — by design and by resource constraint, not as a simplification of a larger effort. We treat the small scale as a feature worth being honest about, not a limitation to obscure: our headline methodological result is precisely about what happens (and what doesn't survive) at this scale, and what changes when the scale grows.

RareRAG Bench is not a medical chatbot, diagnostic tool, or patient-facing system. See §14 (Safety) below and research/safety.md for the constraints this project operates under.

2. Related Work

We do not claim to be the first rare-disease RAG benchmark. A literature scan (research/literature_matrix.csv, 27 papers, search-verified where marked) surfaced two works close enough that they must be cited and differentiated explicitly, not passed over:

• GraphRareBench (arXiv:2607.24878) is an auditable, evidence-traceable rare-disease benchmark. It overlaps us on evidence-provenance requirements but evaluates phenotype-driven diagnosis ranking (HPO query → candidate disease pool), not literature-grounded generative RAG, and does not grade citation support of free-text claims.
• "One Score, Two Decisions: Selective Prediction on the Rare-Disease Tail" (arXiv:2608.14683) stratifies explicitly by disease prevalence, including "ultra-rare," and evaluates selective prediction/risk-coverage — directly overlapping our abstention axis. It evaluates diagnostic ranking systems (Exomiser-style phenotype matching), not a RAG pipeline answering literature questions with claim-level citation grading.

Our narrower, defensible contribution is the combination: a rare-disease literature corpus (not phenotype/diagnosis data), explicit stratification by evidence/corpus scarcity, a full retrieve→generate RAG pipeline, and claim-level citation-support grading tied to calibrated abstention. Individual pieces of this combination have close precedent — BioASQ and MIRAGE for the biomedical RAG-benchmark shape, PopQA and MINTQA for the rarity-stratification convention (which we adapt from popularity terciles to document-count terciles), and AttributionBench/ALCE/AutoAIS for the NLI-style citation-grading methodology we build on. Full detail and the novelty reasoning are in research/novelty_memo.md.

3. Corpus

The v1 frozen corpus contains 832 real PubMed abstracts (metadata + abstract text via NCBI E-utilities) spanning 35 rare diseases, content-hashed and versioned (data/corpus/MANIFEST.json). Disease identifiers are MONDO IDs, resolved programmatically from the actual MONDO ontology file (rarerag/ontology/obo_lookup.py) rather than hand-typed — this caught two real near-miss errors during construction (a legacy synonym cross-link, and a hyphenation mismatch between a disease's general MONDO node and a narrower subtype node; both documented in research/implementation_plan.md).

Source licensing was verified per-source before inclusion (research/data_provenance.md): PubMed metadata/abstracts, MONDO, HPO, Gene Ontology, ClinVar, and ClinicalTrials.gov v2 are used; OMIM is excluded (its terms bar third-party redistribution without a Johns Hopkins license); PMC full text is deferred (its bulk-retrieval channel was mid-transition at construction time).

Rarity stratum is a corpus-coverage proxy, not a claim about true biological prevalence: diseases are binned into high/medium/sparse terciles by their own document count within this corpus (11/12/12 diseases respectively in v1). We are explicit that this measures publication-count-in-this-corpus, confounded with publication bias, not disease severity or true prevalence (research/limitations.md).

4. Benchmark Construction

45 tasks across 6 types: phenotype retrieval (A, n=10), gene association (B, n=10), gene→disease inverse framing (C, n=4), mechanism literature (D, n=10), differential retrieval between similar diseases (E, n=1), and insufficient-evidence questions (H, n=10). Task types F (evidence for/against), G (clinical-trial retrieval), and I (multi-hop) from the original specification were not built in v1 — an honest scope gap, not a hidden one.

Every gold document reference was manually verified against the frozen corpus (enforced by a repo test, tests/test_tasks_manifest.py, that fails the build if any gold reference doesn't exist). Every H-task's claimed evidence-absence was verified by full-text grep of the relevant disease's corpus subset before the task was written, not assumed — and two H-tasks (H-009, H-010) were deliberately constructed as a harder test (same disease has some relevant evidence, but not for this specific product), with the predicted failure mode written into the task file's ambiguity_notes before the pipeline was ever run against them (see §11).

5. Retrieval

Three baselines: BM25 (lexical), dense (sentence-transformers/all- MiniLM-L6-v2), and hybrid (reciprocal rank fusion of the two). On the 35 gold-bearing v1 tasks:

A cross-encoder reranker (cross-encoder/ms-marco-MiniLM-L-6-v2) applied to the hybrid first-stage pool (size 20, ceiling Recall@20=0.890) raises Recall@10 to 0.848, MRR to 0.613, and Success@5 to 0.886.

Statistical significance. Descriptive differences are not treated as findings until bootstrap-tested (paired resampling over tasks, rarerag/eval/bootstrap.py). At the initial v0 scale (n=18 gold tasks), no comparison survived — every 95% CI included zero. This was reported as the headline result of that phase, not hidden. At v1 scale (n=35), two comparisons became real:

• Hybrid beats BM25 on Recall@10: diff=+0.090, 95% CI=[+0.007, +0.186] (excludes zero).
• Reranking beats pre-rerank hybrid on Recall@10: diff=+0.114, 95% CI=[+0.048, +0.190] (excludes zero).

Hybrid vs. dense, and any comparison on nDCG@10/MRR, remain not distinguishable from noise even at n=35 — we do not claim these are resolved, only that Recall@10 comparisons specifically now have power.

6. RAG Generation and Citation Grading

Pipeline: hybrid retrieval → cross-encoder rerank (top 6) → LLM generation of a claim-decomposed answer (each claim carries its own cited evidence doc IDs and a self-reported confidence) → automated citation grading. Grading follows an NLI-style methodology (adapted from AttributionBench/ALCE/AutoAIS): for each cited claim, an LLM judge scores whether the passage supports, contradicts, or is silent on the claim, with citation-existence and citation-was-actually-retrieved checked mechanically first (no LLM call needed for those two checks).

Grader validation was performed, not assumed. A first-pass grader prompt achieved only 33% manual-audit precision (2/6) on its "contradicts" label — it was conflating "passage doesn't mention this detail" with "passage actively contradicts the claim." A revised prompt raised precision to 80% (4/5) on the same audited cases, preserving the one genuine contradiction in the set. Full audit trail: research/grader_validation_log.md.

v1-scale results (45 tasks, real LLM calls, total cost across all generation+grading runs this session: $0.032): claim-level support rate 0.896 (n=125 claims across 38 answered tasks: 112 supported, 9 contradicted, 4 unverifiable). This is consistent with — slightly better than — the v0-scale figure (0.877, n=57 claims), suggesting the earlier number was not a small-n fluke.

A methodological finding about non-determinism. Two full pipeline runs at v0 scale, differing only in generation/grader prompt text (not retrieval), produced opposite-direction correctness flips on two individual tasks. This means single-run per-task correctness at this project's scale is not a fully stable measurement, and any future architecture comparison should run multiple trials or fix temperature/seed rather than trust one run's per-task verdicts as ground truth. We consider this a genuine, reportable finding in its own right — not just a caveat — since it is broadly relevant to any small-n LLM-based benchmark, not specific to rare-disease RAG.

A pipeline engineering finding. A batch of 45+ sequential external API calls is long enough that a single transient failure (one HTTP 400 from the upstream gateway) can occur and, if unhandled, destroy an entire run's completed work. We hit this for real (a 28-task-deep run was lost) and fixed it with retry-with-backoff and per-task checkpointing. We report this because it is a real, generalizable lesson for anyone building a similar pipeline, not because it is a research result.

7. Long-Tail (Rarity-Stratified) Analysis

Dense retrieval Recall@10 by rarity stratum, v1 scale: high=0.854 (n=12), medium=0.750 (n=4), sparse=0.719 (n=19) — monotonically declining with rarity, consistent with the project's central hypothesis and consistent in direction with the earlier v0-scale observation. We do not claim this is statistically confirmed: stratum sizes are small and uneven (4-19), and no bootstrap CI has been computed per-stratum. This remains the least-resolved of our central research questions (RQ7) and is the clearest candidate for a future benchmark-expansion round, specifically targeting the medium stratum's low task count.

8. Abstention

We evaluate whether a RAG system's own reported confidence is a usable signal for deciding when to abstain, via a risk-coverage curve: at each confidence threshold, what fraction of tasks does the system answer (coverage), and of those, what fraction have zero unsupported/ contradicted claims (selective accuracy)?

At v1 scale (n=45): selective accuracy rises monotonically from 0.868 (threshold 0.0-0.8, 38/45 answered) to 0.923 (26/45) to 0.952 (21/45) to 1.000 (15/45, threshold=1.0). This is not a guaranteed shape — a poorly calibrated model could show a flat or non-monotonic curve — and it replicated cleanly from the v0-scale run (which showed the same monotonic pattern on a smaller n), which we take as reasonable evidence the finding is real rather than a small-n artifact.

By contrast, raw first-stage retrieval score (BM25 top-hit score) is not a usable abstention signal on its own: H-task top scores and gold-task top scores occupy fully overlapping ranges in our data. Model self-reported confidence at generation time is a meaningfully better abstention signal than retrieval score at retrieval time, on this benchmark.

A sharper, designed-in-advance finding on abstention granularity. Overall H-task (should-abstain) correctness was modest: 5/10 at v1 scale. But the v1 task set specifically isolated why: three H-tasks tested clean topic-level absence (a real, recently-approved-or-trialed drug verified entirely absent from the relevant disease's corpus subset) — the system correctly abstained on all three. Two H-tasks tested a harder distinction (the disease has some gene-therapy literature in the corpus, but not about this specific named product) — the system failed both, answering instead of abstaining. This was the predicted failure mode, written into the task files before the pipeline was run against them. The system's insufficient-evidence detection appears to operate at the topic level, not the entity/product level — a more precise and more actionable characterization than a single aggregate abstention rate.

9. Failure Taxonomy

Full detail with task/document IDs in research/failure_taxonomy.md. We highlight three categories with quantified, real examples:

• Citation mismatch: a generation-time bug where the model cited bracket-index numbers instead of real document IDs (task A-004, round 1), silently making all of that task's claims ungradeable. Found via manual inspection, fixed, and kept in the taxonomy as a documented example of a real failure mode, not removed once fixed.
• Grader-side silence-vs-contradiction confusion: detailed in §6 above — a citation grader systematically over-applying "contradicts."
• Same-topic-different-product abstention failure: detailed in §8 above — the sharpest, most actionable finding of the v1 expansion.

We did not build a single bar chart comparing all taxonomy category counts, since their denominators are not comparable (some are rates over 5 H-tasks, some over per-citation counts) — presenting them side-by-side in one chart would visually imply a comparability that isn't real.

10. Limitations

Full detail in research/limitations.md; summarized here:

• Scale. 35 diseases, 45 tasks is proof-of-concept scale. All comparisons are reported with bootstrap CIs specifically because point estimates at this n are not trustworthy on their own — as demonstrated concretely by the v0→v1 null-to-significant transition (§5).
• Gold-evidence completeness. "Gold documents" mean known relevant evidence identified during curation, not an exhaustively verified complete set. Absence from the gold set does not certify irrelevance.
• Disease selection is chosen for rarity-proxy spread and ontology-ID availability, not randomly sampled from all rare diseases — long-tail findings generalize only as far as this selection does.
• Publication-count-as-rarity-proxy reflects corpus coverage, not biological prevalence or severity, and is itself subject to publication bias (research/limitations.md, research/failure_ taxonomy.md §46 discussion in the original spec).
• Generation non-determinism (§6 above): single-run per-task correctness is not a stable measurement at this scale.
• Citation grader reliability: validated once (33%→80% precision after a fix) on a small manually-audited subset — one residual failure case remains, and the grader should not be treated as ground truth without expanding that validation.
• Task type coverage gaps: types F, G, and I from the original specification were not built.
• No clinical validity claim. Nothing here should be read as a statement about performance in an actual clinical-research workflow or any patient-facing context.

11. Safety

RareRAG Bench is a retrieval/RAG research benchmark, not a medical device, diagnostic tool, or patient-care system, and is never presented as one. Every UI surface displays: "FOR RESEARCH AND INFORMATION- RETRIEVAL EVALUATION ONLY. NOT A MEDICAL DEVICE. NOT FOR DIAGNOSIS, TREATMENT, OR PATIENT CARE." No patient data is used — only public PubMed literature and public ontologies. No API keys are exposed client-side. Full policy: research/safety.md.

12. Conclusion

At the scale we could build in this session, the most defensible claim is methodological rather than substantive: small-n LLM/RAG benchmarks should expect null results, should report them honestly rather than overclaim from descriptive differences, and expanding the benchmark is a real, testable fix — which we demonstrated directly by watching two null comparisons become significant after a targeted, motivated expansion. Within that scale, we also produced two genuinely positive, replicated findings (self-reported confidence is a usable abstention signal; reranking and hybrid-over-BM25 retrieval both help, with statistical support) and one genuinely actionable weakness (fine-grained, product-level evidence-absence detection is markedly worse than topic-level detection). Future work should prioritize further benchmark expansion — particularly the medium-rarity stratum, task types F/G/I, and multi-run trials to address the non-determinism finding — before treating any of RQ1-RQ8 as conclusively answered.

All results, code, and data provenance are in the accompanying repository. See docs/HANDOFF.md for the full project state and research/claims_registry.md for the complete, itemized evidentiary basis of every claim in this paper.