About

Methodology

A reader-friendly summary of research/benchmark_spec.md — task types, metrics, corpus construction, and chunking. This benchmark's numbers scope: v1 is 35 diseases, 832 documents, 45 tasks.

Corpus construction

Disease identifiers (MONDO ID, cross-referenced to Orphanet where available) are resolved programmatically via an ontology lookup tool, never hand-typed. Each disease has a synonym table pulled from MONDO’s own synonym/xref fields — query normalization always resolves through this table before hitting the retriever.

Each document record carries: doc_id, source, external_id (PMID), title, passages (abstract at minimum), publication date, authors, disease/gene entity links, source type, license, and retrieval timestamp. The corpus is frozen at a single retrieval timestamp and content-hashed for release reproducibility — no document is silently added, removed, or edited after a version is tagged.

v1 uses PubMed abstracts only (metadata + abstract text via NCBI E-utilities), capped around 24 documents per disease, giving 832 documents across 35 diseases. PMC full text was deferred (an FTP retirement transition was in flight at data-collection time), and OMIM was excluded entirely (its terms bar redistribution).

Rarity strata

A reproducible proxy computed from the frozen corpus itself, not an external prevalence claim: doc_count is the number of corpus documents tagged with a disease’s MONDO ID after synonym expansion. Diseases are split into terciles — high (top third by doc_count), medium (middle third), sparse (bottom third, the long tail this benchmark is designed to probe).

Task types

v1 task-type distribution (45 tasks total):

A: 10B: 10D: 10H: 10C: 4E: 1

Code	Task	Gold shape
A	Disease → phenotype retrieval	Gold doc/passage set citing a phenotype-disease link
B	Disease → gene association retrieval	Gold docs supporting a gene-disease link
C	Gene → disease evidence	Inverse of B
D	Disease → mechanism literature	Gold docs describing pathophysiology
E	Differential retrieval between similar rare diseases	Gold docs distinguishing disease X from confusable disease Y
H	Insufficient-evidence questions	Gold answer is abstention; constructed from genuinely uncovered gaps in the frozen corpus, not invented gaps

Type H construction rule: an insufficient-evidence question is only valid if a curator confirms, at corpus-freeze time, that the frozen corpus genuinely contains no supporting passage — verified by full-text grep, not a lazy search — documented in each task’s ambiguity_notes.

Retrieval pipeline stages

1. BM25 (lexical)
2. Dense (biomedical sentence embeddings)
3. Hybrid (reciprocal rank fusion of BM25 + dense)
4. + Cross-encoder reranking (optional stage)

Chunking variants

Chunking is treated as an independent variable in the spec, not fixed in advance:

Abstract-level (one chunk = full abstract)
Fixed-token windows (e.g. 256 tokens, with overlap)
Sentence-group chunks
Section-aware chunks (where full text with section markers is available)

Metrics

Retrieval

Recall@k, Precision@k, MRR, nDCG@k, Success@k. Recall@k is the headline metric for sparse strata by design.

Generation / citation

Support rate, contradiction-acknowledgment rate, unsupported-claim rate, selective accuracy at a given abstention rate.

Statistics

Paired bootstrap over questions for all comparative claims, reported as 95% CI — never a bare point estimate.

Claim-level answer schema

Every generated answer is decomposed into atomic claims, each with its own evidence and grading:

{
  answer_text,
  claims: [
    {
      claim_text,
      evidence: [doc_id...],
      confidence,       // model-reported, not benchmark ground truth
      support_status,   // supported | unsupported | contradicted | unverified
    }
  ]
}