Failure taxonomy

MONDO's own ontology file carries “fibrodysplasia ossificans progressiva” as a legacy synonym on the unrelated “myositis ossificans” node (MONDO:0003964), even though FOP has its own dedicated node (MONDO:0007606) — discovered while resolving the disease list (rarerag/ontology/obo_lookup.py). A naive synonym-lookup entity linker would conflate the two. The E-001 benchmark task exists specifically to probe this.

Not yet observed as a retrieval failure. The synonym expansion (scripts/build_corpus.py's build_query) uses up to 4 synonyms ≥ 6 characters per disease; no case has yet been found where a synonym miss caused a retrieval failure, but the corpus was built with expansion already applied — this category needs a dedicated ablation (expansion on vs. off) to actually test, which has not been run.

The gold documents for “how does CAG expansion in HTT lead to neurodegeneration” describe the mechanism at a fairly high level; a fully mechanistic answer would need documents outside the ~24-doc Huntington disease sample — the corpus itself, not the retriever, is the limiting factor here.

The hybrid first-stage recall ceiling at pool size 20 is 0.889 (claims_registry.md C-011) — roughly 11% of gold evidence is never retrieved by hybrid at all, so no reranker downstream can recover it. This is a retrieval-miss propagating forward, not a reranker defect per se.

Not yet cleanly isolated — see “citation mismatch” for the closest observed relative. A pure case of the model synthesizing a claim with no traceable evidence basis at all (as opposed to citing the wrong identifier for real evidence) hasn't been isolated yet in this benchmark's runs.

Round 1 pipeline: the model generated 5 claims about Fabry disease's affected organ systems, all citing bracket-index numbers (“1”, “3”, “4”, “6”) from the prompt's [i] doc_id=… display format instead of the actual doc_id strings — making every claim “unverifiable” to the grader even though the underlying content was plausible. Root-caused and fixed (C-014) with an explicit “don't cite the bracket index” instruction in the generation prompt. Kept in this taxonomy as a real, reproduced example even after the fix.

Asked specifically about migalastat (Fabry disease) and zilganersen (Alexander disease) — both verified absent from the corpus by grep before the tasks were written — the model did not abstain. Instead it answered a nearby-but-different question (general disease treatment) without flagging that the specific drug asked about had no evidence in the retrieved passages.

Correctly caught by the (fixed) grader, not a model failure in the grading sense: E-001's claim that FOP “differentiates from” myositis ossificans is contradicted by pubmed:36798559, which literally states FOP “is also called … myositis ossificans” — a real naming-convention conflict within the corpus itself. The generation model asserted a clean differentiation without acknowledging this real source of disagreement in its own cited evidence.

Not yet tested — this benchmark has no temporal-cutoff tasks yet (deferred; no ClinicalTrials.gov data in v1).

v0 round 1: 2/5 H-tasks (H-003, H-004) failed to abstain (“overgeneralization”, same failure filed under its other name). Also 1/5 D-tasks (D-002) abstained when it should have answered — a false-positive abstention, the opposite error direction, equally real. At v1 scale (45 tasks), overall H-task correctness held roughly flat at 5/10, but a sharper sub-pattern emerged: two v1 H-tasks (H-009, H-010) deliberately testing “this disease has SOME gene-therapy evidence, but not THIS specific named product” both failed — the model answered instead of abstaining. In contrast, all three v1 H-tasks testing a clean topic absence (H-006, H-007, H-008 — real, recently-approved-or-trialed drugs verified absent from the corpus) correctly triggered abstention. This suggests the model's insufficient-evidence detection works at the topic level but not the fine-grained product/entity level.

“Silence mistaken for contradiction”: the automated citation grader (round 1 prompt) marked a citation “contradicts” whenever the cited passage simply didn't mention the specific fact in the claim, rather than reserving “contradicts” for passages making an actively conflicting statement. Manually audited precision on the “contradicts” label was 33% (2/6) before a prompt fix, 80% (4/5) after. This is a property of the grading pipeline, not the generation pipeline — a benchmark's correctness depends on its grader being accurate too.