Evidence & citations
Claim-level citation grading: for every atomic claim in a generated answer, does the cited document exist, get retrieved, and actually support (or contradict) the claim?
Citation support breakdown
Overall support rate: 89.6% (n=125 graded claims across all 45 tasks, total generation+grading cost $0.015966).
Example claims
Real contradicted and unverifiable claims from the v1 run, not illustrative fabrications — each links to its full task detail.
Classic Fabry disease affects the neurological system.
Classic Fabry disease affects the cutaneous system.
The enzyme homogentisate 1,2-dioxygenase is deficient in alkaptonuria.
Behavioral disorders such as hyperactivity, aggression, and self-injury are reported in Angelman syndrome.
Abstention & selective prediction
Abstention (“insufficient evidence”) is a first-class, gradeable answer in this benchmark, not a failure mode. RQ6 asks whether the model’s self-reported claim confidence is a usable signal for deciding when to abstain.
Type H (should-abstain) correctness is the weak point at 5/10 (50%). The v1 task set sharpens why: the model distinguishes topic-level absence well (3/3 correct when a whole drug topic is absent from the corpus) but not product-level absence within a covered topic (0/2 on “this disease has some gene-therapy evidence, but not this specific named product”) — see the failure taxonomy for the concrete task IDs.
| Confidence threshold | Coverage | Selective accuracy | n answered |
|---|---|---|---|
| 0.00 | 84.4% | 86.8% | 38/45 |
| 0.30 | 84.4% | 86.8% | 38/45 |
| 0.50 | 84.4% | 86.8% | 38/45 |
| 0.60 | 84.4% | 86.8% | 38/45 |
| 0.70 | 84.4% | 86.8% | 38/45 |
| 0.80 | 84.4% | 86.8% | 38/45 |
| 0.90 | 57.8% | 92.3% | 26/45 |
| 0.95 | 46.7% | 95.2% | 21/45 |
| 1.00 | 33.3% | 100.0% | 15/45 |
Selective accuracy rises monotonically as the threshold tightens — at threshold 1.0 (only the model’s maximum-confidence claims), accuracy reaches 100% at the cost of answering only 15/45 tasks. Replicated cleanly at 2x scale from the v0 run (claims_registry.md C-023).