How these explanations are checked
No language model writes factual content here. Explanations are rendered deterministically from typed facts pulled from MyGene, ClinVar, dbSNP and gnomAD, and every sentence carries the ids of the source fields it was built from. That design is only worth anything if the citations are actually checked — so they are, over the whole validation matrix, and the results are published below including the parts that miss.
Claim provenance
Every claim that ships is re-audited from scratch against the evidence list the pipeline actually fetched: do all the fact ids it cites resolve, and would the validator still accept the sentence? Measured across 24 gene and variant cases and 928 evidence facts on 2026-08-31.
244
claims audited
Every sentence rendered across the matrix, not a sample.
0
ungrounded claims shipped
A claim citing a fact id that does not resolve, or that the validator would reject. This is the number that matters; anything above zero fails the build.
0
claims authored by a language model
Structurally zero since the model was removed from the factual path — measured rather than asserted, because that is the claim a reader is most entitled to see checked.
1 of 245
claims the gate removed
Sentences the renderer produced that did not survive validation. Non-zero is the gate working: it fails closed, dropping a sentence rather than showing an unsupported one.
What was dropped, and by which rule
- 1×uncurated_identity
A metric that had to be retired
An earlier harness measured a “grounded rate”: how often the language model’s structured output passed the validation gate instead of falling back to sources alone. When the model was removed from the factual path, that harness kept running and kept reporting a grounded rate of zero — on a system behaving exactly as designed. The number was not wrong so much as answering a question the product no longer asks.
It was replaced rather than quietly kept, because a stale metric that still renders is worse than no metric: it looks like evidence. The provenance audit above asks the question that survives the architecture change — is every shipped sentence bound to a source field that really exists?
Retrieval
Search over the corpus is hybrid — BM25 lexical scoring fused with local embeddings by reciprocal rank fusion. It is evaluated separately from explanation quality, on a golden set of 20 plain-language questions where the correct record is known in advance. Measured 2026-08-30.
0.950
Recall@10
Share of questions whose correct record appears in the top results.
0.775
MRR
Mean reciprocal rank — rewards putting the right record first, not merely somewhere.
5%
retrieval failure rate
Questions where the correct record was never retrieved at all.
What it currently misses
- “why some adults can digest milk without discomfort” — expected rs4988235, not retrieved.
Listed rather than trimmed from the golden set. A phrasing that shares no vocabulary with the record it should match is the honest weak point of lexical-plus-embedding retrieval at this corpus size.
Reproduce
Both harnesses run against live sources with no API key and no model call. The published artifact is committed, so the number on this page and the number in the repository are the same file.
npm run measure:provenance # audits every shipped claim against its cited evidence npm run measure:retrieval # scores the golden question set npm run measure:publish # writes src/data/validation/report.json