Clinical Note Audit

Abstract

Clinical-note evaluation pipelines can confuse a usable response with a correct judgment. We present Clinical Note Audit, a tool that combines an offline evaluation audit with live access to the original evaluator models. Response validity, evidence conditions and reference agreement remain separately inspectable. Users can filter a recorded evaluation, inspect unchanged notes and responses, diagnose the first failed structural check, compare paired evidence conditions, and validate a pasted response against the frozen contract. A packaged evaluation contains 1,530 first attempts on 85 MedVAL mock consultations, with 17 development and 68 held consultations. The application reports coverage, fallback frequency, pipeline Brier and valid-response-only Brier with their denominators. Its replay preserves the study predictions and reference labels. A contract lab checks serialization and quotation, without certifying medical meaning. The tool supports measurement audits of clinical NLP evaluators; it does not establish the evaluators' clinical reliability.

Keep three questions separate.

Did the response satisfy its output contract? Which evidence was available to the model? How well did a returned probability agree with the fixed reference target? The interface keeps response coverage, pipeline loss and valid-response-only loss visible together.

q̃ = q for valid outputs; 0.5 otherwise
Coverage = valid responses / scheduled responses
Pipeline Brier = mean (q̃ − y)²

The archive contains mock consultations. Its physician target is risk grade 1 versus grades 2–4. A verbatim quotation passes a structural check; it does not certify that a medical verdict is supported.

Inspect the recorded evaluation.

Filter 1,530 first attempts by model, output contract, split and evidence condition. Inspect the source, note and unchanged response; exercise the contract lab and export the exact active analysis.

This archive view replays recorded inference. The separate live-model page sends new inputs to the H200 and runs the original language model.

From the page to the code.

The download includes this website, numerical samples, research reference code, provenance and executable checks. The examples are usable without access to a hospital or private account.

unzip clinical-note-audit.zip
cd clinical-note-audit
python -m http.server 8766 --directory dist
# Browse the static website locally:
python -m http.server 8766 --directory dist

Download the complete installation package · Inspect provenance

Saved results are computational evidence. This artifact does not establish independent clinical validation, medical suitability or acceptance of a paper.