Clinical Note Audit

Abstract

Clinical-note evaluation pipelines can confuse a usable response with a correct judgment. We present Clinical Note Audit, a tool that combines an offline evaluation audit with live access to the original evaluator models. Response validity, evidence conditions and reference agreement remain separately inspectable. Users can filter a recorded evaluation, inspect unchanged notes and responses, diagnose the first failed structural check, compare paired evidence conditions, and validate a pasted response against the frozen contract. A packaged evaluation contains 1,530 first attempts on 85 MedVAL mock consultations, with 17 development and 68 held consultations. The application reports coverage, fallback frequency, pipeline Brier and valid-response-only Brier with their denominators. Its replay preserves the study predictions and reference labels. A contract lab checks serialization and quotation, without certifying medical meaning. A post hoc comparison applies four validation policies to every first response. Whole-object probability extraction lowers the held macro pipeline Brier by 0.0078 relative to the frozen contract, demonstrating that validation choices can change the reported score without establishing better medical reasoning. The additional source-access comparison retains all 1,020 fresh first responses. Grammar constraints increase held contract coverage by 0.1740 [0.1422, 0.2034] while pipeline Brier changes by +0.0152 [−0.0022, +0.0329] (exploratory pointwise 95% intervals). All 24 groups, 21 contrasts and failure categories are available above. A prospective interface-comparison pilot is prepared; no participant results are claimed. The tool supports measurement audits of clinical NLP evaluators.

Keep three questions separate.

Did the response satisfy its output contract? Which evidence was available to the model? How well did a returned probability agree with the fixed reference target? The interface keeps response coverage, pipeline loss and valid-response-only loss visible together.

q̃ = q for valid outputs; 0.5 otherwise
Coverage = valid responses / scheduled responses
Pipeline Brier = mean (q̃ − y)²

The archive contains mock consultations. Its physician target is risk grade 1 versus grades 2–4. A verbatim quotation passes a structural check; it does not certify that a medical verdict is supported.

Inspect the recorded evaluation.

Filter 1,530 first attempts by model, output contract, split and evidence condition. Inspect the source, note and unchanged response; exercise the contract lab and export the exact active analysis.

This archive view replays recorded inference. The separate live-model page sends new inputs to the H200 and runs the original language model.

From the page to the code.

The download includes this website, numerical samples, research reference code, provenance and executable checks. The examples are usable without access to a hospital or private account.

unzip clinical-note-audit.zip
cd research-artifact
python -m http.server 8766 --directory dist
# Open http://localhost:8766

Download the complete installation package · Inspect provenance

Saved results are computational evidence. This artifact does not establish independent clinical validation, medical suitability or acceptance of a paper.