Large language models for accurate disease detection in electronic health records: the examples of crystal arthropathies
Authors' conclusion
LLMs accurately detected disease diagnoses from EHRs, even in non-English languages.
Study context
- Population
- Paragraphs mentioning 'gout' from randomly selected electronic health record (EHR) documents of a tertiary university hospital in Geneva, Switzerland
- Participants
- 700 paragraphs
- Comparison
- Large language model (LLM) (Llama-3-8B)-based framework vs regex-based method for detecting gout in French-language EHR paragraphs
- Outcome
- Accuracy of detecting true gout vs non-disease (gold standard: manual classification)
Methodological detailssingle-centre, few-shot and chain-of-thought prompting, manual review by two healthcare professionals as gold standard, validation set
- Single-centre
- Few-shot and chain-of-thought prompting
- Manual review by two healthcare professionals as gold standard
- Validation set
Performance
As reported in the abstract
Detection of gout
| Test or model | PPV | NPV | Accuracy |
|---|---|---|---|
| LLM | 92.7% (88.7%-95.4%) | 96.6% (94.6%-97.8%) | 95.4% (93.6%-96.7%) |
LLM
PPV92.7% (88.7%-95.4%)
NPV96.6% (94.6%-97.8%)
Accuracy95.4% (93.6%-96.7%)
Further findings
Accuracy, validation set on CPPD (600 paragraphs)
- Accuracy
- 94.1% (90.2%-97.6%)
Dependence on model parameters
- The LLM framework performed well over a wide range of parameter values
Original publication
Large language models for accurate disease detection in electronic health records: the examples of crystal arthropathies. RMD Open. Published 20 December 2024. doi:10.
Composed by AI from the published abstract and checked against it. How Briefs are made →