Doctor Versus Artificial Intelligence: Patient and Physician Evaluation of Large Language Model Responses to Rheumatology Patient Questions in a Cross‐Sectional Study
Authors' conclusion
Rheumatology patients rated AI-generated responses to patient questions similarly to physician-generated responses in terms of comprehensiveness, readability, and overall preference. However, rheumatologists rated AI responses significantly poorer than physician-generated responses, suggesting that LLM chatbot responses are inferior to physician responses, a difference that patients may not be aware of.
Study context
- Population
- Rheumatology patients, Edmonton, Alberta, Canada; responses also rated by four rheumatologistsplus 4 rheumatologists
- Participants
- 17 patients
- Exposure
- Large language model (LLM) chatbot (artificial intelligence (AI)) vs physician-generated responses to patient-generated rheumatology questions
- Outcome
- Quality of responses: comprehensiveness and readability (patients), plus accuracy (rheumatologists), Likert scale 1–10 (1 poor, 10 excellent)
Methodological detailssingle-centre, survey
- Single-centre
- Survey
Main finding
As reported in the abstract
Patient-rated comprehensiveness (Likert 1–10)
AI
7.12
mean · SD 0.99
Physician
7.52
SD 1.16
P = 0.1962
Observational data: associations, not effects.
Further findings
| Outcome | AI | Physician | P value |
|---|---|---|---|
| Patient-rated readability (Likert 1–10)No significant difference | 7.90 (SD 0.90) | 7.80 (SD 0.75) | P = 0.5905 |
| Rheumatologist-rated comprehensiveness (Likert 1–10)AI significantly poorer | 5.52 (SD 2.13) | 8.76 (SD 1.07) | P < 0.0001 |
| Rheumatologist-rated readability (Likert 1–10)AI significantly poorer | 7.85 (SD 0.92) | 8.75 (SD 0.57) | P = 0.0003 |
| Rheumatologist-rated accuracy (Likert 1–10)AI significantly poorer | 6.48 (SD 2.07) | 9.08 (SD 0.64) | P < 0.0001 |
| Proportion of preference for AI vs physician responses, by rater | P = 0.0106 | ||
| Proportion correctly identifying AI-generated answers, by rater · after learning one answer per question was AI generatedPatients vs Rheumatologists · patients lower | P = 0.0183 | ||
| Average word count of answers | average 69.10 (SD 25.35) words | 98.83 (SD 34.58) words | P = 0.0008 |
- Proportion of preference for AI vs physician responses, by rater: Patients 0.45 (SD 0.18)
- Proportion of preference for AI vs physician responses, by rater: Rheumatologists 0.15 (SD 0.08)
- Proportion correctly identifying AI-generated answers, by rater: Patients 0.49 (SD 0.26)
- Proportion correctly identifying AI-generated answers, by rater: Rheumatologists 0.97 (SD 0.04)
Original publication
Doctor Versus Artificial Intelligence: Patient and Physician Evaluation of Large Language Model Responses to Rheumatology Patient Questions in a Cross‐Sectional Study. Arthritis Rheumatol. Published 18 January 2024. doi:10.
Composed by AI from the published abstract and checked against it. How Briefs are made →