Man vs. Machine: Comparing Physician vs. Electronic Health Record-Based Model Predictions for 30-Day Hospital Readmissions

Oanh Kieu Nguyen; Colin Washington; Christopher R Clark; Michael E Miller; Vivek A Patel; Ethan A Halm; Anil N Makam

doi:10.1007/s11606-020-06355-3

Man vs. Machine: Comparing Physician vs. Electronic Health Record-Based Model Predictions for 30-Day Hospital Readmissions

J Gen Intern Med. 2021 Sep;36(9):2555-2562. doi: 10.1007/s11606-020-06355-3. Epub 2021 Jan 14.

Authors

Oanh Kieu Nguyen^{1

2

3}, Colin Washington⁴, Christopher R Clark⁵, Michael E Miller⁶, Vivek A Patel⁴, Ethan A Halm^{4

6}, Anil N Makam^{4

6

7}

Affiliations

¹ Department of Internal Medicine, UT Southwestern, Dallas, TX, USA. Oanh.Nguyen@ucsf.edu.
² Department of Population and Data Sciences, UT Southwestern, Dallas, TX, USA. Oanh.Nguyen@ucsf.edu.
³ Division of Hospital Medicine at San Francisco General Hospital, University of California, San Francisco, San Francisco, CA, USA. Oanh.Nguyen@ucsf.edu.
⁴ Department of Internal Medicine, UT Southwestern, Dallas, TX, USA.
⁵ Department of Research Administration, Parkland Health and Hospital System, Dallas, TX, USA.
⁶ Department of Population and Data Sciences, UT Southwestern, Dallas, TX, USA.
⁷ Division of Hospital Medicine at San Francisco General Hospital, University of California, San Francisco, San Francisco, CA, USA.

Abstract

Background: Electronic health record (EHR)-based readmission risk prediction models can be automated in real-time but have modest discrimination and may be missing important readmission risk factors. Clinician predictions of readmissions may incorporate information unavailable in the EHR, but the comparative usefulness is unknown. We sought to compare clinicians versus a validated EHR-based prediction model in predicting 30-day hospital readmissions.

Methods: We conducted a prospective survey of internal medicine clinicians in an urban safety-net hospital. Clinicians prospectively predicted patients' 30-day readmission risk on 5-point Likert scales, subsequently dichotomized into low- vs. high-risk. We compared human with machine predictions using discrimination, net reclassification, and diagnostic test characteristics. Observed readmissions were ascertained from a regional hospitalization database. We also developed and assessed a "human-plus-machine" logistic regression model incorporating both human and machine predictions.

Results: We included 1183 hospitalizations from 106 clinicians, with a readmission rate of 20.8%. Both clinicians and the EHR model had similar discrimination (C-statistic 0.66 vs. 0.66, p = 0.91). Clinicians had higher specificity (79.0% vs. 48.9%, p < 0.001) but lower sensitivity (43.9 vs. 75.2%, p < 0.001) than EHR model predictions. Compared with machine, human was better at reclassifying non-readmissions (non-event NRI + 30.1%) but worse at reclassifying readmissions (event NRI - 31.3%). A human-plus-machine approach best optimized discrimination (C-statistic 0.70, 95% CI 0.67-0.74), sensitivity (65.5%), and specificity (66.7%).

Conclusion: Clinicians had similar discrimination but higher specificity and lower sensitivity than EHR model predictions. Human-plus-machine was better than either alone. Readmission risk prediction strategies should incorporate clinician assessments to optimize the accuracy of readmission predictions.

Keywords: electronic health records; hospitalization; logistic models; patient readmission; safety-net providers.

Publication types

Research Support, N.I.H., Extramural
Research Support, U.S. Gov't, P.H.S.

MeSH terms

Electronic Health Records
Hospitalization
Humans
Male
Patient Readmission*
Physicians*
Prospective Studies
Risk Factors

Abstract

Publication types

MeSH terms

Grants and funding