Performance of risk models to predict mortality risk for patients with heart failure: evaluation in an integrated health system

Abstract

Background: Referral of patients with heart failure (HF) who are at high mortality risk for specialist evaluation is recommended. Yet, most tools for identifying such patients are difficult to implement in electronic health record (EHR) systems.

Objective: To assess the performance and ease of implementation of Machine learning Assessment of RisK and EaRly mortality in Heart Failure (MARKER-HF), a machine-learning model that uses structured data that is readily available in the EHR, and compare it with two commonly used risk scores: the Seattle Heart Failure Model (SHFM) and Meta-Analysis Global Group in Chronic (MAGGIC) Heart Failure Risk Score.

Design: Retrospective, cohort study.

Participants: Data from 6764 adults with HF were abstracted from EHRs at a large integrated health system from 1/1/10 to 12/31/19.

Main measures: One-year survival from time of first cardiology or primary care visit was estimated using MARKER-HF, SHFM, and MAGGIC. Discrimination was measured by the area under the receiver operating curve (AUC). Calibration was assessed graphically.

Key results: Compared to MARKER-HF, both SHFM and MAGGIC required a considerably larger amount of data engineering and imputation to generate risk score estimates. MARKER-HF, SHFM, and MAGGIC exhibited similar discriminations with AUCs of 0.70 (0.69-0.73), 0.71 (0.69-0.72), and 0.71 (95% CI 0.70-0.73), respectively. All three scores showed good calibration across the full risk spectrum.

Conclusions: These findings suggest that MARKER-HF, which uses readily available clinical and lab measurements in the EHR and required less imputation and data engineering than SHFM and MAGGIC, is an easier tool to identify high-risk patients in ambulatory clinics who could benefit from referral to a HF specialist.

Keywords: Heart failure; Machine learning; Outcomes; Risk prediction model.