Importance: National surgical quality improvement programs lack tools for early detection of quality or safety concerns, which risks patient safety because of delayed recognition of poor performance.
Objective: To compare the risk-adjusted cumulative sum (CUSUM) with episodic evaluation for early detection of hospitals with excess perioperative mortality.
Design, setting, and participants: National, observational, hospital-level, comparative effectiveness study of 697 566 patients. Identification of hospitals with excess, risk-adjusted, quarterly 30-day mortality using observed to expected ratios (ie, current criterion standard in the Veterans Affairs Surgical Quality Improvement Program) was compared with the risk-adjusted CUSUM. Patients included in the study underwent a noncardiac operation at a Veterans Affairs hospital, had a record in the Veterans Affairs Surgical Quality Improvement Program (January 1, 2011, through December 31, 2016), and were aged 18 years or older.
Main outcome and measure: Number of hospitals identified as having excess risk-adjusted 30-day mortality.
Results: The cohort included 697 566 patients treated at 104 hospitals across 24 quarters. The mean (SD) age was 60.9 (13.2) years, 91.4% were male, and 8.6% were female. For each hospital, the median number of quarters detected with observed to expected ratios, at least 1 CUSUM signal, and more than 1 CUSUM signal was 2 quarters (IQR, 1-4 quarters), 8 quarters (IQR, 4-11 quarters), and 3 quarters (IQR, 1-4 quarters), respectively. During 2496 total quarters of data, outlier hospitals were identified 33.3% of the time (830 quarters) with at least 1 CUSUM signal within a quarter, 12.5% (311 quarters) with more than 1 CUSUM signal, and 11.0% (274 quarters) with observed to expected ratios at the end of the quarter. The CUSUM detection occurred a median of 49 days (IQR, 25-63 days) before observed to expected ratio reporting (1 signal, 35 days [IQR, 17-54 days]; 2 signals, 49 days [IQR, 26-61 days]; 3 signals, 58 days [IQR, 44-69 days]; ≥4 signals, 49 days [IQR, 42-69 days]; trend test, P < .001). Of 274 hospital quarters detected with observed to expected ratios, 72.6% (199) were concurrently detected by at least 1 CUSUM signal vs 42.7% (117) by more than 1 CUSUM signal. There was a dose-response relationship between the number of CUSUM signals in a quarter and the median observed to expected ratio (0 signals, 0.63; 1 signal, 1.28; 2 signals, 1.58; 3 signals, 2.08; ≥4 signals, 2.49; trend test, P < .001).
Conclusions: This study found that with CUSUM, hospitals with excess perioperative mortality can be identified well in advance of standard end-of-quarter reporting, which suggests episodic evaluation strategies fail to detect out-of-control processes and place patients at risk. Continuous performance evaluation tools should be adopted in national quality improvement programs to prevent avoidable patient harm.