Background: Risk-adjusted mortality rates are commonly used in quality report cards to compare hospital performance. The risk adjustment depends on models that are assessed for goodness-of-fit using various discrimination and calibration measures. However, the relationship between model fit and the accuracy of hospital comparisons is not well characterized.
Objectives: To evaluate the impact of imperfect model calibration (miscalibration) on the accuracy of hospital comparisons.
Methods: We constructed Monte Carlo simulations where a risk-adjustment model is used in a population with a different mortality distribution than in the original model. We estimated the power of calibration metrics to detect miscalibration. We estimated the sensitivity and specificity of a hospital comparisons method under different imperfect model calibration scenarios using an empirical method.
Results: The U-statistics showed the highest power to detect intercept and slope deviations in the calibration curve, followed by the Hosmer-Lemeshow, and the calibration intercept and slope tests. The specificity decreased with increased intercept and slope deviations and with hospital size. The effect of an imperfect model fit on sensitivity is a function of the true standardized mortality ratio, the underlying mortality rate, sample size, and observed intercept and slope deviations. Poorly performing hospitals can appear as good performers and vice versa, depending on the deviation magnitude and direction.
Conclusions: Deviations from perfect model calibration have a direct impact on the accuracy of hospital comparisons. Publishing the calibration intercept and slope of risk-adjustment models would allow the users to monitor their performance against the true standard population.