Objective: Prognostication is central to developing treatment plans and relaying information to patients, family members, and other health care providers. The degree of confidence or certainty that a health care provider has in his or her mortality risk assessment is also important, because a provider may deliver care differently depending on their assuredness in the assessment. We assessed the performance of nurse and physician mortality risk estimates with and without weighting the estimates with their respective degrees of certainty.
Methods: Subjective mortality risk estimates from critical care attendings (n = 5), critical care fellows (n = 9), pediatric residents (n = 34), and nurses (n = 52) were prospectively collected on at least 94% of 642 eligible, consecutive admissions to a tertiary pediatric intensive care unit (PICU). A measure of certainty (continuous scale from 0 to 5) accompanied each mortality estimate. Estimates were evaluated with 2 x 2 outcome probabilities, the kappa statistic, the area under the receiver operating characteristics curve, and the Hosmer and Lemeshow goodness-of-fit chi(2) statistic. The estimates were then reevaluated after weighting predictions by their respective degree of certainty.
Results: Overall, there was a significant difference in the predictive accuracy between groups. The mean mortality predictions from the attendings (6.09%) more closely approximated the true mortality rate (36 deaths, 5.61%) whereas fellows (7.87%), residents (10.00%), and nurses (16.29%) overestimated the mean overall PICU mortality. Attendings were more certain of their predictions (4.27) than the fellows (4.01), nurses (3.79), and residents (3.75). All groups discriminated well (area under receiver operating characteristics curve range, 0.86-0.93). Only PICU attendings and fellows did not significantly differ from ideal calibration (chi(2)). When mortality predictions were weighted with their respective certainties, their performance improved.
Conclusions: The level of medical training correlated with the provider's ability to predict mortality risk. The higher the level of certainty associated with the mortality prediction, the more accurate the prediction; however, high levels of certainty did not guarantee accurate predictions. Measures of certainty should be considered when assessing the performance of mortality risk estimates or other subjective outcome predictions.