A Novel Application of the Modified Angoff Method to Rate Case Difficulty in Simulation-Based Research

Abstract

Introduction: Simulation use in research is often limited by controlling for scenario difficulty when using repeated measures. Our study assesses the feasibility of the Modified Angoff Method to reach expert consensus regarding difficulty of medical simulations. We compared scores with participant physiologic stress.

Methods: Emergency medicine physicians with expertise in simulation education were asked to review 8 scenarios and estimate the percentage of resident physicians who would perform all critical actions using the modified Angoff method. A standard deviation (SD) of less than 10% of estimated percentage correct signified consensus. Twenty-five residents then performed the 6 scenarios that met consensus and heart rate variability (HRV) was measured.

Results: During round 1, experts rated 4/8 scenarios within a 10% SD for postgraduate year 3 (PGY3) and 3/8 for PGY4 residents. In round 2, 6/8 simulation scenarios were within an SD of 10% points for both years. Intraclass correlation coefficient was 0.84 for PGY3 ratings and 0.89 for PGY4 ratings. A mixed effects analysis of variance showed no significant difference in HRV change from rest to simulation between teams or scenarios. Modified Angoff Score was not a predictor of HRV (multiple R2 = 0.0176).

Conclusions: Modified Angoff ratings demonstrated consensus in quantifying the estimated percentage of participants who would complete all critical actions for most scenarios. Although participant HRV did decrease during the scenarios, we were unable to significantly correlate this with ratings. This modified Angoff method is a feasible approach to evaluate simulation difficulty for educational and research purposes and may decrease the time and resources necessary for scenario piloting.