Purpose: This investigation aimed to explore the measurement properties of scores from a patient simulator exercise.
Methods: Analytic and holistic scores were obtained for groups of medical students and residents. Item analysis techniques were used to explore the nature of specific examinee actions. Interrater reliability was calculated. Scores were contrasted for third year medical students, fourth year medical students and emergency department residents.
Results: Interrater reliabilities for analytic and holistic scores were 0.92 and 0.81, respectively. Based on item analysis, proper timing and sequencing of actions discriminated between low- and high-ability examinees. In general, examinees with more advanced training obtained higher scores on the simulation exercise.
Conclusion: Reliable and valid measures of clinical performance can be obtained from a trauma simulation provided that care is taken in the development and scoring of the scenario.