Formative Assessment of Performance in Diagnostic Ultrasound Using Simulation and Quantitative and Objective Metrics

Abstract

Background: We developed simulator-based tools for assessing provider competence in transthoracic echocardiography (TTE) and vascular duplex scanning.

Methods: Psychomotor (technical) skill in TTE image acquisition was calculated from the deviation angle of an acquired image from the anatomically correct view. We applied this metric for formative assessment to give feedback to learners and evaluate curricula.Psychomotor skill in vascular ultrasound was measured in terms of dexterity and image plane location; cognitive skill was assessed from measurements of blood flow velocity, parameter settings, and diagnosis. The validity of the vascular simulator was assessed from the accuracy with which experts can measure peak systolic blood flow velocity (PSV).

Results: In the TTE simulator, the skill metric enabled immediate feedback, formative assessment of curriculum efficacy, and comparison of curriculum outcomes. The vascular duplex ultrasound simulator also provided feedback, and experts' measurements of PSV deviated from actual PSV in the model by <10%.

Conclusions: Skill in acquiring diagnostic ultrasound images of organs and vessels can be measured using simulation in an objective, quantitative, and standardized manner. Current applications are provision of feedback to learners to enable training without direct faculty oversight and formative assessment of curricula. Simulator-based metrics could also be applied for summative assessment.

Keywords: diagnostic ultrasound; duplex scanning; echocardiography; feedback; formative assessment; medical education; simulation.