Validity evidence for the Anesthesia Clinical Encounter Assessment (ACEA) tool to support competency-based medical education

Alayne Kealey; Fahad Alam; Lisa A Bahrey; Clyde T Matava; Graham A McCreath; Catharine M Walsh

doi:10.1016/j.bja.2021.12.012

Validity evidence for the Anesthesia Clinical Encounter Assessment (ACEA) tool to support competency-based medical education

Br J Anaesth. 2022 Apr;128(4):691-699. doi: 10.1016/j.bja.2021.12.012. Epub 2022 Jan 10.

Authors

Alayne Kealey¹, Fahad Alam², Lisa A Bahrey³, Clyde T Matava⁴, Graham A McCreath⁵, Catharine M Walsh⁶

Affiliations

¹ Department of Anesthesia, Sunnybrook Health Sciences Centre, Toronto, ON, Canada; Department of Anesthesiology and Pain Medicine, Temerty Faculty of Medicine, University of Toronto, Toronto, ON, Canada. Electronic address: Alayne.Kealey@sunnybrook.com.
² Department of Anesthesia, Sunnybrook Health Sciences Centre, Toronto, ON, Canada; Department of Anesthesiology and Pain Medicine, Temerty Faculty of Medicine, University of Toronto, Toronto, ON, Canada; The Wilson Centre, Temerty Faculty of Medicine, University of Toronto, Toronto, ON, Canada.
³ Department of Anesthesiology and Pain Medicine, Temerty Faculty of Medicine, University of Toronto, Toronto, ON, Canada; Department of Anesthesia, University Health Network-Toronto General Hospital, Toronto, ON, Canada.
⁴ Department of Anesthesiology and Pain Medicine, Temerty Faculty of Medicine, University of Toronto, Toronto, ON, Canada; SickKids Research Institute, Hospital for Sick Children, Toronto, ON, Canada; Department of Anesthesia and Pain Medicine, Hospital for Sick Children, Toronto, ON, Canada.
⁵ SickKids Research Institute, Hospital for Sick Children, Toronto, ON, Canada.
⁶ The Wilson Centre, Temerty Faculty of Medicine, University of Toronto, Toronto, ON, Canada; SickKids Research Institute, Hospital for Sick Children, Toronto, ON, Canada; Division of Gastroenterology, Hepatology and Nutrition, Hospital for Sick Children, University of Toronto, Toronto, ON, Canada; Department of Paediatrics, Temerty Faculty of Medicine, University of Toronto, Toronto, ON, Canada; SickKids Learning Institute, Hospital for Sick Children, Toronto, ON, Canada.

PMID: 35027168
DOI: 10.1016/j.bja.2021.12.012

Abstract

Background: Workplace-based assessment (WBA) is key to a competency-based assessment strategy. Concomitantly with our programme's launch of competency-based medical education, we developed an entrustment-based WBA, the Anesthesia Clinical Encounter Assessment (ACEA), to assess readiness for independent practice of competencies essential to perioperative patient care. This study aimed to examine validity evidence of the ACEA during postgraduate anaesthesiology training.

Methods: The ACEA comprises an eight-item global rating scale (GRS), an overall independence rating, an eight-item checklist, and case details. ACEA data were extracted for University of Toronto anaesthesia residents from July 2017 to January 2020 from the programme's online assessment portal. Validity evidence was generated following Messick's validity framework, including response process, internal structure, relations with other variables, and consequences.

Results: We analysed 8664 assessments for 137 residents completed by 342 assessors. From generalisability analysis, 10 independent observations (two assessments each from five assessors) were sufficient to achieve a reliability threshold of ≥0.70 for in-training assessments. A composite GRS score of 3.65/5 provided optimal sensitivity (93.6%) and specificity (90.8%) for determining entrustment on receiver operator characteristic curve analysis. Test-retest reliability was high (intraclass correlation coefficient [ICC_2,1]=0.81) for matched assessments within 14 days of each other. Composite GRS scores differed significantly between residents based on their training level (P<0.0001) and correlated highly with overall independence (0.91, P<0.001). The internal consistency of the GRS (α=0.96) was excellent.

Conclusions: This study supports the validity of the ACEA for assessing the competence of residents performing perioperative care and supports its use in competency-based anaesthesiology training.

Keywords: Messick's validity framework; anaesthesia postgraduate training; competency-based medical education; entrustment scale; workplace-based assessment.

MeSH terms

Anesthesia*
Anesthesiology* / education
Clinical Competence
Education, Medical*
Educational Measurement
Humans
Internship and Residency*
Reproducibility of Results