Validity Evidence for a Novel, Comprehensive Bag-Mask Ventilation Assessment Tool

Allison M Whalen; Matthew H Merves; Priyanka Kharayat; James S Barry; Kristen M Glass; Robert A Berg; Taylor Sawyer; Vinay Nadkarni; Donald L Boyer; Akira Nishisaki

doi:10.1016/j.jpeds.2022.02.017

Validity Evidence for a Novel, Comprehensive Bag-Mask Ventilation Assessment Tool

J Pediatr. 2022 Jun:245:165-171.e13. doi: 10.1016/j.jpeds.2022.02.017. Epub 2022 Feb 16.

Authors

Affiliations

¹ Division of Pediatric Critical Care Medicine, Department of Pediatrics, Medical University of South Carolina, Charleston, SC. Electronic address: whalen@musc.edu.
² Division of Neonatology, Department of Pediatrics, University of Arkansas for Medical Sciences and Arkansas Children's Hospital, Little Rock, AR.
³ Department of Pediatrics, Albert Einstein Medical Center, Philadelphia, PA.
⁴ Section of Neonatology, Department of Pediatrics, University of Colorado School of Medicine, Aurora, CO.
⁵ Division of Neonatal-Perinatal Medicine, Department of Pediatrics, Penn State College of Medicine, Milton S. Hershey Medical Center, Hershey, PA.
⁶ Division of Critical Care Medicine, Children's Hospital of Philadelphia, Philadelphia, PA; Department of Anesthesiology & Critical Care, Perelman School of Medicine at the University of Pennsylvania, Philadelphia, PA.
⁷ Division of Neonatology, Department of Pediatrics, University of Washington School of Medicine, Seattle Children's Hospital, Seattle, WA.

PMID: 35181294
DOI: 10.1016/j.jpeds.2022.02.017

Abstract

Objective: To develop a comprehensive competency assessment tool for pediatric bag-mask ventilation (pBMV) and demonstrate multidimensional validity evidence for this tool.

Study design: A novel pBMV assessment tool was developed consisting of 3 components: a 22-item-based checklist (trichotomized response), global rating scale (GRS, 5-point), and entrustment assessment (4-point). Participants' performance in a realistic simulation scenario was video-recorded and assessed by blinded raters. Multidimensional validity evidence for procedural assessment, including evidence for content, response-process, internal structure, and relation to other variables, was assessed. The scores of each scale were compared with training level. Item-based checklist scores also were correlated with GRS and entrustment scores.

Results: Fifty-eight participants (9 medical students, 10 pediatric residents, 18 critical care/neonatology fellows, 21 critical care/neonatology attendings) were evaluated. The pBMV tool was supported by high internal consistency (Cronbach α = 0.867). Inter-rater reliability for the item-based checklist component was acceptable (r = 0.65, P < .0001). The item-based checklist scores differentiated between medical students and other providers (P < .0001), but not by other trainee level. GRS and entrustment scores significantly differentiated between training levels (P < .001). Correlation between skill item-based checklist and GRS was r = 0.489 (P = .0001) and between item-based checklist and entrustment score was r = 0.52 (P < .001). This moderate correlation suggested each component measures pBMV skills differently. The GRS and entrustment scores demonstrated moderate inter-rater reliability (0.42 and 0.46).

Conclusions: We established evidence of multidimensional validity for a novel entrustment-based pBMV competence assessment tool, incorporating global and entrustment-based assessments. This comprehensive tool can provide learner feedback and aid in entrustment decisions as learners progress through training.

Keywords: assessment; checklist; medical education; procedural competency; simulation; workplace-based assessment.

Publication types

Research Support, Non-U.S. Gov't
Research Support, U.S. Gov't, P.H.S.

MeSH terms

Checklist / methods
Child
Clinical Competence
Critical Care
Educational Measurement
Humans
Internship and Residency*
Reproducibility of Results
Students, Medical*

Abstract

Publication types

MeSH terms

Grants and funding