Assessing the inter-rater and intra-rater reliability of the Physical Therapy Competence Assessment for Airway Suctioning (PT-CAAS)

Erin Miller; Dina Brooks; Kelly K O'Brien; Lindsay Beavers; Paul Stratford; Mika Nonoyama; Brenda Mori

doi:10.1002/pri.1944

Assessing the inter-rater and intra-rater reliability of the Physical Therapy Competence Assessment for Airway Suctioning (PT-CAAS)

Physiother Res Int. 2022 Apr;27(2):e1944. doi: 10.1002/pri.1944. Epub 2022 Feb 17.

Authors

Erin Miller^{1

2}, Dina Brooks^{1

2

3

4

5}, Kelly K O'Brien^{1

2

6}, Lindsay Beavers^{2

7}, Paul Stratford³, Mika Nonoyama^{1

2

8

9}, Brenda Mori^{1

2}

Affiliations

¹ Rehabilitation Sciences Institute, University of Toronto, Toronto, Ontario, Canada.
² Department of Physical Therapy, University of Toronto, Toronto, Ontario, Canada.
³ School of Rehabilitation Science, McMaster University, Hamilton, Ontario, Canada.
⁴ West Park Healthcare Centre, Toronto, Ontario, Canada.
⁵ Department of Medicine, University of Toronto, Toronto, Ontario, Canada.
⁶ Institute of Health Policy, Management and Evaluation (IHPME), University of Toronto, Toronto, Ontario, Canada.
⁷ Unity Health Toronto, Toronto, Ontario, Canada.
⁸ Faculty of Health Sciences, Ontario Tech University, Oshawa, Ontario, Canada.
⁹ Respiratory Therapy & Child Health Evaluative Sciences, Hospital for Sick Children, Toronto, Ontario, Canada.

PMID: 35174940
DOI: 10.1002/pri.1944

Abstract

Background and purpose: The Physical Therapy Competence Assessment for Airway Suctioning (PT-CAAS) is a recently developed measure to assess the clinical competence of physiotherapists who perform airway suctioning with adults. The purpose of this study was to assess the inter-rater and intra-rater reliability of the PT-CAAS.

Methods: Scoring rules were developed through expert consultation. Reliability was then assessed using nine videos of suctioning performed in a simulated learning environment. A repeated measures design was used, with two replicate sets of measurements made by each participant for all videos. Data were analyzed using a repeated measures model for the concurrent assessment of inter-rater and intra-rater reliability. Participants were physiotherapists with suctioning experience.

Results: Twenty physiotherapists completed initial scoring and re-scoring for all nine videos; their data were included in the analysis. Intraclass correlation coefficients (ICCs) for inter-rater reliability ranged from 0.569 [lower one-sided 95% confidence interval (CI): 0.395; standard error of measurement (SEM): 0.963] for infection control to 0.759 (lower one-sided 95% CI: 0.612; SEM: 0.722) for post-suctioning assessment and care. The inter-rater ICC for overall performance was 0.752 (lower one-sided 95% CI: 0.602; SEM: 0.660). ICCs for intra-rater reliability ranged from 0.759 (lower one-sided 95% CI: 0.197; SEM 0.721) for infection control to 0.860 (lower one-sided 95% CI: 0.544; SEM: 0.550) for post-suctioning assessment and care. The intra-rater ICC for overall performance was 0.867 (lower one-sided 95% CI: 0.559; SEM: 0.483).

Discussion: Evidence of moderate to good inter-rater and good intra-rater reliability was found; however, the results should be interpreted with caution given the wide CIs and relatively large SEMs. Improved assessor training and assessments of reliability using a larger sample size are recommended.

Keywords: clinical competence; observer variation; physical therapists; suction.

MeSH terms

Adult
Clinical Competence
Humans
Observer Variation
Physical Therapists*
Physical Therapy Modalities
Reproducibility of Results

Abstract

MeSH terms

Grants and funding