Interpretability, credibility, and usability of hospital-specific template matching versus regression-based hospital performance assessments; a multiple methods study

Brenda M McGrath; Linda Takamine; Cainnear K Hogan; Timothy P Hofer; Amy K Rosen; Jeremy B Sussman; Wyndy L Wiitala; Andrew M Ryan; Hallie C Prescott

doi:10.1186/s12913-022-08124-w

Interpretability, credibility, and usability of hospital-specific template matching versus regression-based hospital performance assessments; a multiple methods study

BMC Health Serv Res. 2022 Jun 3;22(1):739. doi: 10.1186/s12913-022-08124-w.

Authors

Brenda M McGrath¹, Linda Takamine¹, Cainnear K Hogan¹, Timothy P Hofer^{1

2}, Amy K Rosen^{3

4}, Jeremy B Sussman^{1

2}, Wyndy L Wiitala¹, Andrew M Ryan⁵, Hallie C Prescott^{6

7}

Affiliations

¹ VA Center for Clinical Management Research, Ann Arbor, MI, USA.
² Department of Internal Medicine, University of Michigan, Ann Arbor, MI, USA.
³ VA Center for Healthcare Organization and Implementation Research, VA Boston Healthcare System, Boston, MA, USA.
⁴ Department of Surgery, Boston University School of Medicine, Boston, MA, USA.
⁵ Department of Health Management and Policy, School of Public Health, University of Michigan, Ann Arbor, MI, USA.
⁶ VA Center for Clinical Management Research, Ann Arbor, MI, USA. hprescot@med.umich.edu.
⁷ Department of Internal Medicine, University of Michigan, Ann Arbor, MI, USA. hprescot@med.umich.edu.

Abstract

Background: Hospital-specific template matching (HS-TM) is a newer method of hospital performance assessment.

Objective: To assess the interpretability, credibility, and usability of HS-TM-based vs. regression-based performance assessments.

Research design: We surveyed hospital leaders (January-May 2021) and completed follow-up semi-structured interviews. Surveys included four hypothetical performance assessment vignettes, with method (HS-TM, regression) and hospital mortality randomized.

Subjects: Nationwide Veterans Affairs Chiefs of Staff, Medicine, and Hospital Medicine.

Measures: Correct interpretation; self-rated confidence in interpretation; and self-rated trust in assessment (via survey). Concerns about credibility and main uses (via thematic analysis of interview transcripts).

Results: In total, 84 participants completed 295 survey vignettes. Respondents correctly interpreted 81.8% HS-TM vs. 56.5% regression assessments, p < 0.001. Respondents "trusted the results" for 70.9% HS-TM vs. 58.2% regression assessments, p = 0.03. Nine concerns about credibility were identified: inadequate capture of case-mix and/or illness severity; inability to account for specialized programs (e.g., transplant center); comparison to geographically disparate hospitals; equating mortality with quality; lack of criterion standards; low power; comparison to dissimilar hospitals; generation of rankings; and lack of transparency. Five concerns were equally relevant to both methods, one more pertinent to HS-TM, and three more pertinent to regression. Assessments were mainly used to trigger further quality evaluation (a "check oil light") and motivate behavior change.

Conclusions: HS-TM-based performance assessments were more interpretable and more credible to VA hospital leaders than regression-based assessments. However, leaders had a similar set of concerns related to credibility for both methods and felt both were best used as a screen for further evaluation.

Keywords: Benchmarking; Hospital mortality; Quality of health care; Risk adjustment.

MeSH terms

Delivery of Health Care
Diagnosis-Related Groups*
Hospital Mortality
Hospitals*
Humans
Surveys and Questionnaires

Grants and funding

I01 HX002546/HX/HSRD VA/United States