Background: Quality improvement studies can provide important insight to learning health systems. The Agency for Healthcare Research and Quality (AHRQ) could devote resources to collate and assess these quality improvement studies to support learning health systems (LHS) but there is no reliable data on the consistency of data extraction for important criteria.
Methods: We identified quality improvement studies in asthma and evaluated the consistency of data extraction from two experienced independent reviewers at three time points: baseline where only a rudimentary description of the criteria was available, first revision where explicit instructions for each criterion were created, and final revision where the instructions were revised. Six investigators looked at the data extracted by one of the systematic reviewers and then the other for the same criteria and determined the extent of similarity on a scale of 0 to 10 (where 0 represented no similarity and 10 perfect similarity). There were 42 assessments for baseline, 42 assessments for the first revision, and 42 assessments for the final revision. We then asked two LHS participants to assess the relative value of our criteria in a pilot phase.
Results: We went through two refinements of the data extraction instructions for each criterion and were able to improve the consistency of extraction from 1.17±1.85 at baseline to 6.07±2.76 after revision one (P<0.001) and to 6.81±1.94 out of 10 for the final revision (P<0.001). However, the final revision was not significantly improved over revision one (p=0.14). In the pilot phase, our two LHS participants felt that some of our 33 criteria were more valuable than others were.
Discussion/Conclusion: Creating explicit instructions for extracting data for quality improvement study helps enhance the consistency of data extraction. Future studies with a larger cadre of LHS participants should help determine the most important criteria.