Purpose of Report
To determine if the consistency of data extraction improves after creating explicit instructions for quality improvement study assessment criteria.
- Findings: Creating explicit instructions to accompany quality improvement study assessment criteria greatly enhances the consistency of data extraction.
- Lessons Learned for EPC Program: While inconsistency in data extraction was a major barrier to moving forward with identifying criteria crucial for the understanding of quality improvement studies, an iterative process to assess the consistency and then refine the instructions can be successful.
- Utility for Health Systems: Learning health systems can expect to see extracted data for different criteria that would look consistent regardless of the person doing the extraction and enhancing the consistency from report to report as well.
Background. Quality improvement studies can provide important insight to learning health systems. The Agency for Healthcare Research and Quality (AHRQ) could devote resources to collate and assess these quality improvement studies to support learning health systems (LHS) but there is no reliable data on the consistency of data extraction for important criteria.
Methods. We identified quality improvement studies in asthma and evaluated the consistency of data extraction from two experienced independent reviewers at three time points: baseline where only a rudimentary description of the criteria was available, first revision where explicit instructions for each criterion were created, and final revision where the instructions were revised. Six investigators looked at the data extracted by one of the systematic reviewers and then the other for the same criteria and determined the extent of similarity on a scale of 0 to 10 (where 0 represented no similarity and 10 perfect similarity). There were 42 assessments for baseline, 42 assessments for the first revision, and 42 assessments for the final revision. We then asked two LHS participants to assess the relative value of our criteria in a pilot phase.
Results. We went through two refinements of the data extraction instructions for each criterion and were able to improve the consistency of extraction from 1.17+1.85 at baseline to 6.07+2.76 after revision one (P<0.001) and to 6.81+1.94 out of 10 for the final revision (P<0.001). However, the final revision was not significantly improved over revision one (p=0.14). In the pilot phase, our two LHS participants felt that some of our 33 criteria were more valuable than others were.
Discussion/Conclusion. Creating explicit instructions for extracting data for quality improvement study helps enhance the consistency of data extraction. Future studies with a larger cadre of LHS participants should help determine the most important criteria.
Hernandez-Diaz AH, Roman YM, White CM. Developing criteria and associated instructions for consistent and useful quality improvement study data extraction for health systems. J Gen Intern Med. 17 Aug 2020. [Epub ahead of print.]
Suggested citation: Hernandez AV, Roman YM, White CM. Developing Consistent and Useful Quality Improvement Study Data Extraction for Health Systems. Methods Research Report. (Prepared by the University of Connecticut Evidence-based Practice Center under Contract No. 290-2015-00012-I.) AHRQ Publication No. 19(20)-EHC025-EF. Rockville, MD: Agency for Healthcare Research and Quality; September 2020. Posted final reports are located on the Effective Health Care Program search page. DOI: 10.23970/AHRQEPCMETHQUALIMPRCRITERIA.