Purpose of Project
To evaluate evidence summaries of Evidence-based Practice Center systematic reviews for their content, components, and usability among health systems decisionmakers.
- Decisionmakers (i.e., leaders within units of a large health system that help develop and implement evidence-informed health improvement strategies) thought the summaries would be foundational for other products such as clinical pathways.
- Decisionmakers preferred a 3-page summary with key messages, details on results, meaningful figures/tables, and strength of evidence; detailed methods and contextual information were less important.
- Development of a summary involved about 90 hours of person-time, and required expertise in systematic reviews and meta-analysis, qualitative analysis, and graphic design.
- Three-page summaries could be produced for most EPC reports and have value for end-users; however, consideration should be given to their purpose and format (e.g., "one size fits all," tailored for specific decisions).
Objectives. To develop dissemination summaries based on AHRQ EPC evidence reviews, and to evaluate the summaries, and their components, for usability among health systems decisionmakers.
Methods. For each of two reviews, we designed a three-page summary with the intent that the first page could potentially be a stand-alone one-page summary. Summaries included various report elements (e.g., key questions, methods, eligibility criteria, analytic framework, summary tables, forest plots) and varied in layout and design (e.g., text only vs. use of images). We conducted ‘think aloud’ telephone interviews with six decisionmakers from Strategic Clinical Networks, which are clinician-led teams that develop and implement evidence-informed health improvement strategies within a province-wide fully integrated health system. We analyzed the data thematically and translated themes into recommendations.
Results. Interviews yielded information on content, layout and design, and usability. With respect to content, themes included purpose, methods, summary data, and overall message. Recommendations regarding content were to: include a purpose statement and define the intended audience; discuss the methods briefly; when possible, provide data to support the findings (e.g., summary estimates with confidence intervals and strength of evidence, use standard well-organized tables and figures); state the overall message on the first page (include key findings, knowledge gaps, and take home message(s)). For layout and design, themes included readability, visual appeal, figures, tables, and credibility. Recommendations were to: use brief titles and subheadings; avoid acronyms; organize content logically; balance text with images and white space; use standard figures to present findings and when feasible, use figures instead of tables; use standard tables to present findings and when feasible, use tables instead of text; include the organization and program logos; and employ professional design choices. Themes regarding usability included language, length, and content. Recommendations were to: use plain language consistently; keep the summary to three pages or less; provide a link to the full text; organize the findings for maximum use; and consider usability in greyscale.
Conclusions. Decisionmakers preferred a three-page summary that presented key messages, details on results, and strength of evidence; detailed methods and contextual information were less important. Decisionmakers preferred summaries with use of meaningful images, graphs, figures, and tables to convey information. Decisionmakers thought the summaries would be foundational for other knowledge translation products such as clinical pathways.
Suggested citation: Hartling L, Gates A, Pillay J, Nuspl M, Newton AS. Development and Usability Testing of EPC Evidence Review Dissemination Summaries for Health Systems Decisionmakers. Methods Research Report. (Prepared by the University of Alberta Evidence-based Practice Center under Contract No. 290-290-2015-00001-I). AHRQ Publication No. 18(19)-EHC027-EF. Rockville, MD: Agency for Healthcare Research and Quality. November 2018. Posted final reports are located on the Effective Health Care Program search page. https://doi.org/10.23970/AHRQEPCMETHENGAGEDEVELOPMENT.