Guidance from within the Agency for Healthcare Research and Quality’s (AHRQ) Evidence-based Practice Center (EPC) Program has long recognized the need for systematic reviews of interventions impacting health to provide balanced assessments that include evaluation of harms as well as benefits. However, synthesizing evidence on harms poses unique challenges: the assessment and reporting of harms is often suboptimal,1,2 the duration of studies is often too short to evaluate important long-term harms and have inadequate statistical power to evaluate serious but uncommon harms,3, 4 patients enrolled in research studies are frequently at lower risk for harms than those encountered in clinical practice,5 and important data on harms may be unpublished or selectively reported.6,7
In 2005, AHRQ funded a series of white papers on challenges in evidence synthesis that included an article on evaluation of harms.3 It highlighted unique challenges in finding and selecting data on harms, rating the quality of harms assessment and reporting, and synthesizing and displaying data from studies reporting harms. Subsequently, recommendations for synthesizing evidence on harms were further developed by a Methods Workgroup of EPC investigators convened by AHRQ; these recommendations were codified in 2010 as a chapter in the AHRQ EPC Program Methods Guide.8 Issues addressed by the Workgroup included the need to consider a broad range of data sources to evaluate harms, including observational studies as well as randomized controlled trials and unpublished as well as published data; the importance of using consistent and precise terminology on harms; the need to evaluate the quality of harms assessment and reporting distinctly from the rigor for assessing benefits; and challenges in synthesis, including evaluation of rare events, use of indirect comparisons, and pooling methods.
In 2015, AHRQ convened an EPC Methods Workgroup to update or expand upon prior guidance for assessing harms. Following initial deliberations, the Workgroup elected to focus on updating guidance on prioritization and selection of harms to assess in systematic reviews. The Workgroup determined that guidance from the prior harms chapter (Appendix A) remains current. However, although the 2010 harms chapter recommended that EPC systematic reviews “always assess harms that are important to clinicians and patients,” the Workgroup found that it lacked more specific recommendations regarding how to select the harms to be included in an EPC review, and noted that selection and prioritization of harms in EPC reviews poses important challenges. EPC reviews frequently address many interventions, which could result in many potential harms (e.g., dozens) to review. Unlike benefits, which are often similar across interventions used to treat a given condition (e.g., medications, non-pharmacological therapies, and surgery for low back pain are all aimed at improving pain and function), different interventions given for the same condition are frequently associated with a large number of diverse harms. For example, medications for low back pain are typically associated with a set of harms distinct from those associated with surgery, and different medications are each associated with unique harms. Other issues include whether to assess composite harms (e.g., “serious harms” or “withdrawal due to adverse events”), which might facilitate comparisons between interventions with dissimilar harms, and how to address harms that are not specified in the original protocol but encountered during the review process. Workgroup members noted that including all potential harms in these cases is not feasible, and can make it difficult for users of EPC reviews to reach bottom-line conclusions regarding harms or determine the balance of benefits to harms. Workgroup members noted that clearer methods would be helpful for supporting the decisions made regarding selection of harms and help focus EPC reviews on the outcomes of greatest importance, potentially increasing their usability.
The purpose of this report is to provide guidance on prioritization and selection of harms for inclusion in systematic reviews. The immediate intended audience of this guidance is the EPC program, though we hope it may be useful to all systematic reviewers and those who commission or use systematic reviews.
Chou R, Baker WL, Bañez L, Iyer S, Myers ER, Newberry S, Pincock L, Robinson KA, Sardenga L, Sathe N, Springs S, Wilt TJ. Prioritization and Selection of Harms for Inclusion in Systematic Reviews. Methods Guide for Comparative Effectiveness Reviews. (Prepared by the Scientific Resource Center under Contract No. 290-2012-0004-C). AHRQ Publication No. AHRQ Pub No. 17(18)-EHC-034-EF. Rockville, MD: Agency for Healthcare Research and Quality; February 2018. DOI: https://doi.org/10.23970/AHRQEPCMETHGUIDE1.
- Haidich AB, Birtsou C, Dardavessis T, et al. The quality of safety reporting in trials is still suboptimal: survey of major general medical journals. J Clin Epidemiol. 2011;64(2):124-35. doi: 10.1016/j.jclinepi.2010.03.005.
- Hodkinson A, Kirkham JJ, Tudur-Smith C, et al. Reporting of harms data in RCTs: a systematic review of empirical assessments against the CONSORT harms extension. BMJ Open. 2013;3(9):e003436-2013-. doi:10.1136/bmjopen-2013-003436.
- Chou R, Helfand M. Challenges in systematic reviews that assess treatment harms. Ann Intern Med. 2005;142(12 Pt 2):1090-9. doi: 10.7326/0003-4819-142-12_Part_2-200506211-00009.
- Bhaumik DK, Amatya A, Normand SL, et al. Meta-Analysis of Rare Binary Adverse Event Data. J Am Stat Assoc. 2012;107(498):555-67. doi:10.1080/01621459.2012.664484.
- Rothwell PM. External validity of randomised controlled trials: “to whom do the results of this trial apply?”. Lancet. 2005;365(9453):82-93. doi:10.1016/S0140-6736(04)17670-8.
- Hodkinson A, Gamble C, Smith CT. Reporting of harms outcomes: a comparison of journal publications with unpublished clinical study reports of orlistat trials. Trials. 2016;17(207). doi:10.1186/s13063-016-1327-z.
- Saini P, Loke YK, Gamble C, et al. Selective reporting bias of harm outcomes within studies: findings from a cohort of systematic reviews. BMJ. 2014;349:g6501. doi:10.1136/bmj.g6501.
- Chou R, Aronson N, Atkins D, et al. AHRQ series paper 4: assessing harms when comparing medical interventions: AHRQ and the effective health-care program. J Clin Epidemiol. 2010;63(5):502-12. doi: 10.1016/j.jclinepi.2008.06.007.
*Only references cited in this Introduction are included here. Go to the full report for the complete reference list.