New Methods Guide Chapter Focuses on Observational Studies for Systematic Reviews
A newly published chapter of the Methods Guide for Effectiveness and Comparative Effectiveness Reviews offers a conceptual framework for Agency for Healthcare Research and Quality (AHRQ) researchers who are considering observational studies to determine the benefits of pharmacotherapeutic, device, or procedural interventions. Posted to the Effective Health Care (EHC) Program Web site on June 14, 2010, the chapter offers researchers a clear set of decision criteria for whether to include or exclude observational studies when conducting systematic reviews.
While most researchers agree that observational studies are appropriate for identifying and quantifying adverse events, the use of data from observational studies to answer questions about intended effects or benefits is more controversial. Some researchers believe that observational studies cannot provide valid or useful evidence of benefit.
The framework focuses on two questions: (1) Are there gaps in the evidence from randomized controlled trials (RCTs)? (2) Will observational studies provide valid and useful information? The chapter then describes in detail how to answer these questions in order to make a final determination about the inclusion of observational studies, such as refocusing the review questions on the gaps in the evidence from RCTs, and assessing the risk of bias in the observational studies. A set of criteria are provided to assist researchers in making an assessment of the data in an RCT to determine if gaps exist. The chapter also includes scenarios where decisions to include or exclude observational studies were made, as well as a flow diagram for considering observational studies for answering comparative effectiveness questions.
The authors — a team of scientists from seven of the Evidence-based Practice Centers (EPCs), AHRQ, and the VA Quality Enhancement Research Initiative (QUERI) — remind readers that the framework of comparative effectiveness reviews (CERs) differs from traditional systematic reviews. As they state, CERs “more closely parallel the decisions facing clinicians, patients, and policymakers, who must choose among a variety of alternatives in making diagnostic, treatment, and health care delivery decisions.” The authors suggest, however, that comparative effectiveness reviewers should routinely assess whether the inclusion of observational studies is appropriate for questions of benefit, and that reviewers should explicitly state their rationale for including or excluding observational studies in the reviews.