This is a chapter from "Methods Guide for Effectiveness and Comparative Effectiveness Reviews."
This paper has also been published in edited form: Slutsky J, Atkins D, Chang S, et al. AHRQ Series Paper 1: Comparing medical interventions: AHRQ and the Effective Health Care Program. J Clin Epidemiol 2010; 63, 481-483.
Comparative Effectiveness Reviews are systematic reviews of existing research on the effectiveness, comparative effectiveness, and harms of different health care interventions. They provide syntheses of relevant evidence to inform real-world health care decisions for patients, providers, and policymakers. Strong methodologic approaches to systematic review improve the transparency, consistency, and scientific rigor of these reports. Through a collaborative effort of the Effective Health Care (EHC) Program, the Agency for Healthcare Research and Quality (AHRQ), the EHC Program Scientific Resource Center, and the AHRQ Evidence-based Practice Centers have developed a Methods Guide for Comparative Effectiveness Reviews. This Guide presents issues key to the development of Comparative Effectiveness Reviews and describes recommended approaches for addressing difficult, frequently encountered methodological issues.
The Methods Guide for Comparative Effectiveness Reviews is a living document, and will be updated as further empiric evidence develops and our understanding of better methods improves. Comments and suggestions on the Methods Guide for Comparative Effectiveness Reviews and the Effective Health Care Program can be made at https://effectivehealthcare.ahrq.gov.
None of the authors has a financial interest in any of the products discussed in this document.
Suggested citation: Slutsky J, Atkins D, Chang S, et al. Comparing medical interventions: AHRQ and the Effective Health Care Program. In: Agency for Healthcare Research and Quality. Methods Guide for Comparative Effectiveness Reviews [posted November 2008]. Rockville, MD. Available at: https://effectivehealthcare.ahrq.gov/products/cer-methods-guide/overview/.
This paper has also been published in edited form: Slutsky J, Atkins D, Chang S, et al. Comparing medical interventions: AHRQ and the Effective Health Care Program. J Clin Epidemiol 2008 Sep 30 [Epub ahead of print].
Authors
Jean Slutsky, P.A., M.S.P.H.a
David Atkins, M.D., M.P.H.b
Stephanie Chang, M.D., M.P.H.a
Beth A. Collins Sharp, Ph.D.a
a Agency for Healthcare Research and Quality, Rockville, MD.
b Veterans Health Administration, Health Services Research and Development Service, Washington, DC.
The findings and conclusions in this document are those of the authors, who are responsible for its contents; the findings and conclusions do not necessarily represent the views of AHRQ, the Veterans Health Administration, or the Health Services Research and Development Service. Therefore, no statement in this report should be construed as an official position of these entities, the U.S. Department of Health and Human Services, or the U.S. Department of Veterans Affairs.
Key Points
Health care expenditures are growing faster than incomes for most developed countries, jeopardizing the stability of health care systems globally.1 This trend has led to interest in knowledge about the most effective use of health care worldwide. To increase the value of health care services, many countries have established programs or independent agencies that inform health care decisionmaking through systematic reviews of technologies, pharmaceuticals, and other health care interventions. A few examples include the National Institute for Health and Clinical Excellence (NICE) in the United Kingdom, the Institute for Quality and Efficiency in Health Care (IQWiG) in Germany, the Haute Autorité de Santé (HAS) in France, and the Canadian Agency for Drugs and Technologies in Health (CADTH). Some international consortiums and collaborations are also committed to increasing the use of evidence in health care decisionmaking. The Cochrane Collaboration has received international recognition for its sustained efforts at developing and disseminating systematic reviews. Additionally, Health Technology Assessment International (HTAi) is an organization with global membership that promotes evidence-based technology assessments.
By any measure, health care expenditures in the United States are increasing much faster than the health of the population and at a faster rate than in any other industrialized nation. Driven by the same goals as other countries and organizations—improving the quality, effectiveness, and efficiency of health care delivery—the U.S. Agency for Healthcare Research and Quality (AHRQ) created the Effective Health Care (EHC) Program in 2005.
A series of articles to be presented here in upcoming months give guidance on the methods to be used in conducting systematic reviews of technologies and interventions under the EHC Program, and together they form the Methods Guide for Comparative Effectiveness Reviews. While the various international programs and agencies mentioned here are united in their goal of providing objective assessments of effective health care interventions through systematic reviews, the varied health care system environments necessitate differences among the programs. For example, with the presence of a universal health system, NICE conducts cost-effectiveness studies, which are more difficult in a decentralized health care system. It is important to understand the context, principles, and philosophies of each program or agency, since they carry implications for the various approaches, methods, and end products of systematic reviews from the various groups.
The United States spent an estimated $1.8 trillion dollars in 2005 on health care, including $342 billion under its Medicare program, with an annual estimated cost growth of 2.4 percent above the Gross Domestic Product.2 Potential solutions for long-term solvency of the Medicare program for seniors and the disabled have been the cause of much political debate. This debate led to a series of Medicare reforms passed by Congress in 2003.3 These reforms included a new drug benefit for seniors as well as new funding of $15 million annually for AHRQ (subsequently doubled to $30 million) to conduct and support research with a focus on the outcomes, comparative clinical effectiveness, and appropriateness of pharmaceuticals, devices, and health care services. Underlying this effort is a realization that improving value and controlling Medicare costs can be achieved only by understanding the relative effectiveness of the different health care interventions at our disposal—both old and new. The EHC Program is guided by 14 priority conditions that are important to beneficiaries of the Medicare, Medicaid, and State Children's Health Insurance Program but would resonate with health care programs throughout the world.
The EHC Program involves the collaborative efforts of three major activities: systematic review, new research, and translation of findings for different audiences. Like the majority of the programs throughout the world, the EHC Program relies on systematic review methods to provide guidance on the effectiveness of therapeutics. The EHC program commissions 14 Evidence-based Practice Centers to perform the systematic reviews that provide an essential foundation from which to understand what we know from existing research and what critical research gaps remain. The Evidence-based Practice Centers undertake a broad variety of reviews that assess the effectiveness, comparative effectiveness, and comparative harms of different health care interventions. Some of these reviews are especially challenging in breadth and depth because the questions of most interest to decisionmakers often require complex comparisons. The EHC Program is supported by a Scientific Resource Center, which provides scientific and technical support to maintain consistency in the methods used across the different centers.
The EHC Program reflects in many ways the decentralized nature of the U.S. health care system. The audience includes not only policymakers in government and private health plans but also clinicians, patients, and members of industry, all of whom play a major role in health care decisionmaking. All of these stakeholders provide input and guidance to the program, all may contribute suggestions of new topics for assessment, and all have provided comments on drafts of the guidance given in this series. The EHC Program is meant to provide understandable and actionable information for patients, clinicians, and policymakers.
In order to provide useful information on effective health care interventions, the EHC Program follows three key principles that guide the EHC Program and, thus, the conduct of systematic reviews by the Evidence-based Practice Centers. First, reviews must be relevant and timely in order to meet the needs of decisionmakers. The questions being addressed in reviews must answer emerging and complex health care questions at the time when decisionmakers need the information. This means identifying the most important issues under the priority conditions and the optimal time to initiate a review. It also requires a conscientious effort to complete the review as quickly as possible without sacrificing the quality of the product.
Second, reviews must be objective and scientifically rigorous. To maintain the objectivity of a review, lead authors on the reports are barred from having any significant competing interests. In addition, although Evidence-based Practice Center staff, consultants, subcontractors, and other technical experts may not be disqualified from providing comments, they must disclose any financial, business, and professional interests that are related to the subject matter of a review or other product or that could be affected by the findings of the review. With respect to the types of financial interests to be disclosed, AHRQ is guided by the U.S. Department of Health and Human Services Regulations 45 CFR Part 94. Directors of the Evidence-based Practice Centers are responsible for the scientific integrity of all members of the review team by ensuring that they comply with AHRQ policy and by providing opportunities for training in rigorous scientific methods. There are a variety of sources for training in systematic review scientific methods in the United States and elsewhere. In addition to having the Methods Guide for Comparative Effectiveness Reviews as a resource, AHRQ and the Scientific Resource Center have regularly scheduled conference calls with Evidence-based Practice Centers and face-to-face meetings biannually to discuss scientific methods and other aspects of producing scientifically sound and credible systematic reviews. The Evidence-based Practice Centers participate in many scientific forums, and the work they do in methods informs the process and helps in collaborating with the work of similar groups in other countries.
Finally, public participation and transparency increase public confidence in the scientific integrity and credibility of reviews and provide further accountability to the Evidence-based Practice Centers. Reviews commissioned under the EHC Program are posted publicly at different stages of the review process, including the stage of proposed Key Questions and the draft report stage. Public posting of the processes and methodological approaches used in developing systematic reviews ensures that the reports are accessible, clear, and credible. The publication of this series of methods articles in the Journal of Clinical Epidemiology and the posting of the Methods Guide for Comparative Effectiveness Reviews on the EHC Web site (https://effectivehealthcare.ahrq.gov) are fundamental ways of clearly laying out the EHC approach to conducting systematic reviews of comparative effectiveness.
The Evidence-based Practice Centers' work on Comparative Effectiveness Reviews builds on nearly 10 years of experience doing systematic reviews of diverse topics, including drugs and devices, diagnostic tests, and health care system interventions.4 Unlike many other programs or agencies producing systematic reviews, which focus on evaluating individual interventions, the AHRQ EHC Program focuses on health care questions that require comparisons of alternative interventions for a given clinical condition.
In addition to the familiar issues raised in a systematic review or meta-analysis of a single intervention, there are specific challenges encountered in conducting Comparative Effectiveness Reviews. The methods papers in this series were written in response to these specific challenges.
The aim of a Comparative Effectiveness Review is to depict how the relative benefits and harms of a range of options compare, rather than to answer a narrow question of whether a single therapy is safe and effective. This requires a clear understanding of the clinical context to ensure that the review focuses on the appropriate population and interventions among which clinicians are currently choosing. As an example, our review of coronary artery bypass surgery vs. percutaneous coronary intervention for stable coronary disease focused on patients who have stable angina and two-vessel disease and on other subgroups for which clinicians might currently consider either option. It did not address patients at either clinical extreme, for whom the benefits of one option might be clear cut.
There is rarely a sufficient body of head-to-head trials to support easy conclusions about comparative benefits and harms. Providing useful information requires examining a broader array of literature, including placebo-controlled trials and observational studies; the latter are especially useful for looking more completely at harms, adherence, and persistence. In addition, reviews may examine whether, in the absence of head-to-head trials, indirect comparisons may be useful (e.g., comparing results of placebo-controlled trials of A and placebo-controlled trials of B).
Carefully examining the applicability of evidence is especially important. A useful review compares the tradeoffs of multiple alternatives, each of which may vary with the underlying population and setting. For example, the results of trials comparing the abilities of different oral diabetes drugs to control blood glucose may depend in important ways on the populations being studied. Evidence on harms is often hard to determine from tightly controlled randomized trials. Observational studies provide another check on whether results observed in trials appear to hold up under more representative settings and populations.
Finally, the interpretation of the evidence and the limits of interpretation are important. Equivalence of different treatments for a group of patients on average does not necessarily imply they are equivalent for all individuals. Attempts to explore subgroups for which benefits or harms of specific interventions vary may be needed. Often, however, there is limited evidence to support strong conclusions about the specific benefits of a particular intervention for subgroups.
The articles in this series reflect the final individual chapters of the EHC Methods Guide for Comparative Effectiveness Reviews. Written by AHRQ Evidence-based Practice Center investigators with the intention of improving both consistency and transparency in the EHC program, they were initially posted as one draft document for public comment on the EHC Web site in late 2007 and have been revised in response to public comment. Where there is an inadequate empiric evidence base, the articles review the existing guidance produced by different organizations and collaborations and build on these activities, focusing on issues specific to conducting Comparative Effectiveness Reviews. As the research methodologies develop, the EHC Program will continue to assess the need to update the current Methods Guide for Comparative Effectiveness Reviews.
Building a stronger empiric base for methods will increase transparency and consistency within and among the various groups that produce reviews of comparative effectiveness. In areas where empiric research is lacking, collaboration is paramount to determine best practices and to set a methods research agenda. Uniform guidance based on validated methods is essential to providing quality and consistent evidence for patients, clinicians, and policymakers, no matter where they live.
References
1. Kaiser Family Foundation. Healthcare Spending in the United States and OECD Countries. Available at https://www.kff.org/. Accessed January 2007.
2. Congress of the United States, Congressional Budget Office. The Long-Term Outlook for Health Care Spending. Available at: https://www.cbo.gov/ftpdocs/87xx/doc8758/11-13-LT-Health.pdf. Accessed November 2007.
3. Medicare Prescription Drug, Improvement, and Modernization Act of 2008. Sec. 1013. Research on Outcomes of Health Care and Services. Public Law 108-173. 108th Congress.
4. Atkins D, Fink K, Slutsky J. Better information for better health care: the Evidence-based Practice Center Program and the Agency for Healthcare Research and Quality. Ann Intern Med 2005;142:1035-41.
