This is a chapter from "Methods Guide for Effectiveness and Comparative Effectiveness Reviews."

Comparative Effectiveness Reviews are systematic reviews of existing research on the effectiveness, comparative effectiveness, and harms of different health care interventions. They provide syntheses of relevant evidence to inform real-world health care decisions for patients, providers, and policymakers. Strong methodologic approaches to systematic review improve the transparency, consistency, and scientific rigor of these reports. Through a collaborative effort of the Effective Health Care (EHC) Program, the Agency for Healthcare Research and Quality (AHRQ), the EHC Program Scientific Resource Center, and the AHRQ Evidence-based Practice Centers have developed a Methods Guide for Effectiveness and Comparative Effectiveness Reviews. This Guide presents issues key to the development of Comparative Effectiveness Reviews and describes recommended approaches for addressing difficult, frequently encountered methodological issues.

The Methods Guide for Comparative Effectiveness Reviews is a living document, and will be updated as further empiric evidence develops and our understanding of better methods improves. Comments and suggestions on the Methods Guide for Effectiveness and Comparative Effectiveness Reviews and the Effective Health Care Program can be made at https://effectivehealthcare.ahrq.gov.

This document was written with support from the Effective Health Care Program at AHRQ.

None of the authors has a financial interest in any of the products discussed in this document.

Suggested citation: Eder M, Feightner A, Webber E, Guirguis-Blake J, Whitlock E. Developing and Selecting Topic Nominations for Systematic Reviews. Methods Guide for Comparative Effectiveness Reviews. Agency for Healthcare Research and Quality. November 2012. AHRQ Publication No. 12(13)-EHC153-EF. https://effectivehealthcare.ahrq.gov/

Authors

Michelle Eder, Ph.D.^a

Alisha Feightner, M.P.H.^a

Elizabeth Webber, M.S.^a

Janelle Guirguis-Blake, M.D.^a

Evelyn P. Whitlock, M.D., M.P.H.^a

^a Oregon Evidence-based Practice Center, Center for Health Research, Kaiser Permanente Northwest

The findings and conclusions in this document are those of the authors, who are responsible for its contents; the findings and conclusions do not necessarily represent the views of AHRQ. Therefore, no statement in this report should be construed as an official position of the U.S. Department of Health and Human Services.

Acknowledgments

The authors gratefully acknowledge Smyth Lai for her assistance in refining the searching processes for topic development, and Michele Solloway and Rebecca Holmes for their work in adapting the topic development process for evaluation of nominations for new primary research.

Peer Reviewers

Mary Butler, Ph.D, M.B.A. Minnesota Evidence-based Practice Center Minneapolis, MN

David Doggett, Ph.D. Johns Hopkins Evidence-based Practice Center Baltimore, MD

Amy Kendrick, R.N., M.S.N., M.S.-C.R.M. Duke Evidence-based Practice Center Durham, NC

Melissa McPheeters, Ph.D., M.P.H. Vanderbilt University Evidence-based Practice Center Nashville, TN

Carol Woodell RTI-UNC Evidence-based Practice Center Research Triangle Park, NC

Introduction

The Effective Health Care (EHC) Program of the Agency for Healthcare Research and Quality (AHRQ) was created under Section 1013 of the Medicare Prescription Drug, Improvement, and Modernization Act of 2003 to conduct comparative effectiveness research, including comparative effectiveness reviews of scientific evidence on health care interventions. Nominations for comparative effectiveness review topics are received via the EHC Program Web site. Given the extent of health care needs and constraints on the resources available to address these needs, methods to identify the most important topics for synthesized research are essential.

The research process includes topic identification, topic nomination development, topic selection, and topic refinement (https://effectivehealthcare.ahrq.gov/about/epc/nomination/). Topic identification is the receipt of nominations for a specific topic that occurs via submissions to the EHC Program Web site or through topic generation activities involving interactions with multiple stakeholders to elicit topics for systematic review. Topic nomination development is the evaluation of a nomination’s fit with EHC Program selection criteria. Topic selection is the selection of nominations for further refinement as a systematic review based on the nomination’s fit with EHC Program selection criteria. Topic refinement is further scoping of a selected topic, including development of Key Questions and an analytic framework, to guide the technical conduct of the systematic review. A 2009 AHRQ Series Paper outlined the principles underlying the selection of topics for systematic reviews within the EHC Program.¹ This followup paper describes current methods for topic nomination development to support the selection of topics for systematic reviews within the EHC Program. Topic identification and topic refinement are not addressed in this paper. Topic refinement is addressed in a separate methods chapter.²

The initial step in formulating the methodology for topic nomination development involved defining the criteria used to select topics. The 2009 AHRQ Series Paper mentioned above outlined the EHC Program selection criteria against which all nominations are evaluated (see Table 1).¹ Application of these criteria allows selection of topics for research reviews that (1) fit within the mandate and priority conditions of the EHC Program, (2) are important to the U.S. population and health care system, (3) are not already covered by a high-quality review,³ (4) represent a large enough evidence base to be feasible for a new review, and (5) have potential for significant clinical impact. The appropriateness criteria are specific to the EHC Program and seek to align selection of topics for systematic review with the overall purpose and mandate of the EHC Program. The other criteria are more generalized and could be applied to the research topic selection activities of other programs, along with the majority of the processes for topic nomination development described below.

Ongoing evaluation and revision are integral parts of the topic nomination development process. As part of this ongoing evaluation, AHRQ staff and Evidence-based Practice Centers (EPCs) conducting topic nomination development were asked to complete an anonymous evaluation of the topic nomination development processes in 2011. There was general agreement among those completing the evaluation that having consistent processes, forms, and criteria that can be used across centers are the most valuable aspects of the current topic nomination development process. EPCs and AHRQ staff identified development and evaluation of nominations that are too broad, vague, or ill-suited to the existing process for selecting research reviews (e.g., nominations for new research) as a challenge (Figure 1). Nominations vary greatly in terms of clarity, the nominator’s perspective, clinical condition, and scope. The 429 nominations submitted to the EHC Program from March 2008 to February 2012 represent a wide variety of clinical conditions (Figure 2) and the perspectives of a diverse set of nominators, including patients/consumers, clinicians, researchers, policymakers/payers, professional associations, and industry. The methods for topic nomination development described below have been developed and refined to address this wide variety of nominations and produce the necessary information for all nominations to guide topic selection.

Figure 1. Challenging diversity of topic nominations

PICO = populations, interventions, comparators, and outcomes

Figure 2. Nominations by priority condition (March 2008 to February 2012)

EHC = Effective Health Care
*None: Do not represent any clinical condition (e.g., methods topics) or represent a condition that is not a current priority condition for the EHC Program (e.g., Morgellon's disease, laser burn imaging)
†Most/all: Cross-cutting areas such as care delivery and management

Topic Nomination Development

The goal of topic nomination development is to apply a consistent, transparent process for evaluating all nominations against EHC Program selection criteria to inform the selection of topics for systematic reviews.

Topic Nomination Development Team

Topic nomination development is typically conducted by a small team consisting of a team lead, research associate, librarian, and clinical team member. The team lead is often a doctorally-trained person with a strong epidemiology, health services research, and systematic review background who provides guidance on the overall content and logic of topic briefs. The research associate is usually a master’s level or higher researcher with an epidemiology, biological sciences, or public health background. S/he does the bulk of the work, including the background searching, definition of the topic scope, documentation of the existing guidance, synthesis of the systematic review search and feasibility scan, and evaluation of the topic’s fit with the EHC Program selection criteria. A master’s level research librarian conducts the systematic review searches and feasibility scans.

The team should also include a generalist clinical team member with expertise in systematic reviews. This team member dedicates 1–5 hours for each topic nomination answering questions from research associates, consulting clinical specialists, and reviewing topic briefs. This team member helps interpret the nomination and clarifies practice variation, clinical uncertainty, appropriate comparators, important subpopulations and outcomes, and other aspects of the topic necessary to understand the current practice or health policy context underlying the need for synthesized research. Generalist physicians can address many questions, supplemented by specialist input for clinical issues not typically handled in primary care. After completion of the topic brief, it is extremely helpful to ask this clinical team member to review the logical flow of evidence that supports the team’s recommendation for the topic’s disposition. Clinical team members can also help identify potential partners for topics.

Topic Nomination Development Process Overview

The topic nomination development process begins with the receipt of a nomination via the EHC Program Web site (https://effectivehealthcare.ahrq.gov/get-involved/suggest-topic/). The steps in this process are geared towards evaluating how a topic nomination fits the aforementioned EHC Program selection criteria (Table 1). The five main domains of criteria include (1) the appropriateness of a topic, including how it fits within the mandate and priority conditions of the EHC Program (Figure 2); (2) the importance to the U.S. population and health care system; (3) desirability (or lack of duplication) of a new systematic review; (4) feasibility; and (5) potential impact of a new research review for the topic. These five domains of the criteria are evaluated in order (Table 1). If the topic meets none of the appropriateness criteria, the other criteria are not considered. If a topic meets appropriateness and importance criteria, but is duplicative with an existing review or is not feasible for a new review, the potential impact of a new review is not relevant so these criteria are not assessed. Evaluating each nomination’s fit with these selection criteria using a consistent, transparent process helps ensure that all nominations are treated equitably.

Figure 3 shows the steps in the topic nomination development process; each step is discussed in detail below. The process is not as linear as this diagram implies; many of the steps in the process are integrated and may occur simultaneously. The process includes background searching, definition of the topic scope, a search for systematic reviews, documentation of existing guidance on the topic, a feasibility scan for primary research, and completion of a three part topic brief summarizes information relevant to the topic’s evaluation against EHC Program selection criteria.³

Figure 3. Topic nomination development and topic selection processes

The topic brief, comprising an Existing Guidance document, Selection Criteria table, and Cover Sheet, allows consistent organization of information to assist orderly, efficient consideration of the topic. The Existing Guidance document lists available and in-process research on the topic. The Selection Criteria document is a table detailing how the topic meets or does not meet each of the EHC Program selection criteria in Table 1. The Cover Sheet, typically seven to eight pages long, includes a description of the nomination, background information on the topic, and a summary of the topic’s fit with EHC Program selection criteria. Thus, the three main documents are related—the existing guidance on a topic helps determine the topic’s fit with the selection criteria, and the topic’s fit with the selection criteria is summarized in the Cover Sheet. Supplementary materials, such as summary tables of existing systematic reviews and/or clinical guidelines on the topic, may be included as appendices in the topic brief.

Figure 3 also shows the steps in the topic selection process. Selection of topics for systematic review occurs at monthly meetings of a “topic triage” group representing stakeholder and scientific perspectives, as well as the programmatic authority vested in AHRQ, and is informed by the information presented in the topic briefs. Results of the topic selection process are described in a Nomination Summary Document (described below) to communicate the disposition of nominations to the public.

Prior to nominations beginning topic nomination development, the Scientific Resource Center and AHRQ conduct an initial assessment of nominations to determine whether they meet EHC Program appropriateness criteria and contain enough information to proceed with topic nomination development. Some nominations to the EHC Program lack sufficient information to evaluate the topic against EHC Program selection criteria so do not undergo topic nomination development. Many of these nominations consist of only a few words or are extremely broad, with no indication of how the nomination could be translated into a feasible topic with well-defined populations, interventions, comparators, and outcomes (PICO). In such cases, we refer to an established checklist for the minimum amount of information needed to evaluate a nomination. This set of minimum information includes the populations, interventions, comparators, and outcomes of interest to the nominator, as well as the policy and/or clinical context. It is sometimes possible to logically conclude what these parameters are, based on the literature and consultation with clinical members of the topic nomination development team, so the nomination can go through the topic nomination development process. In other cases, further input from the nominator is necessary, but is not always possible if the nomination was made anonymously or the nominator does not respond to requests for clarification. In addition, the scope of some nominations may be too broad to develop given limited EHC Program resources.

Background Searching

After reviewing the information provided in the topic nomination, topic nomination development begins with a brief background scan to get a firm understanding of the context, clinical practice and health implications of the topic, what tests or treatments are available, the terms and language used to describe the topic, and affected individuals or populations (with attention to health disparities). Suggested sources for this search include PubMed for recent narrative reviews, clinical library sources (e.g., DynaMed, First Consult, MD Consult, BMJ Clinical Evidence), U.S. FDA Web site, Centers for Disease Control (CDC) and National Center for Health Statistics fast stats for epidemiology and health statistics, National Cancer Institute Physician Data Query, and relevant professional society Web sites. This background information informs the nomination’s fit with the appropriateness and importance criteria. This information also aids in the determination of whether the scope of the topic as described in the nomination is clinically appropriate and relevant, and informs development of the preliminary PICO for the topic, which guides the remainder of the topic’s development.

Definition of the PICO/Scope

In addition to variation in clinical context and nominator perspective, nominations differ greatly in their scope (i.e., the parameters of the research question to be included as defined by the PICO). A poorly defined PICO can lead to development of a topic that may miss important populations, lack clinical relevance or logic, or is not feasible for a systematic review. During topic nomination development, a number of different sources can be used to further define a topic’s PICO, including published literature, clarification from the nominator, and consultation with clinical experts; these sources are used to ensure that the PICO is clinically logical and relevant, includes a realistic set of parameters for a potential review, and would result in a review that is useful to important stakeholders. For the purposes of topic nomination development, the formulation of a PICO is done routinely; timing and setting(s) (PICOTS) may be included if these details are important to the context of the nomination. The literature usually suggests the relevant parameters for a topic, which are compared with the PICO proposed by the nominator. Substantial differences can be reconciled during discussions with the nominator to ensure that the nominator’s interests are reflected in the PICO, and consultation with clinical experts can serve to confirm or revise the PICO to be certain that it fits with the current clinical context.

A vague PICO also presents scoping challenges and may lead to a review that is too inclusive or too exclusive. A narrow PICO may reflect proprietary or individual interests that are not broadly generalizable. A broad PICO is often too imprecise for careful consideration, masking important questions or topics for systematic review. There is an inherent tension in the scoping process between fidelity to the original nomination and broadening the scope of the topic to be more relevant to a larger audience. Discussions with the nominator and other important stakeholders serve to ensure that the nominator’s interests are clearly articulated in the topic brief along with the evidence needs of other key stakeholders for the topic, such as clinicians or policymakers. For example, a nomination on physical therapy for acute ambulatory conditions was too broad to develop or evaluate against EHC Program selection criteria because the interventions, assessments, and outcomes are heavily dependent upon the specific condition for which there is an indication for physical therapy. The physical therapy literature helped identify the most common conditions for which physical therapy is used. Conversations with the nominator facilitated by a clinical expert in the field of physical therapy clarified that the condition of most interest to the nominator was knee pain secondary to osteoarthritis. Further consultation with the nominator narrowed the nominator’s questions to focus on issues such as the relationship between intermediate outcomes and improvement in patient functional performance. On the other hand, a nomination on the effectiveness of a combination of IV diphenhydramine, ketorolac, and metoclopramide in addition to saline intravenous fluids for treating acute migraines in emergency settings was too narrow based on the lack of literature on this drug combination and clinical input. The topic was expanded to more broadly address interventions for the treatment of acute migraines, thus scoping the topic in a manner suited to a review that would be useful for multiple stakeholders, including patients, clinicians, policymakers, and guideline developers. Another common scoping problem is that children and other relevant subpopulations may be omitted in the nomination.

Determination of the clinical context or clinical logic has also been a challenge. For example, in a nomination on benign prostatic hyperplasia, the nominator was mainly concerned with the use of complementary and alternative medicine for benign prostatic hyperplasia for the reduction of prostate-specific antigen levels. However, prostate-specific antigen levels are not a clinically relevant outcome for the topic. Our workup was revised to reflect relevant outcomes we found in the literature and confirmed by clinical consultation.

During consultations with clinical experts, it is useful to ask questions such as where the nominator’s intervention of interest falls within the usual management of the given condition; what other interventions are potential comparators; what outcomes are clinically meaningful for a given intervention or comparator; and whether the intervention of interest is currently used in clinical practice and, if so, how often and in what patient populations it is used most widely.

To aid topic selection decisions, a well-defined PICO should include the following—

• Details on the population (e.g., age, sex, disease stage/severity, subpopulations of interest)
• Comprehensive list of interventions and comparators when the nominator has only provided a general category or class
• Definition of usual standard of care if used as a comparator
• List of intermediate and health outcomes, including potential benefits and harms of interventions and comparators, with particular attention to patient-oriented, clinically relevant, and long-term outcomes

Table 2 presents examples of a poorly-defined and a well-defined PICO.

Search for Systematic Reviews

Searching for literature to answer the nominator’s question is usually conducted in a sequential manner, beginning with synthesized literature identified from a formal search of medical literature databases, then research products and activities identified from searches of specific organization and agency Web sites described below under Existing Guidance Documentation, and later moving to formal searches for trials and other study designs as described below under Feasibility Scan.

Searching begins with identification of existing and in-process systematic reviews and meta-analyses. This search is conducted by a librarian, but it is helpful to provide the librarian with a list of suggested search terms, including Medical Subject Headings (MeSH) and key words, based on the initial background scan, as well as the databases to search (e.g., MEDLINE, PsycINFO, Cumulative Index to Nursing and Allied Health Literature (CINAHL)) and citations that are good illustrations of the topic (e.g., high-quality narrative or systematic review identified in initial background scan). The search strategies for existing systematic reviews are a good starting point for development of this search. Based on prior experience, a search for synthesized literature over the past 5 years is often sufficient, although a search of the past 10 years is necessary for some topics, such as those related to well-established interventions that have not been the focus of recent research activity or topics with limited existing research. In some cases, search dates are dependent upon when the technology or intervention was first developed.

After receiving synthesized literature search results from the librarian and reviewing relevant abstracts, an iterative process begins to determine if the search for synthesized research is adequate and captured the questions raised in the nomination or needs to be narrowed or refined. Citations for the most recent, relevant systematic reviews should be listed in the Existing Guidance document, including the search dates, methods, and overall fit with the nomination.

The ultimate goal of this step is evaluation of the duplication selection criterion. In order that EHC Program resources are put to the best use, the EHC Program may decide not to pursue systematic reviews on topics that are already addressed by existing or in-process high-quality reviews. Such a decision does not constitute endorsement of non-AHRQ systematic reviews, but rather the recognition that there are many important topics in health care that would benefit from systematic evidence reviews and only limited resources with which to do those reviews. The EHC Program may consider a topic as adequately covered by a recent review performed or commissioned by a U.S. government agency (e.g., AHRQ, U.S. Preventive Services Task Force [USPSTF], National Institutes of Health [NIH], Department of Veterans Affairs [VA], CDC) or an independent center, academic institution, or government (e.g., Cochrane Collaboration, National Institute for Health and Clinical Excellence [NICE], Canadian Agency for Drugs and Technologies in Health [CADTH], other center or independent group) using acceptable methodology for evidence grading and conflict of interest management. In some cases, the EHC Program may decide to undertake a review despite possible duplication for reasons such as—

• A U.S. government product is needed for development of guidelines, policy, or translational products for patients or clinicians.
• Impact will be ensured by use of the AHRQ dissemination infrastructure.
• There are potential benefits from expanding or revising the methodology or better managing conflict of interest in the existing review.
• The existing review was conducted in another country where practice patterns or epidemiology are significantly different than what would be found in the United States or conclusions are not consistent with U.S. guidelines.
• Current clinical practice diverges from consistent conclusions from recent systematic reviews.
• Existing systematic reviews have conflicting conclusions.
• The nominator confirms that current reviews do not meet stated needs.

Existing Guidance Documentation

This step focuses on searching for available and in-process research (e.g., reviews, guidelines, studies) and activities (e.g., Centers for Medicare & Medicaid Services [CMS] policies, NIH conferences) related to the topic, which is recorded in the Existing Guidance document along with the results of the more formal librarian searches for systematic reviews (described above) and primary studies (described below under Feasibility Scan). The existing guidance informs the evaluation of the topic’s fit with some of the EHC Program selection criteria, such as duplication, feasibility, and potential impact. Documentation of existing guidance on the topic typically begins while the librarian is conducting the search for systematic reviews. The sources searched for existing guidance include—

• In-process and completed AHRQ products
  • Evidence reviews (from Evidence-based Practice Center and EHC Programs)
  • Technology assessments
  • USPSTF recommendations
  • Developing Evidence to Inform Decisions about Effectiveness (DEcIDE) Network projects
  • Translational products (e.g., patient and clinician guides)
• NICE guidelines
• Cochrane Collaboration reviews and protocols
• Drug Effectiveness Review Project drug class reviews
• Health technology assessments (from Centre for Reviews and Dissemination database, which includes content from the International Network of Agencies for Health Technology Assessment and 20 other health technology assessment organizations)
• PROSPERO database of registered systematic reviews and protocols
• VA products (technology assessments from the VA Technology Assessment Program, systematic reviews from the Evidence Synthesis Program, and VA/Department of Defense Clinical Practice Guidelines)
• NIH consensus statements and upcoming conferences
• CDC Guide to Community Preventive Services publications and recommendations for public health topics
• CMS policies and coverage updates
• ClinicalTrials.gov for active, recently completed, or recruiting studies
• National Guideline Clearinghouse at guidelines.gov and other searches (e.g., PubMed) for clinical practice guidelines

In order to show the breadth of existing or in-process AHRQ activities in the clinical domain, AHRQ products that are related to but don’t directly overlap with the nomination should be documented. For example, there may not be any AHRQ products addressing a nomination for complementary and alternative medicine (CAM) therapies for sleep apnea, but all AHRQ products on sleep apnea should be documented. A comprehensive list of related AHRQ products can also serve as a reference of those who have worked on similar topics and could potentially serve as experts during later stages of the topic nomination development, refinement, or review process. It is also helpful to document the Key Questions for all relevant AHRQ reviews to illustrate whether the existing AHRQ reviews appear to address the full scope of the nomination.

Feasibility Scan

After the search for systematic reviews, a search for controlled trials is conducted by the librarian to determine the feasibility of a new review on the topic. The dates for this scan can begin from the last search date of the most recent high-quality systematic review. The results of the feasibility scan will show whether the most recent systematic review fully covers the topic. If there are landmark studies or a significant number of studies that have not been captured in the most recent systematic review, the need for a new review on the subject should be considered. The recent introduction of new interventions or technologies for which there is published evidence may also underscore the need for a review on the topic. In the absence of a recent high-quality systematic review, a feasibility scan of the last 5 or 10 years will be needed to determine the adequacy, type, and volume of primary research recently published on the topic that would be available for a review. For those topics with a very limited literature base, a search may need to be completed without date limits. If very few controlled trials are found or for topics that are not appropriate for controlled trials, the feasibility scan should be expanded to include study designs such as case-control, cohort, before-after, case series, and other observational designs. The sufficiency of available studies to warrant a review will partly depend on the topic. For topics where controlled trials are possible but only observational studies are available, a review may not have significant clinical impact until there is higher quality evidence on the topic. For other topics, such as those focused on potential harms, data from observational studies may be sufficient for a review.

The aim of the feasibility scan is only to provide a sense of the volume of the available literature that could potentially be included in a review. This scan is geared toward efficiency and is not meant to be as rigorous as a search for primary literature that is conducted during the course of a systematic review. For topics that are selected for a systematic review, more precise searches will be conducted during the conduct of the systematic review that reflect scope revisions made during the topic refinement process. Synthesis of the feasibility scan results is limited to a summary of the number of relevant studies available for inclusion in a review and documentation of any landmark studies. Unlike synthesis of the results of searches for primary research conducted during a systematic review, synthesis of feasibility scan findings during topic nomination development does not include quality rating of articles or an assessment of the results of the studies. High volume or very broad feasibility scan results are a challenge for some nominations. These cases require organization of the results by the most important parameters of the particular topic, such as setting, population, outcomes, comparators, study design, or length of followup, to aid in the determination of whether the existing literature covers all aspects of the nomination. For example, the feasibility scan results for a topic on fibromyalgia treatment were categorized by the type of intervention studied, including pharmacological, psychological, exercise, and CAM therapies, and for a topic on seasonal allergy treatments they were divided by studies addressing adults versus children.

Completion of Topic Brief

Existing Guidance Document

At this point, the Existing Guidance document should be completed. All available and in-process research identified from the search for systematic reviews, feasibility scan, and searches of specific organization and agency Web sites described above under Existing Guidance Documentation should be listed in the Existing Guidance document.

Selection Criteria Document

Details of how the topic meets or does not meet each of the EHC Program selection criteria should be recorded in the Selection Criteria document. The appropriateness and importance criteria are informed by background searching on the topic, the duplication criterion is determined by the results of the search for systematic reviews, and the feasibility criterion is based on the results of the feasibility scan for primary research. The potential of a new review to have significant health impact is the last set of criteria considered and is influenced by the amount of clinical uncertainty and practice variation surrounding the topic. The need for translational products geared toward patients, clinicians, and policymakers also affects the potential for impact from the review. If recent high-quality reviews and/or practice guidelines exist, the added value of an AHRQ review on the topic should be addressed.

Cover Sheet

The Cover Sheet includes a description of the nomination, comprising a summary of the nominator’s interests, the nominator’s PICO, the policy or clinical context of the nomination, and any Key Questions provided by the nominator. A section on key considerations and points for discussion contains the following information:

• Summary of nomination’s fit with appropriateness and importance criteria
• Disease burden
• Description of the condition
• List of relevant drugs, devices, therapies, technologies, or services
• Clinical logic of the nominator’s PICO
• Reason for any changes to the scope of the original nomination
• Clinical uncertainty and practice variation
• The most recent, relevant clinical practice guidelines on the topic, including a summary of conflicting recommendations, areas lacking sufficient evidence for a recommendation, and whether the guidelines are based on a systematic review
• Existing high-quality systematic reviews beginning with AHRQ products, including the number of studies included and a statement of whether the reviews agree or disagree in their conclusions
• How the topic is or is not covered by existing work
• Results of the feasibility scan, including the number of in-process studies identified on ClinicalTrials.gov to give a full picture of how much literature would be available for a new review and if the topic represents an active area of ongoing research
• Related Institute of Medicine comparative effectiveness research priorities⁴
• Suggestions for individuals and organizations to consult if the topic is voted forward for a review or other EHC Program product
• Concluding bullet on the rationale for the team’s recommendation on the topic’s disposition, including assessment of the potential impact of a new research review if applicable

Key points and considerations in the Cover Sheet should have a logical flow leading to the team’s recommendation for the disposition of the topic (described further below). If there are multiple relevant categories within the nomination (e.g., diagnosis and treatment, subpopulations such as children and adults), the topic brief should be clearly divided into sections with subheadings that identify each area of the nomination. Table 3 lists questions that should be considered when summarizing information on the topic in the Cover Sheet. This list is divided into questions relating to the PICO, the nominator, clinical practice, existing literature and feasibility, impact, and program/product fit.

The final step in completing the topic brief is assigning a team recommendation for the disposition of the nomination based on its fit with the EHC Program selection criteria, which is voted on by a topic triage group during topic selection (see set of potential topic dispositions in text box under Topic Selection below). For nominations with multiple aspects addressed in the topic brief (e.g., diagnosis and treatment), it is often necessary to assign separate recommended dispositions for each aspect of the topic. A topic’s disposition may reflect the fact that it does not meet appropriateness or importance criteria, is already covered by an existing review, or is not feasible for a new review. For some topics, ongoing research or activities may be underway that impact the timing for developing the topic. For example, there may be large, in-process clinical trials whose results will heavily influence any conclusion from a systematic review. In such cases, the Cover Sheet should include details on what the ongoing activity is, how it will affect the topic’s disposition, and the date when the results are expected to be available so the topic can be reconsidered at that time.

There are a number of different AHRQ products for which topics may be selected, including a technical brief, comparative effectiveness or effectiveness review, or update to an existing AHRQ review. The context and purpose of each of these products is described in Table 4. In addition to these products, topics are sometimes recommended for other activities, such as referral to the team conducting an in-process review on the topic to be considered for inclusion in the review’s scope, for refinement as a review of reviews, or for a potential methods project. When the topic brief is completed, its contents should be discussed with members of the topic nomination development team who have clinical expertise to ensure that the team’s recommendation for the topic’s disposition is clinically logical.

Stakeholder Engagement

Table 5 shows the points of stakeholder engagement during topic nomination development. In this context, stakeholders are defined as clinicians, policymakers, guideline developers, professional societies, consumers, and patients; the individual nominator may represent one or more of these stakeholder groups. Input from nominators is sometimes needed to clarify the population, interventions, or outcomes of interest when the nomination includes a broad scope or less-defined PICO. In addition, if a topic is deemed duplicative with in-process or existing reviews or programmatic activities, it is sometimes important to verify that the existing products meet the nominator’s needs. This can occur before the topic is presented to the topic triage group, after presentation to the topic triage group but before final disposition of the topic, or during topic refinement, and EHC Program staff usually determine the appropriate time for this engagement with the nominator. As mentioned above, discussions with local, regional, or national clinical experts are often necessary to appropriately scope a topic at the beginning of topic nomination development, and these discussions occur at the discretion of the topic nomination development team. Experts are generally identified by the clinical team member, who communicates with these experts via email or phone. EHC Program staff may provide guidance to the topic nomination development team as to whether and when the nominator, policymakers, or professional society representatives should be consulted.

Stakeholder input can often be solicited via email, although longer conversations are sometimes required that are better handled on the phone after an initial request for information over email. More formal telephone conferences facilitated by clinical team members are occasionally appropriate to clarify nominations from professional societies or policymakers. For topics voted forward for a systematic review, it can be useful to establish a partnership with a group that will develop clinical practice guidelines based on the review to ensure clinical impact and facilitate dissemination. In such cases, communication with the partnering organization is essential to ensure that the timing of the review’s completion is coordinated with guideline development.

Efficiency

The need for and importance of topic nomination development to identify the most important topics for systematic review is unquestionable. But allowing a longer timeline for in-depth topic nomination development comes at the expense of extending the time between submission of nominations and their disposition. Ultimately, spending more time on topic nomination development may lengthen the timeline for completion of any commissioned reviews and translational products or clinical practice guidelines produced from the reviews. Topic nomination development for the EHC Program is time intensive because it requires a universal perspective given the public funding for products that could be important to several segments of the population. As mentioned above, nominations to the EHC Program cover a broad range of clinical conditions and are submitted by a wide array of stakeholders with varying perspectives and needs, thus, a significant amount of effort is required to find a clear context for each topic. The time needed to complete the steps in the current topic nomination development process varies considerably depending on the complexity and breadth of the topic nomination, with the total time for completion of a topic brief ranging from 16 to 68 hours. This estimate does not include time needed for feedback loops such as going back to the nominator for clarification or getting expert feedback. The EHC Program receives an average of nine nominations per month. Eight nominations on average are triaged per month, and the mean time from nomination submission to triage is 7 months. Balancing efficiency with the need for a comprehensive, effective process will continue to be a challenge and will require exploration of potential process revisions, such as instituting a streamlined process for nominations that are clearly covered by existing programmatic activities (e.g., in-process EHC Program reviews, USPSTF recommendations).

Evaluation of Nominations for New Research

Another challenge encountered in topic nomination development for the EHC Program is presented by nominations for new primary research, which are ill-suited to the existing process for selecting topics for research reviews. In 2011, the Scientific Resource Center and AHRQ adapted the EHC Program’s process for evaluation of topics for systematic review to distinguish topics appropriate for potential new research. Potential new research topics are characterized by the existence of a significant research gap that is important to clinician, policymaker, and/or patient decisionmaking. In this process, research gaps and the potential impact of new research on clinical practice and policy are identified by examining the following:

• Systematic reviews⁵ and editorials for any discussion of research gaps
• Clinical practice guidelines for areas reported as having insufficient evidence to make a recommendation
• Recently published studies to determine to what extent research gaps have been filled
• In-process studies and newly funded Federal research or funding opportunities to get a sense of whether it is an active area of research
• Coverage determinations that provide a perspective on uncertainty surrounding a topic.

Clinical consultation is used to confirm a lack of evidence and the need to rely solely on clinical judgment. This background information on the need for new research on a topic is included in the Nomination Summary Document that is sent to the nominator and posted on the EHC Program Web site (see below). Evidence generation programs at AHRQ, such as the DEcIDE Network, as well as researchers, funders, and programs outside of AHRQ, can access this information to support their primary research agendas.

Topic Selection

Selection of topics for further development as a research review occurs during monthly “topic triage” meetings. During each meeting, topic nominations are presented to a topic triage group consisting of members from various components of the EHC Program and AHRQ. These members represent various stakeholder and scientific perspectives, as well as the programmatic authority vested in AHRQ. At the beginning of each topic triage meeting, voting members are asked to disclose any potential financial, business, professional, or intellectual conflicts of interest related to any of the topics that will be discussed and voted on during that meeting. Members disclosing potential conflicts of interest are asked to abstain from voting on the relevant topic(s) and in some cases may recuse themselves from any discussion on the topic. After a brief presentation of the topic by a member of the topic nomination development team and discussion, the facilitator polls all members for a vote on the recommended disposition of the topic. Potential dispositions that can be recommended for topics are shown in Box 1 below. Group members are asked to indicate their enthusiasm for the recommended action on a scale of 1 to 5 (1 = no enthusiasm, 3 = neutral, 5 = complete enthusiasm). Recommendations with an average vote of less than 3 result in further discussion to arrive at an alternate disposition for another vote. These recommendations are not binding, but are highly weighted in the final decision by AHRQ as to the research topics selected for further development as a research review, along with considerations of other programmatic needs and resources.

Box 1. Potential topic dispositions

• Topic is outside the purview of the EHC Program and does not meet EHC Program appropriateness criteria
• Topic is already addressed by existing research review(s) or programmatic activities
• Topic is important, but current research is too limited for appropriate program product development
• Topic should be tabled because ongoing research or activities are underway that impact the timing for determining the topic’s disposition
• Topic will return to a future topic triage meeting with more information that is necessary to determine the topic’s disposition, such as nominator or stakeholder feedback
• Topic will go forward for further refinement as a systematic review or technical brief
• Topic will be considered for potential new primary research

Topic Selection Results Reporting

Transparency is an important aspect of the topic nomination development and selection processes. General information about the topic nomination development and selection processes is available on the EHC Program public Web site, including health care service and patient population priorities, priority conditions, and the EHC Program selection criteria (https://effectivehealthcare.ahrq.gov/about/epc/nomination/). All nominations submitted to the EHC Program are also posted on the public Web site. In addition, decisions regarding whether a nomination is selected for a systematic review are briefly summarized in a one to three page Nomination Summary Document. This document is completed for all nominations and is sent to the nominator and posted on the EHC Program Web site. This document includes the following:

• Results of topic selection process and next steps
  • Summary of disposition of topic (e.g., topic does not meet EHC Program appropriateness criteria, topic is covered by an existing research review or programmatic activities, topic is not feasible for a systematic review, topic will go forward for refinement as a new or updated systematic review)
  • For all reports that are considered as addressing the topic, a full citation, with a link to the report if publically available
  • For topics that are addressed by in-process AHRQ reports, a link to sign up for notification when relevant in-process AHRQ reports are posted
  • For topics going forward as a systematic review, a statement that the final scope of a review may change during topic refinement, and a link to sign up for notification when Key Questions are posted for public comment
• Topic description
  • Nominator identified by category only (e.g., individual, health care professional association, public payer, organization)
  • Nomination summary, including PICO
  • Key Questions provided by the nominator
• Considerations
  • How topic fits with EHC Program selection criteria, with link to all criteria
  • Rationale for topic disposition (e.g., why topic does not meet selection criteria, how a topic is covered by existing review[s], summary of insufficient evidence to address topic, importance and potential impact of topics going forward as a systematic review)
  • Key Questions or inclusion criteria for all reports that are considered as addressing the topic

Future Directions

Several potential avenues for expansion of topic nomination development and selection activities within the EHC Program exist. The EHC Program continues to work with stakeholders to identify issues of high interest to the general public, areas where evidence gaps hinder high-quality care, and topics where systematic review might clarify care for high-priority populations. This stakeholder engagement in topic identification often results in a number of topics in a single clinical domain that have been given a high priority for systematic review by a diverse set of stakeholders. The number of topics voted forward for a research review within the EHC Program is likely to grow significantly, making it necessary to go beyond selection of topics to prioritization of the topics expected to have the highest clinical impact.

In consideration of this potential expansion of selected topics, the EHC Program may explore prioritization techniques such as incorporating a value of information (VOI) approach or minimal analysis as a sequential step after topic selection to prioritize among topics voted forward by the topic triage group.6 VOI may also be considered for prioritizing among multiple research topics addressing a single clinical condition identified in topic identification projects, or for assessing the need for new primary research. This quantitative approach includes a conceptual VOI analysis that considers data, some of which could be taken from the topic brief, including the number of patients that might potentially be affected by a new research review on the topic; the distribution of possible health outcomes, costs, and net benefits of alternative health interventions; reduction in uncertainty from a new review; the likelihood that a review would change clinical practice; and the durability of a review’s relevance. One unresolved difficulty in applying a VOI analysis would be determining relative value across the breadth of topics that are selected to go forward, including 13 priority conditions, multiple subgroups (e.g., adults, children, minorities, acute, chronic), and a range of stakeholder perspectives.

Another potential revision to the current topic nomination development process is inclusion of information about how a nomination relates to the national priorities for comparative effectiveness research outlined by the Patient-Centered Outcomes Research Institute (PCORI). The EHC Program currently conducts a type of research gap analysis when evaluating nominations for new primary research. The EHC Program may consider expansion of the methods for evaluation of nominations for new research to include a more formal evidence gap analysis, such as that proposed by PCORI.⁵ Finally, the EHC Program will soon incorporate evaluations of the need to update reviews conducted by the EHC Program into the current topic selection process.

Transparency of the topic selection process will soon be further enhanced by the posting of Cover Sheets and Existing Guidance documents on the EHC Program public Web site. Because these documents will be available to the public, consistency across topics in the information presented will be especially important. If the EHC Program implements a prioritization process for selected topics, clear communication of prioritization decisions to the public will need to be considered.

Box 2. Key points

• The goal of topic nomination development is to apply a consistent, transparent process for evaluating all nominations against EHC Program selection criteria to inform the selection of topics for systematic reviews.
• Application of the selection criteria allows selection of topics for research reviews that fit within the mandate and priority conditions of the EHC Program, are important to the U.S. population and health care system, are not already covered by a high-quality review, represent a large enough evidence base to be feasible for a new review, and have potential for significant clinical impact.
• The process includes background searching, definition of the topic scope, a search for systematic reviews, documentation of existing guidance on the topic, a feasibility scan for primary research, and completion of a three part topic brief that summarizes information relevant to the topic’s evaluation against EHC Program selection criteria.

Conclusion

Given the extent of health care needs and constraints on the resources available to address these needs, methods to identify the most important topics for synthesized research are essential. The consistent, transparent process for evaluating topics described in this paper is designed to identify the topics most appropriate for a review by the EHC Program. This process was developed and refined over the past 4 years and has been applied to more than 400 nominations representing a wide range of clinical conditions and nominator perspectives. Although some of the selection criteria are specific to the EHC Program, many of the criteria and topic nomination development processes used by the EHC Program are generalizable and could inform the research topic selection activities of other programs.

References

1. Whitlock EP, Lopez SA, Chang S, et al. Identifying, selecting, and refining topics. In: Agency for Healthcare Research and Quality. Methods Guide for Comparative Effectiveness Reviews. Rockville, MD: 2008. PMID: 21433404.
2. The refinement of topics for systematic reviews: Lessons and recommendations from the Effective Health Care Program. Rockville, MD: Agency for Healthcare Research and Quality. [In press].
3. Institute of Medicine. Finding what works in health care: standards for systematic reviews. Washington, DC: National Academies Press; 2011.
4. Institute of Medicine. Initial national priorities for comparative effectiveness research. Washington, DC: The National Academies Press; 2009.
5. Carey T, Yon A, Beadles C, et al. Prioritizing future research through examination of research gaps in systematic reviews. (Prepared for the Patient-Centered Outcomes Research Institute). 2012.
6. Meltzer DO, Hoomans T, Chung JW, et al. Minimal modeling approaches to value of information analysis for health research. Methods future research needs. Report No. 6. (Prepared by the University of Chicago Medical Center and the Blue Cross and Blue Shield Association Technology Evaluation Center Evidence-Based Practice Center under Contract No. 29007-10058.) AHRQ Publication No. 11-EHC062-EF. Rockville, MD: Agency for Healthcare Research and Quality; 2011. PMID: 21977528.

Glossary

Comparative effectiveness and effectiveness reviews—research reviews that outline the effectiveness—or benefits and harms—of treatment options.

Feasibility scan–a brief search for primary studies to evaluate the sufficiency of available evidence to warrant a new review on the topic.

PICO–populations, interventions, comparators, and outcomes.

PICOTS–populations, interventions, comparators, outcomes, timing, and setting.

Technical brief–a research review that explains what is known and what is not known about new or emerging health care tests or treatments.

Topic brief–a summarization of information obtained as a result of the topic nomination development process consisting of the Cover Sheet, Selection Criteria document, and Existing Guidance document.

Topic identification–receipt of nominations for a specific topic by the EHC Program.

Topic nomination–topic suggestion from individual or group for a comparative or clinical effectiveness research review.

Topic nomination development–evaluation of a nomination’s fit with EHC Program selection criteria using a process that includes background searching, definition of the topic’s scope, a search for systematic reviews, documentation of existing guidance on the topic, a feasibility scan for primary research, and completion of a three part topic brief.

Topic prioritization–relative ranking of topics according to the expected level of clinical impact from a review.

Topic refinement–following topic selection, further scoping of a topic in response to input from key stakeholders and technical experts that culminates in the development of Key Questions and an analytic framework to guide the technical conduct of the review and define the targeted patient populations, interventions, comparators, outcomes, timing, and clinical settings.

Topic selection–selection of topics for further development as a research review.

Topic triage group–a group representing stakeholder and scientific perspectives, as well as the programmatic authority vested in AHRQ, which selects topics for further development as a research review.

Topic triage meeting–monthly meeting during which topics are selected for further development as a research review.

Update review–a research review that focuses on the original questions of a previously completed AHRQ research review.