Skip to main content
Effective Health Care Program

Evaluation of the AHRQ Healthcare Horizon Scanning System

Research Report

This report is available in PDF (2.5 MB) only. People using assistive technology may not be able to fully access information in this file. For additional assistance, please contact us.

Executive Summary

With funding from the American Recovery and Reinvestment Act of 2009 (ARRA) and the Patient-Centered Outcomes Research Institute (PCORI) Trust Fund, the Agency for Healthcare Research and Quality (AHRQ) implemented and continues to oversee the Healthcare Horizon Scanning System. The Horizon Scanning System seeks to identify, monitor, and evaluate new and emerging technologies, off-label uses, and new uses of existing technologies and services that may have a significant clinical, system, or cost impact on the provision of health care in the United States. The Horizon Scanning System is the first public horizon scanning system to focus on emerging health interventions and innovations within the unique political, regulatory, cultural, and economic context of the U.S. health care system.

The goals of this evaluation are to assess the performance of the three primary functions of the Horizon Scanning System and to discern ways to improve its processes. The evaluation used multiple methods, metrics, and data sources to address four research questions relevant to scanning for emerging health care interventions:

  1. How successfully did the AHRQ Healthcare Horizon Scanning System identify and prioritize interventions for monitoring?
  2. How successfully did the Horizon Scanning System monitor the selected target interventions?
  3. How accurately did the Horizon Scanning System assess the potential for high-impact of the interventions?
  4. How can processes for identification, prioritization, monitoring, and assessment of potential for high-impact of the interventions be improved?

Methods

We used multiple methods, metrics, and data sources to address these questions. Using Status Update Reports, which ECRI produces and AHRQ publishes on its Web site, we examined whether there were any potentially late-identified interventions the Horizon Scanning System should have identified earlier in their development. Via semistructured interviews with staff and domain experts, we received detailed input on what has worked well or was problematic across every stage of the Horizon Scanning System protocol. We also surveyed experts to aid in assessing the accuracy, completeness, and usability of the Potential High-Impact Intervention reports (also produced by ECRI and published by AHRQ). In addition to the expert survey, we also conducted a survey of stakeholders to evaluate the credibility and usability of Potential High-Impact Intervention reports. Finally, we evaluated the variability in the high-impact potential (HIP) assessments using the metric of the proportion of Potential High-Impact Interventions for which the HIP assessment was unchanged from 2013 to 2014. We also evaluated the usability of the HIP assessment through the stakeholder survey. These analyses were supplemented by perspectives of the Horizon Scanning System staff on the activities to collect and synthesize expert comments to develop an overall assessment of HIP.

Key Findings

The findings of this evaluation present evidence that the Horizon Scanning System effectively identified, monitored, and assessed the potential for high-impact of emerging health care interventions.

Identification of interventions

  • Of a sample of 200 interventions subject to either Medicare coverage or FDA approval, the study identified only 2 interventions that were identified by the Horizon Scanning System after the receipt of FDA approval.
  • One instance was due to internal operations (a change in the criteria for a priority condition and a determination by the Horizon Scanning System analyst that the intervention failed to meet the Horizon Scanning System criteria).
  • The other instance was due to the absence of public information on this intervention until attainment of FDA approval—a factor outside of the Horizon Scanning System’s control.

Monitoring of interventions

  • The 26 experts who provided feedback on Horizon Scanning System reports generally found no substantive inaccuracies and no missing information that they viewed as important.
  • These experts indicated that the reports’ descriptions of the clinical use of interventions was consistent with the prevailing view.
  • A majority of the 65 stakeholders who reviewed a Horizon Scanning System report found the reports highly credible and relatively easy to understand.
  • Among the stakeholders for whom the intervention was relevant to their work, most noted it was easy to find the information of interest.

Assessment of intervention potential for high-impact

  • Surveys of 7 cancer experts found high agreement with interventions rated by the Horizon Scanning System as having potential for high impact. However for interventions rated in the Horizon Scanning System as having no potential for high- impact, some were assessed by responding experts as having some high impact potential.
  • A majority of the stakeholders indicated that they found the report section on the high- impact potential and summary of expert comments useful.
  • Likewise, a similar proportion of the stakeholders indicated the high-impact potential rating was consistent with the other information in the reports.

Potential improvements to the Horizon Scanning System

In line with the findings above, interviews of external experts and internal staff identified several potential improvements. These included: refining the inclusion criteria for the Functional Disability and Limitations condition area, including additional information in the topic profiles, providing more guidance to experts on applying the rating scale, addressing the difficulty of assessing the potential impact of interventions on health disparities, and implementing outreach to raise the visibility of the Horizon Scanning System among the general public and experts.