Skip Navigation
Department of Health and Human Services www.hhs.gov
 
 

Slide 1

Engaging Stakeholders in the
Effective Health Care Program
Information and tools for researchers and investigators

Speaker notes: The Agency for Healthcare Research and Quality presents the following e-learning tool “Engaging Stakeholders in the Effective Health Care Program” to provide researchers and investigators with foundational information on, and tools for, engaging a broad range of stakeholders in the Effective Health Care Program.  The information presented is adapted from content developed by the Scientific Resource Center at the Oregon Evidence-based Practice Center.

Slide 2

Full Course Outline

  • Module I: AHRQ’s Vision for, Theory, and Principles of Stakeholder Engagement
  • Module II: Stakeholders and Stakeholder Engagement
  • Module III: Points of Engagement in the EHC Program
  • Module IV: Skills for Successful Engagement

Speaker notes: This is the 3rd module as part of a four module e-learning series:

Slide 3

Module III
Points of Engagement in the EHC Program

Slide 4

What to Expect in Module III

Module III will outline opportunities for stakeholders to be engaged in the Effective Health Care (EHC) Program. For each EHC Program phase this module will discuss:

  • Points of stakeholder involvement
  • Purpose of points of involvement
  • Engagement opportunities
  • Potential types of stakeholders to involve

Speaker notes: For each EHC Program phase there are multiple opportunities for engagement with stakeholders. Stakeholders may be involved in one or multiple points. This section reviews the EHC Program phases, engagement purpose, and opportunities for engagement at each phase. It also discusses the types of stakeholders that investigators should consider involving.

Slide 5

Module III Learning Objectives

After completing Module III participants will be able to:

  • Identify at least three opportunities to involve stakeholders in EHC Program research
  • Identify the objective of:
    • Each point of involvement in the EHC Program research process
    • Involving stakeholders at each point
  • Describe AHRQ’s goals for stakeholder involvement

Speaker notes: After completing Module III, participants will be able to:

  • Identify at least three opportunities to involve stakeholders EHC program research
  • Identify the objective of:
    • Each point of involvement in the EHC Program research process
    • Involving stakeholders at each point
  • Describe AHRQ’s goals for stakeholder involvement


Slide 6

Skills for Engagement

The Agency for Healthcare Research and Quality (AHRQ) firmly believes that involving all stakeholders in the research enterprise from the beginning improves the end product and facilitates the diffusion and implementation of the findings by getting early buy-in from users. Involving stakeholders also helps to ensure that the research reflects the various needs of all diverse users.”
(AHRQ, 2009)

Speaker notes: AHRQ has held the involvement of stakeholders as central to its EHC Program since its inception, and has clearly outlined the continued expectation for broad stakeholder engagement as a core principle in supported activities. AHRQ defines EHC Program stakeholders as those individuals and organizations who have an interest—personal or professional—in the findings and applications of the research completed under the Program.

Slide 7

EHC Program Phases of Research 

Starting with “Prioritize research topics”, each of these phases are a part of the research cycle; they are listed in order below. All the phases pointing into “EHC Program” at the center:

  • Refine research topics
  • Conduct research
  • Comment on draft research results
  •  Create audience-specific information
  • Disseminate information
  • Use research
  • Suggest a research topics

Speaker notes: The phases of research conducted in the EHC Program are presented here.  Stakeholders have the opportunity to get involved at many different points throughout the research process – from the nomination of research topics through the dissemination and translation of research findings.  At each point of the process, researchers involve stakeholders recognizing their ability to contribute valuable input to ensure that the research is relevant to real world decision making.

Slide 8

Stakeholder Activities in EHC Program Research

  • Suggest a Research Topic
  • Identify real-world questions for health & health care decisions
  • Provide Insights on Research Priorities
    • Provide input to develop and prioritize research gaps
  • Refine Topics & Develop Key Questions
    • Ensure key questions accurately describe decisional dilemmas  
  • Help Develop Research Approach
    • Provide input on methods, clinical relevance and appropriate outcomes to assess
  • Review Draft Research Findings
    • Provide input to improve quality and responsiveness of research reports
  • Shape & Inform Research Products & Tools
    • Help craft meaningful messages
  • Distribute Products and Lead in Implementation
    • Share information with others; take the lead in using evidence to inform health care practices

Each of the above bullets is encapsulated in a box to represent a separate bucket of stakeholder activities.

Speaker notes: Stakeholders may be engaged at only one point, while others will interact with the Program at multiple points during the research process depending on the activity. Each point of engagement serves a different purpose, and provides distinct opportunities for involvement. The “art” of engaging stakeholders involves matching their interests with research objectives and the purpose of each Program phase.

AHRQ envisions ongoing relationships between its research institutions and stakeholders so that continued collaboration and knowledge exchange occurs throughout topic identification and nomination, topic refinement, research development and translation.

Slide 9

Topic Identification and Nomination

  • Purpose
    • Identify and select relevant topics based on priority-setting criteria for diverse group of stakeholders
  • Opportunities
    • Nominate topics online or by mail
    • Frame clinical context of topic
  • Stakeholders
    • Patients & consumers, clinicians, policymakers, experts, citizens, others
    • , others

Speaker notes: Topic identification and nomination is an important component of the Effective Health Care Program. All stakeholders are encouraged to identify and nominate topics for research, including patients and consumers, clinicians, policymakers, clinical and methodological experts, funders, researchers, and other health care decision-makers. Participation by decision-makers at this point ensures that research responds to the key dilemmas and information gaps faced by end-users. The EHC Program accepts nominations from individuals and organizations, and nominations can be submitted through the website or by mail.

Slide 10

Topic Prioritization

  • Purpose
    • To provide input to develop and prioritize research gaps
  • Opportunities
    • Participate in discussions to describe research gaps and prioritize research needs
  • Stakeholders
    • Researchers, funders, patients & consumers, clinicians, others

Speaker notes: To make the best use of limited budget and resources, it is often necessary to focus EHC Program efforts and narrow research topics to meet these high-priority areas.  Stakeholders can play a role in shaping these priorities in the following ways:

  • Stakeholders may be invited to participate in conference calls or in-person meetings to discuss priorities for research topics. 
  • Stakeholders may be invited to take part in a group discussion that provides insights to researchers over the life of a research project.   


Slide 11

Topic Refinement

  • Purpose
    • Clarify and refine key questions to accurately reflect decisional dilemmas
  • Opportunities
    • Provide real-world input as key informants
    • Provide input on clinical knowledge, context & controversy during multi-stakeholder meetings
    • Provide public comment via web
  • Stakeholders
    • Nominator of topic, patients & consumers, clinicians, policymakers, experts, citizens, others

Speaker notes: Topic refinement, one of the first steps in the research process, is concerned with identifying and defining key questions that are important to the myriad of stakeholders in health care. During topic refinement, the research institution is expected to seek input and guidance from consumers, clinicians and policymakers on real-world dilemmas and context, as well as to seek input on clinical logic from experts.

The use of key informants during topic refinement helps ensure the subsequent research addresses the questions most important to decision-makers, represents an accurate scope of issues, and produces the most valuable and usable product. Key informants may include patients and caregivers, clinicians (both generalist and specialty), representatives of relevant professional and consumer organizations, insurers and health plan representatives, public policymakers, and others with experience making health care decisions relevant to the topic.

In addition, for research reviews, all stakeholders are welcome to provide public comment on draft key questions through the website. This helps to further ensure a broad range of interests are represented, and that the final products are responsive to the public and meet the needs of end-users.

Slide 12

Topic Refinement & Key Informants
Ways to find key informants include contacting:

  • Advocacy organizations and support groups
  • Professional organizations and societies
  • Certification boards
  • Professional networks
  • Private insurers

Speaker notes: Identifying and recruiting stakeholders to participate as key informants during topic refinement can be challenging. Some tips for locating potential key informants include:

  • Look for patients and other consumers through advocacy organizations and support groups
  • Look for clinicians through professional organizations and societies, or certification boards
  • Look for policymakers through professional networks such as Medicaid Medical Directors Learning Network, Drug Effectiveness Review Project, or through private insurers


Slide 13

Research Protocol Development

  • Purpose
    • Clinical, methodological, and public input into research and its protocol
  • Opportunities
    • Provide methodological & clinical input as a Technical Expert Panel (TEP) member
    • Provide input on important outcomes for assessment
  • Stakeholders
    • Patients & consumers, clinicians, policy-makers, researchers

Speaker notes: Opportunities for stakeholder involvement are more limited during the research process itself. During this phase, the role of stakeholders is primarily to provide clinical and methodological guidance on research protocols, methodological and clinical appropriateness of study selection criteria. Stakeholders involved in this phase will typically be experts, either clinical or methodological, end-users of the reports, and others to help the researcher maintain fidelity to the applicability of issues and end-user perspectives.

Slide 14

Peer and Public Review

  • Purpose
    • Invite both scientific peer and public review of research reviews and reports
  • Opportunities
    • Comments requested via web and online
  • Stakeholders
    • Patients & consumers, clinicians, policymakers, experts, citizens, others

Speaker notes: Stakeholders may provide peer and public review of both research reviews and reports online.  The intent of the peer and public review process is to ensure scientific integrity and quality of research reports. Organizations such as professional societies are encouraged to suggest experts to participate in peer review. 

Slide 15

Report Translation & Dissemination

  • Purpose
    • Test and provide input on summary guide development
  • Opportunities
    • Participate in focus groups
    • Participate in individual interviews
    • Provide expert review of summary guides
    • Partner to distribute research products
  • Stakeholders
    • Patients & consumers, clinicians, policymakers, experts, citizens, others

Speaker notes: Although research institutions are not directly involved in the translation of reports and dissemination of the resulting summary guides, the important work of involving stakeholders in the earlier EHC Program phases can assist with this final phase. Stakeholder input is essential for the final translation of the research reviews and development of summary guides, as well as for strategic dissemination partnerships. Early involvement of stakeholders in the research process, increases the likelihood of continued engagement and interest in dissemination activities and the Program overall.

Stakeholders may be involved with topic decision-context groups, summary guide reviews, and product user feedback. All potential end-users are invited to participate during this phase, including consumers, clinicians, and policymakers. Experts will be consulted as well to ensure scientific accuracy of translated materials.

The Eisenberg Center is responsible for conducting stakeholder engagement at this phase.

Slide 16

Summary of Module III

  • Identify and Involve stakeholders early in the process
  • There are multiple opportunities for stakeholder engagement in the EHC Program
  • Each opportunity serves a unique purpose
  • In order to ensure valuable and constructive knowledge exchange, consider the most appropriate stakeholders to involve
  • There are many ways to locate and recruit stakeholders

Speaker Notes: For each phase of the EHC Program research process there are multiple opportunities for engagement with stakeholders. Stakeholders may be involved in one or multiple points. Because each point of involvement serves a unique purpose, investigators should consider the most appropriate stakeholders to involve in order for valuable and constructive knowledge exchange to occur.

Slide 17

Module III Quiz
To help you review the information presented in Module III, please complete the following quiz. Answers are provided at the end of the module.

Slide 18

Module III Quiz:
Question 1
1) Which of the following activities are opportunities for stakeholder involvement in the EHC Program? Select all that apply:

    • Suggesting a research idea
    • Providing input on inclusion/exclusion criteria
    • Discussing key questions
    • Illustrating clinical context for a topic


Slide 19

Module III Quiz:
Question 1: ANSWER
Which of the following activities are opportunities for stakeholder involvement in the EHC Program?

    • Suggesting a research idea
    • Providing input on inclusion/exclusion criteria
    • Discussing key questions
    • Illustrating clinical context for a topic.


Slide 20

Module III Quiz:
Question 2
2. Match the engagement purpose with the appropriate EHC Program phase.  Identify nominations relevant to real-world health care decisions:

    • Topic Identification & Nomination
    • Topic Prioritization
    • Topic Refinement
    • Research Protocol Development
    • Peer/Public Review
    • Report Translation & Dissemination


Slide 21

Module III Quiz:
Question 2: ANSWER
2. Match the engagement purpose with the appropriate EHC Program phase.  Identify nominations relevant to real-world health care decisions:

    • Topic Identification & Nomination
    • Topic Prioritization
    • Topic Refinement
    • Research Protocol Development
    • Peer/Public Review
    • Report Translation & Dissemination


Slide 22

Module III Quiz:
Question 3
3.  Match the engagement purpose with the appropriate EHC Program phase. Provide clinical and methodological input:

    • Topic Identification & Nomination
    • Topic Prioritization
    • Topic Refinement
    • Research Protocol Development
    • Peer/Public Review
    • Report Translation & Dissemination


Slide 23

Module III Quiz:
Question 3: ANSWER
3.  Match the engagement purpose with the appropriate EHC Program phase. Provide clinical and methodological input:

  • Topic Identification & Nomination
  • Topic Prioritization
  • Topic Refinement
  • Research Protocol Development
  • Peer/Public Review
  • Report Translation & Dissemination


Slide 24

Module III Quiz:
Question 4
4. Match the engagement purpose with the appropriate EHC Program phase. Provide clinical and methodological input:

    • Topic Identification & Nomination
    • Topic Prioritization
    • Topic Refinement
    • Research Protocol Development
    • Peer/Public Review
    • Report Translation & Dissemination


Slide 25

Module III Quiz:
Question 4: ANSWER
4. Match the engagement purpose with the appropriate EHC Program phase. Provide clinical and methodological input:

  • Topic Identification & Nomination
  • Topic Prioritization
  • Topic Refinement
  • Research Protocol Development
  • Peer/Public Review
  • Report Translation & Dissemination


Slide 26

Module III Quiz:
Question 5:
5. List three ways in which you could locate and recruit stakeholders.

Slide 27

Module III Quiz:
Question 5: ANSWER

  • List three ways in which you could locate and recruit stakeholders.

Potential ways to recruit and identify stakeholders include

  • Patients & consumers
    • Advocacy organizations
    • Support groups
  • Clinicians
    • Professional organizations and societies
    • Certification boards
  • Policymakers
    • Professional networks (MMDLN, MED, DERP, NASADD)
    • Through private insurers


Slide 28

Works Cited

Agency for Healthcare Research and Quality (2009). Methods Guide for Effectiveness and Comparative Effectiveness Reviews. Rockville, MD. Available at: http://effectivehealthcare.ahrq.gov/index.cfm/search-for-guides-reviews-and-reports/?pageaction=displayproduct&productid=318

Agency for Healthcare Research and Quality (2009). Operating Plan for American Recovery and Reinvestment Act Funds for Comparative Effectiveness Research as the Agency for Healthcare Research and Quality.