Skip Navigation
Department of Health and Human Services

Community-Based Participatory Research: Lessons for Stakeholder Engagement in Patient-Centered Outcomes Research


MS. PENNY MOHR: Good afternoon. This is Penny Mohr, and I’m the senior vice president of program development at the Center for Medical Technology Policy. It’s my pleasure today to welcome you to the webcast sponsored by the Agency for Healthcare Research and Quality Effective Health Care Program that’s titled Community Based Participatory Research Lessons for Stakeholder Engagement in Patient Centered Outcomes Research. We have a wonderful turnout today with over 260 people participating by phone. Just a brief word about our sponsorship for our webcast today, it was funded as part of the AHRQ’s Community Forum Project for which I am a subcontractor. The Community Forum is charged with working with AHRQ’s Effective Health Care Program components to expand stakeholder involvement in research processes and activities.

By stakeholder we mean the broad array of groups with a vested interest in outcomes of comparative effectiveness research or patient centered outcomes research such as patients, clinicians, payers, policy makers, and healthcare administrators. They are also developing and testing methods for gathering public input on value based healthcare questions. That said the presentation and panel discussion that we’re going to have today does not represent the official policy of either AHRQ or the U.S. Department of Health and Human Services. I also want to note today that we’re recording the webcast and it will be made available on the Effective Health Care Program website for your review for you to pass on to those who are not able to attend today.

Before we begin I’d like to set the stage for why we felt this was an important webcast to host. Community Based Participatory Research or CBPR has a rich history of community engagement that reaches back almost a half a century. Meanwhile a fundamental feature of Patient Centered Outcomes Research or PCOR, which has only recently entered our lexicon, is the engagement of users of research in research in helping to select priorities, designing and implementing research and translating and communicating the findings. Given that researchers involved in both fields interact infrequently we thought it would be terrific to host a dialog between experts in both fields to talk about some of the relevant lesions.

Today we’re hoping to bring together experts to discuss some of the lessons from CBPR and how can it enhance stakeholder engagement in PCOR, and to encourage a greater dialog and resource sharing between the CBPR and PCOR research communities going forward. I’m thrilled today to introduce our panelists. We have an absolutely expert and excellent panel joining us today. Ms. Pat Deverka is a senior research director from the Center for Medical Technology Policy and she has led CMTP’s efforts to develop and test robust methods for involving stakeholders across a variety of PCOR applications and has published extensively in that area. She oversees CMTP’s Patient and Consumer Advisory Committee which provides ongoing advice about how to more effectively engage patients and consumers in CER, and they have contributed to the development of principles in stakeholder engagement in PCOR in a conceptual framework which she is going to be sharing today.

Another of our PCOR experts is Dr. Dan Mullins. He is a professor in the Pharmaceutical Health Services Research Department at the University of Maryland School of Pharmacy. His research has used both aspects of CBPR and PCOR. He is serving as the principle investigator for a PCOR contract to enhance methods for engaging hard to reach populations in PCOR, and as a recipient of one of six NIHCER methods development grants examining ways to improve the efficiency of clinical trials. In addition to his research as PI Daniel has been responsible for the evaluation of components of a number of community based interventions for other PI grants including an NHLDI grant aimed at reducing hypertension among black males and a current state of Maryland grant to create a health enterprise zone in west Baltimore to reduce cardiovascular health disparities.

Ms. Dwyan Monroe brings the patient perspectives to this dialog. She is living with diabetes and hypertension both and she’s also a community outreach trainer and advocate and facilitates the Community Health Action Team in Baltimore. She has experience with CBPR directly with working with John’s Hopkins University in chronic disease management, and also has been a community advisor for Dr. Mullins grant which I mentioned previously for PCOR. Our experts in CBPR include Dr. Madeleine Shalowitz who is the director of health services research and section head for child and family health studies for NorthShore Research Institute, which is an affiliate of NorthShore University Health Systems, and integrated health system serving the Chicago area. She’s an experienced health services research specifically in reducing health disparities in pediatric and maternal health and she also brings a perspective of working through NorthShore’s Community Health Hospital Community Benefits program to improve public health in the community. She’s also the lead author of an AHRQ’s funded review of the literature on strategies for community engagement with CPDR.

Dr. Nina Wallerstein is professor in the department of family and community medicine at the University of New Mexico School of Medicine and director of the university’s Center for Participatory Research and director of the community engagement and research component of the clinical translational science center. She has 25 years of CBPR research experience and her work is focused on developing culturally centered research in tribal communities.

Our agenda for today’s webcast will be a little bit innovative and highly interactive. We are engaging people in a dialog and have set this up as a conversation among the panelists. We’re going to begin with a short didactic presentation from our CBPR experts Dr. Nina Wallerstein and Dr. Shalowitz giving us an idea about an overview of CBPR and it’s use in health system settings, and Dr. Pat Deverka will provide an introduction to stakeholder engagement in PCOR. Next we’re going to move into an open dialog across our panelists about some of the similarities and dissimilarities between CBPR and stakeholder engagement in PCOR, and some of the lessons that we might learn. We’ve reserved a portion of the webinar at the end to focus on the specific issue of how to engage under represented populations, and I’ll be calling on two of our panelists, specifically Dr. Mullins and also Dr. Shalowitz, Dr. Wallerstein to discuss the issue of how to engage under represented populations, and we will also be asking for Dwyan to provide a patient perspective throughout and we’ll ask her to be a reactant through the different discussions that we will be having today.

I would like to point out that there is also a 15 minute Q&A period at the end of the webinar. If you could we would like to make sure that this is interactive so if you could use the chat feature you could type in and submit your questions as we go through this webinar and then the webinar facilitator will share your question with the audience and ask the speakers to respond at the end. We’ll do our best to respond to everyone but our time is limited and we’ll follow up with any questions that we’re unable to address after the webinar has concluded. Without further adieu it’s my pleasure to turn this over to Dr. Nina Wallerstein to provide you with an introduction to CBPR. Nina?

Community Based Participatory Research: Principles and Conceptual Framework

DR. NINA WALLERSTEIN: Thank you so much. I was asked to basically set the stage of the principles and framework for CBPR so I want to start with the definition. As we all know W.K. Kellogg Foundation CBPR definition has been highly used as a collaborative approach equitably involving all partners using and benefiting from community partners strength and starting with a research topic of importance to the community with the goal of improving community health and improving health equity as a key action orientation to this research process. It’s based on looking at CBPR as part of the continuum engagement. Often we do research which is on communities or sometimes in community settings in clinical practice settings for example, but to move towards the CBPR perspective we have to ask the question how much are we involving our community members in a with perspective? How much are we working with community members, with community agencies, with community providers rather than on them with our own determined agenda?

CBPR has a set of principles that will be useful to looking at PCOR research recognizing the community’s idea of identity, a long term commitment approach, balancing research and action. In my own experience with tribal communities I recognize the importance of working under tribal sovereignty where tribal governments have the authority to approve you being on the…to work with that community and also to approve any kind of data that will be published. They own the data and although most communities may not most people on this telephone call or conference may not be actually working with tribal communities. These ethical principles of working with community members as colleagues and co-publishers and co-sharing those kind of data in a very genuine way I think has applicability for all communities these tribal principles.

I wanted to share with you today mostly this conceptual model that colleagues and I in a three part partnership have been developing. It’s an NIH study from the last we’re on our fourth year. That is a collegial partnership with ourselves at the University of New Mexico, the University of Washington, and the National Congress of American Indian Policy Research Center in developing this model that I think would be useful not just for CBPR but for PCOR processes as well, and let me just walk you through it. It starts at the left side, looks complicated but it’s very straightforward. We all live and do our research in a context whether it’s the social economic context, a historic collaboration context of the relationship between the clinical setting and the health setting or the university and the community setting and the university, the capacity of the community, even the university capacity to engage in CBPR or community engaged research.

That context sets the stage for whatever partnership is developed whether there are agency members involved, community members, CBOs, clinical providers, academics, and those partnering processes are both based on decision making and relationships and how you work well together, the trust that evolves but also the structure. What are the guidelines, the norms, the memory [phonetic] end of the agreement? The essence of this model is if you have community partnership of some sort it will change and inform the research, which moves you to the next oval. It will change the intervention. It will change the understanding of how this fits within the culture and organizational context and climate, a lot of the implementation science agenda comes right in here of the context, and that change will then inform a set of outcomes that might be not just only improved research outcomes but also greater capacity of the clinical setting or the community setting, policies that may improve health at a larger level capacity, et cetera.

This model of engagement we expect and hope could be useful in thinking about PCOR. Let me just take you through a few more examples of how we use this model to set up metrics and measures of partnering asking the question of how do you know if you have an effective partnership that’s going to make a change in our outcome. This as I said is a research for improved health study. We’ve been testing the model. We’ve developed a literature review of existing measurement tools, but we developed our own instruments bringing together the best skills we could of the country that we found and develop new ones to do a data collection of close to 300 partnerships in an internet survey and some in depth case studies, and I want to share with you just a few metrics and measures out of our research and we’ll be able to get these access to these instruments, et cetera, through these websites when you get off this webinar.

Just some examples, we have examples of community capacity context. You can see the number is the number of questions within scales. We have found that through our beginning analysis of our data over the partnerships that we’ve collected data from that alignment with the principles that if the partners are aligned with their values and principles it actually is associated with better outcomes, power dynamics, how much that’s shared, the process of dialog and listening, how much the partners involved in different aspects of the research, not just the design and the research design or the questions but in the data collection, in the data analysis which of course you have different roles of partners but most importantly the data interpretation and the usefulness of this back to the community as well as to the clinical setting, and then of course trust is a major element of how partnerships work together so those are some skills that should be available to you.

We also have found that if those partnership dynamics are working well then there will be impact on set of proximal outcomes of partnership synergy and cultural centeredness of the work itself as well as some more distal outcomes of improved personal and professional levels outcomes of the partners themselves, and then concrete changes in community policy level outcomes, improved services and health improvement. We have measures as well of these outcomes. Finally I just want to give you two examples of what these very important measures mean. This is an example of cultural centeredness which says in this case this is a billboard on the Navajo nation. Have you noticed a change in your harmony, breath, energy? It may be TB shows that you need cultural sensitiveness. This is not a billboard you would find in New York City. This could only be a billboard for the Navajo nation and that’s for any of our interventions.

How to think about what does it mean to be doing it within this context, this clinical context or even this community context, and finally just a short showing you of our metrics of trust. We’ve developed this hipology that goes from no trust up through what we call critical reflective trust, and we’ve been looking at how trust both can be nurtured and can grow within a partnership and just to finish these metrics we suggest and measures could be used with all members of a partnership to be able to assess how much they’re in alignment with each other, what’s the level of trust in this partnership now, where do you want to be in the future, and how for example can you get there, best get there, and given that trust of course we consider an important partnership dynamic for them reaching the kind of outcome that might be important not just in a community setting but again through PCOR in the clinical settings as well. With that I’ll end and look forward to the dialog. Thank you.

MS. MOHR: Great, thank you very much Nina and I think that it’s specifically great that you have these links to resources that are embedded in your slides so that participants will be able to go back on to the AHRQ website and find this material. I’d like to move now to Madeleine Shalowitz who’s going to be talking about her specific experience working within an integrated health system and using CBPR. Thank you, Madeleine.

CBPR Value System for Program Development, Evaluation and Information Sharing in a Health System Environment

DR. MADELEINE SHALOWITZ: The purpose of this transition is to move from the community setting to a healthcare setting and harness community wisdom in a somewhat different way to the academic rigor of the medical environment, and in that regard I’d like to acknowledge my academic partner and affiliation in this which is the University of Chicago because the CTSA at the University of Chicago has been really supportive of the work that I’m going to be showing you, and the health services research background that I bring to this is now being applied in the Center for Clinical Research Informatics at NorthShore. What Nina presented to you is a very balanced view of community and researchers, but really in a health system the balance is very different. The health system is very concerned about it’s patients and also about those who pay the bills, the payers, but family and cultural in society is rarely if ever considered in traditional medicine, traditional medicine meaning the U.S. healthcare system, not tribal medicine, and perhaps will need to be incorporated more as we look to more integration of health outcomes across sites and services.

I do want to point out to you that more in line with Nina’s presentation is that the perspective or the patient or on the pediatrician so you’re seeing the perspective of the parent with regards to the child is really about they’re alive outside the treatment setting and you’ll see that the health system is located at the periphery of their life. Whereas the health system sees itself as the center to the family their lives outside the treatment setting is primary. The program I would like to describe to you is Be Well Lake County which is NorthShore University HealthSystem’s signature community program. It is a direct donation from the hospital and various other funders to the Lake County Health Department and it provides comprehensive diabetes management for adult medically underserved Type 2 diabetics in one of the northern counties of Illinois. It’s a partnership between NorthShore University Health System and the Lake County Health Department, which operates five community health centers in Lake County.

The purpose of presenting this to you is to show how patient input and provider input has changed the clinical programs that are provided and also change the questions we ask in the evaluation. Clinical program was in fact a collaborative development among NorthShore Community Health Department Service Provider input and data in an iterative process over time. What NorthShore brought to the table was funding for comprehensive primary care visits, medication and testing supply systems because community health systems have so much difficulty with subspecialty care access the NorthShore physicians were provided subspecialty care to the health center patients as part of the proposal, and thirdly in medical nutrition therapy and diabetes education. With patient input—which I’ll show you a little bit more about in a minute—over time and through philanthropic funding we’ve added fitness programs, a community garden and retinal screenings.

One of the concerns of the community is that this diabetes program is housed in a food desert, and so based on provider and patient input we added a food security measure to the standard care, and what we learned was almost 57% of the patients with Type 2 diabetes in Lake County in fact did not have adequate access to food. We asked the question do patients who don’t have enough food have worst diabetes control because that certainly makes a whole lot of sense since diabetes involves substantial lifestyle management. I’m showing this to you as a pilot data analysis because I think it’s very important to understand that medical care is in a larger context and in fact the community context may influence the outcomes of Cadillac care. In affect this program brought all of these resources to families and essentially eliminated most of the access issues. At the beginning of the program the patients had pretty similar hemoglobin A1Cs but by nine months in the program you can see a diverging difference between those who are food secure and those who are food insecure, insecure is the top curve, and secure is the bottom curve.

In essence those who have food security who have adequate access profited from the program whereas those who are food insecure were not able to do that, the interference with glucose control increased over time. I wanted to present to you a way to bring together all the stakeholders environment and the top part of the slide shows you an organizational structure that I’ve in fact used and I’m still using with the Lake County Health Department partners in order to run a CBPR program and then I’d like to talk to you about the adaptation to PCOR. You’ll see that all of the stakeholders interrelate to one another through two advisory committees. The balance of the advisory committees is different. On one side the community but that has representation from the academic side, and then an academic steering committee with representation from community.

In essence we operate as a complex network so that we make sure that all voices are heard all the time even though the specific agendas might be different. I propose to you that a sample PCOR adaptation might look quite similar where the health system and patients are the primary stakeholders in the center but that we bring in payers and policy makers, family and communities and two way funding and service provision side and then a community advisory committee in a network model similarly. In summary I think the adaptations for healthcare are clear for CBPR although the overlap is not completely 100%, and with that I’d like to turn back to Penny.

MS. MOHR: Great, thank you very much Madeleine. I just wanted to remind people that we are going to sending out the slides for today’s presentation. We’ve been getting a lot of questions about that at the conclusion of the webinar so you will be getting that everybody who’s registered, and without further adieu I’d like to turn this over to Pat to give us her presentation on PCOR and stakeholder engagement.

Patient-Centered Outcomes Research and Stakeholder Engagement

DR. PATRICIA DEVERKA: Thanks Penny. My cast today is to briefly describe the role of stakeholder engagement in patient centered outcomes research, which admittedly has a much shorter track record than CBPR but I believe shares some of this objective. First I’d like to ensure that we’re all clear about the relationship between comparative effectiveness research which I refer to on several of my slides and in the reference materials and Patient Centered Outcomes Research or PCOR the focus of today’s webinar. One the left hand side of the slide is a familiar Institute of Medicine definition of Comparative Effectiveness Research or CER which is uniquely characterized from other types of clinical research by its purpose which is to provide comparative evidence to help stakeholders make better decisions.

On the right hand side of the slide is a more detailed description of Patient Centered Outcomes Research or PCOR. PCOR I think can be viewed as a subset of CER or more specifically PCOR is comparative effectiveness research focused on and guided by the needs and concerns of patients. Just like CER PCOR requires the active involvement of stakeholders to ensure that the research will meet its stated goals. As you can see from the bullets I’ve listed patients and other key stakeholders should be actively engaged throughout the research process, which includes the results dissemination phase, an aspect of the research continuum that has traditionally been neglected after scientific publication but is an area of intense research currently.

I think it’s important to highlight the context for involving stakeholders in PCOR. There was a great deal of initial enthusiasm I think for stakeholder engagement that was super charged in 2009 with the 1.1 million dollar RF funding for CER. These stakeholder engagement efforts will focus primarily on the areas of priority setting and identifying future research needs, but there are also some examples of involving stakeholders in designing studies. In parallel a number of researchers including our team at CMTP took a step back and evaluated what was working well with the process and what were some of the barriers which I’ve listed here. Many of these may sound familiar to you and many of these have begun to be systematically addressed by experts in the field but briefly the literature has allowed consistent terminology and definition to stakeholders and allow consistency and processes for stakeholder engagement. There also is a new shared understanding between stakeholders and researchers of what it means to affectively or successfully involve stakeholders in research, primarily stemming from a lack of common principles and best practices but more fundamentally because there is no agreed upon conceptual model for stakeholder engagement in CER.

Perhaps not surprisingly there’s limited data regarding the impact of involving the stakeholders in research and systematic evaluation of the process such as we saw described by Nina is rare. Researchers have an expectation that they often need stakeholder input in projects with short timelines or immediate start dates and it’s difficult to coordinate schedules with all relevant stakeholders. There are also extensive training needs for all of the stakeholder’s groups and I would put researchers in that category, and so you need to ensure full participation in the meaningful experience for stakeholder engagement which I’m going to define later. All of these issues translate into concerns primarily on the part of researchers and funders. The stakeholder engagement will add time and cost to projects, and without clear measures of impact and effectiveness I’d argue that there’s no good way to put these concerns into the proper perspective to assess the true value of stakeholder engagement.

That’s why our group of researchers at CMTP undertook an effort to help address some of these barriers as you’ll see on the next slide. There’s a lot of information here and if you’re interested in understanding these concepts in greater detail I’ve provided a reference to the article that goes into the full background and development of the definitions and conceptual model. In the interest of time I’d like to highlight just a few key points. First given the identified problem of confusing terminology we worked with an advisory group of patient consumers and patient advocates to define what we mean by the term stakeholder. I’d only point out here that our definition includes the notion of communities so there appears to be overlap with the target audience for CBPR. Then after reviewing the literature on deliberative methods and other approaches for engaging stakeholders in research we took the step of defining what we mean by stakeholder engagement and you can see that definition on the lower left hand side of the slide.

Here I’d like to emphasize that after extensive discussion and debate with our advisors and project team we settled on specifying two key aspects of the process. First it actively solicitous knowledge, experience, judgment and values of the people that participate therefore we really wanted to signal that each of these sources of information are seen as valuable and desirable for the process. Second that it’s focused on two important outcomes that ideally require collaboration to achieve, creating a shared understanding among stakeholders and making decisions that are viewed as actionable and credible to all parties. Again there seems to be overlap with the goals of CBPR although the differences between them appear to occur in how the method for engaging stakeholders translated into practice.    

Finally I want to point out that we also recognize the need for a conceptual model for engaging stakeholders so that we could begin to measure the impact at both the level of process outcomes and final or CER outcomes. This model is adapted from the literature on risk based decision making. They use both analytic methods and deliberative methods to inform choices, solve problems, and make stakeholder driven decisions. We felt it was highly relevant to the situation of CER where research experts and non-experts come together to use data, experience, and values combined through a variety of quantitative and qualitative methods to make decisions that ultimately are translated into both immediate project related outcomes or more long term health and policy related outcomes. You’ll see in some of the process outcomes notions such as trust, respect, fairness, which appear comparable to some of the concepts covered by Nina. While successfully applied to one three year project this model requires further testing on a broader range of projects and requires the robust validation of the sort that Nina shared during her remarks.

Nevertheless we feel that agreement regarding the definitions, terminology and conceptualization of stakeholder involvement is essential for understanding the contributions of stakeholders to CER and evaluating the effectiveness of engagement practices. Only then can we measure and evaluate the influence of stakeholders on particular CER projects and using through results to inform process improvements and conceptual model refinements. The next slide provides a graphical depiction of how we currently see stakeholder engagement occurring in practice. While this slide was developed as part of a white paper commissioned by PCORI to describe the international experience with methods for engaging the public and topic generation for research it’s useful for describing the continuum of patient or stakeholder engagement in research as well. At the top of the slide is the traditional approach to clinical research where patient representatives are occasionally invited to meetings or asked for input. Here the information flow is primarily from the researchers to the patients. At the level of consultation the information flow is primarily from the patients to the researchers in the form of focus groups or questionnaires or even mediated conversations, but typically patients don’t participate in decision making and the goal is not to have the process result in new shared understanding of the issue on the part of both the patients and the researcher.

I call this stakeholder engagement light as there are certainly valid methods for involving patients in the research process through this type of engagement. However, it does not reflect the type of interaction described in our definition. The third level describes two collaborations where the information flow is intended to be bidirectional, and this type of interaction would occur as part of an ongoing relationship where trust would be built over time. Under these circumstances shared understanding and effective decision making would be the result of this true partnership. The h