Skip Navigation
AHRQ--Agency for Healthcare Research and Quality: Advancing Excellence in Health Care

Back to "About This Webcast"

Integrating Comparative Effectiveness Research into Everyday Practice

Webcast Transcript, October 12, 2010

>>DR. STEPHANIE CHANG: Good afternoon, everyone. Good morning to those who are on the west coast. Good evening to those calling in from other countries east of us. Thanks for standing by. Sorry for the short delay.

I'd like to welcome you to today's Web conference titled Integrating Comparative Effectiveness Research into Everyday Practice. This conference is sponsored by the Agency of Healthcare Research and Quality, also known as AHRQ, through the Effective Health Care Program. This event today is an example of how we hope to build on our commitment to creating and disseminating resources and information about comparative effectiveness research to commission to support everyday practice. Thank you for partnering with us in this effort. I'm Stephanie Chang. I direct the Agency's evidence-based practice center program, and I'll be facilitating today's event. Next slide, please.

Before we get started, I just want to review some information about the Web conference technology today so that you can interact and participate and ask us questions throughout. If you have any questions throughout the program, you can submit them electronically by accessing the "ask a question" button. That's located on the bottom of your screen on the righthand side. You can click on the button and a box will appear on your screen that will request your e-mail address and your question. Once you complete that information, you press the submit button. We'll be having three moderated question-and-answer sessions where the speakers will be responding to some of the questions. Submit them as soon as you think of them. Don't wait until the end.
Also, if you are experiencing any technical difficulties now or later, you can also hit that same button and submit your technical question or problem to the "ask a question" box, and someone will get back to you through your e-mail.

Biosketches of this conference are available. You can find these in the resource library that's located next to the "ask a question" button. If you click on that button, you'll find the slides and a document with the speaker biosketches. There's also a link to the AHRQ Effective Health Care Program Web page that we really would like you to visit after this event.

For those of you who need it or find it helpful, today's Web conference includes closed captioning. The captioning appears in the box below the slides. And for those of you who missed part of it, the presentation will be recorded and made available on the Effective Health Care Program Web site.

Next slide. There's an agenda up on your screen. And I'll just review it really quickly right now. During this Web conference, the presenters will highlight the ways that comparative effectiveness research has been used to improve health care delivery and describe the ways that the various organizations have partnered with AHRQ to improve the delivery of health care.

Dr. Carolyn Clancy will be the first speaker. She's the director of AHRQ. And she'll start by describing how you as clinicians and those involved in the health care field can really find comparative effectiveness research to improve decisionmaking. Immediately after the presentation, she will be answering questions, so please do submit the questions throughout her presentation, and don't wait until the end.

Following Dr. Clancy, Jean Slutsky, director of AHRQ's Center for Outcomes and Evidence, will discuss the Effective Health Care Program specifically and highlight clinician resources available, as well as findings from a couple of research studies funded by the Program. At the end of her program she'll also answer questions from the audience. Please submit those throughout her presentation, as well.

The last part of the Web conference will feature three presentations that will be done in a series. First, these three organizations are some examples of how they have implemented research findings and related materials from the Effective Health Care Program into the delivery of health care through their program or policy work.

Dr. Amir Qaseem will go first, from the American College of Physicians, ACP. And he'll describe how they used the comparative research findings to create evidence-based clinical guidelines.

Second up will be Mary Jo Goolsby from the AANP, the American Academy of Nurse Practitioners, who will describe how they've created modules based upon the Effective Health Care Program clinician guides.

And the last speaker will be Dr. Nilay Shah from the Mayo Clinic, who will talk about how they developed a patient decision aid using the findings from the Effective Health Care Program. After all three of these speakers have finished, we'll have another moderated question-answer session for questions for any of the three of them. Please do submit those any time throughout those three presentations.

We have an exciting lineup. And I don't want to delay this any further. As I mentioned, the full descriptive profiles can be found on the resource library at the bottom of the page. But otherwise, we will move on.

Next slide. I have the pleasure of introducing the first speaker, Dr. Clancy, who is, hopefully, on the line.

>>DR. CAROLYN CLANCY: Great. Thank you for the introduction. I really appreciate it. And I want to welcome everyone here this afternoon, as well. I am really thrilled that there are so many clinicians joining us today to learn more about a growing branch of patient-centered health research previously called comparative effectiveness research and how this work can help improve the quality of health care you deliver every day.

In fact, I would say very directly that in our view, the success of our investments in this research will actually be judged by how much help it gives to clinicians and patients as they are struggling to make decisions every day.

My last information is that we had close to 1,000 registrants for today's event, so I am really excited. Next slide.

Before getting into the details about comparative effectiveness or patient-centered outcomes research, I wanted to provide just a little bit of background about AHRQ. Very simply stated, our mission is to improve the quality, safety, efficiency and effectiveness of health care for all Americans. Next slide.

This next slide shows you, rather than an organizational chart of offices and centers—if anyone feels deprived, I would be happy to get you one — but the real engine of how we do our work, our portfolio, is shown on this slide. And you can see that the upper right-hand wedge of the pie is labeled comparative effectiveness research and that's what we're going to be talking about today. But I think it's also important to recognize that each of these portfolios has great potential to help transform the health care system of the 21st century into one that is information rich but also patient focused, which doesn't necessarily feel like health care today, at least every day.

So, in addition to the fact that we're talking about comparative effectiveness today, again, we think it's very important the intersections among these areas are also recognized.

At the end of the day, we provide support for independent research that is always informed by the needs of people using the research, that is designed to help people and organizations at all levels make a wide range of informed choices. Next slide.

The ultimate goal here is informed choices. We're not telling people what to do; this is not prescriptive. It is actually descriptive. But I think many of us on the phone today together would agree that as a nation, we don't make informed choices about health care often enough.

Whether you're a doctor, a pharmacist, a nurse, or anyone else who has to make health care decisions, you can benefit from patient-centered outcomes research. This research is designed to help you, the busy and way too frequently time-challenged clinician find out what the latest and best science says about treating various conditions. Next slide.

This slide just shows a clinician with I'll say a mature female patient. If you would imagine for a moment that this woman is seeking guidance from you and wants to know, how can I relieve my pain without suffering side effects? What's the best medicine? Next.

The next one shows a clinician with an older gentleman. If you could imagine having the conversation, as I would imagine a number of you have with friends and family members and colleagues and so forth, about his newly diagnosed prostate cancer. He really wants an honest appraisal of his options. Should I have surgery? Chemotherapy? Radiation? Hormones? Watchful waiting? What is the best choice for me? And what are the risks?

Now, obviously people don't necessarily articulate their questions in that very clean, organized way. But at the heart of many, many clinical conversations between clinicians and patients, those are the ultimate questions. Next.

As a practicing clinician, I know you wake up every day wanting to provide the best, safest, and most appropriate care to every patient that you see. And a big, big part of that, a big part of the currency of health care, is talking to patients, listening to their concerns, and answering their questions with information they can use to figure out what is the best decision for their treatment options?

We know today that many patients are getting more involved in their health care decisions. It feels like every 10 minutes I'm getting another e-mail about how many Americans go online to find out more about potential treatments and so forth. And I'm really glad to see that happening. The health care system is much better off for it, and it's a trend that should be encouraged. And we also recognize that every patient is different: different life circumstances, different medical histories, and different values and preferences. Patients need reliable and understandable information that's relevant and applicable to their circumstances, Information that allows them to be a partner with you to make the best possible health care decisions.

I think you're getting the drift that we believe that the role of the patients in this equation is extremely important and, therefore, my Agency is making a big investment to educate patients and their families about taking an active role in partnering with their doctor, pharmacist, nurse, or other health care professional.

So, that brings us to today's topic. Where do you get the information to work with patients to make these critical decisions? Next.

Comparative effectiveness, or patient-centered outcomes research, focuses on patient-centered outcomes. It focuses on providing unbiased and practical evidence-based information. And it ultimately compares drugs, devices, tests, surgeries and other approaches to health care in terms of benefits and harms. And also what is known and what is not known.

The work that we support is not done generally within a category of type of service. Usually, we are doing this under the framework of looking at particular conditions. Again, similar to prostate cancer. Should I have surgery? Should I do watchful waiting? Should I have chemotherapy? Should I do some combination? And what does it mean for me if I make one or another decision? At the end of the day, we believe that this research to be most helpful to you has to be descriptive rather than prescriptive. Next.

So, I think it's very important to recognize both the tremendous advantages that this research offers as well as, quite honestly, what research can't necessarily do. This research can actually help make decisions more consistent and transparent and so forth, and at an organizational level can sometimes help clarify the nature of disputes over practice and policy.

We believe that it is very, very tightly linked to efforts to improve the quality of care and it can help patients make decisions about their own care. Next.

Now, this past year and a half has been amazingly busy for us at AHRQ. We have been the lead federal agency for this kind of work up until 2009 when the American Recovery and Reinvestment Act was passed. Now, up till that time, over 5 years, we received a total of $129 million from Congress for this type of research. And then when the Recovery Act was passed, that included a total of $1.1 billion—that's billion with a —for comparative effectiveness research, including $300 million directly to AHRQ.

Now, we thought we had to make the very, very most of this opportunity and took it very, very seriously. It's been an exciting and exhausting year. And we're very excited about the work that's really just getting started. And we're excited about it because of what it can do for you.

Again, the guiding vision for our efforts is that this is translated as good information that's available when and where you need it for clinicians and patients when they're struggling with a lot of very, very important decisions.

In order to make the best possible use of these resources, my colleagues, led by Jean Slutsky, whom you will hear from in a moment, and including Stephanie, who is moderating today's discussion, thought it was incredibly important that we actually have a very clear framework for how we would make these assessments.

The process, all of our research, begins with stakeholder input and involvement. In other words, we do our best to articulate and support others to articulate what we think are important questions. But we also put those questions out for public comment so that we know and have much better confidence in the fact that we're supporting the best possible research.

Interestingly, many countries have tried—many developed countries, I should say—have tried to figure out what's coming online or over the horizon. The kind of thing you might hear about at a medical meeting, or for those of you who provide care in the hospital, in the cafeteria or overhear in conversations. So, this year we've been able to make an investment in developing a methodology for what we're calling horizon scanning.

A very important linchpin and foundation of our efforts is about synthesizing existing evidence. There are actually renowned researchers who believe that no clinical trial should ever be conducted until a very rigorous systematic review or synthesis of existing studies has been done so that we know with some precision what are the most important areas for future research.

Now, anytime you do a systematic review or synthesis of existing research, whether it's the kind of rigorous work that we support or even a kind of back-of-the-envelope, you immediately stumble onto unanswered questions. And, again, this past year, we thought it was incredibly important to be able to be as clear as possible about those evidence gaps, so we actually made some investments in a strategy to figure out how to make the identification of important research gaps as consistent as possible.

When we do identify important gaps then, of course, we need to turn to a variety of different approaches to generate new, important evidence. And throughout all this, we're very squarely focused on developing and nurturing a workforce of researchers, including clinicians, who will continue to do this work throughout their careers. And we're also focused on supporting the development of the tools that they will need. Next slide.

Now, some of you may have been wondering, "How does this relate to the Affordable Care Act?" Let me just say there is one very explicit section, very, very well written in the Affordable Care Act, Section 6301, which establishes a new Patient-Centered Outcomes Research Institute, which, of course, we have shortened to PCORI here in the nation's capital. Very importantly, this is an independent, nonprofit institute which will be funded by a combination of public- and private-sector funding. It will set priorities and coordinate with existing agencies, such as AHRQ and NIH, that support patient-centered outcomes research.

By law, the director of AHRQ and the director of the National Institutes of Health are on the board of this new institute. The board was just announced several weeks ago. So, it'll be some time before you hear very much about the interworkings of this institute.

It's also important to recognize that the bill was very clear in prohibiting the findings to be construed as mandates on practice guidelines or coverage decisions, and includes some important patient safeguards. This is a comment more on how research is used than on the research itself.

At the end of the day, we know that improving health care and getting to a place where no matter where you land for health care, you'll have full confidence that the care you're getting is superb quality, is by definition a team sport. We look forward to working with you to improve the quality of care as we find new and important ways to conduct and apply patient-centered outcomes research. And we appreciate your continuing interest in practicing evidence-based medicine.

I want to urge you to use this information and share it with your patients to help you work together as a team to make the best possible treatment choices for each individual patient. I can tell you quite directly that I use the consumer guides a lot with my friends, with my family members. And to a person, they have found the information very empowering.

For example, the consumer guide for men confronting decisions about what to do about localized prostate cancer actually has a list of questions that they can take with them when they go to see a cancer specialist. And I can tell you every one of my friends has been very, very excited about this.

Stephanie, I think you said that people could send in questions now?

>> CHANG: That's right. People can submit questions in the "ask a question" box. I think we're running a little behind. We'll just have 5 minutes for questions. Dr. Clancy, one of the questions is you talked about how the comparative effectiveness research is to be used for the individual patient. How does comparative effectiveness research fit with personalized medicine tailoring it to the individual?

>>CLANCY: My shorthand answer is hand in glove. Personalized medicine, although I'm not entirely clear what the complete definition is, does actually have the potential to bring online new kinds of treatments that really are customized for your disease. Now, today, if we were to have a breakthrough in research, what we'd see is that some people would benefit dramatically, but we wouldn't have a systematic way of identifying all of the people who could benefit so that we could make sure that all of those folks had the opportunity to benefit from this new breakthrough kind of treatments. And sometimes we might use it in patients who wouldn't experience such a great benefit.

We think that a big part of the infrastructure necessarily for this research—for example, patient registries—will help us identify which patients are likely to benefit so that all who are likely to benefit can be identified and given the opportunity to benefit and will also help us identify which patients are probably doing just as well on standard treatments.

>>CHANG: Thank you very much. We probably just have time for one more question. Can you tell us a little bit about how the research topics are chosen and how that is decided between AHRQ and PCORI and how those two institutions might interact?

>>CLANCY: First of all, I am so glad you asked that question. We have, at the Effective Health Care Web site, which is—if you land yourself on, you can find it very easily, as well—lots of opportunities to hear from you. And we very, very much want to. You can weigh in on research priorities. You can respond to explicit requests for comments on the key questions that have been posted. You can actually submit comments in response to draft reports.

Occasionally, these draft reports even generate their own news coverage in the trade press. And we think that is actually fantastic, because we believe that it's very important to support research that's both transparent, credible and that's ultimately trusted.

>>CHANG: Thank you, Dr. Clancy. One last question I think we can fit it in. Are you going to be encouraging other organizations around the country to also conduct effectiveness or comparative effectiveness studies?

>>CLANCY: Well, I'm pleased to say that as a result of all the research investments we've made this past year—we're still catching our breath a little bit—in patient-centered outcomes research, there are many, many organizations around the country now conducting some of this work. And those efforts range from efforts to improve care for individuals with multiple chronic conditions to support for registries, which are an important research tool, but also a vital tool to assess and improve quality of care. To say I'm excited about all of these investments is a profound understatement.

>>CHANG: Thank you, Dr. Clancy. I'm afraid we're out of time. I'm sorry for those we didn't get your questions. But, hopefully, if there's time in Jean's, some of them may be relevant there, as well.

>>CLANCY: I was honored to serve as the warm up act to Jean Slutsky, who has really been a fantastic leader for this Program. I'm happy to turn it over to her.

>>CHANG: Thank you, Dr. Clancy. Jean? I'll introduce her briefly. She's the Director for the Centers of Outcomes and Evidence and my boss. The Center oversees the Effective Health Care Program, which she'll review today and all the work that's going on there. Jean?

>>MS. JEAN SLUTSKY: Hi, good afternoon. As Stephanie said, I'm Jean Slutsky, and I direct the Center for Outcomes and Evidence at AHRQ and the agency's Effective Health Care Program. I would also like to thank all of you for joining us today. I know there's been some technical difficulty, so I just want to reassure you that this will be taped, so you will be able to see it with the slides after this program.

Dr. Clancy talked generally about comparative effectiveness or patient-centered outcomes research and the large investment that the Federal government has made in it to improve the health care quality of our patients.

I would like to provide a little bit more detail about the efforts that AHRQ has made in this area over the last several years, especially those things that may be useful to you with your care for your patients and ways that you can actually interact with us to make this program even more effective. Next slide, please.

Actually in 2005, AHRQ established the Effective Health Care Program. This program was initially authorized under the Medicare Modernization Act of 2003. And for the last 5 years, the Effective Health Care Program has been conducting comparative effectiveness research in order to help serve people who have to make decisions either because they’re patients or consumers, clinicians, policymakers, and to improve the quality effectiveness and efficiency of health care delivered. Initially, the program was intended to meet the needs of the Medicare population as well as the Medicaid and the State child health insurance programs, which are really cradle-to-grave programs. But obviously these issues are important to patients who are cared for under different programs and insurance in the United States.

Even though it has a new name, patient-centered outcomes research, the focus of the research doesn't change. It is always focused on clinical effectiveness and treatments and interventions and strategies for providing health care and their impact on patient-centered outcomes.

The research focuses primarily on what we know now and what are important research gaps that are critical to fill and how we can translate this information for users. Next slide, please.

This slide presents a framework for how AHRQ has made investments in comparative effectiveness research. And this framework is how we've built our program. And what you can see is, it really starts with horizon scanning or understanding what might be important things for which we need to make investments in comparative effectiveness research, as well as evidence synthesis, which is very good at letting us know what we know from previous investments in biomedical research, and then can very effectively identify research gaps. And those research gaps are important for both researchers to plan their research programs and for funders of research, as well as understanding which priorities we might want to set for new research.

And then we fund programs of evidence generation or new research on strategies to provide health care, different clinical interventions, and on conditions and populations.

And then one of the most important things is a translation and dissemination and implementation program. The legislation that actually authorized the Effective Health Care Program was very specific that research findings should be translated into usable and understandable products.

And, finally, we hope that this leads to an improvement in care and back to the horizon scanning. Underpinning this whole framework is actually a research platform which helps support the infrastructure to do research by both methods development and training of new researchers, as well as stakeholder input, getting the needs of people who have to make decisions in the health care arena. Next slide.

This program is actually focused around 14 priority conditions that were established by the Secretary of Health and Human Services. And as you can see, these are very, very important conditions that affect a large number of Americans. And these 14 priority conditions do guide the work that we do within the Program. Next slide, please.

Also, it's very important to understand that the Effective Health Care Program has a research focus on priority populations, which include low-income groups, minority groups, women, children, the elderly, and individuals with special health care needs, such as those with disabilities and those who need chronic care or end-of-life care, or those who live in intercity or rural areas. So, patient populations that may not be represented in traditional clinical studies are sometimes hard to study because of the complexity of their conditions. Next slide, please.

Since the inception of the Effective Health Care Program, AHRQ has funded and completed dozens of patient-centered comparative effectiveness research projects. These projects include comprehensive reviews of diagnostic or treatment options for breast and prostate cancers, atrial fibrillation, diabetes, osteoarthritis, depression and many other conditions. To date we completed over 25 research reviews, as well as 16 original research reports. Many, many more are in the pipeline.

We've also made great progress in the areas of methods research. We have completed and posted about 36 methods research reports, including guidance on methods for conducting comparative effectiveness reviews as well as designing, implementing a registry for evaluating patient outcomes, conducting diagnostic tests for improved health outcomes, heterogeneity of treatment effects, and many other research projects on quasi-experimental or observational studies. Next slide, please.

The Effective Health Care Program creates a variety of products that are based on research reviews and reports. And this is part of our mandate to actually translate our findings for a variety of different audiences. This includes executive summaries of systematic reviews, which are manuscript length. Plus, summary guides that are written for clinicians, specifically just the facts, consumers and patients and policymakers. In fact, the clinician guides are designed so that they can be used with the clinician and the patient or alone.

We have recently added to our portfolio a number of materials to support clinician education, including continuing education modules that can be used onsite or transposed to your own site, interactive case studies, and faculty slides sets. We'll soon be adding patient decision aids, as well. Next slide.

I would like to highlight more of our consumer guides that summarize evidence in plain language in easy-to-read formats. These guides are paired with our clinician guides to promote shared decisionmaking. Most of our consumer guides have also been translated into Spanish. And the consumer guides are also available in audio files. The consumer guides can be found online or are available in print. And we also have a mechanism where you can order them in bulk, as well.

Most importantly, in echoing Dr. Clancy's comments, we want to encourage you to get involved in the Effective Health Care Program. Your participation is mutually beneficial. There are multiple points of involvement in our program before, during, and after the research is completed. You can nominate topics for research on our Web site. That's very important to us to know what research questions are important to you.

You can also give input on draft key questions and reports. This kind of involvement helps you get the type of research that will really help answer those controversial questions or just nagging clinical questions. And it helps us by getting the research right. If it isn't relevant and applicable, then we can't expect to have an impact. If it doesn't meet your needs, you're not going to use it.

After the research is completed, you can help disseminate the information to your colleagues and patients. And you can implement the findings in your clinical decisions. This helps both you and us by creating opportunities for better and more informed decisionmaking and making an impact on the quality of health care. We also want you to give us feedback on how effective these documents and products were for you.

We also recognize the importance of information sharing among colleagues, friends, and family when it comes to health care decisionmaking. All of our materials are available for free on our Web site and can be shared by a variety of social media tools, buttons, and widgets. I say this thinking I'm not really sure what a widget is, but I know that those of you who are maybe younger than me or have teenagers are very familiar with widgets. We encourage you to share all of this information with your colleagues and patients and maybe even your teenagers. Next slide.

At this point, I'd like to shift gears a bit and focus on some of the actual comparative effectiveness research findings from AHRQ's Effective Health Care Program. These are still referred to as comparative effectiveness because the reports were published prior to the change in vernacular to patient-centered outcomes research, but these two terms can be used interchangeably. Next slide.

As you can see by the key questions identified here, this research review focused on a population of patients with stable ischemic heart disease and preserved left ventricular systolic function. The clinical trial evaluated in this comparative effectiveness research review included ischemic heart disease patients without left ventricular systolic function, which is defined as having a left ventricular ejection fractions greater than 40 percent. The key questions centered around the comparative effectiveness of one, adding an angiotensin-converting enzyme inhibitor, ACEI, or an angiotensin 2 receptor blocker are the standard medical therapy versus standard medical therapy alone.

Combining ACEI with an ARB and adding it to standard medical therapy versus adding an ASEI alone to standard medical therapy, or adding an ACEI or an ARB to standard medical therapy versus standard medical therapy alone in patients with stable ischemic heart disease and preserved left ventricular systolic function who are in close proximity to a revascularization procedure. The comparative effectiveness of ACEI versus ARBs was not a focus of this evidence review. And I should say, this is actually one of our CME modules. Next slide.

After an analysis of the evidence, the review provided several important insights. In an ancillary clinician guide developed to distill the findings, the clinical bottom line was summarized this way. Adding an ACEI or an ARB can provide additional clinical benefits for some patients. Adding an ACEI may increase the risk of cough, syncope, or hyperkalemia. Adding an ARB may increase the risk of hyperkalemia. And adding an ACEI does not impact cardiovascular mortality in patients with end-stage renal disease and left ventricular hypertrophy.

And as I said, I'm highlighting this particular evidence report, partly because we have recently released two CME activities and faculty slide sets based upon this report, and you can find these tools on the Effective Health Care Web site for free. Next slide.

Next, I'd like to highlight one of our most popular reports—the comparative effectiveness and safety review on oral diabetes medications with adults with type 2 diabetes. It was one of our first research reviews. It summarized evidence on the effectiveness and risk of all approved oral medications commonly used in the U.S. for type 2 diabetes in 2007. In that report it was shown that metformin was less likely to cause weight gain—always a good thing—and more likely to decrease LDL than other oral type 2 diabetes medications.

I'm highlighting this particular report partly because of its date. That is to say we recognize science and medicine are evolving and that our evidence reviews cannot become stagnant. With that in mind, ARHQ and the Effective Health Care Program are committed to regularly updating evidence reviews to keep pace with the science and to bring you the latest and best evidence. This particular report, as well as others, is currently being updated, as you will see noted on the Web site. Please keep an eye out for these updates and other developments by signing up for e-mail alerts on the Effective Health Care Web site.

The significant investment in patient-centered outcome research and the Effective Health Care Program has really bolstered our research pipeline. We have more than 100 research topics that are now in progress or in draft stage, including evidence syntheses, future research needs documents—of which several are all up on the Effective Health Care Web site for public comment—original research reports, and methods research.

Please keep in mind all of our research reports, products, and materials are available to you or your colleagues and your patients for free. You can get them at the Effective Health Care Web site, which is You can also call AHRQ publications clearinghouse at 1-800-358-9295.

I encourage you to learn more about the Effective Health Care Program and to become more involved by exploring the Web site, signing up for e-mail alerts and reading our newsletter. These resources will give you up-to-date information on new and updated clinical resources, plus provide opportunities to get involved with the Program. Remember, your involvement makes this program relevant and vital. Thank you so much.

>> CHANG: Thank you, Jean, for a great presentation and overview. I think we have about 5 minutes for questions. We'll see how many we get through. I'm going to combine a couple questions. One was, has there been any research around chronic pain and use of narcotic medications, and are there any plans to update the priority conditions?

>>SLUTSKY: Yes. We have received several nominations for chronic pain. And there are some activities ongoing now. And we also have some reports on pain for osteoarthritis, pain medications for osteoarthritis, as well as some work on palliative pain. A question about updating our priorities? Right now, we are looking at all the different possibilities, including the role that PCORI will play in setting priorities for patient-centered outcomes research.

>>CHANG: You mentioned the CME modules, which are going to be really useful for people that probably not everyone knew about. How did you select what CME modules are done?

>>SLUTSKY: That's a great question. We hope to actually have CME modules on all of our systematic reviews and to have a faculty slide set for a good subset of our new research. We are increasing the funding in this area of CME and CE. So, we really hope to increase this as we go forward. And those were just the first few that we had funded.

>>CHANG: You have a lot of resources. How do you actually disseminate the results of these studies or resources? How do they get to the patient individuals?

>>SLUTSKY: Yes. We use a variety of different mechanisms, including the translation products that I've told you about. Working with partners, other patient groups, sending it out to our listservs, publicizing it through Web conferences, health FAQs, podcasts, and going to health fairs where patients congregate to make this information available. We actually have podcasts that show up at grocery stores and so forth.

>> CHANG: There are so many issues that need to be studied. How many topics can AHRQ study at any one time?

>> SLUTSKY: That's a great question. We were very fortunate because of the ARRA dollars to be able to increase the production of what we could fund. But, of course, as we all know, that's not sustainable. What we try to do is pick those topics that are very important to users, people who actually need the information as they make health care decisions. Your involvement in letting us know what those topics are is very important in helping us make those important funding decisions.

>> CHANG: I guess a related question to that was asking about what criteria will be used to determine the most important conditions to study.

>> SLUTSKY: Yes. Good question. And that information is actually available on our Web site at www, which takes into account the burden of illness, the importance in terms of does it have a disproportionate effect on a certain population? And so forth. And those are the criteria that we take into account when we decide which is an important topic to go forward with.

>> CHANG: Another specific question about what CER research is being done on cancer types. For example, is there comparative research being done on different chemotherapy regimens versus other treatment modalities?

>> SLUTSKY: There is comparative effectiveness research that's ongoing on different cancers. For example, we are doing research on prostate cancer. We also have done reports on chemo prevention for breast cancer, the role of biopsy, DCIS. If you look at the Effective Health Care Program Web site, under cancer, searching under cancer, you'll see the entire portfolio of comparative effectiveness research that we're doing in cancer.

>> CHANG: Another question about the scope of research. Will your center also review and compare behavioral standards of care? If not, is there another agency tasked with this type of research?

>> SLUTSKY: Behavioral standards of care are certainly something that we can look at. Oftentimes, when we look at a topic that has multiple interventions, including pharmaceutical, behavioral interventions are also appropriate. So, we will compare across those interventions, including behavioral interventions.

>> CHANG: I think we have time for one more question, just handed to me right now. I guess this is related to a previous question. Other than the Web site, how do you plan on submitting CER research?

>> SLUTSKY: We use a variety of different mechanisms, including speeches, podcasts, audio files, publications, and partnerships leveraging with other organizations. AARP disseminates many of our consumer guides. We work with many of the medical organizations to disseminate guides to their constituents. We think that working with people who actually use the information is very important.

>> CHANG: Thanks. I guess this Web conference is another example of ways that we're trying to disseminate the information. Thank you for answering those questions and for your overview, Jean. I'm sure that the audience is excited to learn about the many resources that we have available. We hope that people will go to the Web site and check it out and continue to invite questions and give us feedback on it.

>> SLUTSKY: Thank you, Stephanie.

>> CHANG: Thank you. Next slide. Hopefully, on your screen is an overview of the agenda again. As a time check, we are about halfway through the Web conference, and the next section of this Web conference will feature presentations from the partners of the Effective Health Care Program. The first speaker up is Dr. Amir Qaseem, who is from the ACP. He's the director of the department of clinical policy at the American College of Physicians. Some of his responsibilities include overseeing the evidence-based medicine and clinical practice guidelines development. He'll share about how the ACP is using the AHRQ comparative effectiveness research to develop those for physicians. Dr. Qaseem?

>>DR. AMIR QASEEM: Thank you, Dr. Chang. Good afternoon, everyone. Thank you for attending this Web conference. Hopefully you can all see the slides in front of you.

I'm really glad that AHRQ actually picked this topic for a Web conference because I think it's really important to increase the awareness regarding the excellent and very valuable work I believe that's being done by Jean and Stephanie and their group in this field.

As Stephanie mentioned, my name is Amir Qaseem. I'm the director of clinical policy at the American College of Physicians. And I'm here to give you a brief overview of our guidelines program as well as our relationship with the AHRQ Effective Health Care Program.

Next slide, please. To understand the depth and the breadth of the impact of the Effective Health Care Program, I think it's really important for me to give you a brief background on ACP and let you know who we are. We are the largest medical specialty society in the United States with 130,000 members. Our membership includes internists, internal medicine, subspecialists, residents, fellows in training in internal medicine, and medical students. Our headquarters is in Philadelphia. We have an office down in DC, as well.

Next slide, please. ACP 's evidence-based guideline program was established in 1981 when evidence-based medicine was still in its stage of infancy in the United States. The American Cancer Society was the first who got on this train. And we, actually, within 2 weeks, were the second one to establish the program.

Our clinical practice guidelines were developed by a clinical guidelines committee, and the program has evolved over the years. Initially our guidelines focused on diagnostic tests and technologies, and now our guidelines address screening, diagnosis, and treatment.

Next slide, please. For ACP clinical policies, there are three products developed by us. Clinical guidelines for which we do a systematic literature review and gather evidence from randomized controlled trials, meta analysis, et cetera. And this is one of the products we really rely heavily on from AHRQ's Effective Health Care Program.

We also develop guidance statements where we look for available guidelines on the topic. I'm not going to go into details about describing these products. You can find the details on our Web site.

Our third product is a high-value, cost-conscious care initiative. It is something very new where what we do is we regulate the benefits, harms, and costs of various diagnostic and therapeutic interventions and translate them into value of an intervention. Again, for this one, we really will be relying on AHRQ's Effective Health Care Programs.

So, we do evidence reviews. We follow three pathways. One is, of course, ACP-funded. We fund the evidence reviews ourselves. Second one is we work together with other societies. What I'm going to focus on today is the AHRQ and their evidence-based practice centers and their Effective Health Care Program. EPCs are the academic centers that do the AHRQs comparative effectiveness research, and they're located at various places, the nation's top medical schools, universities, medical centers that conduct AHRQ's research reviews.

Next slide, please. We have a very long and successful history of working together with AHRQ. We have been working together with AHRQ since 1999, and we have utilized several EPC reports in the past. And now since the establishment of the Effective Health Care Program in 2005, we now work utilizing these reports, as well.

The way it works is we nominate a topic to AHRQ. If AHRQ decides it's an important topic, they assign the topic to an EPC. We're usually part of the technical expert panel. And again I will save you the details. But we work together with the evidence-based practice centers in the development of the key questions, the refinement, the methodology, et cetera. And these EPC reviews, many of you who have seen it or those you haven't seen, they're very extensive. They're excellent evidence reviews. They have evolved a lot. It's become a science on its own. It's a very advanced, challenging field.

And I believe that EPCs do an excellent job with these evidence reports. First of all, they're very expensive. It's very difficult for us to develop multiple guidelines on different topics and be able to afford the cost. Again, don't quote me on this. I believe each EPC can cost potentially up to $400,000. So, if you imagine the amount of work that goes into these EPC reports, I consider these reports the gold standard of systematic reviews. I've studied many times in the past, as well, that EPC reports are just so detailed, and they cover every single detail on the key questions that are being addressed.

A great majority of the guidelines, about 50 to 66 percent, of our guidelines to be exact, are based on various EPC or Effective Health Care Program reports.

Now to give you a feel for what happens or what do folks think of our guidelines that are based on this Effective Health Care Program and EPC report, our guidelines are the top three most valued products of the college. It consistently had been there for a long time. Next slide, please. I'm sorry I'm not keeping up with the slides.

They're the most common reason that our members and nonmembers and other clinicians come to our Web site and look for information. They are 25 of the top 100 most read articles ever in the Annals of Internal Medicine, which I'm sure most of you are aware of. Our guidelines are also many times the top most read article in Internal Medicine, Medscape. And our guidelines are regularly covered by print, TV, radio, and online stories. Next slide, please.

You will have a table in front of you right now. And just to give you a feel for our recent guidelines, starting the first one is our erectile dysfunction diagnosis and management guidelines. It was covered by 121 print and Internet reports and total audience of 52 million. I'm going to save you details. You can see all these numbers in front of you. Some of our guidelines receive very good coverage from TV, as well as our osteoporosis guidelines, our COPD guidelines. So, this table, the intent is to just give you a feel for a flavor for how our guidelines are being covered. Next slide, please.

And this slide just shows you the various evidence reports that we have used in the past. If you want to look at the details of these evidence reports, they're available on AHRQ's Web site. Next slide, please.

As far as feedback from our membership, we have sometimes asked our membership to evaluate our guidelines. In a recent survey, a majority of the membership, almost 90 percent, told us our guidelines are very high-quality guidelines and they're based on scientific evidence. And that's the key. Because as I mentioned, the majority of our guidelines are being developed based upon the EPCs reports. They find them a helpful source of advice, not rigid or difficult to apply and not difficult to understand or use. You can see how practical these reports are.

Next slide, please? And here again is just a list. I'm not going to go over them. Here's a list of various topics that we are currently working on. And these guidelines will be based on the EPC research reports, management of patients with chronic kidney disease, obstructive sleep apnea, urinary incontinence, et cetera. Just as you can see our relationship with the AHRQ has been so successful that we are now actually increasing the number of guidelines that are now being based out of the AHRQ evidence reports.

And at this point, I am going to pass it back to Stephanie.

>>CHANG: Thank you, Dr. Qaseem. We're going to hold questions until the panel of the three speakers are done. But please do be submitting your questions in the “ask a question” box, and we'll take them all at the end together.

Our next presenter is Dr. Goolsby, Mary Jo Goolsby, who is the director of education and research at the American Academy of Nurse Practitioners. She oversees all of the ANP research initiatives in the Continuing Ed Program. She'll explain how she sees the Effective Health Care's clinician guides in incorporating them into the nurse practitioner Continuing Ed curriculum. Dr. Goolsby?

>>DR. MARY JO GOOLSBY: Thank you, Dr. Chang. And thanks to AHRQ for letting us talk about our opportunities or our experience in helping to disseminate the information about the Effective Health Care Program. As background, back in 2009—well, earlier than that, we started promoting the information that the comparative effectiveness research reports were available. And after a while, we realized that one way to help our nurse practitioners access this information would be to tie some CE credit to it, which would be some sort of an incentive. But also we know from experience in surveys that nurse practitioners consistently say that they most value information about new drugs, innovations, et cetera, when they receive it through accredited CE first and then peer review journals secondly.

So, we developed a series of CE programs, or are developing a series. And I believe this was before the other CME programs were available. We were not aware of them at the time. But our intent was to really provide resources that would help nurse practitioners and other clinicians have informed decisionmaking. But also help them be aware of the same sorts of resources that would help their patients make informed decisions.

The next two slides will just talk about the nurse practitioner role because I'm not sure how much learners may or attendees may be familiar with nurse practitioners. And these are really our learners. Nurse practitioners are independently licensed providers. And the point is that we diagnose and treat a full range and scope of conditions based on whatever our prepared specialty is. And we do prescribe in all 50 states which some people are not familiar with.

The next slide gives you a sense of the growth of the nurse practitioner role over the past few years. There are currently 140,000 nurse practitioners in the U.S. And the majority of nurse practitioners continue to go into primary care so that the full range of conditions that people present at different stages in their life are certainly relevant to our members.

The next slide we talk about is the CE center. And the point is that AANP is the largest nurse practitioner organization and we certainly value the ability, the opportunity to help enhance our members' education and practice. We've had a long-time experience in offering continuing education. But 3 years ago we launched our online CE center which now has over 16,000 learners. We have about 65 programs at any particular time.

The comparative effectiveness programs are just one of many different types. We have monographs, multimedia, simulations—you can imagine the full range. But these comparative effectiveness clinical guidelines or clinician guidelines really are helpful, because they provide very practical point of care, almost learning that people can complete at any time during their day and pretty easily.

The next slide shows the three programs that we have currently taken up as CE. We started with ACEIs versus ARBs in the premix insulin analogs because diabetes and hypertension are among the highest, most frequently treated conditions by nurse practitioners. They're also among the highest requested CE type of information that our members are asking for. We recently, within the last couple of weeks, launched the program on fracture prevention, postmenopausal women, and osteoporosis. The next slide provides just a summary of the number of people who have completed the CE programs in each of the—at least the two that have been up there for approximately a year. In looking back, we actually have more heavily promoted the one on hypertension or ACEIs and ARBs that we need to get to work on promoting the pre-mixed insulin analogs a little bit more heavily.

The next slide just wanted to summarize that our evaluations and comments we received through our standard program evaluation have been universally favorable. Before we launch any CE program, we typically put it through a pilot process. We have 1,000 volunteers. Not all complete every program we put up. But they do if it's relevant to their practice. By the time we launch, we expect that we're going to get positive comments. And this was no exception.

The objectives that we chose, the learners said that the programs matched those objectives and that the content was definitely relevant to their practice and that they found the resources useful.

In preparing for this presentation, we actually last week did a quick survey. It was up for about a week. Almost 1,200 NPs that had finished the posttest in at least one of these programs and had a good response rate for a one-week survey. Ninety percent of our respondents were family or adult nurse practitioners. You could see that 98 percent found that the discussions were very helpful in their clinical practice, and over half had actually gone on to access the full report in the patient guide from the related programs.

In the next slide, I just have some excerpts from the comments that people added to the survey. They found that the programs were certainly practical and relevant. They liked the concise delivery of the information. And they appreciated that the programs were certainly based on evidence.

And then the final slide gives some more excerpts. And the thing that I think is really interesting about this, the last bullet there is really something that shows how these kinds of programs and dissemination can take legs. Really, this person had used the information from the very brief clinician guides to prepare lectures for her community. She discussed it with her patients. She used the information to discuss with colleagues and certainly to inform her practice. And that's exactly what these very relevant and practical guides are meant to do.

In summary, there's a slide just to reiterate that we found the online process of promoting these clinician guides to be very successful. We plan to continue it. We believe that it's a good way to enhance the uptake of this information in the clinical setting. And so we thank you for your attention.

>>CHANG: Thank you, Dr. Goolsby. Again, don't forget to submit your questions through the “ask a question” button. And we'll take questions after this last presenter, who is Dr. Nilay Shah, who is an assistant professor of health service research at Mayo Clinic. He is also the codirector of the Knowledge Translation Research Unit, which is a part of the Mayo Clinic's CTSA, Center for Translational Science Activities. And he's going to talk about how his unit developed the patient decision aid using the comparative effectiveness research by AHRQ on oral medications for with type 2 diabetes. Dr. Shah?

>>DR. NILAY SHAH: Excellent, Stephanie. This actually ties in really well relative to the last few presentations and sort of puts a lot of the information that was discussed earlier in how we would translate it into a patient decision aid.

I work in the knowledge and encounter research unit, which is a patient-centered knowledge translation unit. A lot of what we do is develop tools that are useful in the patient clinician encounter. A lot of the material from the Effective Health Care Program comes in very handy in developing these tools.

The next slide lists the main disclosures. The development of the decision aid was funded by the American Diabetes Association. Currently, AHRQ is funding the larger trial that's ongoing to look at further, broader implementation. There's a broad multidisciplinary research team that was involved in developing and testing the decision aids.

The next slide gets down to the reason where we started thinking about developing this decision aid. Going back to the comparative effectiveness and safety of oral diabetes medications, the report that was initially published on the Effective Health Care Program Web site and subsequently published in the Annals of Internal Medicine provided a lot of information, but basically said that for most oral diabetes medications, the decrease in hemoglobin A1C was similar across medications. However, there was limited information on all cause mortality cardiovascular morbidity, microvascular outcomes less functional status and quality of life. It's pretty important, and yet it's not covered in clinical trials, so the synthesis in the evidence reviews are not able to have good enough evidence to comment on these specific aspects.

However, what they did find with these reviews is there was quite a bit of differences in side effects in how you took these medications, some things that would be important to the patients. So, that's where we sort of started thinking about developing a decision-making tool for the patient-clinician encounter.

The next slide shows that historically the comparative effectiveness research work focused more on the coverage decision. A lot of peers have used it to decide what they're going to cover, what they're not going to cover, and so forth. Our goal was to create a tool that can be used in the clinical encounter. However, we are now testing how it can be used outside the clinical encounter. But so far a lot of our use has been within the clinical encounter, and that seems to be the best place, at least based on our initial work.

The next slide shows the rationale of why we went this route. Oftentimes what happens, especially with diabetes medications, patients and clinicians discuss the medications and then they decide on an option. The patient leaves the consultation with the prescription. And they make a decision about the medication somewhere down the line.

A lot of recent data suggests about 20 percent primary nonadherence related to medications for diabetes that about one in five people don't even fill the initial prescription for their diabetes medication. The hope is, can we bring this conversation back into the clinical encounter and involve the patient in decisionmaking and hopefully lead to higher adherence? And to do this, we want to develop a decision aid which incorporates two main issues. One, the research evidence, which was from the evidence review that was published through the Effective Health Care Program. And the patient values and preferences that would be important in the exam room as decisions related to diabetes medications were made.

On the next slide, this sort of presents the basic process we went through in developing this decision aid. We took the research of it as well as reviewed the practice. We observed from encounters where these decisions were made about diabetes medications and what was happening. And then we also included diabetes patients we'd like to recruit as well as the live clinical setting to iterate these decision aids.

Our first iteration of this decision aid is what we call the baseball cards. And we thought it looked great. It would be useful to patients and so forth. But what we found out when we actually observed it in a live clinical setting, patients and clinicians thought, oh, these look wonderful, they're great. But they just put it to the side and didn't really use it to create the conversation we were hoping.

Then on the next slide, it sort of shows the second iteration of this decision aid. This is what we call the narrative card, something that we thought the patients and clinicians might find useful. However, in general, it led to the same issue. While it looked great, most patients and clinicians thought they looked great, but they didn't really use it in the clinical encounter.

We went through a multiple iteration process and came up with what we called, finally what's on the next slide, the initial iteration of the issue card. On this slide, this is what we tested in our initial pilot randomized trial, and I'll talk about the overall results of these issue cards. But now we are testing a broader version of it in a larger 20-site trial in primary care practices right now. And this is on the next two slides.

One of the additional issues that came up or that we observed in our initial trial, that we incorporated into our new set of cards was the issue of cost. That patients and clinicians wanted to discuss cost but they didn't have good material. On the following slide, you can see we created a cost slide. We'll learn how this works in the actual clinical encounter.

In summary, our pilot trial of this decision aid, which is on the next slide, the use of the decision aid in primary care practice was acceptable and efficient. Most of the concerns that we always hear from primary care clinicians, we don't have time to incorporate this into our routine practice. And what they found was, on average, it increased encounter length by no more than two minutes, and oftentimes it was either less or about the same timeframe.

Overall, the patients found the tool helpful, and this is from patient surveys after the visit. There was improved knowledge for the patients about the different medication options. There was increased involvement. Not a whole lot on decisional conflict and not much on nature of choices made. Now again, this is the pilot trial, about 100 patients. But we also videotaped a lot of the encounters to understand the nature of the conversations that were taking place. They really told a different story, and we're working on a qualitative study of this. You could really observe a greater patient participation in different sets of conversations, different issues that were being brought up.

In the pilot trial, we didn't see any difference in medication adherence, which is one of the other issues that we hoped this would help with. The problem there was our control group had 100 percent adherence so we weren't surprised to see much difference there.

Finally, just to give you some next steps we are working on is trying to take this out to a more diverse population. We are testing this on 20 sites, half of them randomized at the site level. And looking at longitudinal followup, what happens with repeated use and as patients gain more knowledge about this tool.

We also want to study the implementations. What happens when the study goes away? We don't recruit the patients, but do clinicians continue using the tool? That's one of the keys here. We can develop a lot of tools, but if clinicians don't use it, they aren't as helpful. We're using normalization process theories developed by Carl May to address how practices use this tool further. We are also in the process; we recently received an award from AHRQ through the IDAP mechanism to design and test the decision aid for depression medications. Again, using the reviews from the Effective Health Care Program, we are in the process of working on that.

The main issue here on the next slide as you can see as we wrap this up, the patient important outcomes are challenging because they aren't captured in clinical trials as my colleague Victor Montori has a paper describing that. That leads to not having all the information for the tools that may be valuable for patients. And then, of course, there are practice challenges and costs associated with the decision aid for multiple perspectives, from a peer perspective as well as a practice perspective. But we are trying to learn more about the feasibility in everyday practice.

On the next slide you can learn more about the actual study, and you can, of course, feel free to e-mail me with further questions. The study was published in Archives of Internal Medicine last year. And the flash version of the cards is available at the Web site there. However, it's not the most recent version of the cards, but it will be up pretty soon. Thank you.

>>CHANG: Thank you, Dr. Shah. At this point, we'll take questions for all three speakers who have implemented some of the AHRQ's comparative effectiveness research. We have about 10 minutes for questions. We'll start out with a question for all three of our speakers. Can you talk a little bit more about the time, the burden that your organization's invested into developing these resources from the research in terms of the cost, the time, the staffing level? What is the amount of work it would take for other organizations considering the same thing?

>> GOOLSBY: I can start from the AANP's perspective because ours was probably the simplest process because we really merely repurposed the PDFs of the clinician guides and put them on our CE center. Of course, there was the development of the objectives and the post tests and the maintenance of the records and the certifications and that type of thing. But in reality, it was not a huge burden, and it benefited our members and we believe that it benefits their patients. It was certainly a worthwhile process.

>> SHAH: From our development of the patient decisioning, obviously the Effective Health Care Program, the review really reduced the amount of work we had to do in synthesizing the evidence since it was all there. However, when you start thinking about the amount of work that gets into developing the decision aid, it turned out to be, I think, quite a bit more than we expected, to at least have it be a tool that can be meaningfully used in practice. It really took a lot more iterations than we had initially expected. But having the evidence reviews significantly helped.

>> QASEEM: Dr. Chang, it's a really good question, but it's a little difficult question to answer because those of us who are in the guideline development business, we understand. It's a very extensive process. There's a lot that goes into the development of the guideline. But one thing I can say is that with this Effective Health Care Program, though, it makes it simpler because at least we have a good evidence review at this point to deal with. And then we can take it to the next stage of development of the guideline and translating this into guideline. That takes a huge workload off.

There is still a lot of stuff that goes into the specifics of the recommendation and development of the guidelines. And there's a whole guideline committee that's involved. There are staff-level resources that are utilized at the American College of Physicians. It's a short answer for one minute.

>>CHANG: Dr. Qaseem, when you are deciding which guidelines to do, how do you make sure that they're not redundant with other guidelines? How do you decide which guidelines you're going to do? And specifically do you have any guidelines related to alcohol withdrawal for inpatients?

>>QASEEM: Starting out with your second question, we don't have a guideline on alcohol or anything to do with alcohol abuse. But the way our guideline process works out is we have topic selection criteria. There is a whole list of them: Impact on mortality, morbidity, prevalence of a disease, whether effective health care is available, areas of uncertainty, whether evidence that current practice might be deficient costs. We take those all into account. And if there is a belief there is a need to develop a guideline on the topic, we take it from there.

Now, how do we decide whether there are other guidelines available on the same topic, whether we're going to go ahead and develop a guideline or not is we look at the guidelines that are currently available. If we feel like they do not serve the needs of the internists, at that point we take it up to develop a guideline. And also we have the guidance statement, as I mentioned, where we summarize the guidelines that have been developed by other organizations.

If someone wants to look at the details, we just recently had a paper published in the Annals of Internal Medicine versus the ACP's development process, and you can find the details on it there.

>>CHANG: Thank you. Obviously, sometimes things need to be tailored to each organization's groups. And sometimes we can repurpose other resources. Dr. Goolsby, Jean Slutsky talked about the CME modules that AHRQ has developed. How have the nurse practitioners used this? How have they received the other continuing education resources that have been developed?

>>GOOLSBY: Well, admittedly I don't know if they've used it or not. The AHRQ CME doesn't — it has AMA credit and nurse practitioners typically prefer having AANP credit. We sort out the CE that's relevant to pharmacology which is important to them in many States. I would imagine, though, that many are completing them. Those would be longer modules compared to ours, which would probably take about 20 or 30 minutes, including the posttest. It’s just a different style of learning. And because there's such a variety, I would imagine that there's a good uptake there, as well.

>>CHANG: Thank you. Dr. Shah, the tools that you developed for physicians were really interesting. Especially interesting was the learning process of which ones were useful and what was not. You mentioned costs as being one of the major factors that made it more well-received. Two questions. How did you account for the differences in the insurance prices? Or what measure of cost did you use? And another thing I noticed was that the format was different. It was more pictures, much more simplified. How did you avoid oversimplifying the messages?

>>SHAH: Yeah, I think that's a great question, actually. It’s one of the important learnings of the process that we went through in developing the tools. To answer your first question about costs, this was really difficult. That's part of the reason we didn't have a cost card in the initial iteration because there's no single cost for a whole population. In the iteration where we did use the cost, we tried to use essentially what would be the expected out-of-pocket price if you were going to pay for the medication.

In terms of the second question of oversimplifying, it actually ended up being where a question of less is more in that it allowed patients and clinicians to not focus on sort of the details, which is what our first two iterations had, and get the overlying message and then help that to create the conversation between the clinician and the patient.

>>CHANG: I think I'm going to have time for one more question. I will give that question to Dr. Qaseem, which is how and when do you decide to update a guideline?

>>QASEEM: That's another excellent question, Dr. Chang. And that's something that we have been discussing at the college. We're trying to figure out the guideline process. Again, the short answer, because I could go on for an hour to describe it, is currently ACP's guidelines are valid only for 5 years. That's their shelf life. But generally what we do is we are trying to develop a formal process when we decide whether or not to update the guideline based upon if the evidence has changed or not. But for now, it's an informal process.

We keep our eyes open in terms of if there are some good studies that came out that might need for us to revisit our guidelines, but we are going toward the direction of formalizing that process. And also we are going to be working very closely with the AHRQ's Effective Health Care Program, as well. If there's a need for a literature update, we will work with AHRQ and utilize those resources and help update our guidelines. Thanks.

>>CHANG: I think our time is just about up. Thank you. If we didn't get to your question today, I just wanted to remind you that you can e-mail us at EHC underscore I think we will try to post answers to the questions on the Web site later on.

I hope everyone will thank all of our speakers for presenting today. Dr. Carolyn Clancy and Jean Slutsky, thanks for giving a great overview of the patient-centered outcomes research and the AHRQ Effective Health Care Program. Dr. Qaseem, Dr. Goolsby and Dr. Shah, thank you for sharing with us how much you value the Effective Health Care Program and the research and translation projects. And thank you for partnering with us to develop the practice and use for your practitioners.

And I especially want to thank our many participants on the phone for joining us today. While we strive to produce high quality research, it's really only as useful as clinicians, patients, and other health care partners and providers find it. We hope that the information presented today was enlightening and was useful and stimulating for how you might implement this type of research in your everyday practice.

We hope that you're aware of the variety of Effective Health Care Program resources that are available to assist you. Please go to the Web site, and look at the tools, including the reports, the guides, the continuing education modules. Share them with your colleagues in practice using the social media tools. Share the information and the guides with your patients as you work together as a team to make the best treatment choices.

As Jean mentioned, if you'd like more information, the Web site's there. You can download and print all of our materials. And if you want bulk quantities of the summary guides to share with your patients or colleagues, you can order them for free through the AHRQ publications clearinghouse.

We hope you've been intrigued about the way that you can be involved in the program and continue to improve the quality of health care in America. We invite you to participate in future AHRQ and Effective Health Care Program events like this. Suggest topics for research, provide inputs on research questions, draft research reports. You can do this by signing up for our e-mail listserv and your e-mail inbox will be notified for the latest evidence, opportunities to comment, to partner, to nominate things. And, again, just a reminder, if you have any questions about this conference or how to implement research into your practice, e-mail us at EHC underscore

Thank you for participating in the Web conference. Let me remind you that this event will be archived on the Effective Health Care Web site in a few weeks.

Before you leave, please do answer the one feedback question that will appear on your screen. Your feedback is important to us as we develop more new services and plan similar events like this. We'd like to hear about how it was helpful or what we can do to meet your needs. Thank you very much. And thank you to the team here that's been coordinating and working through this.

Have a good day.