Background and Objectives for the Technical Brief

Provisions of the Affordable Care Act (2010) require the use of validated quality outcomes to evaluate health care programs and services. The impetus for program evaluation stems from a desire to optimize the balance between patient health and health care costs. The need for such a balance is crucial in serious mental illness (SMI), where adequate treatment requires long-term follow-up and places a heavy demand on often-strained health care resources. Although stakeholders have proposed a variety of quality measures (QMs), no single validated measure is used consistently across all treatment sites or all forms of SMI.

Several key issues highlight the importance of this topic. First, the prevalence and morbidity of SMI are striking. SMI, most commonly referring to a diagnosis of psychotic disorders, bipolar disorder, and either major depression with psychotic features or treatment-resistant depression, is a long-term illness involving substantial functional impairment over multiple symptom domains. These impairments often lead to an inability to work, poor social relations, substance abuse, dangerous and reckless behaviors, repeated psychiatric hospitalizations, poor self-care, and homelessness. Rates of SMI for adults range from 4 percent to 6 percent, affecting more than 11 million adults.^1,2 Second, SMI and its comorbidities are frequently untreated or undertreated. Among adults with an SMI in 2008, less than 60 percent used any mental health services in the past year, and only 40 percent used any outpatient services.³ Third, even in those receiving treatment, the quality of care received for both psychiatric⁴ and general medical conditions⁵ is frequently inadequate. SMI patients die 10 to 25 years earlier than patients without these illnesses, primarily from cardiovascular disorders.^6,7 Fourth, the costs of SMI are substantial. Many SMI patients require inpatient hospitalization at multiple times in their lives, and emergency department visits are not uncommon.⁸ The cost of antipsychotic medications is typically the single largest cost to most state pharmacy budgets.⁹ The cost in terms of lost productivity is an estimated $193 billion per year in lost earnings.¹⁰

By improving access to and delivery of quality care, the use of QMs is one means to improve outcomes in SMI populations. Several groups have developed QMs and guidelines that may be suitable for SMI. In 2004, an international expert group developed consensus-based measures for the Organization for Economic Cooperation and Development (OECD) countries that address aspects of care for SMI. In 2006, the Institute of Medicine (IOM) described an approach for developing, testing, and validating quality measures for mental health, and two key organizations are involved in developing and endorsing quality measures based on the IOM recommendations. In keeping with this approach, the National Committee on Quality Assurance (NCQA) develops measures, or identifies any existing measure that appears appropriate for use as a QM. These are subsequently forwarded to the National Quality Forum (NQF) for rigorous review. Measures that meet NQF review criteria are then endorsed. Many measures are not endorsed because they lack sufficient evidence to support them. Both NQF and NCQA have endorsed several measures that are applicable to SMI. Seven of these are based on the Healthcare Effectiveness Data and Information Set (HEDIS) measures established by the Centers for Medicare & Medicaid Services (CMS) for mental health care provided to Medicaid patients.

Multiple barriers hinder the use of QMs in the SMI population. The level of agreement among experts is low about how to define a QM; what outcomes QMs should measure; which QMs should be applied; how to select the appropriate QM; how QMs should be developed, endorsed, and administered; and how their use affects patient-relevant outcomes. The evidence base linking quality measures to improved health outcomes is limited, a constraint made more challenging by the thin evidence base addressing the long-term effectiveness of interventions. Not surprisingly, although stakeholders have proposed a variety of such measures for SMI, no single measure or metric is used consistently across all treatment sites (e.g., inpatient vs. outpatient settings, community mental health vs. private settings) or all forms of SMI. Key knowledge gaps for SMI include an agreed-upon list of relevant QMs and robust assessments of whether use of such measures improves medical, psychiatric, and patient-centered outcomes.

The goal of the proposed technical brief is to describe the current state of the art of various QMs and to describe the evidence supporting their use. The technical brief will interview key informants—including quality measure developers, quality measure endorsers, and quality measure implementers—and it will survey the current QM literature relating to SMI. Our brief will examine (1) the measures, (2) their context, (3) ongoing research, and (4) future research directions and other issues.

Guiding Questions

1. Describe Quality Measures
   1. How should QMs be defined, identified, and implemented?
   2. Regarding outcomes against which QMs should be evaluated
      • What are the important psychiatric, medical, and patient-centered outcomes?
      • Should QMs be more strongly linked with fidelity to best practices or with outcomes?
   3. What are the different types of QMs currently used to assess quality of care and outcomes in patients with SMI?
   4. What factors affect the implementation of currently used QMs?
      • What are the potential advantages of each of these QMs in patients with SMI?
      • What are the potential disadvantages of each of these QMs in patients with SMI?
      • What are barriers to their use?
      • What are the facilitators to their use?
   5. How useful are the currently available QMs?
      • Are QMs currently being used for patient care and clinical decisionmaking?
      • Are the correct QMs being used?
      • What are the necessary steps to improve the use and usefulness of QMs?
   6. How should QMs be used to measure outcomes over short-term, intermediate, and long-term periods?
   7. What are the harms of QM implementation?
2. Describe the Context in Which QMs for SMI Are Used
   1. What organizational components or characteristics of a hospital or other mental health care delivery facility affect selection of QMs?
   2. What clinical characteristics of the SMI diagnosis (e.g., schizophrenia vs. bipolar disorder) affect selection of QMs?
   3. What clinician characteristics (e.g., training, specialty, setting) affect selection of QMs?
   4. What nonclinical patient characteristics (e.g., sociodemographic characteristics, insurance status, urban/rural setting) affect selection of QMs?
   5. What resources (e.g., organizational structure, staffing, appointment availability) are needed to implement QMs?
3. Describe Current Evidence on Whether the Use of QMs for SMI Affects Outcomes
4. Current Key Issues in Future Uptake, Use, Evidence Gaps and Research Priorities for QMs
   1. What are the trends in uptake of QMs for SMI?
   2. Can the use of QMs alter uncertainty in clinical decisionmaking?
   3. What gaps exist in the evidence base for best practices or interventions for QMs for SMI?
   4. How should one prioritize the key areas for future research?
   5. What are the expected short-term developments in this field that may address these uncertainties or influence diffusion of QMs?
   6. What are potential long-term (10-year) developments in this field?

Methods

1. Data Collection

Information to address our Guiding Questions (GQs) will come from three sources: Key Informants (KIs), gray literature searches, and published literature searches.

A. Discussions with Key Informants

KIs are particularly vital to shaping this technical brief because little agreement exists on which QMs are most useful; data addressing the validity of QMs are limited; information on the real world use of QMs is scarce; and there is little evidence identifying which components of care are most strongly associated with improved outcomes (and, hence, should be measured by QMs). Therefore, KIs can identify the best available and most useful QMs, provide information on whether their implementation affects outcome, and identify key research gaps for QM development and implementation.

Specifically, our responses to GQs 1 (description and definition of QMs in SMI), 2 (context for use of QMs in SMI), and 4 (key remaining QM issues in SMI) will be based primarily on KI discussions. Subquestions under guiding questions 1, 2, and 4 serve as prompts to discuss issues further: we may follow new avenues of discussion, should conversations with KIs reveal new insights that require further exploration. Also, we do not anticipate that all KIs can or should answer all GQs. We anticipate asking KIs questions relevant to their specific expertise and exploring general topics as time permits. For GQs 1, 2, and 4, we will review the literature in parallel with KI interviews and explore points of commonality or departure between KI insights and the published literature in our analysis. Our review of the literature will be targeted and will rely on the best and most recent evidence available. For GQ 3, we will conduct a comprehensive and systematic search of the peer-reviewed and gray literature and present all available and eligible evidence. KI insights will help us identify ongoing or planned research.

Identifying Experts Without Conflicts of Interest (COIs). We will determine possible COIs for research teams and stakeholders at the start of a project and will consult with AHRQ about disclosure or removal from the project for those individuals with clear financial or intellectual COIs. These specific steps may be insufficient, however, for ensuring freedom from bias. Other requirements include ensuring balance in perspectives and interests for stakeholder groups and our core teams. Our aim is to provide AHRQ with a technical brief that is as objective and unbiased as possible.

Engaging Relevant Stakeholder Groups. Stakeholder and partner engagement ensures usability and applicability of EPC products and, therefore, is critical to AHRQ’s mission. When engaging stakeholders, we will aim to ensure a balance of viewpoints. We will engage KIs via teleconference, with targeted email communication as needed. We will provide materials for review 1 week before calls, with reminder emails to KIs 2 to 3 days before the scheduled teleconference. We generally have specific questions for stakeholder input, but we will also provide time on calls for suggestions about our GQs. Further, we will obtain input from diverse stakeholders through peer review and public comment.

In consultation with AHRQ, we will identify the distinct perspectives that are essential for informing a well-rounded and balanced technical brief about QMs for SMI. Specifically, we will seek to recruit the following as KIs: QM developers, QM endorsers, QM implementers, mental health providers, policymakers, patient advocacy groups, and payers.

To facilitate broad participation, we will use staff with substantial experience in moderating calls, follow semistructured guides with built-in places for various stakeholders to provide input, call on silent individuals to elicit their views, redirect conversations as needed, and offer opportunities for feedback through other media (e.g., via email). We will adhere to all Office of Management and Budget (OMB) requirements and limit our standardized questions to no more than nine nongovernment-associated individuals so that we will not need to obtain OMB clearance for the interview activities.

B. Gray Literature Search

We will use the gray literature to identify information beyond the published literature on quality measures outcomes in treatment of SMI. Sources for the gray literature include the following:

• HAPI: Health and Psychosocial Instruments provide bibliographic access and descriptions of tests, manuals, rating scales, and other instruments used to assess health and behavior. Assists researchers and others in locating instruments used in the health fields, psychosocial sciences, occupational sciences, library and information science, and education.
• OpenSIGLE: Operated by GreyNet, the OpenSIGLE Repository preserves and makes openly accessible research results originating in the International Conference Series on Grey Literature. GreyNet together with the Institute for Scientific and Technical Information-National Center for Scientific Research designed the format for a metadata record, which encompasses standardized PDF attachments for full-text conference preprints, PowerPoint presentations, abstracts and biographical notes. All 11 volumes (1993–2009) of the GL Conference Proceedings are available in the OpenSIGLE Repository.
• ClinicalTrials.gov: ClinicalTrials.gov offers up-to-date information for locating federally and privately supported clinical trials for a wide range of diseases and conditions. The site currently contains approximately 12,400 clinical studies sponsored by the National Institutes of Health, other federal agencies, and private industry. Studies listed in the database are conducted in all 50 States and in more than 100 countries.
• Academic Search Complete: This source provides information from a wide range of academic areas, including business, social sciences, humanities, general academic, general science, education and multicultural topics. This multidisciplinary database features full-text for more than 4,000 journals with many dating back to 1975, abstracts and indexing for more than 8,200 scholarly journals, and coverage of selected newspapers and other news sources.
• NIH RePORTER: The information found in RePORTER is drawn from several extant databases (eRA databases, Medline®, PubMed Central, the NIH Intramural Database, and iEdison), using newly formed linkages among these disparate data sources.

We will also search Web sites of the National Guidelines Clearinghouse (NGC), the National Quality Measures Clearinghouse (NQMC), The Joint Commission, and other relevant organizations.

C. Published Literature Search

We will systematically search the published literature for information to address our Guiding Questions.

Planned Databases. We propose to conduct searches in PubMed (MEDLINE), Excerpta Medica (EMBASE), PsycINFO, and the Cochrane Library, and to perform gray literature searches (see above).

Draft Search Strategy. An experienced research librarian developed our draft search strategy during our refinement of this topic (Table 1). In a second round of searches, we will look for more studies in PubMed, PsycINFO, and HAPI for each specific measure or instrument identified in the first round of searches. We will review the reference lists of identified papers and reviews to identify additional relevant papers. We will update the literature review by repeating the initial search concurrent with the peer review process. In addition, we will examine any literature suggested by KIs, Peer Reviewers, or public commenters and, if appropriate, incorporate it into the final work.

Proposed Eligibility Criteria. All identified citations will be imported into an EndNote database. Table 2 describes our proposed eligibility criteria. Two trained members of the research team will independently review all abstracts for eligibility based on the pre-established inclusion/exclusion criteria. Studies marked for possible inclusion by either reviewer will undergo a full-text review. Any study with inadequate information in the abstract will also undergo full-text review.

We will retrieve and review the full text of all articles included during the title/abstract review phase. Each full-text article will be independently reviewed by two trained members of the research team for inclusion or exclusion on the basis of the eligibility criteria (Table 2). Disagreements about inclusion will be resolved by discussion or consensus with review by the full research team as needed.

All results will be tracked in the EndNote database. We will record the reason that each excluded full-text publication did not satisfy the eligibility criteria so that we can later compile a comprehensive list of such studies in the final work.

2. Data Organization and Presentation

A. Information Management

Information collected to inform quality measures for SMI outcomes includes information gleaned from discussions with Key Informants, comprehensive searches of the peer-reviewed literature, and targeted searches of the gray literature.

Data Abstraction: We will abstract data from each included study, using a standardized template organized to address the Guiding Questions. One member of the research team will collect the data, and a second team member will review it for accuracy and completeness. The following information will be obtained from each study, where applicable: author, year of publication, source of study funding, study design characteristics, study population (including study inclusion and exclusion criteria), interventions employing quality measures, how documentation affects use of quality measures, duration of patient followup, outcomes assessed (specific measures used, as well as timing of assessment), and other pertinent information.

Two researchers will independently review each of the KI interview summaries in preparation for the qualitative analysis. We will use NVivo qualitative software for the analysis.

Integration of Information: Data from the published literature will be integrated with information from the gray literature and KI discussions. We anticipate that GQs 1 and 2 will be informed primarily by information from KI discussions and secondarily by gray literature or nonsystematic published reviews. Parts of these questions may also be informed by published literature or peer-reviewed evidence, particularly the following:

• What organizational components or characteristics of a hospital or other mental health care delivery facility affect selection of QMs?
• What clinician characteristics (e.g., training, specialty, setting) affect selection of QMs?

In instances where evidence from empirical studies informs the response, we will first provide a summary of the empirical evidence, followed by a summary of information from other sources. Responses to GQ 3 will be based primarily on peer-reviewed, published literature and may be combined with information from the gray literature. Responses to Question 4 will be shaped primarily by information from KIs; we will interpret their feedback in light of our responses to GQs 1 through 3.

Conceptual Framework: A conceptual framework will be developed mapping to the PICOTS criteria (i.e., population, interventions/quality measures, comparators, outcomes, timing and setting).

B. Data Presentation

Our findings will be presented in the order of Guiding Questions. We will qualitatively summarize findings from gray literature searches and KI interviews. For questions with empirical evidence or in-progress studies to inform the results, we will build on study-specific tables to generate cross-cutting tables describing the state of evidence on study characteristics (number and types of study designs addressing serious mental illness interventions using quality measures), quality measures, and types of outcomes. We will explore ways to present data graphically based on the availability and appropriateness of the information that we find.

References

1. Epstein J, Barker P, Vorburger M, et al. Serious mental illness and its co-occurrence with substance use disorders, 2002. Substance Abuse and Mental Health Services Administration, Office of Applied Studies (DHHS Publication No. SMA 04-3905, Analytic Series A-24). Rockville, MD: 2004.
2. National Institute of Mental Health. Schizophrenias. Washington, DC: U.S. Department of Health and Human Services www.nimh.nih.gov/statistics/1SCHIZ.shtml. Accessed May 13, 2013.
3. Substance Abuse and Mental Health Services Administration. Results from the 2008 National Survey on Drug Use and Health: National findings. Department of Health and Human Services NSDUH Series H-36, HHS Publication No. SMA 09-4434. Rockville, MD: 2009.
4. Wang PS, Demler O, Kessler RC. Adequacy of treatment for serious mental illness in the United States. Am J Public Health. 2002 Jan;92(1):92-8. PMID: 11772769.
5. Druss BG, Bornemann TH. Improving health and health care for persons with serious mental illness: The window for US federal policy change. JAMA. 2010 May 19;303(19):1972-3. PMID: 20483975.
6. Gierisch JM, Nieuwsma JA, Bradford DW, et al. Interventions to Improve Cardiovascular Risk Factors in People With Serious Mental Illness. Comparative Effectiveness Review No. 105. (Prepared by the Duke Evidence-based Practice Center under Contract No. 290-2007-10066-I.) AHRQ Publication No. 13-EHC063-EF. Rockville, MD: Agency for Healthcare Research and Quality; April 2013.
7. Parks J, Svendsen D, Singer P, et al. Morbidity and Mortality in People with Serious Mental Illness. National Association of State Mental Health Program Directors, Medical Directors Council; 2006. Monograph available at http://www.nasmhpd.org/docs/publications/MDCdocs/ Mortality%20and%20Morbidity%20Final%20Report%208.18.08.pdf. PowerPoint presentation available at http://www.nasmhpd.org/docs/publications/ MDCdocs/Morbidity%20and%20Mortality%20Slides.final-8152008.pdf.
8. Reeves WC, Strine TW, Pratt LA, et al. Mental illness surveillance among adults in the United States. MMWR Surveill Summ. 2011 Sep 2;60 Suppl 3:1-29. PMID: 21881550.
9. Verdier M, Bagchi AD, Esposito D. Prescription Drug Use and Cost Among Medicaid Beneficiaries with Disabilities and Chronic Illnesses. Mathematica Policy Research; 2011.
10. Kessler RC, Heeringa S, Lakoma MD, et al. Individual and societal effects of mental disorders on earnings in the United States: results from the National Comorbidity Survey replication. Am J Psychiatry. 2008 Jun;165(6):703-11. PMID: 18463104.

Definition of Terms

Quality measure (QM): Broadly defined, quality measures are tools that help measure or quantify health care processes, outcomes, patient perceptions, and organizational structure and/or systems that are associated with the ability to provide high-quality health care and/or that relate to one or more quality goals for health care. These goals include effective, safe, efficient, patient-centered, equitable, and timely care. (Source: http://www.cms.gov/Medicare/Quality-Initiatives-Patient-Assessment-Instruments/QualityMeasures/index.html)

Serious Mental Illness (SMI): Based on work done during topic refinement, we are defining SMI as (1) schizophrenia or schizoaffective disorder (or other related primary psychotic disorder), (2) bipolar disorder, or (3) current major depression with psychotic features, per DSM-IV or their ICD-9-CM equivalent [and subsequent revisions]. A diagnosis of SMI requires that individuals experience functional impairment that substantially interferes with or limits one or more major life activities.

Summary of Protocol Amendments

In the event of protocol amendments, the date of each amendment will be accompanied by a description of the change and the rationale.

Key Informants

Within the Technical Brief process, Key Informants serve as a resource to offer insight into the clinical context of the technology/intervention, how it works, how it is currently used or might be used, and which features may be important from a patient of policy standpoint. They may include clinical experts, patients, manufacturers, researchers, payers, or other perspectives, depending on the technology/intervention in question. Differing viewpoints are expected, and all statements are crosschecked against available literature and statements from other Key Informants. Information gained from Key Informant interviews is identified as such in the report. Key Informants do not conduct analysis of any kind nor do they contribute to the writing of the report and have not reviewed the report, except when given the opportunity to do so through the public review mechanism

Key Informants must disclose any financial conflicts of interest greater than $10,000 and any other relevant business or professional conflicts of interest. Because of their unique clinical or content expertise, individuals are invited to serve as Key Informants and those who present with potential conflicts may be retained. The TOO and the EPC work to balance, manage, or mitigate any potential conflicts of interest identified.

Peer Reviewers

Peer reviewers are invited to provide written comments on the draft report based on their clinical, content, or methodologic expertise. Peer review comments on the preliminary draft of the report are considered by the EPC in preparation of the final draft of the report. Peer reviewers do not participate in writing or editing of the final report or other products. The synthesis of the scientific literature presented in the final report does not necessarily represent the views of individual reviewers. The dispositions of the peer review comments are documented and will be published three months after the publication of the Evidence report.

Potential Reviewers must disclose any financial conflicts of interest greater than $10,000 and any other relevant business or professional conflicts of interest. Invited Peer Reviewers may not have any financial conflict of interest greater than $10,000. Peer reviewers who disclose potential business or professional conflicts of interest may submit comments on draft reports through the public comment mechanism.