The National Healthcare Quality and Disparities Report (NHQDR) assesses the performance of the U.S. healthcare system and identifies areas of strengths and weaknesses using measures related to priority areas of healthcare quality that include access to care, affordable care, care coordination, effective treatment, healthy living, patient safety, and person-centered care.¹ The report presents the latest available data on care quality and access to healthcare stratified by diseases and conditions, as well as disparities related to factors such as race and ethnicity or health insurance status. The objective of this technical brief is to support an update of the criteria and process for selecting measures to align with these priority areas as well as focus areas of equity and social determinant of health.

The NHQDR audience include but is not limited to Congress, policymakers, quality improvement professionals in healthcare organizations, researchers, and the public. The NHQDR is based on measures of quality and disparities that cover a broad array of healthcare services and settings. The sources combine different perspectives and insight into the quality of healthcare and current health disparities. Sources include patient surveys, provider surveys, administrative data from facilities, and medical records, as well as data from registries, surveillance systems, and population statistics. Settings of care covered in the report include ambulatory care, health centers, emergency departments, hospitals, nursing homes, hospices, and home health. For each area, numerous measures of quality of care have been proposed and a central question for the NHQDR is which quality indicators should be selected for the report to provide a valid and meaningful snapshot of care quality and disparities.

There are both conceptual and practical measurement issues that will impact any change to the criteria for selecting measures. Extensive research has been conducted regarding operationalizing and measuring equity in recent years.² Various definitions of health equity exist, but most focus on the conditions under which all persons have the opportunities and resources they need to achieve their optimal health.³ There are many important issues to consider, including data availability and quality, the validity and reliability of measurement schemes, the possibility of unforeseen adverse effects on vulnerable populations, and how best to achieve long-term impact and sustainability. There has been extensive research on how social determinants of health impact healthcare quality and contribute to disparities in health outcomes. Health disparities can be defined as gaps in the quality of health or the quality of healthcare that mirror differences in socioeconomic status, racial and ethnic background, geographic location, and education level,⁴ which encompasses various non-medical factors that influence health outcomes, including housing, education, employment, and neighborhood conditions.^5,6 The focus on disparities highlights inherent measurement issues associated with identifying differences in care processes, outcomes, or experiences.

The brief will be facilitated by guiding questions (GQs), documenting research and key informant input:

GQ1. Which prioritization criteria for health care quality measures have been proposed?

• What settings and intended use were the criteria developed for?
• How are the criteria defined and operationalized?
• In what context have these criteria been used?
• How are the criteria similar or different from the current NHQDR criteria?

GQ2. How should the current NHQDR measure selection prioritization criteria be updated?

• What is the operationalized definition of each updated prioritization criteria? 
• What type of health care quality measures would help the NHQDR’s primary audience monitor the effectiveness of health policy levers?

GQ3. How should the new NHDQR measure selection prioritization criteria be applied?

The methods for this technical brief will follow the Methods Guide for Evidence-based Practice Center (EPC) Program. This technical brief aims to answer the guiding questions with information from interviews with Key Informants, grey literature searches, and published literature.

1. Data Collection

A. Discussions With Key Informants

We will hold Key Informant (KI) calls to discuss key aspects of the brief’s topic. We selected nine scopes of interest for which we will identify suitable representatives: disparities measurement, race equity, gender equity, geographic equity, quality of care measurement, patient safety, health insurance (end user), digital health, and health information technology, as well as representatives from across the Department of Health and Human Services (HHS). We will use the KIs as a source of information that is not captured in the scientific literature, but that may be a critical consideration for updating the criteria to select measures. We will seek the help of the KIs in understanding the conceptual complexity of selecting measures of quality of care and disparities in healthcare delivery systems. In addition, we will seek practical support by asking for input to refine the guiding questions and search strategy for relevant literature on the topic. The KI interviews will follow a semi-structured format. KI interviews will be conducted as web conferences, and we will invite key informants to individual interviews as their schedule allows. KI interviews will be documented during each call by a designated member of the project team. Notes will be reviewed and discussed by the investigators to evaluate how the input provides insight on the key questions.

The interviews will be supported by an online survey that informs KIs of our questions in advance and that will also give participants the opportunity to add information which may have been lost during the call. The online survey will allow KIs to have the opportunity to express their individual preferences and feel heard. We will also provide the KIs with this technical brief protocol to enable meaningful input and discussions during the call.

B. Gray Literature Search

We will search the websites of health services organizations, such as CMS (Centers of Medicare and Medicaid Services), HHS (Department of Health and Human Services), HRSA (Health Resources and Services Administration) and VHA (Veterans Health Administration). We will also search ESHPI (AcademyHealth's Evidence-Informed State Health Policy Institute) and NAM (National Academy of Medicine) for criteria to select and prioritize quality of care and disparities measures.

C. Published Literature Search

We will search the following literature sources. We will search the research databases PubMed (biomedical literature), CINAHL (allied nursing), and PsycINFO (psychosocial literature). Preliminary searches showed the challenge of balancing the search yield and not missing relevant studies. There is no established study design describing the process of prioritizing measures, the prioritization process may not be the central aspect of the publication, and the terminology for quality indicators is not standardized and nomenclature varies across clinical fields. The literature searches will use a set of general quality of care indicator terms ("Quality Indicators, Health Care"[Mesh]) combined with search terms for approaches to developing, selecting, applying, comparing, evaluating, or prioritizing measures. Searches will use controlled vocabulary where applicable as well as text words as not to miss newer studies not indexed yet.

We will reference-mine existing reviews where available.⁷ Reviews will also be used to continuously refine the search strategy, in particular to locate grey literature. We will search the same databases used for primary research to identify potentially relevant reviews. In addition, we will search the Cochrane Database of Systematic Reviews and PROSPERO to systematically identify existing research syntheses. The draft search strategy for research databases is documented in the appendix. The search strategy will undergo peer review to ensure relevant documents will be identified. Finally, AHRQ will set up a portal for submissions of Supplemental Evidence And Data for Systematic Reviews (SEADS) and publish a notice on the Federal Register to encourage SEADS submissions.

Table 1 describes the eligibility criteria in a population, concept, context, and other limiters framework.

Table 1. Criteria for Inclusion/Exclusion of Studies in the Review

Literature searches targeted to each guiding question will be designed, executed, and documented by the EPC Medical Librarian. Searches will be conducted without date restriction.

2. Data Organization and Presentation

A. Information Management

Literature screening and data abstraction will be conducted in an online database designed for systematic reviews (DistillerSR). Literature reviewers will screen citations supported by machine learning. All identified citations will be reviewed by at least one human reviewer. All citations that at least one reviewer determined to be potentially relevant to the technical brief will be obtained as full text. All citations not selected for full text review will be screened by a machine learning algorithm monitoring for reviewer errors. Full text studies will be screened by two independent reviewers against the explicit eligibility criteria; disagreements will be resolved by consensus. The literature searches will be updated during the peer and public review process before finalization of the technical brief. Any identified data meeting the eligibility criteria will be incorporated into the final technical brief.

The data abstraction will capture detailed information about the identified process and criteria to prioritize quality of care measures. We will record the identification process for potential measures, any eligibility requirement for measures, any stakeholder involvement in the prioritization process, consensus finding methods, and procedures used to prioritize measures. To better understand how the prioritization criteria were used, we will abstract the nature of the measures (structure, process, outcomes, patient perception), and the broad system or clinical area for which measures were proposed. We will also capture any information pertaining to required feasibility, reliability, and validity of measures (e.g., reviewer agreement on measures, whether the measures had to have been applied empirically, restrictions regarding the procedural effort to obtain measures, evidence of content validity for the measure selection, or the ability to capture care disparities).

B. Data Presentation

The synthesis for the technical brief will include summary tables and figures to clearly convey the identified research. We will account for the literature flow by documenting included studies and excluded studies with reasons for exclusion. An evidence table resulting from the data abstraction will document all identified publications in detail.

A component table documenting the elements of the identified approaches will facilitate comparing and contrasting different approaches to prioritize quality of care measures. We will also tabulate the identified criteria sets for a concise overview. A narrative summary will document common elements across identified approaches. A summary of findings table will document findings across studies to facilitate answers to the guiding questions.

1. Agency for Healthcare Research and Quality. National Healthcare Quality and Disparities Reports, 2023. Accessed on September 6, 2023.
2. Agniel D, Martino SC, Burkhart Q, et al. Incentivizing Excellent Care to At-Risk Groups with a Health Equity Summary Score. J Gen Intern Med. 2021 Jul;36(7):1847-57. doi: 10.1007/s11606-019-05473-x. PMID: 31713030.
3. The White House. Executive Order on Further Advancing Racial Equity and Support for Underserved Communities Through The Federal Government. February 16, 2023. Accessed on September 6, 2023.
4. National Institute of Allergy and Infectious Diseases. Minority Health & Health Disparities. 2022. Accessed on September 6, 2023.
5. Centers for Medicare & Medicaid Services (CMS). Social Determinants of Health at CDC. 2022. Accessed on September 6, 2023.
6. Centers for Medicare & Medicaid Services (CMS). Health Disparities. 2023. Accessed on September 6, 2023.
7. Kanaoka K, Iwanaga Y, Tsujimoto Y, et al. Quality indicators for acute cardiovascular diseases: a scoping review. BMC Health Serv Res. 2022 Jul 5;22(1):862. doi: 10.1186/s12913-022-08239-0. PMID: 35790957.

N/A

Within the Technical Brief process, Key Informants serve as a resource to offer insight into the clinical context of the technology/intervention, how it works, how it is currently used or might be used, and which features may be important from a patient of policy standpoint. They may include clinical experts, patients, manufacturers, researchers, payers, or other perspectives, depending on the technology/intervention in question. Differing viewpoints are expected, and all statements are crosschecked against available literature and statements from other Key Informants. Information gained from Key Informant interviews is identified as such in the report. Key Informants do not do analysis of any kind nor contribute to the writing of the report and will not review the report, except as given the opportunity to do so through the public review mechanism.

Key Informants must disclose any financial conflicts of interest greater than $5,000 and any other relevant business or professional conflicts of interest. Because of their unique clinical or content expertise, individuals are invited to serve as Key Informants and those who present with potential conflicts may be retained. The TOO and the EPC work to balance, manage, or mitigate any potential conflicts of interest identified.

Peer reviewers are invited to provide written comments on the draft report based on their clinical, content, or methodologic expertise.  Peer review comments on the draft report are considered by the EPC in preparation of the final report.  Peer reviewers do not participate in writing or editing of the final report or other products.  The synthesis of the scientific literature presented in the final report does not necessarily represent the views of individual reviewers. The dispositions of the peer review comments are documented and may be published three months after the publication of the Evidence report.

Potential Reviewers must disclose any financial conflicts of interest greater than $5,000 and any other relevant business or professional conflicts of interest.  Invited Peer Reviewers may not have any financial conflict of interest greater than $5,000.  Peer reviewers who disclose potential business or professional conflicts of interest may submit comments on draft reports through the public comment mechanism.

EPC core team members must disclose any financial conflicts of interest greater than $1,000 and any other relevant business or professional conflicts of interest. Related financial conflicts of interest that cumulatively total greater than $1,000 will usually disqualify EPC core team investigators.

This project was funded under Contract No. 75Q80123F32008 from the Agency for Healthcare Research and Quality, U.S. Department of Health and Human Services. The AHRQ Task Order Officer reviewed contract deliverables for adherence to contract requirements and quality. The authors of this report are responsible for its content. Statements in the report should not be construed as endorsement by the Agency for Healthcare Research and Quality or the U.S. Department of Health and Human Services.

PubMed

"Quality Indicators, Health Care"[Mesh] OR "Healthcare Quality Indicator" [Title/Abstract] OR "Healthcare Quality Indicators" [Title/Abstract] OR "quality measure" [Title/Abstract] OR "care indicator" [Title/Abstract] OR "care indicators" [Title/Abstract] OR "care quality" [Title/Abstract] OR "quality of care" [Title/Abstract] OR "Health metric" [Title/Abstract] OR "Health metrics" [Title/Abstract] OR "care metric" [Title/Abstract] OR "care metrics" [Title/Abstract] OR "Benchmarking"[Mesh] OR benchmarked[Title/Abstract] OR benchmarking[Title/Abstract] OR benchmark[Title/Abstract] OR benchmarks[Title/Abstract] AND "measurement criteria"[Title/Abstract] OR "National Healthcare Quality and Disparities Reports" OR NHQDR[Title/Abstract] OR disparities assessment[Title/Abstract] OR disparities measurement[Title/Abstract] OR care equity[Title/Abstract] OR equity measurement[Title/Abstract] OR equity assessment[Title/Abstract] OR consensus [Title/Abstract] OR Delphi [Title/Abstract] OR RAND/UCLA [Title/Abstract] OR ((selecti* [Title] OR priorit* [Title] OR rank*[Title] OR appropriateness [Title/Abstract] [Title/Abstract]) AND (method OR process OR procedure OR approach OR committee* OR panel OR expert*))