Authors and Affiliations
Richard E. Gliklich, M.D.
Michelle B. Leavy, M.P.H.
Jannette Karl, M.B.A., P.M.P.
Daniel M. Campion, M.B.A.
Daniel Levy, M.S.
The investigators are affiliated with the Outcome DEcIDE Center, Cambridge, Massachusetts.
None of the investigators have any affiliations or financial involvement that conflicts with the materials presented in this report.
Because standardized outcome measures do not exist for most condition areas, clinical studies often use different outcome measures or different definitions for the same outcome measures. The use of different definitions can have a substantial impact on study findings and introduce challenges when comparing or aggregating data across studies, leading to uncertainty when interpreting study findings in the context of existing evidence. The primary objectives of this project were to create an Outcome Measures Framework (OMF) to serve as a conceptual model for development of standard outcome measures and to design and pilot test a tool for collecting and displaying information about outcome measures in a system such as the Registry of Patient Registries (RoPR).
Design requirements for the OMF were gathered and refined through in-person meetings and user acceptance testing. Over 110 stakeholders participated from a broad range of backgrounds, including clinicians, registry sponsors, patients, researchers, payers, and regulatory and funding agencies
The proposed design for the OMF meets the requirements identified by stakeholders. The OMF provides a model and tool for the collection and display of information on outcome measures currently being used in patient registries, supports searching for and comparing identified outcome measures, and minimizes user burden. Further, as a content model, the OMF can serve as a standard approach to developing outcome measurement systems in multiple disease areas.
The OMF tool is intended to collect and display information on outcome measures used in patient registries, with the goals of characterizing what registries currently collect and supporting long-term efforts to standardize outcome measures. If the OMF is incorporated into the RoPR or another similar system, the data collected would enable future projects to accurately characterize the current use of outcome measures in registries and to develop informed, feasible approaches to standardization.
A patient registry is defined as “an organized system that uses observational study methods to collect uniform data (clinical and other) to evaluate specified outcomes for a population defined by a particular disease, condition or exposure, and that serves one or more predetermined scientific, clinical, or policy purposes.”1 Common purposes for patient registries include evaluating the safety, effectiveness, or quality of medical treatments, products, and services, and studying the natural history of diseases. Some registries are developed and maintained solely to assist in care delivery, coordination, and quality improvement, but many serve broader research purposes. When properly designed and conducted, patient registries can provide unique insights into real-world clinical practice, effectiveness, safety, and quality.
Interest in and use of patient registries have increased in recent years. Despite this interest, there was no central database designed specifically to list patient registries until recently. ClinicalTrials.gov is a database and public Web site that provides information about research studies, but it is designed primarily for providing information about experimental studies, such as randomized trials. Not all data fields in ClinicalTrials.gov are applicable or relevant to patient registries, and some data that would be useful for describing registries are not collected. A database and searchable public Web site designed specifically to provide information about patient registries would support research collaborations, reduce redundancies, encourage the efficient use of resources, and improve transparency in the use of patient registries.
The primary goal of the Registry of Patient Registries (RoPR) project is to engage stakeholders in the design and development of a RoPR database system that is integrated with ClinicalTrials.gov and meets the following objectives:
- Provides a searchable database of existing patient registries in the United States;
- Facilitates the use of common data fields and definitions in similar health conditions to improve opportunities for sharing, comparing, and linkage;
- Provides a public repository of searchable summary results, including results from registries that have not yet been published in the peer-reviewed literature;
- Offers a search tool to locate existing data that researchers can request for use in new studies;
- Serves as a recruitment tool for researchers and patients interested in participating in patient registries.
The RoPR launched in December 2012 and is accepting registrations of patient registries currently.
As noted above, the second objective of the RoPR system is to facilitate the use of common data elements and outcome measures across registries. A major effort in creating any new registry is the development of data elements and definitions. Currently, few standards exist across registries, and substantial effort is duplicated as each registry develops unique data elements and definitions. For example, multiple registries focusing on cardiovascular disease may collect myocardial infarction as a primary outcome measure, but each may use a slightly different definition of myocardial infarction. Two registries focusing on cancer may each define “significant disability” differently and use unique validated instruments to measure this outcome measure. Not only does this represent an inefficient use of resources, but the variations in data elements and definitions make it challenging to link and compare data across registries. Standardization of data elements and outcome measures, including both common definitions and common syntax, would result in reduced effort in developing a registry and increased opportunities to link and compare data across registries.
As registries collect a broad range of data for multiple purposes, much work is needed to standardize all data elements for registries. Standardizing patient- and population-level outcome measures, which represent a subset of registry data elements, would provide substantial benefits by allowing results from individual registries to be compared and aggregated. The outcome measures collected in patient registries vary widely depending on the purpose of the study and may include events (e.g., death, hospitalization), laboratory test results, or patient-reported outcomes. Within this document, the term “patient outcome measure” refers to an individual patient-level outcome (e.g., HbA1C > 9.0%), while the term “population outcome measure” refers to a population-level outcome (e.g., proportion of patients with HbA1C > 9.0%). Some examples of patient and population outcome measures that may be used within a patient registry are listed in Table 1.
|Response to rituximab treatment||Reduction of ≥3 in score on the Safety of Estrogens in Lupus Erythematosus: National Assessment (SELENA) version of the Systemic Lupus Erythematosus Disease Activity Index (SLEDAI)2||Patient|
|Significant disability||A score of ≤40 on the Short-Form 12 (SF-12)3||Patient|
|Acute myocardial infarction inpatient mortality||Proportion of acute myocardial infarction (AMI) patients who expired during hospital stay4||Population|
|HbA1c in poor control||The percentage of patients 18-75 years of age with diabetes (type 1 or type 2) who had HbA1c in poor control (>9.0%)5||Population|
An important first step towards standardization is the collation of patient and population outcome measures already used by registries. Characterizing the range of outcome measures currently used within registries will allow for the identification of commonly used items that may become standards for new registries. In areas where there are no commonly used outcome measures, characterization will provide the foundation for informed discussions about how to build consensus and move towards standardization.
Purpose and Objectives of the OMF
The purpose of this project is to design and develop a prototype of an Outcome Measures Framework (OMF). The OMF is a mechanism for the collection and display of information on existing outcome measures used in patient registries in a standardized way that supports searching for those outcome measures. Users of the OMF will fall into two major types: those with a registry who are providing information on the data they collect in their registry, and those who are searching for information about how a particular type of outcome measure is collected within patient registries. The first group of users--registry holders--must be able to enter information into the system easily and efficiently. The second group of users--registry seekers--must be able to find sufficient information on outcome measures to identify items for use in their own registry. Meeting the needs of both sets of users is an important consideration in the design of the OMF.
The initial objective of the OMF is to collect sufficient information to characterize the types of outcome measures that are currently used in patient registries. The long-term objective of the OMF is to support efforts to standardize outcome measures and to facilitate access to that information.
This document describes the proposed design of the OMF and discusses how the OMF could be incorporated into a system such as the RoPR.
Approach to Designing the OMF
Existing systems that collect and present information on data elements, outcome measures, and/or quality measures were identified and reviewed, including the Common Data Elements initiative at the National Institute of Neurological Disorders and Stroke,6 the United States Health Information Knowledgebase,7 the Consensus Measures for Phenotypes and eXposures (PhenX) project, the National Quality Measures Clearinghouse (NQMC), and the National Quality Forum (NQF). Of the reviewed systems, the three systems that are most relevant for this project are PhenX, NQMC, and NQF. The most relevant of these examples is the PhenX project. PhenX aims to provide standard measures related to complex diseases, phenotypic traits, and environmental exposures, with the goals of facilitating the linkage of data from multiple studies and helping to integrate genetics and epidemiological research.8 Some of the PhenX measures are outcomes that would be suitable for inclusion in the OMF, and the system provides an excellent example of the Toolkit approach, in which a user can browse or search for relevant measures and create a selection of measures to place in a bucket for later use.
Both the NQMC and the NQF provide information on quality measures, rather than outcome measures. The NQMC is a database of evidence-based health care quality measures and measure sets.9 Sponsored by the Agency for Healthcare Research and Quality (AHRQ), the system aims to provide detailed information on quality measures to a broad set of stakeholders to support dissemination, implementation, and use of these measures. While the system focuses on quality measures rather than outcome measures, it contains many similar features to the proposed OMF, such as search and compare tools and detailed measure summaries. In particular, the NQMC provides relevant examples of archived entries, topics for browsing, and a measure compare tool. The NQF is a nonprofit organization focused on improving the quality of health care by building consensus on national quality improvement priorities, endorsing national consensus standards for quality measures, and publically reporting on performance.10 The organization manages a directory of NQF-endorsed standards, which provides relevant examples of displaying search results and browsing options. All three systems provided useful examples of how to display individual measures and what information should be included for each measure.
In addition to the review of existing systems, illustrative examples of outcome measures that are currently used by patient registries were assembled. The examples cover multiple disease areas (e.g., diabetes, myocardial infarction, heart failure) and outcome types (e.g., laboratory tests, events, patient-reported outcomes). Forty-three examples were assembled, and 10 illustrative examples are presented in Appendix A. The examples were used to explore the complexities around displaying and searching for measures within the OMF.
Five stakeholder meetings were held in the first quarter of 2011 to collect information on how outcome measures are collected currently in existing patient registries and to learn about how stakeholders would like to see information on outcome measures presented within the RoPR or another similar system. Registry sponsors, clinicians, and clinical researchers were invited to participate in the meetings, which were organized around the following AHRQ priority condition areas:
- Cardiovascular disease, including stroke and hypertension
- Diabetes mellitus
- Infectious diseases including HIV/AIDS
- Peptic ulcer disease and dyspepsia
- Pulmonary disease/asthma
- Arthritis and nontraumatic joint disorders
- Depression and other mental health disorders
- Developmental delays, attention-deficit hyperactivity disorder, and autism
- Functional limitations and disability
A final meeting was held in April 2011 to discuss the preliminary design of the OMF. Stakeholders at this final meeting represented health care provider organizations, professional societies, academia, research and consulting organizations, government agencies, patient/consumer organizations, journal editors, payers, and pharmaceutical companies. In total, 117 stakeholders participated in the series of meetings, as depicted in Figure 1.
User Acceptance Testing
Following the stakeholder meetings, the project team developed a proposed design for the OMF and presented this design to stakeholders through a series of user acceptance testing (UAT) activities. The UAT activities focused on several areas of complexity that arose during the stakeholder meetings, such as clinical equivalency, collection of timeframe, and minimizing user burden. These issues are discussed in detail below. Between July and November 2012, three rounds of UAT were conducted; this iterative approach allowed the project team to refine the proposed OMF design multiple times. UAT activities included participation in Web conferences, completion of Web-based structured questionnaires, and review of this document. Stakeholders participating in UAT represented health care provider organizations, professional societies, academia, research and consulting organizations, government agencies, patient/consumer organizations, payers, and industry. In total, 61 stakeholders participated in these UAT activities, as depicted in Figure 2.
Incorporation of Stakeholder Feedback
Through the meetings and UAT, stakeholders provided valuable input regarding what outcome measures they currently collect in their registries, how they would like to search for outcome measures within the RoPR, what type of information they would be willing to provide on outcome measures when entering a registry into the RoPR, and what type of information they would like to find in the OMF. In particular, the stakeholder discussions and feedback shaped the plans described here for displaying information within the OMF, collecting and sharing information on the data elements that comprise an outcome measure, and entering and updating content within the OMF.
Design and Implementation Considerations
The background research and stakeholder activities identified several major issues that must be taken into account in the OMF design and related implementation and maintenance plans. These issues are discussed below.
Outcome measures can vary significantly in type, complexity, and definition. These variations introduce several challenges that must be considered in the design of the OMF. In particular, the issues of variation in type of information to be displayed, variation in definitions for the same concept, and variation in timeframe must be considered.
Variation in Type of Information To Be Displayed
As noted above, outcome measures encompass multiple types of data, including laboratory tests, patient-reported outcomes, events, and derived (i.e., calculated) measures. Depending on the type of data, different information must be presented within the OMF. For example, at the most basic level, a patient outcome measure collected in a registry may be a laboratory test such as “HbA1c control (<8.0%).”11 To provide useful information for users searching for a patient outcome measure, the OMF would need to collect and display a small amount of information on this measure, such as the title and the source. A more complex example is a patient-reported outcome, such as “Significant disability,” which can be defined as a score of ≤40 on the Short-Form 12 (SF-12).33 To display this example, the OMF would need to include, at minimum, the title, the definition, and the source. Events that are collected as outcome measures, such as myocardial infarction, are similar, in that the title, definition, and source must be displayed. Some events may also require the use of subterms to provide further clarity to users searching for an outcome measure.
The most complex example in terms of the information displayed is a derived measure. Derived measures are calculated from data collected within the registry. An example of a derived measure is “Acute myocardial infarction inpatient mortality,” which can be defined as the percentage of acute myocardial infarction (AMI) patients who expired during hospital stay.4 In addition to the title, definition, and source, the information displayed for this example must include details on how the measure is calculated, such as the inclusion and exclusion criteria for the measure and the specification of the numerator and denominator. A derived measure is generally based on a patient outcome measure—AMI in this example. The patient outcome measure should be defined in the derived measure definition; in some cases, the definition is included as part of the inclusion criteria or in the specification of the numerator. The OMF could provide the ability to define the patient outcome measure as one entry and link that entry to the derived measure that is based on the patient outcome measure.
Variation in Definitions for the Same Concept
A second area of complexity is the lack of consistent definitions for the same concept. As noted above, many registries collect similar data in different ways. While a major goal of the OMF is to standardize definitions for the same concept across registries, characterization of the outcome measures currently used by registries is an important first step. To accomplish this first step, however, the OMF must display information on different definitions for the same concept in a logical, understandable way. Definitions vary in three major ways: (1) different wording for the same concept; (2) different definitions for the same concept; and (3) varying levels of detail such that it is not possible to determine if the definitions are equivalent.
In the first scenario, the definitions may be worded differently, but they refer to the same concept. These are considered clinically equivalent definitions. For example, an important outcome measure in diabetes care is management of HbA1c. Two registries may both collect information on this measure, but may use different wording for the concept, as seen in Table 2 below. In the table, Example 1 and Example 2 are considered clinically equivalent definitions. Both definitions may be entered into the OMF as valid examples of how HbA1c Management is collected as an outcome measure within patient registries. However, users of the OMF may find it confusing to review multiple definitions of HbA1c Management that are clinically equivalent. Therefore, the OMF should identify these clinically equivalent definitions in such a way that the user knows that both definitions exist in the OMF and that these definitions are equivalent.
|1||HbA1c Management: Poor Control5||The percentage of patients 18-75 years of age with diabetes (type 1 or type 2) who had HbA1c in poor control (>9.0%).|
|2||Diabetes: HbA1c Poor Control12||The percentage of patients 18-75 years of age with diabetes (type 1 or type 2) who had HbA1c >9.0%.|
In the second scenario, outcome measures may have the same title and refer to the same concept, but the definitions are substantively different. In Table 3, Example 2 and Example 3 are both examples of how myocardial infarction may be collected as an outcome measure within a patient registry. However, the definitions are not clinically equivalent. Example 2 encompasses acute and established myocardial infarctions, while Example 3 refers only to acute events. Within the OMF, these need to be presented as distinct entries, so that the user can understand that both of these examples may be used to collect myocardial infarction in a patient registry, but they are not clinically equivalent.
|1||Acute myocardial infarction13||Includes: myocardial infarction specified as acute or with a stated duration of 4 weeks (28 days) or less from onset. Excludes: (1) certain current complications following acute myocardial infarction; (2) (I23.-) myocardial infarction: old (I25.2), specified as chronic or with a stated duration of more than 4 weeks (more than 28 days) from onset (I25.8); subsequent (I22.-); and (3) postmyocardial infarction syndrome (I24.1).|
Criteria for acute, evolving or recent Myocardial Infarction (MI). Either one of the following criteria satisfies the diagnosis for an acute, evolving or recent MI:
Criteria for Acute Myocardial Infarction. The term myocardial infarction should be used when there is evidence of myocardial necrosis in a clinical setting consistent with myocardial ischaemia. Under these conditions any one of the following criteria meets the diagnosis for myocardial infarction:
Lastly, in the third scenario, outcome measure definitions referring to the same concept may vary in the level of detail provided. In Table 3, Example 1 provides less detail than Examples 2 and 3. This makes it more difficult to assess clinical equivalency, since the definitions must be equivalent in both directions. In other words, if one item is a broad definition and a second item describes a subset of that broad definition, the definitions are not equivalent. This difference is displayed graphically in Figure 3.
An additional question related to this complexity is how clinical equivalency will be determined. This issue is discussed further in the Maintenance section below.
Variation in Timeframe
In addition to variations in definition, outcome measures may vary in the timeframe of interest. For example, the readmission rate for heart failure patients may be an outcome measure of interest for two registries. Both registries may use the same definition of heart failure, but one registry may collect readmissions within 45 days of the initial hospitalization, while the other registry collects readmissions within 30 days of the initial hospitalization. This type of variation in timeframe is common with outcome measures, which are often collected for a specified follow-up period or measured over time (e.g., change in blood pressure from baseline to 60 days). Through discussions with stakeholders and clinical experts, it was determined that outcome measures that are identical with the exception of the timing of their collection or calculation are clinically equivalent. However, these discussions also revealed that there is interest in understanding what timeframe registries are using when collecting a specific outcome measure. Therefore, the OMF needs to collect information on the outcome measure that is used by the registry and the specific timeframe of collection.
During the stakeholder meetings, participants provided examples of patient and population outcome measures that they currently use in their registries. The examples exhibited the wide variations in definition discussed above. Due to these variations, many systems that collect this type of information do so using “free text” data entry. Users enter into the system the information they feel is relevant for their outcome measures. As a result, entries have varying levels of detail, and it is difficult for users to understand whether registries are collecting the same outcome measures. These types of entries may contain other variations, such as inconsistent use of acronyms or a lack of clarity around their intended meaning. Stakeholders discussed these issues in the series of meetings and indicated that it would be preferable to select from an existing list of outcome measures that contained sufficient detail for them to make relevant selections. Therefore, the OMF is envisioned as a curated library, rather than a library of non-administered user-entered information. The OMF will need to use a controlled process for collecting new data and reviewing the data for completeness and clinical equivalency before updating the OMF. This process is discussed further in the Maintenance section below.
OMF Inclusion and Exclusion Criteria
Because the OMF will use a controlled process for collecting and entering new content, clear inclusion criteria for the OMF are needed. Submitted patient and population outcome measures should be reviewed against the inclusion and exclusion criteria and for completeness before being added to the dataset. As currently envisioned, the OMF will include any outcome measure that is or was used by a patient registry and is submitted with all required information. The OMF focuses specifically on outcome measures, rather than all data collected by a registry. Data elements that do not represent outcome measures will be excluded, as will measures that focus on quality of care rather than patient outcomes. To be considered complete, an outcome measure must include the following data: Source, Source version/date, Title, Subterms (if applicable), Definition, Reference, and Keywords. In addition to these seven items, a complete outcome measure must include the Denominator, Numerator, and Exclusion Criteria. Outcome measures that are published in multiple places (e.g., the NQMC) may be listed in the OMF. In these cases, the original source of the measure should be cited.
Minimizing Burden for Registry Holders
A major consideration of the design for the OMF has been actual usage of the system by registry holders and those seeking to start a new registry. The OMF needs to be scalable so that it may eventually include outcome measures from more disease/condition areas and from a broad group of registries. It must also ensure that the additional content does not overwhelm the users or make the burden of selecting relevant outcome measures too high for actual use. The realities of usage must be balanced with a purely theoretically constructed system. In particular, the stakeholder discussions identified an issue with collecting and presenting all relevant data elements for an outcome measure. Outcome measures can be broken down into separate data elements, each of which could be defined and described within the OMF. For example, the outcome measure “30 day readmission for heart failure” could be broken down into individual data elements: date of hospitalization, date of readmission, and heart failure. However, the burden of selection of this level of detail becomes overwhelming for the user, and, based on stakeholder feedback, this level of detail is not included in the proposed OMF. Options to filter or browse the OMF content would help to reduce the burden of selecting relevant items from the OMF. Filters could limit the content displayed by disease/condition area (e.g., cardiovascular disease, oncology) or by type of outcome measure (e.g., laboratory result, patient-reported outcome measure). These features would provide flexibility for users to search for content in different ways.
Maintenance and Governance of the OMF
As a curated system, the OMF will require dedicated maintenance resources to review and add new entries to the OMF and update or archive existing entries as needed. When reviewing new entries, maintenance personnel will need to consider the relevance of the entry for the OMF (e.g., Is it a patient or population outcome measure? Is it used by a patient registry?), the completeness of the entry (Are all relevant data included?), and the equivalency of the entry to other entries in the OMF. The last step is the most complex and will require personnel with clinical expertise to compare the submitted entry to data already existing in the OMF. This step may be time consuming and will require highly knowledgeable resources. If the entry is determined to meet the inclusion requirements and is unique (i.e., not clinically equivalent to any other outcome measures in the OMF, as defined in the section, Variation in Definitions for the Same Concept), the OMF staff will add it to the OMF system. Incomplete entries will need to be returned to the submitter for further information. Clinically equivalent entries will need to be appended to the equivalent definition and designated as equivalent, as shown in the section, Proposed OMF Design. In addition to adding new content, maintenance staff for the OMF must manage existing content. For example, a widely used definition of an outcome measure may be revised. In this scenario, since users may have selected the existing definition and the new definition may not be clinically equivalent, the OMF staff would need to treat the revised definition as a new entry and compare it to the existing definition. The definitions may be distinct, in which case both may remain in the OMF.
An additional maintenance consideration relates to the long-term supportability of the framework. The framework is intended to be used in other systems, such as the RoPR. The RoPR may subscribe to the OMF, in which case it receives all updates. An update schedule and versioning plan will need to be developed to support clear communication with the RoPR or other systems implementing the OMF. This will provide transparency for any new outcome measures added to the OMF and any revisions to existing outcome measures included in prior OMF versions.
The OMF should be governed by stakeholders with clinical expertise, experience in registry design and conduct, and information technology system design. The governing body for the OMF will be responsible for ensuring that the content of the OMF remains relevant and useful to registry holders and registry seekers and maintaining the balance between the need for complete information and the burden on users. The governing body will also be responsible for promoting the objectives of the OMF and disseminating information about its purpose and use to encourage submission of outcome measures; the issues related to submission are discussed further in the Strengths and Limitations section below. Policies and procedures should be developed for determining clinical equivalency, identifying the dominant entry when clinically equivalent entries exist, displaying equivalent outcome measures, and updating and archiving content. The governing body of the OMF will manage the release of updates to the OMF and should give priority to users’ interests when considering changes or revisions to the OMF. The governing body of the OMF may also benefit from consultation with an advisory committee that includes representatives of the stakeholder groups who participated in the design of the OMF (e.g., researchers, regulators, payers, and patient/consumer advocates).
Proposed OMF Design
The proposed design for the OMF must meet the basic requirements of providing a tool for the collection and display of information on outcome measures used in patient registries. The OMF must support searching for and comparing identified outcome measures. In addition, the OMF must take into account the major issues noted above, such as variation in content, clinically equivalent outcome measures, and burden of use. Easing the burden of use for registry holders is a major priority of the OMF. While it is possible to create a theoretically sound framework to catalogue outcome measures, if the burden of use is too high the framework will not be used in a third-party system such as the RoPR, where participation is voluntary.
The proposed OMF provides a construct for storing and displaying multiple types of patient and population outcome measures. All items in the OMF include a title, source, and definition. Data that are categorized as patient outcome measures are also able to include subterms and references. Data that are categorized as population outcome measures are able to display information on numerators, denominators, and inclusion/exclusion criteria, as well as subterms and references. Users of the OMF are able to search for an outcome measure using various search tools. Search results are displayed in a summary view, and users then have the option to select a single outcome measure for a more in-depth review or select multiple entries to compare. The in-depth view displays all information on the outcome measure contained in the OMF.
The following mock-ups display the potential implementation of the OMF within a system such as the RoPR. They display sample content that a user might see when listing a registry in the RoPR and choosing what outcome measures are collected within that registry. The ‘Add’ buttons throughout the mockups would be used by RoPR registrants to select the outcome measures that they collect within their registries. The workflow presented in these mock-ups is as follows: (1) the user searches for outcome measures related to myocardial infarction; (2) the search results are displayed, and the user selects five entries to compare (Figure 4); (3) the user reviews the selected entries on the Compare page (Figure 5); (4) the user views the complete record for one patient outcome measure and one population outcome measure (Figures 6 and 7); and (5) the user adds four items to their registry and indicates the timeframe of interest for each item (Figure 8).
As shown in Figure 4,“myocardial infarction” may be included in the title of an outcome measure within the OMF, or it may be a phrase related to the content comprising a complete OMF entry. The summary view displayed here is an example of the shortened content set that is accessible immediately after searching. From here, a user could choose to compare selections to see more information.
In Figure 5, the user has selected several entries to compare. The Compare page displays information on each selected measure. While not shown in the mock-up below, the Numerator, Exclusion Criteria, Keyword, and Reference fields are also visible on the Compare page.
The display of outcome measures from multiple sources that have clinically equivalent definitions is a challenge in the Compare view. Displaying separate entries for all clinically equivalent definitions would create visual clutter and may be confusing to users. In Figure 5, the first two entries have clinically equivalent definitions, and text is used to indicate the number of additional clinically equivalent measures. The measure displayed here is the dominant measure, which appears in search results and when browsing. The clinically equivalent measures are displayed on the full record for the outcome measure (see Figure 6 below). As noted in the Governance section, the procedures for identifying the dominant entry when clinically equivalent entries exist should be clearly defined and transparent. From the Compare view, the user can select individual outcome measures to review in detail. Clicking on an individual entry brings the user to the complete record of the OMF entry. The complete record displays all information in the OMF for the outcome measure. Figure 6 shows a complete record for a patient outcome measure entry, while Figure 7 shows a complete record for a population outcome measure entry.
The complete record view also shows full details on the clinically equivalent entries. There is no limit to the number of clinically equivalent entries that could be listed for a given entry. Stakeholders noted that separate listings for all clinically equivalent entries would clutter the initial search results unnecessarily and increase the burden on users to sift through equivalent content before identifying what outcome measures are collected within their registry. The designation of clinical equivalency will be determined when content is added to the OMF, as discussed above. Figure 6 provides an example of how clinically equivalent entries would appear in the OMF. While not shown in Figure 7, clinically equivalent entries would appear in the same manner for population outcome measures.
As noted above, once users identify the relevant entry for their registry, they could select the “Add” button to add the entry to their registry profile within the RoPR system. Users who are searching for entries to facilitate building a new registry could find sufficient information on the complete record page to determine if the outcome measure is relevant for their registry and use the references to find more information, if needed, to incorporate the measure into their registry.
As discussed in the previous section, some outcome measures require the entry of a timeframe of collection or calculation. In Figure 8, the user has added four outcome measures to their registry profile in RoPR. The user now has the ability to specify the timeframe of collection or calculation for each measure. Stakeholders preferred an approach to entering timeframe that allowed for maximum flexibility. As shown in Figure 8, users can specify the timeframe by using a combination of a number and a unit of time (e.g., 1 month, 2 years, 3 days). If the available options do not sufficiently describe the timeframe used in their registry, users can enter free-text information.
Strengths and Limitations of the OMF Design
The proposed design of the OMF presented here has several strengths. First, the proposed design is flexible and scalable. These are important attributes, as the OMF will need to collect and display a large number of heterogeneous outcome measures. Flexibility is particularly critical to accommodate the variations in types of data and level of detail seen in the examples in Appendix A. The flexible design also allows for the presentation of clinically equivalent entries in a consistent way.
Second, the proposed design simplifies searching for entries by including comprehensive lists of keywords for each entry. Users who enter a keyword into the search field will see all of the entries with that keyword in their search results, which facilitates searching for measures using synonyms. For example, a search for “heart attack” will return entries for acute myocardial infarction, as “heart attack” is a keyword for these entries. The keyword function also supports searching for clinically equivalent measures. As an example, the PhenX definition of myocardial infarction is clinically equivalent to the definition of the Joint European Society of Cardiology/American College of Cardiology Committee. The Joint European Society definition is the dominant definition that appears in search results, while the PhenX definition appears as a clinically equivalent definition. However, users who search for “Consensus Measures for Phenotypes and eXposures” or for “PhenX” will find the Joint European Society definition and the clinically equivalent PhenX definition because these terms are included as keywords.
Finally, an important strength of the proposed design is its ease of use. A registry holder who is entering a registry into the RoPR system can search for and select the outcome measures used in the registry with relatively few steps. The level of complexity in the proposed workflow aligns with what stakeholders suggested would be feasible within a voluntary system, such as the RoPR.
The proposed design does have some limitations. First, the individual data elements that comprise an outcome measure are not collected and displayed in the OMF. For example, the outcome measure “30 day readmission for heart failure” could be broken down into date of hospitalization, date of readmission, and heart failure. Each of these individual data elements could be defined within the OMF, and registry holders selecting this outcome measure could specify how they collect each component of the measure (e.g., How is date of hospitalization defined and collected? How is date of readmission defined and collected? How is heart failure defined?). This level of detail was not included in the proposed OMF design due to stakeholder concerns that the burden of entry would be too high for a voluntary system. However, it is possible that two registries that appear to collect the same measure based on the OMF data may define the measure differently. In the “30 day readmission for heart failure” example, the registries may define heart failure differently. Alternately, they may define readmission differently, with one registry only counting patients who are readmitted for heart failure, and another registry counting all readmissions for any reason. These variations may not be apparent based on the level of detail collected and displayed in the OMF. The issue of including the individual data elements may need to be reassessed after a pilot period of data collection through the OMF. The pilot period would provide both information on burden of entry for registry holders and usefulness of the data for those searching for outcome measures.
A second limitation relates to the use of a curated system. The variations in definitions and the discussions with stakeholders led to the decision to propose a curated system to ensure that content is entered consistently and that clinically equivalent items are identified as such. While this approach has advantages in terms of the quality of the content, there are also disadvantages related to the timeliness of updates to the system and the required amount of resources. Submitted entries for the OMF will need to be reviewed against the inclusion criteria and for completeness, as well as for clinical equivalency. As a result, the process of adding an entry to the OMF may take some time. Users who do not find their outcome measure within the existing OMF content set and who submit it for consideration would need to go back into the RoPR system to select their entry once it is available. Some users may not take the time to do this or may find this extra step too burdensome. While reviewing the submitted entries against the inclusion criteria and for completeness will be relatively straightforward, determining clinical equivalency requires time and expertise. Both levels of review—and particularly clinical equivalency—will increase the costs of maintaining and updating the OMF. In addition, outcome measures contained within the OMF may be revised (e.g., as new definitions are developed or when components of the measure change, such as the transition from ICD-9 to ICD-10 coding systems), and resources will be needed to ensure that information presented in the OMF remains accurate and current.
Next Steps for the OMF
The goal of this project is to design and develop a prototype of the OMF. While designing the OMF, some issues were encountered that were beyond the scope of this project. In particular, two issues that must be addressed prior to a full-scale implementation of the OMF are the process for soliciting outcome measure submissions and the process for determining clinically equivalent entries and selecting a dominant entry. Once the OMF design has been finalized following public comment, the OMF will be populated with a starter set of content, comprised of outcome measures that are publically available and may be used within patient registries. However, additional content will need to be solicited from registry holders. The OMF will need a plan for disseminating information about the project and encouraging registry holders to submit information on their outcome measures. Strategies for incentivizing registry holders to share this information will be needed. The OMF will also need an interface to accept and review these submissions. Second, once the OMF receives submitted measures, the governing body will need to develop a clear and transparent approach to determining if entries are clinically equivalent and selecting the dominant measure.
Additional issues that may need to be considered include measures that are not publically available and the expansion of the initial content set beyond the priority conditions. Some registries consider their outcome measures proprietary and only share the measure definitions with participants. The OMF may need to consider ways to encourage these registries to share information on their measures publically. In addition, the OMF will eventually need to expand to include disease areas outside the priority condition areas noted above. A re-evaluation of the data collected and displayed within the OMF should be completed after a pilot period of full-scale use. The inclusion of a broader range of entries may identify the need for additional data elements. As information from the OMF becomes available in the RoPR system, registry seekers (users who search for registries within the RoPR) may begin searching for registries based on the specific outcome measures that the registry collects. For example, a researcher who is seeking a registry to combine with an existing data source may only be interested in registries that collect myocardial infarction using a specific definition. These new users of the OMF content may also identify data elements that should be added or entered by registry holders.
Finally, plans to implement the OMF should include consideration of collaboration opportunities. Several external initiatives with similar or related objectives were mentioned by stakeholders. In order to prevent duplication of effort, it will be important for the OMF to collaborate with these other initiatives where possible. For example, the International Consortium for Health Outcomes Measurement (ICHOM) recently launched a database of registries and outcome measures in 16 priority condition areas. ICHOM, which was founded by the Boston Consulting Group, Michael Porter’s Institute for Strategy and Competitiveness at Harvard Business School, and the Karolinska Institutet, aims to “provide a global resource of in-use outcome measures and risk-adjustment factors by medical condition” and “advance international standardization around the best outcome measures by condition.”16 While the ICHOM is similar in overall concept to the OMF, other projects, such as the Cancer Data Standards Registry and Repository17 and the CSHARE project,18 may provide useful opportunities to collaborate on specific components of the OMF. Additional collaboration opportunities could be identified through the Common Data Element Resource Portal hosted by the National Institutes of Health.19 A related question is whether the OMF should collaborate with initiatives that do not directly address patient registries. For example, some stakeholders suggested that the OMF should work with initiatives addressing outcome measures collected in clinical trials or electronic health records. While the first priority of the OMF will be to work with registry-focused initiatives, the OMF governing body may find it useful to consider collaboration with other types of initiatives once the OMF has been implemented.
In summary, the OMF is intended to collect and display information on outcome measures used in patient registries, with the goals of characterizing what registries currently collect and supporting long-term efforts to standardize outcome measures. The OMF design proposed here provides a flexible, scalable approach to collecting and displaying this information in a way that minimizes user burden and is suitable for inclusion in a system such as the RoPR. In addition, the proposed design addresses many of the complexities around collecting and displaying outcome measures identified through discussions with stakeholders and noted here. While this design has many strengths, it may be improved through pilot testing prior to full-scale implementation.
Within the RoPR, the OMF will provide a tool for collecting and displaying information on outcome measures in a standard format. The OMF will need to be populated with a starter set of content and then expanded as contributors submit additional outcome measures for inclusion. Registry holders who enter a registry in the RoPR can then indicate which outcome measures they collect. The collection of this information in a standard format will support efforts to describe the fragmented landscape of outcome measures used within patient registries. The data collected through the OMF can then be used to identify disease areas where some consistent measures are already being used in similar registries. These areas may be targeted for initial efforts to move towards standardization of outcome measures. Other disease areas that require more substantial, long-term efforts to develop and promote the use of standard outcome measures may also be identified through these data. Overall, the data collected through the OMF will enable future projects to accurately characterize the current use of outcome measures within registries and to develop informed, feasible approaches to standardization.
- Gliklich RE, Dreyer NA, eds. Registries for Evaluating Patient Outcomes: A User's Guide. (Prepared by Outcome DEcIDE Center [Outcome Sciences, Inc. dba Outcome] under Contract No. HHSA29020050035I TO1.) AHRQ Publication No. 07-EHC001-1. Rockville, MD: Agency for Healthcare Research and Quality. April 2007. p. 9.
- Terrier B, Amoura Z, Ravaud P, et al. Safety and efficacy of rituximab in systemic lupus erythematosus: results from 136 patients from the French AutoImmunity and Rituximab registry. Arthritis Rheum. 2010 Aug;62(8):2458-66. PMID: 20506527.
- Fourquet J, Baez L, Figueroa M, et al. Quantification of the impact of endometriosis symptoms on health-related quality of life and work productivity. Fertil Steril. 2011 Jul;96(1):107-12. PMID: 21621771.
- The Joint Commission. Specifications Manual for National Hospital Inpatient Quality Measures. Section 2k: NQF-Endorsed Voluntary Consensus Standards for Hospital Care: Measures Information Form. Version 3.3. www.jointcommission.org/assets/1/6/Specs_Manual_3.3b_PDF_Zip.zip. Accessed February 24, 2012.
- National Committee for Quality Assurance. NQF-Endorsed National Voluntary Consensus Standards for Physician-Focused Ambulatory Care. Appendix A - NCQS Measure Technical Specifications. April 2008. Version 0.7. www.ncqa.org/LinkClick.aspx?fileticket=POLoMIAi3Mo%3d&tabid=59&mid=1604&forcedownload=true. Accessed February 29, 2012.
- National Institute of Neurological Disorders and Stroke. Common Data Elements Initiative. www.commondataelements.ninds.nih.gov. Accessed March 29, 2012.
- Agency for Healthcare Research and Quality. United States Health Information Knowledgebase. http://ushik.ahrq.gov/. Accessed March 29, 2012.
- RTI International. PhenX Toolkit. PhenX Project Background. https://www.phenxtoolkit.org/index.php. Accessed March 19, 2012.
- U.S. Department of Health and Human Services. National Quality Measures Clearinghouse. About. www.qualitymeasures.ahrq.gov/about/index.aspx. Accessed March 19, 2012.
- National Quality Forum. About NQF. www.qualityforum.org/About_NQF/About_NQF.aspx. Accessed March 19, 2012.
- National Committee for Quality Assurance (NCQA). HEDIS® 2011: Healthcare Effectiveness Data and Information Set. Vol. 1, narrative. Washington (DC): National Committee for Quality Assurance (NCQA); 2010. various pp.
- MA Patient Centered Medical Home Initiative. PCMH Mandatory Measures. http://hcfgkc.org/sites/default/files/documents/5_Houy_Performance_Measurement_Final.pdf. Accessed March 28, 2012.
- National Institutes of Health. National Cancer Institute. NCIthesaurus: Myocardial Infarction (Code C27996). http://ncit60.nci.nih.gov/ncitbrowser/pages/concept_details.jsf?dictionary=NCI Thesaurus&code=C27996. Accessed February 24, 2012.
- World Health Organization. International Statistical Classification of Diseases and Related Health Problems, 10th Revision. Acute Myocardial Infarction. http://apps.who.int/classifications/icd10/browse/2010/en#/I21. Accessed February 24, 2012.
- Thygesen K, Alpert JS, White HD. Universal definition of myocardial infarction. J Am Coll Cardiol. 2007 Nov 27;50(22):2173-95
- International Consortium for Health Outcomes Measurement. http://ichom.org. Accessed December 18, 2012.
- National Cancer Institute. Cancer Data Standards Registry and Repository (caDSR). http://ncicb.nci.nih.gov/revised-content-infrastructure/cacore_overview/cadsr. Accessed December 18, 2012.
- Jiang G, Solbrig HR, Iberson-Hurst D, et al. A Collaborative Framework for Representation and Harmonization of Clinical Study Data Elements Using Semantic MediaWiki. https://wiki.nci.nih.
gov/download/attachments/63242249/CSHARE_AMIA_CRI_2010_By_GuoqianJiang.pdf?version=1&modificationDate=1323299030000. Accessed December 18, 2012.
- National Institutes of Health. U.S. National Library of Medicine. NIH Common Data Element (CDE) Resource Portal. www.nlm.nih.gov/cde. Accessed January 24, 2013
Gliklich RE, Leavy MB, Karl J, Campion DM, Levy D. Outcome Measure Framework Design Document. Effective Health Care Program Research Report No. 43. (Prepared by the Outcome DEcIDE Center under Contract No. 290-2005-0035-1). AHRQ Publication No. 14-EHC019-EF. Rockville, MD: Agency for Healthcare Research and Quality. May 2014.
|Type of Entry||Source||Title||Sub-terms||Definition||Denominator||Numerator||Exclusion Criteria|
|Patient||National Committee for Quality Assurance||HbA1c in poor control1||None||HbA1c of >9.0% for patients 18-75 years of age with diabetes (Type 1 or Type 2)||N/A||N/A||N/A|
|Patient||National Committee for Quality Assurance||HbA1c control (<8.0%)2||None||HbA1c of < 8.0% for patients 18 - 75 years of age with diabetes (Type 1 & Type 2)||N/A||N/A||N/A|
|Patient||National Committee for Quality Assurance||HbA1c < 7.0% (controlled)3||None||HbA1c of < 7.0% for patients 18 - 64 years of age with diabetes mellitus (Type 1 & Type 2)||N/A||N/A||N/A|
|Population||National Committee for Quality Assurance||Comprehensive Diabetes Care: HbA1c control (<8.0%)4||None||The percentage of members 18 - 75 years of age with diabetes (type 1 and type 2) who had HbA1c control (<8.0%).||Members 18 - 75 years of ages with diabetes.||HbA1c level is <8.0% during the measurement year.||Members with a diagnosis of polycystic ovaries who did not have any face-to-face encounters with a diagnosis of diabetes, in any setting, during the measurement year or the year prior to the measurement year. Diagnosis can occur at any time in the member’s history, but must have occurred by December 31 of the measurement year.|
|Patient||French AutoImmunity and Rituximab (AIR) Registry||Response to rituximab treatment, as measured by the SELENA-SLEDAI5||None||Reduction of ≥3 in score on the Safety of Estrogens in Lupus Erythematosus: National Assessment (SELENA) version of the Systemic Lupus Erythematosus Disease Activity Index (SLEDAI)||N/A||N/A||N/A|
|Patient||Endometriosis Patient Registry||Average health status, as measured by the SF-126||None||A score of ≥50 on the Short-Form 12 (SF-12)||N/A||N/A||N/A|
|Patient||Endometriosis Patient Registry||Significant disability, as measured by the SF-126||None||A score of ≤40 on the Short-Form 12 (SF-12)||N/A||N/A||N/A|
|Population||Agency for Healthcare Research and Quality||Congestive Heart Failure (CHF) Mortality Rate7||None||None provided.||All discharges, age 18 years and older, with a principal diagnosis code of CHF.||Number of deaths (DISP=20) among cases meeting the inclusion and exclusion rules for the denominator.||
|Population||Joint Commission||Acute myocardial infarction (AMI) inpatient mortality8||None||Acute myocardial infarction (AMI) patients who expired during hospital stay||Discharges with an ICD-9-CM Principal Diagnosis Code for AMI||Inpatient mortality of AMI patients||
|Population||United HealthCare||Risk-Adjusted 30-Day All-Cause Readmission Rate9||None||Estimates a hospital 30-day risk-adjusted readmission rate by measuring percentage of acute inpatient hospitalizations during the measurement period that were followed by an acute readmission for any diagnosis from any hospital within 30 days.||Total inpatient discharges from acute care hospitals with discharge dates during the measurement period.||The number of acute inpatient stays that are admitted within 30 days of a prior acute discharge||
Denominator – Index Discharges:
Appendix A References
1. National Committee for Quality Assurance. NQF-Endorsed National Voluntary Consensus Standards for Physician-Focused Ambulatory Care. Appendix A - NCQS Measure Technical Specifications. April 2008. Version 0.7. www.ncqa.org/LinkClick.aspx?fileticket=POLoMIAi3Mo%3d&tabid=59&mid=1604&forcedownload=true. Accessed March 28, 2014.
2. National Quality Forum. Comprehensive Diabetes Care: HbA1c control (<8.0%). www.qualityforum.org/MeasureDetails.aspx?actid=0&SubmissionId=944. Accessed March 30, 2012.
3. National Committee for Quality Assurance (NCQA). HEDIS® 2011: Healthcare Effectiveness Data and Information Set. Vol. 1, narrative. Washington (DC): National Committee for Quality Assurance (NCQA); 2010.
4. National Quality Forum. Comprehensive Diabetes Care: HbA1c control (<8.0%). www.qualityforum.org/MeasureDetails.aspx?actid=0&SubmissionId=944. Accessed March 30, 2012.
5. Terrier B, Amoura Z, Ravaud P, Hachulla E, Jouenne R, Combe B, et al. Safety and efficacy of rituximab in systemic lupus erythematosus: results from 136 patients from the French AutoImmunity and Rituximab registry. Arthritis Rheum. 2010 Aug;62(8):2458-66.
6. Fourquet J, Baez L, Figueroa M, Iriarte RI, Flores I. Quantification of the impact of endometriosis symptoms on health-related quality of life and work productivity. Fertil Steril. 2011 Jul;96(1):107-12.
7. Agency for Healthcare Research and Quality. Inpatient Quality Indicators #16. Technical Specifications. Congestive Heart Failure (CHF) Mortality Rate. www.qualityindicators.ahrq.gov/Downloads/Software/SAS/V43/TechnicalSpecifications/ IQI%2016%20Congestive%20Heart%20Failure%20(CHF)%20Mortality%20Rate.pdf. Accessed February 23, 2012.
8. The Joint Commission. Specifications Manual for National Hospital Inpatient Quality Measures. Section 2k: NQF-Endorsed Voluntary Consensus Standards for Hospital Care: Measures Information Form. Version 3.3. www.jointcommission.org/assets/1/6/Specs_Manual_3.3b_PDF_Zip.zip. Accessed February 24, 2012.
9. United HealthCare. NQF Readmission Measure Resubmission. Risk-adjusted 30-day All-Cause Readmission Rate - NQF Resubmission Documents. www.uhc.com/live/uhc_com/Assets/Documents/Detailed_Risk_Model_2a1_14.pdf. Accessed March 28, 2014.