Development of Harmonized Outcome Measures for Use in Patient Registries and Clinical Practice: Methods and Lessons Learned

White Paper Feb 11, 2019

This item is available in PDF only (Full Report [556 KB]). People using assistive technology may not be able to fully access information in these files. For additional assistance, please contact us.

Executive Summary

Significant variation exists in both the types and definitions of outcome measures used in patient registries, even within the same clinical area. This variation, makes it difficult to compare, link, and aggregate data across a range of registries reflective of the spectrum of clinical care and reporting, thus reducing the potential utility of registry data. To address these limitations, the Agency for Healthcare Research and Quality (AHRQ) developed the Outcome Measures Framework (OMF), a conceptual model for classifying outcomes that are relevant to patients and providers across most conditions; it is intended to serve as a content model for developing harmonized outcome measures for specific clinical areas.1

AHRQ contracted L&M Policy Research, and its partners OM1 and AcademyHealth, to assess the feasibility of using the OMF to develop standardized libraries of outcome measures in five clinical areas: (1) Atrial fibrillation, (2) Asthma, (3) Depression, (4) Non-small cell lung cancer, and (5) Lumbar spondylolisthesis. These clinical areas were selected during the course of the project to represent a range of populations and care settings, different treatment modalities, and varying levels of existing measure harmonization. For each clinical area, the relevant registries and observational studies meeting project criteria were identified, and registry sponsors, informaticists, and clinical subject matter experts were invited to participate in a registry workgroup that focused on harmonizing outcome measures through a series of in-person and web-based meetings. A stakeholder group, primarily composed of patient and provider representatives and Federal partners, was also assembled for each clinical area to discuss challenges and provide feedback on the harmonization effort.

Key goals of this effort were to identify a minimum set of priority measurement concepts for each condition, standardize the definitions of the outcome measures within those concepts, and, where differences remain in the definitions, identify the specific elements or components that are dissimilar. As a final step in the harmonization process, clinical informaticists mapped the narrative definitions generated by the workgroups to standardized terminologies to produce a library of common data definitions that can eventually be implemented within electronic health records to facilitate standardized outcome data collection.

Challenges and Lessons Learned

This feasibility effort confirmed that the OMF is a robust model that supports classification and harmonization of outcome measures across an intentionally diverse group of clinical areas. While at least some level of harmonization was feasible in each of the five clinical areas, several factors affected the degree to which harmonization was reached in each of the workgroups, ranging from the implementation of the workgroups to outcome measure-related issues, such as gaps in existing evidence, lack of validated measurement tools, and variations in clinical practice. Broadly speaking, the challenges and lessons learned fall into four categories: differences across clinical areas, registry participation and variation among participating registries, recruiting and integrating stakeholders into the harmonization process, and translating narrative definitions into standardized terminologies.

Next Steps

Although each workgroup was focused on a specific clinical area, some common themes emerged around the implementation, dissemination, and ongoing governance of the harmonization process. Barriers to implementation exist, as registries often have limited funding and resources that prevent retroactively adopting harmonized measure sets. However, the development of the value proposition of adopting such a measure set would be beneficial to nascent registries. Additionally, publicizing the harmonized measures aggressively and as widely as possible would give the measures the greatest chance of being adopted by new registries. Strategies such as publishing articles and editorials in journals, blog posts, and presenting at society and association meetings were identified as priority methods. Coordination with other efforts, such as OMERACT and ICHOM, was also identified and prioritized, not only for dissemination purposes, but also so that all parties can leverage each other's work and limit duplication of efforts.

While beyond the scope of this project, a long-term plan for the ongoing curation and updating of outcome measures would appear vital to the ultimate success of any harmonization and standardization of outcome measures effort. A clear ongoing coordinating and governance process is necessary to oversee the harmonized measures, determine when updates and revisions are needed, and determine if new measures should be adopted and harmonized. Regular review and updates are also necessary to reflect changing treatment paradigms and to address challenges encountered by registries that implemented the measure set.

Finally, convening workgroups to harmonize additional clinical areas will allow AHRQ to build upon the experiences and lessons learned from this effort, as well as continually improve the overall harmonization process. Lessons learned during this project reflect the experiences of the five harmonized clinical areas. It is likely that other clinical areas will experience varying challenges for pursuing harmonization and generate additional learnings not yet observed by the project team.


  1. Gliklich RE, Leavy MB, Karl J, et al. A framework for creating standardized outcome measures for patient registries. J Comp Eff Res. 2014 Sep;3(5):473-80. PMID: 25350799.


Suggested citation: Leavy MB, Schur C, Kassamali FQ, Johnson ME, Sabharwal R, Wallace P, Gliklich RE. Development of Harmonized Outcome Measures for Use in Patient Registries and Clinical Practice: Methods and Lessons Learned. Final Report. (Prepared by L&M Policy Research, LLC under Contract No. 290-2014-00004-C) AHRQ Publication No. 19-EHC008-EF. Rockville, MD: Agency for Healthcare Research and Quality; February 2019. Posted final reports are located on the Effective Health Care Program search page.

Project Timeline

Registry of Patient Registries (RoPR)

May 31, 2011
Topic Initiated
May 31, 2012
May 31, 2012
Oct 31, 2012
Dec 4, 2012
Jun 2, 2014
Sep 27, 2016
Mar 5, 2018
Mar 5, 2018
Feb 11, 2019
White Paper
Dec 6, 2019
Dec 6, 2019
Nov 30, 2020
Page last reviewed December 2019
Page originally created February 2019

Internet Citation: White Paper: Development of Harmonized Outcome Measures for Use in Patient Registries and Clinical Practice: Methods and Lessons Learned. Content last reviewed December 2019. Effective Health Care Program, Agency for Healthcare Research and Quality, Rockville, MD.

Select to copy citation