Based on the evidence published during the included dates, how effective is the implementation of high reliability organization (HRO) principles on patient safety outcomes, and what are their unintended effects?

Contextual Questions

1. How does the PSP prevent or mitigate harms?
2. What are common barriers and facilitators to implementation?
3. What resources (e.g., cost, staff, time) are required for implementation?
4. What toolkits are available to support implementation?

The Agency for Healthcare Research and Quality (AHRQ) Making Healthcare Safer (MHS) reports consolidate information for healthcare providers, health system administrators, researchers, and government agencies about PSPs that can improve patient safety across the healthcare system—from hospitals to primary care practices, long-term care facilities, and other healthcare settings. In Spring of 2023, AHRQ launched its fourth iteration of the MHS Report (MHS IV).

HRO implementation was identified as a high priority topic for inclusion in the MHS IV reports using a modified Delphi technique by a Technical Expert Panel (TEP) that met in December 2022. The TEP included 15 experts in patient safety with representatives of governmental agencies, healthcare stakeholders, clinical specialists, experts in patient safety issues, and a patient/consumer perspective. See the MHS IV Prioritization Report for additional details.¹

HROs are organizations that operate in complex, high-hazard domains for extended periods without serious accidents or catastrophic failures.² Healthcare organizations have increasingly implemented HRO principles to improve safety and quality; however, there have been few examinations of how implementation of HRO principles directly impacts patient safety.³

In an HRO, systems consistently perform safely and efficiently, and they produce high quality outcomes even in the face of complex challenges with high hazard potential.³ HRO culture can be described as “collective mindfulness,” such that all workers share a sense of responsibility and accountability for safety and reliability; they anticipate, early detect, and respond to unsafe conditions before they result in adverse events.⁴ Respectful interaction and heedful interrelating are critical elements, as they foster trust and empower workers to honestly speak up without hesitation.⁵ HROs develop over time through continual feedback, analysis, reflection, and refinement.6 Healthcare leaders look to replicate HROs to manage risk and reduce harm.

HRO implementation is designed to change thinking about patient safety through the following 5 principles: (1) sensitivity to operations; (2) reluctance to simplify; (3) preoccupation with failure; (4) deference to expertise; and (5) commitment to resilience.2 These principles are outlined in two categories: anticipation (sensitivity to operations, reluctance to simplify, preoccupation with failure) and containment (deference of expertise, commitment to resilience).² Through the two pillars of anticipation and containment, an HRO intends to prevent all types of harm. Activities and practices that embody these principles differ depending on a team, unit, or organization's goals and tasks, unique context, resources, and constraints.^{6, 7}

An evidence brief conducted by the Department of Veteran's Affairs (VA) in 2019 identified various published frameworks (multicomponent and multidisciplinary strategies) that guide implementation of HRO principles into a healthcare system.⁸ From those frameworks, five commonly reported implementation strategy components were observed: developing leadership; supporting a culture of safety; building and using data systems to measure progress; providing training and learning opportunities for providers and staff; and implementing quality improvement interventions to address specific patient safety issues. Complementary practices to strengthen implementation included the need to incorporate an awareness of justice, equity, and patient-centeredness into all elements of HRO implementation; the importance of involving a variety of stakeholders involved in healthcare delivery, including patients and families; and the value of integrating change management strategies into HRO delivery. Examples of implementation activities included basic error prevention training for staff and leadership training for leaders; enhanced root cause analysis processes using an electronic tracking system; provider peer safety coaches to coach their peers in the use of error prevention techniques; routine sharing of good catches and lessons learned; and increased communication through safety huddles. Successful facilitators to implementation included hiring an outside consultant, leadership commitment, and enacting policies to facilitate data-sharing. Barriers to implementation included competing priorities and high costs.

The evidence brief also described metrics for measuring a health system’s progress towards becoming an HRO.8 Metrics varied in terms of concepts measured and ranged from surveys on culture of safety to extent of integration of HRO principles into practice. The Joint Commission’s High Reliability Health Care Maturity (HRHCM) model/OroTM 2.0 emerged as the most rigorously developed and validated tool and comprehensively addressed all five of the HRO implementation strategy components. Progress measurement is essential but challenging, and no single metric for issues such as safety can provide a clear indication of how a system is performing.⁹

Although HRO implementation was not covered as a PSP in previous MHS reports, the topic of HROs was discussed in the introduction of “Cross-cutting Patient Safety Topics/Practices” in MHS III. HROs were highlighted as a recent trend in healthcare quality improvement and described as “organizations that operate in complex environments while maintaining high levels of safety for extended periods of time.”² Other HRO characteristics were described, including strong leaders that are committed to safety; staff that speak up when a deviation in safety processes or practices is observed without fear of blame or punishment; and process improvement tools that systematically solve safety issues, like reliable assessments of the problem’s scope, identification of root causes associated with the problem, and application of the most appropriate solutions. The importance of context was also noted, such that what works in one organization may not work in another.

The purpose of this review is to determine the effectiveness of HRO principles implementation on patient safety outcomes.

For this rapid review, strategic adjustments will be made to streamline traditional systematic review processes and deliver an evidence product in the allotted time. We will follow adjustments and streamlining processes proposed by the AHRQ Evidence-based Practice Center (EPC) Program. Adjustments include being as specific as possible about the questions, limiting the number of databases searched, modifying search strategies to focus on finding the most valuable studies (i.e., being flexible on sensitivity to increase the specificity of the search), restricting the search to studies performed in the United States and published in English since 2019 when the search was done for the VA’s rapid review on implementation of HRO principles, and having each study assessed by a single reviewer.8 To efficiently identify studies that meet the eligibility criteria, we will distribute citations from the literature search to team members, with plans to have the title and abstract of each citation reviewed by a single team member. A single team member will review the full text of each remaining potentially eligible article to confirm eligibility and extract data. Depending on the expected volume of literature, the EPC team may opt to have a randomly selected 10% sample of articles checked by a second reviewer.

We will search for recent high quality systematic reviews published since 2019 with strength of evidence assessment and will rely primarily on the content of any such systematic review that is found. We will not perform an independent assessment of original studies cited in any such systematic review.

For contextual questions 1 (rationale), 2 (barriers and facilitators), and 3 (resources), we will draw on information reported in the studies identified by the Review Question.

For contextual question 4, we will identify publicly available patient safety toolkits developed by AHRQ or other organizations that could help to support implementation of the PSPs. To accomplish that task, we will review AHRQ’s Patient Safety Network (PSNet) and AHRQ's listing of patient safety related toolkits and we will include any toolkits mentioned in the studies we find for the Review Question. We will identify toolkits without assessing or endorsing them.

We will search for original studies and systematic reviews on the Review Question according to the inclusion and exclusion criteria presented in Table 1.

Table 1. Inclusion and Exclusion Criteria

Our search strategy will focus on databases expected to have the highest yield of relevant studies, including PubMed and the Cochrane Library, supplemented by a narrowly focused search for unpublished reports that are publicly available from governmental agencies or professional societies having a strong interest in high reliability.

Data will be extracted by a single reviewer. Information will be organized according to the review questions, and will include author, year, study design, frequency and severity of the harms, measures of harm, characteristics of the PSP, rationale for the PSP, outcomes, unintended consequences, implementation barriers and facilitators, required resources, and description of toolkits. To streamline data extraction, we will sort eligible studies by specific PSP (if the report covers more than one specific type of approach to HRO implementation), and focus on extracting information about characteristics, outcomes, and barriers/facilitators most pertinent to a specific PSP.

For studies that address the Review Question about the effectiveness of PSPs, the primary reviewer will use the Cochrane Collaboration’s tool for assessing the risk of bias of randomized controlled trials (RCTs) or the ROBINS-I tool for assessing the Risk Of Bias In Non-randomized Studies – of Interventions.^{10, 11} When assessing RCTs, we will use the 7 items in the Cochrane Collaboration’s tool that cover the domains of selection bias, performance bias, detection bias, attrition bias, reporting bias, and other bias. 10 When assessing non-randomized studies, we will use specific items in the ROBINS-I tool that assess bias due to confounding, bias in selection of participants into the study, bias in classification of interventions, bias due to deviations from intended interventions, bias due to missing data, bias in measurement of outcomes, and bias in selection of the reported results.¹¹ The risk of bias assessments will focus on the main outcome of interest in each study.

If we identify a recent eligible systematic review, the primary reviewer will use the criteria developed by the United States Preventive Services Task Force Methods Workgroup for assessing the quality of systematic reviews.¹²

• Good - Recent relevant review with comprehensive sources and search strategies; explicit and relevant selection criteria; standard appraisal of included studies; and valid conclusions.
• Fair - Recent relevant review that is not clearly biased but lacks comprehensive sources and search strategies.
• Poor - Outdated, irrelevant, or biased review without systematic search for studies, explicit selection criteria, or standard appraisal of studies.

The Task Leader will review the risk of bias assessments and any disagreements will be resolved through discussion with the team. We will rely primarily on the reviews of good quality.

Selected data will be compiled into evidence tables and synthesized narratively. We will not conduct a meta-analysis. For the Review Question about the effectiveness of PSPs, we will record information about the context of each study and whether the effectiveness of the PSP differs across patient subgroups. If any of the PSPs have more than one study of effectiveness, we will grade the strength of evidence for those PSPs using the methods outlined in the AHRQ Effective Health Care Program (EHC) Methods Guide for Effectiveness and Comparative Effectiveness Reviews.¹³ Evidence grading would not add value for PSPs that do not have more than one available study.

We will report if the effectiveness of the PSP differs across patient subgroups, but will not conduct subgroup analyses.

We will submit the protocol to AHRQ and to the PROSPERO international prospective register of systematic reviews.

EPC core team members must disclose any financial conflicts of interest greater than $1,000 and any other relevant business or professional conflicts of interest. Related financial conflicts of interest that cumulatively total greater than $1,000 will usually disqualify EPC core team investigators from participation in the review.

Peer reviewers are invited to provide written comments on the draft report based on their clinical, content, or methodological expertise. The EPC considers all peer review comments on the draft report in preparation of the final report. Peer reviewers do not participate in writing or editing of the final report or other products. The final report does not necessarily represent the views of individual reviewers.

We will ask at least one clinical content expert and one methodological expert to review the draft report. Potential peer reviewers must disclose any financial conflicts of interest greater than $5,000 and any other relevant business or professional conflicts of interest. Invited peer reviewers may not have any financial conflict of interest greater than $5,000.

This project is funded under Contract No. 75Q80120D00003/75Q80122F32009 from the Agency for Healthcare Research and Quality (AHRQ), U.S. Department of Health and Human Services. The AHRQ Task Order Officer will review contract deliverables for adherence to contract requirements and quality. The authors of this report are responsible for its content. Statements in the report should not be construed as endorsement by AHRQ or the U.S. Department of Health and Human Services.

1. Rosen M, Dy SM, Stewart CM, et al. Final report on prioritization of patient safety practices for a new rapid review or rapid response. Making Healthcare Safer IV. (Prepared by the Johns Hopkins, ECRI, and Southern California Evidence-based Practice Centers under Contract No. 75Q80120D00003). AHRQ Publication No. 23-EHC019-1. Rockville (MD): Agency for Healthcare Research and Quality (AHRQ); 2023.
2. Weick KE, Sutcliffe KM. Managing the unexpected: Resilient performance in the age of uncertainty, 2nd ed. San Francisco (CA): Jossey-Bass; 2007.
3. Phillips R, Schwartz R, Sostman D, Boom M. Development and expression of a high-reliability organization. NEJM Catal Innov Care Deliv 2021;2(12).
4. Weick K, Sutcliffe K, Obstfeld D. Organizing for high reliability: Processes of collective mindfulness. In: Sutton R, Staw B, eds. Research in organizational behavior. Vol. 21. Stanford (CA): Elsevier Science/JAI Press; 1999:81-123.
5. Sutcliffe KM. Building cultures of high reliability: lessons from the high reliability organization paradigm. Anesthesiol Clin. 2023 Dec;41(4):707-17. PMID: 37838378.
6. Vogus TJ, Sutcliffe KM, Weick KE. Doing no harm: Enabling, enacting, and elaborating a culture of safety in health care. Acad Manag Perspect. 2010;24(4):60-77.
7. Christianson MK, Sutcliffe KM, Miller MA, Iwashyna TJ. Becoming a high reliability organization. Crit Care. 2011;15(6):314. PMID: 22188677.
8. Veazie S, Peterson K, Bourne D. VA Evidence-based Synthesis Program Reports.  Evidence Brief: Implementation of high reliability organization principles. Washington (DC): Department of Veterans Affairs (US); 2019.
9. Hines S, Luna K, Lofthus J. Becoming a high reliability organization: operational advice for hospital leaders. (Prepared by the Lewin Group under Contract No 290-04-0011) AHRQ Publication No 08-0022 Rockville, MD: Agency for Healthcare Research and Quality. April 2008.
10. Higgins JP, Altman DG, Gøtzsche PC, et al. The Cochrane Collaboration's tool for assessing risk of bias in randomised trials. BMJ. 2011;343:d5928. PMID: 2011599455, 22008217.
11. Sterne JA, Hernán MA, Reeves BC, et al. ROBINS-I: A tool for assessing risk of bias in non-randomised studies of interventions. BMJ (Clinical research ed). 2016;355:i4919. PMID: 27733354.
12. U.S. Preventive Services Task Force Procedure Manual. Appendix VI. Criteria for assessing internal validity of individual studies [Internet]. Rockville (MD): U.S. Preventive Services Task Force (USPSTF); 2017.
13. Methods guide for effectiveness and comparative effectiveness reviews. AHRQ publication no. 10(14)-EHC063-EF. Rockville (MD): Agency for Healthcare Research and Quality (AHRQ); 2014.