1. What is the frequency and severity of harms that are addressed by report cards and outcomes measurements to improve the safety of surgical care?
2. What measures or indicators have been used in surgical report cards used to examine the safety of surgical care?
3. What report cards and outcomes measurements to improve the safety of surgical care have been used to prevent, report, or mitigate harms to patients, and in what settings have they been used?
4. What is the reported rationale for the use of report cards and outcome measurements to prevent, report or mitigate the harms associated with surgical care?
5. What studies assessing the effectiveness and unintended effects of report cards and outcome measurements to improve the safety of surgical care have been published since the Making Healthcare Safer (MHS) II report of 2012?
6. What are common barriers and facilitators to implementing report cards and outcome measurements to improve the safety of surgical care?
7. What resources (e.g., cost, staff, time) are required for implementation of report cards and outcome measurements to improve the safety of surgical care?
8. What toolkits are available to support implementation of report cards and outcome measurements to improve the safety of surgical care?

The Agency for Healthcare Research and Quality (AHRQ) Making Healthcare Safer (MHS) reports consolidate information for healthcare providers, health system administrators, researchers, and government agencies about patient safety practices (PSPs) that can improve patient safety across the healthcare system—from hospitals to primary care practices, long-term care facilities, and other healthcare settings. In Spring of 2023, AHRQ launched its fourth iteration of the Making Healthcare Safer Report (MHS IV).

The use of surgical report cards and outcome measurements to improve the safety of surgical care was identified as a PSP to be included with high priority in the MHS IV reports using a modified Delphi technique by a Technical Expert Panel (TEP) that met in December 2022. The TEP included 15 experts in patient safety with representatives of governmental agencies, healthcare stakeholders, clinical specialists, experts in patient safety issues, and a patient/consumer perspective. See the MHS IV Prioritization Report for additional details.¹

The need for transparency and accountability in healthcare has led to the development of outcomes reporting, a tool aimed at informing patients and healthcare providers about the quality of care provided in a specific healthcare setting. Public reporting of surgical outcomes began in the 1980s in New York State, given concerns over the variation in mortality rates following coronary artery bypass surgery (CABG).² Data collected from this registry were published widely, and subsequent analyses reported reduced CABG mortality in New York State, likely because of the transparency.^3,4 Physician report cards – the prospective collection of clinical data that are used to provide risk-adjusted assessments of outcomes which are fed back to hospitals and surgeons for comparative purposes – have evolved substantially in the last 40 years, and progress has accelerated in the last 10 years following the implementation of the Affordable Care Act (ACA). The ACA established the Hospital Readmission Reduction Program which started a process to reduce hospital payments for certain 30-day readmissions.⁵ Surgical volumes have greatly expanded since the 1980s. A 1998 study estimated that over 40-50 million operations are performed in the USA each year in hospital settings and ambulatory care centers.⁶ Postoperative complications occur frequently and can increase hospitalizations, costs and lengths of stay.^7-9 Surgical report cards have the potential to improve operative morbidity and mortality by providing usable clinical data to highlight areas in need of improvement, as well as providing feedback across participating sites so centers can benefit from each other’s strengths and weaknesses.¹⁰ Reporting risk-adjusted postoperative outcomes can provide benchmarks intended to spur local and larger-scale quality improvement efforts to produce better patient outcomes.

Surgical report cards involve defining and reporting a wide range of outcomes (e.g., surgical site infections, post-operative venous thromboembolism) related to surgical care. Outcomes data can be compared across institutions and fed back to participating institutions to help them develop best practices intended to promote patient safety. The largest and best-known program for measuring and reporting surgical outcomes in the U.S. is the American College of Surgeons (ACS) National Surgical Quality Improvement Project (NSQIP). Born out of efforts initiated by Veterans Affairs (VA) Health System researchers and clinicians (the Veterans Affairs National Surgical Quality Improvement Project [VASQIP]) in the late 1980s, this multicomponent intervention provided a method to feedback data to facilities and surgeons on their performance. This served as a stimulus for quality improvement and increased patient safety. The current ACS NSQIP collects prospective, clinical data that provides risk-adjusted assessments of outcomes which are fed back to hospitals and surgeons for comparative purposes, with the goal of quality improvement. A bench-marked, peer-controlled database allows hospitals to compare 30-day outcomes across hospital types. With support from ACS NSQIP, individual sites work to design quality initiatives to achieve better outcomes and care in the areas of need. While ACS NSQIP is the largest and best known, there are many other databases from other groups, such as the Society of Thoracic Surgeon (STS) national databases, the VASQIP, the Trauma Quality Improvement Program (TQIP), the Vascular Quality Initiative (VQI), the American College of Surgeons Metabolic and Bariatric Surgery Accreditation and Quality Improvement Program (MBSAQIP), the ACS NSQIP Pediatric, the American Hernia Society Quality Task Force (ACHQC), and the Collaborative Endocrine Surgery Quality Improvement Program (CESQIP).

MHS II concluded that use of report cards and outcome measurements improved the quality and safety of surgical care, however, information on the impact on patient outcomes was limited. During the MHS IV prioritization process, the TEP noted that use of report cards and outcome measurements to improve safety of surgical care should be included in MHS IV with the availability of new published studies.

For the purposes of this review, we included evidence for the benefits or harms of any of the above-named programs, and quantitative information describing how these programs were implemented. The proposed report cards focused on any intervention that is specifically designed to promote patient safety, increase reporting of outcomes, or offer feedback to institutions to reduce patient safety events and associated harms. For the purposes of this review, surgery is defined to be a therapeutic or diagnostic procedure involving incision or excision or suturing of tissue that requires an operating room and anesthesia.

The overall purpose of this rapid response is to summarize the most relevant and recent literature on the use of report cards and outcome measurements to improve the safety of surgical care and how these can be implemented.

For this rapid response, strategic adjustments will be made to streamline traditional systematic review processes and deliver an evidence product in the allotted time. We will follow adjustments and streamlining processes proposed by the AHRQ Evidence-based Practice Center (EPC) Program. Adjustments include being as specific as possible about the questions, limiting the number of databases searched, modifying search strategies to focus on finding the most valuable studies (i.e., being flexible on sensitivity to increase the specificity of the search), and restricting the search to studies published since 2011 when the search was done for the MHS II report in English and performed in the United States, and having each study assessed by a single reviewer. Depending on the expected volume of literature, the EPC team may opt to have a randomly selected 10% sample of articles checked by a second reviewer or use the artificial intelligence (AI) feature of DistillerSR (AI Classifier Manager) as a second reviewer at the title and abstract screening stage.

We will search for recent high quality systematic reviews and will rely primarily on the content of any such systematic review that is found. We will not perform an independent assessment of original studies cited in any such systematic review.

We will answer Review Questions 1 and 2 by focusing on the harms and patient safety measures or indicators that are addressed in the studies we find for Review Question 5. For Review Question 2, we will focus on identifying relevant measures that are included in the Centers for Medicare & Medicaid Services (CMS) patient safety measures, AHRQ’s Patient Safety Indicators, or the National Committee for Quality Assurance (NCQA) patient safety related measures.

We will ask our content experts to answer Review Questions 3 and 4 by citing selected references, including PSPs used and explanations of the rationale presented in the studies we find for Review Question 5.

For Review Questions 6 and 7, we will focus on the barriers, facilitators, and required resources reported in the studies we find for Review Question 5.

For Review Question 8, we will identify publicly available patient safety toolkits developed by AHRQ or other organizations that could help to support implementation of the PSPs. To accomplish that task, we will review AHRQ’s Patient Safety Network (PSNet) and AHRQ’s listing of patient safety related toolkits and we will include any toolkits mentioned in the studies we find for Review Question 5. We will identify toolkits without assessing or endorsing them.

We will search for original studies and systematic reviews for Review Question 5 according to the inclusion and exclusion criteria presented in Table 1.

Table 1. Inclusion and Exclusion Criteria

MHS = Making Healthcare Safer

We will search PubMed and the Cochrane Library for systematic reviews published since 2020 that address the review questions. If no recent high quality systematic review that will adequately address the review questions is identified, we will conduct searches of PubMed for original studies published from 2011 onwards that address the review questions. To efficiently identify articles that meet the eligibility criteria, we will distribute citations from the literature search to team members, with plans to have the title and abstract of each citation reviewed by a single team member. The team will decide whether it has enough time and resources to ask a second team member to check a 10% sample of citations to verify that important studies were not excluded after the review of titles and abstracts. Alternatively, the team may opt to use the DistillerSR AI Classifier Manager as a semi-automated screening tool to conduct the review efficiently at the title and abstract screening stage. In that case, the title and abstract of each citation will be reviewed by a team member, and then the AI Classifier Manager will serve as a second reviewer of each citation.

To efficiently describe eligible studies, the full text of each potentially eligible article will be reviewed by a single team member to confirm eligibility and prepare a summary of the study, including author, year, study design, number of study participants, and main findings relevant to the review questions. Since Review Question 5 calls for identification of studies on the effectiveness of PSPs, we will describe the objectives and basic characteristics of those studies without conducting a detailed analysis of the findings of those studies. The team will decide whether it has enough time and resources to ask a second team member to check a randomly selected 10% sample of the articles to verify that important studies were not excluded and confirm the accuracy of extracted data.

To describe eligible systematic reviews, a single team member will prepare a summary including the author, year, number of studies by study design, and main findings relevant to each of our review questions. For Review Question 8, we will list the name and source of each relevant toolkit along with a 1-2 sentence description of each toolkit. We will not endorse any specific toolkit.

For studies that address Review Question 5 about the effectiveness of PSPs, the reviewer will use the Cochrane Collaboration’s tool for assessing the risk of bias found in randomized controlled trials (RCTs) or the ROBINS-I tool for assessing the Risk Of Bias In Non-randomized Studies of Interventions.^11,12 When assessing RCTs, we will use the Cochrane Collaboration’s tool that cover the domains of selection bias, performance bias, detection bias, attrition bias, reporting bias, and other bias.¹¹ When assessing non-randomized studies, we will use specific items in the ROBINS-I tool that assess bias due to confounding, bias in selection of participants into the study, bias in classification of interventions, bias due to deviations from intended interventions, bias due to missing data, bias in measurement of outcomes, and bias in selection of the reported results.¹² The risk of bias assessments will focus on the main outcome of interest in each study.

If we identify a recent eligible systematic review, the reviewer will use the criteria developed by the United States Preventive Services Task Force Methods Workgroup for assessing the quality of systematic reviews.¹³

• Good - Recent relevant review with comprehensive sources and search strategies; explicit and relevant selection criteria; standard appraisal of included studies; and valid conclusions.
• Fair - Recent relevant review that is not clearly biased but lacks comprehensive sources and search strategies.
• Poor - Outdated, irrelevant, or biased review without systematic search for studies, explicit selection criteria, or standard appraisal of studies.

The Task Leader will review the risk of bias assessments and any disagreements will be resolved through discussion with the team.

EPC core team members must disclose any financial conflicts of interest greater than $1,000 and any other relevant business or professional conflicts of interest. Related financial conflicts of interest that cumulatively total greater than $1,000 will usually disqualify EPC core team investigators.

This project is funded under Contract No. 75Q80120D00003/75Q80122F32009 from the Agency for Healthcare Research and Quality (AHRQ), U.S. Department of Health and Human Services. The AHRQ Task Order Officer will review contract deliverables for adherence to contract requirements and quality. The authors of this report are responsible for its content. Statements in the report should not be construed as endorsement by AHRQ or the U.S. Department of Health and Human Services.

The report will follow the most recent template approved by AHRQ at the time of approval of the protocol.

1. Rosen M, Dy SM, Stewart CM, Shekelle P, Tsou A, Treadwell J, Sharma R, Zhang A, Vass M, Motala A, Bass EB. Final Report on Prioritization of Patient Safety Practices for a New Rapid Review or Rapid Response. Making Healthcare Safer IV. (Prepared by the Johns Hopkins, ECRI, and Southern California Evidence-based Practice Centers under Contract No. 75Q80120D00003). AHRQ Publication No. 23-EHC019-1. Rockville, MD: Agency for Healthcare Research and Quality. July 2023. DOI: https://doi.org/10.23970/AHRQEPC_MHS4PRIORITIZATION. Posted final reports are located on the Effective Health Care Program search page. .
2. Hannan EL, Cozzens K, King SB, 3rd, et al. The New York State cardiac registries: history, contributions, limitations, and lessons for future efforts to assess and publicly report healthcare outcomes. J Am Coll Cardiol. 2012 Jun 19;59(25):2309-16. doi: 10.1016/j.jacc.2011.12.051. PMID: 22698487.
3. Peterson ED, DeLong ER, Jollis JG, et al. The effects of New York's bypass surgery provider profiling on access to care and patient outcomes in the elderly. J Am Coll Cardiol. 1998 Oct;32(4):993-9. doi: 10.1016/s0735-1097(98)00332-5. PMID: 9768723.
4. Altman LK. Heart surgery death rates decline in New York. New York Times. December 5, 1990;Sect. B10.
5. Mann DL, Tomaselli GF. The Affordable Care Act's Hospital Readmissions Reduction Program: Has the Biology Had Time to Catch Up With the Regulation? JACC Basic Transl Sci. 2016 Dec;1(7):687-9. doi: 10.1016/j.jacbts.2016.10.004. PMID: 30167551.
6. Dobson GP. Trauma of major surgery: A global problem that is not going away. Int J Surg. 2020 Sep;81:47-54. doi: 10.1016/j.ijsu.2020.07.017. PMID: 32738546.
7. Englesbe MJ, Brooks L, Kubus J, et al. A statewide assessment of surgical site infection following colectomy: the role of oral antibiotics. Ann Surg. 2010 Sep;252(3):514-9; discussion 9-20. doi: 10.1097/SLA.0b013e3181f244f8. PMID: 20739852.
8. Saint S, Elmore JG, Sullivan SD, et al. The efficacy of silver alloy-coated urinary catheters in preventing urinary tract infection: a meta-analysis. Am J Med. 1998 Sep;105(3):236-41. doi: 10.1016/s0002-9343(98)00240-x. PMID: 9753027.
9. Dimick JB, Chen SL, Taheri PA, et al. Hospital costs associated with surgical complications: a report from the private-sector National Surgical Quality Improvement Program. J Am Coll Surg. 2004 Oct;199(4):531-7. doi: 10.1016/j.jamcollsurg.2004.05.276. PMID: 15454134.
10. Maggard-Gibbons M. The use of report cards and outcome measurements to improve the safety of surgical care: the American College of Surgeons National Surgical Quality Improvement Program. BMJ Qual Saf. 2014 Jul;23(7):589-99. doi: 10.1136/bmjqs-2013-002223. PMID: 24748371.
11. Higgins JP, Altman DG, Gotzsche PC, et al. The Cochrane Collaboration's tool for assessing risk of bias in randomised trials. BMJ. 2011 Oct 18;343:d5928. doi: 10.1136/bmj.d5928. PMID: 22008217.
12. Sterne JA, Hernan MA, Reeves BC, et al. ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions. BMJ. 2016 Oct 12;355:i4919. doi: 10.1136/bmj.i4919. PMID: 27733354.
13. U.S. Preventive Services Task Force. U.S. Preventive Services Task Force Procedure Manual. Appendix VI. Criteria for Assessing Internal Validity of Individual Studies. U.S. Preventive Services Task Force. July 2017.