Historical Context for Measuring Organizational Characteristics in Care Delivery

A critical part of understanding the quality of medical care as outlined by Donabedian,¹ Andersen,² and others is understanding the key characteristics of organizations that deliver care while taking into consideration the multilevel nature of health care delivery. As demonstrated by the Donabedian, Andersen, and other frameworks, organizational characteristics can influence patient care-seeking behaviors, access to care, clinician decision-making, collaboration and coordination, patient and clinician experience, quality of care, and disparities in care.³ It is therefore critical to understand organizational characteristics when designing and disseminating interventions.³ Yano describes how inattention to organizational characteristics has historically limited clinical and delivery system interventions.⁴ Specifically, when interventions are tested in a single or small number of institutions, organizational characteristics are less likely to vary, so they are either ignored or controlled. As a result, knowledge of the organizational characteristics that can influence the success (or failure) of the intervention is lacking. Thus, when trying to disseminate the intervention, organizations are unaware of the characteristics critical to the intervention’s effectiveness—or whether and how the intervention needs to be adapted to different organizational contexts. This limitation weakens the generalizability of the findings derived from one setting to other settings and to population-level interventions.

To provide a historical perspective, a 1966 paper from Avedis Donabedian includes his now classic structure-process-outcome framework, and the interaction among these features, for evaluating the quality of medical care.¹ Structural characteristics include the levels of care, types of care available, and size (e.g., number of beds in a hospital). Process characteristics include specific aspects of care delivery (e.g., whether cancer screening occurs according to guidelines). Outcome characteristics include the end results of care (e.g., survival, health-related quality of life). Over the past half century, efforts have sought to expand our understanding of the quality of care, measurement frameworks to assess it, and how to produce it. Organizational context (as the fungible internal and external elements of setting) has been added to the classic structure-process-outcomes triad. A 2011 review of organizational characteristics influencing patient safety practices noted its importance.⁵ In 2009, as part of a team of safety researchers, Marsteller laid out critical organizational aspects to measure in the production of safe care.⁶ In 2014, Marsteller and colleagues also extrapolated from existing frameworks to elucidate the range of organizational and other influences on implementation of provider behavior change to desired practices in health care organizations.⁷ It is now widely recognized that group traits developed through interactions among members of an organization (such as shared mental models and psychological safety) are critical to the success of care delivery and yet are distinct from organizational "process" characteristics in the original framework.^1,8,9

Similarly, Ronald Andersen’s Behavioral Model of Health Service Use developed in the late 1960s has evolved over time, as described in his 1995 article.² The initial model focused on the person and family, including predisposing characteristics (demographic, social structure, health beliefs), enabling resources (at the personal/family and community levels), and perceived and evaluated needs, all culminating in health service use. The version of the model presented 25 years later adds the critical role of the healthcare system and external environment at the front end and the outcomes that result from health service use on the back end. Thus, the 1995 version starts with the healthcare system and external environment and traces how they interact with population characteristics (including enabling resources and needs) to influence health behaviors (i.e., personal health practices, use of health services), and outcomes (i.e., health status and satisfaction).

Opportunities and Challenges in Measuring Healthcare Organization Characteristics

The value of understanding organizational characteristics is equaled by the challenges in doing so, and use of organizational theories to inform measurement approaches is lacking in many cases. For example, organizational characteristics influencing the efficiency and quality of health care delivery involve multiple levels (e.g., clinic, system, and local community environment) and multiple perspectives (e.g., patient, provider, administrative).³ Also, changes in care delivery models are influenced by contextual characteristics within the organization, system, and community surrounding the organization.10 Measuring these non-structural characteristics using externally valid constructs is challenging.³ For example, while measuring the number of beds and patient volume may be straightforward, assessing organizational change (e.g., readiness to adopt a new health care delivery model) is a much more complex undertaking, generally requiring labor intensive surveying or interviewing to appropriately characterize. If characteristics are social creations produced organically in real time, then one might question how they can even be measured. Another challenge stems from the fact that studies on the effects of organizational characteristics are often observational and highly variable in their design and methods. This can lead to difficulty homing in on generalizable information. Further, as the health system rapidly evolves, it becomes increasingly difficult to determine how older studies apply to the current healthcare system. This has implications for how far back literature is still relevant, and how we plan to account for changes in the evidence base over time.

Assessing organizations that deliver cancer care involves even greater challenges.³ Multiple types of organizations are involved in cancer care delivery, ranging from solo practices to large integrated systems. The same patient may be seen and cared for by multiple types of organizations. For example, one breast cancer patient may be diagnosed at a stand-alone imaging facility, have surgery at an academic medical center, and receive chemotherapy close to home at a small medical oncology practice. A different breast cancer patient may experience her entire cancer journey in a single, integrated healthcare delivery system. This continuum between modular and integrated care delivery raises questions within and across delivery systems regarding how care will be coordinated within and between systems and who will be responsible for providing supportive services such as psychosocial care. Thus, tools to evaluate coordination of multidisciplinary care are increasingly important.^11,12

Although patient characteristics can explain some variance in care delivery and outcomes, much of the variance may be attributable to the hospitals' negotiation power with insurers, system-level infrastructure readiness, variable leadership and teamwork, organizational culture differences, or the hospitals' workforce network within the community. Therefore, incorporating organizational characteristics, contextual elements, and processes in cancer care delivery research is critical to enhance our knowledge of the context and environment where the care is delivered, improve the reliability of interventions as they are disseminated in new settings, and reduce waste in public investment. Ultimately, such knowledge will help to improve patient outcomes over the continuum of cancer care.

Research conducted to date reflects both the opportunities and challenges in assessing organizational characteristics. In their review of cancer care delivery research protocols from the National Cancer Institute (NCI) Community Oncology Research Program (NCORP), Weaver et al. found that assessment of organizational characteristics was common (15/19 protocols, 79%), with all 15 protocols including some assessment of structural characteristics, and almost all (14/15) assessing at least one process measure; organization-level outcomes were assessed in 80% of protocols (12/15).³ While assessing organizational characteristics was common, the extent to which measurement approaches were based on organizational theories was more variable. Most intervention protocols referred to implementation science frameworks,³ which call on many, although often not all, relevant healthcare organization characteristics or theories. In addition, many measures used to assess organizational characteristics were investigator-developed, with little or no information about the instrument's validity or psychometric performance.³

Refining an Organization Characteristics Framework

As part of their review, Weaver and colleagues³ considered a number of frameworks identifying organization characteristics, including those by Donabedian,¹ Andersen,² and the Consolidated Framework for Implementation Research (an implementation science approach),^13,14 as well as a review of organizational- and system-level characteristics related to shared decision making.¹⁵ Two frameworks emerged as being most relevant and complementary and were used in their review: Yano’s Organizational Research Framework and Piña's Health Care Delivery Organizations and Systems Framework.^4,16 After integrating those two frameworks for their review of NCORP protocols, Weaver, Breslau, and colleagues continued to develop it by cross-walking with additional frameworks (Ferlie and Shortell's Quality Improvement Framework, Wagner's Chronic Care Model, Garvin's framework for Building a Learning Organization, and the Agency for Healthcare Research and Quality (AHRQ) Learning Health System Framework), as well as obtaining internal and external input.

Figure 1. Draft Integrated Framework for Organizational Characteristics Influencing Cancer Care

Figure 1 presents the resulting draft integrated framework for organizational characteristics influencing cancer care, based on adaptations by Weaver, Breslau and colleagues of the previous framework.³ The framework includes three domains covering 47 specific characteristics. Some of these characteristics are straightforward to assess. For example, under organizational context, size and volume can be measured using the number of beds and number of patients served. Other characteristics present substantial measurement challenges. For example, under organizational processes, measuring communication requires assessing processes and procedures to communicate and, ideally, the quality of that communication. This measurement challenge was evident in the Weaver et al. review, which found that investigators frequently had to rely on study-specific survey items to assess certain characteristics because established and psychometrically sound measures were unavailable.³ Some of the characteristics are purely descriptive (e.g., having more or less beds is not inherently good or bad), whereas other characteristics have a normative component (e.g., an organization wants to have better communication).⁴ Frequently, it is these normative characteristics that are hard to measure and prone to introducing measurement or classification bias.⁴ Identifying existing validated instrumentation and developing new measurements using rigorous psychometric approaches can provide researchers and organizations with the tools needed to conduct stronger research and implement more successful care processes and practices.

Improving Measurement of Healthcare Organization Characteristics in Cancer Care Delivery

Several important points emerge from the above background. First, the assessment of organization characteristics would benefit from greater guidance from organizational theory. Second, there is a need for better conceptual and definitional clarity of organization characteristics. Third, while measuring some characteristics is straightforward, other characteristics present measurement challenges. Thus, there is a need to explore how organizational theory can inform healthcare organization characteristic conceptualization, clarify the definitions of these characteristics in the context of cancer care, and identify existing standardized measures for assessing these characteristics, as well as elaborate on gaps and areas for further research. Doing so will improve our understanding of how organization characteristics influence cancer care delivery at the patient, provider, and system levels. This information, in turn, will inform our ability to improve access, quality, and outcomes, and reduce disparities in care. The decisional dilemma is "how can we define and measure organization characteristics to improve research on cancer care delivery and enhance cancer patient care and outcomes?"

The purpose of this Technical Brief is to (1) catalog organizational characteristics, context and process measures relevant to cancer care delivery research, (2) assess the extent to which measurement has been standardized, (3) evaluate how useful these measures have been, or could be, in cancer care delivery research, and (4) identify areas requiring further research. The results of this Technical Brief will inform a compendium that can serve as a resource to the cancer care delivery research community. This compendium will complement the existing AHRQ Compendium of U.S. Health Systems by providing information about important healthcare organization characteristics that cannot be obtained from administrative or claims data.¹⁷

We will use these questions to guide our work:

KQ 1: What frameworks have been developed or applied to examine the effects of organizational characteristics on the delivery of and outcomes associated with cancer screening, diagnosis, and treatment?

KQ 1a: How do these existing frameworks compare to the draft integrated framework developed by Weaver, Breslau and colleagues³?

KQ 2: What approaches have been used to measure or understand the organizational context and process characteristics related to the delivery of cancer screening, diagnosis, and treatment?

KQ 3: Which healthcare organization context and process characteristics have been examined in studies assessing the delivery of cancer screening, diagnosis, and treatment?

KQ 3a: For each identified study, what were the following: (i) Study design; (ii) Setting; (iii) Population; (iv) Measures of organizational context and process characteristics (e.g., measurement instrument name and type, number of items, references, etc.); and (v) Primary and secondary clinical outcomes studied?

KQ 4: What are the evidence gaps and future research needs?

KQ 4a: What are the evidence gaps in the current understanding of how organizational characteristics impact cancer care delivery and cancer-related outcomes?

KQ 4b: What methodologic approaches or measurement tools are needed to better understand the impact of organizational context and processes on the delivery of and outcomes associated with cancer screening, diagnosis, and treatment?

1. Data Collection

A. Discussion With Key Informants (KIs)

We will recruit experts in cancer care delivery research and organizational science as applied to cancer care, as well as government representatives, to provide perspectives on what important influences on cancer care can be attributed to organizational context or process. Specifically, we will obtain KI perspectives on the protocol for our Technical Brief via a brief feedback form and group Zoom call. We will ask KIs for their input on our protocol for conducting the systematic literature search (including the most relevant grey literature resources) and for using the draft integrated framework to guide extraction and synthesis of information from relevant literature. We also will ask the KIs to identify frameworks, models, or theories (other than the draft integrated framework) that describe how organizational characteristics may influence cancer care (screening, diagnosis, or treatment) or health care delivery in general.

Prior to convening the KIs, we will circulate the protocol and a brief feedback form. We will also share the questions we expect to discuss with them. To accommodate the KI's constraints and schedules, we may hold more than one Zoom session. Each Zoom meeting will last 1 hour, and any unfinished items will be followed up via email. In addition to open discussion, group facilitators will use a round-robin process to ensure that all members of the panel have an opportunity to contribute their thoughts and perspectives. AHRQ and NCI project officers will attend the KI meetings.

After the meeting, the team will share minutes and an edited protocol with the KIs to let them know how we incorporated their advice. The KIs will be invited to review the draft report during the public review period. KIs will receive an honorarium for their contributions (except for government members of the review panel) and will be officially acknowledged in the Technical Brief for transparency.

B. Grey Literature Search

We will search the grey literature (e.g., LexisNexis, websites, white papers) for relevant reports from governmental agencies or nongovernmental organizations. A list of proposed sources is included in Table 1. We will also refer to existing resources and compendia, such as the American Hospital Association's (AHA) annual survey and Medicare Cost Report Data that contain information about hospitals' predisposing characteristics, utilization, costs, and other financial measures. They complement AHRQ's Compendium of U.S. Health Systems.17 Supplementing the literature review with these other resources will inform our understanding of the gaps in research conducted to date and help identify the work needed to develop and refine the organizational characteristic framework. Table 1 lists some of the supplementary sources we will use to address challenges in identifying and measuring relevant organizational characteristics and processes.

Table 1. Supplementary Sources that will be Used to Address Challenges in the Literature Review

Abbreviations: AHRQ = Agency for Healthcare Research and Quality; CHSP = Comparative Health System Performance; CPCRN = Cancer Prevention and Control Research Network; VA SOTA = Veterans Affairs State of the Art

C. Published Literature Search

We will conduct a systematic search for published evidence using PubMed, CINAHL, SCOPUS, PsycINFO, and the Cochrane Central Register of Controlled Trials. We will develop an initial search strategy for PubMed based on an analysis of medical subject headings (MeSH) terms and key words and modify this initial search strategy for use in the other database. A preliminary search strategy is included in Appendix A. Search strategy will be peer reviewed by a JHU librarian. We will limit the search to the last 12 years because older studies have little relevance to modern cancer care delivery. A 12-year cut-off corresponds roughly to the implementation of the Affordable Care Act (ACA) (circa 2010). The ACA featured value-based payment and promoted new care models such as the patient centered medical home and accountable care organizations that led to alterations in care delivery in many forms. In their recent review of overlaps between the health care and organizational science literature, Mayo, Myers and Sutcliffe²⁶ noted that:

"the passing of the ACA … [was] a turning point in the relevance of and interest in [organizational science and health care] scholarship. The decade following the passage of the ACA has seen tremendous shifts in the fundamental organization, coordination, and provision of health care in the US…" (p. 538.)

We will conduct separate searches by key questions: (1) to identify relevant frameworks, which may rely more heavily on healthcare literature from organizational journals (KQ1); and (2) to identify approaches, designs, and measures that have been considered specifically in the case of cancer care delivery (where we will require that studies had as a stated objective the investigation of an organizational influence on cancer care delivery) (KQ2 and (3).The literature from the KQ1–3 searches will also inform KQ4 (evidence gaps, research needs), particularly the discussion sections of papers and grey literature from above.

Unique citations identified by the search strategies will be independently assessed using the inclusion and exclusion criteria outlined in Table 2. We will use Distiller AI as a semi-automated screening tool to conduct this review more efficiently at the abstract screening stage. First, 10 percent of the unique citations identified by the search strategies will be screened by a pair of reviewers at the abstract screening stage, with the remainder of the abstracts to be screened by the AI program based on the results of our screening of the initial sample. Reviewers will independently check 10 percent of the screening by the AI program for accuracy and consistency. We will update the literature search during the public posting phase and incorporate any new information into the report.

Table 2. Inclusion and exclusion criteria

Abbreviation: KQ=Key Question

2. Data Organization and Presentation

A. Information Management

We will use DistillerSR to extract information about the characteristics of the frameworks for examining effects of organizational characteristics on cancer care delivery and outcomes (KQ 1), the approaches that have been used to measure organizational characteristics and processes related to cancer care delivery (KQ 2), organizational characteristics that have been examined in studies assessing cancer care delivery, along with information about the study design, setting, population, specific organizational measures, and outcomes used (KQ 3), and evidence gaps or research needs identified in studies (KQ 4). For KQ 1, we will determine whether identified frameworks have been applied to cancer care (screening, diagnosis, or treatment). If a framework has been applied to cancer care, we will extract detailed information about the framework. If a framework mentioned by the KIs has not been applied to cancer care, we will comment on that information separately from the information about frameworks that have been applied to cancer care. A preliminary abstraction form for KQ 2, KQ 3, and KQ 4 is included in Appendix B.

Paired reviewers will independently assess the quality of information about the organizational characteristic measures used in studies. That assessment will focus on whether the measure is defined clearly (i.e., in a manner that can be readily replicated), whether the measure can be classified as being nominal, ordinal or continuous; and whether the measure is intended to be descriptive or normative (i.e., based on data that is compared to the distribution from a reference population). We will classify studies according to whether they apply to cancer screening, diagnosis, and/or treatment so that we can determine the extent to which findings apply to different aspects of cancer care.

B. Data Presentation

We will present data in evidence tables in appendices. To help convey the findings of our review in a compelling and useful manner, we will explore the use of innovative data visualizations, such as bubble plots that show how our findings on KQ 2 and 3 differ according to whether the studies focused on screening, diagnosis, or treatment.

1. Donabedian A. Evaluating the quality of medical care. The Milbank Memorial Fund Quarterly. 1966;44(3):166-206.
2. Andersen RM. Revisiting the behavioral model and access to medical care: does it matter? J Health Soc Behav. 1995 Mar;36(1):1-10. PMID: 7738325.
3. Weaver SJ, Verhoeven DC, Castro KM, et al. Thematic Analysis of Organizational Characteristics in NCI Community Oncology Research Program Cancer Care Delivery Research. JNCI Cancer Spectr. 2022 Mar 2;6(2)doi: 10.1093/jncics/pkac008. PMID: 35603839.
4. Yano EM. The role of organizational research in implementing evidence-based practice: QUERI Series. Implement Sci. 2008 May 29;3:29. doi: 10.1186/1748-5908-3-29. PMID: 18510749.
5. Taylor SL, Dy S, Foy R, et al. What context features might be important determinants of the effectiveness of patient safety practice interventions? BMJ Qual Saf. 2011 Jul;20(7):611-7. doi: 10.1136/bmjqs.2010.049379. PMID: 21617166.
6. Pronovost PJ, Goeschel CA, Marsteller JA, et al. Framework for patient safety research and improvement. Circulation. 2009 Jan 20;119(2):330-7. doi: 10.1161/circulationaha.107.729848. PMID: 19153284.
7. Kahan S, Gielen AC, Fagan PJ, et al. Health behavior change in populations: Johns Hopkins University Press; 2014.
8. Mathieu JE, Heffner TS, Goodwin GF, et al. The influence of shared mental models on team process and performance. J Appl Psychol. 2000 Apr;85(2):273-83. doi: 10.1037/0021-9010.85.2.273. PMID: 10783543.
9. Edmondson AC. The fearless organization: Creating psychological safety in the workplace for learning, innovation, and growth: John Wiley & Sons; 2018.
10. Rafferty AE, Jimmieson NL, Armenakis AA. Change Readiness: A Multilevel Review. J Manage. 2013;39(1):110-35. doi: 10.1177/0149206312457417.
11. Pluimers DJ, van Vliet EJ, Niezink AGH, et al. Development of an instrument to analyze organizational characteristics in multidisciplinary care pathways; the case of colorectal cancer. BMC Res Notes. 2015;8:134. doi: 10.1186/s13104-015-1084-1.
12. McDonald KM, Schultz E, Albin L, et al. Care Coordination Atlas Update Version 4 (Prepared by Stanford University under subcontract to American Institutes for Research on Contract No. HHSA290-2010-00005I). AHRQ Publication No. 14-0037- EF. Rockville, MD: Agency for Healthcare Research and Quality. June 2014. 2014.
13. Kirk MA, Kelley C, Yankey N, et al. A systematic review of the use of the Consolidated Framework for Implementation Research. Implement Sci. 2016 May 17;11:72. doi: 10.1186/s13012-016-0437-z. PMID: 27189233.
14. Leeman J, Baquero B, Bender M, et al. Advancing the use of organization theory in implementation science. Prev Med. 2019 Dec;129s:105832. doi: 10.1016/j.ypmed.2019.105832. PMID: 31521385.
15. Scholl I, LaRussa A, Hahlweg P, et al. Organizational- and system-level characteristics that influence implementation of shared decision-making and strategies to address them—a scoping review. Implement Sci. 2018 Mar 9;13(1):40. doi: 10.1186/s13012-018-0731-z. PMID: 29523167.
16. Piña IL, Cohen PD, Larson DB, et al. A framework for describing health care delivery organizations and systems. Am J Public Health. 2015 Apr;105(4):670-9. doi: 10.2105/ajph.2014.301926. PMID: 24922130.
17. Agency for Healthcare Research and Quality. (2021). Compendium of US Health Systems, 2018 (Updated 2021).
18. Care Coordination Measures Atlas Update. Rockville, MD: Agency for Healthcare Research and Quality; 2014. Accessed on October 6, 2022.
19. Journal of General Internal Medicine. Springer Nature Switzerland AG; 2022. Accessed on October 6, 2022.
20. Comparative Health System Performance Bibliography. Rockville, MD: Agency for Healthcare Research and Quality. Accessed on October 6, 2022.
21. Oncology Care Model. Baltimore, MD: U.S. Centers for Medicare & Medicaid Services; 2022. Accessed on October 6, 2022.
22. Choose Measures. Seattle, WA: University of Washington; 2022. https://impsciuw.org/implementation-science/research/measures/#link-target-1, https://journals.sagepub.com/topic/collections-irp/irp-1-systematic_reviews_of_methods_to_measure_implementation_constructs/irp. Accessed on October 6, 2022.
23. Organizational Theory for Implementation Science (OTIS) Workgroup. CPCRN; 2020. Accessed on October 6, 2022.
24. AHA Annual Survey Database. American Hospital Association; 2022. https://www.ahadata.com/aha-annual-survey-database. Accessed on October 6, 2022.
25. Cost Reports. Baltimore, MD: U.S. Centers for Medicare & Medicaid Services; 2022. Accessed on October 6, 2022.
26. Mayo A, Myers C, Sutcliffe K. Organizational Science and Health Care. Acad Manag Ann. 2021;15(2):537-76. doi: 10.5465/annals.2019.0115.

Not applicable.

If the EPC needs to amend the protocol, we will give the date of each amendment, describe the change, and provide rationale in this section. Changes will not be incorporated into the protocol.

Within the Technical Brief process, Key Informants serve as a resource to offer insight into the clinical context of the technology/intervention, how it works, how it is currently used or might be used, and which features may be important from a patient or policy standpoint. They may include clinical experts, patients, manufacturers, researchers, payers, or other perspectives, depending on the technology/intervention in question. Differing viewpoints are expected, and all statements are crosschecked against available literature and statements from other Key Informants. Information gained from Key Informant interviews is identified as such in the report. Key Informants do not do analysis of any kind nor contribute to the writing of the report and will not review the report, except as given the opportunity to do so through the public review mechanism.

Key Informants must disclose any financial conflicts of interest greater than $5,000 and any other relevant business or professional conflicts of interest. Because of their unique clinical or content expertise, individuals are invited to serve as Key Informants and those who present with potential conflicts may be retained. The TOO and the EPC work to balance, manage, or mitigate any potential conflicts of interest identified.

Peer reviewers are invited to provide written comments on the draft report based on their clinical, content, or methodologic expertise. Peer review comments on the draft report are considered by the EPC in preparation of the final report. Peer reviewers do not participate in writing or editing of the final report or other products. The synthesis of the scientific literature presented in the final report does not necessarily represent the views of individual reviewers. The dispositions of the peer review comments are documented and may be published three months after the publication of the Evidence report.

Potential Reviewers must disclose any financial conflicts of interest greater than $5,000 and any other relevant business or professional conflicts of interest. Invited Peer Reviewers may not have any financial conflict of interest greater than $5,000.

Peer reviewers who disclose potential business or professional conflicts of interest may submit comments on draft reports through the public comment mechanism.

EPC core team members must disclose any financial conflicts of interest greater than $1,000 and any other relevant business or professional conflicts of interest. Related financial conflicts of interest that cumulatively total greater than $1,000 will usually disqualify EPC core team investigators.

This project was funded under Contract No. 75Q80120D00003 from the Agency for Healthcare Research and Quality, U.S. Department of Health and Human Services. The AHRQ Task Order Officer reviewed contract deliverables for adherence to contract requirements and quality. The authors of this report are responsible for its content. Statements in the report should not be construed as endorsement by the Agency for Healthcare Research and Quality or the U.S. Department of Health and Human Services.

Table A1. KQ1: PubMed search strategy

Table A2. KQ2 and KQ3: PubMed search strategy

Table B1. Proposed data abstraction form for KQ2, KQ3, and KQ4