Patient safety culture is the extent to which an organization’s culture supports and promotes patient safety. It refers to the beliefs, values, and norms shared by healthcare practitioners and other staff throughout the organization, which influence their actions and behaviors. Patient safety culture can be measured by determining what is rewarded, supported, expected, and accepted in an organization as it relates to patient safety¹. Establishing a culture of patient safety with a systems-based approach is essential to the delivery of high quality and safe care within healthcare organizations. To further emphasize the importance of patient safety culture, the 2020 National Action Plan to Advance Patient Safety identified culture as one of the foundational areas which are essential to create total systems safety².

Understanding the importance of measuring patient safety culture using a standardized tool to enable comparisons across hospitals, the Agency for Healthcare Research and Quality (AHRQ) funded the development of the Surveys on Patient Safety Culture® (SOPS®) Hospital Survey (HSOPS 1.0)³. The 1.0 survey was released in 2004 to enable hospitals to assess patient safety culture from the perspectives of providers and staff. AHRQ also established a survey benchmarking database, released tools and patient safety culture improvement resources, and offered webinars and in-person survey user meetings to further support hospitals in their efforts to advance and improve patient safety culture.

The HSOPS 1.0 survey has demonstrated reliability and construct validity⁴ and research has established relationships between the HSOPS 1.0 and other important measures. More positive HSOPS 1.0 patient safety culture scores have been linked to a number of positive outcomes, including lower rates of surgical site infections after colon surgery⁵; lower rates of hospital acquired conditions⁶; lower readmission rates⁷; better financial performance/operating margin⁷; higher performance on hospital value-based purchasing metrics⁸; lower rates of adverse events⁹; higher Hospital Consumer Assessment of Healthcare Providers and Systems (HCAHPS) patient experience scores^10,11; and higher Consumer Reports Hospital Safety Scores.¹²

The HSOPS 1.0 survey has been widely adopted in the U.S. and internationally¹³. However, over the years since its initial release, survey users and stakeholders provided AHRQ with feedback about suggested changes to the survey. Under contract with AHRQ, researchers at Westat gathered this feedback along with input from a technical expert panel (TEP) at several key points in the development process along with input from hospital providers and staff, researchers, patients, and other constituents, to help develop and pilot test an updated version of the survey, the SOPS Hospital Survey 2.0 (HSOPS 2.0). The new 2.0 version sought to incorporate the following changes: (1) reduce the number of survey items to shorten the survey; (2) reword complex survey items and survey items that were difficult to translate because of item phrasing; (3) add a “Does not apply or Don’t know” (NA/DK) response option which was included in more recently updated SOPS surveys; (4) revise the Response to Error composite measure to include a focus on learning rather than blame¹⁴ and support for staff involved in patient safety errors¹⁵; (5) revise the staff positions and units/work areas for more specificity; and (6) reduce the number of negatively worded items to improve item comprehension and response as well as the reporting and interpretation of results. In this article, we summarize the development of the 2.0 survey, pilot testing, and psychometric analysis of the survey composite measures and single item measures.

All study procedures were approved by the Institutional Review Board at Westat (FWA 00005551). The study protocol was carried out in accordance with relevant guidelines and regulations.

The version of the HSOPS 2.0 survey pilot tested in the second pilot study in 2018–2019 included 43 survey items measuring 10 unique but conceptually related areas of patient safety culture (Table 1). Composite measures included two or more survey items that measure the same area of patient safety culture. We drafted multiple survey items to assess areas of patient safety culture more comprehensively and to provide sufficient information to make the survey feedback actionable for improvement purposes.

[INSERT Table 1 HERE]

Likert scales were used for draft survey items. This included either 5-point agreement response scales (1 = Strongly disagree to 5 = Strongly agree) or frequency response scales (1 = Never to 5 = Always). A Does not apply or Don’t know (DNA/DK) response option was added to all the composite measure items. The draft survey also included two single item measures: Unit/work area patient safety rating (1 = Poor to 5 = Excellent) and the number of patient safety events reported by the respondent in the past year (1 = None to 5 = Eleven or more).

The final pilot study was conducted from November 2018 to January 2019 in 25 hospitals across 16 states. Pilot hospitals were recruited to vary by multiple characteristics (e.g., bed size, geographic region, ownership). Similar to the initial pilot test, hospitals provided staff lists that indicated the staff position and work area for each person. We randomly assigned providers and staff within units within each hospital to receive a web-based version of either HSOPS 1.0 or HSOPS 2.0 to enable comparisons of scores across the two versions. There were 4,393 respondents for HSOPS 1.0 (39% response rate) and 4,345 respondents for HSOPS 2.0 (38% response rate). A comparison of results on items measuring the same or similar patient safety culture constructs showed that HSOPS 2.0 scores were higher than HSOPS 1.0 scores on most items, with an average of seven percentage points higher. But on a few items, HSOPS 2.0 scores were lower, with an average of four percentage points lower. There was no discernable pattern for when 2.0 scores were higher versus lower than 1.0 scores on crosswalked items. Because score differences between the versions were deemed acceptable, that is, not large enough to cause concern for hospitals adopting the new version, Westat, AHRQ, and the SOPS TEP, decided to proceed with psychometric analysis of the HSOPS 2.0 data from the second and final pilot study.

Psychometric analyses included (1) item analysis and examining missing data patterns, (2) internal consistency reliability, (3) site-level reliability, (4) confirmatory factor analysis, and (5) hospital-level correlations on the survey’s composite measures and single items. Each analysis is described in the sections that follow.

Item Analysis and Missing Data Patterns. We examined respondent-level item frequencies to review the variability of responses and identify items with high percentages of missing data or Does not apply/Don’t know (DNA/DK) responses. Items with low variability are not useful when attempting to differentiate between higher-scoring and lower-scoring hospitals. Items with more than 90 percent of respondents responding positively (e.g., those answering Strongly agree/Agree or Always/Most of the time for positively worded items and Strongly Disagree/Disagree for negatively worded items) were considered to have low variability. If more than 30% of respondents left an item missing or answered DNA/DK, we considered dropping the item since it was not relevant to a large proportion of respondents.

Internal Consistency Reliability Analysis. Internal consistency reliability indicates the extent to which items within a composite measure assess the same construct or area of patient safety culture by assessing how closely those items are correlated. Internal consistency reliability was assessed using Cronbach’s alpha (α), which ranges from 0.00 to 1.00, with higher values indicating greater internal consistency. The minimum criterion for acceptable reliability is an alpha of 0.70¹⁷.

Site-Level Reliability. We calculated site-level reliability to examine the variability of item and composite measure scores among hospitals. Site-level reliability indicates the extent to which responses within the same hospital were more similar to each other than they were to responses from other hospitals. It assesses how well a measure differentiates hospitals by comparing between-hospital variability to within-hospital variability. Site-level reliability was computed using the following formula:

In the formula, is the number of respondents in a given hospital. Site-level reliability for each measure was first calculated for individual hospitals and then averaged across hospitals. Similar to internal consistency reliability, values of 0.70 or higher are considered acceptable for site-level reliability¹⁷.

Confirmatory Factor Analysis (CFA). CFA is used to confirm a particular pattern of relationships among survey items based on past research and theory by assessing how well a proposed factor structure fits the data¹⁸. A CFA was conducted on the proposed composite measures and their associated items. We examined standardized factor loadings for each item on its respective composite measure. Factor loadings above 0.40 indicate that the item’s relationship to the composite measure is acceptable¹⁹.

Several model fit indices were also examined to determine how well the hypothesized factor structure fit the data including chi-square divided by its degrees of freedom (criteria: values less than 5.0²⁰; comparative fit index (CFI) (criteria: values 0.95 or greater²¹), root mean square error of approximation (RMSEA) (criteria: values less than 0.06²²), and the standardized root mean square residual (SRMR) (criteria: values less than 0.08²³). The indices and their criteria are displayed in Table 2.

[INSERT Table 2 HERE]

Hospital-Level Correlations on the Survey Composite Measures and Single Items. Hospital-level percent positive scores were calculated for each item as the percentage of respondents within a hospital who answered positively (e.g., % Strongly agree/Agree or Always/Most of the time for positively worded items, and % Strongly Disagree/Disagree for negatively worded items). Percent positive scores can range from 0 to 100. Hospital-level composite scores were calculated by equally weighting and averaging the percent positive responses for each item included within the proposed composite measures. We examined Spearman’s rank order correlations among the composite measures and single item measures. The survey measures should be correlated because they are designed to assess aspects of patient safety culture. Moderate to moderately high correlations are evidence of convergence among similar concepts.

Of the 11,292 providers and staff invited to participate across 25 pilot hospitals, 4,345 completed the HSOPS 2.0 survey, for an overall response rate of 38%. On average, there were 174 respondents per hospital (range: 29 to 527) with an average response rate across hospitals of 42% (range: 21% to 67%).

The largest proportion of hospital respondents were nursing staff (40%), followed by support staff (19%); other clinical staff (18%); other positions (13%); supervisor, manager, clinical leader, or senior leader (8%); and physician, physician assistant, or resident (3%) (Table 4). The largest percentage of respondents worked primarily in patient care units (30%), while the smallest worked primarily in surgical services (6%).

[INSERT Table 4 HERE]

Table 5 shows the average percent positive, percent missing, and Does not apply/Don’t know (DNA/DK) responses for all survey items at the respondent level. None of the HSOPS 2.0 items were over 90 percent positive (range: 40% to 88%). None of the items had high levels of missing data (range: < 1% to 10%). The percentages of DNA/DK responses ranged from < 1% to 35%. Five items had high percentages of DNA/DK responses (i.e., > 30%). Three of these items were from Handoffs and Information Exchange: “When transferring patients from one unit to another, important information is often left out” (34%); “During shift changes, important patient care information is often left out” (32%); and “During shift changes, there is adequate time to exchange all key patient care information” (34%). The remaining two items were from Reporting Patient Safety Events: “When a mistake is caught and corrected before reaching the patient, how often is this reported?” (35%); and “When a mistake reaches the patient and could have harmed the patient, but did not, how often is this reported?” (35%). Further investigation revealed that most of the respondents who did not answer these items had no interaction with patients, suggesting that the DNA/DK response was being selected appropriately based on item applicability.

[INSERT Table 5 HERE]

The final HSOPS 2.0 survey was the result of extensive development and testing. We obtained survey user feedback; conducted a literature review; conducted multiple rounds of cognitive interviews with 56 respondents; sought input from the SOPS TEP throughout the development process; and conducted two, multi-site pilot studies. Overall, the psychometric properties of the HSOPS 2.0 survey were good for both reliability and construct validity. All but one of the composite measures had acceptable internal consistency reliability, and site-level reliability was also acceptable for all but one composite measure, which was slightly below the criterion. Construct validity was supported by acceptable CFA fit indices and statistically significant, moderate to high hospital-level correlations among most of the measures.

The 2.0 survey includes a new item assessing the extent to which healthcare professionals who have been involved in a patient safety error feel supported, which can be used to identify organizational needs in this important area. AHRQ programs such as the Communication and Optimal Resolution (CANDOR) program^25,26 and the RISE peer support program for caring for the caregiver²⁷ can be implemented by hospitals to address the support needs of healthcare professionals as well as the needs of patients and their family members.

Given that HSOPS 2.0 is a relatively new survey, there has been limited research on its relationship with outcomes. However, one study conducted hospital-level regression analyses with data from 178 hospitals and found that hospitals with more positive patient safety culture scores on HSOPS 2.0 had patients who reported better experiences of care on the CAHPS Hospital Survey²⁸. Hesgrove²⁹ found that provider and staff perceptions of their hospital’s workplace safety culture are significantly related to their perceptions of patient safety culture using HSOPS 2.0. The research also discovered that providers and staff with higher work stress and burnout had worse perceptions of patient safety culture on HSOPS 2.0. While these studies show promising results, more research is needed to examine relationships between HSOPS 2.0 and other important safety and quality outcomes.

In addition to the HSOPS 2.0 core survey, SOPS supplemental items have been developed to measure additional areas of culture related to patient safety and quality. Supplemental items on hospital workplace safety culture for providers and staff assess protection from workplace hazards, moving and lifting patients, workplace aggression from patients or visitors, work stress/burnout, and turnover intentions³⁰. Additional SOPS hospital supplemental measures focus on health information technology and patient safety³¹ and measures of the culture of value, efficiency, and waste reduction³².

Assessing patient safety culture with the HSOPS 2.0 survey can help hospitals address the underlying cultural aspects of their organizations that affect patient safety. As co-lead in the development of the National Action Plan to Advance Patient Safety, AHRQ played a major role in identifying culture as one of the foundational areas for safety. Recommendations in the action plan include ensuring that safety is a core value, assessing capabilities and resources to advance safety, widely sharing information about safety to promote transparency, and implementing competency-based governance and leadership to ensure that board members and senior leaders demonstrate the necessary competencies in safety, equity, and data literacy² (IHI report).

The AHRQ HSOPS 2.0 survey can also enable hospitals to meet external requirements, including Joint Commission Leadership Standards and The Leapfrog Group requirement to regularly assess the culture of patient safety using a validated tool. In addition, the Centers for Medicare and Medicaid Services (CMS) recently introduced an attestation-based quality measure, the Patient Safety Structural Measure (PSSM)³³, in which hospitals report the structural and cultural strategies and practices they have implemented to strengthen their systems for safety. Scores will be publicly available in the hope of driving hospitals to be more proactive and systematic in their approach to patient safety and reducing preventable harm. The PSSM Domain 3, Culture of Safety and Learning Health System, includes an attestation that hospitals conduct a hospital-wide culture of safety survey using a validated instrument annually, or every two years, with pulse surveys on target units during non-survey years. The AHRQ HSOPS survey is mentioned as one of the measures that can be used to fulfill this requirement. The PSSM also stipulates that results should be shared with the governing board and hospital staff and used to inform unit-based interventions to reduce harm (CMS PSSM), emphasizing the importance of following up on the survey results and taking action.

Despite a longstanding, national focus on the importance of patient safety culture and improving patient safety in healthcare, much work remains. Bates³⁴ found that adverse events were identified in almost one in four admissions, and approximately one fourth of the events were preventable, underscoring the importance of continuing to focus on patient safety improvement. In an editorial referring to the Bates study, Berwick³⁵ noted that while senior executives and boards in health care systems may feel overwhelmed by numerous priorities, “first do no harm” remains a sacred obligation, and success requires “constancy of purpose for improvement.” Since patient safety culture serves as the foundation upon which improvement efforts are based, it is critical to assess it regularly using validated instruments and implement improvement initiatives.

The goals for the HSOPS 2.0 survey were to improve upon the original 1.0 survey with a streamlined and updated instrument. The final HSOPS 2.0 survey has 11 fewer items than HSOPS 1.0 and measures similar core concepts as the 1.0 version. The HSOPS 2.0 survey demonstrates good psychometric properties and is a reliable and valid instrument for assessing patient safety culture. Health systems, hospitals, and researchers can use the HSOPS 2.0 to raise awareness about patient safety among hospital providers and staff, assess patient safety culture to identify areas of strength and opportunities for improvement, and to ultimately improve the overall safety and quality of care.