Theories, Models, and Frameworks (TMFs) in implementation science (IS) are a critical means through which we can describe and assess the complex implementation process.^1–3 When appropriately selected and operationalized, TMFs can ensure that we are addressing the most important implementation questions and considering all pieces of the implementation puzzle including the intervention, implementation strategies, context, and key outcomes.⁴ There is an increasing appreciation for understanding how interventions and implementation strategies need to align with the often dynamically changing implementation context and how they impact the most relevant implementation outcomes when assessing success.^6,7 Additional questions also remain: how do different elements or dimensions of implementation context interact with each other, what is the relationship among key implementation outcomes, and how do context domains and implementation outcomes relate.^8,9 Few studies have explored these relationships quantitatively^10–12 and we have not found any publications assessing these associations longitudinally and across multiple, diverse programs.

Much of the existing literature discusses context domains and implementation outcomes separately. For example, Nilsen and Bernhardsson (2019) conducted a scoping review of 17 determinant frameworks to explore how “context” is operationalized in implementation science and identified common context dimensions but noted considerable inconsistency in terminology and definitions across TMFs.¹³

Other scholars have developed meta-frameworks to integrate contextual domains from multiple TMFs,¹⁴ explore how one key domain is operationalized across multiple TMFs,¹⁵ or provide guidance on how to conduct contextual analysis.^16,17 In parallel, Proctor and colleagues coined definitions for key implementation outcomes¹⁸ and in a more recent publication discussed nuances associated with their operationalization.⁹ Other work has explored how implementation outcomes across different TMFs can be cross walked to support comparison and synthesis across studies.¹⁹ There remains a critical gap in our understanding of the dynamic relationships between context and outcomes, and how these connections influence the effectiveness, sustainability, and equity of implementation efforts.

To explore the above outlined questions about complex relationships, it makes sense to look at data from studies that used one of these TMFs (or a combination of TMFs that involve both context and outcomes). One such example is the Practical, Robust Implementation and Sustainability Model (PRISM) which includes both multilevel and multi-perspective consideration of the implementation context through the PRISM context domains as well as a set of implementation (RE-AIM) outcomes.^8,22,23 PRISM was developed to address specific contextual factors related to successful adoption, implementation, and sustainability. PRISM includes RE-AIM (Reach, Effectiveness, Adoption, Implementation, and Maintenance) outcomes and is a pragmatic, contextual expansion of the widely used RE-AIM framework.^8,22–24 It focuses on four contextual factors that influence RE-AIM implementation outcomes. As shown in Fig. 1, these factors are: organizational and patient perspectives on the intervention; organizational and patient characteristics; external environment factors (e.g., reimbursement policies, reporting guidelines), and the implementation and sustainability infrastructure (e.g., resources, aligned organization priorities, audit and feedback processes). While PRISM was created on the premise that context predicts or is associated with implementation and effectiveness outcomes, this presumed relationship has not been formally tested to our knowledge.

The purposes of this paper are to: (1) describe and summarize data collected on PRISM contextual domains and RE-AIM outcome variables; (2) investigate specific hypotheses concerning relationships among and between contextual and outcome constructs; (3) examine implications for refining a standard set of assessment items for the Iterative PRISM (iPRISM) Webtool²⁶ to enhance their pragmatic utility; and (4) recommend directions for future research on PRISM and related frameworks. Specific hypotheses and research questions were defined a priori and are detailed in the Methods section and Table 3. The overarching aim was to deepen understanding of how PRISM domains interact. This will inform both the refinement of the iPRISM Webtool and the development of a standardized, pragmatic set of items to operationalize PRISM constructs and outcomes.

Data were collected using the iPRISM Webtool,^24,26 which is freely and publicly available for researchers and implementers to use. The iPRISM Webtool was developed to increase accessibility of PRISM, thus is available in English and Spanish and is designed to be used by a broad audience, including those without IS expertise. The electronic format facilitates real time prompts and guides users through the process of applying PRISM to their projects and includes embedded education. It can be used by individuals or teams, with each member providing their independent perspective on the PRISM factors. When used as a team, individual responses are collated and summary feedback provided.

The iPRISM Webtool operationalizes PRISM using 21 assessment questions that are mapped on the PRISM contextual domains and RE-AIM outcomes (see Additional File 1). It is designed to be used across all phases of a project. Each assessment question is rated on a 6-point sliding Likert scale anchored by the terms from 1= ‘not at all likely’ to 6= ‘very likely’ for RE-AIM items and from 1= ‘not aligned’ to 6= ‘very aligned’ to the question ‘How aligned is your program with (PRISM characteristic)?’ and has an option to select not applicable. Except for Implementation, all of RE-AIM dimension assessment questions have both a general question and a question to assess representativeness or equity on that dimension. Users can decide whether to stop after completion of the assessment questions and feedback; or continue with a section that guides users through the process of identifying and prioritizing implementation strategies (adaptations) to optimize the contextual alignment and/or enhance the impact of their project on RE-AIM outcomes. All responses and data entered in iPRISM are anonymous, but general characteristics about the project and person completing it are optional fields.

This study was classified as exempt by the Colorado Multiple Institutional Review Board under protocol number 23–0179.

We developed four hypotheses and one general research question involving comparisons of average scores on RE-AIM dimensions and alignment scores on PRISM domains.

Ho 1: The mean value on the Maintenance dimension will be lower than the other RE-AIM dimensions (i.e., Reach, Effectiveness, Adoption, Implementation). Rationale: Maintenance is contingent on success at earlier stages and notoriously difficult to achieve.

Ho 2: The mean score on Implementation and Sustainability Infrastructure (ISI) will be lower than other PRISM context domains. Rationale: Resources are frequently limited, especially in low-resource settings and few organizations have robust procedures for audit and feedback.

Ho 3: Among RE-AIM dimensions, mean scores for the Reach and Adoption dimensions will correlate most highly with each other; and Reach and Effectiveness the lowest. Rationale: Definitions of Reach and Adoption are the same, just at different levels. Often actions taken to maximize effectiveness can limit reach, and vice versa.

Ho 4: Relationships among the three components of Implementation: fidelity, adaptation, and costs will be only modestly correlated (e.g., 0.3 or lower). Rationale: These are different constructs and are not expected to be strongly related, but conceptually all are part of the multifaceted construct of implementation.

Exploratory Research Question 1: What are the correlations among PRISM context measures? We believe this is an important issue to explore that could advance our understanding and assessments.

Ho 5: Scores on the representativeness/equity items for each RE-AIM dimension (e.g. Reach) will be lower than the general item for that dimension. Rationale: It is more challenging to produce results that are similar across all groups given societal, structural and social needs issues,^27–29 than to produce a good overall (average) outcome.

Ho 6: There will be a high correlation between equity and general scores- (r = .6 or higher) on each RE-AIM dimension. Rationale: As the questions relate to the same construct (e.g., Reach) they should be strongly related, although the mean level on the equity items are hypothesized to be lower (Ho 5).

Ho7 : Responses to the RE-AIM equity items will be significantly related to each other. Rationale: a) Programs that produce differential results across one dimension will be likely to also do so on other issues and b) respondents are likely to conceptualize equity as a global issue.

Ho 8: Organizational and recipient perspectives on the intervention/implementation strategy will be most strongly related to Adoption and Maintenance among the RE-AIM dimensions. Rationale: Experiences of a) organizations and b) individual recipients with similar programs in the past are likely to heavily influence adoption decisions and propensity to continue a program after research support is no longer available.

Ho 9: Patient (recipient/participant) Characteristics will be more strongly related to Reach than other RE-AIM dimensions. Rationale: Individual characteristics include social and structural factors that can heavily influence willingness to participate in a program.

Ho 10: Implementation and Sustainability Infrastructure (ISI) will be more strongly related to Implementation and Maintenance than other RE-AIM outcome dimensions. Rationale: As its title implies, the ISI domain was developed to include factors presumed to strongly impact these RE-AIM outcomes.

Ho 11: External Environment scores will most highly correlate with Adoption and Maintenance among the RE-AIM outcome dimensions. Rationale: Higher level policy, structural and incentive issues included in the External Environment domain of PRISM should strongly influence organizational Adoption and Maintenance decisions.

Exploratory Research Question 2: Do implementers and front-line staff have lower ratings (less positive) than Investigators or Supervisors? Rationale: This question has implications for future implementation, but we do not have specific hypotheses.

Exploratory Research Question 3: How much congruence is there among team members? Rationale: This has implications for future implementation research and practice, including use of the team function on the iPRISM webtool.

To our knowledge, this was the largest study to date evaluating assessment items related to PRISM and the first to assess the relationship between contextual and outcome constructs of an IS TMF. These data from the sizable and moderately diverse sample of implementation and health services researchers and practitioners in this report provides initial norms for the PRISM assessment items. These questions (Appendix) can be used via the iPRISM Webtool or separately. This evaluation serves to not only refine a standardized set of assessment questions that operationalize one TMF; the findings can also be used to inform other IS TMFs and implementation science approaches that aim to pragmatically improve contextual fit and optimize equitable outcomes of evidence-based programs. While TMFs are used in the majority of IS papers and grants, they are seldom if ever evaluated to understand the complex interactions between the different elements of implementation context and outcomes.^1,10,13,30 This evaluation represents one such effort.

The scope of this study was limited to PRISM-defined categories of context and its RE-AIM implementation outcomes. These factors may be labeled differently across other IS TMFs, but many constructs are similar and can be cross walked.^2,19 Although PRISM is a framework, not a formal theory, there is sufficient information about its purpose, development and use to justify forming hypotheses.. Further research is needed to understand the complex relationships between the dynamically changing context and outcomes in real world settings.

There were some key findings related to the RE-AIM equity specific questions. As predicted, ratings on the RE-AIM equity items were consistently lower than the general RE-AIM ratings and there were moderate to high correlations between the equity components and the general dimensions. Given these results, we considered whether the equity items on the iPRISM Webtool add sufficient value that they should be retained. Although the differences were small, the equity items did significantly differ from their respective general RE-AIM dimension score, and the correlations of .55 to .66 only account for 35–43% of the variance in equity ratings. Our decision, pending further data, is to allow programs to decide whether. to include the equity questions, but to recommend they do.

This study also explored the relationship between PRISM contextual domains and implementation outcomes.^8,22 Two of four hypotheses about contextual factors were moderately supported (including multi-part predictions summarized in Table 3). As predicted, ISI ratings were strongly related to the RE-AM Implementation and Maintenance outcome dimensions. We also hypothesized PRISM’s Recipient Characteristics would be most strongly correlated with Reach but found that it was most correlated with Effectiveness (r = 0.51), followed by Reach (r = 0.48). The two hypotheses that were not supported were that 1) the two PRISM ‘perspective scores’ (Organizational and Recipient) would be more highly related to Adoption and Maintenance than other RE-AIM outcomes; and 2) External Environment ratings would be most highly related to Adoption and Maintenance.

Overall, the correlations between ratings of contextual alignment and RE-AIM outcomes were moderate. Conceptually, these correlations would not be predicted to be much higher due to other key factors (moderators and mediators) of different problems being addressed, implementation strategies used, types of interventions, and different settings and populations. In addition, having only 1–2 items per construct substantially diminishes the chances of finding strong relationships with a given construct.

It is notable that responses a) within the RE-AIM outcome dimensions and b) within the PRISM context domains were moderately correlated, the PRISM context ratings more so than the RE-AIM ones. Although this was not a specific prediction, it makes sense given that neither RE-AIM outcome nor PRISM context constructs were ever conceptualized to be independent.^22,24,31 The PRISM context ratings were all significantly correlated, with the strongest correlations between the Perspectives and Characteristics within both levels of the Recipients (r = 0.71) and the Organization (r = 0.74). We considered whether to keep the Perspective and Characteristic items as separate assessment questions, and whether to reword them, given these high correlations. We recommend keeping them as separate ratings for now, and to qualitatively assess respondents’ thought processes in answering these questions to make a more informed decision. In other work evaluating a set of 29 items assessing the PRISM context questions, there was moderate internal consistency within the 3–6 items comprising each of the PRISM domains.³³

Our hypotheses concerning relationships among the RE-AIM outcome dimensions were partially supported. We hypothesized that the correlation between Reach and Effectiveness would be the lowest based on our experience with ‘objective data’ in prior research,³⁴ but this was not what we found. It is possible that in the absence of more detailed information on the definition of different PRISM and RE-AIM terms that respondents have a general impression about the program they are implementing and do not differentiate sharply among the different PRISM factors. The components of implementation – cost, adaptation and fidelity were significantly correlated, perhaps because they may be perceived as related, given they are part of the same overarching outcome dimension. In future iterations of the PRISM assessment items, we will consider collapsing the 3-part implementation items into one to improve pragmatism. As predicted, mean ratings for Maintenance were lower than the composite of the other four RE-AIM dimensions; this may be in part due to most respondents being in the planning stage of their project or that sustainability of the project was not prioritized.

Although we did not articulate a specific research question about ratings across project phases a priori, these analyses produced very clear and consistent findings. Mean ratings across all constructs were lower for the Implementation (program delivery) phase than either Planning or Sustainment. This is likely because of the high probability of encountering numerous barriers to program delivery during the Implementation phase.³⁵ In the future, with larger sample sizes we will evaluate the relationship between the PRISM context factors and RE-AIM outcomes within each project phase.

We found that individual team members responded differently to the assessment items. This lack of agreement across team members does not necessarily imply low reliability but is likely a reflection of different individuals observing different issues and outcomes. These differences among individuals within teams support the importance of partner engagement across levels and seeking diverse representation.^27–29 These findings also support the team-based option of the iPRISM Webtool.

We conclude that this relatively brief set of questions assessing many components of PRISM is an appropriate, relatively brief, pragmatic set of items for the purposes of the Iterative PRISM webtool. Prospective use of the iPRISM Webtool in a large project with a standard intervention and objective data on at least some RE-AIM outcomes (e.g., participation rate, implementation fidelity, heterogeneity of outcomes), would facilitate understanding of the usefulness of these subjective ratings. Research is needed on the use of the webtool over time; The incremental value of the equity-specific RE-AIM items and the three component Implementation items deserve further study.

Finally, there is a need for continual evaluation of IS TMFs, including PRISM, to understand the complex relationships between implementation contexts and outcomes. Such evaluations are important to enhance pragmatism of IS efforts to align with context.³⁹ They can also help to study the importance of assessing all constructs in a TMF and if some constructs that may be more essential than others. Enhancing the pragmatism of IS TMFs, including brief standardized assessment items that operationalize TMFs can assist in making IS methods more accessible for broad use.

In conclusion, we encourage replication of our findings and further research to evaluate IS TMFs and pragmatic assessment tools to operationalize TMFs for both practice and research. Further investigation of the relationship of PRISM and other TMF contextual factors to implementation and clinical effectiveness outcomes can help advance the field.