Desktop version

Home arrow Engineering arrow Behavioral Intervention Research: Designing, Evaluating, and Implementing

FIDELITY CONSIDERATIONS ALONG THE INTERVENTION PIPELINE

The specific aspects of fidelity that should be considered may vary depending upon the particular phase of an intervention along the pipeline (see Chapter 2). Nevertheless, there are neither documented best practices nor evidence concerning the preferred approach to fidelity at each development, evaluation, or implementation phases. Regardless, it seems reasonable to suggest that some attention be conferred to fidelity when developing and evaluating an intervention, possibly as early as in Discovery and Phase I (selection of a theory base, treatment elements, and delivery characteristics, determining feasibility) and Phase II (pilot testing, evaluation of outcomes and effect sizes). Developing a plan for enhancing and monitoring fidelity and evaluating the feasibility of this plan, as well as the validity of fidelity measures, would best occur in tandem with identifying the essential ingredients of an intervention protocol. Furthermore, attending to fidelity early on would provide some measure of confidence that the intervention is worthy of advancing. For example, examining whether interventionists are able to learn a complex intervention and adhere to its delivery, or that participants adequately receive intervention components, would provide important preliminary evidence for validating the value of the approach and also allow for “real-time” changes to be made early on in the intervention development process. Nevertheless, although there may be much value in attending to fidelity during the initial stages of intervention development, practical limitations such as inadequate funding, minimal staffing, and limited resources may make this the ideal rather than a reality

In the evaluation phases, particularly in an efficacy trial (Phase III), the role of fidelity is clear. It is to assure that an intervention is implemented per protocol and that adherence to the established study design features and procedures is achieved. The goal of a fidelity plan in this phase is to maximize internal validity by minimizing “noise” from external sources such as differences in interventionists’ backgrounds and approaches to delivering an intervention.

The importance of measuring and monitoring fidelity in an efficacy trial cannot be overstated. Poor fidelity can have a critical impact on the interpretation of results at any developmental phase of an intervention, but particularly in a definitive efficacy study. Without proper documentation and/or measurement of fidelity, it is not possible to evaluate whether inconsistent, ambiguous, or unsuccessful outcomes from a trial reflect a failure of the intervention or failure to implement it as intended (Chen, 1990; Hohmann & Shear, 2002). Failed implementation is the most common reason for the lack of positive outcomes (Mills & Ragan, 2000). This is aptly illustrated by early psychotherapy research in which therapists did not always adhere to treatment techniques yielding studies with inconsistent outcomes (Bond et al., 2000).

Another way to understand the impact of failed implementation in an efficacy trial is through the lens of Type I and Type II errors. The lack of adequate fidelity monitoring and evaluation may lead to the risk of a Type I error or accepting positive outcomes as a signal that the intervention works when unknown contaminants may actually be responsible for the desired effects (Hohmann & Shear, 2002; Spillane et al., 2007). As such, undetected errors in delivery can result in positive results for ineffective treatments, yet such treatments may not be replicable. Alternately, if results are not significant, researchers run the risk of a Type II error by erroneously rejecting a treatment that may have been poorly or inconsistently implemented.

Fidelity is also important in effectiveness studies (Phase IV). In Phase IV evaluations, the focus is on testing an intervention in different settings and with diverse populations that may differ from those included in the original efficacy trial. The concern at this evaluative phase is with external validity or being able to generalize the intervention to a broader arena including other settings and populations. With regard to fidelity, the focus is with obtaining a balance between streamlining or modifying the intervention to better fit the practice context and maintaining the integrity of the intervention for which it was originally designed and tested. Maintaining fidelity to the identified core immutable principles and features of an intervention is essential; however, the demands of a delivery context may require that adaptations to the intervention be made. A balance between being flexible and maintaining treatment integrity can be difficult to achieve and, to date, there is no consensus as to how much flexibility or deviation from a protocol can be allowed or for what aspects of the protocol. As to the latter, it may be that changing dose, visit schedule, or level of expertise of an interventionist is required to enable the intervention to be embedded into a practice setting. Yet, the effect of such changes may not be well understood or previously evaluated in earlier phases of the intervention’s development. It may be that adapting interventions leads to better and more appropriate adoption and reach; yet, an adapted intervention may then need to be submitted to further rigorous testing if it is transformed too much from its original form (Washington et al., 2014). Figure 12.1 summarizes the role of fidelity in the different evaluative phases of a behavioral intervention and the tension between the demands for internal and external validity.

In implementation and sustainability phases, the emphasis of fidelity is on the accurate replication of an intervention and identifying the barriers to, and facilitators of, implementation integrity within a delivery context. Without attention to fidelity in previous evaluative phases and the in-depth knowledge of how an intervention has been delivered, received, and enacted, it is not possible to replicate or generalize to other settings (Bass & Judge, 2010). Further, the effectiveness of scaling up and rolling out an intervention will depend upon the ability of other practice sites to replicate the original intervention or make adaptations to fit their context. In this respect, standardization of the intervention is critical (see Chapter 6 on standardization).

Although assuring fidelity and accurate replication of an intervention is essential in the translation and implementation phases, efforts to do so are challenging. The fidelity approach used in the evaluation phases may need to be streamlined to reflect the realities and resources of the practice setting. For example, whereas monitoring and rating treatment sessions via audio or video for adherence are a

Role of fidelity in development, testing, and implementation phases

Figure 12.1 Role of fidelity in development, testing, and implementation phases.

common fidelity practice in an efficacy trial, these are impractical in a translation phase. Clinical sessions are not typically audiotaped, and in the few cases where this is possible, neither agencies nor clinical personnel typically have the time to review and code recordings to assess fidelity

Consider the case of the Skills2CareR program. This intervention was initially tested as part of the NIH REACH I initiative. In this efficacy trial, audiotapes of 10% of intervention sessions were listened to and rated by two research staff along various dimensions (Gitlin et al., 2003) using a monitoring form similar to the one shown in Table 12.2. A score was derived reflecting the level of adherence achieved for that session; also, strengths and concerns were documented, and then shared with interventionists in one-on-one supervisory sessions in order to provide course corrections early in the trial. Subsequently, common errors detected across interventionists were discussed at group supervisory sessions. Reasons for their occurrences were explored, and solutions were derived and documented.

However, when the Skills2CareR program was translated and implemented in busy home care practices (Gitlin, Jacobs, & Earland, 2010), this approach was not feasible. Other approaches had to be adopted, such as having supervisors review session-by-session checklists that documented the elements delivered in sessions. Indicators of fidelity were subsequently built into the documentation of each contact with clients, and checklists were developed for ease of use by clinical supervisors. Understanding the challenges that present at the implementation phases can inform the development of fidelity strategies and measures early on in the process of building an intervention. Integrating fidelity processes into the intervention protocol may help researchers avoid the need to reconfigure fidelity approaches in the latter phases of rolling out an intervention.

The role of fidelity in dissemination and then in the maintenance or the sustainability phase of an intervention is unknown. In disseminating a proven program, of importance is to specify what can and what cannot be modified in terms of treatment delivery characteristics. As to sustainability, it is unclear as to what constitutes an effective fidelity plan when an intervention is fully integrated and being maintained in a delivery setting. The goal of sustainability is to normalize an intervention in a particular setting such that it becomes habituated and part of everyday practice. Doing so, however, presents a new set of challenges for assuring that ongoing implementation is within the parameters of the original intervention. When seeking sustainability, fidelity needs to be aligned with quality indicators, supervisory structures, and an organization’s quality control procedures to assure ongoing adherence. To normalize an intervention in practice, fidelity monitoring needs to be built into the expectation and operations of that delivery context.

To summarize our discussion thus far, attention to fidelity is critical at every phase of the pipeline and doing so confers important advantages. First, monitoring and measuring fidelity enable errors to be detected and course corrections to be instituted. Second, fidelity monitoring can improve consistency in the delivery of the treatment and prevent drift, omissions (e.g., omitting a particular treatment element), and co-missions (e.g., augmenting an intervention with a new treatment element) or their co-occurrence. Third, attending to fidelity helps to further define and refine an intervention.

Interventionist: Date of Session: Reviewer: Date Reviewed:

Effectively Met (2)

Partially Met (1)

Not Met (0)

N/A

Comments

Preparedness

  • 1. Did interventionist appropriately greet the caregiver and the person with dementia?
  • 2. Did interventionist accurately describe the Tailored Activity Program and the specific purpose of Session 1?
  • 3. Did interventionist have all necessary materials available?

a. Documentation binder

b. Education materials

c. Assessment tools

  • 4a. Was interventionist able to answer questions asked by caregiver concerning the session and/or research study?
  • 4b. If not, did he or she indicate that he or she would check with supervisor and get back to respondent within a day or two?

Professionalism

  • 1. Did interventionist speak in a clear audible voice?
  • 2. Was the volume of interventionist's voice appropriate?
  • 3. Did interventionist use a calm voice?

Interventionist: Date of Session: Reviewer: Date Reviewed:

Effectively Met (2)

Partially Met (1)

Not Met (0)

N/A

Comments

  • 4. Was interventionist polite?
  • 5. Did it appear that sufficient rapport was established between interventionist and caregiver/person with dementia?
  • 6. Were interruptions and/or other unexpected occurrences handled professionally?
  • 7. Was the caregiver/person with dementia treated with respect and his or her wishes adhered to?
  • 8. Did interventionist use nonmedical and nontechnical language?
  • 9. Did interventionist express confidence and enthusiasm in the intervention?

Flow and Compliance: Did the interventionist . . .

  • 1. Demonstrate knowledge of intervention goals and objectives?
  • 2. Clearly introduce session goals?
  • 3. Stay focused on session goals?
  • 4. Indicate the intervention includes eight sessions over 3 months, discussing flexibility of meeting schedule?

Interventionist: Date of Session: Reviewer: Date Reviewed:

Effectively Met (2)

Partially Met (1)

Not Met (0)

N/A

Comments

  • 5. Actively engage caregiver in telling his or her story (e.g., asking what a typical day is like)?
  • 6. Confirm and discuss behaviors identified in interview and/or new or upsetting behaviors?
  • 7. Periodically ask caregiver if he

or she had any questions or understood the purpose of the session and/ or points being made?

  • 8. Refer to information caregiver provided during interview?
  • 9. Provide and review educational materials and discuss in context of each target behavior?
  • 10. Obtain closure for the session (e.g., reviews what was accomplished, sets up next session, provides homework, reviews strategies to try)?

Score:

Summary of strengths, concerns: Action plan:

 
Source
< Prev   CONTENTS   Source   Next >

Related topics