Intervention Implementation:

Evaluating Data and Treatment

Integrity

University or l(f
CINCINNATI
\

This presentation discusses the importance of evaluating data and treatment integrity
in order to determine if the intervention being implemented is effective and to guide
next steps.

1
 Problem Solving Process
(-~:~.=--:-.
: (Is it working?) :
Assessment
(Identify the
I
'
,_ ________ _, I
I
concern)

t
Implementation
(What we are
Planning
l
(Why is it
going to do?) occurring?) « l(i['
°cim::tNNATI

Data collection and evaluation is a critical step in the problem solving process. Data is
how the team will determine if the intervention is working and if it is to a socially
significant level. Depending on the data the team will decide the best next steps
which may be to continue with the intervention as designed, make adjustments to
the intervention, or to begin to fade the intervention.
 Evaluating Data

University or l(f
CINCINNATI
\

The specifics of how to visually analyze data is beyond the scope of this presentation,
but we will begin to discuss key components of evaluating data and the types of data
that help inform decision making. If a concern was identified, resources were
allocated towards an intervention, and addressing the concern efficiently and
effectively is important, then we must know whether the intervention being
implemented is being effective.

3
 Collecting Data: Baseline
• Baseline: The level of the target behavior prior to the
independent variable (IV), or intervention, being implemented
– Basis for determining the effects of the behavior change
program/IV on the target behavior/ dependent variable
– Does not necessarily mean absence of instruction or treatment

• Establishing baseline is important for:

Intervention Setting Initial Maintaining

Planning Reinforcement
Criteria
Objectivity
University or l(f
CINCINNATI

After choosing the target behaviors and an appropriate measurement system, it is

important to collect baseline data. A couple key terms to remember is that the
independent variable is the intervention or behavior change program and the
dependent variable is the target behavior being analyzed. During baseline the
intervention (or independent variable) is not implemented, as to serve as a control
condition. Baseline data allows behavior analysts to determine effectiveness of
intervention by comparing data from baseline phase to intervention phase. I always
make a point to encourage the teacher, RBT, parent, whoever will be the implementer
to continue doing what they have been doing in order to get an accurate idea of level,
trend, and variability prior to implementing the intervention. Baseline data doesn’t
mean that no interventions are in place, but just the new intervention that is going to
be implemented and evaluated. In other words, the teacher or parent is very likely
already using strategies to address the target behavior and those should remain
consistent in the baseline phase and it will likely be helpful in determining if those
strategies are effective. It is best if baseline data is collected prior to discussions of
the new intervention or intervention package, because it is very easy for the
implementer to begin incorporating aspects of the intervention (or what they
anticipate to be the intervention) and you truly just want an accurate picture of the
natural environment.

4
These data are important for planning
interventions (e.g., analyzing antecedent‐behavior‐
consequence correlations), setting initial criteria
for reinforcement (criteria as too high or too low
can impact results), and merits of objective
measurement. Baseline data demonstrate the value and
necessity of behavior change procedures to relevant
stakeholders for them to decide if intervention is warranted.

4
 SMART Goal Setting
What skill are is How is the target skill Is the goal realistic?
being targeted? being measured? (not too hard, or
easy to attain)

Specific Measurable Attainable

Is the goal socially What is the time-

significant? frame for the goal
to be accomplished
(Short vs. long-
term)?
Time-
Relevant
Bound
Additional Information at: http://www.huronhs.com/Downloads/smart-goals.pdf

After establishing baseline rates of the dependent variable (target behaviors) by

collecting a minimum of 3‐5 data points. It is time to make a goal. Using a SMART
goal format, goals should be specific, measurable, attainable, realistic, and time‐
bound.

Goals should be specific, so that they explicitly mention what skill or behavior
(dependent variable) is to be targeted. For example, we may target inappropriate
verbalizations during class, this should have a clear, operational definition.

Additionally, we want the goal to quantifiable. With inappropriate verbalizations, we

could be targeting counting the number of occurrences for a specific class (e.g., social
studies).

This ties in with the goal being attainable. For instance, frequency data was collected
over 5 social studies class periods for inappropriate verbalizations. The median
number was 12 times during baseline for the target student. However, peer
comparisons indicated that students within the class had an average of 1
inappropriate verbalization. Using this information, a short‐term and long‐term goal
can be set. The short term‐goal could be a decrease in inappropriate verbalizations

5
from 12 to 8 at the end of 2 weeks. The long‐term goal could be that the
verbalizations will decrease from 12 to 1 verbalization over 6 weeks. It would be
unrealistic to expect the student’s verbalizations to decrease from 12 to 1 in a week,
or the behavior may be too easy of a goal for the behavior to decrease to 12 to 10
over 6 weeks.

Smart goals should also be relevant. This is often assessed during the identification
and planning phases of problem‐solving when selecting a target behavior.
Inappropriate verbalizations is socially significant to the student, as this is interrupting
his learning in class, as well as the teacher and fellow students as the verbalizations
could be distracting during class. Additionally, a replacement behavior for this
behavior could be hand‐raising.

Lastly, goals should be time‐bound. With high counts of inappropriate verbalizations

(12 per class), it is important to establish both short term and long‐term goals. This is
important for progress monitoring and accountability purposes.

5
 Decision Rules
. 2.J
-✓--.
·n..:..--
Needed to Contributes to Can be based Example: If the
guide objectivity and on time, trend frequency of
evaluator in accountability estimation, or inappropriate
determining combination verbalizations
progress remains above
aim line for three
consecutive data
points, intervention
will be modified.

University or l(f
CINCINNATI

After goal setting, it is important for the team to establish decision rules. Decision
rules are used to guide decision‐making in regard to continuing or modifying the
intervention based on progress monitoring data. These rules are often based on time,
such as, “If the frequency of inappropriate verbalizations does not decrease to 8 in
two weeks, then the team will reconvene to discuss the intervention and modify as
necessary.” Or these rules can be based off trend estimation, or the aim line, for
example, “If the frequency of inappropriate verbalizations remains above the aim line
for 3 consecutive data points, then the team will reconvene to discuss the
intervention and modify, as necessary.” The purpose of the decision rule is to allow
for data‐based decisions along the way and an evaluation of why an intervention may
not be effective so it can be addressed in a timely manner rather than getting to the
end of the school year and everyone being disappointed that the intervention was
not effective. It is okay if something isn’t working, we just need to address it. These
decision rules should also plan for fading of the intervention. For example, “after 3
data points of inappropriate verbalizations at the goal level or below then the team
will meet to discuss fading the intervention.” Typically a decision rule is based on one
of the target variables, but the team will evaluate all of the data when making
decisions.

6
 Graphing Data
Inappropriate Verbalizations
14 Baseline Intervention
Key features:
12
 Time on x axis ~
~ 10
 Measure on y axis g_ Aim Line
~ 8
 Baseline 0 6
~

 Intervention ..0
~ 4
z
 Phase change lines
 Goal line
I 2 3 4 5 6 7 8 9 m ll U Ll M 8 ffi IT IB
 Aim line Sessions

University or l(f
CINCINNATI

Graphing is not covered in this presentation, but in practice and research behavior
analysts use visual analysis to determine if an intervention is effective from baseline
to intervention implementation. In this graph, we can use trend estimation with the
aim line, to determine whether our client is making adequate progress towards their
goal of 1 inappropriate verbalization during class. Here, we can see that the client’s
number of responses is decreasing over time after the intervention was
implemented. It should be noted that this graph is fictious and progress may not look
this clean. Additionally, this graph highlights key features which makes it easier for
stakeholders to understand what behavior change is occurring through visual
presentation. Throughout the program you will learn ways to demonstrate
experimental control, but this is the first step in evaluating whether it appears that
the intervention is being effective in addressing the target behavior.

7
 Treatment Integrity

University or l(f
CINCINNATI
\

The team can design the perfect intervention, but if it isn’t implemented or if it not
implemented as designed then the data on the intervention effectiveness is
irrelevant. Many times a team evaluates data, but it is unclear whether or not the
intervention is just not working or if it hasn’t been implemented with integrity. It is
important to collect data on treatment integrity, so the team knows when evaluating
data if the intervention needs to be modified since it isn’t being effective or if
additional supports need to be provided or modifications made for the intervention
to be implemented as designed.

8
 Treatment Integrity
• Extent to which an
Definition

Importance
ii===== intervention is
implemented as planned

• Increases confidence in
findings
• Adds merit to research
studies

Threats
ii___ • Experimenter Bias
• Treatment Drift

Poor intervention implementation can act as a confound when interpreting results.

We want to rule out poor implementation as a reason for why an intervention is
unsuccessful. High treatment fidelity allows us to assert with confidence that the
independent variable (intervention) is functionally related to the results (e.g., change
in behavior). Treatment integrity, by definition, is the extent to which an intervention
is implemented as planned. This is different than procedural fidelity which is the
extent to which procedures in all conditions (e.g., baseline, intervention), are
implemented correctly. Procedural adherence is another term that is often used
synonymously with procedural fidelity and some suggest that it is a more acceptable
term to intervention implementers.

Furthermore, adequate treatment integrity is important when conducting research as

studies with insufficient treatment integrity data (less than 25% across sessions) may
be excluded from systematic research reviews. Ongoing measurement of
implementation is also recommended when conducting single‐case designs.

There are several threats to treatment integrity which include experimenter bias in
which the intervention is implemented in such as way to skew results between
conditions. Another threat is treatment drift when the implementation of the

9
intervention changes overtime after the onset of the study or intervention
implementation.

9
 Promoting Treatment Integrity

Use precise, Simplify, Provide

operational standardize, training,
definitions and practice and
for steps of automate the feedback
the intervention
intervention procedures

University or l(f
CINCINNATI

Just like definitions for dependent variables, or target behaviors, we want our
intervention steps operationally defined. That is, we want the steps to be clear,
concise, unambiguous and objective. If given an intervention script, we want
implementers to be able to understand what and how to implement the intervention
procedures. Furthermore, this assists observers when collecting treatment fidelity
data as they know exactly what to look for when procedures are implemented .

Research suggests that brief, simple, and easy to implement procedures are more
likely to be implemented with consistency and accuracy than those that are
extremely, or unnecessarily complex. For example, a teacher or parent may be more
likely to implement behavior specific praise initially than a token economy as this
would require less effort and steps involved in implementation. If procedures are
more complex, standardizing procedures can also aid in treatment fidelity. Providing a
script for implementers is an effective way in boosting implementation for
procedures that are more complex, and daunting, by taking the guess work out.
Additionally, implementation can be improved by automating procedures when
possible, this entails using a device, or technology, to delivery the intervention. Here,
we can control for potential variations in implementation, such as tone used or
overall presentation, which could impact delivery. Furthermore, automating some

10
aspects of implementation may not limit desirability of the intervention or impact
practicality or acceptability of intervention (using a pre‐recorded instructions or
models when teaching a specific skill) Providing training and practice opportunities to
implementers also impacts the way in which interventions are delivered. Incorporate
best practice through modeling and talking through intervention steps, allowing
implementers to practice steps, and then providing performance feedback. With
performance feedback, the behavior analyst may either observe the implementer in
vivo or taped, and then assess the treatment integrity but determining if all the steps
were adhered to. The behavior analyst then may provide the implementer feedback
on areas of strength and steps to improve on before the next implementation.
Another practice is self‐monitoring, here, the implementer can then record their own
implementation and rate it on the extent to which the intervention was provided with
integrity, and monitor their own performance.

10
 Assessing Treatment Integrity
Data collected specifically to determine if the intervention is being
implemented as intended
• Examples: Direct observation, adherence checklists, permanent
product review (e.g., video self-monitoring)
Treatment Integrity Ch ecklist

Place a check mark if the teacher completes the step. If the teacher does not perform th e
action, leave the box empty or write . /A.
□ Introduction:
• (Insert child's name), let's watch a v ideo about what we do during circle
time.
• (Insert child 's name), let's look at some pictures about what we do during
circle time.
D Allowed the child to start the video or photos.
D Showed the video/ pictures on the iPad.
D If the child was distracted fo r more than 3 seconds, made a comment about the
video/ photo (i.e. "I see you are s inging') as a redirection (check if needed or
'"TiteN/A).
D Practice Opportunity:
• Now, it is time for circle time and your turn to do what was in the video/
photos.
University or l(f
N umber of steps completed: CINCINNATI
McCoy et al. (2017)

Assessing treatment integrity uses similar behavior assessment methods of those to

measure target behaviors. For example, it is common to use direct observation or
permanent product review. Intervention scripts outlining the intervention steps for
the implementer can easily be modified into a checklist that allows an observer or
the implementer to check off the completed steps of the intervention. You can see a
sample treatment integrity or adherence checklist on this slide.

Additionally, acceptable rates of treatment adherence is roughly 80%. This is

especially important when determining whether to continue or modify an
intervention. Often, if inadequate progress is being made for a client, it is important
to review the treatment integrity to rule out poor implementation as a factor in
inadequate progress.

11
 Reporting Treatment Integrity
Graph data to provide a visual display of intervention effectiveness to
stakeholders and to guide implementation feedback

a11,wlina
NC«~CnaKw&Nrd

.
-
, ... _ . , _ _ Co= ,us.

.
'
...
Darupav.lleh.awiol

~~ .
·---~r--~::
· orl(([
UniverS>ty
CINC
INNATI

Furthermore, visual analysis of implementation can be used to display changes in

adherence relative to student or client performance.

12
 Reporting Treatment Integrity
Graph data to provide a visual display of intervention effectiveness to
stakeholders and to guide implementation feedback

a11,wlina
NC«~CnaKw&Nrd

.
-
, ... _ . , _ _ Co= ,us.

· orl(([
INNATI
.. L..----:----:i--=:..... .
·---~r--~::
UniverS>ty
CINC

In the highlighted baseline phase you see the solid squares represent engagement at
a low level and the solid triangles are disruptive behavior at a moderate level.

13
 Reporting Treatment Integrity
Graph data to provide a visual display of intervention effectiveness to
stakeholders and to guide implementation feedback

a11,wlina
NC«~CnaKw&Nrd

.
-
, ... _ . , _ _ Co= ,us.

.
'
...
Darupav.lleh.awiol

~~ .
·---~r--~::
· orl(([
UniverS>ty
CINC
INNATI

The next phase is the intervention phase where non‐contingent reinforcement and a
choice board intervention package was implemented. Engagement increased and
disruptive behaviors decreased, but not to the goal level and so the team evaluated
potential factors. They co‐plotted the treatment integrity percentages next to the
engagement data points. During this phase the intervention was implemented with
15% and 14% accuracy.

14
 Reporting Treatment Integrity
Graph data to provide a visual display of intervention effectiveness to
stakeholders and to guide implementation feedback

a11,wlina
NC«~CnaKw&Nrd , ... _ . , _ _ Co= ,us.

.
.
'
...
Darupav.lleh.awiol

~~ • * *
CINC
· orl(([
UniverS>tyINNATI

The team determined that the intervention package should be implemented with a
higher level of accuracy before deciding it was ineffective and making adjustments to
the intervention. In the final phase highlighted you see that additional adherence
consultation was provided to the implementer to support a higher level of treatment
integrity. You see that the data demonstrate an immediate increase in the level of
engagement and disruptive behavior drops to 0%. Treatment integrity data are also
reported showing the intervention implementation accuracy increased to 61%‐91%.
This visual display of treatment integrity and how it influenced student performance
is very powerful when making decisions regarding the effectiveness of an
intervention.

Discussions and consultation with an implementer about treatment integrity can be

uncomfortable or challenging‐ especially if you approach the situation
confrontationally. I encourage you to allow the data to guide the discussion by saying
something along the lines of “Looking at this data it appears that performance is
much higher on the days you are able to implement all of the intervention
components.” or “I’ve observed that you are typically able to implement steps 1 and
2 of the intervention script, but not able to implement the rest.” Then you can
approach the situation by asking how the team can support the implementer in

15
increasing the implementation. You can ask questions like, “What is the different
about the days you are able to implement all of the intervention compared to the
other days?”, “Are you comfortable with the remaining steps of the intervention or
would you like to see me model them again?”. The implementer will likely highlight
some significant concerns that need to be addressed in order for the intervention to
be implemented which may include the need for additional training or clarification,
schedule adjustments, additional support, etc. and we shouldn’t just assume that the
implementer is intentionally not implementing the intervention. If the implementer
has had input in the process of developing the intervention along the way then they
will likely feel much more comfortable implementing the intervention and trouble
shooting issues that arise along the way.

15
 Treatment Integrity Overview

How To Improve Treatment Integrity In ABA: Hacking Applied

University or l(f
CINCINNATI
Behavior Analysis (5:15): https://youtu.be/KMGTXXiHxMo
Note: 5th Task List Item H‐6

This short video discusses ways to improve treatment integrity.

16
 Treatment Integrity Overview

University or l(f
How To Improve Treatment Integrity In ABA: Hacking Applied CINCINNATI
Behavior Analysis (5:15): https://youtu.be/KMGTXXiHxMo

17
 Social Validity

University or l(f
CINCINNATI
\

Social validity is the extent to which target behaviors are appropriate, the
intervention is acceptable, and important and significant changes in target and
collateral behaviors are produced. These are key concepts and should be assessed
throughout the problem solving process.

18
 Assessing Social Validity

Behavior Change Goals

Appropriateness of the
Intervention

Intervention Outcomes

University or l(f
CINCINNATI

Assessing social validity is accomplished several ways and is important when

developing goals, interventions, and evaluating the intervention outcomes and client
progress.

19
 Social Validity of Behavior Change Goals

Begin with a clear, descriptive goal

Use experts or individuals who use

the skill in the natural environment

Determine social validity of goals

First, assessing the social validity of behavior change goals starts with a clear,
descriptive goal. Social goals should be precise at all levels. Broad level (e.g.,
improved parenting), categories of behavior hypothesized to be related to the broad
goal (e.g., parenting‐ providing feedback, using time‐out, etc.), and responses that
comprise the behavioral category of interest (e.g., what time‐out procedures look
like).

Selecting a socially valid goal can be accomplished by asking an expert in that area.
For example, if the target behavior is to draw, you may ask an artist on what a goal for
performance would be. Additionally, you can select socially‐valid goals by consulting
those who use the skill in the natural environment. For example, you may consult
with a grocery store employee when determining a vocational goal for community‐
based work.

You can also determine the social validity of goals by assessing performance of those
deemed competent. You don’t want to set a goal that exceeds or falls short of typical
performance. For example, you may want to observe a preschool classroom to
identify typical phrases young children use when entering play‐ they may be different
than phrases you would assume. You can also manipulate levels of performance to

20
determine which produces the optimal results.

20
 ~u•• uu c,..,..p,1111111 ~-ad,ap-111.., t'-n-,...,t ,1,«11JU•boli'l~h4 ntr1111n--11...-.i

Social Validity of
~ - • n d WIO'itl'. l!llllll ID(>Ploal~ _ _..CIC',N(>III or 1... ~Llrllllltyqf 1111~.--t•kln
~•DCe!l..,n- ID IIHJ -•'OlnQ b~,.._.01111f~•l)'>l.-il1.,,.llll-•~ d-111!,n

the Intervention
Acceptability of the
intervention can be
...
....
assessed formally through
surveys and
questionnaires
- Example:
-
ll.Hl,wfll....-.-Ulna l"CICINll,re7

Treatment lt.-mu:hdap . . ll'wl'fCIII""-~

Acceptability Rating
Form (TARF)

Social validity of the intervention, or the acceptability can be assessed formally,

through surveys and questionnaires or informally by asking the implementer or
person of interest their preference for intervention. Additionally, the social validity
can be gauged by the extent to which procedures align with best practice and ethical
guidelines. These findings can then be reported and graphed to relevant stakeholders.

21
 Assessing Social Validity of
Intervention Outcomes

Consumer Expert Normative

Opinion Evaluation Comparison

University or l(f
Standardized Real-world CINCINNATI
Tests Tests

The most common way to assess social validity of an intervention’s outcome or

behavior change is to ask the client or consumer if they think change occurred and if
so, if it was valuable and important. Consumers include the person of interest, as well
as, all relevant stakeholders, such as teachers, related service staff, and caregivers.

Additionally, you can gauge the value of behavior change outcomes by asking an
expert in the field to rate the outcomes. Broadly speaking, you might consult with
educators on outcomes related to academic goals for you client. For example, you
might consult with secondary teachers when determining appropriate note‐taking
techniques. Or, if you are teaching sport skills, such as basketball, you could allow a
professional or college player to review the form of your players to deem if the
technique is appropriate.

Normative comparison is often used to assess socially‐valid outcomes within the

classroom. This is accomplished by determining the amount of time peers are on‐task
compared to the student(s) of interest. For example, a behavior analyst can compare
rates of engagement between one student and a normative sample of peers’
engagement to use to both determine goals and behavior change.

22
Standardized tests, such as academic achievements tests, can also be socially valid
indicators of behavior change. For example, individuals may validate outcomes of a
Curriculum Based Measure (CBM) with high‐stakes achievement tests. Behavior
analysts may also validate outcomes of socio‐emotional learning outcomes using
standardized behavioral rating scales.

Arguably, the most socially valid way to assess behavior change, is to assess the new
behavior within the natural environment. For example, a behavior analyst assess the
validity of teaching their client to initiate conversation by observing the client
demonstrate this behavior with peers on the playground. However, a caveat here is
the observing performance in the natural environment does not necessary mean
there is a mismatch between the target behavior differs from the behavior of
ultimate value in natural situations or the behavior changes observed are situations in
which the behavior is not crucial. For example, we may analyze self‐initiations using a
task analysis, and see the client successfully demonstrate all the steps on the task
analysis, does not necessarily mean the behavior leads to the child being accepted
into play with peers. The peers could still possibly reject the initiation.

In closing, there are a variety of ways to assess social validity and it is best to monitor
in several different ways to make sure we are developing appropriate target
behaviors, acceptable interventions, and socially meaningful changes are made in
target and replacement behaviors.

22