Chapter 50 * The Assessment of Executed Policy Solutions 341

50
The Assessment of Executed
Policy Solutions
Stella Z. Theodoulou and C. Kofinis

Concerns over accountability, efficiency, and makers became more concerned with -whether
effectiveness in policymaking are intensifying at the different welfare and anti poverty programs
all levels. Elected officials, policy makers, commu- were having the effect they were supposed to and
nity leaders, bureaucrats, and the public at large whether tax dollars were being spent efficiently
want to know what policies work and what ones and effectively, From the late 1960s, requirements
are not working, and why. More often than not for program evaluation were written into almost
policies and the programs they create often fail to all federal programs.!
“achieve their intended effects and have unintended Subsequently, Congress established evalu-
outcomes. The purpose of evaluation is to deter- ating organizations such as the Congressional
-mine whether an implemented program is doing Budget Office and The General Accounting Office.
_ what it is supposed to. If it is, then evaluation will Funding support for such organizations has varied
assess how well it is achieving its intended objec- with each subsequent presidential administration.
tives and goals. If the program or policy is not . For example, the Reagan administration made
performing well, evaluation will determine what deep budgetary cuts that affected the federal gov-
effects the program is having. Through evaluation ernment’s ability to conduct program evaluation.’
we can determine whether a policy's effects are _ In this chapter we will identify the differ-
intended or unintended and whether the results ent types and approaches to evaluation, discuss
are positive or negative benefits for the target the process of evaluation, identify who carries
population and society as a whole. out evaluation, and the obstacles that face evalu-
Over the years evaluation has become more ators. The chapter’s objective is to offer readers a
common place, although it is not new for govern- brief, simple, and clear introduction to program or
ments to assess whether their programs are cost policy evaluation. It is not intended to point the
effective and are achieving desired benefits. For reader to any one approach or type of evaluation
example, in the Roman Empire, it was common over another. Rather the reader should understand
to alter tax policies in response to fluctuations in that the best evaluation is the one which meets
revenues. This was an early form of policy evalu- the needs of the program or policy that is being
ation. American policy makers became more con- evaluated. We begin by distinguishing program
cerned with judging the effects of policies in the evaluation from other related forms of assessment
1960s with the advent of the War on Poverty pro- activity.
grams. From this point in time American policy

From Stella Z. Theodoulou and Chris Kofinis, The Art of the Game: Understanding American Public Policy Making
(Belmont: Wadsworth, 2001).
i+
- 342 «Part Five * Making Public Policy

What Is Policy Evaluation? or policy evaluation is that it allows for account-

ability to be measured empirically. Conducting an
Policy evaluation consists of reviewing an im-
evaluation allows policy makers to be provided
plemented policy to see if it is doing what was
with accurate information on key policy ques-
amandated. The consequences of such policies pro-
tions that arise from the implementation of any
grams are determined by describing their impacts,
policy or program. Such information is of course
or by looking at whether they have succeeded or
provided by an evaluation study within a given
failed according to a set of established standards. 3
set of real world constraints, such as time, bud-
Within the field of public policy a number of per-
get, ethical considerations, and policy restric-
spectives as to what evaluation exactly is can be
tions. The usefulness of conducting an evaluation
found. The first perspective defines’ evaluation as study of a program or policy is that it provides
the assessment of whether a set of activities im- information to policy makers on whether the pol-
plemented under a specific policy has achieved a
icy or program in question is achieving its stated
given set of objectives. Thus, overall policy effec-
goal and at what costs these are being achieved.
tiveness is assessed,’ A second perspective defines
The effects of a program or policy will also be
evaluation as any éffort that renders a judgement
ascertained and if the evaluation is conducted
about program quality.’ The third perspective de-
correctly policy makers will be able to deter-
fines evaluation as information gathering for the mine whether those effects are intended or un-
purposes off making decisions about the future of
intended. Of course, policy makers want to know
a program. fA final perspective found in the lit-
if programs are being administered and managed
erature views evaluation as the use of scientific in the most efficient, accountable, and effective
methods to determine how successful implemen- manner. An evaluation study can determine if this
tation and its outcomes have been.’
is true or not. Evaluation is also useful because *
The General Accounting Office (GAO)
it can eventually stimulate change. Finally, the _
defines program evaluation as the provision of
utility of conducting an evaluation is that it can
sound information about what programs are ac-
discover flaws in a program that policy designers
tually delivering, how they are managed, and the were never aware of in the abstract.
extent to which they are cost-effective.’ For our
purposes none of these definitions are necessar-
ily unacceptable. We believe policy evaluation can TYPES OF POLICY EVALUATION
be better defined as a process by which general
judgments about quality, goal attainment, program There are a variety of models or frameworks that
effectiveness, impact, and costs can be determined. fuse theoretical content with practical guidelines
What differentiates policy evaluation from for conducting a program or policy evaluation.
other informal types of assessment is, first, its Most models arose in the 1960s and 1970s and
focus on outcomes or consequences.” Next, were early attempts to conceptualize what evalu-
ation was and how it should be conducted. Thus,
evaluation is done post implementation. In other
words the program must have been implemented they offer varying understandings as to the goals
for a certain period of time. Third, the goals of of an evaluation, the role of the evaluator, the
the policy or program are provided to the evalua- scope of an evaluation, as well as how it is or-
tors. The main purpose of evaluation is to gather ganized and conducted. Subsequent practitio-
information about a particular program’s perfor- ners have taken the models and adapted them to
mance so as to assist in the decision to continue, changing times, contexts, and needs. Often two
change, or terminate. or more models will be used in conjunction with
each other. The result is that there are several
different types of evaluation models that vary in
The Usefulness of Evaluation
complexity.” There are, however, four types that
One way that programs or policies may be as- are most commonly applied: Process Evaluation,
sessed in terms of their accountability is through Outcome Evaluation, Impact evaluation, and
_ formal evaluation. Thus, the real value of program Cost-Benefit Analysis.
 a
© wy

Chapter 50 * The Assessment of Executed Policy Solutions 343

PROCESS EVALUATION. This type focuses on intended target population. The major difference
the concrete concerns of program implementa- between an impact evaluation and an outcome
tion. It assesses how a program or policy is being evaluation is that the latter are solely concerned
delivered to target populations or how it is being with whether the program or policy's goals and
managed and run by administrators. A process objectives are being achieved. In comparison the
evaluation should address the following: impact evaluation is concerned with assessing
whether the target population is being affected
* determine why a program or policy is per-
in any way by the introduction and implemen-
forming at current levels. tation of the policy. There is also concern with
* identify any problems
the impact of the program on the original prob-
* develop solutions to the problems
lem being addressed. The benefit of an impact
‘improve program performance by
evaluation is it is suited to the needs of both pro-
recommending
gram level managers and policy designers, for it
* how solutions should be implemented and -
is important for both to ascertain whether target
evaluat once carried out.
populations are appropriately receiving delivery
With this type of evaluation the focus is not on of a program. A successful impact evaluation
whether the program is meeting specified goals, _ must help to identify the following:
but is solely to develop recommendations to im-
* theoretical goals of the program/policy
prove to implementation procedures. This type of
* the actual goals
evaluation is best suited to the needs of program
managers and has the objective of helping man- .
* the program or policy objectives
agers overcome barriers to achieving the goals of * program or policy results and whether they
“are intended, unintended, positive or negative
the program policy being implemented.
in effect. °
OUTCOME EVALUATION. This type focuses
on the degree to which a policy is achieving its - COST-BENEFIT ANALYSIS. This type of evalua-
intended objectives with regards to the target. tion focuses on calculating the net balance of the —
population, It is concerned with outputs and benefits and costs of a program. Essentially, cost
whether the policy is producing the intended benefit analysis is a method with which to evalu-
results. This can lead to assessment of effective- ate and assess the effectiveness of a policy's costs,
ness, including cost. Outcome evaluation is not benefits, and outcomes. Evaluators identify and
well suited to the needs of program level man- quantify both the negative costs and positive bene-
agers because it does not provide operational fits in order to determine the net benefit. For many
guidelines on how to improve the implementa- it is a controversial evaluation technique because
tion of the program. Rather it is best suited to of the difficulty of applying it to the public sector.
the needs of policy designers because it identifies It ignores qualitative concerns at the expense of
whether there is consistency between policy out- quantitative information. For example, if we take a
puts and program intent. An outcome evaluation cost benefit analysis approach to assessing certain
must determine the following: policy issue areas, it is sometimes easier in certain
issue areas to calculate the immediate real dollar
* legislative intent costs then the tangible benefits. For certain types
* program goals of programs such as education or the environ-
* program elements and indicators ment, one could argue that the real benefits do
* measures of indicators not materialize for years or decades. Hence a cost
* program outcomes and outcome valences benefit analysis may evaluate a program for being
(Whether they are positive or negative). inefficient in terms of monetary expenditures when
it may in fact be effective in realizing its long term
IMPACT EVALUATION. This type focuses on goals and in delivering benefits that in the long
whether a program is having an impact on he term far exceed the dollar costs. There are simply
344 Part Five * Making Public Policy

some things, such as quality of life, that cannot be policy being evaluated. This can be accomplishe
quantified. If used alone, cost benefit analysis can through the evaluator asking him or herself
color discussion on whether a program or policy is series of questions. The first question attempt
successful or not. It is most useful as a tool in con- to clarify the goals and objectives of a prograr
junction with one of the other types of evaluation. or policy. This is not always easy to do sinc
the legislative mandate for the policy may hav
How Policy Is Evaluated ambiguously expressed goals, or multiple goal:
or conflicting goals. Next, the evaluator mu:
Evaluation of ‘policy is fairly complex and in-
determine the relationship of the program bein
cludes initial activities that must be undertaken
evaluated to other similar programs. Third, th
to ensure the success of the overall evaluation.
evaluator must identify the major stakeholders i
Intrinsic to this success is the duty of the evalu-
and the target populations of the program.
ato to communicate findings and conclusions to
Stakeholders are individuals, agencies, c
the client. Evaluation can be viewed as a three-
gtoups who hold stakes in-the outcome of th .
stage sequence: planning, data gathering, and dis-
evaluation. Target populations are those whor
‘semination. Across these three stages a series of
the policy affects. Non target groups should als
essential activities must take place (see Box 50.1).
be considered because they may potentially b
affected by the policy.’ Finally, the evaluatc
Stages in the Evaluation Process
must learn the ongoing and recent history of th
STAGE ONE - PLANNING. This stage consists of program. Once all of these questions have bee
three steps. Step one is familiarity with the pro- answered then the evaluator can move on to th
gram. Step two is deciding the focus of the evalu- next activity in the planning stage. |
ation, and step three is developing evaluation In step two of the planning stage evalu: ©
measures. In step one evaluators must become _ tors must decide what they are actually assessin;
aware and familiar with the actual program or Specifically what is the focus of the evaluation? |

BOX 50.1
| Essential Activities in the Evaluation Process

i wid Ors ea
Who is the-audience?
** $
A Are “fie Wieasuites*setiCatbrs 8 a
ute appropriate oe heeds ot ei.

the pose of the beakiion? ecr

6. ying to build theoretical kriowledge or
i Seeding if makimum service Is pralided?
° s Danblopent of esae 0: How will the study affect funding of theprprdgram?
actual program/policy goals fromrotor actu Can we realistically on a Vali oor es
a8 ~ ally achieved: ; ids aan ai
oe

?
of data or. ‘atti reeastieher 2 rig @ What we are slipposed e2 :
ook =) tar -
.-Whet-am: doing, how: am | doing it, and whe
ss and interpreta
of dot
tiaon Oe “preswhah Lia them? Hits Ah EeH
 4
*

Chapter 50 * The Assessment of Executed Policy Solutions 345

it the policy’s impact, is it its outcomes, is it the There is no essential agreement on which
costs and benefits, or is it the way the policy is type of analysis to utilize. Often the best way to.
being delivered? Once the focus is decided upon determine which. methods to apply is to look at
then the evaluator can conduct step three which the program's size and scope, the intended audi-
is the development of measures for the focus. ence for the evaluation, the program’s goals, the
Such measures should include estimating the cost evaluator’s own skills, and the resources available -
of the policy, in both dollar and non dollar terms. to conduct ‘the evaluation. Once the methods have
been determined, the evaluator must develop’.
STAGE TWO - DATA GATHERING. ‘Two types of the research desigri and confirm the instruments
data must be collected by evaluators. First, data (data collection devices) that will be applied (see
must be collected that allows for the program’s Box 50.3). Research designs can be seen as strate-
overall configuration and structure to be better gies that can help the evaluator to improve the
understood. Thus, information on how the pro- validity and reliability of the evaluation. Designs —
gram is delivered, to whom, and how many cli- must be rigorous so as to avoid validity or reli-
ents are served must be gathered. The second ability threats but must also be appropriately
’ type of data gathering deals with the degree to applicable to the complexity and needs of the pro-
which program goals and objectives are being gram. Once a research instrument is selected and .
achieved. The evaluator must also collect data data gathered the evaluator may use a number of
on other effects, both intended and unintended, statistical techniques to analyze and interpret the
that can be attributed to the policy. How data data, Such techniques allow the evaluator to de-
is gathered will be determined by the evalua- termine the potential associations or correlations
tor’s decision to apply quantitative or qualitative of the variables under analysis.
methods, Quantitative methods refer to a range of
techniques that involve the use of statistics and — STAGE THREE - DISSEMINATION. ‘The final stage
statistical analysis for systematically gathering involves the dissemination of the findings of the
and analyzing information. Qualitative methods evaluation to those who commissioned the evalu-.
are aimed at understanding underlying behavior ation, specifically the client. In some cases evalu-
through comprehending how and why certain ation findings are also forwarded to stakeholders,
actions are taken by implementers, clients, and target groups, or the public at large. ‘The goal of
target populations. Box 50.2 highlights the differ- any evaluation is to provide useful information.
ences between quantitative and qualitative meth- Usefulness depends upon a number‘of:factors, in-
ods of analysis. cluding timeliness, accuracy, and completeness.
All evaluation reports should report assumptions a
as well as real indicators which affect data inter-
pretation. In sum, every evaluation will include
BOX 50.2 the perceptions and assumptions that the evalu-
Differences Between Qualitative ator derives from the assessment. Additionally,
and Quantitative Analysis”

BOX 50.3
: teas Research Instruments .
JExperiencing
dey
SEG ithe ATID Fs 1A i
Testing « es Ae Exploring: ry Wee tects
SR Pa ot
“Obsenng m #4 eidiathe: artude Sur
“pe Ros
ee
Pevereers Tar
Finding what is eal pa Exploring Multiple alts “Petsonel Observatigas: =
 346 = Part Five * Making Public Policy

there should be alternative explanations for all organization being evaluated. They are per-
observed outcomes, the separation of fact from ceived as “outsiders.” External evaluators are
opinion, and the findings should be clear and un- often used when the evaluation is authorized
. ambiguous. Finally, an evaluation should, when by an entity other than the organization itself.
‘appropriate, include recommendations for the For example, if the City of Los Angeles wanted
policy's continuation, change or termination. to evaluate the Los Angeles Police Department,
Another dimension critical to effective dis- it would be an authorizing entity outside of
semination is the relationship between the eval- the organization being evaluated. In this case
uator and the client. Clients in many ways can it is more than likely the city would use exter-
‘influence an evaluation study's outcome by bring- nal evaluators on the assumption that external
ing pressure to bear. For example, a client may evaluators would provide objective information
have already made up his or her mind about the because they have no vested interest or agenda
program and may pressure the evaluator to pro- to fulfill. The major advantage of external evalu-
duce an assessment a predetermined finding. In ation is that it is perceived to be impartial be-
response to such concerns, professional organiza- cause evaluators supposedly have no stake in the
tions in recent years have clarified the rights and outcome of the evaluation. This is particularly
responsibilities of evaluators in publishing stan- useful when controversial programs or policies
dards and guiding principles for program evalua- are being assessed. A further strength of utiliz-
tion practitioners. The standards are principles ing external evaluators is that they are usually
rather than rules that evaluators should adhere professional consultants who are trained in the
to. They simply highlight what are acceptable and requisite skills and methods of evaluation tech-
unacceptable practices. Thus, they are a bench- niques. In the past this was undoubtedly true. —
mark for practitioners. Inevitably evaluators must Recently, however, many individuals working in
decide for themselves what practices are ethical the public sector are educated in administration
and justifiable. and management programs that train students in
both policy analysis and program evaluation and
are capable of conducting an evaluation.
Who Evaluates?
The major disadvantage of opting for an ex-
The choice for any agency or group that wishes ternal evaluation is cost, in both money and time.
to be evaluated is who should conduct the evalu- Some would also argue that it can prove costly in
ation. In many ways this is the most critical de- terms of organizational politics because of its po-
cision in the evaluation process. The choice is tentially disruptive nature. A further disadvantage
between internal and external evaluators. Neither could be that external evaluators also have an
choice is inherently better then the other. The key agenda. For instance, they may wish to please the
to who should be used as an evaluator depends client in order to secure future jobs. This is poten-
upon the needs of the organization that is com- tially a dilemma. However, in theory professional
missioning the evaluation study. Both types of ethics ensure that evaluators, although mindful of
evaluators have their strengths and weaknesses. client needs, should stay true to their impartiality.
Internal evaluators have an overall advantage of Another weakness of utilizing external evaluators
being familiar with the program, the organiza- is they may face resistance from within the or-
tion, the actors, and the target population. This ganization and between actors and other stake-
can save time in the planning stage of the study. holders who might have a vested interest in the
However, it can also prove to be a disadvantage outcome of the evaluation.
in that internal evaluators, because of their ties to In conclusion it is interesting to consider
the organization, might be “too, close” to identify two general laws formulated by James Q. Wilson,
problems, to place blame, or recommend major ’ which put into perspective concerns about the
changes or termination. evaluation process. !4 Wilson’s first law is that all
External evaluators are individuals who policy interventions in social problems produce
have no internal connection or ties to the the intended effect—if the research is carried
 Chapter 50 * The Assessment of Executed Policy Solutions 347
RO

BOX 50.4
inked emir cd fine rtd

Validity Types
Rie

internal Validty Eternal Validity ° . ‘« Programmatic Validity: Ls

* Does evaluation ease what mh Len
¢ a gered? “¥ Does evaluation generate ifortae
it intends?- , “fs ay 8 Mas useful to:program officials? oi
5 Requires correct idetfeaton and: “e+ Con fis et " Requires designing an evaluation ae
Senisit* a _toall audiences Sat ah

out by those implementing the policy or by their - the need for validity can prove to be a problem
friends. Wilson’s second law is that no policy for evaluators.
df ad

intervention in social problems produces the There are three broad categories of validity
intended effects—if the research is carried out by that evaluators must be concerned with: internal,
mae op reeeag paRaR nied

independent third parties, especially those skep- external, and programmatic. Box 50.4 highlights
tical of the policy. Wilson’s two laws help explain each of these factors in achieving a valid design.
just how difficult the evaluation process is. If evaluators do not pay close attention to such
factors in the formulation and conduct of an
evaluation then the very findings of the evalua-
Obstacles and Problems in Evaluation
tion will be invalidated. The obstacles to validity
and Utilization of Evaluation Research
are numerous and range from elements within
ah ae

There are several factors that pose serious prob- the environment to methodological errors by the
lems during the evaluation of a policy.’ The first evaluator.6
factor that clearly causes problems in any evalu- Overall, obstacles to evaluation are im-
ation is ambiguity in the specification of the ob- portant factors that can prevent successful
jectives and goals of a policy. It is common for evaluation and hinder the evaluator’s recom-
objectives and goals to be sometimes unclear or mendations being utilized by policy makers.
equivocal and this can cloud the assessment of Quite often, because of contextual factors,
whether the goals and objectives have been met. human factors, or technical factors, decision
A second problem can occur when objectives makers may be prevented from utilizing the
have been stated, but there is no clearly defined results of the study. Contextual factors involve
way to measure the success of the objective. factors within the environment which will be af-
A third problem is the presence of side effects fected in unacceptable ways if decision makers
from other policies that interact with the pro- act on the recommendations of the evaluation.
gram being evaluated. In essence the problem is Technical factors refer to the problems caused
how to weigh outside factors relative to the op- by methodological considerations. Human fac-
eration of the program being evaluated. A fourth tors are obstacles posed by the personality and
problem is that the necessary data is often not psychological profile of the decision makers,
available, or if it is available, it is not in a-suit- evaluators, the client, and other internal actors.
able state for the purposes of the study. Fifth, the In reality, evaluation is fraught with problems ~
politics of the situation will often interfere with and weaknesses.
the evaluation process. For example, there may
be resistance by administrators or other policy
Summary
actors to an evaluation being conducted or to its
findings, A sixth problem is determining if suf- In this chapter, we have discussed what policy eval-
ficient resources are being allocated to conduct uation is, how it is carried out, who does it, the prob-
the most appropriate type of evaluation. Finally, lems that may be encountered and the obstacles to
 ®
€

48 = Part Five * Making Public Policy

he utilization of recommendations made by pro- 7. Ibid. p. 3. é

ram or policy evaluation studies. Over the past 8. General Accounting Office, Federal Evaluation
hirty years, policy evaluation has attracted consid- Issues (Washington, D.C.: GAO, 1989), p. 4,
rable interest among policy makers at all levels in 9. FG. Caro, ed. Readings in Evaluation Research,
he public sector. It is important to remember that 2™4 ed. (New York: Russell Sage Foundation,
valuation is essential, for it often tells policy mak- 1977), p. 6.
ts what is working and what does not work. 10. J. R. Sanders, The Program Evaluation
Standards (Thousand Oaks: Sage, 1994),
pp. 8-12.
11. R. D, Sylvia, K. M. Sylvia and E. M. Gunn,
rnd Notes Program Planning and Evaluation for the
Public Manager 274 ed. (Prospect Heights;
1. R. Haverman, “Policy Evaluation Research
After Twenty Years,” Policy Studies Journal 16, Waveland Press, 1997), pp. 171-174.
no.2 (winter 1987), pp. 191-218.
12. E. R. House, Evaluating with Validity
(Thousand Oaks: Sage, 1980), pp. 20-33.
2. M.E. Rushefsky, Public Policy in The United
13. J. R. Sanders, The Program Evaluation
States, (Belmont: Wadsworth, 1990), p. 16.
Standards (Thousand Oaks: Sage, 1994),
3. M.J. Dubnick and B. A. Bardes, Thinking About
pp. 8-12.
Public Policy (New York: Wiley,1983), p.203.
14. J. Q. Wilson, “On Pettigrew and Armor,” The
4. J. S. Wholey et al, Federal Evaluation Policy
(Washington, D.C.: The Urban Institute,
Public Interest 30 (Winter 1973), pp. 132-134.
15. B. W. Hogwood and L. A. Gunn, Policy
1970), p.15.
Analysis for the Real World (New York: Oxford
5. R. Haveman, “Policy Evaluation Research After
Twenty Years,” Policy Studies Journal 16, no. 2 University Press, 1984), pp. 220-227,
16. R. D. Sylvia, K. M. Sylvia and E. M. Gunn,
- (Winter 1987), pp. 191-218.
6. R. D. Bingham and C. L. Felbinger, Evaluation Program Planning and Evaluation for the
Public Manager 2°¢ ed. (Prospect Heights:
in Practice: A Methodological Approach | (New
Waveland Press, 1997), pp. 117-127.
York: Longman, 1989), p. 4.

& 1.