A framework for public policy analysis and policy evaluation M.

Theo Jans
IES research colloqium 4 September 2007

9/6/07 | pag. 1

Policy analysis
Public policy focuses on the public and its problems (Dewey, 1927) The study of how, why and to what effect governments pursue particular courses of action and inaction (Heidenheimer, 1990) What governments do, why they do it, and what difference does it make (Dye, 1976) The study of the nature, causes, and effects of public policies (Nagel, 1990)

Policy analysis
Public policy concerned with:
How are problems and issues defined and constructed? How are they placed on political and policy agenda? How policy options emerge? How and why governments act or do not act What are the effects of government policy?

No single discipline, integrates what seems useful for understanding

Policy analysis
Multi-method (quantitative, qualitative) Multi-disciplinary (social sciences) Problem-focused Mapping the context Options and effects Analysis:
Of policy => theories (determination, content, evaluation) For policy => prescriptive, applied (techniques)

Agenda-setting (Howlett & Ramesh: adapted and altered)

Nature of public involvement
High Low

Public

Outside initiation

Consolidation

Initiator of debate
State

Mobilisation

Inside initiation

Agenda-setting Reasons for inclusion

Serious, important, real problems Issues that manage to overcome the gatekeepers (discontent, demands, solutions, public agenda, extension, inclusion) = policy entrepreneurs Convergence thesis: Issues on the agenda are determined by the time and context. States with similar welfare levels will deal with similar agenda items Economic cycles determine issues: crisis= labour flexibility / efficiency increases; growth= quality of life / worker welfare

Agenda-setting Reasons for inclusion

Electoral cycles: issues on agenda determined by election timing / party in office Issues that fit the policy mood or policy paradigm or the leading policy ideas (eg. competitiveness, global warming,) Multiple streams and policy windows (Kingdon)

Agenda-setting: Kingdons multiple streams

Research Kingdon: 300 interviews period 4 years Executive, parliament, interest groups: policy community What are the main policy issues and why? What policy issues, problems are of lesser importance and why? Compare year per year.

Agenda-setting: Kingdons multiple streams

Origin is necessary but unclear, follow-up is much more important for inclusion I can trace the paths of ideas. But my personal theory is that people plant seeds every day. There are a lot of ideas around, and there is no lacking of ideas. The real question is, which of these ideas will catch hold ? When you plant a seed you need rain, soil, and luck. (Kingdon, 81) Streams of recognised problems, of policy solutions, stream of political events.

Agenda-setting: Stream of recognised problems

Stream problem recognition by actors in- and outside government Indicators: volume, change (IM scoreboard, week-end traffic accidents) Focus events: 9/11 Policy feedback: evaluation, complaints (speeding fines, small arms legislation) Conditions become problems Clash with norms, values, principles Unfavourable comparisons with other countries/situations Problems can go back to being social conditions again Something was done, time and effort was spent on the issue (BHV) Indicators decrease, crisis dissolves Acceptance and getting used to problem (high unemployment levels, racism) Other problems take priority and absorb the attention

Agenda-setting: Stream of policy solutions

Stream Policy Community
Administration, academics, policy advisors, think thanks, interest groups: formulate policy proposals

Primeval policy soup: multitude of ideas are debated, combined, tested, exchanged, and evaluated Selection mechanism: survival of the fittest
Technical and administrative feasibility Congruence with dominant values Anticipation of probable resistance (budget, public opinion, political receptiveness)

Agenda-setting: Stream of politics

Events (ex. dioxine in poultry, prison escape Dutroux, coalition negotiations) Elections Governmental majorities Activities of interest groups The political climate ...

Agenda-setting: When streams meet

The three streams exist and develop relatively independent from each other Events (political streams) may occur for which the policy community is unprepared (ideas, solutions) , and vice versa. The separate streams come together at critical times. A problem is recognized, a solution available, the political climate make the time right for change, and the constrains do not prohibit action. (Kingdon, 94)

Agenda-setting: Policy window

Policy windows are the moments when the three streams meet, and the issue can achieve high agenda priority Policy entrepreneurs need to act in the policy window:
Policy entrepreneurs have solutions and are waiting for problems and right climate to implement them Policy entrepreneurs have problems, are waiting for solutions and the opportunity to settle them.

Policy windows are opened by:

Problems, Political stream, coincidence

Agenda-setting: Policy window

Policy window allows for full coupling Partial coupling is also possible:
Problem and a solution: but is politically not interesting There is a solution and a political willingness to apply it, but no real problem Problem and political will to tackle it, but no policy solutions

Policy formulation
The definition, evaluation, acceptance and discarding of policy options. Policy formulation is both a technicalrational as well as a competitive phase. Who develops options? issue networks, iron triangles, advocacy coalitions Openness to new ideas/actors

Policy formulation (Howlett & Ramesh)

Entrance of new actors
Yes Yes No

Entrance of new ideas

Policy renewal Policy goals (open subsystem) Experimentation with instruments (resistant subs.)

Program reform Specification (contested subs.) Instrument tinkering (components) (closed subs.)

No

Decision-making Three models

The rational model The incremental model The irrational model

Decision-making: rational model

Problem identification, organization and classification of values, goals, and objectives relevant to problem List all possible ways to solve problem and to realise goals List possible consequences for each policy alternative with probability of occurence Compare consequences with previously formulated goals and objectives

Decision-making: rational model

Select the policy solution:
1. With consequences most closely aligned to goals 2. That provides highest level of problem resolution 3. That provides most benefits at equal costs 4. That provides lesser costs in case of equal benefits

The reasoning constitutes the basis of most rational decision models

Decision-making Bounded rationality (H. Simon)

Policy makers are limited in the information and policy alternatives they can process Policy makers lack complete information and knowledge of all policy options Consequences of options are unknown and educated guesses at best Individuals have cognitive limitations (memory, attention, processing) Complete rationality can not be assumed in policy making === satisfycing

Decision-making: incremental model (C. Lindblom)

Lindblom starts from bounded rationality but draws different conclusions Rational, planned goal realisation is not possible nor desirable in decision making Decision making requires bargaining and negotiation systematic evaluation of policy options hampers that dynamic Policy requires feasible and supported decisions rather than decisions that maximally established a desired state of affairs

Decision-making: incremental model

Rational model does not correspond to real decision making processes Decision making occurs step-by-step, piecemeal, through trial-and-error Incremental decisions are to be preferred:
smaller scale, less radical and ambitious policy measures can be tested and adjusted as they are implemented modesty of decisions limits possible negative consequences

Decision-making: incremental model

Gradual policy changes Decisions have limited applicability: they are amended, adjusted, re-adopted, etc. Only familiar policy options are considered (no radical change or innovation) Policy options dissappear as result of lacking consensus rather than rational selection Policy makers concentrate on avoiding disadvantages or problematic situations rather than goal achievement Gradual decision processes stimulate policy learning Decision making is a constant negotiation and adjustment process

Decision-making: irrational model

March, Cohen, Olsen: analysis of decision making in universities Non-rational, nor incremental decisionmaking Rather coincidental congruence between problems, solutions and choice opportunities Problems, solutions and choice moments are dumped in garbage cans

Decision-making: organised anarchies

Unclear preferences and goals: organisation is a loose collection of ideas with few clear goals (multiple goals)
No clear preferences: would lead to conflict Preferences and strategy are developed through action, rather than that action is guided by preferences or strategies Often organisations have multiple goals (quality research, education, many students, income, smooth running administration, happy alumni)

Decision-making: organised anarchies

Incomplete knowledge of technology and organisation: own role is known, partial understanding of other roles and of procedures within organisation
Knowledge of how institution is acquired through trial-and-error or is deduced from experiences and responses in crisis situations

Varying participation: participation and time investment varies considerably among participants: impact on outcome

Implementation
Pressman & Wildawsky: Implementation how great expectations in Washington are dashed in Oackland; or why it is amazing that Federal programs work at all, this being a saga of the economic development administration as told by two sympathetic observers who seek to build morals on a foundation of ruined hopes
(1973 federal program for unemployed inhabitants of Oackland)

Pressman - Wildavsky Top-down implementation

Frustration felt with the failures of the war on poverty and great society programmes (in late sixties) Sentiment that rationalized decision making (Simon, McNamara) was not leading to desired policy outcomes Not because bad decisions were taken but rather because good decisions were badly implemented

Pressman - Wildavsky Top-down implementation

Multiple intermediary actors (govs., agencies) for implementation require perfect co-operation (Wildavsky) Less than perfect co-operation leads to an accumulation of small mishaps which trigger policy large failures Careful implementation design is the key to succes = top-down approach Monitoring and control of implementation (Military chain of command)

Michael Lipsky The bottom-up perspective

Implementation: analysis of front-line staff in policy delivery agencies or street-level bureaucrats Policy is made as it is being administered The decisions of street-level bureaucrats, the routines they establish, and the devices they invent to cope with uncertainties and work pressures, effectively become the public policies they carry out

Michael Lipsky The bottom-up perspective

Street-level bureaucrats start with high service ambitions but large classes or huge caseloads, inadequate resources, uncertainties of method, unpredictability of of clients defeat their initial aspirations To cope with pressures, implementers develop methods of processing people in a routine and stereotyped way = techniques to salvage service and decision making values

Implementation and ambiguity: R. Matland.

Policy ambiguity
Low Low High

Level conflict
High

Administrative implementation (planning and resources) Political implementation (power and feedback)

Experimental implementation (context, variation, learning) Symbolic implementation (local coalitions)

Policy evaluation
The stage of the policy process at which it is determined how a public policy has actually fared in action (Howlett & Ramesh) = evaluation of means being employed and objectives being served Problem no universal and fixed criteria
Spectacular failures Substantive failures Procedural failures

Policy evaluation
Evaluate output in function of expectations/goals BUT failure or succes is a judgement of events (not inherent to the event) Goals often vague, multiple, no ranking, shifting throughout policy stages. Evaluation inherent, build-in biases

Policy evaluation
Administrative:
Effort evaluation: screen and monetarise inputs what does it cost Performance: screen outputs (graduates, publications) - what is it doing Effectiveness: is it doing what it is supposed to be doing goals vs. outputs Efficiency: input evaluation / output evaluation and seek to reduce input (lower cost) Process evaluation: organization methods used and possibilities for process re-engineering

Policy evaluation
Judicial review Political evaluation Policy evaluation = increasingly perceived as policy learning MBO functioning of EC Four evaluative styles (Cohen & Levinthal)

Policy evaluation styles (Cohen & Levinthal)

Dominant actors in subsystem
Societal actors High State actors

State admin. capacity

Social Learning (fundamental acknowledgement) Non-learning (Political evaluations)

Instrumental Learning (Lesson-drawing) Limited learning (Technical evaluations)

Low

Policy styles (Howlett & Ramesh: adapted and altered)

Agenda-setting Policy formulation Decision making Implementation Policy evaluation
Outside Initiation Policy Renewal Incremental adjustment Adminstrative Social learning Inside Initiation Program reform Satisfycing adjustment Political Instrumental learning Consolidation Policy experimentation Garbage can emergence Symbolic Limited learning Mobilization

Policy Tinkering

Rational search

Experimental

Non learning

Bibliography
M. Howlett, M. Ramesh (2003) Studying public policy. Policy cycles and policy subsystems. OUP, Canada. M.D. Cohen, J.G March, J.P. Olsen (1972) A garbage can model of organizational choice. Administrative Science Quarterly, 17, 1: 1-25. W.M. Cohen, D.A. Levinthal (1990) Absorptive capacity: a new perspective on learning and innovation. Administrative Science Quarterly, 35: 128-52. B.W. Hogwood, L.A. Gunn (1984) Policy analysis for the real world. New York: Oxford University Press. J.W. Kingdon (1984) Agendas, alternatives and public policies. Boston, Little, Brown. C.E. Lindblom (1979) Still muddling through. Public Administration Review, 39, 6: 517-25. J.L. Pressman, A.B. Wildavsky (1984) Implementation: how great expectations in Washington are dashed in Oakland. 3rd edn. Berkeley, University of California Press. M. Lipsky (1980) Street-level bureaucracy: dilemmas of the individual in public services. New York: Russell Sage Foundation. R.E. Matland (1995) Synthesizing the implementation literature: the ambiguity-conflict model of policy implementation. Journal of Public Administration Research and Theory, 5, 2. P.A. Sabatier (1999) The need for better theories. in P.A. Sabatier (1999) ed. Theories of the policy process. Boulder Colorado: Westview Press. H.A. Simon (1957) Models of man: social and rational. John Wiley: New York.