RTPI

Research
Paper

November
2020

MEASURING WHAT
MATTERS
Planning Outcomes Research Handbook

[Type here]
The Royal Town Planning Institute (RTPI)
RTPI champions the power of planning in creating prosperous places and vibrant communities. We have over 25,000 members in the
private, public, academic and voluntary sectors. Using our expertise and research we bring evidence and thought leadership to
shape planning policies and thinking, putting the profession at the heart of society's big debates. We set the standards of planning
education and professional behaviour that give our members, wherever they work in the world, a unique ability to meet complex
economic, social and environmental challenges. We are the only body in the United Kingdom that confers Chartered status to
planners, the highest professional qualification sought after by employers in both private and public sectors.

Report authors
Kevin Murray Associates, University of Dundee, McCabe Durney Barnes, Cardiff University, Yellow Book

About this paper

This Handbook is part of a suite of documents forming the Measuring What Matters Planning Outcomes Research. This Handbook offers
practical guidance alongside the research toolkits that provide the practical framework for implementing this new process of measurement.
Other documents forming part of this research are:
 Measuring What Matters: Planning Outcomes Research Report and Toolkit (this toolkit can be adapted and applied to any nation)
 Scottish Report and Toolkit
 Irish Report and Toolkit
 Wales Report and Toolkit

Front and back cover image

Photo credit: Alistair Battles
Handbook contents
1. The handbook introduction ....................................................................................... 4
2. Measuring What Matters: the process ...................................................................... 5
3. The toolkit framework ................................................................................................ 6
4. The toolkit steps ......................................................................................................... 8
5. Scoring ...................................................................................................................... 10
6. Visualisation options ............................................................................................... 11
7. Review and feedback loop ....................................................................................... 14

Measuring What Matters: Planning Outcomes Handbook 3

1. The handbook introduction
This document is a supplement to Measuring What Matters: Planning Outcomes Research Report and sets out the process for use of the
toolkit. The toolkit is a way of bringing all the necessary information and data into a single location, to be ‘diagnosed’ and reviewed as part of
the process in shifting output monitoring towards tracking impact. Four versions of the toolkit exist - for Scotland, Ireland, Wales and a common
toolkit that can be applied in other planning jurisdictions. Use this handbook, alongside any of the toolkits, as a guide through the process and
to help make the most of your use of the toolkit.

Measuring What Matters: Planning Outcomes Handbook 4

2. Measuring What Matters: the process
The research highlighted that there is a need to develop a toolkit that enables a better understanding of the impact of planning. The toolkit is
built around addressing the question – what difference is planning making? To understand the impact of planning, the immediate activities and
direct outputs of planning need to be understood, alongside the wider outcomes from the planning process.

The three levels below provide a guide, showing a progression from understanding policy effectiveness in application determinations (Level 1),
to direct physical development outcomes (Level 2), to wider cumulative direct and indirect impacts of planning (Level 3).

Measuring wider impacts – understanding

Level 3: Evaluating
the difference planning makes, directly and
impact
indirectly, to what matters

Monitoring direct change to environment

Level 2: Development
through development or protection – and
outcomes
whether in line with policy/objectives

Level 1: Inputs, Monitoring how policy and objectives

Activities, Outputs influence planning decisions that follow

Measuring What Matters: Planning Outcomes Handbook 5

3. The toolkit framework
The toolkit is a flexible tracking framework that can operate at multiple spatial scales. It has been tested in a range pilots, ranging from regional
plans to urban development frameworks. This framework is used to pull together all the information needed to undertake the assessment for
any level. By drawing plan information together with monitoring information, it acts as a diagnostic tool, moving beyond simply monitoring to
aiding an assessment of progress towards objectives/targets. A full-size version of the framework below, with examples, is provided in the
Measuring What Matters: Planning Outcomes Research Toolkits.

Theme Ref No. Policy/Objective Objectives/Targets Monitoring Indicators Data Achievement Score

Place, Design & People

Health & Wellbeing

Environment,
Conservation &
Improvement
Climate Change

Homes & Community

Economy & Town

Centres

Movement

Process & Engagement

Measuring What Matters: Planning Outcomes Handbook 6

 Toolkit Input Columns (see columns on page 6)

Themes The eight themes reflect the sectoral areas over which spatial planning has an influencing role and impact

A numbering system for users to refer and cross-refer with a data library (see data library tab in toolkit). This is
Ref No.
important for tracking

Identify theme-related policies or objectives to be monitored. This material may be drawn from approved statutory
Policy/Objective
plans and their constituent policies or from other strategies/masterplans at a variety of scales (e.g. city, regional)

Objectives/Targets Clearly identify the purpose of the plan or policy, including any specific objectives and/or targets

From available monitoring material, identify appropriate performance measurement indicators that are relevant to
Monitoring Indicators
the policy/objective being tracked

Insert data associated with the indicator, beginning with a ‘baseline’ position. Note likely tracking intervals (e.g. 1, 2,
Data
5 years)

Assessment of the data against objective/target. An aggregate score for the theme can be presented using
Achievement Score
diagrams. More detail is provided on scoring at page 10
Note: The data library tab in the toolkit can be used to help record available data sources

Measuring What Matters: Planning Outcomes Handbook 7

4. The toolkit steps
Central to the process is the working through a series of steps over time to measure progress. The five steps below form a full evaluation cycle:
Step 1 is preparation, Steps 2 to 4 use the toolkit (from collecting data to diagnostic review and assessing impact) while Step 5 guides the
future actions arising from evaluation. These steps can be applied at each of the three levels (shown on page 5).

 Identify clearly what plan and policy is to be tracked and its performance monitored, including reasons
 Define the spatial scope and timeframe of assessment and note what stage the plan/policy is at in its current cycle
 Explain the purpose and stage of assessment (e.g. mid-term review of policy, thematic review, baseline, 5-year review)
Step 1
 Identify any current monitoring processes and evaluation feedback to benchmark policy effectiveness and data
Scoping exercise
 Reflect on how the toolkit will be used to identify activities and select indicators to measure outputs and outcomes
 State any working assumptions underpinning the use of the toolkit
 Establish a project team and agreed procedures (e.g. for sharing and tracking data across organisations/teams)

 Confirm selection of plan and policies to be tracked - these may be drawn from a mix of site/locational/area specific, to
strategic/sectoral and indirect impact (e.g. health and wellbeing)

Step 2  Assign selected policies to one of the eight toolkit themes

Collation of data  Identify tracking indicators from available monitoring material
 Publicly available data connected to impact, either directly or indirectly attributable to planning, is required
 Draw together all data sources relating to monitoring of plans/ policies and their objectives/targets

Measuring What Matters: Planning Outcomes Handbook 8

  Analyse trends in measurable data to gain picture of performance over timeframe
 Support measurement with qualitative assessment - include via group review - to reflect on context of application and
Step 3 limitations
Diagnostic review
 Identify and understand effectiveness of the policy in operational decision-making - e.g. applications and appeals
 Identify any gaps in data and monitoring knowledge. This can influence future data and performance enhancement

 Evaluation and scoring of performance to be undertaken as either internal, external (or combined) team assessment
 Assign a score of between 1 and 5 to reflect performance against target, policy effectiveness, achievement of objective
Step 4  Aggregate scoring allows for visualisation and composite presentation of findings for multiple policy strands in same
Assessing impact theme. (See pages 10 - 13 for advice on the visualisation of scores)
 Consideration may be given to weighting of different policies depending upon their strategic importance before producing
finalised aggregate score

 Review the strengths and weaknesses in performance of plan/strategy/policy in the round

 Consider how policies align with higher level strategy and objectives, such as national or regional

Step 5  Review the effectiveness and outcomes of the policy, and if the policy is working
Review, resolve,  Review and amend strategy/policy/plan objectives, if required from analysis
implement  Update indicators and monitoring process including additional data gathering requirements
 Identify and share all lessons learnt for the assessment process
 Implement the lessons learnt the next cycle of performance measurement in the chosen timeframe

Measuring What Matters: Planning Outcomes Handbook 9

5. Scoring
A method of comparative scoring is needed to give an indication of achievement against objectives that can be read across policies, plans and
the eight themes. To provide such comparability, the scoring is based on a five-level ranking, each with performance benchmarks and
proposed action.

To be rigorous and fair in measuring impact, the scoring needs to be an assessment of data against targets. This ensures that the
measurement is a real gauge of progress over time, not an arbitrary ranking.

To influence future performance the achievement score needs to be associated with some action in the next steps of the process.

Score Performance benchmark Action

5 Excellent progress towards outcomes Share learning with others

4 Good progress towards outcomes Consider increasing ambition in policy objectives/targets

3 Fair progress towards outcomes Review and aim to improve implementation and raise ambition in policy

2 Poor progress towards outcomes Review policy and understand implementation

1 Not achieving progress towards outcomes Immediate review and revisit of policy and its delivery

The first cycle in the application of the toolkit will establish baseline scores. It may be that, in the course of this initial cycle, accurate evaluation
is not possible due to a lack of clear goals or reliable indicator data.

Because the process relates to sharing lessons and guiding future action, this should aid not only better performance, but also better policy and
future measurement too.

Measuring What Matters: Planning Outcomes Handbook 10

6. Visualisation options
Visualisation of Scores

Communication of the impact of planning by scoring is a

key visual product of the toolkit. We have created two
options for illustrating this.

By aggregating the scoring within in each of the eight

themes, and visually representing on a radar diagram
(opposite) or graphic equaliser matrix (below on page 13)
we can provide quick ‘easyread’ feedback on the impact
planning for a specified plan, area, policy.

As both forms show, performance can be linked to the UN

SDGs and also to a range of national objectives or
outcomes.

The circular ‘radar’ diagram scores performance from the

inside (low, under-achievement) outwards to the outside
(excellent). It is useful for communicating online, in
presentations or other instances where early visual impact
is important.

The linear ‘graphic equaliser’ diagram can be more

nuanced for formal reporting and other uses where space
and visual impact are not at a premium.

Measuring What Matters: Planning Outcomes Handbook 11

 This shows an example of a radar diagram with
aggregate scores from each theme inserted.

The themes with no scores showing reflect that

either (1) no specific policies were identified under
these themes for measurement, or (2) no data
was able to be identified.

The expectation is that, over time, all theme

categories will be able to be address at a local
authority area plan level by at least some
trackable policies, ideally several.

However, with some specialist plans or policies,

such as economic, housing or health, they may
not trigger policy objectives or responses across
all eight themes.

Measuring What Matters: Planning Outcomes Handbook 12

This version below shows the matrix/graphic equaliser method of representing the performance within each theme. The right choice will depend
upon the occasion and audience. Typical audiences for communicating the analysis might include internal team members, elected members,
funding agencies and the wider public, such as civic groups.

Measuring What Matters: Planning Outcomes Handbook 13

7. Review and feedback loop
The toolkit is intended to be used in more than one full evaluation cycle,
embracing monitor, review and application of the learning. Only by
cyclical repetition of the process with a critical review at Stage 5, can
improvement be tracked and delivered.

The process is set up to ensure that the basic building blocks are in place
to assess impacts, address issues and improve performance.

The key elements of the Stage 5 review stage are:

Review the assessment

 Are inputs being monitored?

 Are activities and outputs being monitored?
 Are development outcomes being monitored?
 Are place impacts (quality and value) being monitored?
 Does the scoring indicate strong alignment with
local/regional/national plans and national outcomes?
 Are Key Performance Indicators, SMART targets or other
markers being established where needed?

Figure 1 Handbook: Aligning and Setting Targets

Measuring What Matters: Planning Outcomes Handbook 14

Resolve issues arising

 Inputs1 – what data needs to be captured to monitor effectively?

 Activities and outputs – what data needs to be captured to monitor effectively?
 Outcomes – what data needs to be captured to monitor effectively?
 Impacts – what data needs to be captured to monitor effectively?
 What policy areas need to be developed to create stronger alignment with local/regional/national plan and national outcomes?

Implement actions

 Develop policy to align with local/regional/national plans and national outcomes

 Develop objectives and targets that can be measured
 Identify existing data to begin monitoring
 Identify future data requirements

This completes a feedback loop of any one cycle and feeds into the next cycle of monitoring. Evidence for how to take forward to higher levels
of performance, including national and international, are provided in Appendix B of the research report.

Congratulations on completing your first cycle on the process towards measuring the direct and indirect impacts of planning.

1
Inputs refer to the human, financial (including fees from planning income) and other resources deployed by planning policymakers and Planning Authorities.
The other resources may include legal powers, statutory plans, planning policies and published guidance. The inputs are deployed to deliver planning
activities.

Measuring What Matters: Planning Outcomes Handbook 15

For more information about this paper, visit:
www.rtpi.org.uk

RTPI - Royal Town Planning Institute

[email protected] OR [email protected]
Tel: 020 7929 9494

Report contact
Rhian Brimble

Royal Town Planning Institute,

41 Botolph Lane, London EC3R 8DL.
Registered Charity in England (262865) & Scotland (SC037841)