Chapter 1
Chapter 1
Performance auditing
Performance auditing carried out by SAIs is an independent, objective, and reliable examination of
whether government undertakings, systems, operations, programmes, activities, or organizations are
operating in accordance with the principles of economy, efficiency and effectiveness and whether
there is room for improvement.
A performance audit is one of three main types of public-sector audits defined in the
International Standards of Supreme Audit Institutions (ISSAI) 100/22. It is distinct from the
other two main types, financial audits and compliance audits, as discussed later in this
chapter.
13
A performance audit covers the full range of government activities, including organisational,
financial and administrative systems (INTOSAI-P-1, Section 4). A performance audit may focus
on a single programme, policy, entity or fund, or may focus on outcomes or systems, looking
across programmes, policies and entities that contribute to the outcome or system. It can
focus on:
14
Economy: Keeping the cost low
The Standard
Economy is minimising costs of resources used in performing an activity. The resources used should be
available in due time, in and of appropriate quantity and quality and at the best price.
Source: ISSAI 300/11
Auditing economy focuses the audit on how the audited entities succeeded in minimising
the cost of resources (input), taking into account the appropriate quality of these
resources. This part of the audit focuses only on the input by asking: “Are the resources
used available in due time, of appropriate quantity and quality, and at the best price?”
(GUID 3910/38).
When conducting audits of economy, the auditor may provide answers to such questions
as1:
• Have the best prices been obtained for consultancy services?
• Is there potential for reducing the cost of sickness absences?
• Are there procedures in place to ensure that transport costs of food aid are the lowest
available?
• Has there been a waste of resources in achieving an output?
Considerations of economy often lead to examining processes and management decisions
regarding the procurement of goods, works and services.
The Standard
Efficiency is getting the most from available resources. It is concerned with the relationship between
resources employed (the inputs) and outputs delivered in terms of quantity, quality and timing.
Source: ISSAI 300/11
Efficiency assesses the relationship between inputs and outputs. Auditing efficiency means
asking whether the inputs have been put to optimal or satisfactory use or whether the
same or similar outputs (in terms of quantity, quality and turnaround time) could have
been achieved with fewer resources. In other words, “Are we getting the most output – in
terms of quantity and quality – from our inputs?” (GUID 3910/39). Therefore, efficiency is
about the maximum output obtained for a given level of input or the minimum level of
input required for a given output level. Quality is an important concept on the input side,
both in efficiency and economy (GUID 3910/38).
15
Efficiency is a relative concept, meaning that a process, instrument or programme is either
more or less efficient than another. For an audit on efficiency, you, need to conduct some
comparison. You may, for example, compare similar activities in comparable entities; one
process (in one entity) with the same process at an earlier point in time; a process before
and after the adoption of a policy or procedure; the efficiency of an organisation with an
accepted set of characteristics of efficient organisations. Audits of efficiency can also
examine the processes leading from input to output to expose shortcomings in these
processes or their implementation. This can lead to a better understanding of why
processes are efficient, even without measuring efficiency itself. (GUID 3910/41)
In audits of efficiency, you might ask questions such as 2:
• How does the cost per job created by a training programme for the unemployed
compare with similar costs per job elsewhere?
• Could project X have been implemented differently that would have resulted in
improved timeliness and quality?
• Are adequate procedures and criteria for prioritising and selecting transport
infrastructure projects to ensure maximum impact in place?
• Are schools maximising the use of their information technology equipment?
When the audit objective of efficiency considers outputs, you will usually focus on
processes by which an organisation transforms inputs into outputs.
The Standard
Effectiveness is meeting the objectives set and achieving the intended results.
Effectiveness deals with outputs, results or impacts. It is about the extent to which policy
objectives have been met in terms of the generated output. It is concerned with the
relationship between goals or objectives on the one hand and outcome on the other. The
question of effectiveness consists of two parts: first, to what extent the objectives are met
and second, if this can be attributed to the output of the policy pursued (GUID 3910/42).
It focuses on questions such as:3
• Have infrastructure projects contributed to increased traffic flow while improving safety
and reducing journey times?
16
• Have suitable measures to monitor and mitigate the environmental impact in sector X been
set up and properly implemented?
• Are departments or entities achieving their objectives for all sectors of the community?
Audit of effectiveness will concentrate on outputs, results or impacts. When assessing
effectiveness, SAIs consider whether and how a government policy, programme or activity
is meeting its goals. Sometimes SAIs may split effectiveness into two distinct aspects:
• The attainment of specific objectives in terms of outputs (this is called efficacy in some
SAIs).
• The achievement of intended results in terms of outcomes.
For example, you may be auditing a Ministry of Education programme designed to improve
the skills of students who have left school to fill anticipated skills gaps in the workforce. If
you focus purely on outputs, your focus will probably be on the changes in indicators, such
as the number and proportion of students leaving school with the target qualifications. A
more ambitious audit, looking at outcomes, might consider more complex questions such
as:
• Has the policy led to any change in the skills gap in the labour market?
• How well is the Ministry able to predict and respond to gaps in the labour market?
In that case, when you look at effectiveness in terms of outcomes, it would be necessary
to look at connections among entities and institutions. You need to consider a larger
environment. The expected outcome will not depend just on one programme or initiative.
In the example above, it might involve entities related to employment, transport, finance,
besides the entity directly responsible for the programme.
SAIs working on effectiveness can benefit from approaches drawn from disciplines such as
programme evaluation – the activity of examining the implementation and impacts of
policy interventions to identify and assess their intended and unintended effects and costs.
Where appropriate, SAIs and audit teams have to consider the impact of the regulatory or
institutional framework on the performance of the audited entities.
Auditing the effectiveness of performance in relation to the achievement of the audited
entities’ objectives entails auditing the actual impact of activities compared with the
intended impacts.
Effectiveness can be measured by various methods. The most sophisticated methods
compare the situation being addressed before and after the introduction of the policy or
programme and involve measuring the behaviour of a control group, which has not been
subject to the policy or programme (the counterfactual) through a randomized trial or as a
quasi-experiment.4 However, this type of method is not always feasible. Sometimes more
4A quasi-experiment studies the impact of an intervention on a target population, but uses methods other than random
assignment to select which members of the population are chosen for participation in the study.
17
qualitative methods are better suited to gain insight into causal relations between policy
or programme and effect. When concluding the causal relation between policy or
programme and effects, it is important to clearly communicate the strengths and
limitations of the methods used. There are various documents providing guidance in
choosing the right methods (GUID 3910/45).
In practice, it will be difficult for you to make these comparisons, partly because suitable
comparative material is often lacking, and it can be extremely difficult to isolate the
impacts of the policy or programme being audited from other outside factors. More
commonly, you could assess the plausibility of the assumptions on which the policy is
based. This is sometimes called testing the programme theory. You could also assess if
earlier steps in the programme – especially steps necessary for the final impact – have been
achieved. Often, a less ambitious audit objective will need to be chosen, such as assessing
to what extent the entities´ objectives have been achieved, target groups have been
reached, or the desired level of performance has been attained.
18
Figure 1: Relationship among the 3Es
19
What value do performance audits bring?
Performance auditing focuses on areas in which it can add value for citizens and which have
the greatest potential for improvement and provides constructive recommendations for the
audited entities to take appropriate action to improve performance. (ISSAI 300/12)
Public sector auditing, as championed by the SAI, is an important factor in making a difference to the
lives of citizens. The auditing of government and public sector entities by SAIs has a positive impact on
trust in society because it focuses the minds of the custodians of public resources on how well they use
those resources. Such awareness supports desirable values and underpins accountability mechanisms,
which in turn leads to improved decisions.
Once SAIs’ audit results have been made public, citizens can hold the custodians of public resources
accountable. In this way, SAIs promote the efficiency, accountability, effectiveness and transparency of
public administration.
Source: INTOSAI-P-12
INTOSAI-P-12 explains ways in which SAIs can make a difference in the lives of citizens. Figure
2 shows the specific contributions that performance auditing can make.
20
Figure 2: How performance audits can add value
Performance audit Relevant INTOSAI-P-12 principle How might the Supreme Audit Institution
activity that adds value (SAI) perform this activity?
Integrity 2 - Carrying out audits to ensure that Examining whether government financial
Providing independent government and public sector entities intervention in the housing market has
assurance on success are held accountable for their encouraged buyers who would not have
claimed by government stewardship over, and use of, public otherwise entered the market.
resources. Help to Buy: Equity Loan scheme – progress
review. National Audit Office (UK), 2019.
Accountability 2 - Carrying out audits to ensure that Assessing whether government negotiates a
Helping to hold the government and public sector entities good deal when purchasing medical equipment.
executive to account for are held accountable for their
stewardship over, and use of, public Performance audit report on procurement of
its performance
resources. medical equipment and surgical instruments by
the Department of Clinical Services. Office of the
Auditor General Botswana, 2012.
Transparency 4 - Reporting on audit results and Publishing regional performance data that had
By publishing new thereby enabling the public to hold only been available internally.
information, the SAI can government and public sector entities
shine a light on how public accountable for performance. NHS waiting times for elective and cancer
resources are used treatment. National Audit Office (UK), 2019.
New insights 7 - Being a credible source of Using multiple regression analysis to see which
Applying analytical independent and objective insight and factors have a statistically significant effect on
guidance to support beneficial employee performance.
techniques that have not
yet been used by change in the public sector.
Federal Workforce: Additional Analysis and
government Sharing of Promising Practices Could Improve
Employee Engagement and Performance.
Government Accountability Office, 2015.
Sharing best practice from 7 - Being a credible source of Comparing how different countries manage
home and abroad independent and objective insight and the same activity.
Offering insight based on guidance to support beneficial
experience of auditing similar change in the public sector. Healthcare across the UK: A comparison of the
activities in other NHS in England, Scotland, Wales and Northern
departments. SAIs may Ireland. National Audit Office (UK), 2012.
analyse their individual audit
reports to identify themes, L’accès des jeunes à l’emploi : construire des
common findings, trends, root parcours, adapter les aides (Employment
causes and audit access for young people – building pathways,
recommendations, and adapting state support), Cour des comptes.
discuss these with key (French Court of Auditors), 2016.
stakeholders. SAIs may also
use their engagement in the
international public-sector
auditing profession to draw
lessons from other countries
Making practical 3 - Enabling those charged with public Assessing the root causes of shortfalls in
recommendations sector governance to discharge their performance, then basing their
Including recommendations responsibilities in responding to audit recommendations on this evidence to suggest
in performance audit reports findings and recommendations and how to perform better.
that enable the audited taking appropriate corrective action.
entity to improve its
performance
Clarifying complexity 4 - Reporting on audit results and Writing performance audit publications in a
Providing an easy-to-digest thereby enabling the public to hold simple and clear manner, using language that
summary of complex topics government and public sector entities is understood by all intended users.
accountable.
It is important for you as the auditor to think early about whether and how you can aim to
provide value through your performance audit. These considerations will help you design
methods, analyses and communication strategies that maximise the impact of your work.
21
What types of reports result from performance audits?
The objectives of performance audits – promoting the 3Es and addressing accountability and
transparency – mean that the potential scope of a performance audit may be wide. However,
some themes appear more frequently than others. Figure 3 illustrates some of the common
themes you will likely find in performance audits.
22
Figure 3: Themes that appear in performance audits
Example of a Supreme Audit
Theme Example of an audit objective Institution (SAI) report addressing
this theme
Assessing the extent to which the actions Jamaica's Preparedness for
implemented by the Government of Implementation of Sustainable
Preparedness Jamaica at the national level, since the Development Goals (SDG). Auditor´s
for endorsement of the 2030 Agenda in General Department of Jamaica, 2018.
implementation September 2015, are adequate to
of SDGs support preparedness for the
achievement of the SDGs.
23
Assessing whether the implementation Elimination of violence against women.
of Women’s Plan of Action, in particular SAI Fiji, 2019.
Gender on elimination of violence against
equality women, is effective by examining: the
existing legal and policy framework;
the process by which the framework
has been implemented; the monitoring
and reporting arrangements over the
implementation of the framework, and
whether improvements can be
demonstrated.
What is the difference between performance audit and other types of public
audits?
Performance auditing is a specific discipline with its own standards and conventions. It is
important to understand the differences between performance auditing and the other two
main types of public sector audits: financial audits and compliance audits.
It is usually easy to distinguish a financial audit from a performance audit. A financial audit
involves determining, through the collection of audit evidence, whether an entity´s financial
information is presented in its financial statements following the financial reporting and
regulatory framework applicable (ISSAI 200/7). SAIs conduct financial audit annually, in which
auditors certify an audited entity’s financial statements. A financial audit adds value by
24
providing the intended users of the financial statements with confidence in the reliability and
relevance of information presented in the audited statements.
It can be more challenging to understand the difference between a compliance audit and a
performance audit because they sometimes overlap. Compliance audits cover a broad
spectrum of audits, with different characteristics, examining activities, financial transactions
or information.
Compliance auditing is the independent assessment of whether a given subject matter
complies with applicable authorities identified as criteria. Compliance audits are carried out
by assessing whether activities, financial transactions and information comply, in all material
respects, with the authorities which govern the audited entity (ISSAI 400/12). These
authorities may include rules, laws and regulations, budgetary resolutions, policy, established
codes, agreed terms or the general principles governing sound public-sector financial
management and the conduct of public officials. (ISSAI 400/29)
Some performance audits can include compliance questions to the extent that these are
necessary and relevant to examining 3Es of the subject matter.
A performance audit is a direct reporting engagement (ISSAI 100/29-30). In direct reporting
engagements, the auditor selects the subject matter and criteria and measures or evaluates
the subject matter against the criteria, considering risk and materiality. The outcome of the
measure is presented in the audit report in findings, conclusions, recommendations, or an
opinion. (ISSAI 100/29)
The other type of engagement is attestation engagement, where the responsible party
measures the subject matter against the criteria and presents the subject matter information.
The auditor gathers sufficient and appropriate audit evidence to provide a reasonable basis
for expressing a conclusion. Financial audits are always attestation engagements, and
compliance audits may be attestation or direct reporting engagements, or both at once. (ISSAI
100/29-30)
25
How does being classified as a direct reporting engagement influence the conduct of
performance audits?
As performance auditing is a direct reporting engagement, it will be part of your role as
auditor to select and define the subject matter of your report and conclusion. It is also part
of your role to identify the relevant criteria, and it will be your task to measure or evaluate
the subject matter against these criteria in order to elaborate an audit report that provides
relevant and reliable information to the users of your audit. You will have a much more active
role in asking the relevant audit questions and in selecting and applying the methods that are
relevant for obtaining audit evidence for the subject matter.
A performance audit may include some checking of the procedures of the audited body, but
you should make sure that the whole audit does not just become a ‘box-ticking’ exercise.
Testing procedures to identify gaps in them does not provide the necessary understanding
for assessing performance. Measuring performance is the process of assessing what the
audited entities do to implement policies. In doing so, you may well need to explain how the
procedures you are checking contribute to a successful outcome. For example, a performance
audit assessing how a Ministry procures vehicles for official use might check that staff follow
procurement procedures. However, it would go on to collect evidence on outcomes, such as:
• How often are the vehicles left unused?
• Did the Ministry pay a fair price for the vehicles?
• Are private businesses able to acquire vehicles more cheaply than the Ministry?
• How can the Ministry reduce the costs of maintaining its vehicles?
• Would it be more cost effective to hire vehicles as and when they are needed?
In a direct reporting engagement, the onus is on you, the auditor, to communicate to the
reader:
• what the objective(s) of the performance audit is (are);
• what criteria you have chosen, and why;
• what evidence you have gathered;
• what strengths and weaknesses exist in performance;
• what has caused the weaknesses and why;
• how compelling the evidence is;
• what conclusion you have reached and why;
• what is the impact or consequence of the finding reported; and
• how much assurance the reader can place on the conclusion.
26
Who are the three parties in a performance audit?
The Standard
The auditor shall explicitly identify the intended users and the responsible parties of the audit and
throughout the audit consider the implication of these roles in order to conduct the audit accordingly.
The three parties in public-sector audits are the auditor, responsible party and intended users.
They may assume distinct characteristics in performance auditing.
The auditor's role is fulfilled by the Head of the SAI and by persons to whom the task of
conducting the audits is delegated (ISSAI 100/25). This definition elapses from the different
SAI models. In the Westminster model, the SAI is usually called National Audit Office and the
reports are signed only by the Auditor General, who takes responsibility for the audit. In the
Court model and Board (or Collegiate) model, auditors conduct audits under the supervision
of management level. Thus, the rules have to be interpreted according to these institutional
designs (TCU, 2020).
Auditors in performance audits typically work in a team offering different and complementary
skills (ISSAI 300/16).
The responsible party may refer to those responsible for the subject matter, for providing the
auditor with information, and also for addressing the recommendations. In performance
audits, this role may be shared by individuals or organisations. A responsible party may also
be an intended user, but it will typically not be the only one (ISSAI 100/25; ISSAI 300/17; ISSAI
3000/27).
Intended users are the individuals, organisations or classes thereof for whom the auditor
prepares the audit report. The legislature, executive, government agencies, third parties
concerned by the audit, and the public are examples of intended users. (ISSAI 100/25; ISSAI
3000/26)
It is important that you, the auditor, consider the needs and interests of the intended users
and responsible parties. It will help the audit report to add value and to be understandable to
these entities. However, this should not undermine your independence and objective attitude
throughout the audit. (ISSAI 3000/28)
27
What is subject matter and subject matter information?
The Standard
The auditor shall identify the subject matter of a performance audit.
Subject matter refers to the information, condition or activity that is measured or evaluated
against certain criteria. The subject matter relates to the question ‘what is audited’ and is
defined in the audit scope, which is the boundary of the audit. The subject matter of a
performance audit may be programmes, undertakings, systems, entities or funds. They may
comprise activities (with their outputs, outcomes and impacts) or existing situations, including
causes and consequences. The subject matter is determined by the audit objective and
formulated in the audit questions. (ISSAI 100/26; ISSAI 300/19; ISSAI 3000/30)
Subject matter information refers to the outcome of evaluating or measuring the subject
matter against the criteria (ISSAI 100/28). In performance audit, it is the auditor who produces
the subject matter information. It is different in a financial audit, where the responsible party
presents the subject matter information (the financial statements). The auditor then obtains
audit evidence to support an opinion. (TCU, 2020)
28