About the Canadian Audit and Accountability Foundation

The Canadian Audit & Accountability Foundation is a premier Canadian research and education foundation.
Our mission is to strengthen public sector performance audit, oversight and accountability in Canada and
abroad. We build capacity in legislative audit offices, oversight bodies, and departments and crown
corporations by developing and delivering:

 Training workshops and learning opportunities;

 Methodology, guidance and toolkits;
 Applied and advanced research;
 Information sharing events and community building initiatives.

Visit us at http://www.caaf-fcar.ca for more information about our products and services.

Practice Guide to Auditing Oversight

© 2013 CCAF-FCVI Inc. (now the Canadian Audit and Accountability Foundation)

All rights reserved. No part of this publication, or its companion products, may be reproduced by any means,
electronic, mechanical, photocopying, recording or otherwise, without the prior written permission of the
publisher.

Published by:
Canadian Audit and Accountability Foundation
100-1505 Laperriere Avenue
Ottawa, Ontario CANADA
K1Z 7T1
Tel: 613-241-6713
[email protected]
http://www.caaf-fcar.ca

ISBN: 978-1-926507-03-3

This publication is available in French under the title:

Guide pratique sur l’audit de la surveillance

2
Acknowledgements
The Canadian Audit & Accountability Foundation’s mission is to promote and strengthen public sector
performance audit, oversight and accountability in Canada and abroad through research, education and
knowledge sharing. To support this, we provide capacity development for public sector auditors and
oversight committees, helping them to work with other public officials for accountable government.

The Practice Guide to Auditing Oversight is part of our performance audit capacity-building program and
is the second in a planned series of such guides. It has been made possible with funding provided by the
Foundation’s members.

The consultative process undertaken for the Practice Guide project was extensive and included
consultations with many leaders and professionals, on both an individual and collective basis, through the
Canadian Council of Legislative Auditors (CCOLA) and other stakeholder networks.

On behalf of our Board of Directors, we acknowledge the support, thought leadership, and active
contributions of the members of the core project team who guided the design and development of this
Practice Guide 1:

 Maryanna Basic, Director Value-For-Money, Corporate Audit Division, BMO Financial Group
 Maria Capozzi, Principal, Strategic Initiatives and Board Governance, Office of the Auditor
General of Manitoba
 Davide Cargnello, Senior Researcher, Institute on Governance
 Gus Chagani, Assistant Auditor General, Office of the Auditor General of Ontario
 Evangeline Colman-Sadd, Assistant Auditor General, Office of the Auditor General of Nova
Scotia
 Vincent Daluz, Chief Audit Executive, Employment and Social Development Canada
 Terry Hunt, Executive Director, Policy and Liaison, Treasury Board of Canada Secretariat
 Trena Keats, Audit Manager, Office of the Auditor General of Newfoundland and Labrador
 Ray Kostuch, Deputy Auditor General, Office of the Auditor General, City of Ottawa
 Larry Munroe, Auditor General of Halifax Regional Municipality
 Paul Nyquist, Director, Office of the Auditor General of British Columbia
 Karl Salgo, Executive Director, Institute on Governance
 Regan Sommerfeld, Principal, Office of the Provincial Auditor of Saskatchewan

We also acknowledge the many other audit professionals who supported this project and provided
comments and suggestions, including:

1
Titles and organizations of individuals included in this publication were those in effect at the time of original publishing. The Canadian
Audit & Accountability Foundation was then known as CCAF-FCVI Inc.

3
  Gordon Beal, Vice-President, Research, Guidance and Support, Chartered Professional
Accountants of Canada
 Régent Chouinard, Principal, Office of the Auditor General of Canada
 Gigi Dawe, Principal, Corporate Governance and Oversight, Chartered Professional Accountants
of Canada
 Judy Ferguson, Acting Provincial Auditor, Provincial Auditor of Saskatchewan
 Serge Giguère, Acting Assistant Auditor General, Auditor General of Quebec
 Juli-Ann Gorgi, Principal, Research, Guidance and Support, Chartered Professional Accountants
of Canada
 Rebecca Yosipovich, Professional Practices Manager, Office of the Auditor General of Ontario

This project and the final Practice Guide would not have been possible without:

 the leadership provided by John Reed 2, CCAF-FCVI’s Vice-President for Performance Audit, who
chaired the core Practice Guide project team;
 the contribution of Pierre Fréchette, CCAF-FCVI’s Research Officer and Lead Author of the Practice
Guide; and
 the support of CCAF-FCVI staff members Julien Raynaud, Project Officer, Lynne Casiple,
Computer Specialist and Webmistress, and James Oulton, Corporate Officer.

Finally, we would like to extend our thanks to the individuals who contributed their skills to the process
of producing all the material for the Practice Guide website: Nicole Plamondon (translation), Laurel
Hyatt (editing) and Paul Edwards (graphic design).

Strong oversight is important to the success of every public sector organization in both delivering public
services effectively and in promoting accountable government. This Practice Guide aims to assist public
sector auditors in designing, carrying out and reporting performance audits of oversight practices. We
hope you find it helpful and we welcome your feedback.

Brian Bost, Chair, CCAF-FCVI Board of Directors

Paul Lohnes, President and CEO, CCAF- FCVI

2
Comments, suggestions and new ideas can be provided to John Reed at the Canadian Audit & Accountability Foundation
([email protected])

4
Table of Contents
Purpose of the Practice Guide to Auditing Oversight ...................................................... 6
Part 1 Concepts and Context ......................................................................................... 7
What Is Oversight and How Does it Relate to Governance? ........................................................8
What Is Oversight? .......................................................................................................................... 8
What Is Governance? ..................................................................................................................... 10
The Importance of Effective Oversight ....................................................................................13
Oversight Bodies and Functions ..............................................................................................16
The Hierarchy of Oversight Responsibilities................................................................................. 16
Oversight Bodies ........................................................................................................................... 20
Oversight Functions ...................................................................................................................... 23
Oversight of Public Agencies, Boards and Authorities ................................................................. 25
Oversight of Major Initiatives in Departments and Ministries ..................................................... 26
Part 2 Audit Methodology ........................................................................................... 28
Introduction to Auditing Oversight .........................................................................................29
Selecting an Audit Topic .........................................................................................................31
Determining the Degree of Focus on Oversight ........................................................................34
Planning an Audit of Oversight of a Public Agency, Board or Authority .................................... 36
Acquiring Knowledge of Business and Assessing Risk ................................................................. 37
Determining the Audit Approach .................................................................................................. 43
Drafting Audit Objectives.............................................................................................................. 44
Selecting Audit Criteria ................................................................................................................ 48
Planning an Audit of Oversight of a Major Initiative in a Department....................................... 56
Acquiring Knowledge of Business and Assessing Risk ................................................................. 57
Determining the Audit Approach .................................................................................................. 62
Drafting Audit Objectives.............................................................................................................. 63
Selecting Audit Criteria ................................................................................................................ 67
Conducting the Examination Phase .........................................................................................73
Documentary Evidence ................................................................................................................. 76
Testimonial Evidence .................................................................................................................... 76
Physical Evidence ......................................................................................................................... 78
Analytical Evidence....................................................................................................................... 79
Reporting the Results of an Audit of Oversight ........................................................................81
Setting the Context ....................................................................................................................... 82
Audit Observations........................................................................................................................ 83
Recommendations ........................................................................................................................ 84
References ................................................................................................................. 85
Guidance and Good Practices ..................................................................................................85
Audits and Public Accounts Committee Reports Cited in the Practice Guide ............................ 86
Other References on Governance and Oversight ......................................................................87
Glossary ..................................................................................................................... 90

5
Purpose of the Practice Guide to Auditing Oversight
The purpose of this Practice Guide is to provide contemporary guidance for public sector auditors, both
internal and external, on how to select, plan, carry out, and report on performance (or value-for-money)
audits of oversight bodies and functions.

Some audit offices have published informative material about governance and oversight best practices and
principles. However, little practical guidance on how to audit oversight is readily available. This Practice Guide
aims to fill this gap: it includes guidance for each phase of the audit process, as well as examples of audit
objectives and criteria for different types of oversight bodies and functions.

Scope of the Practice Guide

This Practice Guide’s focus is on oversight in the public sector. Specifically, the Practice Guide includes
guidance on how to audit oversight bodies and functions responsible for:

 oversight of public agencies, boards and authorities (such as Crown agencies, school boards, and
health authorities) and
 oversight of major initiatives in departments and ministries (critical programs, projects, services, or
horizontal initiatives managed internally or outsourced to a private sector provider).

The Practice Guide does not discuss parliamentary oversight and does not include guidance on how to audit
day-to-day management controls in departments and agencies. However, auditors may find that some if its
contents (objectives, criteria, etc.) can be used or modified to audit different oversight mechanisms in
ministries and departments.

Using the Practice Guide

The Practice Guide is a flexible tool to be used within each audit office’s existing processes and procedures, in
accordance with existing auditing and assurance standards. It is therefore a complement to current audit
methodology.

Readers of the Practice Guide do not have to read all its sections in order. Rather, the Guide has been
designed to provide easy access to any section of interest and to allow readers to jump rapidly from one
section to any other. Auditors are thus free to consult only the sections that best meet their needs.

Back to Table of Contents

6
Practice Guide to Auditing Oversight

Part 1 Concepts and Context

7
What Is Oversight and How Does it Relate to Governance?
Definitions of “oversight” and “governance” vary across public and private sector organizations, but they
share many similar elements. This Practice Guide recognizes that oversight is a component (or subset) of good
governance and adopts definitions of these terms suited to public sector organizations.

What Is Oversight?
Oversight refers to the actions taken to review and monitor public sector organizations and their policies,
plans, programs, and projects, to ensure that they:

 are achieving expected results;

 represent good value for money; and
 are in compliance with applicable policies, laws, regulations, and ethical standards.

Oversight is a critical governance function performed by boards of directors, committees, councils, and
external bodies.

Oversight is composed of “over,” meaning above, and “sight,” meaning looking, but not touching.
Indeed, those in charge of oversight functions are asked to look at a process, program, or project from
above, but not to get involved in its day-to-day management.

In other words, oversight (or watchful care) is a safety net to ensure the following:

 Due diligence takes place before key decisions are made.

 Policies and strategies are being implemented as intended.
 Key risks are identified, monitored, and mitigated.
 Business processes and systems are working well.
 Expected results are being achieved.
 Value for money is obtained.
 Activities comply with policies, laws, regulations, and ethical standards.
 Developing areas of concern are being dealt with.
 Assets are being safeguarded.
 Continuous improvement is taking place.

8
In practice, oversight can be conducted through various functions, including:

 Planning
 Defining information needs
 Challenging
 Advising
 Approving
 Deciding
 Monitoring
 Reviewing
 Taking corrective action

Different oversight bodies will fulfill different oversight functions, in accordance with their specific mandates.
Some oversight bodies will play a more active role in guiding management than others, while still staying
away from day-to-day management of the organization’s activities.

Back to Table of Contents

9
What Is Governance?
While governance includes oversight, it is a broader concept. Governance refers to the structures, systems,
and practices an organization has in place to:

 assign decision-making authorities, define how decisions are to be made, and establish the
organization’s strategic direction;
 oversee the delivery of its services; the implementation of its policies, plans, programs, and projects;
and the monitoring and mitigation of its key risks; and
 report on its performance in achieving intended results and use performance information to drive
ongoing improvements and corrective actions.

A simplified governance framework is presented in Figure 1.

Figure 1 – A Simplified Governance Framework

10
Much has been written about what constitutes good governance, and “good practice” guides have been
published in recent years by a number of organizations, including audit offices. (See, for example, the
Australian National Audit Office’s 2014 Better Practice Guide Public Sector Governance: Strengthening
Performance Through Good Governance.)

While this Practice Guide does not explore all aspects of governance in the public sector, it is useful to
highlight the basic principles that support good governance, and therefore oversight, too.

The basic principles of good governance are:

 Accountability
 Leadership
 Integrity
 Stewardship
 Transparency.

These five principles are briefly defined in Figure 2.

Figure 2 – Principles of Good Governance

Accountability is the obligation of an individual, a group, or an organization to answer for a
responsibility that has been conferred.

Leadership is setting the “tone at the top,” which plays a crucial role in encouraging an organization’s
personnel to embrace good governance practices.

Integrity is acting in a way that is impartial, ethical, and in the public interest. Integrity is reflected in
part through compliance with legislation, regulations, and policies, as well as through the instilling of
high standards of professionalism at all levels of an organization.

Stewardship is the act of looking after resources on behalf of the public and is demonstrated by
maintaining or improving an organization’s capacity to serve the public interest over time.

Transparency is achieved when decisions and actions are open, meaning that stakeholders, including
the public and employees, have access to full, accurate, and clear information on public matters.

Source: Modified from Public Sector Governance: A Guide to the Principles of Good Practice, Office of the Auditor General of
British Columbia.

It is also useful for auditors to have a clear understanding of the distinct roles played by both oversight bodies
and management. As a general principle, the roles of an oversight body should be segregated from those of
management. To illustrate this principle, Table 1 presents the usual roles of boards of directors and
management in public sector agencies, boards and authorities.

11
Oversight bodies are expected to play their respective roles without getting involved in the organization’s
day-to-day management. Members of oversight bodies should also be independent from management in
order to avoid real or perceived conflicts of interest.

Table 1 – The Separate Roles of Boards of Directors and Management

Board’s Roles Management’s Roles

 Select, evaluate, and enable the CEO  Manage the organization in line with board
direction
 Keep the board informed
 Seek the board’s counsel

 Approve strategic organizational goals and  Recommend goals and policies supported by
policies relevant information

 Make strategic decisions  Frame decisions in the context of the

organization’s mission and strategic vision, and
provide the board with well-documented
recommendations

 Establish appropriate risk tolerance levels  Conduct risk assessments, mitigate and monitor
risks, and provide the board with regular
updates on key risks to the organization

 Oversee management and organizational  Provide the board with transparent, complete,
performance timely information in concise, contextual, or
comparative formats
 Be responsive to requests for additional
information

Source: Adapted from B. S. Bader (2008), “Distinguishing Governance from Management,” Great Boards, Vol. VIII, No. 3.

Back to Table of Contents

12
The Importance of Effective Oversight
Effective oversight is important to the success of every public sector organization. There are numerous
oversight bodies within the public sector at the national, provincial, and municipal levels, each of which plays
a role in ensuring that public services are delivered effectively, efficiently, and with due regard for economy.

The importance of strong oversight processes has been illustrated in recent times by a number of high-profile
cases where weak oversight resulted in serious adverse consequences. The weak oversight of financial
institutions in the United States, which contributed (among other factors) to the 2008 global economic crisis,
is a well-known example. In Canada, a number of oversight weaknesses in the public sector have also been
the subject of substantial media coverage (see Figure 3).

Figure 3 – Canadian Examples of Oversight Weaknesses

Air ambulance services in Ontario

In March 2012, the Auditor General of Ontario released a special report on Ontario’s air ambulance services
delivered by Ornge, a not-for-profit provincial corporation created in 2005. The report highlighted a series
of irregular financial transactions, performance issues, and a lack of oversight of the corporation’s activities
by the Ministry of Health and Long-Term Care. Over time, Ornge had created, with the approval of its
board, a network of subsidiary companies that were not subject to the performance agreement signed
between Ornge and the Ministry. The Ministry’s ability to obtain the information it needed to fulfill its
oversight responsibilities in relation to Ornge and its Board was therefore hindered. Furthermore, the Board
did not request the Ministry’s perspective on several significant strategic decisions and failed to take
appropriate actions to investigate questionable transactions.

In 2014, a Standing Committee on Public Accounts summary report concluded that:

“the matters identified in the Auditor General’s report could be attributed primarily to the absence of
due diligence and oversight on the part of the Ministry of Health and Long-Term Care in applying a
robust accountability framework, the lack of transparency and accountability on the part of Ornge’s
management and Board of Directors, compounded by systemic operational issues, as well as
shortcomings in Ornge’s first Performance Agreement.”

Rail safety in Canada

Rail safety issues were brought to the fore by the July 2013 tragedy in Lac-Mégantic, Quebec, in which 47
people died after a runaway train carrying crude oil exploded in the town’s centre. The Transportation
Safety Board of Canada investigation that followed the tragedy found many contributing causes, including
insufficient monitoring and oversight of railway management safety systems by Transport Canada.

In November 2013, the Auditor General of Canada released an audit report on rail safety oversight that also
pointed to oversight weaknesses at Transport Canada. Among other findings, the report noted that the
Department had conducted only 26 percent of the audits of railway safety management systems its own
policy required over the period covered by the audit. The audit concluded that the Department had not
exercised enough oversight over safety management systems.

13
The examples in Figure 3 are only some of the most prominent recent examples; there are many more and
the media regularly bring new ones to the public’s attention. These oversight weaknesses (and other factors)
have led to reforms in both the public (for example, the Federal Accountability Act and Quebec’s Act
respecting the governance of state-owned enterprises) and private sectors (for example, Securities Act
reforms in Ontario) aimed at increasing the accountability of directors and executive managers, as well as
strengthening internal audit functions. In the federal government, for example, departmental audit
committees with external members have been created in the aftermath of the federal sponsorship scandal.

The importance of effective oversight has also been heightened in many jurisdictions where governments
have divested themselves of direct program delivery in a number of sectors by:

 delegating the delivery of programs and services to newly created agencies, boards or authorities; or
 outsourcing the delivery of programs, services or capital projects to private sector partners, through
public-private partnerships or other types of contractual agreements.

Public sector spending happens increasingly outside traditional models of departmental/ministerial

accountability and governance, a situation that may create new risks that must be managed and mitigated.
To manage these emerging risks, public sector organizations have had to adapt in order to maintain or
improve their oversight effectiveness. New governance and oversight arrangements have been developed to
meet the needs of new situations, recognizing that where there is a need for effective governance, there is
also a need for strong oversight.

At the same time, however, a troubled economic situation has created budgetary constraints that have
significantly affected the management of many public services and programs. Concerns have been expressed
among deputy ministers and other public officials about the resources required to support existing oversight
mechanisms and fulfill reporting requirements.

To address the challenge of maintaining strong oversight processes in programs facing budgetary and staff
reductions, public sector organizations need to ensure that they put in place a mix of oversight processes that
strikes the right balance between risk, control, efficiency, and cost. Not doing so can unduly expose an
organization to serious risks or, on the contrary, burden it with unnecessary processes and costly internal red
tape that focus on process instead of results.

By conducting audits of oversight bodies and functions, internal and legislative auditors can play an important
role in helping public sector organizations to achieve this balance between risk and controls, efficiency and
costs. Through their reports, auditors can:

 identify the causes of breakdowns in oversight (audits of oversight are often conducted after a
significant failure, crisis, or scandal);
 highlight weaknesses and inefficiencies in oversight regimes (thus helping auditees to prevent
breakdowns in oversight);

14
  point to best practices;
 make recommendations for improvements; and
 help departments, agencies, boards and authorities to improve their oversight performance and
avoid repeating past mistakes.

Ultimately, conducting audits of oversight bodies and functions is an important manner in which audit offices
can fulfill their mandate to provide their clients (legislative assemblies, audit committees, or others) with
independent information, advice, assurance, and recommendations regarding the stewardship of public
funds.

Back to Table of Contents

15
Oversight Bodies and Functions
The Hierarchy of Oversight Responsibilities
Oversight responsibilities in public sector organizations do not all rest with a single body or a single
hierarchical level. Rather, they can be distributed at different levels.

This Practice Guide focuses on high-level oversight responsibilities in two specific situations (public agencies,
boards or authorities and major initiatives in ministries and departments) and does not include in-depth
consideration of day-to-day management and controls (that is, at the operational and tactical levels). A brief
overview of other situations is presented below to illustrate the diversity of oversight roles in public sector
entities.

For example, within a Crown corporation or agency, a board or an authority, the various oversight
responsibilities are distributed among different managers, functions, and bodies:

 Operational line managers—These managers oversee business operations in which day-to-day

transactions are entered and processed.
 Tactical oversight functions—These functions are centralized competence centres, like finance,
risk management, compliance, and human resources. These tactical oversight functions monitor,
facilitate, and coordinate the activities of business lines, to ensure they are operating effectively,
within budget, and in compliance with corporate policies.
 Executive management—These managers are responsible for running and overseeing the business
of the organization and developing corporate strategies for approval by the board of directors. They
are also expected to provide representations to the board of directors to the effect that the
organization’s objectives are being achieved.
 Board of directors—The board is responsible for governing the organization and overseeing its
activities and the performance of executive management in implementing corporate strategies.

In addition, Crown corporations and other similar organizations are subject to oversight by their responsible
Minister and may also be overseen by one of several independent regulatory agencies, depending on the
economic sector and jurisdiction in which they operate.

For example, Figure 4 illustrates a situation where a provincial Minister of Health has oversight responsibility
for a number of regional health authorities, which themselves have oversight responsibilities for the activities
of several hospitals in their respective region. In this case, the chair of the board of each of these regional
health authorities is accountable to the Minister. In Figure 5, a regulatory agency, in this case an energy
board, has oversight responsibilities for a number of energy producers, distributors and retailers. The energy
board itself is overseen by a Minister.

Similarly, central agencies have their own oversight responsibilities, focusing on the implementation of key
policies by a jurisdiction’s departments and agencies. Figure 6 presents an example where a central agency
has oversight responsibilities for the implementation of a policy on the management of major Crown
projects.

16
Finally, internal oversight mechanisms can be established for major initiatives within departments or agencies.
Figure 7 illustrates a case where a special committee composed of assistant deputy ministers and a deputy
minister has been set up to oversee a major initiative. This committee is independent from the day-to-day
management of the initiative and is accountable to the department’s Minister.

Figure 4 – Example of Ministerial Oversight for Public Agencies, Boards, or

Authorities

In major initiatives (critical public sector projects, programs, and services), oversight responsibilities covered in
the Practice Guide are those exercised by the minister and by any special oversight body put in place to
oversee the project, program, or service.

In public agencies, boards and authorities, oversight responsibilities covered in the Practice Guide are those
exercised by the board of directors or equivalent oversight body, as well as those of the minister charged with
overseeing the board of directors or oversight body.

17
These two distinct situations are covered in more detail in the Concepts and Context part of this Practice
Guide and are also the main focus of the Audit Methodology part. Although the Practice Guide does not
provide guidance specifically tailored for auditing the oversight of regulatory agencies that are not governed
by a board of director or a similar oversight body, auditors interested in this topic can use and modify the
examples (indicators, questions, objectives, criteria) provided in the Practice Guide to meet their own needs.
Some of the guidance may also be useful as a starting point to audit common oversight mechanisms in
ministries and departments.

Figure 5 – Example of a Regulatory Body and its Oversight

18
Figure 6 – A Central Agency and its Oversight of the Implementation of a
Government-Wide Policy on Major Crown Projects

Figure 7 – A Special Oversight Committee Established Within a Department to

Oversee a Major Initiative

19
Oversight Bodies
Oversight responsibilities exist in all public sector organizations and are assigned to managers and personnel
working under a variety of governance structures. While some structures are common and well regulated,
like the boards of directors of Crown corporations and agencies, others are more ad hoc and are subject only
to internal rules.

To add to the complexity of this situation, the nomenclature used to describe oversight bodies is far from
consistent. The term “board,” for example, is not restricted to the boards of directors of public or private
corporations. It can also refer to administrative tribunals, regulatory agencies, investigative and advisory
bodies, operational organizations, and organizations mandated to manage public monies. It is therefore
important for public sector auditors to go beyond names and to clearly define the characteristics of the
oversight bodies they may choose to audit.

In general terms, an oversight body is a group of people with a common oversight purpose acting as an
organized unit. In this Practice Guide, emphasis is put on oversight bodies that have:

 a discrete structure,
 a degree of independence, and
 a clear oversight mandate.

An oversight body may also be an organization’s governance body (a board of directors, for example), or it
may be a committee or other structure that reports directly to the governance body (an audit committee, for
example).

The board of directors of a Crown corporation, the governing body of a local health or education authority,
and a regulatory board for a specific economic sector would all meet the definition of an oversight body.
However, the Practice Guide recognizes that it is possible to audit oversight even when some of these
conditions (discrete structure, independence, and clear mandate) are not met. While some sections of the
Practice Guide may be less applicable in such cases, other sections will be easily adaptable.

Each oversight body has its own unique characteristics. What unites oversight bodies is the nature of their
relationship with their overseen organization and the oversight functions they play. As shown in Figure 8,
the relationship between oversight bodies and overseen bodies is mediated through the exchange of
information from one body to the other. The oversight body communicates its information needs to the
overseen body and the latter provides the required information in return, thus fulfilling its accountability
obligation.

This exchange of information can take place at more than one level. Depending on reporting relationships, an
organization may report to a second organization (a regulatory agency or a health authority, for example),
which itself reports to a third organization (a department or Parliament/legislature). In this case, the second
organization is both an oversight body (overseeing the first organization) and an overseen body (overseen by
the third organization).

20
Figure 8 – Information Flow Between Oversight and Overseen Bodies

The expression “oversight of oversight” can be used to describe such situations where an oversight body
oversees another oversight body. Figures 4, 5, and 6 each illustrate examples of oversight of oversight. In
Figure 4, for example, the Regional Health Authorities are both oversight bodies and overseen bodies. Note,
however, that the oversight relationships illustrated in these diagrams are not necessarily hierarchical; some
are horizontal. p

The role of “oversight of oversight” is an important one, especially where ministers are responsible for
overseeing agencies, boards or authorities. This was recognized by Ontario’s Standing Committee on Public
Accounts in its 2014 report on the Ornge air ambulance service:

“The Committee notes that the events at Ornge confirm that it is not responsible simply to rely
on the boards of transfer agencies to provide appropriate oversight. The Ministry must exercise
its responsibility to ensure that public funds are being properly administered and that boards
are held accountable for their actions.”

In recent years, many audits have reported significant weaknesses in the oversight of certain agencies, boards
or authorities by their responsible minister. Examples include the 2011 New Brunswick audit of the oversight
of wastewater commissions, the 2009 Ontario audit of the Electronic Health Records initiative, and the 2012
Ontario audit of the Ornge air ambulance service.

21
Audits can therefore look at oversight at different levels within a single audit:

 oversight by a board or another body of an organization’s activities,

 oversight of this board or body by its responsible minister, and
 oversight of the organization’s activities by an independent regulatory agency.

Back to Table of Contents

22
Oversight Functions
Oversight bodies are created to fulfill specific mandates. The list in Table 2 defines the main functions
exercised by oversight bodies to fulfill their oversight mandate. The functions are categorized according to
the part of the Plan, Do, Check, Act management model they belong to. (While some of the functions may
be exercised in more than one stage of the management cycle, the table has been simplified.)

Often, an oversight body needs to exercise many functions in order to provide adequate oversight of a single
process. For example, boards of directors and other oversight bodies usually play many oversight functions in
relation to corporate risk management. They can approve risk management policies, make decisions on risk
tolerance levels, review risk profiles, monitor the implementation of risk assessment processes, and
communicate information on corporate risks.

However, not all oversight bodies will exercise all functions. Each oversight body's functions are defined in its
mandate. It is therefore important that auditors have a good understanding of the oversight mandate of the
organization(s) they have decided to audit.

It is also important to note that even where there is no discrete, independent oversight body responsible for
overseeing a major initiative, it can still be reasonably expected that the functions presented in Table 2 would
have to be exercised somehow. In other words, all major initiatives should have effective governance and
oversight. In such situations, auditors could use the oversight functions as a starting point to develop their
audit criteria.

Table 2 – List of Oversight Functions

PLAN Functions

1. Planning Determining how and when oversight actions will be taken by the oversight
body

2. Defining Defining what information is needed by the oversight body to fulfill its
information needs responsibilities

DO Functions

3. Challenging Requesting an explanation or justification; calling into question

4. Advising Offering suggestions about the best course of action to adopt

5. Approving Officially agreeing to or accepting something as satisfactory (or in compliance)

6. Deciding Coming to a resolution after having considered relevant factual information

and potential options

23
 CHECK Functions

7. Monitoring Maintaining regular, systematic surveillance over a process, system, program,

project, or service, and comparing performance against expectations

8. Reviewing Formally examining or assessing some aspects of an organization with the

possibility or intention of instituting change if necessary. This may include
reviewing:

 audit reports,
 evaluation reports, and
 investigation reports.

ACT Functions

9. Taking corrective Taking actions to correct an observed deficiency once its cause has been
actions identified, either directly, by adopting a new rule or policy or amending an
existing one, or indirectly, by ensuring that management effectively implements
adequate measures.

In addition to the functions in Table 2, oversight bodies can play other important roles, including facilitating
continuous improvement, setting the tone at the top, communicating key decisions, and indicating the
preferred behaviour and values (through a code of conduct) that are to be adopted and demonstrated by an
organization’s personnel. As with the functions listed in Table 2, these roles could be audited.

Back to Table of Contents

24
Oversight of Public Agencies, Boards and Authorities
Federal and provincial departments and ministries play significant roles in the delivery of services to
Canadians. However, in the last several decades, there has been a trend in Canada to decentralize the
management of many public services and to delegate the responsibilities for these services to agencies,
boards and authorities (or “distributed governance organizations”). As a result, there are now hundreds of
agencies, boards and authorities in provinces and at the federal level.

These organizations share several common characteristics:

 They are established by the government, but are not part of a ministry.
 They are accountable to the government.
 They were assigned or delegated authority and responsibility by the government, or otherwise have
statutory authority and responsibility to perform a public function or service.

Common examples of agencies, boards and authorities include school boards, health authorities, and Crown
corporations and agencies. Important public services like health care, education, energy production, and
public transportation are delivered every day by agencies, boards and authorities.

These organizations are often (but not always) governed by a board of directors or governing council—an
oversight body modelled after the boards of directors of publicly traded corporations. As in the private sector,
the board or council is responsible for overseeing the organization’s activities.

The boards of directors and governing councils of these agencies, boards and authorities are usually
independent from the management of the organization they oversee and are granted the power to exercise
all or most of the oversight functions listed in the Concepts and Context part of the Practice Guide. In
particular, the board often has an audit committee, which plays an important role in overseeing financial and
performance reporting, compliance, and related controls.

The Audit Methodology part of the Practice Guide includes guidance for auditing the oversight of agencies,
boards and authorities governed by a board or council.

Back to Table of Contents

25
Oversight of Major Initiatives in Departments and Ministries
In addition to creating and delegating to various government agencies, boards and authorities the delivery of
some public services, federal and provincial departments and ministries also retain responsibility for the
delivery of other critical public services, social programs, and capital projects. This Practice Guide collectively
refers to these “major initiatives” (in contrast to more routine programs) and where they exist, stresses the
need for effective governance to ensure the delivery of value for money. As stated previously, where there is
a need for effective governance, there is also a need for strong oversight.

Major initiatives managed by departments and ministries that may warrant special or specific oversight
mechanisms can include:

 large, complex procurement or capital projects (such as transit projects, bridges, and hospitals);
 projects and services outsourced to private sector providers, through traditional contracts or through
public-private partnerships (such as ambulance services, and construction and maintenance of
schools, hospitals, and highways); and
 government-wide initiatives that involve large sums of public money (such as economic stimulus
programs and public safety initiatives).

There is no single definition of what constitutes a major initiative. Each audit office has to define what this
term means in its own context and exercise professional judgment in determining if special or specific
oversight is warranted (whether in place or not). In general, major initiatives will involve a department
managing (or outsourcing the implementation of) a program, project, or service of direct benefit to the
public, as opposed to projects or services of benefit to the department itself (exceptions may be warranted,
however, in the case of high value and/or high risk projects).

In instances where strong governance and oversight are especially important, it is common for governments
to create special governance structures that share several characteristics:

 a discrete body composed of a number of senior officials (for example, assistant deputy ministers,
deputy ministers, and ministers);
 a clear oversight mandate; and
 a degree of independence (no involvement in the day-to-day management of the overseen program,
project, or service).

For example, in the case of the National Shipbuilding Procurement Strategy, which included plans to spend
more than $50 billion over 30 years to recapitalize fleets of Navy and Coast Guard ships, a committee of
assistant deputy ministers was charged with overseeing the Strategy’s development and implementation. (See
the OAG Canada audit on this topic.)

26
Similarly, in the context of Canada’s Economic Action Plan, Infrastructure Canada instituted a project review
panel composed of the Associate Deputy Minister and assistant deputy ministers to review all Infrastructure
Stimulus Fund projects recommended by program staff before forwarding them to the Minister for final
approval. (See the OAG Canada audit on this topic.)

In the context of preparing for the 2015 Pan American Games and the Parapan American Games to be
hosted in Toronto, the Province of Ontario has established a Security Budget Oversight Committee to oversee
the budgets and costs of the large security operations that will be necessary to ensure safety during the
games. This committee includes senior officials from two provincial ministries and from the Pan/Parapan
American Games Secretariat. (See the special report of the Auditor General of Ontario on this topic).

Given their significance, programs, projects, and services overseen by special oversight committees (or similar
structures) will often be of interest to auditors. The Audit Methodology part of this Practice Guide includes
guidance for auditing the oversight of major initiatives in departments and ministries.

Back to Table of Contents

27
Practice Guide to Auditing Oversight

Part 2 Audit Methodology

28
Introduction to Auditing Oversight
Audits of oversight follow the same standards and general process as all performance audits. Auditors are
required to follow the standards and audit processes applicable to their body of practice and office mandate.

An overview of the generic audit process is in Figure 9.

Figure 9 – Overview of the Performance Audit Process

The diversity of governance structures in any jurisdiction’s public sector and the diversity of oversight
functions in any organization mean that auditors will rarely be able to apply the same audit plan to different
organizations. However, auditors can apply a common methodology to plan all their audits of oversight.

When undertaking an audit of oversight, auditors will need to:

 select a significant oversight topic (for example, oversight of food safety, oversight of major capital
projects) to audit and
 select one or more organizations to audit and develop a very good understanding (“knowledge of
business”) of each audited organization’s governance structure, oversight responsibilities, strategic
direction, and performance expectations.

29
Once these decisions are made, auditors will need to determine the extent of focus that the audit will place
on oversight:

 Should the audit deal exclusively with oversight (that is, should it be a “stand-alone” audit of
oversight) or should oversight be part of an otherwise broader performance audit where oversight is
only one of the topics covered by the audit?

Auditors will also need to decide whether their audit approach will be to look at:

 the structures and systems of oversight bodies and functions or

 the results and effectiveness of these bodies and functions.

Alternatively, auditors could decide to combine both of these approaches in the audit in order to provide a
more complete assessment of oversight responsibilities.

In addition to identifying the topic, focus, and approach of the audit, auditors will need to prepare a detailed
audit plan that includes audit objectives, audit criteria, and audit procedures.

This Practice Guide provides information and guidance that will help auditors to complete the successive
steps involved in planning, conducting, and reporting the results of an audit of oversight. This guidance will
be especially useful to auditors who wish to audit:

 oversight of agencies, boards and authorities; and

 oversight of major initiatives in departments and ministries.

The Practice Guide also includes a glossary and a list of references (with hyperlinks to quickly access audit
reports and other relevant documents on oversight).

Back to Table of Contents

30
Practice Guide to Auditing Oversight

Selecting an Audit Topic

The first step in the performance audit process is to select a topic. The specific practices and criteria used to
select audit topics vary from one office to another.

In some cases, audits are mandated by legislation, like the special examinations of federal Crown
corporations under the Financial Administration Act. In other cases, a special request may be made by a
legislature or a minister for an auditor general to conduct a particular audit (as was the case for the 2011
New Brunswick audit of the oversight of wastewater commissions). These requests are often made after a
significant negative event has occurred, with a view to identify the cause and prevent a reoccurrence.

But, in most cases, internal and legislative audit offices in Canada have the flexibility to choose (or at least
propose) their own audit topics. Often, selection of audit topic is done as part of the office's strategic
planning process. The selection process usually involves senior audit executives who make decisions based on
information generated by a risk analysis of some sort (or other method) as well as consideration of any
constraints imposed by the audit’s timing, available resources and skills, and the auditability of the topic. In
some offices, a senior auditor may have the responsibility to select an audit topic (or at least propose one for
approval).

Canadian Audit & Accountability Foundation – www.caaf-fcar.ca 31

This Practice Guide suggests that consideration of the importance of oversight may also influence audit topic
selection. Further, activities related to acquiring knowledge of business and assessing risk are typically applied
to both audit topic selection (see below) as well as in detailed planning of a performance audit (described in
subsequent sections of the Practice Guide), albeit at different levels of detail.

Given that there are oversight responsibilities in every public sector organization, it is unlikely that
offices /senior auditors would first decide to audit oversight and then undertake an analysis to determine in
which department or agency this would be most relevant. Rather, it is far more likely that they would already
have in mind a specific organization, program, or horizontal issue (one for which responsibilities are spread
across several departments). In that instance, the office's or senior auditors’ main task would be to determine
if the audit should cover oversight responsibilities in the chosen organization(s), program, project, or public
service.

In order to make this determination, audit teams will need to:

 develop a preliminary knowledge of business,

 assess the importance of proper oversight to the attainment of stated organizational objectives, and
 assess whether there are indications that oversight has been ineffective and has put the achievement
of these objectives at risk.

There are many indicators that oversight may be weak, including:

 significant cost overruns, delays, high numbers of complaints, escalating risks, and poor performance
against targets;
 irregular board or committee meetings, poor (i.e. absent, incomplete, ambiguous or
inaccurate)documentation to support key decisions, and lack of performance information; and
 failure to take corrective actions or to make significant progress in relation to previous audit
observations and recommendations.

Auditors can look for these and other signs, document them, and then use this information as part of their
analysis to determine whether oversight is an important risk factor for the success of the project, program, or
organization they want to audit.

Among the questions to consider in making this determination are:

 Would weak oversight prevent the organization from achieving its objectives or adequately carrying
out its mandate?
 Would weak oversight result in significant adverse consequences for the organization, its clients, or
the public?

32
In situations where oversight is an important risk factor, auditors should consider including one of more lines
of inquiry on oversight in their audit plan. Lines of enquiry can focus on either:

 the design of oversight structures and systems (oversight body structure, mandate, roles and
responsibilities, independence, skills and experience requirements, and so on) or
 the results and effectiveness of these structures and systems (performance in delivering oversight
mandate; compliance with laws, regulations, and bylaws; performance monitoring; reporting; and so
on).

Back to Table of Contents

33
Determining the Degree of Focus on Oversight
When planning a performance (value-for-money) audit that will integrate oversight questions, auditors will
need to decide on the focus of the audit. Focus relates to the level or degree of attention given to oversight
in a performance audit.

There are many ways in which a performance audit can integrate oversight considerations. Some audits will
focus exclusively on oversight while others will only cover oversight as a secondary topic. This varying level of
effort and focus directed at oversight can be thought of as a spectrum (see Figure 10) along which are
different categories, from “marginal or no focus” to “exclusive focus”:

 Marginal or no focus—There is no formal plan to audit oversight, but the issue comes up during an
audit (for example, weak oversight is identified as the root cause of a performance problem).
 Non-specific focus—Some audit steps touch on oversight even though there is no specific oversight
criterion.
 Specific focus—Structured audit work on oversight is part of a larger audit. Oversight can be a line
of enquiry among others or elements of oversight are looked at under lines of enquiry that primarily
focus on other matters.
 Exclusive focus—This is an audit focused exclusively on oversight (a stand-alone audit of oversight).

Figure 10 – The Spectrum of Audits of Oversight

Since oversight is a subset of governance, oversight is often audited in the context of governance audits. For
example, the Office of the Auditor General of British Columbia has conducted audits of Crown agency board
governance (2012) and university board governance (2014) that included examining the oversight
responsibilities of selected boards of directors, among other governance aspects. These audits focused
exclusively on governance and oversight.

34
In other instances, oversight is included in an audit as part of a line of enquiry on governance, but
governance is only one of several topics being examined. The special examinations of federal Crown
corporations conducted by the Office of the Auditor General of Canada follow this model. In these audit
engagements, auditors are asked to provide assurance that a corporation’s assets are safeguarded, its
resources are managed economically and efficiently, and its operations are carried out effectively. In addition
to governance, special examinations include other lines of inquiry on important corporate areas like human
resources, financial management, performance measurement, and environmental management. The focus on
oversight in special examinations is therefore limited by the requirement to provide assurance on a broad
range of significant corporate activities.

Finally, sometimes oversight issues surface in audits where this topic (or governance) was not initially included
in the audit scope. In such cases, audit teams will need to modify their scope, seek the necessary internal
approvals, and inform the audited organization’s management as appropriate.

Back to Table of Contents

35
Planning an Audit of Oversight of a Public Agency, Board or Authority

This section of the Practice Guide is organized according to the key actions and decisions that need to be
made during the planning phase of the audit process:

§ Acquiring knowledge of business and assessing risk

§ Determining the audit approach
§ Drafting audit objectives
§ Selecting audit criteria

Although these topics are presented in a specific order, planning a performance (value-for-money) audit is
rarely a linear process. In fact, the planning process is often iterative, with decisions in one step requiring the
audit team to review decisions made in previous steps to ensure the audit plan’s overall coherence.

36
Acquiring Knowledge of Business and Assessing Risk
Auditing procedures typically require auditors to acquire knowledge of the organization and subject matter
being audited and to prepare a risk-based audit plan. In practice, this means that the audit team needs to:

 collect knowledge of business information about the governance structure of selected agencies,
boards or authorities, especially their oversight bodies and functions, and
 identify significant areas that would benefit from an examination of oversight.

As in all performance (value-for-money) audits, the auditor’s understanding of significance and risks will be
used to identify particular programs or areas to include in the audit and to develop audit objectives. This
section of the Practice Guide is designed to help auditors acquire a sound understanding of significance and
risks by providing them with examples of:

 general audit questions that can be used to better understand oversight roles and responsibilities in
public sector organizations and
 indicators that oversight may be at risk in a program, project, or organization.

While these tools will be helpful to auditors, they should keep in mind that the Practice Guide does not
foresee all possible situations. Applying professional judgment and knowing the particularities of each
selected organization are key success factors for the planning phase of any audit of oversight.

Acquiring knowledge of business

The early stage of planning a performance audit requires that auditors develop a sound understanding of the
nature, objectives, and activities of the organization or organizations that will be audited. This involves
obtaining basic information on an organization’s mandate, organizational structure, accountability
relationships, programs, resources, key risks, past performance, and so on. It also means gathering more
detailed information on specific systems and practices in areas that auditors are particularly interested in,
including oversight.

Since oversight is a subset of governance, it is usually beneficial for auditors who want to focus on oversight
to first develop a good understanding of the full governance structure of an agency, a board or an authority.
This includes obtaining information on the structure and operation of the board of directors (or governing
council) and all its committees. Table 3 provides a list of questions that auditors can seek answers to early in
the audit. The required information can often be found easily in legislation, bylaws, annual reports, or
organization websites. Auditors can also ask management for any missing information. It should not be
necessary at this point to interview board members to obtain the required information.

37
Table 3 – Questions About the Governance of an Agency, a Board or an
Authority
Questions

 How many directors sit on the board?

 For how long can directors serve on the board?
 What is the process to appoint new directors?
 Do board members receive training or orientation on their roles and responsibilities?
 Is there a board charter?
 Are board and corporate policies (such as a code of conduct) documented?
 Is there a board profile or a skills matrix?
 How many committees does the board have? What are the respective roles of the board
committees? How often do the various committees meet?
 Are board minutes publicly available? Are records of committee meetings kept on file?
 Are board self-assessments conducted regularly?
 Who does the board report to and what information does it provide?
 What performance expectations has the government specified for the organization? What key
outcomes are expected? What would be the impact of not meeting expectations?
 In addition to the President or CEO, how many senior executive positions are there? What are their
respective roles and responsibilities?

Once auditors have a good understanding of the basic governance structure of the agency, board or
authority they have selected, they can move to the next step, which is gaining a better understanding of the
organization’s oversight roles and responsibilities and how they are being discharged in practice. In other
words, how are things supposed to be and how are they in reality? Figure 11 presents an overall oversight
framework that auditors can refer to when they develop their knowledge of business questions.

At this stage of the audit process, auditors can ask questions to get an overview of an organization’s
oversight regime without having to conduct extensive research and file reviews. Auditors typically ask more
detailed questions that would require in-depth review and testing of evidence in the audit’s examination
phase.

Knowledge of business questions specific to oversight responsibilities can be divided in two broad categories:
structures and systems (Table 4) and the results and effectiveness of the oversight regime (Table 5). While
this distinction is practical and often easily made, it does not work in all situations; there are usually links
between systems and results and, in some cases, it may be hard to say where systems stops and where results
begin.

Conducting this preliminary audit work will help auditors to draw an overall picture of the oversight in the
agency, board or authority they have selected. It will also help them determine what the most important
oversight functions and activities are and why. Equipped with this information, auditors will be able to start
considering where the audit could fall on the spectrum of audits of oversight.

38
Figure 11 – Overall Oversight Framework

39
Table 4 – Knowledge of Business: Questions on Oversight Structure and
Systems
Questions

 What are the key oversight bodies? How many members do they include? Who are they accountable
to? Has the government formally provided the oversight body with clear performance expectations
and information on the key outcomes to be achieved?
 Do oversight bodies have clear mandates that set out their authority to conduct specific oversight
functions? What are these oversight functions? How are they organized?
 What are the specific oversight roles and responsibilities of the members of oversight bodies?
 Are there independence requirements for oversight bodies and their members? Are the oversight
functions organizationally independent of management? Are there processes in place to manage
conflicts of interest and other threats to independence?
 Is there a board profile or similar document that makes explicit the skills, knowledge, and experience
that board members should possess in order to exercise their oversight roles and responsibilities?
How does the board ensure that its members collectively meet these skills, knowledge, and
experience requirements? Does the board make use of independent subject matter experts to
supplement any identified skills/experience gaps?
 What information do oversight bodies need to make informed decisions? Have those needs been
documented and communicated to management? What systems has management put in place to
help produce the required information?
 Has the oversight body established a system to monitor the performance of important oversight
activities or functions?
 What resources are allocated to oversight bodies each year? Are there significant resource gaps?

Table 5 – Knowledge of Business: Questions on Results and Effectiveness

Questions

 Are the oversight bodies receiving the information they request from management? If yes, is this
information of good quality?
 How do oversight bodies obtain assurance that their organization is in compliance with laws,
regulations, bylaws, and the organization’s code of ethics? Is compliance monitored regularly?
 Has the oversight body (or governance body) adopted a risk management policy? Has the oversight
body ensured that adequate risk management practices exist within the organization? Is the oversight
body aware of the key risks facing the organization? Are risk profiles and risk mitigation strategies
prepared by management regularly reviewed by the oversight body?
 Is there a process in place for the oversight body to monitor the implementation of recommendations
of internal audits and evaluations? Are actions taken in response to the recommendations of internal
audits and evaluations?

40
  Are the results of important oversight activities or functions measured? Is performance information
available? Is performance data gathered, used, and reported?
 What performance information is reported by oversight bodies and functions to fulfill their
accountability responsibilities? Is the information reported complete and transparent? That is, do the
reports include sufficient information for readers to be able to understand key results and evaluate
organizational performance?
 Does the board or council periodically evaluate its performance in discharging its oversight roles and
responsibilities?
 How do the different oversight functions within the organization interact and collaborate?

Assessing risk
Assessing potential risk is an important task when selecting the most significant oversight issues to audit.
Auditors can review the information they have gathered early in the audit (such as governance structure and
minutes of board or committee meetings) and determine whether they can identify indicators that oversight
may be at risk in specific areas of an agency, board or authority.

A list of common indicators that oversight may be at risk is presented in Table 6. While such indicators can
be useful to target further examination work, their presence should not be indiscriminately accepted as
evidence that an oversight deficiency exists. Auditors must always gather sufficient appropriate evidence to
support a cause-and-effect relationship before concluding that the presence of an indicator means that an
actual deficiency exists.

Table 6 – Indicators that Oversight May Be at Risk

Indicators

 A wholesale change of board members took place or turnover is very high, there is a lack of turnover
of board members or excessively long terms, or replacements of board members are not staggered in
time.
 The board’s relationship with the CEO is overly strained, the CEO is not being transparent with the
board, the board’s relationship with the CEO is too cozy, or the board does not (or rarely) question and
challenge the CEO.
 The chair or the CEO is overdominant at board meetings or management is reluctant to talk at board
meetings.
 Conflicts of interests are a frequent occurrence among the members of the oversight body or actions
taken to manage known conflicts of interest are not documented.
 There is no communication about the organization’s code of conduct or there is no code of conduct,
or board members are not in compliance with the code’s requirements.
 The regulator is too close to the regulatee and independence is compromised.
 The chair of the board is involved in the organization’s day-to-day management or there is no
segregation of duties between the board and management.

41
 The board or its committees rarely meet or they hold short, orchestrated, perfunctory meetings.
 The board has no charter and/or no governance manual.
 Board members do not understand their roles, are not aware of the scope of their oversight
responsibilities, and believe that many aspects are management’s responsibility.
 The organization’s governance structure does not include an audit committee.
 Internal audit recommendations are not, or rarely, implemented, or internal audit is being dismantled
or outsourced.
 The board does not periodically review regulations that apply to boards of directors.
 The board is too passive in defining its information requirements and/or fails to follow up on
information requests.
 There is an absence of risk management policies and processes or risk management policies and
processes are not being implemented.
 There are significant organizational problems: poor performance against operational or strategic
targets; significant delays and cost overruns; a high number of complaints, penalties, and fines; or risks
that are escalating.
 The organization has a history of repeated failures for specific types of projects or initiatives.
 Business activities are not aligned with the organization’s mandate.
 There is poor documentation of oversight activities and decisions.
 There is a lack of or misleading performance information.
 There is failure to take follow-up or corrective actions when significant issues are brought to the
attention of the board or its committees.

Back to Table of Contents

42
Determining the Audit Approach
In addition to deciding how much focus or emphasis to place on oversight in the audit (see the section
Determining the Degree of Focus on Oversight for more on this topic), auditors will need to consider which
audit approach the audit should adopt. Essentially, this means deciding to adopt either an approach focused
on the structures and systems of an oversight body or an approach focused on the results and effectiveness
of an oversight body in exercising its oversight functions, roles, and responsibilities.

Focusing on oversight structures and systems means examining an oversight body’s:

 structure and mandate,

 roles and responsibilities,
 independence requirements, and
 skills and experience requirements

to determine whether they are adequate, in line with best practices, or comparable with an appropriate
benchmark.

Focusing on results and effectiveness aspects means examining the quality of the oversight of an
organization’s actual:

 performance,
 risk management,
 compliance,
 reporting, and so on.

Given enough time and resources, auditors can combine both approaches and conduct a more complete
audit that will provide additional assurance to the report’s recipient. The British Columbia audits of Crown
agency board governance (2012) and university board governance (2014) are examples of this combined
approach.

Back to Table of Contents

43
Drafting Audit Objectives
All performance audits need clearly stated objectives that are worded in a manner that allows auditors to
conclude against them. Audit objectives should be realistic and achievable and give sufficient information to
audited organizations about the focus of the audit.

Audits can have one or several objectives depending on the extent of their scope and their complexity. Office
practice will also influence the number of objectives and whether or not sub-objectives are used. (Some audit
offices never use sub-objectives.) Sub-objectives can be included in audit plans (for example, one for each line
of enquiry), but auditors who decide to do so will still be expected to conclude on their main audit objective.

Objectives for audits of oversight are generally of three different types.

 The first type focuses on the structure and systems of oversight bodies. That is, are oversight
processes well designed?
 The second type focuses on the results or effectiveness of oversight bodies in exercising their
functions, roles, and responsibilities. That is, are oversight processes working as designed?
 The third type combines the structures/systems and results/effectiveness aspects.

Audit objectives can be either broad in scope, encompassing the overall oversight framework, or narrow in
scope, covering only a specific oversight requirement. Selecting one type or the other may depend on audit
office practices and available resources to conduct the audit. Table 7 provides examples of broad and narrow
audit objectives for both structures/systems and results/effectiveness audits. These examples cover key
structures/systems aspects of oversight bodies (clear roles and responsibilities, independence, skills and
knowledge, and information flow), as well as a number of important roles usually played by oversight bodies
(overseeing risk management, monitoring compliance and performance, taking corrective actions, and
reporting).

When sufficient time and resources are available, examining both structures/systems and results/effectiveness
aspects is desirable because this approach provides more complete information and additional assurance to
the audit report’s recipient. By examining both aspects, auditors reduce the risk of reaching an incomplete or
irrelevant conclusion. For example, concluding that systems are implemented as designed would be of limited
value if the systems’ design was poor in the first place. Similarly, simply concluding that well-designed
systems are in place would provide only limited value if the systems are not actually used and implemented as
designed.

This being said, focusing solely on the structures/systems aspect is a valid option when it is too early to obtain
result information. It is also possible for auditors who decide to focus on results and effectiveness to cover
structures/systems issues in their report if these issues come up when analyzing the root cause of observed
deficiencies.

44
Table 7 – Examples of Audit Objectives for Audits of Oversight in Agencies,
Boards and Authorities

Topic Structures and Systems Results and Effectiveness

1. Overall oversight To determine whether the structures To determine whether oversight

framework and processes established for the structures and processes are
organization set the framework for implemented as intended and resulting
effective oversight. in effective oversight.

2. Oversight roles To determine whether the board (or To determine whether the board (or
and responsibilities governing body) has clear oversight governing body) is fulfilling its oversight
roles and responsibilities and a clear roles and responsibilities and carrying
mandate to carry out specific oversight out its oversight functions as defined in
functions. its charter (or mandate).

To determine whether the committee To determine whether the committees

structure put in place by the board (or of the board (or governing body) are
governing body) provides for adequate fulfilling their respective oversight roles
oversight of key corporate functions and responsibilities.
and operations.

To determine whether the board (or To determine whether the audit

governing body) has established an committee is fulfilling its assigned
audit committee and clearly defined its oversight roles and responsibilities.
oversight roles and responsibilities.

3. Independence To determine whether the board (or To determine whether the board (or
governing body) has established clear governing body) and its committees are
independence requirements for its effectively managing independence
members and put in place a policy or risks to ensure that they perform their
process to manage perceived and oversight responsibilities objectively.
actual conflicts of interests.
To determine whether the board (or
governing body) and its committees
actively manage conflicts of interest in
accordance with policy requirements (or
best practices).

45
 Topic Structures and Systems Results and Effectiveness

4. Skills and To determine whether the board (or To determine whether the board (or
knowledge governing body) has defined the skills, governing body) and its committees
knowledge, and experience that board collectively possess the skills,
and committee members must possess knowledge, and experience to fulfill
in order to have the capacity to fulfill their oversight responsibilities.
their oversight responsibilities.

5. Sufficient and To determine whether the board (or To determine whether the board (or
appropriate governing body) has defined its governing body) receives the
information information needs and communicated information it needs to fulfill its
those needs to management. oversight responsibilities.

To determine whether the board (or

governing body) regularly assesses the
quality and sufficiency of the
information that management provides
it with.

6. Risk management To determine whether the board (or To determine whether the board (or
governing body) has approved a risk governing body) is aware of the key risks
management policy and clearly allocated facing the organization.
roles and responsibilities in this area.
To determine whether the board ensures
that management has established
adequate processes to monitor and
mitigate key organizational risks.

7. Performance To determine whether the board (or To determine whether the board (or
monitoring governing body) has put in place governing body) is conducting effective
adequate systems and practices to performance monitoring to ensure that
monitor the organization’s performance the organization is meeting its
in meeting its established objectives. established objectives.

8. Compliance To determine whether the board (or To determine whether the board (or
governing body) has put in place governing body) is regularly monitoring
adequate controls to ensure that it is the organization’s compliance with
aware of the organization’s state of laws, regulations, bylaws, and ethical
compliance and of any need for requirements, and taking corrective
corrective actions. actions as necessary.

46
 Topic Structures and Systems Results and Effectiveness

9. Corrective actions To determine whether the board (or To determine whether the board (or
governing body) has put in place governing body) is taking timely
adequate controls to ensure that corrective actions when inefficiencies,
corrective actions are taken in a timely poor performance, substandard results,
manner. or instances of non-compliance are
identified and brought to its attention.

10. External To determine whether the board (or To determine whether the board (or
reporting governing body) has clearly identified governing body) regularly reviews and
the accountability reports it needs to approves key accountability reports.
receive, review, and approve.

11. Performance To determine whether there is an To determine whether the board (or
evaluation adequate process in place to evaluate governing body) regularly evaluates its
the board’s (or governing body’s) own performance in fulfilling its
performance in fulfilling its oversight oversight responsibilities.
responsibilities.

12. Government/ To determine whether the government/ To determine whether the government/
Ministerial oversight Minister has established a clear Minister exercises adequate oversight of
framework for the oversight of the the organization.
organization.

Back to Table of Contents

47
Selecting Audit Criteria
Audit criteria represent the standards expected to be met by an audited organization. Audit criteria are a key
contributor to an audit’s strength and potential impact. Audit procedures focus on determining whether
criteria are met or not met. Suitable criteria are clear, concise, relevant, reliable, neutral, understandable, and
complete.

Finding suitable criteria is a challenge for any performance (value-for-money) audit, not just for audits of
oversight. Each audit is unique due to the auditor’s mandate, audit focus, audit objectives, and the way the
organization being audited approaches the audit’s subject matter. However, the governing bodies of
agencies, boards and authorities usually share many organizational and operational aspects and many studies
have been published on board governance. As a result, guidance already exists about the audit criteria that
can be used to audit oversight in Crown corporations or agencies. The criteria presented as examples in this
section are largely derived from the work of the Canadian Council of Legislative Auditors (CCOLA)
Governance Study Group and the Office of the Auditor General of Canada.

Examples of audit criteria and sub-criteria that can be used to audit oversight structures/systems and their
results/effectiveness in agencies, boards and authorities are presented in Table 8. The criteria and sub-criteria
are divided into 11 categories:

1. Oversight roles and responsibilities

2. Independence
3. Skills and knowledge
4. Sufficient and appropriate information
5. Risk management
6. Performance monitoring
7. Compliance
8. Corrective actions
9. External reporting
10. Performance assessment
11. Government oversight

These categories correspond to the audit objective topic numbers 2 to 12 in Table 7. Objective topic 1 in
Table 7, the overall oversight framework, is very broad and would need, in practice, to be supported by a
selection of criteria taken from these 11 sub-categories.

Auditors are not expected to use all of the suggested criteria. Rather, they can pick and choose those that are
most relevant to the scope of the audit and document the rationale for their selection. They can also develop
additional criteria where needed, in order to conclude on their audit objective(s).

48
Auditors should always use their professional judgment to select audit criteria and to determine whether the
expectations defined by the criteria are reasonable given the nature and operational constraints of the
audited organization. The reasonableness of potential criteria is, in part, a function of the degree to which
they represent a balance between cost, risk, and effectiveness. For example, it would not be reasonable to
expect an organization to adopt an unproven, costly control measure to mitigate a minor risk.

Table 8 – Examples of Audit Criteria that Can Be Used to Audit the Oversight of
Agencies, Boards and Authorities

Topic Structures and Systems Results and Effectiveness

1. Oversight Criterion: The oversight body and its Criterion: The oversight body and its
roles and committees have clearly defined committees fulfill their assigned
responsibilities oversight roles and responsibilities. oversight roles and responsibilities.

Sub-criteria:

 The oversight body has clearly

defined oversight roles,
responsibilities, and authorities.
 Each committee of the oversight
body has terms of reference that
clearly define its areas of
responsibility and level of authority,
and that have been approved by the
board.
 The roles and responsibilities of the
audit committee, set down in its
terms of reference, include:
o maintenance of an effective
internal/external audit
function,
o maintenance of a suitable risk
management and internal
control framework,
o meeting frequency and core
agenda items,
o committee authority, and
o reporting to the oversight
body.

49
 Topic Structures and Systems Results and Effectiveness

2. Independence Criterion: The oversight body and its Criteria:

committees have established systems and
procedures to ensure that members  Members of the oversight body

have, and can demonstrate, the and its committees comply with

independence necessary to perform their applicable independence policies.

oversight responsibilities objectively.  Independent oversight body

members hold regular in camera
Sub-criteria: meetings without management in
attendance.
 The oversight body has established  The internal audit function reports
clear policy and guidance about to the oversight body or its audit
independence requirements. committee, and its independence
Specific prohibitions are listed and from management is supported by
guidance covers the various forms the oversight body.
of independence threats (self
review, self-interest, advocacy,
familiarity, and intimidation) and
how they are to be addressed.
 Oversight body members have to
sign an annual independence
declaration that requires them to
disclose any known independence
threats and confirm their
understanding of the organization’s
independence policy.

3. Skills and Criterion: The skills, knowledge and Criterion: Oversight body members
knowledge experience required of oversight body have the skills, knowledge and
members have been identified and experience they require to effectively
communicated. discharge their oversight responsibilities.

Sub-criteria: Sub-criteria:

 The oversight body has profiled the  The skills and knowledge of
skills and knowledge required of oversight body members are
individual members and for the aligned with those described in the
oversight body as a whole to ensure oversight body profile.
effective oversight of the  The oversight body has access to
organization. The oversight body and uses outside expertise when

50
 Topic Structures and Systems Results and Effectiveness

has shared this profile with the necessary to fill gaps in its skills and
responsible minister. expertise profile.
 An orientation program has been  Committee members have the
developed to provide all new qualifications, skills, and
oversight body members with competencies necessary to
information on: effectively fulfill the committee’s
o the roles and responsibilities of role and responsibilities, as defined
the oversight body and its in its terms of reference.
committees;  All oversight body members receive
o the organization’s mandate, sufficient, appropriate training and
vision, mission, and strategic guidance to provide them with a
plan; working knowledge of their
o the organization’s compliance corporation and the environment
regime; and within which it operates.
o the organization’s
accountability framework.

4. Sufficient Criteria: Criterion: The oversight body and its

and committees have sufficient relevant and
appropriate The oversight body has defined the reliable information to fulfill their
information information and knowledge it needs oversight responsibilities.
from management (on performance,
compliance, risk management, financial Sub-criteria:
management, etc.) to effectively exercise
its oversight role and communicated  The oversight body ensures that it

these needs to management. receives sufficient and appropriate

information on a timely basis to
The oversight body has established a support oversight body decision
process to periodically review the quality making overall.
and quantity of information it receives  The oversight body ensures that it
from management and external sources. receives appropriate (credible,
complete, timely) financial,
performance, and risk information
to allow it to:
o fully assess the corporation’s
performance at regular
intervals;
o ensure that pertinent
legislation, regulations,

51
 Topic Structures and Systems Results and Effectiveness

corporate bylaws, and board

policies are being complied
with; and
o ensure that key risks are being
adequately managed.
 Where additional information is
required to make an assessment or
a decision, the oversight body
requests such information from
management and/or external
sources, and ensures that it is
obtained on a timely basis. The
oversight body defers decisions
when appropriate information has
not yet been received.
 Periodically, the oversight body
looks critically at the quality and
quantity of information it receives
from management and external
sources to ensure that this
information allows the oversight
body to effectively discharge its
oversight responsibilities.

5. Risk Criterion: The oversight body has Criterion: The oversight body and its
management established a risk management policy committees effectively oversee the
framework for the organization. organization’s risk management policies
and processes.

Sub-criteria:

 The oversight body understands the

organization’s key risks.
 The oversight body reviews and
challenges management’s plans on
how to avoid, control, accept, or
transfer key risks to the organization
before approving them.

52
 Topic Structures and Systems Results and Effectiveness

 The oversight body monitors the

organization’s implementation of risk
management policies, processes and
internal controls to ensure they are
working as intended.

6. Performance Criteria: Criterion: The oversight body is

monitoring effectively monitoring the organization’s
The oversight body has established a performance in relation to its mandate
Performance Management Framework and stated objectives.
for the organization.
Sub-criteria:
Performance targets and pertinent
indicators are in place to enable the  The oversight body regularly
oversight body to properly monitor the monitors organizational and
organization’s performance. management performance and
challenges management about the
quality and reliability of the available
performance information.
 The oversight body regularly
monitors and evaluates the CEO’s
performance and takes appropriate
action where that performance is
judged to be below expectations.

7. Compliance Criterion: Systems and practices are in Criteria:

place to monitor the compliance of the
organization with enabling legislation, The oversight body obtains assurance

regulations, bylaws, and oversight body that enabling legislation, regulations,

policies. bylaws, and board policies are being

complied with.

The oversight body ensures that the

organization’s code of conduct is
communicated to all staff, that
compliance with its requirements is
monitored, and that action is taken
when deviations are identified.

53
 Topic Structures and Systems Results and Effectiveness

8. Taking Criterion: The oversight body has put in Criterion: Evidence exists that, based
corrective place adequate controls to ensure that on the information they receive,
actions corrective actions are taken in a timely oversight body members make
manner (to address performance or decisions, provide direction, and follow
compliance issues, weak risk up on actions taken in response.
management or financial management
practices, etc.).

9. External Criterion: The oversight body has Criteria:

reporting determined which accountability reports
it needs to receive, review and approve. The oversight body and its committees
regularly review and approve key
accountability reports.

The audit committee provides an

adequate challenge and review of
financial statements and the associated
management discussion and analysis,
and of any other financial information
and performance information to be
released by the organization, before
their release.

10. Assessment Criteria: Criterion: The performance of the

of the oversight oversight body and its committees in
body’s The oversight body has adopted a policy discharging their oversight
performance that requires it to periodically assess its responsibilities is assessed periodically.
performance.
Sub-criteria:
A process is in place to periodically assess
the performance of the oversight body  The collective performance of the
and its committees in discharging their oversight body, its committees, and
oversight responsibilities. individual members is self-assessed
periodically, and an appropriately
transparent mechanism is used in
reporting the assessment results.
 The oversight body complies with
the corporation’s values and ethics.

54
 Topic Structures and Systems Results and Effectiveness

 The oversight body and its

committees hold a sufficient
number of meetings each year to
fulfill their roles and
responsibilities.
 The oversight body and its
committees keep adequate
meeting minutes and supporting
documentation.
 The oversight body works well as a
team and has effective decision-
making processes in place.

11. Government Criterion: The government has defined Criteria:

oversight and communicated its expectations with
The government exercises adequate
regard to the organization’s performance
oversight of the organization.
and reporting thereof.
Government takes, and follows up on,
Sub-criteria:
corrective actions when significant
 Government provides a letter of issues in the overseen organization are
expectations or similar document brought to its attention.
annually to the overseen
organization that specifies expected
performance for the year, including
the targets that government will use
in evaluating its performance.
 Government clearly communicates
the performance reporting it
requires from the overseen
organization in order to evaluate its
performance.
 The conditions under which the
overseen organization should
consult government for direction
are clearly documented.

Source: These criteria and sub-criteria have been modified from the CCOLA Governance Study Group’s Crown Agency Governance:
Audit Objectives & Criteria and from the Office of the Auditor General of Canada’s Recommended General Criteria & Sub-Criteria (for
special examinations of Crown corporations).
Back to Table of Contents

55
Planning an Audit of Oversight of a Major Initiative in a Department

This section of the Practice Guide is organized in accordance with the key actions and decisions that need to
be taken during the planning phase of the audit process:

§ Acquiring knowledge of business and assessing risk

§ Determining the audit approach
§ Drafting audit objectives
§ Selecting audit criteria

Although these topics are presented in a specific order, planning a performance (value-for-money) audit is
rarely a linear process. In fact, the planning process is often iterative, with decisions in one step requiring the
audit team to review decisions made in previous steps to ensure the audit plan’s overall coherence.

56
Acquiring Knowledge of Business and Assessing Risk
Auditing procedures typically require auditors to acquire knowledge of the organization and subject matter
being audited and to prepare a risk-based audit plan. In practice, this means that the audit team needs to:

 collect knowledge of business information about the governance structure of selected major
initiatives (critical projects, programs, or services), especially regarding oversight bodies and functions
and
 identify significant areas that would benefit from an examination of oversight.

As in all performance (value-for-money) audits, the auditor’s understanding of significance and risks will be
used to identify particular activities or aspects of the major initiative being audited to include in the audit and
to develop audit objectives. This section of the Practice Guide is designed to help auditors acquire a sound
understanding of significance and risks by providing them with examples of:

 general audit questions that can be used to better understand oversight roles and responsibilities
relevant to the major initiative(s) being audited and
 indicators that oversight may be at risk in the major initiative(s) selected for audit.

While these tools will be helpful, auditors should keep in mind that the Practice Guide does not foresee all
possible situations. Applying professional judgment and knowing the particularities of each selected
organization are key success factors for the planning phase of any audit of oversight.

Acquiring knowledge of business

The early stage of planning a performance audit requires that auditors develop a sound understanding of the
nature, objectives, and activities of the organization or organizations that will be audited. This involves
obtaining basic information on an organization’s mandate, organizational structure, accountability
relationships, programs, resources, key risks, past performance, and so on. It also means gathering more
detailed information on specific systems and practices in areas that auditors are particularly interested in,
including oversight.

At this stage, if not already done during the audit selection process, auditors interested in auditing the
oversight of a critical program, project, or service would be expected to clearly document what aspects of the
selected initiative make it an especially important one that requires strong oversight:

 Is the initiative a high-risk one?

 Is it a key government-wide initiative?
 Does it involve large sums of public money?
 Has the initiative’s implementation been delegated to a private sector provider?
 Is the initiative of primary importance to a large proportion of citizens?

In addition to determining why the selected initiative is particularly important, auditors will need to have a
clear understanding of key targets, performance expectations, and outcomes for this initiative. Knowing this

57
will be important for auditors who intend to audit how the department oversees the initiative’s performance
and takes corrective actions when performance issues arise.

Auditors will also need to obtain information on the structures and processes put in place to govern and
oversee the selected initiative. Table 9 provides a list of questions that will help auditors gather information
on oversight structures and systems, while Table 10 provides a list of questions about their results and
effectiveness.

At this stage of the audit process, auditors can ask questions that will provide them with an overview of an
initiative’s oversight regime without requiring them to conduct extensive research and file reviews. Auditors
typically ask more detailed questions that would require in-depth review and testing of evidence in the audit’s
examination phase.

Table 9 – Knowledge of Business: Questions on Oversight Structures and

Systems
Questions

 What structure has been put in place to govern and oversee the selected major initiative (program,
project, or service)? How many senior officials are part of this structure? What are their respective
positions within the department?
 Has the government or department formally provided the oversight body with clear performance
expectations and information on the key outcomes to be achieved?
 Are there terms of reference (or a similar document) that define the oversight body’s mandate and
the specific roles and responsibilities of its members? Does the mandate include clear authority to
conduct specific oversight functions? What are these oversight functions and how are they
conducted?
 Are there independence requirements for the oversight body and its members? Are the members of
the oversight structure also involved in the day-to-day management of the selected initiative? Are
there processes in place to manage conflicts of interest and other threats to independence?
 How often do the oversight body’s members meet? Are records of those meetings kept on file?
 What information does the oversight body need to make informed decisions (business case, expected
benefits, targets, baselines, timelines, etc.)? Have those needs been documented and communicated
to managers of the selected initiative? What systems has management put in place to help produce
the required information?
 Is the oversight body’s performance in fulfilling its roles and responsibilities periodically assessed?
 To whom is the oversight body accountable? What accountability reports and information does it
provide?
 What resources are allocated to the oversight body each year? Are there significant resource gaps?

58
Table 10 – Knowledge of Business: Questions on Results and Effectiveness
Questions

 Is the oversight body receiving the information it requests from the management of the selected
initiative? Is this information of good quality?
 How does the oversight body obtain assurance that the selected initiative is in compliance with laws,
regulations, bylaws, and the organization’s code of ethics? Is compliance regularly monitored?
 Has the oversight body ensured that adequate risk management practices exist for the selected
initiative? Is the oversight body aware of the key risks facing the initiative? Are risk profiles and risk
mitigation strategies prepared by initiative managers regularly reviewed by the oversight body?
 Is there a process in place for the oversight body to monitor the implementation of
recommendations of internal audits and evaluations related to the selected initiative? Are actions
taken in response to the recommendations of internal audits and evaluations?
 Can the results of important oversight activities or functions be measured? Is there a monitoring
system in place? Is performance information available? How is performance data gathered, used, and
reported?
 What performance information is reported by the oversight body to fulfill its accountability
responsibilities? Is the information reported complete, accurate and transparent? That is, do the
reports include sufficient information for readers to be able to understand key results and evaluate
performance?
 Does the oversight body periodically evaluate its performance in discharging its oversight roles and
responsibilities?

Completing the knowledge of business part of their audit planning will help auditors to draw an overall
picture of the oversight of their selected initiative. It will also help them determine what the most important
oversight functions are and why. Equipped with this information, auditors will be able to start considering
where the audit could fall on the spectrum of audits of oversight.

In addition to drawing inspiration from the questions included in Table 9 and Table 10, auditors can use the
overall oversight framework presented in Figure 12 as a reference or when developing their knowledge of
business questions.

59
Figure 12 – Overall Oversight Framework for a Major Initiative

Assessing risk
Assessing potential risk is an important task when selecting the most significant oversight issues to audit.
Auditors can review the information they have gathered early in the audit (governance structure, minutes of
board or committee meetings, and so on) and determine whether they can identify indicators that oversight
of the selected project, program, or service may be at risk.

A list of common indicators that oversight may be at risk is presented in Table 11. While such indicators can
be useful to target further examination work, their presence should not be indiscriminately accepted as
evidence that an oversight deficiency exists. Auditors must always gather sufficient appropriate evidence to
support a cause-and-effect relationship before concluding that the presence of an indicator means that an
actual deficiency exists.

60
Table 11 – Indicators that Oversight May Be at Risk
Indicators

 A wholesale change of oversight body members took place or turnover is very high.
 The oversight body does not (or rarely) question and challenge the managers of the overseen
initiative.
 The chair of the oversight body is overdominant at oversight meetings.
 Conflicts of interests are a frequent occurrence among the members of the oversight body and/or
actions taken to manage known conflicts of interest are not documented.
 Oversight body members are involved in the day-to-day management of the overseen initiative or
there is no segregation of duties between the oversight body and the management of the initiative.
 The oversight body rarely meets or holds short, orchestrated, perfunctory meetings.
 The oversight body has no charter or clear terms of reference.
 Oversight body members do not understand their roles, are not aware of the scope of their oversight
responsibilities, and believe that many aspects are management’s responsibility.
 Internal audit recommendations are not, or rarely, implemented, or internal audit is being dismantled
or outsourced.
 The oversight body does not periodically seek assurance that the overseen initiative is in compliance
with applicable legislation, regulations, and policies.
 The oversight body is too passive in defining its information requirements and/or fails to follow up on
information requests.
 There is an absence of risk management policies and processes applicable to the overseen initiative,
or risk management policies and processes are not being implemented as intended.
 There are significant performance problems in the overseen initiative: poor performance against
operational or strategic targets; significant delays and cost overruns; a high number of complaints,
penalties, and fines; or risks that are escalating.
 The overseen initiative is not aligned with the department’s mandate.
 There is poor documentation of oversight activities and decisions.
 Oversight body provided with too much information, or poorly organized information prior to
oversight meetings
 Oversight body not provided with oversight information sufficiently in advance of oversight meetings
to facilitate meaningful review.
 There is a lack of or misleading performance information.
 There is a failure to take follow-up or corrective actions when significant issues are brought to the
attention of the oversight body.

Back to Table of Contents

61
Determining the Audit Approach
In addition to deciding how much focus or emphasis to place on oversight in the audit (see the section
Determining the Degree of Focus on Oversight for more on this topic), auditors will need to consider which
audit approach they should adopt. Essentially, this means deciding to adopt either an approach focused on
the design of oversight structures and systems or an approach focused on the results and effectiveness of the
oversight body in exercising its oversight functions, roles, and responsibilities.

Focusing on oversight structures and systems means examining the oversight body’s:

 structure and mandate,

 roles and responsibilities,
 independence requirements, and
 skills and experience requirements

to determine whether they are adequate, in line with best practices, or comparable with an appropriate
benchmark.

Focusing on results and effectiveness aspects means examining the quality of the oversight of the selected
initiative’s actual:

 performance,
 risk management,
 compliance,
 reporting, and so on.

Given sufficient time and resources, auditors can combine both approaches and conduct a more complete
audit that will provide additional assurance to the report’s recipient.

Back to Table of Contents

62
Drafting Audit Objectives
All performance audits need clearly stated objectives that are worded in a manner that allows auditors to
conclude against them. Audit objectives should be realistic and achievable and give sufficient information to
audited organizations about the focus of the audit.

Audits can have one or several objectives depending on the extent of their scope and their complexity. Office
practice will also influence the number of objectives and whether or not sub-objectives are used. (Some audit
offices never use sub-objectives.) Sub-objectives can be included in audit plans (for example, one for each line
of enquiry), but auditors who decide to do so will still be expected to conclude on their main audit objective.

Objectives for audits of oversight are generally of three different types.

 The first type focuses on the structures and systems of oversight bodies, functions, and processes.
That is, are oversight processes well designed?
 The second type focuses on the results and effectiveness of oversight bodies in exercising their
functions, roles, and responsibilities. That is, are oversight processes working as designed?
 The third type combines the structures/systems and results/effectiveness aspects.

Audit objectives can be either broad in scope, encompassing the overall oversight framework, or narrow in
scope, covering only a specific oversight requirement. Selecting one type or the other may depend on audit
office practices and available resources to conduct the audit. Table 12 provides examples of broad and
narrow audit objectives for both structures/systems and results/effectiveness audits. These examples cover key
oversight structures/systems (mandate, clear roles and responsibilities, independence, skills, and knowledge),
as well as a number of important roles usually played by oversight bodies (overseeing risk management,
monitoring compliance and performance, taking corrective actions, and reporting).

Audit teams can combine both structures/systems and results/effectiveness objectives in an audit of oversight,
granted they have sufficient time and resources to do so. Examining both design and effectiveness aspects is
desirable since this approach provides more complete information and additional assurance to the audit
report’s recipient. By examining both aspects, auditors reduce the risk of reaching an incomplete or irrelevant
conclusion. For example, concluding that systems are implemented as designed would be of limited value if
the systems’ design was poor in the first place. Similarly, simply concluding that well-designed systems are in
place would provide only limited value if the systems are not actually used and implemented as designed.

This being said, focusing solely on the structures/systems aspect is a valid option when it is too early to obtain
result information. It is also possible for auditors who decide to focus on results and effectiveness to cover
design issues in their report if these issues come up when analyzing the root cause of observed deficiencies.

63
Table 12 – Examples of Audit Objectives for Audits of Oversight of Major
Initiatives in Departments and Ministries

Topic Structures and Systems Results and Effectiveness

1. Overall To determine whether the structures To determine whether the oversight

oversight and processes established for the structures and processes put in place
framework initiative set the framework for for the initiative are implemented as
effective oversight. intended and resulting in effective
oversight.

2. Oversight roles To determine whether the oversight To determine whether the oversight
and body has clear roles and responsibilities body is fulfilling its roles and
responsibilities and a clear mandate to carry out responsibilities and carrying out its
specific oversight functions. oversight functions as defined in its
terms of reference (or mandate).

3. Independence To determine whether the oversight To determine whether the oversight

body has established clear body is effectively managing
independence requirements for its independence risks to ensure that its
members and put in place a policy or members perform their oversight
process to manage perceived and responsibilities objectively.
actual conflicts of interests for the
selected major initiative. To determine whether the oversight
body actively manages conflicts of
interest in accordance with policy
requirements (or best practices).

4. Skills and To determine whether the oversight To determine whether the oversight
knowledge body has defined the skills, knowledge, body members collectively possess the
and experience that its members must skills, knowledge, and experience to
possess in order to have the capacity to fulfill their oversight responsibilities.
fulfill their oversight responsibilities for
the selected major initiative.

64
 Topic Structures and Systems Results and Effectiveness

5. Sufficient and To determine whether the oversight To determine whether the oversight
appropriate body has defined its information needs body receives the information it needs
information and communicated those needs to to fulfill its oversight responsibilities.
initiative managers.
To determine whether the oversight
body regularly assesses the quality and
sufficiency of the information that
initiative managers provide it with.

6. Risk To determine whether the oversight To determine whether the oversight

management body has approved a risk management body is aware of the key risks facing
policy or procedure and clearly the organization in relation to the
allocated roles and responsibilities in selected major initiative.
this area.
To determine whether the oversight
body ensures that management has
established adequate processes to
monitor and mitigate the major
initiative’s key organizational risks.

7. Performance To determine whether the oversight To determine whether the oversight

monitoring body has ensured there are adequate body is conducting effective
systems and practices to monitor the performance monitoring to ensure that
initiative’s performance in relation to the initiative is meeting its established
its established objectives. objectives.

8. Compliance To determine whether the oversight To determine whether the oversight is

body has put in place adequate regularly monitoring the initiative’s
controls to ensure that it is aware of compliance with laws, regulations,
the initiative’s compliance with laws, policies, and ethical requirements, and
regulations, and policies, and of any taking corrective actions as necessary.
need for corrective actions.

9. Corrective To determine whether the oversight To determine whether the oversight

actions body has put in place adequate body is taking timely corrective actions
controls to ensure that corrective when inefficiencies, poor performance,
actions are taken in a timely manner. substandard results, or instances of
non-compliance are identified and
brought to its attention.

65
 Topic Structures and Systems Results and Effectiveness

10. Reporting To determine whether the oversight To determine whether the oversight
body has clearly identified the body regularly reviews and approves
accountability reports it needs to key accountability reports prepared by
receive (from initiative managers), initiative managers.
review, and approve.

11. Performance To determine whether there is an To determine whether the oversight

evaluation adequate process in place to evaluate body’s performance in fulfilling its
the oversight body’s performance in responsibilities is regularly evaluated.
fulfilling its oversight responsibilities.

Back to Table of Contents

66
Selecting Audit Criteria
Audit criteria represent the standards expected to be met by an audited organization. Audit criteria are a key
contributor to an audit’s strength and potential impact. Audit procedures focus on determining whether
criteria are met or not met. Suitable criteria are clear, concise, relevant, reliable, neutral, understandable, and
complete.

Finding suitable criteria is a challenge for any performance (value-for-money) audit, not just for audits of
oversight. Each audit is unique due to the auditor’s mandate, audit focus, audit objectives, and the way the
organization being audited approaches the audit’s subject matter.

The criteria presented as examples in this section are largely derived from the work of the CCOLA
Governance Study Group and the Office of the Auditor General of Canada.

Examples of audit criteria and sub-criteria that can be used to audit the structures/systems and
results/effectiveness of oversight bodies responsible for the oversight of major initiatives in departments and
ministries are presented in Table 13. The criteria and sub-criteria are divided into 10 categories:

1. Oversight roles and responsibilities

2. Independence
3. Skills and knowledge
4. Sufficient and appropriate information
5. Risk management
6. Performance monitoring
7. Compliance
8. Corrective actions
9. External reporting
10. Performance assessment

These categories correspond with the audit objective topic numbers 2 to 11 in Table 12. Oversight topic 1 in
Table 12, the overall oversight framework, is very broad and would need, in practice, to be supported by a
selection of criteria taken from these 10 sub-categories.

Auditors are not expected to use all of the suggested criteria. Rather, they can pick and choose those that are
most relevant to the scope of the audit and document the rationale for their selection. They can also develop
additional criteria where needed, in order to conclude on their audit objective(s).

Auditors should always use their professional judgment in selecting audit criteria and determining whether
the expectations defined by the criteria are reasonable given the nature and operational constraints of the
audited organization. The reasonableness of potential criteria is, in part, a function of the degree to which
they represent a balance between cost, risk, and effectiveness. For example, it would not be reasonable to
expect an organization to adopt an unproven, costly control measure to mitigate a minor risk.

67
While the criteria presented in Table 13 have been designed for situations where there is a clear oversight
structure in place, many can be adapted to audit situations where there is no such structure but it would be
reasonable to expect one. In such situations, auditors could adopt a general objective about whether there is
adequate oversight in place for a major initiative and select and adapt a number of audit criteria based on
what could reasonably be expected in each specific situation, based on good management principles and
best practices.

Table 13 – Examples of Audit Criteria that Can Be Used to Audit the Oversight
of a Major Initiative in a Department or Ministry

Topic Structures and Systems Results and Effectiveness

1. Oversight Criterion: The oversight body has clearly Criterion: The oversight body fulfills its
roles and defined oversight roles and assigned oversight roles and
responsibilities responsibilities. responsibilities.

2. Independence Criteria: Criterion: The oversight body has the

independence necessary to perform its
The oversight body has established clear oversight responsibilities objectively.
policy and guidance about independence
requirements. Specific prohibitions are Sub-criteria:
listed and guidance covers the various
forms of independence threats (self Members of the oversight body comply

review, self-interest, advocacy, with applicable independence policies.

familiarity, and intimidation) and how

Independent members of the oversight
they are to be addressed.
body hold regular in camera meetings

Oversight body members have to sign an without initiative management in

annual independence declaration that attendance.

requires them to disclose any known

independence threats and confirm their
understanding of the applicable
independence policy.

68
 Topic Structures and Systems Results and Effectiveness

3. Skills and Criterion: The skills, knowledge and Criterion: Collectively, oversight body
knowledge experience required of oversight body members have the skills and
members have been defined and knowledge they require to effectively
communicated. discharge their oversight
responsibilities.

Sub-criteria:

 Oversight body members have the

qualifications, skills, and
competencies necessary to
effectively fulfill the committee’s
role and responsibilities, as defined
in its terms of reference.
 The oversight body has access to
and uses outside expertise when
necessary to fill gaps in its skills and
expertise profile.
 All oversight body members receive
sufficient, appropriate training and
guidance to provide them with a
working knowledge of the selected
initiative and the environment
within which it operates.

4. Sufficient Criterion: The oversight body has Criterion: The oversight body has
and defined the information and knowledge sufficient relevant and reliable
appropriate it needs to effectively exercise its information about the selected major
information oversight role. initiative to fulfill its oversight
responsibilities.

Sub-criteria:

 The oversight body ensures that it

receives sufficient and appropriate
information on a timely basis to
support decision making overall.

69
Topic Structures and Systems Results and Effectiveness

 The oversight body ensures that it

receives appropriate (credible,
complete, and timely) financial,
performance, and risk information
to allow it to:
o fully assess the initiative’s
performance at regular
intervals;
o ensure that the initiative
complies with applicable
legislation, regulations, and
policies; and
o ensure that key initiative risks
are being adequately
managed.
 Where additional information is
required to make an assessment or
a decision, the oversight body
requests such information from
initiative management and/or
external sources, and ensures that it
is obtained on a timely basis. The
oversight body defers decisions
when appropriate information has
not yet been received.
 Periodically, the oversight body
looks critically at the quality and
quantity of information it receives
from initiative management and
external sources to ensure that this
information allows it to effectively
discharge its oversight
responsibilities.

70
 Topic Structures and Systems Results and Effectiveness

5. Risk Criterion: The oversight body ensures Criterion: The oversight body
management that appropriate risk management effectively oversees the initiative’s risk
policies and internal controls are put in management policies and processes.
place to mitigate the initiative’s key risks
in a cost-effective manner. Sub-criteria:

 The oversight body understands

the initiative’s key risks and
ensures that a risk assessment
process is in place for the
initiative.
 The oversight body reviews and
challenges management’s plans
on how to avoid, control, accept,
or transfer key initiative risks
before approving them.
 The oversight body monitors the
implementation of risk
management processes and
internal controls applicable to the
initiative to ensure they are
working as intended.

6. Criterion: The oversight body ensures Criteria:

Performance that performance targets and pertinent
monitoring indicators are in place to enable it to The oversight body is effectively

properly monitor the initiative’s monitoring the initiative’s performance

performance. in relation to its stated objectives and

intended outcomes.

The oversight body challenges

management about the quality and
reliability of the available performance
information.

7. Compliance Criterion: Systems and practices are in Criterion: The oversight body obtains
place to monitor the compliance of the assurance that the initiative is in
initiative with applicable legislation, compliance with applicable legislation,
regulations and policies. regulations, and policies.

71
 Topic Structures and Systems Results and Effectiveness

8. Taking Criterion: The oversight body has put in Criterion: Evidence exists that, based
corrective place adequate controls to ensure that on the initiative information they
actions corrective actions are taken in a timely receive, oversight body members make
manner. decisions, provide direction, and follow
up on actions taken in response.

9. External Criterion: The oversight body has Criterion: The oversight body regularly
reporting determined which accountability reports reviews and approves key
it needs to receive, review and approve. accountability reports produced by
initiative managers.

10. Criterion: A process is in place to Criterion: The performance of the

Performance periodically assess the performance of oversight body in discharging its
assessment the oversight body in discharging its oversight responsibilities is assessed
oversight responsibilities. periodically.

 The collective performance of the

oversight body is assessed
periodically.
 The oversight body complies with
the department’s values and
ethical requirements.
 The oversight body holds a
sufficient number of meetings
each year to fulfill its roles and
responsibilities.
 The oversight body keeps
adequate meeting minutes and
supporting documentation.
 The oversight body works well as
a team and has effective decision-
making processes in place.

Source: These criteria and sub-criteria have been modified from the CCOLA Governance Study Group’s Crown Agency Governance:
Audit Objectives & Criteria and from the Office of the Auditor General of Canada’s Recommended General Criteria & Sub-Criteria (for
special examinations of Crown corporations).

72
Conducting the Examination Phase

During the examination phase of a performance audit, audit teams must conduct procedures that will yield
sufficient appropriate evidence to:

 determine whether audit criteria are met,

 conclude on audit objectives, and
 document and support these conclusions.

Audit conclusions can be based on one or more types of evidence, including:

 Documentary evidence—file and document reviews, correspondence, databases, performance

reports, studies, previous audits, and so on
 Testimonial evidence—interviews, focus groups, management assertions
 Physical evidence—personal observations, inspections, walkabouts
 Analytical evidence—calculations, benchmarking, surveys, statistical analysis, data mining, and so on

73
Table 14 provides specific examples of evidence in each category. Each type of evidence can be useful in an
audit of oversight but, in practice, documentary and testimonial evidence tend to constitute the main sources
of evidence for audits that focus on the roles and responsibilities of oversight bodies.

Table 14 – Examples of Evidence Sources for Audits of Oversight

Documentary

 Minutes of board meetings and committee meetings, records of decisions, agendas, board
attendance records
 Information packages prepared by management for board and committee meetings
 Committee debriefs to boards of directors
 Legal mandates, charters, terms of reference, bylaws, board policies, code of ethics
 Strategic plans
 Board-approved delegated authorities
 Correspondence with government officials, correspondence between board and management
 Training material prepared for board members
 Board profile, skills matrix, succession plans
 Board self-assessments and surveys
 Performance reports
 Report cards, monitoring reports, risk management reports
 Internal and external audit reports, evaluation reports, investigation reports, inspection reports,
independent reports by third parties (think tanks, non-profit organizations, and so on)
 Major initiative business case, including baseline data and expected benefits, and related timelines
and targets.

Testimonial
 Interviews with chair of the board, board members, committee members, and senior management
 Assertions (written testimony) by board members or senior management
 Interviews with the minister or other elected officials
 Interviews with stakeholders

Physical
 Observing board meetings
 Observing committee meetings relevant to audit purpose

Analytical
 Benchmarking against best practices or against similar organizations/major initiatives
 Surveys (if auditing multiple organizations)

74
Using documentary and testimonial evidence to support audit observations on oversight and to reach an
audit level of assurance is sometimes relatively straightforward. However, when it comes to questions related
to an oversight body’s effectiveness or dynamics, there may be limited documentary evidence available and
obtaining sufficient and appropriate evidence may therefore represent a challenge.

The remainder of this section briefly discusses the value, limitations, and potential challenges of each type of
evidence that can be used to support conclusions in an audit of oversight. These reflect in large part the
experience of practitioners who have audited oversight. Auditing and assurance standards and associated
guidance materials may also be consulted by auditors for additional information.

Back to Table of Contents

75
Documentary Evidence
Documentary evidence is obtained from the audited organization or from third parties, in hard copy or in
electronic format. Documentary evidence can include documents prepared by an organization for its
oversight body, documents prepared by the oversight body for its own use, and legal documents that set the
organization’s operational context.

Obtaining adequate documentary evidence during an audit of oversight may be challenging for a number of
reasons:

 Some oversight processes may be informal and/or undocumented (“soft controls”) but nonetheless real.
 Some organizations nowadays put less and less information in their board minutes, limiting the
usefulness of this source of evidence.
 Management may be seeking to interpose itself between auditors and board members, controlling
the access to documentation.
 It may be impossible for auditors to obtain documentation about what was discussed during in
camera meetings (when no record is kept).

To work around these challenges, auditors may need to conduct more interviews to obtain additional details
on the nature and results of discussions held during oversight body meetings, and to document informal
controls. Auditors may also need to clearly assert their access to information rights as provided in their
office’s mandate.

In addition, auditors may face situations where minutes of oversight meetings constitute their main source of
documentary evidence for an audit observation. This may be problematic since minutes are considered
secondary evidence and it may be difficult to demonstrate that there are sufficient controls in place to ensure
that the minutes are reliable. In such a case, to avoid overreliance on minutes, the documentary evidence can
be supported by testimonial evidence of some kind (interviews or written assertions). When there is a public
version of the minutes and a more complete, internal version, it is suggested that auditors use the internal
version.

Testimonial Evidence
Testimonial evidence is obtained by conducting interviews, focus groups, surveys, or written assertions.
Testimonial evidence is particularly useful in audits of oversight to document the less tangible aspects of the
oversight environment: soft controls, organizational culture, leadership, and oversight body dynamics.

Testimonial evidence is often very useful to:

 confirm information obtained from other sources of evidence (thus strengthening the support for
audit observations and conclusions),
 confirm the absence of something that was expected to exist,
 place documentary evidence in its proper context, and
 open new leads in an audit and identify further sources of evidence.

76
When auditing boards of directors or other oversight bodies, notes of interviews with oversight body
members may constitute an important source of evidence. Auditors can use the notes as support for their
observations, but should avoid putting too much reliance on interviews alone. Whenever possible,
documentary, physical, or analytical evidence should also be obtained to support key observations.

Special considerations for interviews with board members

Conducting interviews with only one director or governor at a time is key to creating a safe environment.
Using focus group or group interviews would likely not provide complete information for audit teams since
directors and governors might not feel comfortable enough in such settings to express some of their views on
the effectiveness of the oversight in the organization they govern. Having an auditor with a seniority level
matching the interviewee’s conduct the interview is another way to foster a climate of trust.

The experience of auditors suggests that, when planning audit procedures for examining a board of directors
or a governing council, auditors should consider planning sufficient time and resources to interview all
current directors or governors, as well as previous ones who were active during the period covered by the
audit. Limiting the interviews to the chair and key members of a board or a council creates a risk that auditors
may remain unaware of significant facts. Contradictory views and different perspectives can often be
obtained from the “backbencher” members of boards and councils. By interviewing all directors or
governors, auditors can ensure that they obtain and consider as many points of view as possible and so
develop a full understanding of the dynamics of a board or council. However, when time and resources are
limited, auditors may not be able to interview all directors; in such situations, they will need to carefully
consider which directors to interview.

One final aspect to consider when planning interviews with directors or governors is their timing. It is
generally better for auditors not to interview board or council members before they interview management
and develop a good understanding of the risks and issues facing the audited organization. Once this is done,
directors or governors can be interviewed. This way, auditors will be in a position to assess whether directors
or governors are aware of the main issues facing the organization and what they are doing to monitor and
resolve them. Since auditors may often have only one chance to interview individual directors, it is in their
interest to carefully consider the best moment to do so.

Back to Table of Contents

77
Physical Evidence
In audits of oversight bodies, the principal means of gathering physical evidence is to observe board or
committee meetings.

There are benefits to attending these meetings in person (upon request). Doing so enables auditors to
observe board or committee dynamics directly and so obtain a better understanding of context and situations
than would be possible by simply reviewing meeting minutes after the fact. This is especially relevant with in
camera meetings for which management cannot provide minutes.

However, from a practical standpoint, this kind of evidence is unlikely to be relied on frequently, for different
reasons. First, board and committee meetings only occur a few times a year and auditors may have limited
occasions to attend these meetings during the audit. Second, what is observed at one meeting may not
represent what usually takes place at the meetings. This is especially true if the behaviour of the directors or
governors changes because of the auditors’ presence.

For these reasons, observing board or committee meetings may be better considered as a source of
knowledge of business information than as a source of evidence to be used to support audit observations.

Back to Table of Contents

78
Analytical Evidence
Many different procedures can be used to generate analytical evidence in support of audit observations on
oversight. Some can be relatively simple, like reviewing the minutes of board meetings over a precise period
of time in order to determine whether meetings are held regularly, whether directors have a good track
record of attending the meetings and what topics were discussed by the board. Other procedures are more
complex and will often require the assistance of specialists, like surveys and benchmarking exercises. This
section provides information on surveys and benchmarking.

Surveys
Conducting surveys is a useful audit procedure when the scope of an audit of oversight is large, covering
multiple organizations or a whole sector (health or education, for example). Surveys enable auditors to collect
specific, structured information from a well-defined population.

In audits of oversight, surveys can be used to obtain information on the policies, systems, and practices in
place in different organizations. They can also be used to obtain opinions on oversight body dynamics or on
the effectiveness of specific oversight practices and functions.

While surveys can be useful, auditors should note that they are qualitative assessments (especially surveys of
opinions) and are not generally sufficient on their own as audit evidence. Indeed, in its 2013 document
Crown Agency Governance – Obtaining Audit Evidence: Challenges, the CCOLA Governance Study Group
considered that opinion surveys do not generally provide audit-level assurance on a board’s performance, nor
on the quality of its oversight. However, it is possible to use data collected through a survey in combination
with other types of evidence to provide audit-level assurance on specific audit observations. Also, survey data
can be used as the basis of a non-assurance report (see, for example, British Columbia’s 2009 OAG report on
information use by boards of public sector organizations and Manitoba’s 2009 OAG study on board
governance in Crown organizations).

Finally, auditors need to be aware that surveys are complex procedures that require much thought, time,
resources, and expertise. Developing and conducting surveys requires specialized knowledge and skills. For
this reason, auditors are encouraged to consult with an internal specialist or an external expert before
proceeding with a survey as part of the audit of oversight.

Benchmarking
Benchmarking is a method for comparing performance, systems, or processes across and between
organizations, across or between countries. In audits of oversight, benchmarking can be used for three
purposes:

1. to identify best practices that will be used as audit criteria,

2. to assess the design of oversight structures and systems and/or the results and effectiveness of
oversight bodies compared with those of other organizations or with recognized best practices, and
3. to identify best practices that will constitute the foundation for audit recommendations.

79
The principal advantage of benchmarking is that it provides an objective basis from which to derive audit
observations and conclusions. When properly conducted, benchmarking can allow auditors to reach
conclusions on the structures and systems of an organization’s oversight bodies and on their relative
effectiveness compared with best practices or with similar organizations in a sector of activity.

However, conducting and documenting the results of a benchmarking exercise can be time consuming and
challenging, especially when authoritative sources of best practices are not readily available.

Benchmarking the design of oversight structures and systems will generally be easier to do than
benchmarking their effectiveness. Information on mandates, governance structures, policies, and practices
can readily be obtained by conducting a survey of similar organizations and by collecting documentary
evidence available in the public domain or upon request. Public organizations will often be willing to provide
information on the design of oversight structures and systems. Obtaining reliable information on the results
and effectiveness of oversight bodies and their practices will usually be more difficult, especially when there
are significant deficiencies that selected organizations would rather not bring to the attention of auditors.

Beyond obtaining sufficient information, auditors who want to use benchmarking as a source of evidence
must ensure that they are making valid comparisons. They should do the following:

 Compare organizations that operate in the same sector of activity and that share significant
operational characteristics. In general, comparing public sector organizations with private sector ones
will not be appropriate because of the very different goals pursued by each type of organization.
 Use equally reliable data for all the organizations covered by the analysis. Using only annual reports
and website information is insufficient to compare effectiveness unless the reliability of this
information is assessed by the auditors.
 Close all information gaps and clear all uncertainties by obtaining documentary or testimonial
evidence from selected organizations.

Finally, before embarking on a benchmarking analysis, auditors are advised to consult with the audited
organization’s management and with subject experts to discuss which organizations (or countries) would
constitute acceptable comparators. It is preferable to obtain management’s agreement with the methodology
used, but auditors can expect to run into arguments that the audited organization has unique challenges and
cannot fairly be compared with its peers. In such situations, audit teams will need to exercise their
professional judgment and decide whether or not to proceed with a benchmarking analysis.

Back to Table of Contents

80
Reporting the Results of an Audit of Oversight

During the reporting phase of a performance audit, auditors produce a report that presents their audit
observations and conclusions. Audit reports vary considerably in scope and nature. In addition, the formats
and writing styles of performance audit reports are specific to individual audit offices. As a result, there is no
standard way to present audit findings.

However, some common principles and good practices can be applied by performance auditors to present
their audit findings and conclusions more effectively. This section of the Practice Guide discusses some
principles and good practices applicable to audits of oversight, but avoids specific recommendations about
format and writing styles.

81
Setting the Context
When writing the introduction to an audit report on oversight, auditors should clearly state why they carried
out the audit and explain why oversight is important to the success of the selected organization, program, or
project. Doing so will provide an answer to the “so what?” question that readers might pose and will let the
readers know why they should care about the audit topic.

The front end of the report should also provide sufficient context on the organization, program, or project
being audited. In particular, auditors should clearly:

 present the roles and responsibilities of the relevant oversight bodies and functions;
 distinguish the responsibilities of management from those of oversight bodies; and
 explain the key accountability relationships in the organization, program, or project.

Using organizational charts and flow diagrams can effectively present this information without using too
many words. Figure 13 provides an example from the health sector in British Columbia.

Figure 13 – Example of a Flowchart Used to Illustrate Oversight

Responsibilities and Accountability Relationships

Source: Adapted from Oversight of Physician Services, Office of the Auditor General of British Columbia (2014).

82
Audit Observations
In reporting audit observations, it is common practice not to name the senior officials responsible for
oversight, but to simply refer to the position they held at the time (for example, “the chair of the board” or
“the Minister”). Auditors should be particularly cautious when the findings clearly point to the behaviour of a
specific individual, as any misrepresentation of the facts could result in litigation by that individual. In cases
where auditors feel that the actions of a specific individual should be reported, they can use a management
letter to present their observations to the relevant organization.

Auditors can also address a management letter to the audited organization when they have more findings
than they can communicate in a single report or have findings that are not significant enough to be brought
to the attention of their legislature but that should nonetheless be addressed.

Back to Table of Contents

83
Recommendations
Drafting effective recommendations is a challenging task that requires much thought, discussion, and
professional judgment. When drafting a recommendation, auditors can ask themselves the following
questions:

 Is the recommendation addressed to the right organization (that is, the one that can actually
implement it and make change happen)?
 Should the recommendations be directed to the oversight body or to the organization overall?
 Is the recommendation aimed at the root cause of the issue or at its symptoms? (See our discussion
paper on Root Cause Analysis for guidance on this topic.)
 Does the recommendation clearly identify the risk(s) being addressed?
 Is the recommendation consistent with the audit observations?
 What is the cost and feasibility of implementing the proposed action? Are there alternative courses
of remedial actions that would be easier to implement or are more affordable?
 What would be the impact on results, both positive and negative, if the recommendation were
adopted?

Furthermore, auditors can inform their decisions on audit recommendations by seeking the audited
organization’s views on the actions that would be necessary to correct the identified oversight deficiencies. By
discussing audit recommendations with audited organizations before the completion of audit reports,
auditors can increase the likelihood that their recommendations will be implemented and will lead to positive
change.

Back to Table of Contents

84
References
Guidance and Good Practices
Australian National Audit Office (2014). Public Sector Governance: Strengthening Performance Through
Good Governance, available at: https://www.anao.gov.au/work/better-practice-guide/public-sector-
governance-strengthening-performance-through-good

Australian National Audit Office (2015). Public Sector Audit Committees: Independent Assurance and Advice
for Accountable Authorities, available at: https://www.anao.gov.au/work/better-practice-guide/public-sector-
audit-committees-independent-assurance-and-advice

Canadian Council of Legislative Auditors’ Governance Study Group (2012, unpublished document). Crown
Agency Governance – Audit Objectives and Criteria. Available at: www.ccola.ca (CCOLA members only)

Canadian Council of Legislative Auditors’ Governance Study Group (2013, unpublished document). Crown
Agency Governance – Obtaining Audit Evidence: Challenges. Available at: www.ccola.ca (CCOLA members
only)

CCAF-FCVI Inc. (2014). Better Integrating Root Cause Analysis into Legislative Performance Auditing: A
Discussion Paper. Available at: https://www.caaf-fcar.ca/en/performance-audit/research-and-methodology/
discussion-papers

CCAF-FCVI Inc. (1997). Information, the Currency of Corporate Governance – A Board Information Strategy,
25 pp.

Chartered Professional Accountants of Canada (2014). CPA Canada Handbook – Assurance, performance
audit standards:

 CSQC-1 – Quality Control for Firms that Perform Audits and Reviews of Financial Statements, and
other Assurance Engagements
 5025 – Standards for Assurance Engagements Other than Audits of Financial Statements and Other
Historical Financial Information
 PS 5400 – Value-for-Money Auditing in the Public Sector

Institute of Internal Auditors (IIA). International Professional Practices Framework (IPPF):

 Standard 2010 – Planning

 Standard 2100 - Nature of Work
 Standard 2200 - Engagement Planning
 Standard 2300 - Performing the Engagement
 Standard 2400 - Communicating Results

85
Institute of Internal Auditors (IIA) – Global (2014). Assessing Organizational Governance in the Public Sector,
available at: https://na.theiia.org/standards-guidance/recommended-guidance/practice-
guides/Pages/Assessing-Organizational-Governance-in-the-Public-Sector.aspx (IIA members only)

International Organization of Supreme Audit Institutions (2016). Guidelines on Central Concepts for
Performance Auditing (ISSAI 3100), available at: http://www.issai.org/en_us/site-issai/issai-framework/4-
auditing-guidelines.htm

International Organization of Supreme Audit Institutions (2016). Standards for Performance Auditing (ISSAI
3000), available at: http://www.issai.org/en_us/site-issai/issai-framework/4-auditing-guidelines.htm

Office of the Auditor General of British Columbia (2009). Guidelines: Information Use by the Boards of Public
Sector Organizations, available at: http://www.bcauditor.com/online/pubs/592/592

Office of the Auditor General of British Columbia (2008). Public Sector Governance: A Guide to the Principles
of Good Practice, available at: http://www.bcauditor.com/files/publications/2008/report13/report/public-
sector-governance-guide-principles-good-practice.pdf

Office of the Auditor General of Canada (2006, unpublished draft). An Introduction to Gathering and
Analyzing Evidence for Performance Audits.

Office of the Auditor General of Canada (2005). Recommended General Criteria & Sub-Criteria, internal
document.

PricewaterhouseCoopers (2011). Board Effectiveness – What Works Best, 2nd Edition, Institute of Internal
Auditors Research Foundation, 138 pp.

Public Service Commission of Canada (2005). Audit Manual: Public Service Commission of Canada, available
at: http://publications.gc.ca/collections/collection_2011/cfp-psc/SC3-112-2005-eng.pdf

RAND Europe (2009). Performance Audit Handbook: Routes to Effective Evaluation, available at:
http://www.rand.org/content/dam/rand/pubs/technical_reports/2010/RAND_TR788.pdf

Audits and Public Accounts Committee Reports Cited in the Practice Guide
Office of the Auditor General of British Columbia (2014). Oversight of Physician Services, available at:
http://www.bcauditor.com/pubs/2014/report9/oversight-physician-services

Office of the Auditor General of British Columbia (2014). University Board Governance Examinations,
available at: http://www.bcauditor.com/pubs/2014/report10/university-board-governance-examinations

Office of the Auditor General of British Columbia (2012). Crown Agency Board Governance, available at:
http://www.bcauditor.com/pubs/2012/report2/crown-agency-board-governance

86
Office of the Auditor General of British Columbia (2009). Making the Right Decisions: Information Use by the
Boards of Public Sector Organizations, available at: http://www.bcauditor.com/pubs/2009/report6/board-use-
information

Office of the Auditor General of Canada (2013). National Shipbuilding Procurement Strategy, available at:
http://www.oag-bvg.gc.ca/internet/English/parl_oag_201311_03_e_38797.html

Office of the Auditor General of Canada (2013). Oversight of Rail Safety—Transport Canada, available at:
http://www.oag-bvg.gc.ca/internet/English/parl_oag_201311_07_e_38801.html

Office of the Auditor General of Canada (2010). Canada’s Economic Action Plan, available at:
http://www.oag-bvg.gc.ca/internet/English/parl_oag_201010_01_e_34284.html

Office of the Auditor General of Manitoba (2009). Study of Board Governance in Crown Organizations,
available at: http://www.oag.mb.ca/wp-content/uploads/2011/06/board_gov_survey_report_2009.pdf

Office of the Auditor General of New Brunswick (2011). Department of Environment – Wastewater
Commissions, available at: https://www.gnb.ca/oag-bvg/2011v1/chap1e.pdf

Office of the Auditor General of Ontario (2014). 2015 Pan Am/Parapan Am Games Security, available at:
http://www.auditor.on.ca/en/content/specialreports/specialreports/2015panam_june2016_en.pdf

Office of the Auditor General of Ontario (2012). Ornge Air Ambulance and Related Services, available at:
http://www.auditor.on.ca/en/content/specialreports/specialreports/ornge_web_en.pdf

Office of the Auditor General of Ontario (2009). Ontario’s Electronic Health Records Initiative, available at:
http://www.auditor.on.ca/en/content/specialreports/specialreports/ehealth_en.pdf

Standing Committee on Public Accounts, Legislative Assembly of Ontario (2014). Ornge Air Ambulance and
Related Services: Summary Report, available at: http://www.frank-klees.on.ca/wfkp/wp-
content/uploads/2014/06/ORNGE-AIR-AMBULANCE-AND-RELATED-SERVICES-SUMMARY-REPORT-Merged-
Accessible-May-2-2014.pdf

Other References on Governance and Oversight

Bader, B. S. (2008). Distinguishing Governance from Management, Great Boards, Vol. VIII, No. 3, available at:
https://cacnc.org/wp-content/uploads/2016/06/Great-Boards-distinguishing-governance-and-
management.pdf

Canadian Institute of Chartered Accountants (2010). 20 Questions Directors Should Ask About Governance
Committees, available at: https://www.cpacanada.ca/en/business-and-accounting-resources/strategy-risk-and-
governance/corporate-governance/publications/20-questions-for-directors-on-governance-committees

87
Canadian Institute of Chartered Accountants (2005). 20 Questions Directors Should Ask About Governance
Assessments, available at: http://www.yorku.ca/rleblanc/media/20quesGovAssessPub.pdf

Conference Board (2011). Corporate Oversight and Stakeholder Lines of Defense, available at:
http://www.mixprize.org/sites/default/files/media/posts/documents/corporate_oversight_and_stakeholder_line
s_of_defense.pdf

Crown Agencies Secretariat, Board Resourcing and Development Office, British Columbia (date unknown).
Crown Agency Risk Management and Internal Controls: A Good Practices Checklist, available at:
http://www2.gov.bc.ca/gov/DownloadAsset?assetId=FB0FF30AF7BE4788A4769CDC6597B824

Institute of Internal Auditors (2013). IIA position paper: The Three Lines of Defense in Effective Risk
Management and Control, available at: https://na.theiia.org/standards-guidance/Public Documents/PP The
Three Lines of Defense in Effective Risk Management and Control.pdf

Institute of Internal Auditors (date unknown). The Audit Committee: Internal Audit Oversight, available at:
https://na.theiia.org/about-ia/PublicDocuments/08775_QUALITY-AC_BROCHURE_1_FINAL.pdf

Institute on Governance (2014). Towards a Risk-Based Approach to Public Sector Oversight, available at:
http://iog.ca/wp-content/uploads/2014/11/Towards-a-Risk-Based-Approach-to-Public-Sector-Oversight.pdf

Institute on Governance (2013). The Role of Management Boards in the Public Sector: A Public Governance
Exchange Discussion Paper, available at: http://iog.ca/publications/the-role-of-management-boards-in-the-
public-sector/

Institute on Governance (2013). A Risk Lens on Governance: A Public Governance Exchange Discussion Paper,
available at: http://iog.ca/publications/a-risk-lens-on-governance-a-public-governance-exchange-discussion-
paper/

Institute on Governance (2011). The Governance Continuum: Origins & Conceptual Construct, available at:
http://iog.ca/publications/the-governance-continuum-origins-and-conceptual-construct/

KMPG (2012). Enhancing Board Oversight: Avoiding Judgment Traps and Biases, available at:
https://assets.kpmg.com/content/dam/kpmg/pdf/2015/10/enhancing-board-oversight.pdf

Office of the Premier, Province of British Columbia (2005). Board Resourcing and Development, available at:
http://www.brdo.gov.bc.ca/governance/corporateguidelines.pdf

Organization for Economic Co-operation and Development (2005). OECD Guidelines on Corporate
Governance of State-owned Enterprises, available at:
http://www.oecd.org/corporate/ca/corporategovernanceofstate-ownedenterprises/34803211.pdf

88
Privy Council Office (2009). Federal Government Institutions by Organizational Form, available at:
http://www.pco-bcp.gc.ca/index.asp?lang=eng&page=information&sub=publications&doc=gloss/gloss-
eng.htm#part1.1

Treasury Board of Canada Secretariat (2012a). Guide to Integrated Risk Management, available at:
https://www.canada.ca/en/treasury-board-secretariat/corporate/risk-management/guide-integrated-risk-
management.html

Treasury Board of Canada Secretariat (2012b). Oversight in the Government of Canada: An Overview of
Assurance Providers, available at: https://www.canada.ca/en/treasury-board-
secretariat/corporate/reports/report-state-comptrollership-government-canada.html#toc7

Treasury Board of Canada Secretariat (2005). Review of the Governance Framework for Canada’s Crown
Corporations, available at: http://www.tbs-sct.gc.ca/report/rev-exa/gfcc-cgse-eng.pdf

Back to Table of Contents

89
Glossary
Accountability – The obligation of an individual, a group, or an organization to answer for a responsibility
that has been conferred. This usually entails reporting on performance, explaining any variance from agreed
expectations, and taking appropriate corrective actions.

Agency, Board or Authority – A public sector organization that:

 is established by government, but is not part of a ministry;
 is accountable to the government; and
 was assigned or delegated authority and responsibility by the government, or otherwise has statutory
authority and responsibility to perform a public function or service.

Auditability – The ability to carry out an audit in accordance with professional standards and internal audit
policies. Although some areas may be significant, they may not be auditable for the following reasons:
 the audit team does not have or cannot acquire the required expertise,
 the selected area is undergoing significant and fundamental change,
 suitable criteria or approaches are not available to assess performance, or
 the information or evidence required is not available or cannot be obtained efficiently.

Auditee – The organization whose performance is being audited.

Audit conclusion – An informed judgment made by an auditor based on sufficient and appropriate audit
evidence.

Audit focus – The breadth and depth of an audit, the risk areas, and the issues selected. Because different
audit offices use the term “audit scope” in different ways, the Practice Guide avoids this word and instead
uses “audit focus” to refer to the depth and breadth of an audit.

Audit observation – The outcome of an objective evaluation of audit evidence against selected audit
criteria.

Audit program – A detailed outline of the audit work to be undertaken during the audit examination phase
to gather sufficient and appropriate evidence. Each audit activity outlined in the program includes the
applicable criteria to be used and the audit steps, tasks, resources, and time required to complete the work.

Audit recommendation – A measurable statement for corrective action made by the auditor and addressed
to the audited organization. Recommendations must address the causes of deficiencies identified in audit
reports.

Control – Any action taken by management, a board, or other parties to manage risk and increase the
likelihood that an organization’s objectives will be achieved.

90
Due diligence – What occurs when, in support of key decisions and related management activities, an
organization has:
 clarified rules, roles, and responsibilities;
 performed and documented analyses (of benefits and risks, operational requirements, options, and costs);
 consulted with other organizations; and
 obtained the necessary approvals.

Governance – The structures, systems, and practices an organization has in place to:
 assign decision-making authorities, define how decisions are to be made, and establish the
organization’s strategic direction;
 oversee the delivery of its services and the implementation of its policies, plans, programs, and
projects; and
 report on its performance in achieving intended results and use performance information to drive
ongoing improvements and corrective actions.

Outcome – The consequences of a policy, program, initiative, or activity. An intended outcome is the end
result that is being sought by an organization, a policy, a program, or an initiative.

Oversight – The responsibility to review, monitor, and supervise public sector organizations and their
policies, plans, programs, and projects, to ensure that they are achieving expected results and are in
compliance with applicable policies, laws, regulations, and ethical standards. Oversight is a critical governance
function performed by senior management, boards of directors, committees, or other internal or external
bodies.

Oversight body – A group of people with a common oversight purpose acting as an organized unit.

Performance audit – An independent, objective, and systematic assessment of how well government is
managing its activities, responsibilities, and resources in a given sector of activity.

Risk – An event or action that may adversely affect an organization’s ability to achieve its objectives.
Assessing risk involves considering the probability (or likelihood) of the event occurring and the potential
impact of that event.

Significance – The relative importance of a matter within the context in which it is being considered,
including quantitative and qualitative factors. Such factors include the magnitude of the matter in relation to
the subject matter of the audit, the nature and effect of the matter, the relevance of the matter, the needs
and interests of third parties, and the impact of the matter to the audited program or activity.

Value-for-money audit – An assessment of whether an organization has obtained the maximum benefit
from the goods and services it both acquires and provides, within the resources available to it. Value for
money is often described in terms of the “3 Es”: economy, efficiency, and effectiveness.

Back to Table of Contents

91