AI Check-List

Self-Assessment Tool Version 2.0

Step 1. Regulatory Framework

This section focuses on the factors that signal the approach to the use of AI technology is
structured and a part of existing systems that decision-makers, adopters, and end-users can rely
upon.
Choose 0 if NO or 1 if YES
There are relevant AI or data-related programs, plans or provisions adopted or
1.1
considered by the government
Data protection and privacy is officially recognized by
1.2
statements/announcements/policies
1.3 Procurement policy is ready for AI adoption
1.4 AI ethical principles are adopted
AI council/committee or similar institution is established under the responsible
1.5
institution
Total Score per Step

Step 2. Decisions
This section focuses on how to use AI technology for enhancing decision-making processes
Choose 0 if NO or 1 if YES
2.1 Decision-makers use digital data and technologies to make policy decisions

There are existing backlogs of work, cases and lessons learned to use
2.1.1
for decision-making.

There are dashboards or equivalent analytical instruments to use data

2.1.2
for decision-making

Raw datasets are available (by request) for decision-makers that are
2.1.3
good-quality and representative
 2.1.4 Trainings on the use of data for decision-makers are in place

A Chief Data Officer (or similar role) is present in the institution responsible
2.2 for regulation/oversight and this person has authority to initiate and manage
decision-making automatization processes

Cybersecurity measures are provided for decision-making processes by

2.3
internal or external teams

A diverse, multidisciplinary and qualified team (including data scientists and

2.4 representatives of different sectors, gender experts, and professionals of
diverse expertise) has been formed to participate in decision-making processes

An independent AI Ethics team has been formed, or AI Ethics themes are

2.5
specifically covered by the multidisciplinary team for decision-making

Total Score per Step

Step 3. Risk assessment and mitigation

This section focuses on systems for ensuring AI is adopted in a secure and ethical manner and
transparency is provided for all engaged stakeholders, communities, researchers, and observer

Choose 0 if NO or 1 if YES
3.1 A risk assessment for algorithms and data has been developed and operationaliz
3.2 Data Quality is ensured
A documented process is in place to test datasets against biases and
3.2.1 other unexpected outcomes via experience in applying frameworks,
methods, guidelines or other assessment tools
Gender-based analysis of the data is periodically conducted and
3.2.2
information about this analysis is publicly available

Accountability for the design, development, maintenance, and

3.3.3
improvement of the AI system is ensured

Datasets are available on an Open Government Portal (Open Data)

3.4.4
(e.g. https://datos.gob.mx/busca/organization/estado-de-guanajuato).

3.3 Procedural Fairness is upheld

A Fairness Tool is used to assess and fix data bias (e.g. AI Fairness
3.3.1 360 toolkit from IBM, Google's PAIR, Datasheets for Datasets from
Microsoft)
 A system to audit AI technology has been established that includes
access to the audit trail, logged details of all changes made to the
3.3.2
model and the system, and logged details of the system’s reasons for
its decisions or recommendations

A process is in place to develop and approve a project concept case

3.3.3
description for internal management (i.e. AI council/committee)
3.4 Privacy is ensured
Security and privacy principles are built into all systems by default
3.4.1
(e.g. from the concept stage forward)

There are established agreements or arrangements with appropriate

3.4.2
safeguards when sharing of personal information is involved

3.5 Transparency and accountability are ensured

Transparency is ensured. The methodological description of risk

3.4.1
assessment frameworks, tools, datasets is clear and available

Accountability is ensured. Roles and responsibilities for system

3.4.2
compliance and errors are clear

Total Score per Step

Step 4. Project-Based Consultations

This section focuses on maximizing transparency in AI decision-making to give users

confidence that an AI system functions well.
Choose 0 if NO or 1 if YES
A process for consultations with internal stakeholders (e.g. Strategic Policy
4.1
and Planning, Data Governance, Program Policy, etc.) is established

A process for consultations with external stakeholders (e.g. Civil Society,

4.2 Academia, Industry, external groups relevant for each specific project, etc.) is
established.

4.3 Consultations on the application of AI tools are provided at the initial stages of

A feedback system for gathering opinions after consultation is provided in

4.4
multiple forms (e.g. by email, on the specialized portal)

4.5 A system for monitoring and evaluating the system once the project is implemen
 Total Score per Step

Step 5. Procurement
This section focuses on processes related to the procurement, maintenance and change of AI
technologies.
Choose 0 if NO or 1 if YES
Local AI technology suppliers are invited and engaged at early stages within
5.1
the planning phase of any AI project
5.1.1 Multiple, diverse channels are used to identify AI suppliers
An open environment supporting competition in the AI ecosystem
5.1.2
exists
5.1.3 Public, open and blind procurement procedures are established
Procurement documentation focuses on a clear problem statement, rather than
5.2
details or specifications for a solution
All invitations to tender include outcome-based requirements that
5.1.1
focus on describing both challenges and opportunities
The invitation to tender includes an iterative approach to product
5.1.2
development

The procurement problem statement requires both the explanation of

5.1.3
process and interpretation of results from algorithms

The procurement document includes the requirement of periodic

5.1.4
testing for the life of the technology and project

The procurement document includes a requirement for transferring

5.1.5
knowledge and training

The evaluation process of submitted tenders is multidisciplinary to ensure

5.3
there is a broad expertise conducting the tender evaluation
Procurement requirements include explanations of AI technology, algorithms
5.4
and systems for non-specialists
Appropriate support and hosting arrangements are in place for the procurement
5.5
team

Total Score per Step

Step 6. End-of-life
 This section pictures of the future functioning AI system.
Choose 0 if NO or 1 if YES
End-of-life processes for any procured AI system, the data used and results
6.1 created are considered and expectations are agreed to by all parties (e.g
decision-makers, vendors, users)
6.2 There is a list of ethical implications associated with the future AI system
6.3 There is a list of technical risks and flowcharts on how to mitigate identified risks
6.4 Monitoring and evaluation process of the AI system are established
6.5 A transparent, publicly available dialogue with the developers is established

Total Score per Step

Note for Google Drive users: In the online version, the graph may be displayed with errors, so it
is suggested to download the file for use.

Step 1. Regulatory Framework

Step 6. End-of-life Step 2. Decisions

0.5

Step 5. Procurement Step 3. Risk assessment and mitigation

Step 4. Project-Based Consultations

 Note
Check-List Blue cells are protected from editing
and have automatic calculation

sment Tool Version 2.0 Red cells are protected from editing
and have autamtic calculation

egulatory Framework
that signal the approach to the use of AI technology is
ms that decision-makers, adopters, and end-users can rely
upon.
ose 0 if NO or 1 if YES
elated programs, plans or provisions adopted or
0

officially recognized by
0
icies
r AI adoption 0
ed 0
r institution is established under the responsible
0

Total Score per Step 0

ep 2. Decisions
I technology for enhancing decision-making processes
ose 0 if NO or 1 if YES
ta and technologies to make policy decisions 0

klogs of work, cases and lessons learned to use

0

or equivalent analytical instruments to use data

0

able (by request) for decision-makers that are

0
sentative
 f data for decision-makers are in place 0

ar role) is present in the institution responsible

s person has authority to initiate and manage 0
n processes

ovided for decision-making processes by

0

d qualified team (including data scientists and

tors, gender experts, and professionals of 0
med to participate in decision-making processes

m has been formed, or AI Ethics themes are

0
ltidisciplinary team for decision-making

Total Score per Step 0

ssessment and mitigation

nsuring AI is adopted in a secure and ethical manner and

ed stakeholders, communities, researchers, and observer

ose 0 if NO or 1 if YES
ms and data has been developed and operationaliz 0
0
is in place to test datasets against biases and
omes via experience in applying frameworks, 0
other assessment tools
of the data is periodically conducted and
0
analysis is publicly available

design, development, maintenance, and

0
I system is ensured

on an Open Government Portal (Open Data)

0
mx/busca/organization/estado-de-guanajuato).
0
d to assess and fix data bias (e.g. AI Fairness
Google's PAIR, Datasheets for Datasets from 0
echnology has been established that includes
, logged details of all changes made to the
0
and logged details of the system’s reasons for
mendations

o develop and approve a project concept case

0
l management (i.e. AI council/committee)
0
rinciples are built into all systems by default
0
stage forward)

agreements or arrangements with appropriate

0
ng of personal information is involved

ty are ensured 0

ed. The methodological description of risk

0
s, tools, datasets is clear and available

red. Roles and responsibilities for system

0
are clear

Total Score per Step 0

ect-Based Consultations

ng transparency in AI decision-making to give users

at an AI system functions well.
ose 0 if NO or 1 if YES
h internal stakeholders (e.g. Strategic Policy
0
e, Program Policy, etc.) is established

h external stakeholders (e.g. Civil Society,

roups relevant for each specific project, etc.) is 0

n of AI tools are provided at the initial stages of 0

ng opinions after consultation is provided in

0
on the specialized portal)

valuating the system once the project is implemen 0

 Total Score per Step 0

5. Procurement
ated to the procurement, maintenance and change of AI
technologies.
ose 0 if NO or 1 if YES
are invited and engaged at early stages within 0
roject
nels are used to identify AI suppliers 0
supporting competition in the AI ecosystem
0

procurement procedures are established 0

cuses on a clear problem statement, rather than 0
olution
er include outcome-based requirements that
0
th challenges and opportunities
r includes an iterative approach to product
0

lem statement requires both the explanation of

0
ion of results from algorithms

ment includes the requirement of periodic

0
he technology and project

ment includes a requirement for transferring

0
g

mitted tenders is multidisciplinary to ensure

0
ucting the tender evaluation
ude explanations of AI technology, algorithms
0
s
g arrangements are in place for the procurement
0

Total Score per Step 0

p 6. End-of-life
s of the future functioning AI system.
ose 0 if NO or 1 if YES
rocured AI system, the data used and results
ectations are agreed to by all parties (e.g 0
s)
ations associated with the future AI system 0
and flowcharts on how to mitigate identified risks 0
cess of the AI system are established 0
le dialogue with the developers is established 0

Total Score per Step 0

Total Score for AI Checlist 0

ine version, the graph may be displayed with errors, so it

se.

tory Framework
Final Scoring
1 Step 1. Regulatory Framework 0
Step 2. Decisions 0
Step 3. Risk assessment and mitigatio 0
Step 2. Decisions
0.5
Step 4. Project-Based Consultations 0
Step 5. Procurement 0
Step 6. End-of-life 0

Step 3. Risk assessment and mitigation

ased Consultations
Intermediate
scores

Total scores

1
0