Need for interpretability

What does interpretability actually mean ?

Two examples of Interpretable ML approaches

Chapter 5: Interpretable Machine Learning

Christophe Ley

University of Luxembourg, 2023-2024

Statistical Modelling

Christophe Ley University of Luxembourg, 2023-2024 1 / 29

 Need for interpretability
What does interpretability actually mean ?
Two examples of Interpretable ML approaches

Outline

1 Need for interpretability

2 What does interpretability actually mean ?

3 Two examples of Interpretable ML approaches

LIME
Partial Dependence Plots

Christophe Ley University of Luxembourg, 2023-2024 2 / 29

 Need for interpretability
What does interpretability actually mean ?
Two examples of Interpretable ML approaches

Responsible AI

Goal from the OECD : “AI that is innovative and trustworthy and
respects human rights and democratic values”

The following aspects need to be ensured :

• Fairness and robustness (reliability)
• Data privacy
• Human control as much as possible
• AI must be explainable

Christophe Ley University of Luxembourg, 2023-2024 3 / 29

 Need for interpretability
What does interpretability actually mean ?
Two examples of Interpretable ML approaches

Explainable AI or Interpretable Machine Learning

Very active and essential subfield of AI !

Basic idea : do not accept black-box predictions but understand what

are the driving factors of an AI output ⇒ otherwise no control
possible !

Christophe Ley University of Luxembourg, 2023-2024 4 / 29

 Need for interpretability
What does interpretability actually mean ?
Two examples of Interpretable ML approaches

Quality questions

Although a high accuracy (i.e. low prediction error) is an indication for

a reliable Machine Learning model, it does not provide an answer to
the following questions :
Are the results based on a sound logic ?
What covariates are influential in the decision-making process ?
Does the model rely heavily on ’accidental’ patterns present in
the training data ?

Disclaimer : besides being based on the literature on Interpretable

ML, this chapter strongly relies on the Bachelor thesis by my former
student M.J. van Noort.

Christophe Ley University of Luxembourg, 2023-2024 5 / 29

 Need for interpretability
What does interpretability actually mean ?
Two examples of Interpretable ML approaches

Are the results based on a sound logic ?

This question appears within the context of ethics and reliability.
Ethical considerations can be required for real-life applications of
machine learning. Example :

A self-driving car finds itself in a situation where it would have to

choose between the certain death of one out of two individuals, say
an infant and an elderly person (which the car recognizes as such).
The human driver is unable to interfere as his/her reflex time is too
long, but the car’s software can. Depending on the decision of the
car’s algorithm, either the infant or the elder will be hit. What should
the algorithm’s decision be, and why ?

In such cases, the ML decision must be open to human verification.

The information regarding the algorithm’s decision must be
formulated in such a manner that a person is able to verify the
soundness of the decision. In short : the decision-making process
has to be interpretable.
Christophe Ley University of Luxembourg, 2023-2024 6 / 29
 Need for interpretability
What does interpretability actually mean ?
Two examples of Interpretable ML approaches

What covariates are influential in the decision-making

process ?
This is in some sense a follow-up question to the first question. The
question ties in directly with one of the traditional aims of statistics :
significance testing.

Pure prediction enjoys the benefit of being able to exploit all patterns,
possibly involving many covariates and their mutual interactions,
present in the dataset in order to get the best prediction. On the
contrary, significance testing generally focuses on verifying whether
one covariate has a sufficiently large effect on the outcome.

Coming back to the self-driving car example, the answer to the

second question could be that the car would decide to save the infant,
as it has the highest remaining life expectancy, or to save the elder,
due to minimal damage to the car’s infrastructure. In these cases, the
covariates “life expectancy” and “damage” are decisive.

Christophe Ley University of Luxembourg, 2023-2024 7 / 29

 Need for interpretability
What does interpretability actually mean ?
Two examples of Interpretable ML approaches

Does the model rely heavily on ’accidental’ patterns

present in the training data ?

The question anticipates on the problem of overfitting a model. Here,

the advantage of prediction methods of being able to exploit patterns
in the data more easily than significance testing turns into a
weakness. An over-eagerness of prediction methods to exploit all
data patterns plays right into the hands of overfitting, which can
undermine the performance and reliability of the prediction. As
significance testing is less tailored towards exploiting all possible
patterns in the data, it tends to be more robust towards overfitting
than prediction.

Christophe Ley University of Luxembourg, 2023-2024 8 / 29

 Need for interpretability
What does interpretability actually mean ?
Two examples of Interpretable ML approaches

Outline

1 Need for interpretability

2 What does interpretability actually mean ?

3 Two examples of Interpretable ML approaches

LIME
Partial Dependence Plots

Christophe Ley University of Luxembourg, 2023-2024 9 / 29

 Need for interpretability
What does interpretability actually mean ?
Two examples of Interpretable ML approaches

Interpretability itself is a very context-dependent concept. An

explanation that would suffice for one person may be
incomprehensible for another.

A general explanation of the average process behind an algorithm

might not explain how a specific decision has been reached.

Therefore, we need to categorise distinct types of desiderata for

explanations.

Christophe Ley University of Luxembourg, 2023-2024 10 / 29

 Need for interpretability
What does interpretability actually mean ?
Two examples of Interpretable ML approaches

Interpretable covariates

Obviously, an explanation can not be understandable if the “building

blocks” of the explanations are not understandable for its intended
recipient(s).

For example, a patient’s diagnosis could be explained in terms of

his/her medical properties such as blood pressure, saturation levels,
gene expression, etc. Medical experts are able to verify this
information and act accordingly.

Well-known dimensionality reduction techniques, for instance

Principal Component Analysis, can aid in composing features out of a
high number of covariates, but are not designed to yield
understandable properties.

Christophe Ley University of Luxembourg, 2023-2024 11 / 29

 Need for interpretability
What does interpretability actually mean ?
Two examples of Interpretable ML approaches

Comprehensible aggregation

When an interpretable set of covariates is available, a second

prerequisite will be to connect those covariates in a way that is
comprehensible for a user. Familiar operations, such as addition or
logical operators, are in general considered to be understandable.
More advanced operations such as exponential or
log-transformations are less intuitive.

Decision trees and linear models are preferred from this perspective
over Artificial neural networks.

The amount of covariates in a model also plays a big part in the

interpretability of an explanation. Consequently, comprehensible
aggregation of covariates spans both the manner in which the
covariates are combined as well as the quantity.

Christophe Ley University of Luxembourg, 2023-2024 12 / 29

 Need for interpretability
What does interpretability actually mean ?
Two examples of Interpretable ML approaches

Local Versus Global fidelity

Another important precondition is the scope, or scale, at which
explanations are provided.

Think of the self-driving car. Explanations for day-to-day normal traffic

may not be sufficient for this specific situation.

We differ between local fidelity, i.e. an explanation that is accurate

and sufficiently detailed for a model’s prediction in a given situation,
and global fidelity, which concerns understanding the global
decision-making process.

Local fidelity thus allows for explanation of a model’s prediction at a

single point, whilst global fidelity covers explanations of the model
itself, mostly for the whole point space. Explanations that cater
towards one of these two types of fidelity may differ quite drastically,
as covariates important to a local prediction may not be equally
important within the model’s general prediction mechanisms, or the
other way around.
Christophe Ley University of Luxembourg, 2023-2024 13 / 29
 Need for interpretability
What does interpretability actually mean ?
Two examples of Interpretable ML approaches

Interpretable algorithms vs. interpreting an algorithm

Most ML algorithms fall into two categories : (i) algorithms designed

to construct a well-performing, yet interpretable ML model, versus (ii)
techniques that focus on providing an explanation to accompany an
already existing algorithm.

The first category are “interpretable models”, while the second covers
“post-hoc techniques”, where interpretability is regarded as an
addition to a model that was constructed in advance.

Christophe Ley University of Luxembourg, 2023-2024 14 / 29

 Need for interpretability
What does interpretability actually mean ?
Two examples of Interpretable ML approaches

Model-agnosticism

Not every method from category (ii) is able to extend any model. For
instance, guided backpropagation is a method that is limited to the
convolutional neural network model. This method makes clever use of
the model’s structure to provide a qualitative explanation, hence is
tailored to take advantage of the neural network’s structure, and can’t
be applied to other machine learning models.

Model-agnosticism can be referred to as an explainer able to explain

any model.

The benefits of model-agnosticism are mainly practical, as such

methods can be applied without regard for the prediction model
(freedom of choice). This increased freedom may come at an indirect
cost, as one loses the opportunity to make proper use of the
decision-making process underpinning the model to explain.

Christophe Ley University of Luxembourg, 2023-2024 15 / 29

 Need for interpretability
LIME
What does interpretability actually mean ?
Partial Dependence Plots
Two examples of Interpretable ML approaches

Outline

1 Need for interpretability

2 What does interpretability actually mean ?

3 Two examples of Interpretable ML approaches

LIME
Partial Dependence Plots

Christophe Ley University of Luxembourg, 2023-2024 16 / 29

 Need for interpretability
LIME
What does interpretability actually mean ?
Partial Dependence Plots
Two examples of Interpretable ML approaches

Outline

1 Need for interpretability

2 What does interpretability actually mean ?

3 Two examples of Interpretable ML approaches

LIME
Partial Dependence Plots

Christophe Ley University of Luxembourg, 2023-2024 17 / 29

 Need for interpretability
LIME
What does interpretability actually mean ?
Partial Dependence Plots
Two examples of Interpretable ML approaches

Local Interpretable Model-agnostic Interpretations

LIME aims to provide trustworthy explanations for the point
predictions of any machine learning method. It takes a model and
constructs an explanation for one prediction at a time, hence the
name.

Suppose we wish to understand how a Random Forest predicts

outcome y based on set of covariates x . The basic idea of LIME can
be described as follows :
• Create perturbed data around x ; the number of perturbed
observations is a hyperparameter (can be e.g. 5000)
• Predict the outcome for each perturbed observation with the
Random Forest
• Create discretized features, to make continuous covariates
understandable
• Create a similarity score for each perturbed observation to reflect
its “distance” to the original x
• (One may wish to do feature selection)
• Choose an interpretable model and explain the prediction
Christophe Ley University of Luxembourg, 2023-2024 18 / 29
 Need for interpretability
LIME
What does interpretability actually mean ?
Partial Dependence Plots
Two examples of Interpretable ML approaches

The last step requires more details.

An interpretable model I with which we wish to explain the ML model

M needs to satisfy the following two criteria :
1 I has to be interpretable, and simple explanations are favored
over complicated ones.
2 M and I have to behave similarly when it comes to predicting x.

To find the best choice for I among a set I of interpretable models,

one can define a complexity measure Ω(I) and a similarity score
S(M, I, πx ) with πx the distance measure to x. The LIME explanation
for a given model M at observation x is then given by the
interpretable model

argminI∈I {Ω(I) + S(M, I, πx )}.

A popular choice for I is the set of sparse linear regression models

like LASSO or Ridge regression and decision trees.
Christophe Ley University of Luxembourg, 2023-2024 19 / 29
 Need for interpretability
LIME
What does interpretability actually mean ?
Partial Dependence Plots
Two examples of Interpretable ML approaches

With a chosen model I, we then explain the prediction. From the

similarity score, each perturbed observation gets a weight for the final
prediction.

Suppose we choose a linear regression model. For each perturbed

observation, we thus get linear regression coefficients for each
feature. These get weighted with the similarity score, and the result
gives the explanation for the ML model.

Christophe Ley University of Luxembourg, 2023-2024 20 / 29

 Need for interpretability
LIME
What does interpretability actually mean ?
Partial Dependence Plots
Two examples of Interpretable ML approaches

Concrete example
Data about the details of patients who have tested positive for
COVID-19 along with their demographic details, age, sex, symptoms,
and outcome. Using this data an ML model predicts the likelihood of a
patient recovering given the age, sex and symptoms of the patient.

Figure – Image taken from https ://towardsdatascience.com/unboxing-the-

black-box-using-lime-5c9756366faf
Christophe Ley University of Luxembourg, 2023-2024 21 / 29
 Need for interpretability
LIME
What does interpretability actually mean ?
Partial Dependence Plots
Two examples of Interpretable ML approaches

Visualization of LIME by Ribeiro, Singh and Guestrin :

“Why Should I Trust You ?” Explaining the Predictions
of Any Classifier

Christophe Ley University of Luxembourg, 2023-2024 22 / 29

 Need for interpretability
LIME
What does interpretability actually mean ?
Partial Dependence Plots
Two examples of Interpretable ML approaches

Outline

1 Need for interpretability

2 What does interpretability actually mean ?

3 Two examples of Interpretable ML approaches

LIME
Partial Dependence Plots

Christophe Ley University of Luxembourg, 2023-2024 23 / 29

 Need for interpretability
LIME
What does interpretability actually mean ?
Partial Dependence Plots
Two examples of Interpretable ML approaches

Partial Dependence Plots

PDPs help visualize the relationship between a subset of the features

(typically 1-3) and the response while accounting for the average
effect of the other predictors in the model. They are particularly
effective with black box models like random forests and support
vector machines.

The idea is to add information on top of the classical variable

importance measures by averaging out the impact of all other
features.

Christophe Ley University of Luxembourg, 2023-2024 24 / 29

 Need for interpretability
LIME
What does interpretability actually mean ?
Partial Dependence Plots
Two examples of Interpretable ML approaches

Consider the features x = (x1 , . . . , xp ) with associated prediction

fˆ(x). Suppose we are interested in a subset zs of features, and we
write zc = x \ zs its complement. Then the partial dependence of the
prediction on zs is defined as
h i
fs (zs ) = Ezc fˆ(zs , zc )

which in practice is given by

n
1Xˆ
f¯s (zs ) = f (zs , zi,c )
n i=1

with zi,c , i = 1, . . . , n, the values of zc occurring in the training sample.

Christophe Ley University of Luxembourg, 2023-2024 25 / 29

 Need for interpretability
LIME
What does interpretability actually mean ?
Partial Dependence Plots
Two examples of Interpretable ML approaches

Example from Greenwell : “pdp : An R Package for

Constructing Partial Dependence Plots”

Christophe Ley University of Luxembourg, 2023-2024 26 / 29

 Need for interpretability
LIME
What does interpretability actually mean ?
Partial Dependence Plots
Two examples of Interpretable ML approaches

Variable importance from Random Forest with 500 trees. Out-of-bag

R2 of 0.89 is obtained.

Christophe Ley University of Luxembourg, 2023-2024 27 / 29

 Need for interpretability
LIME
What does interpretability actually mean ?
Partial Dependence Plots
Two examples of Interpretable ML approaches

Single PDP

Christophe Ley University of Luxembourg, 2023-2024 28 / 29

 Need for interpretability
LIME
What does interpretability actually mean ?
Partial Dependence Plots
Two examples of Interpretable ML approaches

Multiple PDPs

Christophe Ley University of Luxembourg, 2023-2024 29 / 29