[FREE PDF sample] Introduction To Conformal Prediction With Python : A Short Guide For Quantifying Uncertainty Of Machine Learning Models 1st Edition Christoph Molnar ebooks
[FREE PDF sample] Introduction To Conformal Prediction With Python : A Short Guide For Quantifying Uncertainty Of Machine Learning Models 1st Edition Christoph Molnar ebooks
com
OR CLICK BUTTON
DOWNLOAD NOW
https://ebookmeta.com/product/learn-tensorflow-2-0-implement-machine-
learning-and-deep-learning-models-with-python-1st-edition-pramod-
singh/
ebookmeta.com
https://ebookmeta.com/product/switch-back-elliott-lisbon-cozy-
mystery-0-5-kendel-lynn-et-el/
ebookmeta.com
Fearless Like Us 1st Edition Krista Ritchie
https://ebookmeta.com/product/fearless-like-us-1st-edition-krista-
ritchie/
ebookmeta.com
https://ebookmeta.com/product/learning-ray-fifth-early-release-max-
pumperla/
ebookmeta.com
https://ebookmeta.com/product/cybersecurity-ops-with-bash-attack-
defend-and-analyze-from-the-command-line-1st-edition-paul-troncone-
carl-albing/
ebookmeta.com
https://ebookmeta.com/product/safe-laparoscopic-cholecystectomy-an-
illustrated-atlas-1st-edition-mohammad-ibrarullah/
ebookmeta.com
Introduction To Conformal
Prediction With Python
A Short Guide for Quantifying Uncertainty of Machine
Learning Models
Christoph Molnar
Introduction To Conformal Prediction With
Python
A Short Guide for Quantifying Uncertainty of Machine Learning Models
© 2023 Christoph Molnar, Germany, Munich
christophmolnar.com
For more information about permission to reproduce selections from this book,
write to [email protected].
2023, First Edition
Christoph Molnar
c/o MUCBOOK, Heidi Seibold
Elsenheimerstraße 48
80687 München, Germany
1 Summary 7
2 Preface 9
7 Classification 43
7.1 Back to the beans . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
7.2 The naive method doesn’t work . . . . . . . . . . . . . . . . . . . 45
7.3 The Score method is simple but not adaptive . . . . . . . . . . . 46
4
7.4 Use Adaptive Prediction Sets (APS) for conditional coverage . . . 51
7.5 Top-k method for fixed size sets . . . . . . . . . . . . . . . . . . . 58
7.6 Regularized APS (RAPS) for small sets . . . . . . . . . . . . . . . 59
7.7 Group-balanced conformal prediction . . . . . . . . . . . . . . . . 61
7.8 Class-Conditional APS (CCAPS) for coverage by class . . . . . . 63
7.9 Guide for choosing a conformal classification method . . . . . . . 64
11 Q & A 95
11.1 How do I choose the calibration size? . . . . . . . . . . . . . . . . 95
11.2 How do I make conformal prediction reproducible? . . . . . . . . 95
11.3 How does alpha affect the size of the prediction regions? . . . . . 95
11.4 What happens if I choose a large 𝛼 for conformal classification? . 96
11.5 How to interpret empty prediction sets? . . . . . . . . . . . . . . 96
11.6 Can I use the same data for calibration and model evaluation? . . 96
11.7 What if I find errors in the book or want to provide feedback? . . 97
5
12 Acknowledgements 98
References 99
6
1 Summary
A prerequisite for trust in machine learning is uncertainty quantification. Without
it, an accurate prediction and a wild guess look the same.
Yet many machine learning models come without uncertainty quantification. And
while there are many approaches to uncertainty – from Bayesian posteriors to
bootstrapping – we have no guarantees that these approaches will perform well
on new data.
At first glance conformal prediction seems like yet another contender. But con-
formal prediction can work in combination with any other uncertainty approach
and has many advantages that make it stand out:
Sound good? Then this is the right book for you to learn about this versatile,
easy-to-use yet powerful tool for taming the uncertainty of your models.
This book:
7
• Demonstrates how conformal prediction works for classification and regres-
sion
• Shows how to apply conformal prediction using Python
• Enables you to quickly learn new conformal algorithms
With the knowledge in this book, you’ll be ready to quantify the uncertainty of
any model.
8
2 Preface
My first encounter with conformal prediction was years ago, when I read a paper
on feature importance. I wasn’t looking for uncertainty quantification. Never-
theless, I tried to understand conformal prediction but was quickly discouraged
because I didn’t immediately understand the concept. I moved on.
About 4 years later, conformal prediction kept popping up on my Twitter and
elsewhere. I tried to ignore it, mostly successfully, but at some point I became
interested in understanding what conformal prediction was. So I dug deeper and
found a method that I actually find intuitive.
My favorite way to learn is to teach, so I decided to do a deep dive in the form of an
email course. For 5 weeks, my newsletter Mindful Modeler1 became a classroom
for conformal prediction. I didn’t know how this experiment would turn out. But
it quickly became clear that many people were eager to learn about conformal
prediction. The course was a success. So I decided to build on that and turn
everything I learned about conformal prediction into a book. You hold the results
in your hand (or in your RAM).
I love turning academic knowledge into practical advice. Conformal prediction is
in a sweet spot: There’s an explosion of academic interest and conformal predic-
tion holds great promise for practical data science. The math behind conformal
prediction isn’t easy. That’s one reason why I gave it a pass for a few years. But
it was a pleasant surprise to find that from an application perspective, conformal
prediction is simple. Solid theory, easy to use, broad applicability – conformal
prediction is ready. But it still lives mostly in the academic sphere.
With this book, I hope to strengthen the knowledge transfer from academia to
practice and bring conformal prediction to the streets.
1
https://mindfulmodeler.substack.com/
9
3 Who This Book Is For
This book is for data scientists, statisticians, machine learners and all other mod-
elers who want to learn how to quantify uncertainty with conformal prediction.
Even if you already use uncertainty quantification in one way or another, confor-
mal prediction is a valuable addition to your toolbox.
Prerequisites:
10
4 Introduction to Conformal
Prediction
In this chapter, you’ll learn
11
workers can use the uncertainty estimates to prioritize their review of the
claim and intervene if necessary.
• Uncertainty quantification can be used to improve the user experience in a
banking app. While the classification of financial transactions into “rent,”
“groceries,” and so on can be largely automated through machine learning,
there will always be transactions that are difficult to classify. Uncertainty
quantification can identify tricky transactions and prompt the user to clas-
sify them.
• Demand forecasting using machine learning can be improved by using un-
certainty quantification, which can provide additional context on the con-
fidence in the prediction. This is especially important in situations where
the demand must meet a certain threshold in order to justify production.
By understanding the uncertainty of the forecast, an organization can make
more informed decisions about whether to proceed with production.
ė Note
As a rule of thumb, you need uncertainty quantification whenever a point
prediction isn’t informative enough.
• The model is trained on a random sample of data, making the model itself
a random variable. If you were to train the model on a different sample
from the same distribution, you would get a slightly different model.
• Some models are even trained in a non-deterministic way. Think of random
weight initialization in neural networks or sampling mechanisms in random
forests. If you train a model with non-deterministic training twice on the
same data, you will get slightly different models.
• This uncertainty in model training is worse when the training dataset is
small.
12
• Hyperparameter tuning, model selection, and feature selection have the
same problem – all of these modeling steps involve estimation based on
random samples of data, which adds to uncertainty to the modeling process.
• The data may not be perfectly measured. The features or the target may
contain measurement errors, such as people filling out surveys incorrectly,
copying errors, and faulty measurements.
• Data sets may have missing values.
Some examples:
• Let’s say we’re predicting house values. The floor type feature isn’t always
accurate, so our model has to work with data that contains measurement
errors. For this and other reasons, the model will not always predict the
house value correctly.
• Decision trees are known to be unstable – small changes in the data can lead
to large differences in how the tree looks like. While this type of uncertainty
is “invisible” when only one tree is trained, it becomes apparent when the
model is retrained, since a new tree will likely have different splits.
• Image classification: Human labelers may disagree on how to classify an
image. A dataset consisting of different human labelers will therefore con-
tain uncertainty as the model will never be able to perfectly predict the
“correct” class, because the true class is up for debate.
13
(a) Clearly A Dog (b) Don’t let these dogs bamboozle you.
They want you to believe that they are
ghosts. They are not!
14
One classification is quite clear, because the probability is so high. In the other
case, it was a close call for the “cat” category, so we would assume that this
classification was less certain.
At first glance, aren’t we done when the model outputs probabilities and we use
them to get an idea of uncertainty? Unfortunately, no. Let’s explore why.
15
Random documents with unrelated
content Scribd suggests to you:
Then was the time for General Pavia’s action. Arthur Houghton,
correspondent to The Times at Madrid, gives, in his “French History
of the Restoration of the Bourbons,” the account of this coup in the
General’s own words; for, favoured by the soldiers’ friendship, Mr.
Houghton had the opportunity of hearing the story first-hand, and the
smart General, looking spruce and trim in his well-cut black frock,
would often talk to the Englishman, when he met him in the salons of
Madrid, of the way he took matters into his own hand when the
republican Parliament could not manage the Congress.
“No, no,” said the former Governor of Madrid, “I admitted nobody
into my counsel, but, under the stress of circumstances, I took all the
responsibility upon myself. When I heard how the Assembly had
given voice to a vote of want of confidence in Castelar, I thought the
hour had come; and as the session the next day increased in force
and disorder, whilst the hours of early dawn succeeded those of the
evening and the night in fruitless and violent discussion, I called a
company of the Civil Guard, and another of the Cazadores, and, to
their surprise, I led them to the square in front of the Congress, and
stationed them all round the building. Then, entering the Parliament
with a few picked men, I surprised the deputies by ordering them to
leave the House. A few shots were fired in the corridor on those who
sought to defy the military order, so the members did not long resist,
and by four o’clock in the morning I found myself in complete
command of the House. I called a Committee, with the power to form
a Ministry, of which General Serrano was once more elected
President, and thus ensued the second period of the republic.”
This brilliant and successful coup reminds one of that of our
Oliver Cromwell when he freed the country of a particular
Government; but in this case of military sway in Spain General Pavia
acted from no aims of self-interest, but only for the restoration of
order, which it was his duty as Governor of the city to preserve.
During the second period of the republic, which lasted from
January 4, 1874, till December 30 of the same year, Serrano had his
hands weighted with two civil wars—the never-ceasing one of
Carlism in the Peninsula, as well as that of Cuba—and, as Francisco
Paréja de Alarcon says, in the criticism which he publishes in the
above-mentioned work on this period, the Government formed under
Serrano proved unable to restore order and save Spain from the
dishonour which was threatening it.
So when the Ministers heard of the rising at Sagunto, on
December 29, 1874, for the restoration of the monarchy, they knew
that the movement was really supported by leading military men,
who had been inspired thereto by the ladies of the land, who
resented the irreligion and disorder of the republic; and, as they saw
that resistance would only lead to another disastrous civil war, they
resigned their posts peacefully.
It was thus that the son of Isabella II. was raised to the throne.
And Alarcon says: “The hypocritical banner of ‘the country’s honour’
was set aside; for had it not meant the support of a foreign
monarchy, destitute of prestige; and then an unbridled, antisocial,
impious, and anarchical republic, which was a blot on the history of
our unhappy Spain in these latter days, which have been so full of
misfortunes under the government of the ambitious parties which
harrowed and exploited under different names and banners?”
The Circulo Hispano Ultramarino in Barcelona, agitating
continually for the restoration of Alfonso XII., was a strong agent in
the monarchical movement. Figuerola Ferretti worked strenuously as
secretary of the society, and this officer is the possessor of the only
escutcheon signed by Alfonso XII., in which he paid tribute to the
Colonel’s valiant conduct in the Cuban War of 1872.
It is interesting to see that the opinion of the republic published in
“Contemporaneous Truths” by this Ferretti was echoed by the great
leader of the party himself, for Señor Castelar writes: “There were
days during that summer of 1874 in which our Spain seemed
completely ruined. The idea of legality was so lost that anybody
could assume power, and notify the fact to the Cortes, and those
whose office it was to make and keep the laws were in a perpetual
ferment against them.
“It was no question then, as before, of one Ministry replacing
another, nor one form of government substituting another; but a
country was divided into a thousand parts, like the Kalifat of Cordova
after its fall, and the provinces were inundated by the most out-of-
the-way ideas and principles.”
When the great republican speaks in such a derogatory way of
the republic of which he was the leader, it is not strange that public
opinion turned to the restoration of the Bourbons as the salvation of
the country. Society clamoured for such balls and entertainments as
had formerly taken place at Court, or which had been patronized by
the palace, and the dreary disorder wearied both politicians and
patriots.
The house of the Dukes of Heredia-Spinola never ceased to be
the scene of the reunion of Alfonsists, and as General Martinez
Campos played his daily game of tresillo at their table, many
expressions of hope for the return of the ex-Queen’s son fell upon
his ears; whilst the Countess of Tacon, who had been Lady-in-
Waiting to the little Prince of Asturias as a child, was loud in her
opinions. It is interesting to note that this lady subsequently filled the
same office for the restored King’s little daughter, the Princess of
Asturias, Doña Maria de las Mercedes.
From a social point of view the salon of the old Countess of
Montijo ranked foremost in Madrid, and it assembled within its walls
the frequenters of Court society in the reign of Isabella. Scenes from
“Don Quixote” were given with great success at the Countess’s little
theatre; and the year of the restoration was marked by a very
successful dramatic representation, in which some of the members
of the old nobility took part.
Moreover, the services held every Friday in the private chapel of
the mansion, where great preachers made remarkable orations,
were a protest against the irreligion of the period. On these
occasions ladies of Court society, among whom may be noted Clara
Hunt, wife of one of the diplomats of the English Embassy—who was
quite a notable singer—gave proofs of their talent.
The niece of the Count of Nava de Tajo was another of the
distinguished ladies who frequented the salon of the Countess of
Montijo. The Count was varied in his interests. One afternoon he
paid a series of visits, beginning with the Pope’s Nuncio, going on to
the house of Canovas, then to Roque Barcia, who was asking for
subscriptions for his famous dictionary, and ending with the unhappy
Lopez Bago, who was seeking support for his Review of the Salons,
of which only three or four numbers were ever published.
CHAPTER XV
THE REVIVAL OF COURT LIFE IN SPAIN UNDER ALFONSO XII.
1874–1884
The foregoing brief sketch of the political and social life in Spain
during the republic will have given some idea of the joy which filled
Spanish hearts at seeing the Bourbons once more on the throne of
Spain in the person of Alfonso XII. Madrid indeed was wild with joy
when the little Prince whom we saw at eleven years of age, in his
blue velvet suit and lace collar, leaving his country as an exile, with
his mother and family, re-entered the royal palace as a young man
eighteen years old in January, 1875, having wisely passed through
Catalonia, which Martinez Campos had gained over to the cause,
and pleased the people by saying: “I wish to be King of all
Spaniards.”
As Isabella had abdicated in favour of her son on June 26, 1870,
there was no impediment to his taking the oath of coronation soon
after he was summoned to the Spanish capital. Of a good figure,
gentlemanly, and well cultured, Alfonso added the art of good
dressing to his other attractions, and the excellent taste and cut of
his clothes led to his being called “the Beau Brummell of Spain.”
K I N G A L F O N S O X I I . V I S I T I N G C H O L E R A PAT I E N T S AT
ARANJUEZ
D O N C A R L O S , P R I N C E O F A S T U R I A S , A N D H I S L AT E W I F E , T H E
I N FA N TA M E R C E D E S
It was in 1882 the King and Queen paid a visit to the Duke and
Duchess of Montpensier at their beautiful Palace of Sanlucar de
Barrameda, and the Queen won the hearts of her host and hostess
by her charming manners and the admiration with which she always
spoke of their daughter, the late wife of Alfonso.
On November 12, 1882, the Infanta Maria Teresa was born, and
two days later she was baptized with the customary ceremony.
On April 2, 1883, the King’s sister, Doña de la Paz, was married
very quietly to Prince Lewis Ferdinand of Bavaria. The Prince is a
very able surgeon, and when he comes to Madrid he delights in
going to the military hospital and exhibiting his scientific skill on
some soldier-patient.
The newly wedded pair laid the foundation-stone of the Cathedral
of the Almudena, and, according to the custom, the Princess de la
Paz placed in the casket a poem from her own pen to the Virgin of
the Almudena. The departure of the Infanta de la Paz left the Infanta
Eulalia with no companion in her musical and artistic tastes, for the
sisters had worked, played, painted, and poetized, together.
In September, 1883, Alfonso XII. went to France and Germany.
True to his old friends, the King went to see the Warden of the
Teresian College at his private house. As he was not at home,
Alfonso asked for a pencil and paper to write him a note, which he
handed to the servant. When she saw that the letter ran,
she fell on her knees and entreated forgiveness for her stupidity in
having asked the royal visitor into the kitchen.
But Alfonso, with his usual kindness, expressed interest in this,
the first kitchen he had ever seen. He asked many questions about
the utensils, and showed great curiosity about the use of a ceramic
vessel, which, according to the description he subsequently gave
and the sketch he made of it to show the Court officials, proved to be
an egg-poacher.
The enthusiastic reception accorded to Alfonso at Homburg
excited the ire of the French, and so antagonistic was the exhibition
of public feeling as the young King was crossing Paris alone that he
informed the President of the Republic that he would recall his
Ambassador at once. This prompt act brought the necessary
apology, and the King of Spain subsequently attended the banquet
given in his honour at the Elysée, at which the Minister of War was
absent, as the President of France had asked him to send in his
resignation.
The news of this contretemps reached Spain, and when the
Queen returned from La Granja to Madrid she was at first quite
alarmed at the enthusiasm shown by the people at the station. She
clasped her children to her breast, and seemed to think she was on
the brink of a revolution. But her fears were soon stilled when
somebody shouted: “Señora, the Spanish people are only protesting
against the recent events in Paris.”
The return of the King from France saw an ovation of equal
enthusiasm, and, in defiance of all Court etiquette, the people
pressed up the staircases and into the galleries of the palace, crying:
“Viva el Rey y la Reina!”
It was on Maunday Thursday, 1884, that the Court went for the
last time in state to make the customary visits on foot to the chief
churches of the capital. There was the usual service in the morning
in the chapel of the palace, the washing of the beggars’ feet and
feeding them,[21] and the solemn, imposing public procession at
three o’clock in the afternoon. The streets were strewed with tan to
soften the cobbled stones to the feet of the ladies, whose high-
heeled velvet shoes rather impeded their walk. The streets were
lined with troops, and the Plazas de Oriente, Mayor, and La
Encarnacion, were respectively filled with the regiment of the
Princess of Pavia and the artillery.
[21] This ceremony is described on pp. 332-4.
“Nobody,” says the writer in this appeal, “has the courage to warn
you of the impending evil. When the doctors order you change of
climate, the Government opposes the course for reasons of State.
‘Reasons of State’ imperil the life of a man! And a man to whom we
owe so much!
“Therefore, even as a republican, I beg you, as the occupier of
the throne, to look to your health, if it be only to overthrow some
iniquitous plan, or some unworthy object which is contingent on your
illness; and if scientists think it well for you to pass the winter in
some other place in Spain, or abroad, follow their counsel, and not
that of interested politicians, in sacrificing your life to their ambitions.”
It was certainly true that the King was overborne by the intrigues
of the politicians in the palace. Even in such a little social matter as
that of wishing to go in costume to a fancy ball, the King could not
have his own way, for Canovas showed such aversion to Alfonso
donning fancy attire for the occasion that he had to abandon the idea
and wear his ordinary dress.
If such influence had been used to the prevention of the King
favouring a danseuse like Elena Sanz, which brought so much
sorrow and so many complications in the Royal Family, his life might
certainly have been prolonged. It was true that the doctors advised
the King’s wintering in Andalusia, but “State reasons” led to the
failing Sovereign being exposed to the colder climate and sharp
winds of the Palace of the Pardo, where politicians could use their
influence with the invalid, and remind him continually that he alone
was the arbiter of parties.
Alfonso was only twenty-seven years of age when he felt he was
doomed to an early death; but his natural energy led him to take
horse exercise, despatch business with his Ministers every day, and,
in spite of daily increasing weakness, to do as much as possible.
If his longing for the sea-breezes of San Sebastian had been
gratified, his life might have been prolonged; but politicians gave little
heed to the plea, and their authority was paramount.
On November 24, 1894, the royal invalid was seized with
faintness when he came in from a walk. Queen Maria Cristina,
Queen Isabella, and the Duchess of Montpensier, were called to his
side. Seeing his wife by him when he recovered consciousness, the
King embraced her, and the alarming symptoms vanished for a time;
but the following day he was seized with another fainting fit, which
proved fatal.
We read in La Ilustracion Española of this date, that when Queen
Maria Cristina was told by Dr. Riedel that all was over, she fell
weeping at the head of the bed of her unhappy husband, whilst
covering his hand with kisses.
D E AT H O F A L F O N S O X I I .