Clever Algorithms Nature Inspired Programming Recipes 1ST Edition Jason Brownlee - The ebook in PDF format is ready for download
Clever Algorithms Nature Inspired Programming Recipes 1ST Edition Jason Brownlee - The ebook in PDF format is ready for download
https://ebookultra.com/download/authoritarianism-in-an-age-of-
democratization-1st-edition-jason-brownlee/
https://ebookultra.com/download/sql-server-t-sql-recipes-4th-edition-
jason-brimhall/
https://ebookultra.com/download/awesome-game-creation-no-programming-
required-3rd-edition-jason-darby/
https://ebookultra.com/download/data-algorithms-recipes-for-scaling-
up-with-hadoop-and-spark-1st-edition-mahmoud-parsian/
Handbook of Intelligent and Sustainable Smart Dentistry
Nature and Bio Inspired Approaches Processes Materials 1st
Edition Ajay Kumar
https://ebookultra.com/download/handbook-of-intelligent-and-
sustainable-smart-dentistry-nature-and-bio-inspired-approaches-
processes-materials-1st-edition-ajay-kumar/
https://ebookultra.com/download/new-england-open-house-
cookbook-300-recipes-inspired-by-the-bounty-of-new-england-1st-
edition-sarah-leah-chase/
https://ebookultra.com/download/scala-cookbook-recipes-for-object-
oriented-and-functional-programming-1st-edition-alvin-alexander/
Clever Algorithms
Nature-Inspired Programming Recipes
ii
Cover Image
© Copyright 2011 Jason Brownlee. All Reserved.
Clever Algorithms: Nature-Inspired Programming Recipes
© Copyright 2011 Jason Brownlee. Some Rights Reserved.
First Edition. LuLu. January 2011
ISBN: 978-1-4467-8506-5
Webpage
Source code and additional resources can be downloaded from the books
companion website online at http://www.CleverAlgorithms.com
Contents
Foreword vii
Preface ix
I Background 1
1 Introduction 3
1.1 What is AI . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.2 Problem Domains . . . . . . . . . . . . . . . . . . . . . . . . 9
1.3 Unconventional Optimization . . . . . . . . . . . . . . . . . 13
1.4 Book Organization . . . . . . . . . . . . . . . . . . . . . . . 16
1.5 How to Read this Book . . . . . . . . . . . . . . . . . . . . 19
1.6 Further Reading . . . . . . . . . . . . . . . . . . . . . . . . 20
1.7 Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . 21
II Algorithms 27
2 Stochastic Algorithms 29
2.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
2.2 Random Search . . . . . . . . . . . . . . . . . . . . . . . . . 30
2.3 Adaptive Random Search . . . . . . . . . . . . . . . . . . . 34
2.4 Stochastic Hill Climbing . . . . . . . . . . . . . . . . . . . . 39
2.5 Iterated Local Search . . . . . . . . . . . . . . . . . . . . . . 43
2.6 Guided Local Search . . . . . . . . . . . . . . . . . . . . . . 49
2.7 Variable Neighborhood Search . . . . . . . . . . . . . . . . . 55
2.8 Greedy Randomized Adaptive Search . . . . . . . . . . . . . 60
2.9 Scatter Search . . . . . . . . . . . . . . . . . . . . . . . . . 66
2.10 Tabu Search . . . . . . . . . . . . . . . . . . . . . . . . . . . 73
2.11 Reactive Tabu Search . . . . . . . . . . . . . . . . . . . . . 79
iii
iv Contents
3 Evolutionary Algorithms 87
3.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87
3.2 Genetic Algorithm . . . . . . . . . . . . . . . . . . . . . . . 92
3.3 Genetic Programming . . . . . . . . . . . . . . . . . . . . . 99
3.4 Evolution Strategies . . . . . . . . . . . . . . . . . . . . . . 108
3.5 Differential Evolution . . . . . . . . . . . . . . . . . . . . . 114
3.6 Evolutionary Programming . . . . . . . . . . . . . . . . . . 120
3.7 Grammatical Evolution . . . . . . . . . . . . . . . . . . . . 126
3.8 Gene Expression Programming . . . . . . . . . . . . . . . . 134
3.9 Learning Classifier System . . . . . . . . . . . . . . . . . . . 141
3.10 Non-dominated Sorting Genetic Algorithm . . . . . . . . . . 152
3.11 Strength Pareto Evolutionary Algorithm . . . . . . . . . . . 160
IV Appendix 411
A Ruby: Quick-Start Guide 413
A.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . 413
A.2 Language Basics . . . . . . . . . . . . . . . . . . . . . . . . 413
A.3 Ruby Idioms . . . . . . . . . . . . . . . . . . . . . . . . . . 417
A.4 Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . 419
Index 421
vi Contents
Foreword
Melbourne, Australia
2010
vii
viii Foreword
Preface
ix
x Preface
‘best guess’. At the other end of the spectrum, a badly described algorithm
may be an example of bad science and the failure of the scientific method,
where the inability to understand and implement a technique may prevent
the replication of results, the application, or the investigation and extension
of a technique.
The software I produced provided a first step solution to this problem: a
set of working algorithms implemented in a (somewhat) consistent way and
downloaded from a single location (features likely provided by any library of
artificial intelligence techniques). The next logical step needed to address this
problem is to develop a methodology that anybody can follow. The strategy
to address the open problem of poor algorithm communication is to present
complete algorithm descriptions (rather than just implementations) in a
consistent manner, and in a centralized location. This book is the outcome
of developing such a strategy that not only provides a methodology for
standardized algorithm descriptions, but provides a large corpus of complete
and consistent algorithm descriptions in a single centralized location.
The algorithms described in this work are practical, interesting, and
fun, and the goal of this project was to promote these features by making
algorithms from the field more accessible, usable, and understandable.
This project was developed over a number years through a lot of writing,
discussion, and revision. This book has been released under a permissive
license that encourages the reader to explore new and creative ways of
further communicating its message and content.
I hope that this project has succeeded in some small way and that you
too can enjoy applying, learning, and playing with Clever Algorithms.
Jason Brownlee
Melbourne, Australia
2011
Acknowledgments
This book could not have been completed without the commitment, passion,
and hard work from a large group of editors and supporters.
A special thanks to Steve Dower for his incredible attention to detail
in providing technical and copy edits for large portions of this book, and
for his enthusiasm for the subject area. Also, a special thanks to Daniel
Angus for the discussions around the genesis of the project, his continued
support with the idea of an ‘algorithms atlas’ and for his attention to detail
in providing technical and copy edits for key chapters.
In no particular order, thanks to: Juan Ojeda, Martin Goddard, David
Howden, Sean Luke, David Zappia, Jeremy Wazny, and Andrew Murray.
Thanks to the hundreds of machine learning enthusiasts who voted on
potential covers and helped shape what this book became. You know who
you are!
Finally, I would like to thank my beautiful wife Ying Liu for her unre-
lenting support and patience throughout the project.
xi
xii Acknowledgments
Part I
Background
1
Chapter 1
Introduction
1.1 What is AI
1.1.1 Artificial Intelligence
The field of classical Artificial Intelligence (AI) coalesced in the 1950s
drawing on an understanding of the brain from neuroscience, the new
mathematics of information theory, control theory referred to as cybernetics,
and the dawn of the digital computer. AI is a cross-disciplinary field
of research that is generally concerned with developing and investigating
3
4 Chapter 1. Introduction
Neat AI
The traditional stream of AI concerns a top down perspective of problem
solving, generally involving symbolic representations and logic processes
that most importantly can explain why the systems work. The successes of
this prescriptive stream include a multitude of specialist approaches such
1.1. What is AI 5
Scruffy AI
There have been a number of thrusts in the field of AI toward less crisp
techniques that are able to locate approximate, imprecise, or partially-true
solutions to problems with a reasonable cost of resources. Such approaches
are typically descriptive rather than prescriptive, describing a process for
achieving a solution (how), but not explaining why they work (like the
neater approaches).
Scruffy AI approaches are defined as relatively simple procedures that
result in complex emergent and self-organizing behavior that can defy
traditional reductionist analyses, the effects of which can be exploited for
quickly locating approximate solutions to intractable problems. A common
characteristic of such techniques is the incorporation of randomness in
their processes resulting in robust probabilistic and stochastic decision
making contrasted to the sometimes more fragile determinism of the crisp
approaches. Another important common attribute is the adoption of an
inductive rather than deductive approach to problem solving, generalizing
solutions or decisions from sets of specific observations made by the system.
Evolutionary Computation
A paradigm that is concerned with the investigation of systems inspired by
the neo-Darwinian theory of evolution by means of natural selection (natural
selection theory and an understanding of genetics). Popular evolutionary
algorithms include the Genetic Algorithm, Evolution Strategy, Genetic
and Evolutionary Programming, and Differential Evolution [4, 5]. The
evolutionary process is considered an adaptive strategy and is typically
applied to search and optimization domains [26, 28].
Swarm Intelligence
A paradigm that considers collective intelligence as a behavior that emerges
through the interaction and cooperation of large numbers of lesser intelligent
agents. The paradigm consists of two dominant sub-fields 1) Ant Colony
Optimization that investigates probabilistic algorithms inspired by the
foraging behavior of ants [10, 18], and 2) Particle Swarm Optimization that
investigates probabilistic algorithms inspired by the flocking and foraging
behavior of birds and fish [30]. Like evolutionary computation, swarm
intelligence-based techniques are considered adaptive strategies and are
typically applied to search and optimization domains.
Fuzzy Intelligence
Fuzzy Intelligence is a paradigm that is concerned with the investigation of
fuzzy logic, which is a form of logic that is not constrained to true and false
determinations like propositional logic, but rather functions which define
approximate truth, or degrees of truth [52]. Fuzzy logic and fuzzy systems
are a logic system used as a reasoning strategy and are typically applied to
expert system and control system domains.
selection, negative selection, the dendritic cell algorithm, and immune net-
work algorithms. The immune-inspired adaptive processes vary in strategy
and show similarities to the fields of Evolutionary Computation and Artifi-
cial Neural Networks, and are typically used for optimization and pattern
recognition domains [15].
1.1.4 Metaheuristics
Another popular name for the strategy-outcome perspective of scruffy AI is
metaheuristics. In this context, heuristic is an algorithm that locates ‘good
enough’ solutions to a problem without concern for whether the solution
can be proven to be correct or optimal [37]. Heuristic methods trade-off
concerns such as precision, quality, and accuracy in favor of computational
effort (space and time efficiency). The greedy search procedure that only
takes cost-improving steps is an example of heuristic method.
Like heuristics, metaheuristics may be considered a general algorithmic
framework that can be applied to different optimization problems with
relative few modifications to adapt them to a specific problem [25, 46]. The
difference is that metaheuristics are intended to extend the capabilities
of heuristics by combining one or more heuristic methods (referred to as
procedures) using a higher-level strategy (hence ‘meta’). A procedure in a
metaheuristic is considered black-box in that little (if any) prior knowledge
is known about it by the metaheuristic, and as such it may be replaced with
a different procedure. Procedures may be as simple as the manipulation of
a representation, or as complex as another complete metaheuristic. Some
examples of metaheuristics include iterated local search, tabu search, the
genetic algorithm, ant colony optimization, and simulated annealing.
Blum and Roli outline nine properties of metaheuristics [9], as follows:
Embargo.
And the question being put, that the House do agree to the said
resolution, and, upon the question thereupon, the yeas and nays
being demanded by one-fifth of the members present, and debate
arising, a motion was made by Mr. Macon, that the resolution do lie
on the table; and it was resolved in the affirmative.
A message from the Senate, by Mr. Otis, their Secretary:
Mr. Speaker: The Senate have, in confidence, directed me to
inform this honorable House that they have passed a bill, entitled
“An act laying an embargo on all ships and vessels in the ports and
harbors of the United States,” in which bill they desire the
concurrence of this House.
The said bill was received, read the third time, and committed to a
Committee of the Whole on the state of the Union on this day.
On motion of Mr. Crowninshield,
Resolved, That this House will immediately resolve itself into a
Committee of the Whole on the said bill.
The House accordingly resolved itself into the said committee;
and, after some time spent therein, the Speaker resumed the chair,
and Mr. Masters reported that the committee had had the said bill
under consideration, but not having time to go through the same,
had directed him to ask for leave to sit again.
Resolved, That this House will, to-morrow, again resolve itself into
a Committee of the Whole on the bill, entitled “An act laying an
embargo on all ships and vessels in the ports and harbors of the
United States.”
And then the House adjourned.
Ordered, That the Clerk of this House do carry the said bill, as
amended, to the Senate, and desire their concurrence.
The bill is as follows:
An Act laying an embargo on all ships and vessels in the ports and
harbors of the United States.
Be it enacted, by the Senate and House of Representatives of the
United States of America, in Congress assembled, That an embargo
be and hereby is laid on all ships and vessels in the ports and places
within the limits or jurisdiction of the United States, cleared or not
cleared, bound to any foreign port or place; and that no clearance
be furnished to any ship or vessel bound to such foreign port or
place, except vessels under the immediate direction of the President
of the United States; and that the President be authorized to give
such instructions to the officers of the revenue, and of the navy and
revenue cutters of the United States, as shall appear best adapted
for carrying the same into full effect: Provided, That nothing herein
contained shall be construed to prevent the departure of any foreign
ship or vessel, either in ballast, or with the goods, wares, and
merchandise, on board of such foreign ship or vessel, when notified
of this act.
Sec. 2. And be it further enacted, That during the continuance of
this act, no registered or sea-letter vessel, having on board goods,
wares, and merchandise, shall be allowed to depart from one port of
the United States to another within the same, unless the master,
owner, consignee, or factor of such vessel, shall first give bond with
one or more sureties to the collector of the district from which she is
bound to depart, in a sum of double the value of the vessel and
cargo; that the said goods, wares, and merchandise shall be
relanded in some port of the United States, dangers of the seas
excepted; which bond, and also a certificate from the collector
where the same may be relanded, shall, by the collectors
respectively, be transmitted to the Secretary of the Treasury. All
armed vessels possessing public commissions from any foreign
power are not to be considered as liable to the embargo laid by this
act.
Importation of Slaves.
Mr. Marion presented the petition of sundry merchants and others,
in Charleston, South Carolina, stating that many vessels had cleared
out from thence for the purpose of importing slaves, before the law
was passed by Congress prohibiting the importation of slaves, and
some had cleared out immediately after the passing of the law, and
had been detained by accidents beyond the time limited by law; and
praying that a law may be passed affording them relief.
The question being put on a motion made by Mr. Marion for a
reference of this petition to the Committee of Commerce and
Manufactures—
Mr. Masters said if there was any subject in favor of which a
petition should not be referred, it was the slave trade. These
petitioners knew when the prohibitory law would go into operation,
and they were not entitled to relief by the laws of God or man.
The motion for reference was negatived—yeas 37, nays 39.
Mr. Randolph then rose for the purpose of making a motion, and
giving information to the House which he had just received. This was
a duty which he owed, not only to himself, but to the enlightened
and independent freeholders who gave him a seat on this floor, and
to the country at large. Within a few days, information had been put
into his possession, of a nature and on a subject which he deemed it
proper for the constituted authority to inquire into. Had this
information come earlier into his possession, he should not till now
have delayed giving it publicity. He would first state certain facts,
and those facts would be the ground of his motion, on which he
should offer no argument. Mr. R. then read the following documents:
[TRANSLATION.]
New Orleans, January 20, 1796.
In the galley the Victoria, Bernardo Molina, Patron, there have
been sent to Don Vincent Folch nine thousand six hundred and forty
dollars; which sum, without making the least use of it, you will hold
at my disposal, to deliver it at the moment that an order may be
presented to you by the American General, Don James Wilkinson.
God preserve you many years.
The BARON DE CARONDELET.
To Señor Don Tomas Portell.
“On the 27th of the same month [October last] appeared in the
Richmond Enquirer a certificate given by myself to General Wilkinson
in New Orleans on the 16th of May preceding. Immediately on my
getting sight of this piece, which was the same or the next day, I
addressed a note to his Excellency General Wilkinson, [No. 3.] Of
this I did not keep a copy, and therefore dare not vouch that it is an
exact literal transcript of the original; but I will be bold to say that it
is nearly (or, to make use of the General’s own language,
substantially) the same.
“Between my repeated declarations to many of my friends and
acquaintances (I must say it with a blush) and this certificate, there
is a manifest contradiction. And between this same certificate and
the deductions to be drawn from my declaration before the
Richmond Court, there is an apparent inconsistency, which it is now
my task to clear up and reconcile.
“During General Wilkinson’s residence in New Orleans, last winter,
I used occasionally to visit him. A few days before he left New
Orleans, I waited upon him one morning, and after some
conversation on certain transactions that had taken place at a former
period in the Western country, and on the delicate situation in which
his conduct during the winter was likely to place him, he asked me if
I had any objection to give him a certificate that might help him to
silence that foul-mouthed Bradford, and refute the assertions of the
editor of the Western World. I replied without hesitation that I had
none, and would give him one with pleasure, provided he promised
me it should not be published. On this he assured me that the only
use he proposed to make of it was to lay it before the President,
with the view to prove the falsehood of the charges circulated
against him, vindicate his character, and secure the confidence of
the Executive. This, if not exactly, is substantially what the General
said. He then desired me to sit down and write the certificate. I
observed that I might not make it out entirely to his satisfaction; and
that, as he best knew the points he wished should be embraced in it,
he had better make it out himself, and I would copy it. To this he
agreed. Next morning, I waited on his Excellency, and he presented
me the certificate, which I copied, as it has been published, with a
few alterations. One—a very material one—is that, after these
words: ‘Do most solemnly declare that I have at no time carried or
delivered to Gen. James Wilkinson’—I erased the words, ‘either
directly or indirectly,’ and declared to the General I could not insert
those words. He did not insist, and contented himself with saying
that he wished me to insert them if my conscience would allow it,
but not otherwise. This is ingenuously exactly what passed between
the General and myself at that time.
“Now let me with the same frankness and ingenuousness, without
referring to any preceding or subsequent event, narrate the
transaction of 1796, alluded to in my certificate, and concerning
which I offered to give testimony in the federal circuit court in
Richmond. It is the same that is the subject of the affidavits of
Messrs. Derbigny and Mercier. That of the former gentleman is
correct as to substance, for I actually did receive from Captain Don
Thomas Portell, commandant of New Madrid, the sum of $9,640 for
General Wilkinson, towards the latter end of June or beginning of
July, 1796, which was packed up in the manner described by Mr.
Derbigny, and when I was stopped and my boat searched on the
Ohio by Lieutenant Steele, under the orders of General Anthony
Wayne, I had other sums on board, but this was the only one I had
received for General Wilkinson. On my arrival at Louisville,
determined not to expose myself a second time to military insult,
and fearful of being overtaken by Steele on his return, and of being
Welcome to our website – the ideal destination for book lovers and
knowledge seekers. With a mission to inspire endlessly, we offer a
vast collection of books, ranging from classic literary works to
specialized publications, self-development books, and children's
literature. Each book is a new journey of discovery, expanding
knowledge and enriching the soul of the reade
Our website is not just a platform for buying books, but a bridge
connecting readers to the timeless values of culture and wisdom. With
an elegant, user-friendly interface and an intelligent search system,
we are committed to providing a quick and convenient shopping
experience. Additionally, our special promotions and home delivery
services ensure that you save time and fully enjoy the joy of reading.
ebookultra.com