**Unformatted text preview: **Jason Brownlee Clever Algorithms
Nature-Inspired Programming Recipes ii Jason Brownlee, PhD
Jason Brownlee studied Applied Science at Swinburne University in
Melbourne, Australia, going on to complete a Masters in Information
Technology focusing on Niching Genetic Algorithms, and a PhD in the field
of Artificial Immune Systems. Jason has worked for a number of years as a
Consultant and Software Engineer for a range of Corporate and Government
organizations. When not writing books, Jason likes to compete in Machine
Learning competitions. Cover Image
© Copyright 2011 Jason Brownlee. All Reserved. Clever Algorithms: Nature-Inspired Programming Recipes
© Copyright 2011 Jason Brownlee. Some Rights Reserved.
Revision 2. 16 June 2012
ISBN: 978-1-4467-8506-5
This work is licensed under a Creative Commons
Attribution-Noncommercial-Share Alike 2.5 Australia License.
The full terms of the license are located online at
Webpage
Source code and additional resources can be downloaded from the books
companion website online at Contents
Foreword vii Preface I ix Background 1 1 Introduction
1.1 What is AI . . . . . . . . . .
1.2 Problem Domains . . . . . . .
1.3 Unconventional Optimization
1.4 Book Organization . . . . . .
1.5 How to Read this Book . . .
1.6 Further Reading . . . . . . .
1.7 Bibliography . . . . . . . . . II .
.
.
.
.
.
. .
.
.
.
.
.
. .
.
.
.
.
.
. .
.
.
.
.
.
. .
.
.
.
.
.
. .
.
.
.
.
.
. .
.
.
.
.
.
. .
.
.
.
.
.
. .
.
.
.
.
.
. .
.
.
.
.
.
. .
.
.
.
.
.
. .
.
.
.
.
.
. .
.
.
.
.
.
. .
.
.
.
.
.
. .
.
.
.
.
.
. Algorithms 3
3
10
13
17
20
21
22 27 2 Stochastic Algorithms
2.1 Overview . . . . . . . . . . . . . . . .
2.2 Random Search . . . . . . . . . . . . .
2.3 Adaptive Random Search . . . . . . .
2.4 Stochastic Hill Climbing . . . . . . . .
2.5 Iterated Local Search . . . . . . . . . .
2.6 Guided Local Search . . . . . . . . . .
2.7 Variable Neighborhood Search . . . . .
2.8 Greedy Randomized Adaptive Search .
2.9 Scatter Search . . . . . . . . . . . . .
2.10 Tabu Search . . . . . . . . . . . . . . .
2.11 Reactive Tabu Search . . . . . . . . .
iii .
.
.
.
.
.
.
.
.
.
. .
.
.
.
.
.
.
.
.
.
. .
.
.
.
.
.
.
.
.
.
. .
.
.
.
.
.
.
.
.
.
. .
.
.
.
.
.
.
.
.
.
. .
.
.
.
.
.
.
.
.
.
. .
.
.
.
.
.
.
.
.
.
. .
.
.
.
.
.
.
.
.
.
. .
.
.
.
.
.
.
.
.
.
. .
.
.
.
.
.
.
.
.
.
. 29
29
30
34
40
44
50
57
63
69
76
82 iv Contents 3 Evolutionary Algorithms
3.1 Overview . . . . . . . . . . . . . . . . . . .
3.2 Genetic Algorithm . . . . . . . . . . . . . .
3.3 Genetic Programming . . . . . . . . . . . .
3.4 Evolution Strategies . . . . . . . . . . . . .
3.5 Differential Evolution . . . . . . . . . . . .
3.6 Evolutionary Programming . . . . . . . . .
3.7 Grammatical Evolution . . . . . . . . . . .
3.8 Gene Expression Programming . . . . . . .
3.9 Learning Classifier System . . . . . . . . . .
3.10 Non-dominated Sorting Genetic Algorithm .
3.11 Strength Pareto Evolutionary Algorithm . . .
.
.
.
.
.
.
.
.
.
. .
.
.
.
.
.
.
.
.
.
. .
.
.
.
.
.
.
.
.
.
. .
.
.
.
.
.
.
.
.
.
. .
.
.
.
.
.
.
.
.
.
. .
.
.
.
.
.
.
.
.
.
. .
.
.
.
.
.
.
.
.
.
. 91
91
96
103
113
119
125
131
139
146
157
165 4 Physical Algorithms
4.1 Overview . . . . . . . .
4.2 Simulated Annealing . .
4.3 Extremal Optimization .
4.4 Harmony Search . . . .
4.5 Cultural Algorithm . . .
4.6 Memetic Algorithm . . . .
.
.
.
.
. .
.
.
.
.
. .
.
.
.
.
. .
.
.
.
.
. .
.
.
.
.
. .
.
.
.
.
. .
.
.
.
.
. .
.
.
.
.
. 173
173
175
181
188
194
200 5 Probabilistic Algorithms
5.1 Overview . . . . . . . . . . . . . . . . . . .
5.2 Population-Based Incremental Learning . .
5.3 Univariate Marginal Distribution Algorithm
5.4 Compact Genetic Algorithm . . . . . . . . .
5.5 Bayesian Optimization Algorithm . . . . . .
5.6 Cross-Entropy Method . . . . . . . . . . . . .
.
.
.
.
. .
.
.
.
.
. .
.
.
.
.
. .
.
.
.
.
. .
.
.
.
.
. .
.
.
.
.
. .
.
.
.
.
. 207
207
211
216
220
224
232 6 Swarm Algorithms
6.1 Overview . . . . . . . . . . . . . . . . . . .
6.2 Particle Swarm Optimization . . . . . . . .
6.3 Ant System . . . . . . . . . . . . . . . . . .
6.4 Ant Colony System . . . . . . . . . . . . . .
6.5 Bees Algorithm . . . . . . . . . . . . . . . .
6.6 Bacterial Foraging Optimization Algorithm .
.
.
.
.
. .
.
.
.
.
. .
.
.
.
.
. .
.
.
.
.
. .
.
.
.
.
. .
.
.
.
.
. .
.
.
.
.
. 237
237
241
247
254
261
266 7 Immune Algorithms
7.1 Overview . . . . . . . . . . . . . . . .
7.2 Clonal Selection Algorithm . . . . . .
7.3 Negative Selection Algorithm . . . . .
7.4 Artificial Immune Recognition System
7.5 Immune Network Algorithm . . . . . .
7.6 Dendritic Cell Algorithm . . . . . . . . .
.
.
.
.
. .
.
.
.
.
. .
.
.
.
.
. .
.
.
.
.
. .
.
.
.
.
. .
.
.
.
.
. .
.
.
.
.
. 275
275
280
288
295
304
312 .
.
.
.
.
. .
.
.
.
.
. .
.
.
.
.
. .
.
.
.
.
. .
.
.
.
.
. .
.
.
.
.
. .
.
.
.
.
. .
.
.
.
.
. .
.
.
.
.
. .
.
.
.
.
. .
.
.
.
.
. .
.
.
.
.
. .
.
.
.
.
. v
8 Neural Algorithms
8.1 Overview . . . . . . . . . . .
8.2 Perceptron . . . . . . . . . .
8.3 Back-propagation . . . . . . .
8.4 Hopfield Network . . . . . . .
8.5 Learning Vector Quantization
8.6 Self-Organizing Map . . . . . III .
.
.
.
.
. .
.
.
.
.
. .
.
.
.
.
. .
.
.
.
.
. .
.
.
.
.
. .
.
.
.
.
. .
.
.
.
.
. .
.
.
.
.
. .
.
.
.
.
. .
.
.
.
.
. .
.
.
.
.
. .
.
.
.
.
. .
.
.
.
.
. .
.
.
.
.
. Extensions
.
.
.
.
.
. .
.
.
.
.
. .
.
.
.
.
. .
.
.
.
.
. .
.
.
.
.
. .
.
.
.
.
. .
.
.
.
.
. .
.
.
.
.
. .
.
.
.
.
. .
.
.
.
.
. .
.
.
.
.
. .
.
.
.
.
. .
.
.
.
.
. .
.
.
.
.
. .
.
.
.
.
. .
.
.
.
.
. Appendix A Ruby: Quick-Start Guide
A.1 Overview . . . . . . . .
A.2 Language Basics . . . .
A.3 Ruby Idioms . . . . . .
A.4 Bibliography . . . . . . 319
319
323
328
336
342
348 357 9 Advanced Topics
9.1 Programming Paradigms . .
9.2 Devising New Algorithms .
9.3 Testing Algorithms . . . . .
9.4 Visualizing Algorithms . . .
9.5 Problem Solving Strategies
9.6 Benchmarking Algorithms . IV .
.
.
.
.
. 359
360
370
382
389
402
417 427
.
.
.
. .
.
.
. .
.
.
. .
.
.
. .
.
.
. .
.
.
. .
.
.
. .
.
.
. .
.
.
. .
.
.
. .
.
.
. .
.
.
. .
.
.
. .
.
.
. .
.
.
. .
.
.
. .
.
.
. .
.
.
. 429
429
429
433
435 Errata 437 Index 439 vi Contents Foreword
I am delighted to write this foreword. This book, a reference where
one can look up the details of most any algorithm to find a clear
unambiguous description, has long been needed and here it finally is.
A concise reference that has taken many hours to write but which has
the capacity to save vast amounts of time previously spent digging out
original papers.
I have known the author for several years and have had experience
of his amazing capacity for work and the sheer quality of his output, so
this book comes as no surprise to me. But I hope it will be a surprise
and delight to you, the reader for whom it has been written.
But useful as this book is, it is only a beginning. There are so many
algorithms that no one author could hope to cover them all. So if you
know of an algorithm that is not yet here, how about contributing it
using the same clear and lucid style? Professor Tim Hendtlass
Complex Intelligent Systems Laboratory
Faculty of Information and Communication Technologies
Swinburne University of Technology Melbourne, Australia
2010 vii viii Foreword Preface
About the book
The need for this project was born of frustration while working towards my PhD. I was investigating optimization algorithms and was
implementing a large number of them for a software platform called
the Optimization Algorithm Toolkit (OAT)1 . Each algorithm required
considerable effort to locate the relevant source material (from books,
papers, articles, and existing implementations), decipher and interpret
the technique, and finally attempt to piece together a working implementation.
Taking a broader perspective, I realized that the communication of
algorithmic techniques in the field of Artificial Intelligence was clearly a
difficult and outstanding open problem. Generally, algorithm descriptions are:
• Incomplete: many techniques are ambiguously described, partially
described, or not described at all.
• Inconsistent: a given technique may be described using a variety
of formal and semi-formal methods that vary across different techniques, limiting the transferability of background skills an audience
requires to read a technique (such as mathematics, pseudocode,
program code, and narratives). An inconsistent representation for
techniques means that the skills used to understand and internalize one technique may not be transferable to realizing different
techniques or even extensions of the same technique.
• Distributed : the description of data structures, operations, and
parameterization of a given technique may span a collection of
papers, articles, books, and source code published over a number
of years, the access to which may be restricted and difficult to
obtain.
1 OAT located at ix x Preface For the practitioner, a badly described algorithm may be simply
frustrating, where the gaps in available information are filled with
intuition and ‘best guess’. At the other end of the spectrum, a badly
described algorithm may be an example of bad science and the failure of
the scientific method, where the inability to understand and implement
a technique may prevent the replication of results, the application, or
the investigation and extension of a technique.
The software I produced provided a first step solution to this problem:
a set of working algorithms implemented in a (somewhat) consistent way
and downloaded from a single location (features likely provided by any
library of artificial intelligence techniques). The next logical step needed
to address this problem is to develop a methodology that anybody can
follow. The strategy to address the open problem of poor algorithm
communication is to present complete algorithm descriptions (rather
than just implementations) in a consistent manner, and in a centralized
location. This book is the outcome of developing such a strategy that not
only provides a methodology for standardized algorithm descriptions, but
provides a large corpus of complete and consistent algorithm descriptions
in a single centralized location.
The algorithms described in this work are practical, interesting, and
fun, and the goal of this project was to promote these features by making
algorithms from the field more accessible, usable, and understandable.
This project was developed over a number years through a lot of writing,
discussion, and revision. This book has been released under a permissive
license that encourages the reader to explore new and creative ways of
further communicating its message and content.
I hope that this project has succeeded in some small way and that you
too can enjoy applying, learning, and playing with Clever Algorithms. Jason Brownlee Melbourne, Australia
2011 Acknowledgments
This book could not have been completed without the commitment,
passion, and hard work from a large group of editors and supporters.
A special thanks to Steve Dower for his incredible attention to detail
in providing technical and copy edits for large portions of this book,
and for his enthusiasm for the subject area. Also, a special thanks to
Daniel Angus for the discussions around the genesis of the project, his
continued support with the idea of an ‘algorithms atlas’ and for his
attention to detail in providing technical and copy edits for key chapters.
In no particular order, thanks to: Juan Ojeda, Martin Goddard,
David Howden, Sean Luke, David Zappia, Jeremy Wazny, Andrew Murray, John Wise, Patrick Boehnke, Martin-Louis Bright, Leif Wickland,
Andrew Myers, Paul Chinnery, Donald Doherty, Brook Tamir, Zach
Scott, Diego Noble, Jason Davies, Mark Chenoweth, Markus Stokmaier,
Stefan Pauleweit, and Lai Yu-Hsuan.
Thanks to the hundreds of machine learning enthusiasts who voted
on potential covers and helped shape what this book became. You know
who you are!
Finally, I would like to thank my beautiful wife Ying Liu for her
unrelenting support and patience throughout the project. xi xii Acknowledgments Part I Background 1 Chapter 1 Introduction
Welcome to Clever Algorithms! This is a handbook of recipes for computational problem solving techniques from the fields of Computational
Intelligence, Biologically Inspired Computation, and Metaheuristics.
Clever Algorithms are interesting, practical, and fun to learn about and
implement. Research scientists may be interested in browsing algorithm
inspirations in search of an interesting system or process analogs to
investigate. Developers and software engineers may compare various
problem solving algorithms and technique-specific guidelines. Practitioners, students, and interested amateurs may implement state-of-the-art
algorithms to address business or scientific needs, or simply play with
the fascinating systems they represent.
This introductory chapter provides relevant background information
on Artificial Intelligence and Algorithms. The core of the book provides
a large corpus of algorithms presented in a complete and consistent
manner. The final chapter covers some advanced topics to consider
once a number of algorithms have been mastered. This book has been
designed as a reference text, where specific techniques are looked up, or
where the algorithms across whole fields of study can be browsed, rather
than being read cover-to-cover. This book is an algorithm handbook
and a technique guidebook, and I hope you find something useful. 1.1
1.1.1 What is AI
Artificial Intelligence The field of classical Artificial Intelligence (AI) coalesced in the 1950s
drawing on an understanding of the brain from neuroscience, the new
mathematics of information theory, control theory referred to as cybernetics, and the dawn of the digital computer. AI is a cross-disciplinary
3 4 Chapter 1. Introduction field of research that is generally concerned with developing and investigating systems that operate or act intelligently. It is considered
a discipline in the field of computer science given the strong focus on
computation.
Russell and Norvig provide a perspective that defines Artificial Intelligence in four categories: 1) systems that think like humans, 2) systems
that act like humans, 3) systems that think rationally, 4) systems that
act rationally [43]. In their definition, acting like a human suggests that
a system can do some specific things humans can do, this includes fields
such as the Turing test, natural language processing, automated reasoning, knowledge representation, machine learning, computer vision, and
robotics. Thinking like a human suggests systems that model the cognitive information processing properties of humans, for example a general
problem solver and systems that build internal models of their world.
Thinking rationally suggests laws of rationalism and structured thought,
such as syllogisms and formal logic. Finally, acting rationally suggests
systems that do rational things such as expected utility maximization
and rational agents.
Luger and Stubblefield suggest that AI is a sub-field of computer
science concerned with the automation of intelligence, and like other
sub-fields of computer science has both theoretical concerns (how and
why do the systems work? ) and application concerns (where and when
can the systems be used? ) [34]. They suggest a strong empirical focus to
research, because although there may be a strong desire for mathematical
analysis, the systems themselves defy analysis given their complexity.
The machines and software investigated in AI are not black boxes,
rather analysis proceeds by observing the systems interactions with their
environments, followed by an internal assessment of the system to relate
its structure back to its behavior.
Artificial Intelligence is therefore concerned with investigating mechanisms that underlie intelligence and intelligence behavior. The traditional approach toward designing and investigating AI (the so-called
‘good old fashioned’ AI) has been to employ a symbolic basis for these
mechanisms. A newer approach historically referred to as scruffy artificial intelligence or soft computing does not necessarily use a symbolic
basis, instead patterning these mechanisms after biological or natural
processes. This represents a modern paradigm shift in interest from symbolic knowledge representations, to inference strategies for adaptation
and learning, and has been referred to as neat versus scruffy approaches
to AI. The neat philosophy is concerned with formal symbolic models
of intelligence that can explain why they work, whereas the scruffy
philosophy is concerned with intelligent strategies that explain how they
work [44]. 1.1. What is AI 5 Neat AI
The traditional stream of AI concerns a top down perspective of problem
solving, generally involving symbolic representations and logic processes
that most importantly can explain why the systems work. The successes
of this prescriptive stream include a multitude of specialist approaches
such as rule-based expert systems, automatic theorem provers, and operations research techniques that underly modern planning and scheduling
software. Although traditional approaches have resulted in significant
success they have their limits, most notably scalability. Increases in
problem size result in an unmanageable increase in the complexity of
such problems meaning that although traditional techniques can guarantee an optimal, precise, or true solution, the computational execution
time or computing memory required can be intractable.
Scruffy AI
There have been a number of thrusts in the field of AI toward less crisp
techniques that are able to locate approximate, imprecise, or partiallytrue solutions to problems with a reasonable cost of resources. Such
approaches are typically descriptive rather than prescriptive, describing
a process for achieving a solution (how), but not explaining why they
work (like the neater approaches).
Scruffy AI approaches are defined as relatively simple procedures that
result in complex emergent and self-organizing behavior that can defy
traditional reductionist analyses, the effects of which can be exploited
for quickly locating approximate solutions to intractable problems. A
common characteristic of such techniques is the incorporation of randomness in their processes resulting in robust probabilistic and stochastic
decision making contrasted to the sometimes more fragile determinism
of the crisp approaches. Another important common attribute is the
adoption of an inductive rather than deductive approach to problem solving, generalizing solutions or decisions from sets of specific observations
made by the system. 1.1.2 Natural Computation An important perspective on scruffy Artificial Intelligence is the motivation and inspiration for the core information processing strategy of
a given technique. Computers can only do what they are instructed,
therefore a consideration is to distill information processing from other
fields of study, such as the physical world and biology. The study of
biologically motivated computation is called Biologically Inspired Computing [16], and is one of three related fields of Natural Computing
[22, 23, 39]. Natural Computing is an interdisciplinary field concerned 6 Chapter 1. Introduction with the relationship of computation and biology, which in addition to
Biologically Inspired Computing is also comprised of Computationally
Motivated Biology and Computing with Biology [36, 40].
Biologically Inspired Computation
Biologically Inspired Computation is computation inspired by biological
metaphor, also referred to as Biomimicry, and Biomemetics in other
engineering disciplines [6, 17]. The intent of this field is to devise mathematical and engineering tools to generate solutions to computation
problems. The field involves using procedures for finding solutions abstracted from the natural world for addressing computationally phrased
problems.
Computationally Motivated Biology
Computationally Motivated Biology involves investigating biology using
computers. The i...

View
Full Document

- Fall '08
- BECKMAN
- Artificial Intelligence, Algorithms, Data Structures, Evolutionary Algorithms