CleverAlgorithms.pdf - Jason Brownlee Clever Algorithms...

This preview shows page 1. Sign up to view the full content.

You've reached the end of this preview.

Unformatted text preview: Jason Brownlee Clever Algorithms Nature-Inspired Programming Recipes ii Jason Brownlee, PhD Jason Brownlee studied Applied Science at Swinburne University in Melbourne, Australia, going on to complete a Masters in Information Technology focusing on Niching Genetic Algorithms, and a PhD in the field of Artificial Immune Systems. Jason has worked for a number of years as a Consultant and Software Engineer for a range of Corporate and Government organizations. When not writing books, Jason likes to compete in Machine Learning competitions. Cover Image © Copyright 2011 Jason Brownlee. All Reserved. Clever Algorithms: Nature-Inspired Programming Recipes © Copyright 2011 Jason Brownlee. Some Rights Reserved. Revision 2. 16 June 2012 ISBN: 978-1-4467-8506-5 This work is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 2.5 Australia License. The full terms of the license are located online at Webpage Source code and additional resources can be downloaded from the books companion website online at Contents Foreword vii Preface I ix Background 1 1 Introduction 1.1 What is AI . . . . . . . . . . 1.2 Problem Domains . . . . . . . 1.3 Unconventional Optimization 1.4 Book Organization . . . . . . 1.5 How to Read this Book . . . 1.6 Further Reading . . . . . . . 1.7 Bibliography . . . . . . . . . II . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Algorithms 3 3 10 13 17 20 21 22 27 2 Stochastic Algorithms 2.1 Overview . . . . . . . . . . . . . . . . 2.2 Random Search . . . . . . . . . . . . . 2.3 Adaptive Random Search . . . . . . . 2.4 Stochastic Hill Climbing . . . . . . . . 2.5 Iterated Local Search . . . . . . . . . . 2.6 Guided Local Search . . . . . . . . . . 2.7 Variable Neighborhood Search . . . . . 2.8 Greedy Randomized Adaptive Search . 2.9 Scatter Search . . . . . . . . . . . . . 2.10 Tabu Search . . . . . . . . . . . . . . . 2.11 Reactive Tabu Search . . . . . . . . . iii . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29 29 30 34 40 44 50 57 63 69 76 82 iv Contents 3 Evolutionary Algorithms 3.1 Overview . . . . . . . . . . . . . . . . . . . 3.2 Genetic Algorithm . . . . . . . . . . . . . . 3.3 Genetic Programming . . . . . . . . . . . . 3.4 Evolution Strategies . . . . . . . . . . . . . 3.5 Differential Evolution . . . . . . . . . . . . 3.6 Evolutionary Programming . . . . . . . . . 3.7 Grammatical Evolution . . . . . . . . . . . 3.8 Gene Expression Programming . . . . . . . 3.9 Learning Classifier System . . . . . . . . . . 3.10 Non-dominated Sorting Genetic Algorithm . 3.11 Strength Pareto Evolutionary Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91 91 96 103 113 119 125 131 139 146 157 165 4 Physical Algorithms 4.1 Overview . . . . . . . . 4.2 Simulated Annealing . . 4.3 Extremal Optimization . 4.4 Harmony Search . . . . 4.5 Cultural Algorithm . . . 4.6 Memetic Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 173 173 175 181 188 194 200 5 Probabilistic Algorithms 5.1 Overview . . . . . . . . . . . . . . . . . . . 5.2 Population-Based Incremental Learning . . 5.3 Univariate Marginal Distribution Algorithm 5.4 Compact Genetic Algorithm . . . . . . . . . 5.5 Bayesian Optimization Algorithm . . . . . . 5.6 Cross-Entropy Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 207 207 211 216 220 224 232 6 Swarm Algorithms 6.1 Overview . . . . . . . . . . . . . . . . . . . 6.2 Particle Swarm Optimization . . . . . . . . 6.3 Ant System . . . . . . . . . . . . . . . . . . 6.4 Ant Colony System . . . . . . . . . . . . . . 6.5 Bees Algorithm . . . . . . . . . . . . . . . . 6.6 Bacterial Foraging Optimization Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 237 237 241 247 254 261 266 7 Immune Algorithms 7.1 Overview . . . . . . . . . . . . . . . . 7.2 Clonal Selection Algorithm . . . . . . 7.3 Negative Selection Algorithm . . . . . 7.4 Artificial Immune Recognition System 7.5 Immune Network Algorithm . . . . . . 7.6 Dendritic Cell Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 275 275 280 288 295 304 312 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . v 8 Neural Algorithms 8.1 Overview . . . . . . . . . . . 8.2 Perceptron . . . . . . . . . . 8.3 Back-propagation . . . . . . . 8.4 Hopfield Network . . . . . . . 8.5 Learning Vector Quantization 8.6 Self-Organizing Map . . . . . III . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Extensions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Appendix A Ruby: Quick-Start Guide A.1 Overview . . . . . . . . A.2 Language Basics . . . . A.3 Ruby Idioms . . . . . . A.4 Bibliography . . . . . . 319 319 323 328 336 342 348 357 9 Advanced Topics 9.1 Programming Paradigms . . 9.2 Devising New Algorithms . 9.3 Testing Algorithms . . . . . 9.4 Visualizing Algorithms . . . 9.5 Problem Solving Strategies 9.6 Benchmarking Algorithms . IV . . . . . . 359 360 370 382 389 402 417 427 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 429 429 429 433 435 Errata 437 Index 439 vi Contents Foreword I am delighted to write this foreword. This book, a reference where one can look up the details of most any algorithm to find a clear unambiguous description, has long been needed and here it finally is. A concise reference that has taken many hours to write but which has the capacity to save vast amounts of time previously spent digging out original papers. I have known the author for several years and have had experience of his amazing capacity for work and the sheer quality of his output, so this book comes as no surprise to me. But I hope it will be a surprise and delight to you, the reader for whom it has been written. But useful as this book is, it is only a beginning. There are so many algorithms that no one author could hope to cover them all. So if you know of an algorithm that is not yet here, how about contributing it using the same clear and lucid style? Professor Tim Hendtlass Complex Intelligent Systems Laboratory Faculty of Information and Communication Technologies Swinburne University of Technology Melbourne, Australia 2010 vii viii Foreword Preface About the book The need for this project was born of frustration while working towards my PhD. I was investigating optimization algorithms and was implementing a large number of them for a software platform called the Optimization Algorithm Toolkit (OAT)1 . Each algorithm required considerable effort to locate the relevant source material (from books, papers, articles, and existing implementations), decipher and interpret the technique, and finally attempt to piece together a working implementation. Taking a broader perspective, I realized that the communication of algorithmic techniques in the field of Artificial Intelligence was clearly a difficult and outstanding open problem. Generally, algorithm descriptions are: • Incomplete: many techniques are ambiguously described, partially described, or not described at all. • Inconsistent: a given technique may be described using a variety of formal and semi-formal methods that vary across different techniques, limiting the transferability of background skills an audience requires to read a technique (such as mathematics, pseudocode, program code, and narratives). An inconsistent representation for techniques means that the skills used to understand and internalize one technique may not be transferable to realizing different techniques or even extensions of the same technique. • Distributed : the description of data structures, operations, and parameterization of a given technique may span a collection of papers, articles, books, and source code published over a number of years, the access to which may be restricted and difficult to obtain. 1 OAT located at ix x Preface For the practitioner, a badly described algorithm may be simply frustrating, where the gaps in available information are filled with intuition and ‘best guess’. At the other end of the spectrum, a badly described algorithm may be an example of bad science and the failure of the scientific method, where the inability to understand and implement a technique may prevent the replication of results, the application, or the investigation and extension of a technique. The software I produced provided a first step solution to this problem: a set of working algorithms implemented in a (somewhat) consistent way and downloaded from a single location (features likely provided by any library of artificial intelligence techniques). The next logical step needed to address this problem is to develop a methodology that anybody can follow. The strategy to address the open problem of poor algorithm communication is to present complete algorithm descriptions (rather than just implementations) in a consistent manner, and in a centralized location. This book is the outcome of developing such a strategy that not only provides a methodology for standardized algorithm descriptions, but provides a large corpus of complete and consistent algorithm descriptions in a single centralized location. The algorithms described in this work are practical, interesting, and fun, and the goal of this project was to promote these features by making algorithms from the field more accessible, usable, and understandable. This project was developed over a number years through a lot of writing, discussion, and revision. This book has been released under a permissive license that encourages the reader to explore new and creative ways of further communicating its message and content. I hope that this project has succeeded in some small way and that you too can enjoy applying, learning, and playing with Clever Algorithms. Jason Brownlee Melbourne, Australia 2011 Acknowledgments This book could not have been completed without the commitment, passion, and hard work from a large group of editors and supporters. A special thanks to Steve Dower for his incredible attention to detail in providing technical and copy edits for large portions of this book, and for his enthusiasm for the subject area. Also, a special thanks to Daniel Angus for the discussions around the genesis of the project, his continued support with the idea of an ‘algorithms atlas’ and for his attention to detail in providing technical and copy edits for key chapters. In no particular order, thanks to: Juan Ojeda, Martin Goddard, David Howden, Sean Luke, David Zappia, Jeremy Wazny, Andrew Murray, John Wise, Patrick Boehnke, Martin-Louis Bright, Leif Wickland, Andrew Myers, Paul Chinnery, Donald Doherty, Brook Tamir, Zach Scott, Diego Noble, Jason Davies, Mark Chenoweth, Markus Stokmaier, Stefan Pauleweit, and Lai Yu-Hsuan. Thanks to the hundreds of machine learning enthusiasts who voted on potential covers and helped shape what this book became. You know who you are! Finally, I would like to thank my beautiful wife Ying Liu for her unrelenting support and patience throughout the project. xi xii Acknowledgments Part I Background 1 Chapter 1 Introduction Welcome to Clever Algorithms! This is a handbook of recipes for computational problem solving techniques from the fields of Computational Intelligence, Biologically Inspired Computation, and Metaheuristics. Clever Algorithms are interesting, practical, and fun to learn about and implement. Research scientists may be interested in browsing algorithm inspirations in search of an interesting system or process analogs to investigate. Developers and software engineers may compare various problem solving algorithms and technique-specific guidelines. Practitioners, students, and interested amateurs may implement state-of-the-art algorithms to address business or scientific needs, or simply play with the fascinating systems they represent. This introductory chapter provides relevant background information on Artificial Intelligence and Algorithms. The core of the book provides a large corpus of algorithms presented in a complete and consistent manner. The final chapter covers some advanced topics to consider once a number of algorithms have been mastered. This book has been designed as a reference text, where specific techniques are looked up, or where the algorithms across whole fields of study can be browsed, rather than being read cover-to-cover. This book is an algorithm handbook and a technique guidebook, and I hope you find something useful. 1.1 1.1.1 What is AI Artificial Intelligence The field of classical Artificial Intelligence (AI) coalesced in the 1950s drawing on an understanding of the brain from neuroscience, the new mathematics of information theory, control theory referred to as cybernetics, and the dawn of the digital computer. AI is a cross-disciplinary 3 4 Chapter 1. Introduction field of research that is generally concerned with developing and investigating systems that operate or act intelligently. It is considered a discipline in the field of computer science given the strong focus on computation. Russell and Norvig provide a perspective that defines Artificial Intelligence in four categories: 1) systems that think like humans, 2) systems that act like humans, 3) systems that think rationally, 4) systems that act rationally [43]. In their definition, acting like a human suggests that a system can do some specific things humans can do, this includes fields such as the Turing test, natural language processing, automated reasoning, knowledge representation, machine learning, computer vision, and robotics. Thinking like a human suggests systems that model the cognitive information processing properties of humans, for example a general problem solver and systems that build internal models of their world. Thinking rationally suggests laws of rationalism and structured thought, such as syllogisms and formal logic. Finally, acting rationally suggests systems that do rational things such as expected utility maximization and rational agents. Luger and Stubblefield suggest that AI is a sub-field of computer science concerned with the automation of intelligence, and like other sub-fields of computer science has both theoretical concerns (how and why do the systems work? ) and application concerns (where and when can the systems be used? ) [34]. They suggest a strong empirical focus to research, because although there may be a strong desire for mathematical analysis, the systems themselves defy analysis given their complexity. The machines and software investigated in AI are not black boxes, rather analysis proceeds by observing the systems interactions with their environments, followed by an internal assessment of the system to relate its structure back to its behavior. Artificial Intelligence is therefore concerned with investigating mechanisms that underlie intelligence and intelligence behavior. The traditional approach toward designing and investigating AI (the so-called ‘good old fashioned’ AI) has been to employ a symbolic basis for these mechanisms. A newer approach historically referred to as scruffy artificial intelligence or soft computing does not necessarily use a symbolic basis, instead patterning these mechanisms after biological or natural processes. This represents a modern paradigm shift in interest from symbolic knowledge representations, to inference strategies for adaptation and learning, and has been referred to as neat versus scruffy approaches to AI. The neat philosophy is concerned with formal symbolic models of intelligence that can explain why they work, whereas the scruffy philosophy is concerned with intelligent strategies that explain how they work [44]. 1.1. What is AI 5 Neat AI The traditional stream of AI concerns a top down perspective of problem solving, generally involving symbolic representations and logic processes that most importantly can explain why the systems work. The successes of this prescriptive stream include a multitude of specialist approaches such as rule-based expert systems, automatic theorem provers, and operations research techniques that underly modern planning and scheduling software. Although traditional approaches have resulted in significant success they have their limits, most notably scalability. Increases in problem size result in an unmanageable increase in the complexity of such problems meaning that although traditional techniques can guarantee an optimal, precise, or true solution, the computational execution time or computing memory required can be intractable. Scruffy AI There have been a number of thrusts in the field of AI toward less crisp techniques that are able to locate approximate, imprecise, or partiallytrue solutions to problems with a reasonable cost of resources. Such approaches are typically descriptive rather than prescriptive, describing a process for achieving a solution (how), but not explaining why they work (like the neater approaches). Scruffy AI approaches are defined as relatively simple procedures that result in complex emergent and self-organizing behavior that can defy traditional reductionist analyses, the effects of which can be exploited for quickly locating approximate solutions to intractable problems. A common characteristic of such techniques is the incorporation of randomness in their processes resulting in robust probabilistic and stochastic decision making contrasted to the sometimes more fragile determinism of the crisp approaches. Another important common attribute is the adoption of an inductive rather than deductive approach to problem solving, generalizing solutions or decisions from sets of specific observations made by the system. 1.1.2 Natural Computation An important perspective on scruffy Artificial Intelligence is the motivation and inspiration for the core information processing strategy of a given technique. Computers can only do what they are instructed, therefore a consideration is to distill information processing from other fields of study, such as the physical world and biology. The study of biologically motivated computation is called Biologically Inspired Computing [16], and is one of three related fields of Natural Computing [22, 23, 39]. Natural Computing is an interdisciplinary field concerned 6 Chapter 1. Introduction with the relationship of computation and biology, which in addition to Biologically Inspired Computing is also comprised of Computationally Motivated Biology and Computing with Biology [36, 40]. Biologically Inspired Computation Biologically Inspired Computation is computation inspired by biological metaphor, also referred to as Biomimicry, and Biomemetics in other engineering disciplines [6, 17]. The intent of this field is to devise mathematical and engineering tools to generate solutions to computation problems. The field involves using procedures for finding solutions abstracted from the natural world for addressing computationally phrased problems. Computationally Motivated Biology Computationally Motivated Biology involves investigating biology using computers. The i...
View Full Document

{[ snackBarMessage ]}

What students are saying

  • Left Quote Icon

    As a current student on this bumpy collegiate pathway, I stumbled upon Course Hero, where I can find study resources for nearly all my courses, get online help from tutors 24/7, and even share my old projects, papers, and lecture notes with other students.

    Student Picture

    Kiran Temple University Fox School of Business ‘17, Course Hero Intern

  • Left Quote Icon

    I cannot even describe how much Course Hero helped me this summer. It’s truly become something I can always rely on and help me. In the end, I was not only able to survive summer classes, but I was able to thrive thanks to Course Hero.

    Student Picture

    Dana University of Pennsylvania ‘17, Course Hero Intern

  • Left Quote Icon

    The ability to access any university’s resources through Course Hero proved invaluable in my case. I was behind on Tulane coursework and actually used UCLA’s materials to help me move forward and get everything together on time.

    Student Picture

    Jill Tulane University ‘16, Course Hero Intern