SIMULATION A OF INSECT INTELLIGENCE: ANIMATS WITH EMPIRICALLY BASED LEARNING AND BEHAVIOR
By ALFRED WESLEY BUHL
A THESIS PRESENTED TO THE GRADUATE SCHOOL OF THE UNIVERSITY OF FLORIDA IN PARTIAL FULLFILLMENT OF THE REQUIREMENTS FOR THE DEGREE OF MASTER OF SCIENCE UNIVERSITY OF FLORIDA 1994
To the Invertebrates
ACKNOWLEDGMENTS
I would first like to thank Dr. Paul Fishwick for his patience and support during this rather long-running project. My thanks also go to my cohorts, the other CIS graduate students, for their camraderie, moral support, diversions, and foreign language tutoring. Special thanks to Hanns-Oskar Porr, the Graphite, for his help with the graphical aspects of this project. Thanks to my (mostly) ex-Lakeland support group--you know who you are, and the pizza's on me next time. My thanks also go to my family for their support, especially my father and a certain special grandmother of mine, Ruth T. Vaughn, whose unwavering confidence in my abilities has kept me going strong through this long haul. Thanks also to my supervisor at Animal Science, Dr. Kunkle, for allowing me to reduce my work hours to get this project finished. I would also like to thank Dr. Daystrom, Dr. Forbin, and Dr. Chandrasegarampillai for inspiration early on in my career.
iii
TABLE OF CONTENTS
ACKNOWLEDGMENTS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . iii ABSTRACT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . CHAPTERS 1 INTRODUCTION . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Rationale . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Contribution to the Problem . . . . . . . . . . . . . . . . . . . . . . . . Overview of the Xants Project . . . . . . . . . . . . . . . . . . . . . . 2 1 1 4 6 vi
AN OVERVIEW OF ANIMAL INTELLIGENCE . . . . . . . . 10 The Ethological Perspective . . . . . . . . . . . . The Perspective of Comparative Psychology Habituation in Depth . . . . . . . . . . . . . . . . . Classical Conditioning in Depth . . . . . . . . . Biological Constraints on Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 13 18 20 23
3
AN OVERVIEW OF ANT CAPABILITIES AND BEHAVIOR . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 The Ant Colony Life Cycle . . . . The Ant's Navigational System Pheromones and Behavior . . . . The Alarm Response . . . . . . . . Foraging and Trail-Laying . . . . Other Behaviors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 26 27 27 28 29
4
CLASSIFIER SYSTEMS AND THEIR IMPLENTATION IN XANTS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32 Definitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32 The Xants Implementation of Classifier Systems . . . . . . . . 35 iv
Message Types . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36 An Xant's Brain: Theory of Operation . . . . . . . . . . . . . . . . 43 5 DESCRIPTION OF THE SIMULATOR . . . . . . . . . . . . . . . 58 The Xants World . . . . . . . . . . . . . . . . . . . . . . . . Control Files for the Xants Simulator . . . . . . . . . Syntax of the Xants.brain File . . . . . . . . . . . . . . The Xants Interface--Controls and Options . . . . . The Trace Command--Observing an Xant's Brain Performing Experiments on the Xants . . . . . . . . Generating 3D Animations . . . . . . . . . . . . . . . . . 6 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58 59 61 64 66 68 69
VALIDATION OF THE SIMULATION . . . . . . . . . . . . . . . 73 Validating the Simulation of Nonassociative Learning . . . . 73 Validating the Simulation of Classical Conditioning . . . . . . 79
7
CONCLUSIONS AND RECOMMENDATIONS FOR FURTHER RESEARCH . . . . . . . . . . . . . . . . . . . . . . . . . . . 81 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81 Recommendations for Further Research . . . . . . . . . . . . . . . 81
REFERENCE LIST . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89 BIOGRAPHICAL SKETCH . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92
v
Abstract of Thesis Presented to the Graduate School of the University of Florida in Partial Fulfillment of the Requirements for the Degree of Master of Science A SIMULATION OF INSECT INTELLIGENCE: ANIMATS WITH EMPIRICALLY BASED LEARNING AND BEHAVIOR By Alfred Wesley Buhl April 1994 Chairman: Paul A. Fishwick Major Department: Computer and Information Sciences This paper presents a computer simulation of an ant colony. The simulated ants, "xants", have a set of instincts, based on those of real ants, which they can then add to and modify based on their experiences. The learning abilities of these xants are based on those of real animals, both invertebrate and vertebrate, as described by current research in animal learning. We then compare the results of experiments on the xants to experiments on real animals. (We could not find similar experiments involving real ants, their small size making them difficult research subjects.) The mental abilities of the xants are modeled via a classifier system, with extensions to provide more accurate modeling. Specifically, the
vi
strengths both of the stimulus and of the reward are intrinsically represented in the evaluation and modification of rules, and heuristics based on animal learning data are used to provide rule creation and modification. The simulation was implemented in C++ and works with the X Windows system. The program can additionally generate animation scripts that can then be rendered in 3-D via the ANIMATE program.
vii
CHAPTER 1 INTRODUCTION
Rationale
For the past four decades, researchers in Artificial Intelligence (AI) have tried to create computer systems that exhibited some degree of intelligence. Most of this work, unsurprisingly, has had human-level performance as its goal. There have been successes in many areas-programs to play games such as Chess and Backgammon do so at master levels, and expert systems are capable of diagnosing diseases better than the average physician--but all these successes have been in small, welldefined domains. When confronted with situations outside their expertise, or even novel situations in their normal domain, these systems exhibit a brittleness that comes from both their specialization and their inflexibility. So, despite these accomplishments, the goal of machines with true humanlevel intelligence has not been achieved.1 This failure is not so much a failure of AI as it is a failure of excessive optimism on the part of some AI researchers. Four decades is
Broad, general, human-level intelligence has not been the goal of every AI researcher, but it has been the goal of many. 1
1
2 simply too short a time to reach such a lofty goal. It is as if cavemen, anxious to leave their caves, had started out trying to build skyscrapers instead of the thatched huts that were their logical first step. In order to truly model intelligence, we should start small and work our way up. Instead of trying to model human-level intelligence, why not try to model animal-level intelligence first? Since the goal is simpler, the project is less formidable. Animal behavior and learning have been well studied, More
giving a vast array of empirical data to validate our models.
importantly, animals excel at many tasks that computer programs still do not do very well--navigation, visual processing, and coping with novel situations. By trying to model a broad swath of the animal's behavior, instead of a narrow slice of human-level expertise, we can avoid the brittleness problems associated with small domains. By modeling learning or evolution, we build systems that can adapt to novel situations, instead of repeatedly failing whenever a novel situation is encountered. This approach offers a kind of "Great Chain of Being" approach to AI--start out modeling very unintelligent animals, then build on your past success, creating more and more intelligent behavior as time goes by. This type of work holds the most immediate benefit for people working on autonomous robots--for example, robots exploring a distant
3 planet, far from human guidance. It is also an important step toward creating more intelligent systems in the future. In Meyer and Wilson (1991), autonomous, adapting systems, whether simulated or actually constructed, are referred to as "animats". This source also gives many examples of work in this area, also referred to as the "simulation of adaptive behavior". There is a distinction between the Animat approach and what is called "artificial life" (see Langton (1989) and Langton et al. (1992) for representative research in the latter area). In the animat approach, work is focused on the simulation of behavior and adaptation to the environment via learning. Artificial life is significantly broader and shows more work in the area of modeling the evolution of life itself. One could argue that modeling the intelligence of lower animals may not necessarily provide techniques applicable to modeling human minds-human cognition could be so divorced from its animal heritage as to make animal intelligence irrelevant. As it turns out, we are not as far removed from the animals as we might imagine, or even hope. All of the types of learning exhibited in animals are also present in humans. We have more advanced capabilities, to be sure, and sometimes these capabilities can mask the effect of that simpler framework, but it is still present.
4
Contribution to the Problem
This work most properly fits in the Animat category. However, where this work differs from other research in the area (see Meyer and Wilson, 1991, and Maes, 1990) is the emphasis on empirical validation of the simulation. We wanted these animats to be capable of the same instinctual and learning capabilities of real animals, to show that we are achieving at least animal-level intelligence. We also wanted to be able to compare results from the simulation to results from real animal experiments to validate my model. Empirical validation of this kind of work is rare. For example Sutton (1990) presents a reinforcement learning architecture for animats, but never compares its performance to any living organism. Riolo (1990) created a classifier-based system that was capable of lookahead planning and latent learning, but again did no empirical validation. Beer et al. (1991) developed an cockroach simulator by manually designing a neural net, but did not formally compare the results to those of real cockroaches or other organisms. Admittedly, the way an animal does something may not be the only way to do something (it may not even be the best way), but it is important to understand what animals are doing and how they are doing it, especially
5 since they seem to be doing many things significantly better that AI systems at this time. As it has been pointed out, a submarine does not swim, but both submarines and fish must be streamlined, and the understanding of the common requirements of submarines and fish was necessary for the development of good submarines. This simulation is the only one of its kind that models the "nonassociative" types of learning: habituation, dishabituation, and sensitization. These are the simplest kinds of learning, but they have been largely ignored by philosophers, psychologists, and AI researchers, perhaps because they are so simple. In addition, this simulation models classical conditioning on top of this framework, using the output of the habituation, dishabituation, and sensitization routines to determine the familiarity of encountered stimuli and, therefore, whether new rules may need to be generated to cover a novel situation. This is the most important aspect of the simulation--the combining of a good model of nonassociative learning with a good model of classical conditioning in an Animat/Artificial Life framework that allows experimentation and evaluation. Holyoak et al. (1989) presented a classifier-based model of associative learning with empirical validation. However, this simulation ignored nonassociative learning and constrains association of stimuli to events happening simulataneously or within one iteration of the simulator-learning for longer intervals is not supported.
6
Figure 1-1. Solenopsis invicta. From Hlldobler and Wilson (1990). While working on this project, we have studied material in Ethology (the study of animal behavior in the wild), Comparative Psychology (the study of animal behavior in controlled conditions), Myrmecology (the study of ants), Cognitive Psychology, Psychophysiology (both human and animal), classifier systems, and genetic algorithms, in addition to the more closely related work in artificial life and the simulation of adaptive behavior. We have tried to use all the information we gathered from these various areas to reach our goal.
Overview of the Xants Project
For this project, we decided to model ants, specifically the Red Imported Fire Ant, Solenopsis invicta (Figure 1-1). we chose this animal because, as a major pest in the southern United States, it is well studied. It is also readily available for local observation. In addition, the relatively
7 simple nervous system of an ant (less than a million neurons total) gave us hope that we could capture most of the mental abilities of these animals. We call the resulting simulation "Xants", since it runs under the X Windows system. In order for the xants to exhibit behavior, we had to first create a simulated world in which they could live, and a way of displaying and interacting with this world. The xants live in a simulated, unbounded world of blocks, food sources, and a central nest (Figure 1-2). The xants can detect collisions, smell odors, and have a homing sense that tells them how to find the nest. They are capable of various kinds of learning and can be trained by the user of the xants program in a variety of ways. The 2D user interface can zoom in and out and lock onto a particular ant or object, allowing the view to follow that object through the world. You can also pick up and move objects with the mouse and, via options available from pull-down menus, you can slow down the simulation, trace the internal states of the xant's brain, and expose the whole colony or an individual to a variety of stimuli to perform learning experiments. You can also record the world in the form of a 3D graphics script language that can then be fed into the ANIMATE program developed at the University of Florida to render 3D images of the xants in action. To implement the brains of these creatures, we expanded on the work described by Holyoak et al. (1989). Their system, derived from J. H.
8
Figure 1-2. The Xants program. Holland's classifier systems (Holland et al. 1987), used fixed-length binary message strings, representing the current state of the environment, to be matched by a set of rules, which, when activated, post other messages
9 which then affect the environment. When presented with unusual stimuli, rules are created to cover the novel situations. These rules compete with the other rules present in the system--successful rules being strengthened, unsuccessful ones being weakened. This simulation expands on this technique, adding nonassociative learning and strength parameters to messages to provide more realistic behavior. Chapter 2 provides an overview of animal intelligence, viewed from a variety of perspectives. Chapter 3 is a brief overview of ant capabilities and behavior. Chapter 4 discusses Classifier Systems, and their implementation in the Xants simulator. Chapter 5 is a description of the simulator's interface and capabilities. Chapter 6 contains results from experiments performed with the simulator, and compares those results to the behavior of real animals. Chapter 7 explores conclusions reached and recommendations for further research.
CHAPTER 2 AN OVERVIEW OF ANIMAL INTELLIGENCE
Animal Intelligence has been studied in two quite different ways-observation of animal behavior under natural conditions (Ethology) and studying animals under controlled conditions (Comparative Psychology). The two approaches yield different insights into the brains of animals, but share a substantial common ground.
The Ethological Perspective
Work in ethology has led to numerous insights on the differences in sensory capabilities between humans and animals (Gould 1982). The fact that bees can see ultraviolet light, for example, was discovered by the pioneering ethologist Karl Von Frisch. His additional discovery that most flowers appear as "bullseyes" (dark centers with light surrounds) in the UV part of the spectrum explained how flowers attract bees, and how bees orient to flowers. Ethological studies have shown that much knowledge is "wired" into organisms at birth, and it has explored and quantified the type and extent of that knowledge. Animals have a variety of simple feature detecting systems at birth. These systems are called "innate releasing mechanisms" (IRMs). For 10
11 example, Otto Tinbergen discovered that baby chicks have an innate fear of hawk-shaped shadows falling on them from above. A simple wooden Tshape moving with the cross-bar towards the front can trigger a "run for cover" response. Interestingly enough, if the T-shape is moving so that cross-bar is towards the back (a "swan" configuration) the object is ignored. These kinds of simple detectors abound in nature. Sometimes, these detectors are too simple. The greylag goose, for instance, will roll back an egg that has fallen out of its nest. The detector responsible is not very discriminating. Anything remotely egg-shaped--a beer can, for instance--will trigger the response. In fact, a volley ball will even trigger the response, and will be preferred over real eggs (a so-called "supranormal" stimulus). Once the volley ball is in the nest, however, the goose will remove it, along with other inappropriate objects, since the detector for inappropriate objects in the nest is more discriminating than the egg-out-of-nest detector. It might seem curious that geese are taken in by such un-egg-like objects, but it must be remembered that in the wild, geese would never encounter volley balls or beer cans, so there would be little adaptive advantage for evolving better discriminators. Also, since the discriminator circuitry is specified by the animal's DNA, it would take a lot more genetic information to code the exact appearance of an egg archetype, and a lot simpler to specify the circuitry for detecting something whitish and round.
12 Nature seems to work with the simplest hardware/software required to get the job done. Animals have a variety of pre-wired, complex behaviors that can be triggered by an IRM or other cause. The simplest category of these is called a "fixed action program" (FAP). This is a sequence of movements executed without any feedback. For example, like most animals, mice have a prewired grooming behavior, rubbing their arms over their faces. If you amputate a mouse's front legs, it will still attempt the behavior. Another, more familiar example: a de-clawed cat will still attempt to sharpen its claws on furniture. Other, more complex behaviors require input from the environment. Often, there is a "critical period" when certain types of information are acquired, never to be rewritten. This is especially true during maturation, where many pre-wired behaviors are fine-tuned. The phenomenon of imprinting is an example. Baby ducklings imprint on their mothers, following them wherever they go. In another example of less-than-optimal detectors, ducklings followed the ethologist Konrad Lorenz in the same way due to his frequent presence at their parent-less nest and his high-quality duck call imitations presented during their critical period. Many behaviors are triggered not only by exterior stimuli, but by the animal's internal state. An animal is said to have "motivation" or a "drive"
to perform certain activities or classes of activities. Animals tend to seek
13 food when they are hungry, for instance. However, some types of behavior happen without motivation, during times of extreme deprivation or conflict. These are called "vacuum activities"--the grooming response, for example, is performed by many species in these situations. Motivations may appear only at certain times of the year or points in the life cycle. For instance, some species of geese remove cracked egg shells from the nest, but this activity is suspended shortly before the eggs hatch, presumably to keep the geese from removing hatching offspring. We have come away from these ethological readings with a profound amazement for the simplicity behind most seemingly intelligent animal behavior, and a reinforced skepticism in regard to claims for any given animal's intelligence.
The Perspective of Comparative Psychology
Comparative Psychology hopes to gain insights into human psychology via the study of animals. The emphasis is not so much on the study of existing behaviors, or situations occurring in natural settings, as it is on what can be learned by the animals, and how their learning capabilities may provide insights to human intelligence. A wide variety of animals have been studied, both vertebrates and invertebrates, although much more research has been carried out on the former. However, invertebrates exhibit many of the types of learning seen
14 in higher animals and produce similar changes in response (Bitterman 1988). It may seem surprising that the learning abilities of invertebrates are similar to those of vertebrates, given that the two groups separated 500 million years ago, and since the common ancestor of these groups "probably did not have much brain at all" (Bitterman 1988, p. 252). It is therefore likely that these similarities are due to their basic adaptive usefulness, rather than their common ancestry. There are several different types of learning. The categorization in Thorpe (1963) is most often cited in the literature (with qualifications); We will use his descriptions, although we will use more recognizable terms for some of his categories. The first category is nonassociative learning, which includes three phenomena: habituation, dishabituation, and sensitization. Thorpe defined habituation as "the relatively permanent waning of a response as a result of repeated stimulation which is not followed by any kind of reinforcement. It is specific to the stimulus and relatively enduring" (Thorpe 1963, p. 61). Habituation can occur even though the time between stimuli, the "interstimulus interval" (ISI) is 24 hours or even longer. A human example: reportedly there is a cannon in the town square of Edinburgh, Scotland that is fired daily at noon. It is very easy to tell the residents from the tourists when it fires: the tourists are startled, and the residents calmly check their watches--their startle reflex having been
15 completely habituated for that particular stimulus. A more familiar example: here at UF, new students quickly habituate to the sound of the bells of Century Tower, to the point where even after hearing the bells toll the hour, a student may not know what time it is. Thorpe views habituation as very important, since it keeps the animal "from wasting its energies in responses to stimuli which experience shows to be harmless or of no significance" (Thorpe 1963, p. 70). Habituation has been demonstrated in organisms as simple as Protozoa, which lack even a rudimentary nervous system (Abramson 1990). Protozoa exhibit a contraction response to touch which will habituate with repeated presentations (approximately 25 touches are sufficient). The
presence of habituation even at this rudimentary level of development demonstrates its adaptive value. There are two other phenomenon related to habituation: dishabituation and sensitization. Unsurprisingly, dishabituation is a decrease of habituation, usually due to the presentation of a strong or noxious stimulus. Strong stimuli can also produce sensitization--an increase in response magnitude to unhabituated stimuli. The second category of learning is classical conditioning. The 1949 Round Table Conference on Nomenclature in Animal Behavior defined it as
The process of acquisition by an animal of the capacity to respond to a given stimulus with the reflex reaction proper to another stimulus
16 (the reinforcement) when the two stimuli are applied concurrently a number of times. (Thorpe 1963, p. 80) This phenomenon was first studied in Pavlov's well-known experiments on dogs. When multiple presentations of food (the "unconditioned stimulus", or US) were paired with the ringing of a bell (the "conditioned stimulus", or CS), the dogs would eventually salivate at the sound of the bell alone (salivation being the "unconditioned response", or UR, normally produced by the US). This type of learning has been exhibited in organisms as simple as Planaria, a type of flatworm (Abramson 1990). Repeated presentations of shock paired with light will cause the contraction response normally exhibited from shock to be seen when the light is presented alone. A third type of learning, and the one most studied by Behaviorists, is "operant conditioning". This is where an animal associates a particular behavior with a reward or punishment--for example, a rat pressing a bar to obtain food. This type of behavior can persist if the reward is no longer presented--indeed, if the reward is presented unreliably, the behavior will persist even longer than if it had been presented reliably before. "Latent learning" is learning that occurs without an explicit reward. For example, rats will learn a maze even if there is no food in it--they have a natural tendency to explore and remember features of their environment.
17 In "insight learning", we see the type of learning that is most associated with humans. Thorpe described it as "the sudden production of a new adaptive response not arrived at by trial behavior" or "the solution of a problem by the sudden adaptive reorganization of experience." The "Aha" experience, the sudden insight, so common to human problem-solving (including computer programming) is an example of this type of learning. A category of learning that Thorpe ignored is called observational learning (Liebert and Spiegler, 1970). This is learning by imitation, learning by watching the actions of another animal ("monkey see, monkey do"). This type of learning has been demonstrated in animals as mentally unsophisticated as the mouse (Fioretto and Scotto, 1992), pigeon (Biederman et al., 1986), and octupus (Fiorito and Scotto 1992). In this last study, untrained octopus "observers" were allowed to watch trained octopi "demonstrators" select one of two objects that were presented simultaneously and only differed in color. Later, in isolation, the observers consistently selected the same object as did the demonstrators. In fact, the observers learned the task faster than the demonstrators (the demonstrators were trained via operant conditioning). Observational
learning is very important in learning social behavior of higher mammals (especially humans), but this study showed that even (admittedly very bright) invertebrates can learn by observation.
18 Ants have been shown to exhibit habituation and classical conditioning. Latent learning has also been shown in maze learning and the learning of landmarks. No operant conditioning results have been shown, but operant conditioning has been shown in locusts (see Abramson 1990). No insight or observational learning has been demonstrated in insects. My simulation concentrates on habituation and classical conditioning.
Habituation in Depth
Habituation is a more complex phenomenon than a simple waning in response to a repeated stimulus. In (Thompson and Spencer 1966), nine specific features of habituation were described:
1.
Given that a particular stimulus elicits a response, repeated applications of the stimulus result in decreased response (habituation). The decrease is usually a negative exponential function of the number of stimulus presentations. If the stimulus is withheld, the response tends to recover over time (spontaneous recovery).
2.
(This contradicts the "relatively permanent" aspect of Thorpe's definition, but is a quite common phenomenon).
3.
If repeated series of habituation training and spontaneous recovery are given, habituation becomes successively more rapid (this might be called potentiation of habituation).
19 4. Other things being equal, the more rapid the frequency of stimulation, the more rapid and/or more pronounced is habituation. The weaker the stimulus, the more rapid and/or more pronounced is habituation. Strong stimuli may yield no significant habituation. The effects of habituation training may proceed beyond the zero or asymptotic response level.
5.
6.
(In other words, even after a response is fully habituated, repeated exposes will make it more resistant to dishabituation).
7.
Habituation of response to a given stimulus exhibits stimulus generalization to other stimuli. Presentation of another (usually strong) stimulus results in recovery of the habituated response (dishabituation). Upon repeated application of the dishabituatory stimulus, the amount of dishabituation produced habituates (this might be called habituation of dishabituation).
8.
9.
(Thompson and Spencer 1966, pp. 18-19) The authors go on to say that "in reviewing the behavioral habituation literature, it is striking to find virtually complete agreement on the parametric characteristics of the phenomenon in such a wide variety of animals and responses." (Thompson and Spencer 1966, p. 19) Habituation is, in a sense, a recognition of the familiar. If one can recognize familiar things, one can pay special attention to the unfamiliar. Identifying unfamiliar things is very important for a "situated agent" trying
20 to learn and survive in the real world--for it is the unfamiliar things that can harm you or, at least, give you an opportunity to learn something new.
Classical Conditioning in Depth
Classical conditioning was, until recently, considered a very simple phenomenon. In the late 60's and early 70's, however, further experimentation found that things were not as simple as they once appeared (Domjan 1983, Holyoak et al. 1989, Klein and Mowrer 1989). One belief was that a simple temporal pairing of stimuli was sufficient to produce conditioning. Recent research has shown that temporal pairing is neither necessary or sufficient to produce conditioning. Some stimuli can be associated even though they are presented minutes or even hours later. For example, in taste aversion learning, an animal will associate its sickness to a food or drink it may have eaten hours before, forming a strong aversion to the associated substance. Other phenomena may prevent learning even if a close temporal association is present. One such phenomena is called "blocking". Suppose event CS has been paired with event US so that a strong association has been created. Now suppose that event CS2 is presented with CS, and then US is presented. Even after repeated trials, no association between CS2 and US will be created--the strong CS->US association "blocks" the CS2->US conditioning.
21 "Overshadowing" is the converse of blocking. If you pair events CS and CS2 reliably before event US and then try to train with CS alone before the US, you will find that learning the CS->US relation will be slower than it would be if you had not paired CS with CS2 initially. In "conditioned inhibition", a stimulus is used to trigger an exception to an existing association. For example, training an animal to associate noise with shock and then training with noise paired with light, but no shock, will cause the light to inhibit the fear response normally produced by the noise. Another response, which seems somewhat paradoxical, is the "overlearning-extinction" effect. Extinction refers to the gradual loss of an association previously built up between a CS and a US. If the CS is continually presented without a US, the response will gradually disappear. In general, the higher number of training experiences, the slower the extinction. However, after a certain number of trials, more training may actually produce faster extinction than less training. It has been proposed that some sort of depressive "frustration" reaction, triggered only by strong associations, is causing this effect. Finally, in "latent inhibition", if the CS has been presented previously, before its pairing with the US, its conditioning will be slowed. This is especially true if the CS had been presented in an uncorrelated fashion with the US. Despite its name, this effect can happen with both
22 excitatory and inhibitory learning. This is an example of the effect the animal's familiarity with a stimulus has on learning. Other situational aspects of the training can affect conditioning. The strength of an association between two events depends on how reliably of a predictor the first event was of the second. If an US follows a CS 100% of the time, the association will be strong. If the US and CS are presented in an uncorrelated fashion, so that the CS is not a good predictor of the US, no association will be made. Also, the magnitude of the unconditioned stimulus (the reward) limits the amount of conditioning. A weak reward will condition very slowly and will not produce a strong associative strength with any conditioned stimulus. A simple association may produce a complex sequence of behavior. "Autoshaping" was discovered when it was found that instead of having to elaborately train a pigeon to peck at a key to obtain food, all one had to do was to illuminate the key before food was presented. After about 50 trials the birds would readily approach and peck the key. The key was being associated with food, and so the approach and pecking behavior usually encountered with food was transferred to the key. A more familiar example: a domestic dog or cat will associate the sound of a can opener with food, and will come running when it hears the sound.
23 Biological Constraints on Learning
Contrary to the expectations of earlier researchers, a variety of unusual experimental results seemed to point to some species-dependent limitations in learning. These effects, called biological constraints (Domjan 1983), have these aspects in common: An exception to the rule. Biological constraints appear to contradict some other well-regarded theory of learning. Situational specificity. Learning in one type of task may be different that in other types of tasks. For example, if a rat is made sick, it will quickly learn aversion to a taste but not aversion to an auditory or visual cue, and, contrarily, if a rat is shocked, it will learn aversion to auditory or visual cues but not to a taste. Adaptive specialization. Certain types of learning have evolved because they are adaptive--their presence increases the likelihood of the animal's survival. For example, the long delays exhibited by taste-aversion learning are not seen for other types of stimuli. Animals have acquired specialized learning abilities that facilitate certain kinds of learning and hinder others. The fact that animals are not general-purpose learning machines helps emphasize the importance of the ethological approach. Not only have an animal's instincts been tailor-made for its environment, but its learning capabilities have been specialized as well. By studying the animal in the
24 wild, one can see the usefulness of these specializations in promoting the survival of the animal.
CHAPTER 3 AN OVERVIEW OF ANT CAPABILITIES AND BEHAVIOR
The Ant Colony Life Cycle
An ant colony is founded when a mated queen loses her wings and starts burrowing her first egg chamber--the male dies soon after mating (Hlldobler and Wilson 1990). In Solenopsis invicta, she feeds this first small group of workers by laying infertile "trophic" eggs to feed to the larvae. This keeps her from having to forage, a sound strategy since the mortality rate is high for foragers. Once they leave the larval stage, the
workers, actually infertile females, perform a variety of tasks--grooming and feeding the queen, feeding and caring for the larva, and leaving the nest to forage. However, despite the stereotype of the "industrious" ant, a typical worker ant is in the nest either standing still, grooming, or walking around aimlessly as much as 78% of the time. Once the colony is large enough, fertile females and males are born-these reproductive forms have wings. Eventually, enough are present to trigger a swarm, in which the reproductives leave the colony en masse, mate, and form new colonies. In Solenopsis invicta, the colony may swell to 250,000 ants before the swarm takes place. An S.i. queen may live from 6
25
26 to 7 years--workers live anywhere between 10 and 70 weeks. In S.i., workers range from 2mm to 6mm in size and are classed into three groups on this basis: minor, media, and major castes. Different castes perform tasks in differing amounts (Mirenda and Vinson, 1981). Smaller ants spend more time caring for the brood, while media caste workers are most likely to forage. Major workers are best at subduing and carrying larger prey.
The Ant's Navigational System
When an ant finds a food source on a foraging trip, they will head directly back to the nest, even if they did not take a direct route out to the food. They manage this trick through a variety of navigational methods, all of which rely on input from their visual system. The primary means of navigation in ants is via direct observation of the position of the sun. During the trip out, they observe the position of the sun as they move. Their nervous system is apparently able to perform the vector additions required to determine a direct path back to the nest--the neural mechanism for this is still unknown. Some species of ants can even account for the sun's apparent movement in the sky over the course of time and adjust their trip back accordingly. An ant's eyes can also sense the polarization of light. Since the atmosphere polarizes the light coming from the sun, an ant can deduce the position of the sun even if the sun is blocked by a patch of clouds. In
27 addition, ants can use landmarks for navigation on very cloudy days when a patch of sky is not visible. This cascade of mechanisms creates a robust navigational system, one whose redundency should serve as an example to autonomous robot designers. Far too many robots rely on one navigational method and are totally helpless should that mechanism fail.
Pheromones and Behavior
Much of an ant's behavior are triggered by pheromones, chemicals released by other ants for the purposes of communication. In Solenopsis invicta, there are 11 different classes of behaviors involved (Table 3-1). Ants contain a variety of glands which secrete these pheromones. Some pheromones are given off solely by larvae, others solely by the queen.
The Alarm Response
If an ant is disturbed, it will release an alarm pheromone. In Solenopsis invicta, two different pheremones are involved. One, a longerrange pheromone, attracts any ants in the vicinity--it is the same pheromone used for laying chemical trails and marking the nest. The other pheromone triggers an alarm response (rapid movement, circling behavior, increased aggression). The alarm response will habituate after repeated presentations if the stimulus is not too strong--in fact, even fire ants can be
28
Table 3-1. Alarm
Behavior triggered by pheromones. From Hlldobler and Wilson, 1990.
Simple attraction Trail laying and following Grooming other ants, molting assistance Food exchange (between workers and the feeding of larvae) Group facilitation/inhibition of an act Recognition of nestmates, castemates, and removal of the dead Caste determination Control of competing reproductives (in queen, wing loss and egg-laying are triggered by lack of other queens) Territorial signals and markers Sexual communication (species recognition, timing of swarm)
handled safely after they have had time to become accustomed to being handled.
Foraging and Trail-Laying
To forage, an ant leaves the nest in a random direction and heads out looking for food. It keeps a fairly straight path out, but will wander a bit as it gets farther from the nest. If it finds a food source, the ant will return straight back to the nest via the navigational systems already discussed, laying a chemical trail. In Solenopsis invicta, the chemical trail has a range
29 of about 1 cm and lasts about 100 seconds. This species will even modulate the intensity of the trail left in proportion to the size of the food source.
Other Behaviors
Ants have a fairly rich and complex behavioral repetoire. A list of behaviors and their relative frequency of one species in given in Table 3-2. This table list behaviors for an undisturbed nest; behaviors not included in the table include responses to invasion by alien colonies, flooding, and overheating, as well as emigration to a new nest site and assisting during the initiation of nuptial flights. For the purposes of this simulation, we modeled only alarm and foraging--all other behaviors are currently ignored. Many of the other behaviors could be added to this existing framework, time permitting. Others behaviors would require significant improvements to the simulated environment the xants live in. For example, if a colony of S.i. is flooded during a storm, the workers will carry the queen and brood (eggs and larvae) outside the colony, link together, and form a living raft of ants which will then be carried off by the current to a new (hopefully less flood-prone) location. Burrowing is another type of behavior difficult to model, since simulating dirt in three dimensions with a fine particle size would be quite CPU-intensive. In addition, little
30 research has been done on burrowing in ants, so it would be difficult to know whether the simulation was accurate.
31
Table 3-2. Behaviors of two physical castes of the ant Pheidole dentata in an undisturbed colony with their relative frequencies. From Hlldobler and Wilson, 1990. Frequency of acts Behavior Minor Workers (N=1,222) 0.18003 Major Workers (N=208) 0.56373
Self-Grooming Allogroom Adult Minor worker Major worker Alate or mother queen Brood care Carry or roll egg Lick egg Carry or roll larva Lick larva Assist larval ecloysis Feed larva solid food Carry or roll pupa Lick pupa Assist eclosion of adult Regurgitate With larva With minor worker With major worker With alate or mother queen Forage Feed outside nest Carry food particles inside nest Feed inside nest Lick meconium Carry dead nestmate Carry or drag live nestmate Eat dead nestmate Handle nest material Total
0.04992 0.00573 0.01146
0 0 0
0.01391 0.00245 0.12357 0.09984 0.00409 0.00573 0.03601 0.01882 0.00818
0 0 0 0.02941 0 0 0 0 0
0.02128 0.03764 0.00573 0.00327 0.12111 0.04337 0.05237 0.05810 0.00573 0.01882 0.00246 0.06383 0.00655 1.0
0 0.02941 0 0 0.02941 0.01471 0 0.01471 0 0.04902 0 0.07843 0 1.0
CHAPTER 4 CLASSIFIER SYSTEMS AND THEIR IMPLENTATION IN XANTS
Definitions
A "classifier system" (Holland et al. 1986, Goldberg 1989) is a production rule based system with some distinctive features. Classifiers are condition-action rules. Classifiers are tested against "messages", fixedlength binary strings. Communication with the outside world is via these messages--some are posted by input devices, and others are read by output effectors and can perform external actions. A classifier may be activated via a message due to some external event, by a message posted by another classifier in the previous time step, or by a combination of these elements. In a basic classifier system, if a classifier's conditions are matched with the current list of messages, it posts its action part (another message) to the message list for the next time step. The condition part of a classifier consists of a list of "schemas"--a fixed length ternary string consisting of 0s, 1s, and don't-care symbols usually represented by pound signs (#). For example, the schema #01# would match the messages 1010 and 0011, but not 0100. The action part
32
33 can be either a fixed-length binary or ternary string--the don't-cares here acting as pass-throughs from a matching message. If, for example, the message 000101 matched a classifier with the action part 111###, the resulting message would be 111101. Notationally, each condition in the classifier is separated by a comma, with the action part separated by the rest with a "/." In addition, each condition may have a "-" preceding it, in which case it will be matched if there is no message matching the schema in the message list. A example classifier would be 00##,-1000/1111. This is translated as "if you see a message in the message list with two leading 0s, and you do not see the message 1000, post the message 1111." In other variants of classifier systems, instead of each matching classifier posting its action, it competes with other matched classifiers for the privilege. Each classifier produces a bid, which is calculated by a strength associated with the classifier and the specificity of the classifier (the more don't-cares, the less specific it is). The classifier with the highest bid wins. A classifier's strength may be modified based on its worthiness, determined by the effect that its firing had upon the system. Various algorithms, such as the bucket brigade algorithm, can apportion credit and blame for successful or unsuccessful outcomes even when classifiers are
34 indirectly responsible (a classifier's firing causing another classifier to fire causing an desirable outcome should cause both rules to be strengthened). The classifiers may be subject to modification via "genetic algorithms", which are changes to rules that work like the changes that happen to the codons in DNA. The operations include reproduction (successful rules making copies of themselves), mutation (single symbols in a condition or action changing their value), and crossover (two classifiers exchanging sections after a crossover point). Rules may be generated de novo in a variety of ways as well. A classifier system may consist of several basic classifier systems working together--the classifiers activated from one message list posting to another. Alternatively, the various lists can be combined into a single list via "tagging"--using some of the message bits as a tag to determine the list to which it belongs. The beauty of these systems is that they can be used for any kind of task, as long as you can handle the inputs and outputs to the system as binary strings. Classifier systems have been used for such diverse applications as machine vision, learning complex Boolean functions, mazerunning, and simulated gas-pipeline control.
35
The Xants Implementation of Classifier Systems
In Xants, each message posted to the message list has an associated strength value. This is an important feature, as it makes the implementation of nonassociative learning straightforward and also allows the xant to exhibit preferences to stronger stimuli over weaker ones. These message strength values are used in computing a matching rule's bid. Message strengths have a range from 0 to 255. Some types of messages have a fixed strength. There is system variable called POST_THRESHOLD--any message with a strength below POST_THRESHOLD is not posted to the message list. This simulates a environmental input too faint to notice. Each rule in Xants has up to two conditions, an optional action, and an optional prediction. The action message (which can include pass-through characters) is posted to an action message list to perform various behaviors; the prediction message is posted to the prediction list, which is used to evaluate the validity of the rule. Rules without predictions are not subject to strength revision. However, predictionless rules must compete with rules that are subject to revision and can therefore be supplanted by strong rules with valid predictions.
36
Message Types
This system, like Holland's CS-1 system (described in Goldberg 1989), uses a message tag to distinguish between the different sources of messages in a common message list. A list of message tags is provided in Table 4-1; a detailed description of each message type follows. For each type, only those bits that have defined values are relevent; the other bits in the fixed-length messages are ignored.
Null
Any message with a tag of 0000 is treated as a null message. Null messages are not posted onto a list: they act as place-holders in rules that have no actions (rules that have predictions only, for example).
Odor
For each odor sensed by the olfactory function, an odor message is posted to the message list. The first three bits identify the kind of odor (Alarm, Food, Trail). The next three identify the direction of the source (Left, Straight, or Right). For example, a message that there is a Food odor to the Left would be posted as: 010 100 0001
37 The strength value of an odor message corresponds to the strength of Table 4-1. Message Types and meanings of bits.
Null 0
Tag 0 0 0
Inputs from environment: Odor Type Odor A F T Direction L Tag 0 0 1 0 Tag A n n o y S R 0 Tag 0 0 1
Taste
Type F
Stimulus Type Stimulus Onset S h o c k A i r B u m p S o u n d L i g h t
1
0
0
1
Collision 0 1
Clock Position 2 3 4 5 6 7 0
Tag 1 0 1
38 Table 4-1--continued Internal Messages:
Current State
State H N T R F 0
Tag 0 1 1 Tag 1 1 Tag A n n o y 1 0 1 0 0 1
Homing Direction
At-home A
Direction L S R
Stimulus Trace
Stimulus Type S h o c k A i r B u m p S o u n d L i g h t
Outputs: Move Speed X Set State Motion F State N T Action S D E 1 R F 1 Tag 1 1 1 N B Direction L S R 0 1 Tag 1 1
Tag 1 1 0
General Effector
39 the odor, which varies inversely with distance. If the strength is below POST_THRESHOLD, no message is posted and the odor is ignored.
Taste
When the xant's mouth is in physical contact with an object of type food, a taste message is posted--tasting non-food objects does not cause a taste message. The F bit in the Taste message signifies a food taste, allowing other tastes to be added in the future by adding other taste bits to this message type.
Stimulus Onset
Whenever a shock, puff of air, bump, sound, or flash of light is noticed (at the beginning, or onset, of the stimulus), a stimulus onset message is posted to the message list. The appropriate stimulus type bit is set. If the stimulus type is Shock, Air, or Bump, the last bit (Annoying) is
set. This distinguishes noxious stimuli from other, neutral stimuli. This message type covers stimuli that xants are sensitive to that do not have their own message type. The stimuli in this type are also the only stimuli that are subject to nonassociative learning.
40
Figure 4-1. The Collision Clock. Collision
Every cycle, the collision message is posted, summarizing the positions on the xant's body where it is in contact with another object. The clock position numbers correspond to the "collision clock", which gives an approximate position of the contact (Figure 4-1). Each position on the collision clock handles a 45 sweep around the xant. If an object is colliding with the ant at that position, the appropriate collision clock bit is set; otherwise, it is zero.
Current State
At each tick the xant's current state is posted. The first bit (Hungry) is independent of the others. The other state bits (Nesting, Trail-laying, Returning, and Foraging) are mutually exclusive. Nesting is the state an
41 xant is in when it is in the nest, roaming around. An xant is Foraging when it is looking for food sources outside the nest. An xant in the Traillaying state heads home, leaving a chemical trail. In the Returning state, it heads home without laying a trail. There is an internal hunger value that increases over time. If the hunger value is over the the value HUNGER_THRESHOLD, the H bit is set on the current state. Eating decrements this hunger value. The behavior in each state and the transitions from state to state are controlled by the rules in the rule list, not by any intrinsic actions caused by the message.
Homing Direction
The Direction tags (Left, Straight, Right) give a rough indication of the direction of home, assumed to be derived from sun-compass reckoning and landmarks. The A bit (At-Home) is set when the xant is inside the nest.
Stimulus Trace
Any general stimuli in the recent past are re-posted to the Message List in later ticks, but with a different tag so that they can be distinguished from current events. This allows the xant to have rules that can refer to events in the recent past, as well as current events.
42 Move
This message controls the xant's motion for the next tick. The xant will move Forward, Neutral (stand in place), or Backward depending on the value of the Motion bits. The direction is determined by the Direction bits (Left, Straight, Right). If the X bit is set, motion occurs at twice-normal speed.
Set State
This message sets the current state, depending on the values of the State bits (Nesting, Trail-laying, Returning, Foraging). The xant stays in this state until another Set State message is posted.
General Effector
This message causes other actions (Sting, Drop Alarm, or Eat) to be performed. If an Eat message is posted, the xant's hunger value will be decremented as long as the object being eaten is a food. The Drop Alarm message causes the xant to emit a drop of alarm pheromones, the amount in proportion to the strength of the triggering stimulus message. For the Drop Alarm action, the strength of the message controls the amount of pheromones dropped--lower strengths drop smaller amounts.
43
An Xant's Brain: Theory of Operation
At this point, let me describe in detail the inner structure of an xant's brain (Figure 4-2). There are six major structures, containing various types of information. The Message List is where input from the environment, along with internal state messages, are posted. The Stimulus List is used to gauge the familiarity of stimuli and to keep track of recent stimuli so that stimulus trace messages can be posted. The Rule List contains all the condition/action/prediction rules for the system. The Bid List is used to
Figure 4-2. The Organization of the Xant's brain
44 For each tick in the simulation: Phase I. 1. 2. 3. 4.
5 Phase II. 1.
Clear the Message List Post last tick's actions to the Message List. Post input from environment for all message types except Simulus to Message List. Adjust strength of Stimulus type based on experience; post Stimulus messages with modified strength to the Message List. Post Stimulus Trace messages for any stimulus that occured in the recent past. Match messages in the message list with rules in the Rule List. Post resulting Bids to the Bid List. For each message type: a. Find the highest bid with an action of that message type. b. Post the action and prediction messages for this highest bid to the Action and Prediction Lists. Remove predictions from the Prediction List whose prediction time has lapsed, punishing the posting rules for positive predictions, and rewarding for negative predictions. If a stimulus occurred this tick: If the stimulus was predicted, reward the predicting rule, unless it was a negative prediction, then punish. else try to find a preceding stimulus that might predict this one--if found, generate new rules based on exception creation, covariation detection, and chaining. Perform the actions specified by the message in the Action List.
Phase III. 1.
Phase IV. 1.
2.
Phase V.
Figure 4-3. Pseudocode representation of the xant brain phases of operation.
45 hold bids, which are matches from the rule list/message list evaluation. Actions from winning bids are posted to the Action List, and predictions are added to the Prediction List. There are five distinct phases of processing performed by the xant's brain each tick. Figure 4-3 gives a pseudocode representation of the phases. Each phase is described in detail below. During Phase I, environmental input, along with the input from the xant's internal senses, are posted to the Message List (Figure 4-4). In addition, Stimulus Trace messages, due to stimuli on previous ticks, are posted. A system variable called MAX_ISI controls Stimulus Trace posting-if an event happened more than MAX_ISI ticks ago, it will not be posted to
Figure 4-4. Phase I: Inputs posted to the Message List
46 the message list this tick. In addition, the winning bid messages from last tick are posted, to allow rules to be constructed that have knowledge of last tick's actions. When Stimulus Onset messages are posted, the system looks for the stimulus in the Stimulus List. If the stimulus is of moderate strength and is familiar, its strength will be reduced to simulate habituation. If a stimulus is stronger, it will be habituated less, and in fact, for very strong stimuli, the xants will become sensitized, temporarily decreasing habituation to all stimuli, and increasing the response to stimuli yet-to-be habituated to.
Figure 4-5. Phase II. Rule-message matching and bidding.
47 In Phase II, the messages in the message list are matched to the rules in the Rule list, and bids are made (Figure 4-5). The bids are calculated using both the strength of the rule and the strength of the messages in the message list. The bid also records the strength of the message matching the rule's first condition for later use. The details of bidding are discussed in more detail in the next section. In Phase III, the winning bids for each message tag are calculated (Figure 4-6). Only one winning bid is allowed for each message tag. The action part (if present) of each winning bid is posted to the Action List, and the prediction part (if present) is posted to the Prediction List.
Figure 4-6. Phase III. Posting winning bids to the Action and Prediction Lists.
48 The Action List is created anew each tick. In contrast, a prediction added to the Prediction list remains there until it is fulfilled or invalidated. The strength of the action message is the same as the the strength of the message that matched the bidding rule's first condition. In Phase IV (Figure 4-7), generation of new rules, modification of the strength of existing rules, and evaluation of predictions takes place. The methodology used here is similar to that of Holyoak et al. (1989), with enhancements to allow predictions to span more than one tick. This phase is discussed in more detail below.
Figure 4-7. Phase IV. Rule creation and strength modification via prediction evaluation.
49
Figure 4-8. Phase V. Performing effector actions and setting state variables as determined by the Action List. Finally, in Phase V (Figure 4-8), the effector messages from the winning bids are acted upon. Any effector actions or state changes are now performed.
50
Bid Calculation
For rules with one condition, the bid is calculated using this formula:
Where: Smessage Srule Speccondition Wbid N is the adjusted message strength; is the strength of the rule; is the specificity of the rule's condition part; is the bid-weighting constant (currently 1.0). is the random noise value (an random integer from 0 to 255)
The Specificity value is a number from 0.0 to 1.0, which measures how specific the condition part of the rule is. If the condition has no wildcard characters, it has a specificity of 1.0; if it is all wildcard characters, it has a specificity of 0.0. This gives preferential treatment for more specific rules, allowing the default hierarchy discussed above--specific exception rules protecting more general rules. The random noise value creates what is called a "noisy auction". The noise keeps the strongest rule from winning every time, allowing some
51 weaker rules the chance to prove themselves and be strengthened or weakened even further. For bids with two rules, the formula is similar:
Smess1Smess2SruleSpeccond1Speccond2Wbid N Here, the strengths and specificities of both conditions are present. Every rule that has all of its conditions matched posts a bid to the Bid list. Since the noisy auction would otherwise allow them to participate, rules with a strength of zero are excluded from the bidding competition. This way, invalid rules can be effectively ignored.
Nonassociative Learning: Habituation, Dishabituation, and Sensitization
When a stimulus is encountered, it is checked against the Stimulus List for strength adjustment for habituation, dishabituation, and sensitization. The first time the stimulus is encountered, its strength is unmodified, but the stimulus is added to the Simulus List for future reference. Initially, the stimulus's familiarity factor, Fstimulus, is set to the current value of the Stimulus List's initial_familiarity value, which may vary depending on level of sensitization. If the stimulus is already present in the Stimulus List, it has been encountered before, and the following values are calculated: Astrength= 1.001 - Mstrength/MAX_STRENGTH
52 Atime= 1.001 - Telapsed'/MAX_ISI Acount=1+(# exposures to stimulus)/POTENTIATION_BASIS Telapsed= time since last occurence of stimulus Telapsed'= minimum of (Telapsed,MAX_ISI) If the Telapsed value is below the value of SPONTANEOUS_RECOVERY_TIME and the value of Fstimulus is less than MAX_HABITUATION, then the value of Fstimulus is modified: Fstimulus=Fstimulus*(1+Astrength*Atime*Acount) This causes the familiarity of the stimulus to increase up to double its current value, provided that the strength and the interval are close to minimum values. The terms in this equation give the familiarity some important characteristics--because of them, familiarity is more rapid for lower stimulus strengths, shorter inter-stimulus-intervals, and for stimuli it has been exposed to more often, key features of this phenomenon. Once the new value of Fstimulus has been calculated, then adjusted stimulus strength is calculated: Sadjusted=Sstimulus/Fstimulus*Astrength Here, the familiarity measure is decreased in proportion to the strength of the stimulus this tick, since stronger stimuli habituate less despite their familiarity. The extra strength adjustment is included here instead of already being present in Fstimulus to allow a proportional response to stimuli of different strengths once habituation has taken place.
53 If the elapsed time is greater than the SPONTANEOUS_RECOVERY_TIME, then the systems performs dishabituation to simulate spontaneous recovery:
Aelapsed
Telapsed SpontanousRecoveryTime Fstimulus 2A
elapsed
Fstimulus
Fstimulus maximum(FStimulus,MinHabituation) Regardless of whether the stimulus's strength was adjusted or left the same, if the stimulus's strength is above the SENSITIZATION_THRESHOLD value, then the entire Stimulus List is dishabituated by a factor of 4, and the initial_familiarity value is halved. This causes strong stimuli to sensitize the entire system to all stimuli. In addition, since habituation will generalize to other stimuli, each time a stimulus is encountered, the value of INITIAL_FAMILIARITY is slightly increased, as are the familiarity values of all encountered stimuli-each is multiplied by the system variable FAMILIARITY_ADJUST. The combination of these various manipulations produce a response that has most of the characteristics of a real organism's nonassociative learning.
54 Prediction Evaluation, Rule Strength Modification
Each prediction in the Prediction List contains: a prediction message, with don't-care symbols and an optional "not" flag (signifying a negative prediction--a prediction that a message will not be posted); the index of the predicting rule in the Rule List; the tick when the rule was posted; and the expected delay between the posted time and the time by which the predicted event should occur. The system variable PREDICTION_RANGE allows for the actual time of the event to be PREDICTION_RANGE % later yet still be counted as a valid prediction. An event may occur any time between this cutoff time and the time it was posted and be considered a fulfillment of the prediction. To begin this phase, each prediction in the Prediction List is checked to see if the expected time for the prediction has lapsed. If so, the posting rule is weakened, and the prediction is removed from the list. In addition, the create_exception function, which is described below, is called to try to generate a exception rule. However, if the prediction was a negative prediction, then the prediction can now be considered fullfilled: the posting rule is strengthened, and the prediction is removed from the Prediction List. The system variables REWARD_RATE and PUNISH_RATE control the amount of strength modification--their values are added to the current strength (PUNISH_RATE's value is negative).
55 Now that the expired predictions have been dealt with, the Stimulus List is checked to see if any stimuli were encountered this tick. If a stimulus was encountered, it is checked against the Prediction List to see if it was predicted. If the stimulus was predicted, then the predicting rule is strengthened and the prediction is removed from the list. However, if the prediction was a negative prediction, then the prediction is invalidated; the predicting rule is weakened, and the prediction is removed from the Prediction List. In addition, the create_exception function is called to see if an exception rule can be created. If the stimulus was not predicted, the system tries to find another stimulus that preceded this stimulus--something that could act as a predictor. If one if found, new rules are created via the create_covariation and the create_chaining functions described in the next section.
Rule Generation
There are three types of rule generation in Xants--exception creation, covariation detection, and chaining. These methods used here for rule generation are the same methods used in Holyoak et al. (1989) except that they have been expanded to handle multiple conditions and variable interstimulus-intervals. Exception creation. When the exception-creation function is called, a prediction has failed. The system checks to see if there was a stimulus in
56 the MAX_ISI ticks before the prediction was made. If so, a new rule is created, with this stimulus added as a condition and with the negation of the failed prediction. For instance, if the rule if S then action A, predict P made an invalid prediction P and there was a stimulus T present before the rule fired, the rule If S and trace of T then action A, predict -P would be created and added to the Rule List. The rule will have the same strength as the parent rule, initially, and the new prediction's delay time would be the same as the original prediction. If the rule that failed already had two conditions, no new rule is generated. Covariation detection. When the covariation-detection function is called, an unpredicted stimulus S has occurred. In addition, a stimulus T has occurred within MAX_ISI ticks before stimulus S. Given these two facts, the rule if T, predict S is created and added to the rule list. The covariation-detection rule created does not contain an action part--it merely predicts the new stimulus. Chaining. When the chaining function is called, an unpredicted stimulus S was encountered and a stimulus T preceded S. The function checks the Rule List to see if a rule that posted a winning bid had stimulus S as a condition. If such a rule is found, a new rule is created with stimulus T replacing S. In this way, a previously neutral stimulus T may produce the response normally elicited by S. This is the heart of classical conditioning.
57 Unlike traditional classifier systems, this approach to new rule generation has severe constraints on crossover--crossover only occurs between whole condition and predictions, not at any arbitrary point in the rule string. As it turns out, this may be more similar to true genetics than the traditional Classifier System model, as recent research has suggested that crossover in real DNA is constrained to keep from splitting a genes in two--crossover only occurs in the unused sections of DNA called introns (Young 1993). The combination of new rule generation and strength modification yield a system capable of creating new rules, evaluating their effectiveness, and discarding them if they prove incorrect. This, tied with the nonassociative learning capabilities, produce a system capable of producing quite accurate modeling of a real animal's behavior.
CHAPTER 5 DESCRIPTION OF THE SIMULATOR
The Xants World
There are 7 types of objects in the Xants world. Xants. The purpose of the simulation. They move around, seeking food, laying chemical trails, emitting pheromones, and learning. Foods. What the xants eat, and their primary goals. Foods can be detected at a distance via odor. When eaten, they reduce in size and eventually disappear. Blocks. Obstacles for the xants. Blocks cannot be climbed over--they must be walked around. The Nest. The xants' home. Its location is known by the xants via their homing sense. In addition to these tangible objects, other objects are sensed as odors when in range. Trails. Chemical trails left by the ants. They can be detected by other xants and evaporate over time. Food odors. Each food has a odor cloud around it--if an xant is within the range of the odor cloud, it can smell the food.
58
59 Alarms. If an xant is bothered in some way, it drops two different alarm pheromones, which dissapate over time. The pheromone with the larger range is identical to the trail pheromone and attracts other xants to the area. The other, shorter range pheromone causes an alarm behavior-circling and fast running.
Control Files for the Xants Simulator
When "xants" is typed at the Unix prompt, the Xants programs looks for two files, "xants.init" and "xants.brain." If the program cannot find them in the current directory, it looks for them in the directory containing the Xants executable. The xants.init file contains various settings controlling the execution of the program (Table 5-1). The program starts by placing the requested number of xants, blocks, and food sources at random places in the Xants world. The program creates a simulated world of dimensions 2*MAXDOUBLE on a side. Initially, however, the program places these objects in a square area whose dimensions are specified by "worldsize", which is measured in units called "units"--one unit is approximately equal to .1mm in the real world. The "viewsize" variable controls the size of the 2D viewport on the X Windows screen. This viewsize is in pixels, not units, since the view can be zoomed and unzoomed to provide varying levels of magnification. The magnification factor is initially 1.0, so setting viewsize and worldsize the same allows one
60 Table 5-1. Variables used in xants.init file Variable worldsize Default Value 500 Description The size of the initial deployment area for objects in the world, in units.
viewsize
500
The size of the initial 2D viewport.
numants numblocks numfoods
20 10 5
The number of ants in the world. The number of blocks in the world. The number of food sources in the world.
trailrange
10
The detection range of trails,in units.
slowmotion
1000
The time between ticks when slow-motion is enabled (in ms). Number of ticks between displayed frames.
skipfactor
1
to see all the objects placed in the Xants world initially. The program creates a central nest at location (worldsize/2, worldsize/2). This is centered on the 2D display when the program begins. The format of the xants.init file is simple--a variable's name, a space, the value the variable should have, and a return to finish the line. Any values not set in the xants.init file retain their default values.
61 Once the xants.init file has been read, the xants.brain file is processed. This file contains the initial rule set for each xant in the simulation.
Syntax of the Xants.brain File
The xants.brain file begins with an optional "descriptors" section (Figure 5-1). The purpose of the descriptors section is to allow the trace function to output more descriptive information than simply lists of binary strings. The descriptor section consists of a sequence of descriptor lists, separated by carriage returns. A descriptor list is a series of strings, separated by commas, followed by a colon and a 4-bit binary string. Each string in the descriptor list corresponds to a bit in a corresponding message.
; xants.brain--the xants brain file ; ; First, I'll define some descriptors for message ; output: ; descriptors Alarm, Food, Trail, Left, Straight, Right, Smelling:0001 Food,Tasting:0010 Hungry,Nesting, Trail-laying,Homing,Foraging, I am:0011 Alarm,Timer:0100 Right,Far Right,Middle,Middle,Middle,Middle,Far Left,Left,Blocked at:0101 Fast,Forward,Neutral,Backward,Left,Straight,Right, Move:0111 enddescriptors Figure 5-1. The Descriptors section of an xants.brain file.
62 If the bit is set to 1, the string will be displayed. If the bit is 0, the string will not be displayed. The last string in the descriptor list is the string used to identify the given 4-bit class of messages. For example, given the descriptors for the 0001 tag in Figure 5-1, the message 100 001 0001 would display as "Smelling Alarm Left." This makes understanding the trace output much easier. The descriptions given in the descriptor list should correspond to the meanings assigned to the bits in the messages posted to the message list for the xants, to provide meaningful trace output. Since those meanings are currently hard-coded in the definition of the xants, a simple re-labeling in this file will not change the form of these binary strings. Following the descriptor section are the rules that govern the behavior of the xants. These rule consists of one or two conditions to be matched, the action message to be posted, and an strength value for the rule. Throughout the xants.brain file, you can insert descriptive comments by starting them with a semicolon. To make understanding the rule coding easier, We usually put comments above each rule giving the meanings of the bits. For example: ;AFT LSR odor/FNB LSR move 001 100 0001/100 100 0111,200
63 This rule translates to "if you see a message for a trail odor to the left, move forward to the left (rule strength = 200)." A more useful rule would be: ;AFT LSR odor/FNB LSR move 001 ### 0001/100 ### 0111,200 This rule translates to "if you smell a trail in a certain direction, move forward in that same direction"--the don't-care symbols in the action part acting as pass-through characters from the message matching the condition. Here is an example of a two-condition rule: ;AFT LSR odor,ANTHF stat/FNB LSR move #1# ### 0001,#0001 0011/100 ### 0111,5
This rule translates to "if you smell a food source in a certain direction, and if you are in Foraging state, move forward in that direction, (strength 5)." By convention, the pass-through characters on the action message use the first condition's message to fill in the don't-care values. There is no provision for putting predictions in the xants.brain file, as it would be inappropriate for an xant to re-evaluate its instincts. Instincts can be outbid by generated rules with higher strengths, however.
64
Figure 5-2. The Xants interface, with descriptions of visible objects.
The Xants Interface--Controls and Options
When Xants begins, the opening screen looks like Figure 5-2. Here, we see the 2D view, along with control buttons stretching across the top, and a status bar below them.
65 The 2D view provides a simplified view of the Xants world. By using the mouse and clicking on objects in the 2D view, you can change both change the view and interact with the Xants environment. The left mouse button allows you to "lock on" to any Xants object. When an object is locked onto, it is displayed in the center of the 2D view. If the object moves, the view will follow it, keeping it centered. This is the best way to keep track of the behavior of a single xant. When you locks onto an object, the message "Locked onto <object-name>" will appear on the status line. The middle mouse button allows you to pick up and move any Xants object. Simply move the mouse pointer over the object and click on it. While holding down the middle mouse button, drag it to its new position and release the button. The message "<object-name> moved" will appear on the status line. You should not try to move an object while locked onto it-things move very rapidly that way. The right mouse button will let you rotate any Xants object. Just put the mouse pointer near the object and hold down the button. As you move the pointer around the object, it will rotate. Release the button when the object is in the correct orientation. This is useful in pointing a given xant in the right direction. Several control buttons affect the 2D view. The Zoom In and Zoom
Out buttons magnify and reduce the image. If you hold down the button,
66 the zoom will continue until you lift it. The Pause button stops the simulation--you can change the view even with the simulation paused. On the View menu, there are several options that affect the 2D view. The Slow Motion option slows down the simulation, allow you to view it tick by tick. The value of the "slowmotion" variable in the xants.init file controls the speed while in slow motion. The Reset View option resets the current view to be centered on the nest at normal magnification. The Show Food Odor option makes food odors visible. The Show Alarm Odor makes the alarm pheromones visible. The Show Bounding Boxes shows the bounding boxes the system uses for collision detection.
The Trace Command--Observing an Xant's Brain
With the Trace command on the View menu, you can view the operation of a given xant's brain from tick to tick. The trace output is sent to the Unix standard output--we advise users to maximize their text window so they can see the maximum number of lines at once. You can redirect output to a file by running Xants with output redirection, or pipe the output through the Unix "tee" command to view the trace and store it in a file at the same time. You should be warned that trace output is rather verbose and can create large files. To use the trace option, lock onto an xant with the left mouse button and select Trace on the View menu. When you turn on the Trace command,
67 ************* TRACE OF Ant7 tick 189:****************** Messages: Move Forward Straight Tasting Blocked at Home is Left I am Hungry Foraging Bids: Move Move Move Move Move Move
(510) (255) (510) (255) (255)
Forward Right Forward Left Forward Straight Backward Left Backward Straight Backward Right
(bid=981,input=510) (bid=907,input=510) (bid=1638,input=510) (bid=337,input=510) (bid=140,input=510) (bid=365,input=510)
Winning Bids: Move Forward Straight Predictions: Rule Generation:
(510)
Figure 5-3. Sample output from the Trace command for one tick. the system automatically goes into Slow Motion mode to slow down the trace output to a watchable speed. You can turn slow motion off if you wish. Sample output for the trace command is given in Figure 5-3. Judicious use of the descriptors option in the xants.brain file produces quite readable output. For each tick, each operation of the xant's brain is displayed in detail. The messages section displays all the messages posted to the message list
68 that tick with their strength in parenthesis. The bids section lists all the bids generated from those messages by matching the messages against the rule list. The strength of the bid is shown, along with the strength of the message matching the bidding rule's first condition. The winning bids section shows the winning bids for each message type (in this example, only the move message type was represented), along with the strength of the message matching the first condition, which will be used for the action message. The predictions section lists any pending predictions, and the rule generation section details the creation of any new rules.
Performing Experiments on the Xants
With the options available on the Edit menu, you can expose ants to stimuli, gauge their reactions, and train them to associate stimuli. Before you begin an experiment, lock onto the xant you wish to work with--if you do not, you will be exposing all the xants to the experiments you are performing. You will see a variety of stimuli on the edit menu--light, sound, bump, air, and shock. The last three stimuli trigger an alarm response-consider them to be unconditioned stimuli (USs). The light and sound stimuli trigger no response initially, but with training can be associated with alarming stimuli--they can be the conditioned stimuli (CSs) for any classical conditioning experiments.
69 The Set Intensity control allows you to control the strength of the stimuli. Move the slider to the right for strong stimuli, to the left for weak ones. Click on OK and the status line will display the stimulus intensity you set as a percent of maximum. Set the intensity low and the alarm response will be low or nonexistent. Set it high and the response will be greater. Remember how nonassociative learning works--weaker stimuli will habituate faster and stronger stimuli will sensitize the organism. You can associate two stimuli by repeated presentations of the CS before the US; eventually the CS will produce the desired response. You may wish to run experiments with trace mode on so that you can see the effects of your manipulations.
Generating 3D Animations
Recording Animation Script Files
Using the Record Animate Script option on the File menu, you can create animation scripts that you can then render in 3D using the "Animate" program developed here at UF by Hanns-Oskar A. Porr. Be warned, however, that the Xants program slows down dramatically while generating script files and that the files become very large, very quickly.
70 When you select Record Animate Script, the animation script file is written to the Unix standard output--you can redirect it to a file by specifying the redirection on the command line when you run Xants. If you are locked onto an xant, the view in the animation script will follow that xant through the world. To stop recording, select the option again, or select Quit on the file menu to leave the Xants program. Unlike the 2D view, trails and pheromones are not visible--the Animate program does not allow the dynamic creation of objects.
Rendering the Animation Script
Once you have created the animation script, you can render it and view the results. To render the animation, you will need copies of the ants.dat file, which contains the 3D description of the xant. Once you have a copy of the ants.dat file, you can view a wireframe preview of the animation by using the command: animate scriptfile where "scriptfile" is the name of the animation script file. To see faster preview, use the -bbox option after "animate" to display a bounding-box preview (Figure 5-4). Normally, the camera angle for Xants animations is set to a high altitude, approximating the view from the 2D interface. We wrote a
71
Figure 5-4. Sample Bounding Box preview image of an xant. Camera position was modified from normal view. program called "director" that allows you to edit the camera angles of a given script file, allowing you to zoom in or out, lock onto, and "truck" (move the camera in sync to the xant). The Animate program will render flat, gourand, or Phong shaded image files, stored in Utah Raster Toolkit RLE format. By converting these images to PPM format, you can use the Berkeley MPEG encoder program to create MPEG animations from Xants scripts. Be warned that this does
72
Figure 5-5. Sample 3D rendered image of an xant.
require a large amount of disk space to store the images. The resulting output is quite good, however. Figure 5-5 shows a Phong-shaded rendered image of an xant.
CHAPTER 6 VALIDATION OF THE SIMULATION
Validating the Simulation of Nonassociative Learning
We ran a series of experiments to test various aspects of the habituation, dishabituation, and sensitization routines. All of these experiments dealt with the alarm response, which is triggered by some Stimulus Onset messages. Currently, Stimulus Onset messages are the only messages subject to nonassociative learning. The magnitude of the alarm response is the same as the magnitude of the stimulus--a linear relationship between stimulus intensity and response is common in many different species. Habituation will diminish this response, however. In this section, we compare results generated in my Xants experiments with the features of habituation in Thompson and Spencer (1966) that were described earlier, pointing out which aspects of their description are relevant. Since we were unable to find quantitative studies of ant alarm habituation, we use data collected on the sea squirt, Aplysia, for part of the validation in this section. Since, as we have seen, the characteristics of 73
74 nonassociative learning are common throughout vertebrate and invertebrate life, this should not make any important difference, although the exact timing and change of response in real Solenopsis invicta will of course be different from the values shown here.
Experiment I: Simple Habituation, Spontaneous Recovery, Potentiation of Habituation
In this experiment, we exposed an xant to a 50% intensity stimulus every 10 ticks until the response was completely habituated. We then repeated trials at 2000 ticks and 4000 ticks, re-exposing the xant to the stimulus at the same intensity until the response habituated again. Results are presented in Figure 7-1. Spontaneous recovery is evident, as is the increased rate of habituation (potentiation of habituation) in the 3rd trial--note the lowered 2nd data point as compared to the 2nd.
Figure 7-1. Experiment I--Habituation, Spontaneous Recovery, and Potentiation of Habituation
75
Figure 7-2. Habituation and Spontaneous Recovery in Aplysia. From Kandel (1979). As a point of comparison, Figure 7-2 displays a graph of the siphon withdrawal reflex of the sea squirt, Aplysia from Kandel (1979). This experiment, like mine, habituated a stimuli and then repeated trials after a interval to allow for spontanous recovery. The first three trials in that experiment are quite similar to those for my xant. This experiment validates my model for features 1-3 in the Thompson-Spencer description.
Experiment II: Role of Inter-Stimulus-Interval (ISI) in Speed of Habituation
In this experiment, we exposed an xant to a 50% intensity stimulus with an ISI of 20 ticks. We then compared the results with the first trial of experiment I. The results are in Figure 7-3. There is a noticable difference in the speed of habituation given the different ISIs. This result conforms to characteristic 4 of the Thompson-Spencer description.
76
Figure 7-3. Experiment II--Effect of Inter-Stimulus-Interval on Habituation. Experiment III. Effects of Stimulus Strength on Speed of Habituation
In this experiment, we exposed an xant to an stimulus at a 75% intensity level with a 10 tick ISI until the response was habituated. We then compared the results to the first trial of experiment I, which used a 50% intensity level at the same ISI. The results in Figure 7-4 clearly show the effect of stimulus intensity on speed of habituation. This validates my model for Thompson-Spencer feature 5.
Experiment IV. Effect of Habituation beyond the Zero Response Level
In the first trial of this experiment, a stimulus of intensity 25% was presented with an ISI of 10 ticks until the stimulus produced no alarm response, which required 4 trials. After 500 ticks, the stimulus was represented to observe the spontanous recovery.
77
Figure 7-4. Experiment III--Effects of Stimulus Strength on Speed of Habituation. In the second trial, an experimentally naive xant was exposed at the same intensity and ISI, but the stimulus was presented 8 times. Again, after 500 ticks, the stimulus was re-presented to gauge the amount of spontaneous recovery. The results are presented in Figure 7-5. As you can see, the spontaneous recovery in the second trial was greatly reduced, validating my model with feature 6 of the Thomas-Spencer description.
Figure 7-5. Experiment IV--Effect of Habituation beyond the Zero Response Level.
78 Experiment V. Dishabituation
To observe dishabituation, we first exposed an xant to a 50% intensity bump stimulus every 5 ticks for 4 trials. Then a 100% intensity shock stimulus was presented. Following that, a 50% intensity bump stimulus was re-presented to show the level of dishabituation. The results, given in Figure 7-6, show a clear dishabituation effect. This validates my model against Thompson-Spencer feature 8.
Experiment VI: Generalization of Habituation to Other Stimuli
In experiment VI, we exposed an xant to a 50% intensity bump stimulus at a 10 tick ISI until there was no alarm response. In then exposed the ant to a 50% shock stimulus until there was no response. In Figure 7-7, you can see how the shock stimulus habituated faster than the bump stimulus due to the effects of the bump preexposure. This
Figure 7-6. Experiment V--Dishabituation.
79
Figure 7-7. Experiment VI--Generalization of Habituation to Other Stimuli. validates the model with Thompson-Spencer feature 7.
Conclusions
The model implements all features of habituation described in Thompson and Spencer (1966) except one--the habituation of dishabituation.
Validating the Simulation of Classical Conditioning
To show the effects of classical conditioning, in Experiment VIII we paired a neutral stimulus (Light) with an alarm-triggering stimulus (bump) with a 5 tick ISI. After one presentation, presenting a light alone produced an alarm response. Continued presentation the light without pairing it with the bump, the generated rule is weakened and eventually will have a strength of zero, removing it from the bidding competition, and hence eliminating the response. In Holyoak et al. (1989), the authors reported experimental results due to blocking, overshadowing, conditioned inhibition, and the
80 predictability of rules with this same framework. We have been unable to reproduce those results with my simulation, however, due to some differences in implementation. In conclusion, the implementation of classical conditioning is functional, but lacks features of true classical conditioning.
CHAPTER 7 CONCLUSIONS AND RECOMMENDATIONS FOR FURTHER RESEARCH
Conclusions
This project has shown that the combination of the animat approach with ethology and comparitive psychology can provide a fresh look at an old problem. We also tried to show the value of nonassociative learning in realworld contexts and the usefullness of a bottom-up, incremental approach to artificial intelligence. We have shown the value of empirical validation in AI, ignored by far too many researchers. However, even after these efforts, the xants did not achieve full ant-level functionality, which shows exactly how hard such a modest-seeming goal can be. This says much about the incredible mental feats of very small-brained organisms, as well as the current state of artificial intelligence. We have far to go.
Recommendations for Further Research
A project like this can be expanded on many different axes and in many different ways on each axis. There are several categories of expansion that can be identified.
81
82 Improving the Simulator's Interface and Performance
The interface for this simulator is somewhat crude and the animation not much better. More controls that allowed new types of interaction with the xants' world would be welcome. If the system supported Silicon Graphic's GL graphics language, users could have a 3D real-time window on the Xants world. This would be a lot more visually satisfying the line drawings, and a lot less tedious than the current 3D rendering method, involving multi-megabyte files and many intermediate steps. Given the computational demands of the simulation, parallelization would be quite an improvement. There are several different ways to repartition this project across processors--one processor per xant, or even one processor per rule per xant. Most of the rule evaluation and bidding could easily occur in parallel, and certainly the xants could work in parallel. You would not necessarily have to keep the xants in sync from tick to tick, although this might be desirable to make the input/output of the simulator simpler.
Improving the Accuracy of the Xant Model
It would be interesting to see if the behavioral repetoire could be expanded to be a close match to that of real ants. Some of the other behaviors might require a better method of object location and identification
83 than currently supported (identifying conspecifics, picking up and moving food objects). Right now, the visual system is not modeled at all--it is merely assumed to provide the navigational data required. To develop a good model of this system from the eye up would be instructive, especially if it were based on models of insect or mammalian vision (perhaps the silicon retina described in Mahowald and Mead (1991) might be a good jumping-off point). It would not need fool-proof detectors to be realistic, since, as we have seen, Nature's detectors are often less than fool-proof. There are several big steps between eye and object recognition, however, so this would be a rather complicated undertaking. Due to the diffraction problems in compound eyes, the visual system of ants has a much lower resolution than that of mammals, so perhaps that makes simulating their visual system simpler. Improving the sensory capabilities in other areas would be welcome. The collision clock, while allowing simple interpretation by the rules, does not give very detailed information. The xant, in addition, has no "conscious" knowledge of the position of his limbs or antennae, or which limbs are in contact with the ground at a given moment. The odor discrimination could be taken to a much higher level, allowing taste aversion learning for some types of foods but not others.
84 Expanding the Xant's Mental Capabilities
Improving the modeling of classical conditioning and perfecting the model of nonassociative learning should be the first order of business. Once those encompass all the aspects of real animal's behavior, we can set our sights higher. Currently, the simulation does not pay much attention to the context of events--the surrounding environment in which an event occurs. Animals do take context into account when they learn a behavior--a tone in a testing box may be given a different meaning than a tone in their home cages. Handling locations is very crude, currently (an xant is either in the nest or out of it) so that would have to be enhanced as well. The simulation currently does not pay much attention to motivation, aside from the hunger variable. There could be motivations for behaviors like foraging and grooming. These motivations could gradually build up over time with the strength of the each motivation posted to the message list each tick. Rules would have these motivation tags in their conditions
and would fire when the motivation was sufficiently high. Hopefully, these same rules would cause effector actions that would decrease the level of motivation, just as the hunger value is decreased by eating. Another obvious step toward improving the xants' brains would be to allow for operant conditioning. A connection could be made between a specific sequence of rules firings (and hence, effector actions) and a reward
85 that occurred at the end of such a sequence, with the rules in the sequence strengthened and chained together. This could involve a form of a bucketbrigade algorithm for strength revision. The last rule in the chain would be rewarded first, and the reward it would propagate back in progressively smaller amounts to previous rules. This type of learning corresponds nicely to the way rats learn a maze--they learn the last turn to the end goal first, then the next-to-last, and so on until the entire maze is learned. The massive volume of rat studies could provide a source of empirical validation for this aspect of the model. Observational learning, the "next level" of learning, could also be added, but it would require both recognition of conspecifics and the ability to observe how those conspecifics behave with rules that were able to evaluate the current status of conspecifics--whether they were rewarded or punished for their behavior. This recognition could be simulated, of course-merely have conspecific messages posted with position and actions encoded-but to do it properly modeling the visual system would be a prerequisite. Adding this ability would allow a group of xants a form of culture; a discovery made by one xant could spread throughout a group, even surviving the death of the originator. Insight learning would require much more improvement still. The organism would need to be able to generate and test possible courses of action without actually performing them, using its acquired knowledge of
86 the real world to gauge how effective these actions would be. To do this effectively, very good mental models would be necessary.
Improving the Simulator's Mental Models
Currently, the type and degree of learning in the system is hardcoded for both timing and which message categories may be affected by learning. These constraints could be varied for each rule or collection of rules by attaching time values on the rules for associations formed (longer for taste-aversion learning, for instance) and putting meta-rules that would control rule generation in the brain file. In regards to this type of work, we have taken the classifier-system bit-string framework about as far as it can go. Coding everything as binary
strings made rule evaluation quite fast, but did not enhance the learning techniques used, since they are quite non-binary. In addition, this approach limited the kind of data entering the simulation. Perhaps a different implementation, one based on objects inherit characteristics via their class membership would be better. Each characteristic of an object (size, color, shape, et. cetera) could have a number of distinct values associated with it, allowing for fine discrimination between objects. A rule could be a special container class with objects as conditions, actions, and predictions. Perhaps rule elements could be special containers too--a container for a sequence of objects presented in a certain order, with expected time delays, which could
87 handle complex associations and even operant conditioning. The system would still do the habituation and rule-generation much as it does, but over a more powerful framework. If the organism was capable of creating new classes of objects as experience suggested them, the system might be capable of learning whole new categories of knowledge. Needless to say, implementing such a system could be quite challenging. It would be interesting to modify the neural-network paradigm to support this kind of simulation. Traditional neural-nets do not habituate to stimuli and their style of learning is different, but it would certainly be an empirical validation of the neural-net approach if the behaviors exhibited by such a network were similar to those of real organisms.
Moving the Xants into the Real World
It would be interesting to build a six-legged robot with enough onboard computing and sensory equipment to allow running the xants system on real hardware. One would need to change some of the sensory modalities, unless one wanted to put chemical receptors on the xants-following a chemical trail has been used for some office mail-delivery robots in the past. In Beer et al. (1991), a cockroach simulation that ran on a workstation was successfully ported to a hexapod robot, so this sort of conversion is possible.
88 We view this project as a small step to a lofty goal--the simulation of human-level intelligence. We hope that we or others may use this work, its techniques, and its approaches as a stepping stone to simulating higher levels of intelligence.
REFERENCE LIST
Abramson, C. I. (1990). Invertebrate Learning: A Laboratory Manual and Source Book. Washington, DC: American Psychological Association. Beer, R. D., Chiel, H. J., and Sterling, L. S (1991). An artificial insect. American Scientist, 79(5), 444-452. Biederman, G. B., Robertson, H. A., and Vanayan, M. T. I. (1986). Observational learning of two visual discriminations by pigeons: a within-subjects design. Journal of the Experimental Analysis of Behavior, 46(1), 45-49. Bitterman, M. E. (1986). Vertebrate-invertebrate comparisons. In H. J. Jerison and I. Jerison, Editors, Intelligence and Evolutionary Biology, 251-276. Berlin: Springer-Verlag. Collins, R. L. (1988) Observational learning of a left-right behavioral asymmetry in mice (Mus musculus). Journal of Comparative Psychology, 102(3), 222-224. Domjan, M. (1983). Biological constraints and the pursuit of general theories of learning. In R. L. Mellgren, Editor, Animal Cognition and Behavior, 319-344. Amsterdam: North-Holland. Fiorito, G. and Scotto, P.T.I. (1992). Observational learning in Octopus vulgaris. Science, 256(5056), 545-547. Goldberg, D. E. (1989). Genetic Algorithms in Search, Optimization, and Machine Learning. Reading, MA: Addison-Welsey. Gould, J. L. (1982). Ethology: The Mechanisms and Evolution of Behavior. New York, NY: W. W. Norton and Company Holland, J. H., Holyoak, K. J., Nisbett, R. E. and Thagard, P. R. (1987). Induction: Processes of Inference, Learning, and Discovery. Cambridge, MA: MIT Press.
89
90 Hlldobler, B. and Wilson, E. O. (1990). The Ants. Cambridge, MA: Harvard University Press. Holyoak, K. J., Koh, K. and Nisbett, R. E. (1989). A theory of conditioning: inductive learning within rule-based default hierarchies. Psychological Review, 96(2), 315-340. Kandel, E. R. (1979). Behavioral Biology of Aplysia. San Francisco, CA: W. H. Freeman and Company. Klein, S. B. and Mowrer, R. R., Editors (1989). Contemporary Learning Theories: Pavlovian Conditioning and the Status of Traditional Learning Theory. Hillsdale, NJ: Lawrence Erlbaum Associates. Langton, C. G., Editor (1989). Artificial Life. Redwood City, CA: AddisonWesley. Langton, C. G., Taylor, C., Farm, J.D. and Rasmussen, D., Editors (1992). Artificial Life II. Redwood City, CA: Addison-Wesley. Liebert, R. M., and Spiegler, M. D. (1970). Personality: An Introduction to Theory and Research. Homewood, IL: Dorsey. Maes, P., Editor (1990). Designing Autonomous Agents: Theory and Practice from Biology to Engineering and Back. Cambridge, MA: MIT Press. Mahowald, M. A., and Mead, C. (1991). The silicon retina. Scientific American, 264(5), 76-82. Meyer, J-A, and Wilson, S. W., Editors (1991). From Animals to Animats: Proceedings of the First International Conference on Simulation of Adaptive Behavior. Cambridge, MA: MIT Press. Mirenda, J. T., and Vinson, S. B. (1981). Division of labour and specification of castes in the red imported fire ant Solenopsis invicta buren. Animal Behavior 29(2), 410-420.
91 Riolo, R. L. (1990). Lookahead planning and latent learning in a classifier system. In J-A Meyer and S. W. Wilson, Editors, From Animals to Animats: Proceedings of the First International Conference on Simulation of Adaptive Behavior, 316-326. Cambridge, MA: MIT Press. Sutton, R .S. (1990). Reinforcement learning architectures for animats. In J-A Meyer and S. W. Wilson, Editors, From Animals to Animats: Proceedings of the First International Conference on Simulation of Adaptive Behavior, 288-296. Cambridge, MA: MIT Press. Thompson, R. F., and Spencer, W. A. (1966) Habituation: a model phenomenon for the study of neuronal substrates of behavior. Psychological Review, 73(1), 16-43. Thorpe, W. H. (1963). Learning and Instinct in Animals. Cambridge, MA: Harvard University Press. Young, P., Editor (1993). A DNA structure that tags genetic junk? Science News, 144(2), 121.
BIOGRAPHICAL SKETCH
The author was born in Lakeland, Florida in 1963. He received a bachelor's degree in computer science from the University of Florida in 1986. Currently, he is employed by the University of Florida's Animal Science Department as a senior computer programmer/analyst.
92
I certify that I have read this study and that in my opinion it conforms to acceptable standards of scholarly presentation and is fully adequate, in scope and quality, as a thesis for the degree of Master of Science.
Paul A. Fishwick, Chairman Associate Professor of Computer and Information Sciences I certify that I have read this study and that in my opinion it conforms to acceptable standards of scholarly presentation and is fully adequate, in scope and quality, as a thesis for the degree of Master of Science.
Joseph N. Wilson Assistant Professor of Computer and Information Sciences I certify that I have read this study and that in my opinion it conforms to acceptable standards of scholarly presentation and is fully adequate, in scope and quality, as a thesis for the degree of Master of Science.
Douglas D. Dankel II Assistant Professor of Computer and Information Sciences This thesis was submitted to the Graduate Faculty of the Department of Computer and Information Sciences in the College of Liberal Arts and Sciences and to the Graduate School and was accepted as partial fulfillment of the requirements for the degree of Master of Science. April 1994 Dean, Graduate School