of Distribution Strategies in a Spatial Multi-Agent Game
Frank Schweitzer
Chair of Systems Design, ETH Zurich Kreuzplatz 5, 8032 Zurich, Switzerland
Robert Mach
Chair of Systems Design, ETH Zurich Kreuzplatz 5 8032 Zurich, Switzerland
[e-mail]
[e-mail]
ABSTRACT
We investigate the adaptation of cooperating strategies in an iterated Prisoners Dilemma (IPD) game. The deterministic IPD describes the interaction of N agents spatially distributed on a lattice, which are assumed to only interact with their four neighbors, hence, local congurations are of great importance. Particular interest is in the spatialtemporal distributions of agents playing dierent strategies, and their dependence on the number of consecutive encounters ng during each generation. We show that above a critical ng , there is no coexistence between agents playing different strategies, while below the critical ng coexistence is found.
1.
INTRODUCTION
When an agent chooses its strategy from a xed set of possible ones, its decision has to take several factors into account: (i) locality, i.e. the subset of those agents, the respective agent most likely interacts with, (ii) heterogeneity, i.e. the possible strategies of these agents, (iii) time horizon, i.e. the number of foreseen interactions with these agents. The best strategy for a rational agent would be the one with the highest payo for a given situation. To reach this goal, however, can be dicult for two reasons: uncertainty, i.e. particular realizations are usually subject to random disturbances and can therefore be known only with a certain probability, and co-evolution, i.e. the agent does not operate in a stationary environment, but in an ever-changing one, where its own action inuences the decisions of other agents and vice-versa. To reduce the risk of making the wrong decision, it often seems to be appropriate just to copy the successful strategies of others [16, 12]. Such an imitation behavior is widely found in biology, but also in cultural evolution. A similar kind of local imitation behavior will be used in this paper to explain the spatial dispersion of strategies in a multi-agent system. In order to address the problem systematically, we apply a well dened game-theoretic problem. Game theory embod-
ies many of the features mentioned above and thus has often been employed in the design of mechanisms and protocols for interaction in multi-agent systems [17, 21]. Many investigations on the relation between game theory and multiagent systems focus on mathematical investigations [3], and the learning and coordination of tasks [19, 6, 20, 18]. In contrast our paper mainly deals with the dynamics of strategy distribution. We consider a system of N agents spatially distributed on a square lattice, so that each lattice cite is occupied by just one agent. Each agent i is characterized by two state variables, (i) its position r i on the lattice, and (ii) a discrete variable i , describing its possible actions, as specied in Sect. 3. Agents are assumed to directly interact only with their 4 nearest neighbors a number of ng times. In order to describe the local interation, we use the so-called iterated Prisoners Dilemma (IPD) game a paradigmatic example [1, 15] well established in evolutionary game theory with a broad range of applications in economics, political science, and biology. In the simple Prisoners Dilemma (PD) game, each agent i has two options to act in a given situation, to cooperate (C), or to defect (D). Playing with agent j, the outcome of this interaction depends on the action chosen by agent i, i.e. C or D, without knowing the action chosen by the other agent participating in a particular game. This outcome is described by a payo matrix, which for the 2-person game, i.e. for the interaction of only two agents, has the following form: C D C R T D S P (1)
In PD games, the payos have to fulll the following two inequalities: T >R>P >S 2R > S +T (2)
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for prot or commercial advantage and that copies bear this notice and the full citation on the rst page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specic permission and/or a fee. AAMAS06 May 812 2006, Hakodate, Hokkaido, Japan. Copyright 2006 ACM 1-59593-303-4/06/0005 ...$5.00.
In this paper, we use the known standard values T = 5, R = 3, P = 1, S = 0. This means in a cooperating environment, a defector will get the highest payo. From this, the abbreviations for the dierent payos become clear: T means (T)emptation payo for defecting in a cooperative environment, S means (S)ucker payo for cooperating in a defecting environment, R means (R)eward payo for cooperating in a likewise environment, and P means (P)unishment payo for defecting in a likewise environment. In any one round (or one-shot) game, choosing action D is unbeatable, because it rewards the higher payo for agent i whether the opponent chooses C or D. At the same
time, the payo for both agents i and j is maximized when both cooperate. But in a consecutive game played many times, both agents, by simply choosing D, would end up earning less than they would earn by collaborating. Thus, the number of games ng two agents play together becomes important. For ng 2, this is called an iterated Prisoners Dilemma (IPD). It makes sense only if the agents can remember the previous choices of their opponents, i.e. if they have a memory of nm ng 1 steps. Then, they are able to develop dierent strategies based on their past experiences with their opponents, which is described in the following.
2.
AGENTS STRATEGIES
In this paper, we assume only a one-step memory of the agent, nm = 1. Based on the known previous choice of its opponent, either C or D, agent i has then the choice between eight dierent strategies. Following a notation introduced by Nowak [14], these strategies are coded in a 3-bit binary string [Io |Ic Id ] which always refers to collaboration. The rst bit represents the initial choice of agent i: it is 1 if agent i collaborates, and 0 if it defects initially. The two other values refer always to the previous choice of agent j. Ic is set to 1 if agent i chooses to collaborate given that agent j has collaborated before and 0 otherwise. Id is similarily set to 1 if agent i chooses to collaborate given that agent j has defected before and 0 otherwise. For the deterministic case discussed in this paper, the eight possible strategies (s = 0, 1, . . . , 7) are given in Tab. 1. s 0 1 2 3 4 5 6 7 Strategy suspicious defect suspicious anti-Tit-For-Tat suspicious Tit-For-Tat suspicious cooperate generous defect generous anti-Tit-For-Tat generous Tit-For-Tat generous cooperate Acronym sD sATFT sTFT sC gD gATFT gTFT gC Bit String 000 001 010 011 100 101 110 111
spatial interactions, where agents simultaneously encounter with 4 dierent neighbors. Agents playing strategy gATFT (s = 5) initially also start with cooperation and then do the opposite of whatever the opponent did in the previous move, while agents playing strategy sATFT (s = 1) behave the same way, except for the rst move where they defect. Agents playing strategy sTFT (s = 2) also start with defection, but then imitate the previous move of the opponent, as in gTFT. A closer look at the encounters reveals that sTFT and gTFT will exploit each other alternatively while gTFT will mutually cooperate. Also, sATFT and gATFT exploit each other alternatively, while sATFT will alternatively cooperate. This illustrates that the rst move of a strategy can be vital to the outcome of the game. The number of interactions ng is also a crucial parameter in this game, because, if ng is even, gTFT and sTFT will gain the same, but in case of ng being odd sTFT will gain more than gTFT. Eventually, it can be argued that some of the strategies do not make sense from a human point of view. In particular sATFT or gATFT seem to be lunatic or paranoid strategies. Therefore, let us make two points clear: Such arguments basically reect the intentions, not to say the preconceptions, of a human beholder. We try to avoid such arguments as much as possible. In our model, we consider a strategy space of 23 = 8 possible strategies, and there is no methodological reason to exclude a priori some of these strategies. If they do not make sense, then this will be certainly shown by the evolutionary game theoretic approach used in our model. I.e., those strategies will disappear in no time, but not because of our private opinion, but because of a selection dynamics that has proven its usefulness in biological evolution. So, there is no reason to care too much about a few paranoid strategies. On the other hand, biological evolution has also shown that sometimes very unlikely strategies get a certain chance under specic (local?) conditions. In a complex system, it would be a priori not possible to predict the outcome of a particular evolutionary scenario, simply because of the path-dependence. We come back to this point in our conclusions, where we shortly discuss that in the case of eight strategies and ng = 2 also unpredicted strategies survive.
Table 1: Possible agents strategies using a one-step memory. Depending on the agents rst move, we can distinguish between two dierent classes of strategies: (i) suspicious (s = 0, 1, 2, 3), i.e. the agent initially defects, and (ii) generous (s = 4, 5, 6, 7), i.e. the agent initially cooperates. Further, we note that four of the possible strategies do not pay attention on the opponents previous action, i.e. except for the rst move, the agent continues to act in the same way, therefore the strategies sD, sC, gD, gC (s = 0, 3, 4, 7) can be also named rigid strategies. The more interesting strategies are s = 1, 2, 5, 6. Strategy s = 6, known as (generous) tit for tat (TFT), means that agent i initially collaborates and continues to do so, given that agent j was also collaborative in the previous move. However if agent j was defective in the previous move, agent i chooses to be defective, too. This strategy was shown to be the most successful one in iterated Prisoners Dilemma games with 2 persons [1]. Here, however, we are interested in
3. SPATIAL INTERACTION
So far, we have explained the interaction of two agents with a one-step memory. This shall be put now into the perspective of a spatial game with local interaction among the agents. A spatially extended (non-iterative) PD game was rst proposed by Axelrod [1]. Based on these investigations, Nowak and May simulated a spatial PD game on a cellular automaton and found a complex spatiotemporal dynamics [13, 14]. A recent mathematical analysis [2] revealed the critical conditions for the spatial coexistence of cooperators and defectors with either a majority of cooperators in large spatial domains, or a minority of ooperators in small (non-stationary) clusters. In the following, we concentrate on the iterated PD game, where the number of encounters, ng , plays an important role. We note that possible extensions of the IPD model have been investigated e.g. by Lindgren and Nordahl [7], who introduced players which act erroneously sometimes, allowing a complex evolution of strategies in an unbounded strategy space.
i1
i2 i i4
i3
completed, i can be changed based on a comparison of the payos received. I.e., payo ai is compared to the payos aij of all neighboring agents, in order to nd the maximum payo within the local neighborhood during that generation, max {ai , aij }. If agent i has received the highest payo, then it will keep its i , i.e. it will continue to play its strategy. But if one of its neighbors j has received the highest payo, then agent i will adopt or imitate, respectively, the strategy of the respective agent. If j = arg maxj=0,...,n1 aij (5)
Figure 1: Local neigbhorhood of agent i. The nearest neighbors are labeled by a second index j = 1, ..., 4. Note that j = 0 refers to the agent in the center. In the spatial game, we have to consider local congurations of agents playing dierent strategies (see Fig.1). As explained in the beginning, each agent i shall interact only with its four nearest neighbors. Let us dene the size of a neighborhood by n (that also includes agent i), then the dierent neighbors of i are characterized by a second index j = 1, ..., n 1. The mutual interaction between them results in a n-person game, i.e. n = 5 agents interact simultaneously. In this paper, we use the common assumption that the 5-person game is decomposed into (n 1) 2-person games, that may occur independently, but simultaneously [7, 8, 15], a possible investigation of a true 5-person PD game is also given in [15]. We further specify the i that characterize the possible actions of each agent as one of the strategies that could be played (Tab. 1), i.e. i s = {0, 1, . . . , 7}. The total number of agents playing strategy s in the neighborhood of agent i is given by:
s ki = n1 X j=1
denes the position of the agent that received the highest payo in the neighborhood, the update rule of the game can be concluded as follows: i (G + 1) = ij (G) (6)
We note that the evolution of the system described by eq. (6) is completely deterministic, results for stochastic CA have been discussed in [4]. The adaptation process leads to an evolution of the spatial distribution of strategies that will be investigated by means of computer simulations on a cellular automaton in the following section.
4. EVOLUTION OF SPATIAL PATTERNS OF 3 STRATEGIES
In order to illustrate the spatio-temporal evolution we have restricted the computer simulation here to only three strategies. The more complex (and less concise) case of eight strategies is discussed in detail in [11, 9, 10]. The three strategies were chosen as sD, sATFT and gTFT (s = 0, 1, 6) for the following reason. Strategy sD is known to be the winning strategy for the one-shot game, i.e. ng = 1, while is gTFT known to be the most successful strategy for ng 4. We are interested in the transition region, 1 < ng < 4, thus we include those two strategies in our simulation and xed ng to values of 2 or3. Agents playing sATFT are also added to the initial population, since they behave anti-cyclic, i.e. they defect when the opponent cooperated in the previous encounter and vice versa. The apparent solution to describe the dynamics of the system by an dynamical system approach works only for the so-called mean-eld case, which can be simulated by a random interaction. I.e., each agent interacts with four randomly chosen agents during each generation. In this case the dynamics can well be described by a selection equation of the Fisher-Eigen type [9]. The random interaction is also used as a reference case, to point out diences to the case of local interaction described in the following. The simulation are carried out on a 100 100 lattice with periodic boundary conditions, in order to eliminate spatial artifacts at the edges. Initially, all agents are randomly assigned one of the three strategies. Dening the total fraction of agents playing strategy s at generation G P as fs (G) = 1/N N i s , f0 (0) = f1 (0) = f6 (0) = 1/3 i=1 holds for G = 0 (see also the rst snapshot of Fig. 2). Because each agent encounters with his 4 nearest neighbors ng times during one generation, in each generation (N/2 ng 4) indepentent and simultaneous deterministic 2-person games occur. Fig. 2 shows snapshots of the spatiotemporal distribution of the three stategies for ng = 2, while Fig. 3 shows snapshots with the same setting, but
s i j
(3)
where xy means the Kronecker delta, which is 1 only for x = 0 1 2 7 y and zero otherwise. The vector k i = {ki , ki , ki , . . . , ki } then describes the occupation numbers of the dierent strategies in the neighborhood of agent i playing strategy i . Agent i encounters with each of its four neighbors playing strategy ij in independent 2-person games from which it receives a payo denoted by ai ij , which can be calculated with respect to the payo matrix, eq. (1). The total payo of an agent i after these indepentent games is then simply ai (i ) =
n1 X j=1
a i i j =
X
s
s a i s k i
(4)
We note again that the payos ai s also strongly depend on the number of encounters, ng , for which explicit expressions have been derived. They are concluded in a 8 8 payo matrix not printed here [11]. In order to introduce a time scale, we dene a generation G to be the time in which each agent has interacted with its n 1 nearest neighbors ng times. During each generation, the strategy i of an agent is not changed while it interacts with its neighbors simultaneously. But after a generation is
for ng = 3.
1 0.8 0.6 sD sATFT gTFT
1 0.8 0.6
sD sATFT gTFT
f
0.4 0.2 0
f
0.4 0.2 0 0
G=0
G=1
G=2
0
50
100
150
5
10
15
20
G
G
Figure 4: Global frequencies fs (G) of the three strategies for ng = 2 (left) and ng = 3 (right). For the spatial distribution, see Fig. 2 and Fig. 3, respectively. G=4 G=22 G=150 the fact that the total number of encounters during one generation has increased. The main reason is that for ng = 2 agents playing sATFT are able to locally block the spreading of strategy gTFT, while this is not the case for ng = 3. This is because both of the ng dependence of the agents payo and the local conguration of players: for ng = 2, there is only one local conguration where strategy gTFT can invade sATFT, because of the higher payo. After this invasion, however, the preconditions for further invasion have vanished. For ng = 3, this situation is dierent in that there are more local congurations, where gTFT can invade sATFT. This in turn enables the further takeover. The crossover dynamics mentioned in conjunction with Fig. 4 can be explained in a similar manner. For ng = 2 gTFT can not spread initially because of agents playing sATFT. Only sD is able to invade sATFT and gTFT, therefore its frequency increases. Once sATFT is removed, gTFT can spread [11].
Figure 2: Spatial-temporal distribution of three strategies sD (black), sATFT (white), and gTFT (gray) on a 100 100 grid for ng = 2.
G=1
G=2
G=4
G=11
5. GLOBAL PAYOFF DYNAMICS
The adaptation of strategies by the agents is governed by the ultimate goal of reaching a higher individual payo. As we know from economic applications, however, the maximization of the private utility does not necessarily mean a maximization of the overall utility. So, it is of interest to investigate also the global payo and the dynamics of the payos of the individual strategies [5]. The average payo per agent a is dened as: P N X 1 X i ai (i )i s P ai (i ) = fs (G) as ; as = a= N i=1 i i s s (7) where fs (G) is the total fraction of agents playing strategy s and as is the average payo per strategy, shown in Fig. 5 for the dierent strategies.
3.5 3 2.5 2 3.5 3 2.5 2
sD sATFT gTFT
Figure 3: Spatial-temporal distribution of three strategies sD (black), sATFT (white), and gTFT (grey) on a 100 100 grid for ng = 3. The comparison with Fig. 2 elucidates the inuence of ng . For ng = 2, we see from Fig. 2 that in the very beginning, i.e. in the rst four generations, strategy sD grows very fast on the expense of sATFT and especially on gTFT. This can be also conrmed when looking at the global frequencies of each strategy (see left part of Fig. 4). Already for G=4, strategy sD is now the majority of the population only a few agents playing gTFT and even fewer agents playing sATFT are left in some small clusters. Hence, for the next generation we would assume that the sD will take over the whole population. Interestingly, this is not the case. Instead, the global frequency of sD goes down while the frequency of gTFT starts to increase continuously until it reaches the majority. Only the frequency of sATFT stays at its very low value. On the spatial scale, this evolution is accompanied with a growth of domains of gTFT that are nally separated by only thin borders of agents playing sD (cf Fig. 2 for G = 150). The reasons for this kind of crossover dynamics will be explained later. When increasing the number of encounters ng from 2 to 3, we observe that the takeover of gTFT occurs much faster. Already for G = 13, it leads to a situation where all agents play gTFT, with no other strategy left. Hence, they will mutually cooperate. The fast takeover is only partly due to
as
1.5 1 0.5 0 0 50 100 150
sD sATFT gTFT
as
1.5 1 0.5 0 0 5 10 15
20
G
G
Figure 5: Average payo per strategy, as , eq. (7), vs. time for ng = 2 (left) and ng = 3 (right) We note that the payos per strategy for the 2-person
games are always xed dependent on ng . However, the average payo per strategy changes in the course of time mainly because the local congurations of agents playing dierent strategies change. For ng = 2, we have the stable coexistence of all three strategies (cf Fig. 2 and Fig. 4left), while for ng = 3 only strategy gTFT survives (cf Fig. 3 and Fig. 4right). Hence, in the latter case we nd that the average payo of gTFT reaches a higher value than for ng = 2, while in Fig. 5 the corresponding curves for the other strategies simply end, if one of these strategies vanishes. Eventually, the average global payo is shown in Fig. 6 for dierent values of ng . Obviously, the greater ng , the faster the convergence towards a stationary global value, which is a = 3 only in the ideal case of complete cooperation. As we have already noticed, for ng = 2 there is a small number of defecting agents playing either sD or sATFT left, therefore the average global payo is lower in this case.
3.5 3 2.5 2
1.5 1 0.5 0 0 50 100
ng=9 ng=3 ng=2
(gD, sTFT) or three strategies (sD,gD,gTFT) (given in order of frequency) in the nal state. In particular, the most known strategy gTFT will usually become extinct, which is certainly dierent from the expected behavior. A second point to be mentioned, we nd for dierent runs with the same initial conditions dierent outcomes of the simulations. Hence, random deviations may lead the global dynamics to dierent attractors. Thus, local eects seem to be of great inuence for the nal outcome. This proves our point made at the end of Sect. 2, that path dependence plays an important role in the dynamics and the evolutionary game cannot be completely predicted. A more through analysis of the game three strategy game is given in [11], whereas [9, 10] concentr...