Nash Convergence of Gradient Dynamics in General-Sum Games. Michael Kearns.

Transcription

1 Convergence of Gradient Dynamics in General-Sm Games Satinder Singh AT&T Labs Florham Park, NJ 7932 bavejaresearch.att.com Michael Kearns AT&T Labs Florham Park, NJ 7932 mkearnsresearch.att.com Yishay Mansor Tel Aviv University Tel Aviv, Israel mansormath.ta.ac.il Abstract Mlti-agent games are becoming an increasingly prevalent formalism for the stdy of electronic commerce and actions. The speed at which transactions can take place and the growing complexity of electronic marketplaces makes the stdy of comptationally simple agents an appealing direction. In this work, we analyze the behavior of agents that incrementally adapt their strategy throgh gradient ascent on expected payo, in the simple setting of two-player, two-action, iterated general-sm games, and present a srprising reslt. We show that either the agents will converge to a eilibrim, or if the strategies themselves do not converge, then their average payos will nevertheless converge to the payos of a eilibrim. Introdction It is widely expected that in the near ftre, software agents will act on behalf of hmans in many electronic marketplaces based on actions, barter, and other forms of trading. This makes mlti-agent game theory (Owen, 995) increasingly relevant to the emerging electronic economy. There are many dierent formalisms within game theory that model interaction between competing agents. Or interest is in iterated games, which model sitations where a set of agents or players repeatedly interact with each other in the same game. There is a long and illstrios history of research in iterated games, sch as the stdy of (and even competitions in) solving iterated prisoner's dilemma (Owen, 995). Of particlar interest in iterated games is the possibility of the players adapting their strategy based on the history of interaction with the other players. Many dierent algorithms for adaptive play in iterated games have been proposed and analyzed. For example, in ctitios play, each player maintains a model of the mixed strategy of the other players based on the empirical play so far, and always plays the best response to this model at each iteration (Owen, 995). While it is known that the time averages of the strategies played form a eilibrim, the strategies themselves do not converge to, nor are the averaged payos to the players garanteed to be. Kalai and Lehrer (993) proposed a Bayesian strategy for players in a repeated game that reires the players to have \informed priors", and showed that nder this condition play converges to a eilibrim. A series of recent reslts has shown that the informed prior condition is actally ite restrictive, limiting the applicability of this reslt. These seminal reslts implicitly assme that nbonded comptation is allowed at each step. In contrast, we envision a ftre in which agents may maintain complex parametric representations of either their own strategy or their opponents, and in which fll Bayesian pdating or comptation of best responses is comptationally intractable. In other words, as in the rest of articial intelligence and machine learning, in order to eciently act in a complex environment, agents will adopt both representational and comptational restrictions on their behavior in iterated games and other settings (e.g., Papadimitro and Yannakakis (994) and Frend et al. (995)). Perhaps the most common type of algorithm within machine learning are those that proceed by gradient ascent or descent (or other local methods) on some appropriate objective fnction. In this paper we stdy the behavior of players adapting by gradient ascent in expected payo in two-person, two-action, generalsm iterated games. Ths, here we stdy a specic and very simple adaptive strategy in a setting in which a general mixed strategy is easy to represent. Sch a stdy is a prereisite to an nderstanding of gradient methods on rich, parametric strategy representa-

2 tions. While it is known from the game theory literatre that the strategies compted by gradient ascent in two-person iterated games need not converge, we present a new and perhaps srprising reslt here. We prove that althogh the strategies of the two players may not always converge, their average payos always do converge to the expected payos of some eilibrim. Ths, the dynamics of gradient ascent ensre that the average payos to two players adopting this simple strategy is the same as the payo they wold achieve by adopting arbitrarily complex strategies. In the remaining sections, we dene or problem and the gradient ascent algorithm, and show that the behavior of players adapting via gradient ascent can be modeled as an ane dynamical system. Many properties of this dynamical system are known from control theory literatre, and have been applied before to the somewhat dierent setting of evoltionary game theory (Weibll, 997). Or main technical contribtion is a new and detailed geometric analysis of these dynamics in the setting of classical game theory, and particlarly of the eects of the bondary conditions imposed by game theory on those dynamics (in contrast, evoltionary game theory explicitly and articially prevents the dynamics from reaching the bondaries). 2 Problem Denition and Notation A two-player, two-action, general-sm game is dened by a pair of matrices R = r r 2 r 2 r 22 and C = c c 2 c 2 c 22 specifying the payos for the row player (player ) and the colmn player (player 2), respectively. If the row player chooses action i 2 f; 2g and the colmn player chooses action j 2 f; 2g the payo to the row player is r ij and the payo to the colmn player is c ij. Two cases of special interest are that of zero-sm games, in which the payo of the colmn player and the payo of the row player always sms to zero (r ij + c ij = for i; j 2 f; 2g), and that of team games, in which both players always get the same payo (r ij = c ij for i; j 2 f; 2g). The players can choose actions stochastically, in which case they are said to be following a mixed strategy. Let denote the probability of the row player picking action and let denote the probability of the colmn player picking action. Then V r (; ), the vale or expected payo of the strategy pair (; ) to the row player, is V r (; ) = r () + r 22 ((? )(? )) +r 2 ((? )) + r 2 ((? )) () and V c (; ), the vale of the strategy pair (; ) to the colmn player, is V c (; ) = c () + c 22 ((? )(? )) +c 2 ((? )) + c 2 ((? )): (2) The strategy pair (; ) is said to be a eilibrim (or pair) if (i) for any mixed strategy, V r ( ; ) V r (; ), and (ii) for any mixed strategy, V c (; ) V c (; ). In other words, as long as one player plays their half of the pair, the other player has no incentive to change their half of the pair. It is well-known that every game has at least one pair in the space of mixed (bt not necessarily pre) strategies. 3 Gradient Ascent for Iterated Games One can view the strategy pair (; ) as a point in R 2 constrained to lie in the nit sare. The fnctions V r (; ) and V c (; ) then dene two vale srfaces over the nit sare for the row and colmn players respectively. For any given strategy pair, (; ), one can compte a gradient for the row player from the V r -vale srface and for the colmn player from the V c -vale srface as follows. Letting = (r + r 22 )? (r 2 + r 2 ) and let = (c + c 22 )? (c 2 + c 2 )), we have V r (; ) V c (; ) =? (r 22? r 2 ) (3) =? (c 22? c 2 ): (4) In the gradient ascent algorithm, each player repeatedly adjsts their half of the crrent strategy pair in the direction of their crrent gradient with some step size : k+ = k + V r( k ; k ) k+ = k + V c( k ; k ) (5) where ( ; ) is an arbitrary starting strategy pair. Points on the bondary of the nit sare (where at least one of and is zero or one) have to be handled in a special manner, becase the gradient may lead the players to an infeasible point otside the nit sare. Therefore, for points on the bondary for which the gradient points otside the nit sare, we redene the gradient to be the projection of the tre gradient onto the bondary. For ease of exposition, we do not change the notation in Eation 5 to reect the projection of the gradient at the bondary, bt the behavior there shold be nderstood and is important to or analysis. Note that the gradient ascent algorithm assmes a fll information game that is, both players know both

3 game matrices, and can see the mixed strategy of their opponent at the previos step. (However, if only the actal previos move played is visible, we can dene a stochastic gradient ascent algorithm.) 4 Gradient Ascent as Ane Dynamical System If the row and colmn players were to play according to the gradient ascent algorithm of Eation 5, they wold at iteration k play the strategy pair ( k ; k ), and receive expected payos V r ( k ; k ) and V c ( k ; k ) respectively. We are interested in the performance of the two players over time. In particlar, we are interested in what happens to the strategy pair and payo seences over time. It is well-known in game theory that the strategy pair seence prodced by following a gradient ascent algorithm may never converge (Owen, 995). In this paper we prove that the average payo of both players always converges to that of some pair, regardless of whether the strategy pair seence itself converges or not. Note that this also means that if the strategy pair seence does converge, it mst converge to a pair. For the prposes of analysis, it is convenient to rst consider the gradient ascent algorithm for the limiting case of innitesimal step size (lim!); hereafter we will refer to this as the IGA (for Innitesimal Gradient Ascent) algorithm. Sbseently we will show that the asymptotic convergence properties of IGA also hold in the more practical case of gradient ascent with decreasing nite step size. In IGA, the seence of strategy pairs becomes a continos trajectory in the nit sare (thogh there are discontinities at the bondaries of the nit sare becase of the projected gradient). The basic intition behind or analysis comes from viewing the two players behaving according to IGA as a dynamical system in R 2. In particlar, as we show below the dynamics of the strategy pair trajectory is that of an ane dynamical system. This view does not take into accont the constraint that the strategy pair has to lie in the nit sare. This separation between the nconstrained dynamics and the constraints of the nit sare will be sefl throghot the rest of this paper. Using Eations 3,4 and 5 and an innitesimal step size, it is easy to show that the nconstrained dynamics of the strategy pair as a fnction of time is dened by the following dierential eation: t t = +?(r22? r 2 )?(c 22? c 2 ) : (6) We denote the o-diagonal matrix containing the terms and in Eation 6 as U. Figre : The general form of the dynamics: a) when U has imaginary eigenvales and b) when U has real eigenvales. From dynamical systems theory (Reinhard, 987), it is known that if the matrix U is invertible (we handle the non-invertible case separately below), the nconstrained strategy pair trajectories can only take the two possible alitative forms shown in Figre. Notice that these two dynamics are very dierent: the one in Figre a has a limit-cycle behavior, while the one in Figre b is divergent. Now depending on the exact vales of and, the ellipses in Figre a can become narrower or wider, or even reverse the direction of the ow. Similarly, the angle between the dashed axes in Figre b and the direction of ow along the axes will depend on and. Bt these are the two general forms of nconstrained dynamics that are possible. In the next section we dene the characteristics of general-sm game that determine whether the nconstrained dynamics is elliptical or divergent. The center where the axes of the ellipses meet, or where the dashed-axes of the divergent dynamics meet, is the point at which the tre gradient is zero. By setting the left hand side of Eation 6 to zero and solving for the nie center ( ; ), we get: ( ; (c22? c 2 ) ) = ; (r 22? r 2) (7) Note that the center is in general not at (; ), and it may not even be inside the nit sare. 5 Analysis of IGA The following is or main reslt: Theorem ( convergence of IGA in iterated general-sm games) If in a two-person, two-action, iterated general-sm game, both players follow the IGA algorithm, their average payos will converge in the limit to the expected payos for some eilibrim. This will happen in one of two ways: ) the strategy pair trajectory will itself converge to a pair, or 2) the strategy pair trajectory will not converge, bt the average payos of the two players will nevertheless converge to the expected payos of some pair.

4 The proof of this theorem is complex and involves consideration of several special cases, and we present it in some detail below. Bt rst we give some high-level intition as to why the theorem is correct. First observe that if the strategy pair trajectory ever converges, it mst be that it has reached a point with zero gradient (or zero projected gradient if the point is on the bondary of the nit sare). It trns ot that all sch points mst be pairs becase no improvement is possible for either player. More remarkably, it trns ot that the average payo of each ellipse in Figre a is exactly the expected payo of the center (which is a point with zero gradient). Bt how is all this aected by the constraints of the nit sare? Imagine taking a nit sare and placing it anywhere in the plane of Figre a. The projected gradient along the bondary will be determined by which adrant the bondary is in. We show that if there are some ellipses contained entirely in the nit sare, the dynamics will converge to one sch ellipse, and that if no ellipses are contained in the nit sare (the center is otside the nit sare), then the constrained dynamics mst converge to a point. In either case, by the argments above, the average payo will become. Similarly, imagine taking a nit sare and placing it anywhere on the plane in Figre b. Given the gradient direction in each adrant of the plane we show that the dynamics will converge to some corner of the nit sare. Again, the average payo will become. From dynamical systems (Reinhard, 987) it can be shown that we only need to consider three mtally exclsive and exhastive cases to complete a proof:. U is not invertible. This will happen whenever or or both are zero. Sch a case can occr in team, zero-sm, and general-sm games. Examples of the dynamics in sch a case are shown in Figre U is invertible and its eigenvales are prely imaginary. We can compte the eigenvales by solving for in the following eation: x y = x y yielding 2 =. Therefore we will get imaginary eigenvales whenever <. Sch a case can occr in zero-sm and general-sm games bt cannot happen in team games (becase = and therefore ). Two examples of the dynamics are shown in Figre U is invertible and its eigenvales are prely real. This will happen whenever >. Sch a ; case can occr in team and general-sm games bt cannot happen in zero-sm games (becase =? and therefore ). Example dynamics are shown in Figre 6. Theorem is proved below by showing that convergence holds in all three cases smmarized above. Bt before we analyze these three cases in seence in the next three sbsections, we present a basic reslt common to all three cases that shows that if the ((t); (t)) trajectory ever converges to a point, then that point mst be a pair. Lemma 2 (Convergence of strategy pair implies convergence to eilibrim) If, in following IGA, lim t!((t); (t)) = ( c ; c ), then ( c ; c ) is a pair. In other words, if the strategy pair trajectory converges at all, it converges to a pair. Proof: The strategy pair trajectory converges if and only if it reaches a point where the projected gradient is exactly zero. This can happen in two ways: ) the point is the center ( ; ), where by denition the gradient is zero (this can only happen if the center is in the nit sare), or 2) the point is on the bondary and the projected gradient is zero. Either way, it means that from that point no local improvement is possible. For a proof by contradiction, assme that sch a point is not a pair. Then for at least one of the players, say the colmn player, there mst be a nilateral change that increases their payo. Let the improved point be ( c ; i ). Then for all >, ( c ; (? ) c + i ) mst also be an improvement. This follows from the linear dependence of V c (; ) on and the fact that the nit sare is a convex region. Therefore the projected gradient at c ; c mst be non-zero. 2 Corollary 3 If the center ( ; ) is in the nit sare it is a pair. 5. U is not Invertible Lemma 4 ( convergence when U is not invertible) When the matrix U is not invertible, the IGA algorithm leads the strategy pair trajectory to converge to a point on the bondary that is a pair. Proof: First consider the case when exactly one of and is zero. Withot loss of generality assme that =, i.e., (r + r 22 ) = (r 2 + r 2 ). Then the gradient for the row player is constant (see Eation 3) and depending on its sign, the row player will converge to either = or to =. Once the row player's strategy has converged, the gradient of the colmn player will also become constant (see Eation 4) and therefore

5 it too will converge to an extreme vale and therefore the joint strategy will converge to some corner. If both and are zero, then both the gradients are constant and again we get convergence to a corner of the nit sare. In smmary, if U is not invertible the gradient algorithm will lead to convergence to some point on the bondary of the nit sare, and hence from Lemma 2 will lead to convergence to a pair of the game are determined by the real and imaginary parts of the two eigenvectors, that is, by the vectors j j jj and Note that these two vectors, and hence the axes of the ellipses, are always orthogonal to each other and parallel to the axes of the nit sare. In the zerosm case, becase jj = j j, they are also eal in size which means that the dynamics in the zero-sm case are circlar (we merely observe this bt do not se it hereafter). Note that the ellipses are centered at ( ; ) and that the nit sare may be anywhere in R 2 and therefore the center can be otside the nit sare. : D A Figre 2: Example dynamics with U not invertible. a) In this case is zero and is not. b) Both and are zero. Figre 2 shows the dynamics for two general-sm games in which U is not invertible. Figre 2a is for a case where = and >. The gradient for the row player is constant and points downwards. The gradient for the colmn player depends on, bt once converges to zero it point to the right and therefore from all starting points we get convergence to the bottom right corner. Figre 2b is for a case where both and are zero. In this case both the gradients are constant and we get piecewise straight line dynamics. 5.2 U has Prely Imaginary Eigenvales Prely imaginary eigenvales occr when < in p p which case the two eigenvales are jjj ji and? jjj ji. It can be shown that in sch a case the nconstrained dynamics are elliptical arond axes determined by the eigenvectors of U (Reinhard, 987). See Figre 3a) for an illstration. There are two possible cases to consider: ) > and <, and 2) < and >. However, withot loss of generality we can consider only one case. When < and >, j j jj + is a complex eigenvector corresponding to the eigen- p vale jjj ji, and j j? i jj is a complex eigenvector corresponding to the eigenvale? p jjj ji. The axes of the ellipses in Figre 3a i C B Center Figre 3: a) Unconstrained dynamics when U has imaginary eigenvales. b) The constrained dynamics when the center is in the nit sare. In each case only some sample trajectories are shown. We can solve the ane dierential Eation 6 for the nconstrained dynamics of and to get: and (t) = B p cos( p t + ) + (8) p (t) = B sin( p t + ) + (9) where B and are constants dependent on the initial and. These are the eations for the ellipses of Figre 3a. Note that if an ellipse happens to lie entirely inside the nit sare then these eations also describe the constrained dynamics for any starting strategy pair that falls on that ellipse. Lemma 5 ( Average Payo for Ellipse entirely inside nit sare) For any initial strategy pair ( ; ), if the trajectory given by Eations 8 and 9 lies entirely within the nit sare, then the average payos along that trajectory are exactly the expected payos of a pair. Proof: Under the assmption that the ellipse lies entirely in the nit sare, the average payo for the row player can be compted by integrating the vale

6 obtained by the row-player in Eation where the and trajectories are those specied in eations 8 and 9. It can be shown that the integral of jst the cosine term, jst the sine term, and the prodct of the cosine and sine terms is exactly zero. This leaves jst the terms containing and. Therefore, the average payos are exactly the expected payos of the center which by Corollary 3 is a pair. 2 Therefore when the center point of the ellipses is in the interior of the nit sare, then all ellipses arond it that lie entirely within the nit sare have payos. Finally, we are ready to prove that when U has imaginary eigenvales, the average payos of the two players are always that of some pair. Lemma 6 ( Convergence in the case of imaginary eigenvales) When the matrix U has imaginary eigenvales, the IGA algorithm either leads the strategy pairs to converge to a point on the bondary that is a pair, or else the strategy pairs do not converge, bt the average payo of each player converges in the limit to that of some pair. Proof: Consider again the nconstrained dynamics of Figre 3a. The for adrants have the following general properties: in adrant A the gradient has a positive component in the down and right directions, in adrant B the gradient has a positive component in the down and left directions, in adrant C the gradient has a positive component in the p and left directions, and in adrant D the gradient has a positive component in the p and right directions. The direction of the gradient on the bondaries between the adrants is also shown in Figre 3a. The important observation here is that the direction of the gradient in each adrant is sch that there is a clockwise cycle throgh the adrants. There are three possible cases to consider depending on the location of the center ( ; ).. Center is in the interior of the nit sare. First observe that all bondaries are tangent to some ellipse, and that at least one bondary is tangent to an ellipse that lies entirely within the nit sare. For example, in Figre 3b the tangent ellipse to the left-side bondary lies wholly inside the nit sare, while the other three bondarys' tangent ellipses are not contained in the nit sare. If the initial strategy pair coincides with the center, we will get immediate convergence to a eilibrim becase the gradient there is zero. If the initial strategy pair is o the center point, then one of two things can happen: ) either the ellipse that passes throgh the initial point does not intersect with the bondary, or 2) the ellipse that passes throgh the initial point intersects with a bondary. In the rst case the dynamics will jst follow the ellipse, and by Lemma 5 above, the average payo for both players will be, even thogh the strategy pairs themselves will not converge, bt will follow the ellipse forever. In Figre 3b this will happen if the initial strategy pair is inside or on the dashed ellipse. In case 2) above, the strategy pair trajectory will hit a bondary, and then travel along it ntil it reaches a point at which the bondary is tangent to some ellipse that may or may not lie entirely in the nit sare. If it does, then the trajectory will follow that ellipse thereafter. If it does not, then the trajectory will follow the tangent ellipse to the next bondary in the clockwise direction. This process will repeat ntil the bondary that has a tangent ellipse lying entirely within the nit sare is reached. In Figre 3b, if the initial strategy pair starts anywhere otside the dashed ellipse, the dynamics will eventally follow the dashed ellipse. In all cases, from Lemma 5 we will get asymptotic convergence to the expected payos of some pair. 2. Center is on the Bondary. Consider the case where the center is on the left-side bondary of the nit sare. The rst observation is that all points below the center on the left-side bondary will then have a projected gradient of zero. (Figre 3a shows that the gradient at sch points will point left, and therefore otside the nit sare and perpendiclar to the left-side bondary.) The bottom bondary will have a projected gradient to the left. No matter where we start, we will either hit the bottom bondary, in which case we will get convergence to the lower left corner of the nit sare, or we will hit the left bondary below the center, in which case again we will converge becase of the zero projected gradient there. In either case, from Lemma 2 sch points will be pairs. By symmetry, a similar argment is easily constrcted when the center is on some other bondary of the nit sare. 3. Center otside nit sare. There are two cases to consider: ) the nit sare lies entirely inside one adrant, and 2) the nit sare lies entirely inside two adjacent adrants. No other case is possible. First consider the sitation when the nit sare lies entirely within adrant A. Then the gradient at each point points down and right (see Figre 3a), and hence we

7 will get convergence to the bottom right corner of the nit sare. A similar argment is easily constrcted when the nit sare lies entirely within some other adrant (yielding convergence to some other corner of the nit sare). Next consider the case where the nit sare lies in adrants A and D. In adrant D the gradient points right and p, so either the trajectory will enter adrant A withot hitting the top bondary, or it will hit the top bondary, in which case the projected gradient will be towards the right, and again it will enter adrant A. In adrant A we will converge to the lower right corner of the nit sare as above. A similar argment is easily constrcted when the nit sare lies within some other two adjacent adrants (again yielding convergence to some corner of the nit sare) Figre 4: Example dynamics when U has imaginary eigenvales. a) The center is in the nit sare. b) The center is on the left bondary of the nit sare. In Figre 4 we present examples of strategy pair trajectories for example problems whose U matrices have imaginary eigenvales. The left-hand gre is for a case where the center is contained in the nit sare while the right-hand gre is for a case where the center is on the left-hand bondary of the nit sare. 5.3 U has Real Eigenvales D C A B Figre 5: a) General characteristics of the nconstrained dynamics when U has real eigenvales. b) The possible transitions between the adrants of the left-hand gre. D C A B 2 The nconstrained dynamics of a linear dierential system with real eigenvales are known to be divergent (Reinhard, 987). See Figre b) for an illstration. Ths, withot the constraints of the nit sare, the strategy pair trajectory wold diverge. Figre 5a shows the crcial general properties of the nconstrained dynamics. The center ( ; ) is the point where the gradient is zero. Everywhere in adrant A the gradient has a positive component in the right direction and in the p direction; in adrant B the gradient has a positive component in the p direction and in the left direction; in adrant C the gradient has a positive component in the left and down directions; and in adrant D the gradient has a positive component in the right and down directions. At the bondary between adrants A and D the gradient points left; at the bondary between A and B it points p; at the bondary between B and C it points to the right; and at the bondary between C and D is points down. The nit sare that denes the feasible range of strategy pairs can be anywhere relative to the center. The eigenvectors corresponding to the two real eigenvales, p and? p are ;? respectively (this is for the case that ; > ; the analysis for the case that ; < is analogos and omitted). The eigenvectors are represented in Figre 5a with dashed lines: one by drawing a line throgh the center and the point (; ) and and the other by drawing a line throgh the center and the point (;? ). Note that the general alitative characteristics of the positive components of the gradient in the dierent adrants do not depend on the eigenvectors. However, the eigenvectors are relevant to the detailed dynamics, as we will see in the examples below. Lemma 7 ( convergence in the case of real eigenvales) For the case of U having real eigenvales, the IGA algorithm leads the strategy pair trajectory to converge to a point on the bondary that is a pair. Proof: Consider the graph of possible transitions between the adrants in Figre 5b. From every point inside adrant A, the gradient is sch that the strategy pair will never leave that adrant. Therefore if the strategy pair trajectory ever enters adrant A, it will converge to the top right corner of the nit sare. Similarly, from every point inside adrant C, the gradient is sch that the strategy pair will never leave that adrant. Ths if the strategy pair trajectory ever enters adrant C, it will converge to the lower left corner of the nit sare. If the initial strategy pair is in adrant B or D, the dynamics is a bit more complex

8 becase it depends on the location of the nit sare relative to the center. We consider only the case of adrant D, for by symmetry a similar analysis will hold for adrant B. Unless the nit sare lies entirely in adrant D, the strategy pair trajectory will enter adrant C or adrant A, in which case we will get convergence to the associated corner as above. If the nit sare is entirely within adrant D, then the direction of the gradient in that adrant will lead to convergence to the lower right corner of the nit sare. Finally, if the right-hand bondary of the nit sare is on the bondary between adrants D and A, then all the points on that bondary will have zero projected gradient, and any trajectory from D hitting that bondary will converge there. Similarly, if the bottom bondary of the nit sare is aligned with the bondary between adrants C and D, then any trajectory from D hitting that bondary will converge there. From Lemma 2, if we ever get convergence to a strategy pair, it mst be a pair Figre 6: Example dynamics when U has real eigenvales. a) Center in the nit sare. b) Center on the right bondary of the nit sare. In Figre 6 we present examples of strategy pair trajectories for example problems whose U matrices have real eigenvales. The left-hand gre has the center in the nit sare; the right-hand gre has the center on the right-hand bondary of the nit sare. The locations of the points are shown. 6 Finite Decreasing Step Size The above analysis and Theorem have been abot the IGA algorithm that assmes that both players are following the tre gradient with innitesimal step sizes. In practice, of corse, the two players wold se the gradient ascent algorithm of Eation 5 with a decreasing nite step size k (where k is the iteration nmber). Theorem 8 The seence of strategy pairs prodced by both players following the gradient ascent algorithm of Eation 5 with a decreasing step size (several schedles will work, e.g., k = ) will satisfy one of k 2=3 the following two properties: ) it will converge to a pair, or 2) the strategy pair seence will not converge, bt the average payo will converge in the limit to that of some pair. Proof: (Sketch) Here we provide some intition; the fll proof is deferred to the fll paper. Consider rst the cases in which the IGA algorithm converges to a pair. The proofs in sch cases exploited only the direction of the gradient in the for adrants arond the center. These same proofs will extend withot modication to the case of decreasing step sizes. The one case (with imaginary eigenvales) in which the IGA algorithm does not converge to a point bt instead converges to some ellipse flly contained in the nit sare is more complex to handle. The basic intition is that the strategy pairs cannot get \trapped" anywhere, and as the step size decreases, the dynamics of gradient ascent approaches the dynamics of IGA. 2 7 Conclsion Algorithms based on gradient ascent in expected payo are natral candidates for adapting strategies in repeated games. In this work we analyzed the performance of sch algorithms in the base case of twoperson, two-action, iterated general-sm games, and showed that even thogh the strategies of the two players may not converge, the asymptotic average payo of the two players always converges to the expected payo of some eilibrim. Or proof also provides some insight into special classes of games, sch as zero-sm and team games. In the ftre we will stdy the behavior of gradient ascent algorithms in complex mlti-action and continos action games in which the players se parameterized representations of strategies. References Fred Y., Kearns M., Mansor Y., Ron D., Rbinfeld R., and Schapire R.E. (995). Ecient Algorithms for Learning to Play Repeated Games Against Comptationally Bonded Adversaries. In Proceedings of the 36th Annal Symposim on Fondations of Compter Science. Kalai E. and Lehrer E. (993). Rational Learning leads to Eilibrim. Econometrica. Owen G. (995). Game Theory. Academic Press, UK. Papadimitrio C.H. and Yannakakis M. (994). On complexity as bonded rationality. In Proceedings of the 26th Annal ACM Symposim on the Theory of Compting, pages 226{733. Reinhard H. (987). Dierential Eations: Fondations and Applications. Macmillan Pblishing Company, New York. Weibll J.W. (997). Press. Evoltionary Game Theory. MIT