Constraint Satisfaction Algorithms for Graphical Games

Transcription

1 Constraint Satisfaction Algorithms for Graphical Games Vishal Soni Satinder Singh Computer Science and Engineering Division University of Michigan, Ann Arbor Michael P. Wellman Abstract We present PureProp: a new constraint satisfaction algorithm for computing pure-strategy approximate Nash equilibria in complete information games. While this seems quite limited in applicability, we show how PureProp unifies existing algorithms for 1) solving a class of complete information graphical games with arbitrary graph structure for approximate Nash equilibria (Kearns et al., 21; Ortiz & Kearns, 23), and 2) solving classes of incomplete information games with tree-graph structure for approximate Bayes-Nash equilibria (Singh et al., 24). We also show how PureProp provides a hitherto unavailable algorithm for solving classes of incomplete information graphical games with arbitrary graph structure for approximate Bayes-Nash equilibria, and conclude with some empirical analysis of the new algorithm. 1 Introduction Graphical models for representing games were introduced by Kearns et al. (21) to capture locality of interactions among agents in a static game. Under this model, nodes represent agents and (undirected) edges represent interactions between agents. A missing edge between two nodes implies that the payoffs of the respective agents do not directly depend on the other s action. The central idea is to view the game as being composed of several interacting local games and to exploit this locality, especially in sparsely connected games, by iteratively computing local equilibria and patching them together to obtain global equilibria efficiently. Games with sparsely connected graphs arise quite naturally in various settings because of computer network topologies, social networks of a population of animals, ad hoc networks of mobile devices, etc. The original work by Kearns, Littman and Singh (21) considered acyclic graphical games of complete information and presented a message-passing algorithm (hereafter, the KLS algorithm) for computing approximate Nash equilibria (NE) efficiently. Ortiz and Kearns (23) later presented the NashProp algorithm that extended the KLS algorithm to arbitrary graph structures in complete information games. In other recent work, Vickrey and Koller (22) reformulated the problem of finding approximate equilibria in games of complete information as a constraint satisfaction problem and derived a generalization of the KLS algorithm as a constraint satisfaction algorithm. 1 In our own recent work (Singh et al., 1 Ortiz and Kearns (23) also note but do not establish a connection between their iterative

2 24), we considered acyclic games of incomplete information and presented a messagepassing algorithm to compute approximate Bayes-Nash equilibria (BNE) efficiently. In games of incomplete information, the payoff to an agent depends not only on the actions of the other agents but also on its own private type. Agents have to choose actions without explicit knowledge of the private types of other agents. Thus, a BNE strategy has to be an equilibrium with respect to the known distribution over the types of the other agents. As our main contribution in this paper, we present PureProp(ɛ), a simple constraint propagation algorithm that unifies all of the results mentioned above, and provides a hitherto unavailable algorithm for computing ɛ-bne in two classes of incomplete information games with arbitrary graph structures. Hereafter we drop the argument ɛ from the name of the algorithm for ease of exposition. Specifically, PureProp is an algorithm for finding a pure strategy ɛ-ne in arbitrary complete information games. We show how to induce a restricted game solvable by PureProp from all the classes of graphical games mentioned above. In all these cases, the construction of the induced game will be such that a pure-strategy ɛ-ne will always exist in the induced game and furthermore the equilibria found by PureProp will be ɛ-equilibria in the original game. 2 Preliminaries (Incomplete Information Games) Let n denote the number of agents in the game. For games with discrete types, the realized type of each agent is chosen independently from some discrete set T = T 1, T 2,, T m according to some commonly known distribution P T. Complete information games are a special case in which there is always exactly one type and so there is no uncertainty as to the type of each agent. For games with continuous types, the realized type of each agent is chosen independently from some interval T = [t l, t u ] according to some again commonly known probability density p T. For ease of exposition, we assume that all agents have the same action set A = {a 1, a 2,, a d } available to them. The vector notation a, t, and µ denotes the joint actions, types, and strategies for all n agents. The symbol µ denotes a type-conditional mixed strategy, a mapping from types to probability distributions over actions, while the symbol σ is used to refer to an unconditional (type independent) strategy, a probability distribution over actions. Accordingly, µ V ( t V ) refers to the mixed strategy of V when its type is t V, and µ V (a t V ) refers to the probability of V taking action a when V s type is t V. We use a V, t V, and µ V to refer to the actions, types and strategies of all agents excluding V. Finally, R V ( a t) refers to the payoff to agent V when the joint actions of the n agents are a and the type of V is t. Defining the game requires specifying R V ( a t) for all V, all a A n and all types t. Next we define some quantities of interest for games with discrete types. These can be extended in a straightforward way for continuous types by replacing the summations over discrete types and probability distribution P T with integrals over continuous types and probability density p T. The expected payoff to agent V from playing a strategy σ when its type is t V and the other agents play according to the joint strategy µ V is R V (σ, µ V t V ) = ( ) P T ( t V ) σ(a V ) µ V ( a V t V ) R V ( a t V ). (1) t V a The expected payoff to V when agents play according to µ is R V ( µ) = t V P T (t V ) R V (µ V ( t V ), µ V t V ). (2) A strategy σ is said to be an ɛ-best response strategy for agent V with type t V and other agents strategies µ V if, for all unconditional strategies σ, R V (σ, µ V t V ) message-passing algorithm and constraint satisfaction algorithms (they thank Michael Littman in their note).

3 R V (σ, µ V t V ) ɛ. An ɛ-bne is a strategy profile µ such that for all agents V and strategies µ V, R V ( µ) R V (µ V, µ V ) ɛ. In words, an ɛ-bne is a strategy profile so that no agent by a unilateral change in strategy can benefit more than ɛ in expectation over the types of the other agents. Note that in complete information games many of the above equations would simplify because each agent has only one type. Also, in such games the ɛ-bne would actually be an ɛ-ne. It is well known that exact (or ɛ = ) BNE and NE always exist in incomplete and complete information games respectively. Graphical Games: Graphical games have nodes representing agents and edges representing direct interactions between agents. Each node has a payoff matrix that defines her reward as a function of her type and the actions of her neighbors. We use N V to denote the set of nodes in the neighborhood of V (this includes node V itself) and N V V to denote the set N V \{V }. Let k be the largest number of neighbors of any node in the graph. If n >> k, then the graphical representation leads to a substantially more compact representation of the game than the usual normal form (matrix) representation. In the worst case of a complete graph, the size of the graphical game representation equals that of the normal-form representation. The locality of interaction in a graphical game also reduces the computation required to compute the expected rewards in Equations (1) and (2) because only the neighbors of a node have to be considered in the summations. In Figure 1, we illustrate four graph topologies (borrowed from Ortiz & Kearns, 23), capturing different kinds of locality of interaction among the agents. These are also the graph topologies on which we evaluate the performance of our algorithm. (a) Ring (b) Grid (c) Tree (d) Chordal 3 PureProp(ɛ) Algorithm Figure 1: Example graph topologies. We first describe how to convert the problem of finding pure-strategy ɛ-ne in games of complete information with a finite number of actions into a constraint satisfaction problem (CSP). We call the resulting CSP the game-csp. Then we present our algorithm for solving game-csps. Game-CSP: A CSP is defined by a set of variables, the domain of each variable (i.e., the values each variable can take), and a set of constraints. Each constraint involves some subset of the variables and specifies the allowable combinations of values for that subset. In the game-csp there is a variable for each agent (node in the graphical game). The domain of a variable V, denoted D(V ), is the set of all possible profiles of V. A profile for V is an assignment of (pure) actions to all agents in the neighborhood of V. We briefly note here that our CSP formulation of games differs from the CSP formulation of games in Vickrey and Koller (22) in that we use neighborhood profiles as values in the domains of the variables while they use single-agent actions as values in the domain of the variables. Thus our CSP is a dual transformation (Dechter & Pearl, 1988) of the Vickrey and Koller (22) CSP. We will investigate the computational impact, if any, of this difference as future work. There are two sets of constraints imposed by the CSP. The first set are unary constraints that require that if we assign profile ρ to V, it must be true that the action assigned to agent V in ρ is an ɛ-best response to the actions assigned by ρ to the neighbors of V (where

4 ɛ is an approximation parameter that is an input argument to PureProp). The second set of constraints are binary constraints on every pair of neighbors U and V in the graphical game. This constraint requires that if we assign profile ρ to V and profile γ to U, then ρ and γ must assign the same actions to the neighbors that U and V have in common. The constraint graph of a CSP places an undirected edge between nodes that have a binary constraint between them. Thus, the constraint graph of the game-csp formulated as above for a graphical game will have exactly the same edges as in the graphical game itself. A solution to the game-csp is an assignment of profiles to each variable so that none of the constraints are violated. If there is no such assignment of profiles to variables the game- CSP has no solution. The following result is obtained by construction of the game-csp above. Theorem 1 Any solution to the game-csp for a graphical game of complete information is an ɛ-nash equilibrium of the underlying game. Also, all pure-strategy ɛ-nash equilibria of a complete information graphical game with a finite number of actions are solutions to the game-csp generated by the game. Solving the game-csp: phases. The PureProp algorithm proceeds in following three consecutive 1. Node consistency: This phase considers each variable in turn and removes the profiles from its domain that violate the unary constraint for that variable. For an agent with k neighbors with d actions for each agent, there are d k profiles in the domain and each of them have to be checked for a violation. 2. Arc Consistency: Each undirected edge in the constraint graph of the CSP is treated as two directed arcs in this phase. So for an edge between neighbors V and U, there is an arc [U, V ] as well as an arc [V, U]. Given current domains D(V ) and D(U), the arc [V, U] is said to be arc-consistent if for every profile ρ in D(V ), there is some profile γ D(U) such that the assignment of ρ to V and γ to U satisfies the binary constraint of the CSP. The goal of the arc consistency algorithm is to return domains for the variables such that all arcs are arc-consistent. While there are a number of algorithms for arc consistency in the CSP literature, we implemented the AC-3 algorithm as presented in Russell and Norvig (23). This algorithm maintains a queue of arcs to be examined (initialized to contain all the arcs in the graph) and at each step examines the arc at the head of the queue. For every arc [V, W ] examined AC-3 prunes the domain of node V so that [V, W ] is arc-consistent. If the domain of V is reduced then all incoming arcs to V are inserted into the queue. The algorithm stops either when every arc is arc-consistent (the queue is empty) or when the domain of some node is empty. We denote the domains returned by AC-3 as D. Note that if the domain of any node is empty at any step in either of the above two phases, the game-csp has no solution. Next we show that all pure-strategy ɛ- NE in the underlying game are present in D. For any global profile (an action-assignment to every variable) P, let P[N V ] be the projection of P onto the variables in the neighborhood of V. Lemma 2 Let P be a global profile. Then, P[N V ] D (V ) V, if and only if P is an ɛ-ne in the underlying game. Proof If P[N V ] D (V ) V, then P[V ] must be a ɛ-best response to P[N V ] and therefore P must be an ɛ-ne. Now let s consider the other direction. If P is a ɛ-ne, then for all V, P[V ] is an ɛ-best response to P[N V ] and hence P[N V ] D (V ). Also, for two neighbors V and U, P[N V ] and P[N U ] satisfy the binary constraint for the variables U and V in the CSP. Hence P[N V ] D (V ). Lemma 2 establishes that the arc consistency phase can potentially (and hopefully significantly) reduce the size of the domains of the variables without eliminating any ɛ-ne in the

5 game. 3. Value Propagation: This phase executes a standard backtracking search over the variable domains (post node and arc consistency) in the constraint graph. Following the minimum values remaining heuristic, nodes are visited in ascending order of their domain size. At any stage of the search, a node V picks from its domain a profile ρ that is consistent with the current action assignments of V and its neighbors. If any of V s neighbors have not been assigned an action, V instructs them to play an action according to ρ. If V is not able to find such a profile, the algorithm backtracks. It is known that backtracking search is a complete algorithm that in the worst-case amounts to an exhaustive search over variable domains. The algorithm can stop after finding the first solution or can be extended to find all solutions to the game-csp. Treating the conversion to the game-csp as part of PureProp the following result holds: Theorem 3 PureProp(ɛ) computes (all) pure-strategy ɛ-ne in complete information games with a finite number of actions. Proof The result is directly implied by Theorem 1, Lemma 2, and the completeness of the value propagation phase. So far we have shown that PureProp is able to find pure-strategy approximate NE in complete information games. While this by itself seems quite limited in applicability, we show next the far more interesting and applicable result that PureProp can be used to solve classes of complete and incomplete information games for approximate mixed-strategy equilibria. 4 Algorithms for Finding Approximate Mixed-Strategy Equilibria We consider two classes of incomplete information games: games with discrete types and games with continuous types. In both classes, the space of strategies is infinite and hence PureProp cannot be directly applied. To apply PureProp we have to construct a new reduced game with a finite strategy space such that (a) all pure-strategy ɛ-ne in the reduced game are ɛ-equilibria in the original game, and (b) at least one ɛ-equilibrium of the original game is a pure-strategy ɛ-ne in the reduced game. We refer to such a reduced game as an ɛ-induced game. Discrete Types: In such games, a strategy is a mapping from the discrete type space to probability distributions over actions. We construct an ɛ-induced game by constraining action probabilities to be multiples of some discretization parameter τ. The result is a finite set of τ-discretized type-conditional strategies that can be treated as pure actions of the ɛ-induced game. Theorem 4 For arbitrary graphical games of incomplete information with discrete types, PureProp applied to the ɛ-induced game defined above with discretization parameter τ will converge to an ɛ-bne of the game. ɛ d k (4k log k) ɛ Proof (Sketch) Singh et al. (24) show that for any ɛ, there is a τ d k (4k log k) such that if the mixed strategy space for every type is gridded up at τ intervals, there exists an ɛ-bne in the τ-discretized strategy space (recall that d is the number of actions and k is the maximum neighborhood size). Thus the number of pure actions in the resulting ɛ-induced ( game is of O[ 1 ) ] md τ where m is the number of discrete types. PureProp applied to the ɛ-induced game is thus guaranteed to converge to an ɛ-bne of the original game. Continuous types: In games with continuous types, it is reasonable to require that types have some structured effects on agents payoffs. Otherwise just representing the effect of

6 types exactly can lead to unbounded game descriptions. Quite naturally, the desire is to identify common effects that can be exploited for computational efficiency. In Singh et al. (24), we considered a class of games with continuous types that have what is called the single crossing point property (examples of such games include certain kinds of first price auctions, all-pay auctions, noisy signaling games, oligopoly games, etc. (Athey, 21; Milgrom & Weber, 1985; Topkis, 22)) such games have the interesting property that there exist BNE that are monotone threshold pure strategies. In a threshold pure strategy each type is mapped to a pure action and furthermore each action is played in at most one contiguous region of the continuous type interval. We use the term threshold to refer to these strategies because the points in the type interval at which the strategies switch to new actions constitute thresholds. In monotone threshold pure strategies there is an ordering over actions such that the actions appear in that order in the strategy of all agents. As in Singh et al. (24), we will only consider games with the single crossing point property, i.e., games with monotone threshold pure strategies. Even though distributions over the pure actions don t need to be considered in such games, the number of strategies in such games is infinite because of the continuous types. Theorem 5 For graphical games of incomplete information with continuous types that satisfy the single crossing point property (and thus have threshold pure strategy BNE), PureProp applied to the ɛ-induced game defined above with discretization parameter τ ɛ 4dkρ t will converge to an ɛ-bne of the original game. Proof (Sketch) Singh et al. (24) showed that for any ɛ, there exists a τ = ɛ 4dkρ t, where ρ t is the maximum probability density over type space, such that there exists a τ- discretized strategy that is an ɛ-bne. A τ-discretized strategy only allows thresholds at the [( τ intervals in type space. There will be O 1 τ ) d ] many τ-discretized strategies and these will be the pure actions of the ɛ-induced game. As before, the existence of an ɛ-bne in the τ-discretized strategies of the original game will guarantee the existence of an ɛ-ne in the ɛ-induced game. PureProp applied to the ɛ-induced game with the τ-discretized strategies as actions will find ɛ-bne in the original game. Note that PureProp extended the results of Singh et al. (24) for tree-games to cyclic graphical games without requiring any changes in the discretization parameters. The ability to compute approximate equilibria in the above classes of incomplete information games with arbitrary graph structure is one of the main contributions of this work. Furthermore, PureProp also unifies all the results on graphical games mentioned in the introduction; for lack of space we only briefly discuss this aspect. For tree structured games of incomplete information, it can be shown that with an appropriate ordering of variables during arcconsistency and value assignment, PureProp applied to the ɛ-induced games implements the algorithm of Singh et al. (24). Recall that complete information games can be treated as incomplete information games with just one (discrete) type. The discretization technique employed for discrete type games can thus be applied here to obtain an ɛ-induced game solvable by PureProp for ɛ-ne. This, in fact, is the same discretization scheme as that used by the algorithms of Ortiz and Kearns (23) and Kearns et al. (21). Thus, the resulting algorithm implements the NashProp algorithm of Ortiz and Kearns (23) for arbitrary graphs, and, with an appropriate variable ordering heuristic, the algorithm of Kearns et al. (21) for tree structured games. 5 Experimental Results For our experiments, we ran PureProp on the party game with threshold pure strategies described in Singh et al. (24) and explored the effects of graph size and topology on various phases of PureProp. In the party game, agents have been invited to a party and must

7 decide whether to attend or not (so two actions). Each agent has a social-network of agents she is acquainted with and these constitute her neighbors in the graphical game. An agent s reward for attending is an additive function of her like/dislike for her attending acquaintances. Each agent also incurs a private cost associated with attending the party which has an additive effect on her reward. In the corresponding graph game, nodes represent agents while edges connect an agent to her social network. We used the τ-discretization idea described above to construct an ɛ-induced game and then used PureProp to compute approximate equilibria in party games for the following neighborhood topologies (shown in Figure 1): (a) ring, (b) grid, (c) acyclic with a branching factor of 2 (i.e. each node has 3 neighbors), (d) chordal (1% of edges are chords), (e) chordal (2% of edges are chords). For each topology, we varied the number of players and measured the effect on the runtime 2 of various phases of our algorithm. (a) Runtimes to compute a Single Equilibrium 8 Ring 7 Tree Chordal 1% Chordal 2% 6 (d) x 15 Grid Topology (b) 1 (e) Acyclic Topology (tree) x 15 4 Chordal Topology; 1% chords (c) 14 (f) Ring Topology x Chordal Topology; 2% chords 15 Single equilirium All equilibria, node + arc consistency 5 All equilibria, node consistency Figure 2: Party Game Results. (a) Scaling with number of agents. (b) (f) Runtime analysis of the PureProp algorithm on the following topologies : acyclic (tree), ring, grid, chordal (1%), and chordal (2%). In (b) (f), solid lines indicate the time to compute a single ɛ-bne, dashed-dot lines indicate the time to compute all ɛ-bne, and dashed lines indicate the time to compute all ɛ-bne when the arc consistency step is skipped. Figure 2(a) compares the runtime of the algorithm on various topologies as a function of the number of agents. We observe that regardless of topology, the runtime to compute a single ɛ-bne scales linearly with the number of nodes. This is in accordance with our objective of exploiting the locality of interactions to compute global equilibria in a manner that scales well with the number of agents. Additionally, we see that runtime is strongly affected by the sizes of local neighborhoods. For instance, computation is quickest for ring topologies. For the two chordal graphs, we observe that the graph with fewer chords, and hence fewer cycles, results in faster runtimes. Each plot in Figures 2(b - f) shows three curves that indicate 1) the time to compute a single ɛ-bne, 2) the time to compute all ɛ-bne, and 3) the time to compute all ɛ-bne when the arc consistency phase is skipped, i.e. node consistency is immediately followed by value assignment. 2 We measured runtime for all our experiments on a machine with a 2.8GHz Intel(R) Pentium(R) CPU and approximately 1GB of RAM.

8 In Figure 2(b), we see that for acyclic graphs, the time required for all three computations is pretty much the same, suggesting that most of the work is done by the node consistency step and very little by arc consistency and value propagation this is exactly the behavior we would expect since the number of backtracking steps during value propagation will be minimal. 3 In Figure 2(c), we notice that for ring topologies, circumventing arc consistency results in a big performance hit. This serves to demonstrate the effectiveness of the arc consistency step in reducing domain sizes. In Figure 2(d), however, it is interesting to note that for the grid topology, arc consistency does not prove very beneficial for graphs with fewer than 1 nodes. We speculate the reason for this is high degree of connectivity in this graph. For smaller graphs, this would cause the node consistency phase to eliminate a considerable number of profiles. However, as the ratio of node connectivity to graph size decreases, so does the effectiveness of node consistency. At the same time, the benefits of arc consistency start to become apparent. In Figures 2(e) and 2(f) we observe similar effects, and note again that the algorithm generally ran faster on graphs with fewer chords due to the presence of fewer cycles. 6 Conclusion We presented PureProp(ɛ), a constraint satisfaction algorithm for computing pure-strategy ɛ-ne in complete information graphical games with arbitrary graph structure. We showed how to use PureProp to derive a hitherto unavailable algorithm for approximately solving two classes of graphical games of incomplete information with arbitrary graphical structure. Finally, we showed (briefly) how PureProp unifies the previous algorithms available for finding approximate equilibria in several classes of graphical games: tree-structured graphical games of complete information, arbitrary-structured graphical games of complete information, and tree-structured graphical games of incomplete information. References Athey, S. (21). Single crossing properties and the existence of pure strategy equilibria in games of incomplete information. Econometrica, 69, Dechter, R., & Pearl, J. (1988). Tree clustering schemes for constraint processing. Seventh National Conference on Artificial Intelligence (AAAI-88) (pp ). St. Paul, MN. Kearns, M., Littman, M. L., & Singh, S. (21). Graphical models for game theory. Seventeenth Conference on Uncertainty in Artificial Intelligence (pp ). Seattle. Milgrom, P., & Weber, R. (1985). Distributional strategies for games with incomplete information. Mathematics of Operations Research, 1, Ortiz, L. E., & Kearns, M. (23). Nash propagation for loopy graphical games. In S. T. S. Becker and K. Obermayer (Eds.), Advances in neural information processing systems 15, Cambridge, MA: MIT Press. Russell, S., & Norvig, P. (23). Artificial intelligence - a modern approach. Prentice Hall. Singh, S., Soni, V., & Wellman, M. P. (24). Computing approximate Bayes Nash equilibria in tree-games of incomplete information. In ACM Conference on Electronic Commerce. Topkis, D. (22). Supermodularity and Complementarity. Princeton University Press. Vickrey, D., & Koller, D. (22). Multi-agent algorithms for solving graphical games. Eighteenth National Conference on Artificial Intelligence (pp ). Edmonton, Alberta, Canada: American Association for Artificial Intelligence. 3 In acyclic graphs an ordering over nodes from the root to the leaves would lead to no backtracking in the value propagation phase. We did not specialize the value propagation phase for acyclic graphs and choose node ordering based on domain size as for the cyclic graphs.