Large Scale Parallel Monte Carlo Tree Search on GPU

Transcription

1 Large Scale Parallel Monte Carlo Tree Search on Kamil Rocki The University of Tokyo Graduate School of Information Science and Technology Department of Computer Science 1

2 Tree search Finding a solution by visiting each node in a tree/graph data structure in a systematic way Typical search problems: Shortest path (i.e. Dijkstra, A*), Traveling salesman problem Min-max depth (plies) SAT solver branching complexity = f(branching, depth) 2

3 Typical Applications Planning, scheduling Discrete optimization Database search Games/puzzles (AI) Games and puzzles are a good testbed for tree-search algorithms: - Complexity - Easy to measure results - The algorithm should work in other domains 3

4 Tree search (games) depth branching complexity = f(branching, depth) 4

5 Problem statement Game tree search Nodes represent <state,actions> pair Look ahead to find the best move (minimizing cost/ maximizing reward) Search strategy - defines the way the nodes are expanded Class of problems - decision making 5

6 Analyzed problem Reversi Game - tree represent a sequence of moves Non-uniform tree (variable children number) Depth up to 60 Average number of leaves Initial position Example final position (A) (B) 6

7 Problems complexity Exhaustive search Assuming only 1 ns per state, (in Reversi): 11 moves ahead ~ 21 s 12 moves ahead ~ 3 m 13 moves ahead ~ 27 m 14 moves ahead ~ 4 h 15 moves ahead ~ 1.45 d 7

8 Parallel systems - vs clock speed has been fairly static recently - 3~3.4Ghz Due to the rise of multi-core machines, the computational power has still been increasing s be far ahead of the s in terms of FLOPS 8

9 vs - Dedicated to computation 9

10 Goal Utilize highly parallel (/+) systems to search trees efficiently Increase in + systems share (TOP500 list) But, programming is challenging 10

11 Taxonomy Single Data stream Multiple Single Instruction stream Multiple Single Instruction Single Data SISD Multiple Instruction Single Data MISD Single Instruction Multiple Data SIMD Multiple Instruction Multiple Data MIMD 11

12 SIMD processing - threads in SIMD groups Flynn s Taxonomy (1966) PU - Processing Unit 12

13 SIMD processing - why a problem? Threads a[i] = b[i] + c[i]; if (a[i] > 0) WAIT else a[i] = a[i]*a[i]; a[i] = 0; Tree search - much more complex 13

14 Thread 0 2nd iteration Thread Inner loop 1 Thread divergence problem 14

15 Thread 0 5th iteration Thread Inner loop 1 4 Thread divergence problem 15

16 Thread 0 6th iteration Thread Inner loop Thread divergence problem 16

17 Processing 1. Data needs to be transferred from to 2. Program is executed 3. Output data is transferred back from to s memory 17

18 Processing 1. Data needs to be transferred from to 2. Program is executed 3. Output data is transferred back from to s memory 18

19 Processing 1. Data needs to be transferred from to 2. Program is executed 3. Output data is transferred back from to s memory Latency 19

20 Hardware Global memory (Big - GBs, all threads), Local memory (max 512kB per thread): slow New s (2010-) also have cache Architecture awareness Registers (per thread - max 32k per MP): fastest Shared memory (per block - max 48kB per MP): fast 20

21 Software Model Host = Device = Thread = basic execution unit = SP Block = batch of threads - sharing MP Grid = batch of blocks = MIMD Warp = scheduling unit = SIMD Programmability Thread Warp Block Grid... SIMD Processing 1 warp = 32 threads (currently) 21

22 programming difficulties Easy to write a simple program - just port the code from - bad performance Algorithms need to be rethought and reimplemented - implementation itself can be challenging ( hardware, CUDA software) Hard to achieve good performance (high parallelism) Memory hierarchy/constraints SIMD processing - warp divergence Limited communication Latency Latency Architecture awareness Programmability 22

23 Monte Carlo Tree Search 23

24 What is Monte Carlo Tree Search (MCTS)? A method for making optimal decisions in AI problems Can theoretically be applied to any domain described by {state, action} Games, i.e. GO, Reversi + some other difficult games - Coulom (2006, 2008) Optimization - Gaudel, Sebag (2010) Decision support systems - Chaslot et al. (2006) Alternative for dynamic programming, Markov models - Teytaud et al. (2008) Alternative for A* IDA* algorithms - Single Player MCTS - Schadd et al (2008) Constraint satisfaction problem - Baba et al. (2011) 24

25 MCTS - Coulom (2006) UCT - Kocsis and Szepesvári (2006) The basic MCTS algorithm is simple C - exploitation/exploration ratio factor, tunable mean value of node (i.e. success/loss ratio) standard UCT formula - selection step total simulations number of s node visited Repeated X s Selection Expansion Expension Simulation Backpropagation The selection function is applied recursively until the end of the tree The selection function is applied recursively One One (or (or more) leaf nodes are nodes created are created One simulated game is is played The The result of this this game game is is in the tree backpropagated in the tree 25

26 Exploitation vs exploration High exploitation (greedy) High exploration Quickly gets to a local optimum Greater chance of finding the global optimum 26

27 MCTS 3/6 2 parts 3/5 2/3 1/3 1/3 Tree building Stored in the memory Simulating 1. Temporary - not remembered 0 2. Done by or 3. The results are used to affect Final result: the tree s expansion strategy 0 or 1 27

28 MCTS components 3/5 Tree construction selection 3/4 1/3 2/3 C - exploitation/exploration ratio factor, tunable mean value of node (i.e. success/loss ratio) total simulations number of s node visited choose the highest evaluation 28

29 MCTS 3/5 3/4 1/3 Tree construction 2/3 expansion 1/2 Simulation I initiate the nodes with values 1 win / 2 simulations 29

30 MCTS 3/5 3/4 1/3 Tree construction 2/3 1/2 simulation Simulation 0 The result is remembered only 30

31 MCTS backpropagation 0/1 3/6 3/5 2/3 1/3 1/3 backpropagation 0/1 Tree construction Simulation 0 backpropagation 0 successes / 1 simulation 31

32 MCTS - Coulom (2006) After X iterations - the decision making step Choose the best option from the root s children Typically based on the average score (number of wins/number of simulations) UCT - Kocsis and Szepesvári (2006) What now? The best choice 32

33 The features of MCTS Aheuristic Asymmetric Tree Growth More iterations stronger algorithm Easy to parallelize No intermediate state evaluation is needed 33

34 Parallel MCTS Schemes Chaslot et al. (2008) Complex, not efficient Easy Efficient 34

35 Parallel MCTS Schemes Performance in GO Chaslot et al. (2008) 35

36 Sequential/leaf parallel MCTS Seen as a optimization problem 36

37 Root parallel MCTS - many starting points Greater chance of reaching the global 37

38 Parallel MCTS on Leaf parallelism - Not efficient, but easy to implement on No divergence problem, high throughput (simulation speed) Root parallelism - Very efficient on, but would be inefficient on One thread per tree Thousands of trees, where to store them?, how to manage them? Divergence problem again Solution? 38

39 Proposed solution: Block parallelism 39

40 Block parallelism (a) + (b) = Block parallelism (c) n trees Weakness: sequential tree management part (proportional to the number of trees) n simulations a. Leaf parallelism b. Root parallelism n = blocks(trees) x threads (simulations at once) Advantage: Works well with SIMD hardware, improves the overall result on 2 levels of parallelization c. Block parallelism 40

41 sequential overhead : stores and manages the trees Repeated X s Selection Expansion Expension Simulation Backpropagation : simulates The selection function is applied recursively until the end of the tree The selection function is applied recursively One One (or (or more) leaf nodes are nodes created are created One simulated game is is played The The result of this this game game is is in the tree backpropagated in the tree More trees processing takes more 4 trees 8 trees Sequential parts Parallel Parallel Parallel Parts Parallel Parallel 41

42 sequential overhead More trees: More spent on the sequential part 1 thread iterates over the trees Sequential parts (Backpropagation) (Selection) (Expansion) Possible solution: more threads per Parallel parts (simulation) 42

43 Analysis Leaf parallelism vs block parallelism vs cpu root parallelism More simulations/s = better score? Block parallelism: number of trees vs their management cost 43

44 Proposed solution - Block parallelism - block parallelism vs leaf parallelism speed Simulations/second x 105 Leaf parallelism (block size = 64) Block parallelism (block size = 32) Block parallelism (block size = 128) Average for 2000 games vs 1 cpu thread 1 - around sim/s is much faster! 112 trees 448 trees Threads 44

45 Win ratio Proposed solution - Block parallelism - block parallelism vs leaf parallelism result Average for 2000 games vs 1 cpu thread 112 trees 448 trees Leaf parallelism (block size = 64) Block parallelism (block size = 32) Block parallelism (block size = 128) 112 trees Threads 45

46 Proposed solution - Block parallelism Number of trees/speed/results More trees = higher score More simulations = higher score More trees = fewer simulations Block size needs to be adjusted 1 ~ s (AI strength) 46

47 Proposed solution - Block parallelism Weakness no 1 Still a part of the algorithm relies on weak leaf parallelism Solution: Variance based error estimation 47

48 Variance based error estimation Calculation based on all gathered samples Works better for larger number number of samples utilizes leaf parallelism, need many samples to work Applied to the decision making step (final selection) best average best lower estimate possible decisions values exact value solid lines - averages dotted lines - lower confidence bounds possible errors 48

49 Variance based error estimation - results 1 Tesla C2050 (8192 threads, 64 Trees) vs 1 (TSUBAME) blocks x 128 threads Points Leaf basic Block basic Leaf + error est Block + error est 49

50 Proposed solution - Block parallelism Weakness no 2 - s latency, shallow trees 1 simulation takes ~ 0.1ms = 10 sims/ms simulations take ~ 100ms = 300 sims/ms But it takes at least 50ms to execute any number of simulations on s MCTS iterations are much faster More iterations, more tree nodes, deeper trees, more look ahead Repeated X s Selection Expansion Expension Simulation Backpropagation The selection function is applied recursively until the end of the tree The selection function is applied recursively One One (or (or more) leaf nodes are nodes created are created One simulated game is is played The The result of this this game game is is in the tree backpropagated in the tree 50

51 Simultaneous / simulating While runs a kernel can work too Increases the tree depth, improves the overall result kernel execution call cpu control can work here! kernel execution gpu ready event 51

52 Simultaneous / simulating While runs a kernel can work too Increases the tree depth, improves the overall result kernel execution call cpu control can work here! kernel execution gpu ready event 51

53 Simultaneous / simulating While runs a kernel can work too Increases the tree depth, improves the overall result kernel execution call cpu control can work here! kernel execution gpu ready event 51

54 Simultaneous / simulating While runs a kernel can work too Increases the tree depth, improves the overall result kernel execution call cpu control can work here! kernel execution gpu ready event 51

55 Simultaneous / simulating While runs a kernel can work too Increases the tree depth, improves the overall result kernel execution call cpu control can work here! kernel execution gpu ready event 51

56 Simultaneous / simulating While runs a kernel can work too Increases the tree depth, improves the overall result kernel execution call Processing cpu control can work here! kernel execution gpu ready event 51

57 Simultaneous / simulating While runs a kernel can work too Increases the tree depth, improves the overall result kernel execution call Processing cpu control can work here! kernel execution gpu ready event 51

58 Simultaneous / simulating While runs a kernel can work too Increases the tree depth, improves the overall result kernel execution call Processing cpu control can work here! kernel execution gpu ready event 51

59 Simultaneous / simulating While runs a kernel can work too Increases the tree depth, improves the overall result kernel execution call Processing cpu control can work here! kernel execution gpu ready event 51

60 Simultaneous / simulating While runs a kernel can work too Increases the tree depth, improves the overall result kernel execution call Processing cpu control can work here! kernel execution gpu ready event 51

61 Simultaneous / simulating While runs a kernel can work too Increases the tree depth, improves the overall result kernel execution call Simulating Processing cpu control can work here! kernel execution gpu ready event 51

62 Simultaneous / simulating While runs a kernel can work too Increases the tree depth, improves the overall result kernel execution call Simulating Processing cpu control can work here! kernel execution gpu ready event 51

63 Simultaneous / simulating While runs a kernel can work too Increases the tree depth, improves the overall result kernel execution call Processing cpu control can work here! kernel execution gpu ready event 51

64 Simultaneous / simulating While runs a kernel can work too Increases the tree depth, improves the overall result kernel execution call Processing cpu control can work here! kernel execution gpu ready event 51

65 Simultaneous / simulating While runs a kernel can work too Increases the tree depth, improves the overall result kernel execution call Processing cpu control can work here! kernel execution gpu ready event 51

66 Simultaneous / simulating While runs a kernel can work too Increases the tree depth, improves the overall result kernel execution call Processing cpu control can work here! kernel execution gpu ready event 51

67 Simultaneous / simulating While runs a kernel can work too Increases the tree depth, improves the overall result kernel execution call Processing cpu control can work here! kernel execution Simulating gpu ready event 51

68 Simultaneous / simulating While runs a kernel can work too Increases the tree depth, improves the overall result kernel execution call Processing cpu control can work here! kernel execution Simulating gpu ready event 51

69 Simultaneous / simulating While runs a kernel can work too Increases the tree depth, improves the overall result kernel execution call Processing cpu control can work here! kernel execution gpu ready event 51

70 Simultaneous / simulating While runs a kernel can work too Increases the tree depth, improves the overall result kernel execution call Processing cpu control can work here! kernel execution gpu ready event 51

71 Simultaneous / simulating While runs a kernel can work too Increases the tree depth, improves the overall result kernel execution call Processing cpu control can work here! kernel execution gpu ready event 51

72 Simultaneous / simulating While runs a kernel can work too Increases the tree depth, improves the overall result kernel execution call Processing cpu control can work here! kernel execution gpu ready event 51

73 Simultaneous / simulating While runs a kernel can work too Increases the tree depth, improves the overall result kernel execution call Processing cpu control can work here! kernel execution Simulating gpu ready event 51

74 Simultaneous / simulating While runs a kernel can work too Increases the tree depth, improves the overall result kernel execution call Processing cpu control can work here! kernel execution Simulating gpu ready event 51

75 Simultaneous / simulating While runs a kernel can work too Increases the tree depth, improves the overall result kernel execution call Processing cpu control can work here! kernel execution gpu ready event 51

76 Simultaneous / simulating While runs a kernel can work too Increases the tree depth, improves the overall result kernel execution call cpu control can work here! kernel execution gpu ready event 51

77 Simultaneous / simulating While runs a kernel can work too Increases the tree depth, improves the overall result kernel execution call Finished cpu control can work here! kernel execution gpu ready event 51

78 Simultaneous / simulating While runs a kernel can work too Increases the tree depth, improves the overall result kernel execution call cpu control can work here! kernel execution gpu ready event 51

79 Simultaneous / simulating While runs a kernel can work too Increases the tree depth, improves the overall result kernel execution call cpu control can work here! kernel execution gpu ready event 51

80 Simultaneous / simulating While runs a kernel can work too Increases the tree depth, improves the overall result kernel execution call cpu control can work here! kernel execution gpu ready event 51

81 Simultaneous / simulating While runs a kernel can work too Increases the tree depth, improves the overall result kernel execution call cpu control can work here! kernel execution gpu ready event 51

82 Simultaneous / simulating While runs a kernel can work too Increases the tree depth, improves the overall result kernel execution call cpu control can work here! kernel execution gpu ready event 51

83 Simultaneous / simulating While runs a kernel can work too Increases the tree depth, improves the overall result kernel execution call cpu control can work here! kernel execution gpu ready event 51

84 Simultaneous / simulating While runs a kernel can work too Increases the tree depth, improves the overall result Processing kernel execution call cpu control can work here! kernel execution gpu ready event 51

85 Simultaneous / simulating While runs a kernel can work too Increases the tree depth, improves the overall result Processing kernel execution call cpu control can work here! kernel execution gpu ready event 51

86 Simultaneous / simulating While runs a kernel can work too Increases the tree depth, improves the overall result Processing kernel execution call cpu control can work here! kernel execution gpu ready event 51

87 Simultaneous / simulating While runs a kernel can work too Increases the tree depth, improves the overall result Processing kernel execution call cpu control can work here! kernel execution gpu ready event 51

88 Simultaneous / simulating While runs a kernel can work too Increases the tree depth, improves the overall result Processing kernel execution call cpu control can work here! kernel execution gpu ready event 51

89 Simultaneous / simulating While runs a kernel can work too Increases the tree depth, improves the overall result Processing kernel execution call Simulating cpu control can work here! kernel execution gpu ready event 51

90 Simultaneous / simulating While runs a kernel can work too Increases the tree depth, improves the overall result Processing kernel execution call Simulating cpu control can work here! kernel execution gpu ready event 51

91 Simultaneous / simulating While runs a kernel can work too Increases the tree depth, improves the overall result Processing kernel execution call cpu control can work here! kernel execution gpu ready event 51

92 Simultaneous / simulating While runs a kernel can work too Increases the tree depth, improves the overall result Processing kernel execution call cpu control can work here! kernel execution gpu ready event 51

93 Simultaneous / simulating While runs a kernel can work too Increases the tree depth, improves the overall result Processing kernel execution call cpu control can work here! kernel execution gpu ready event 51

94 Simultaneous / simulating While runs a kernel can work too Increases the tree depth, improves the overall result Processing kernel execution call cpu control can work here! kernel execution gpu ready event 51

95 Simultaneous / simulating While runs a kernel can work too Increases the tree depth, improves the overall result Processing kernel execution call Simulating cpu control can work here! kernel execution gpu ready event 51

96 Simultaneous / simulating While runs a kernel can work too Increases the tree depth, improves the overall result Processing kernel execution call Simulating cpu control can work here! kernel execution gpu ready event 51

97 Simultaneous / simulating While runs a kernel can work too Increases the tree depth, improves the overall result Processing kernel execution call cpu control can work here! kernel execution gpu ready event 51

98 Simultaneous / simulating While runs a kernel can work too Increases the tree depth, improves the overall result Processing kernel execution call cpu control can work here! kernel execution gpu ready event 51

99 Simultaneous / simulating While runs a kernel can work too Increases the tree depth, improves the overall result Processing kernel execution call cpu control can work here! kernel execution gpu ready event 51

100 Simultaneous / simulating While runs a kernel can work too Increases the tree depth, improves the overall result Processing kernel execution call cpu control can work here! kernel execution gpu ready event 51

101 Simultaneous / simulating While runs a kernel can work too Increases the tree depth, improves the overall result Processing kernel execution call cpu control can work here! kernel execution Simulating gpu ready event 51

102 Simultaneous / simulating While runs a kernel can work too Increases the tree depth, improves the overall result Processing kernel execution call cpu control can work here! kernel execution Simulating gpu ready event 51

103 Simultaneous / simulating While runs a kernel can work too Increases the tree depth, improves the overall result Processing kernel execution call cpu control can work here! kernel execution gpu ready event 51

104 Simultaneous / simulating While runs a kernel can work too Increases the tree depth, improves the overall result kernel execution call cpu control can work here! kernel execution gpu ready event 51

105 Simultaneous / simulating While runs a kernel can work too Increases the tree depth, improves the overall result Finished kernel execution call cpu control can work here! kernel execution gpu ready event 51

106 Simultaneous / simulating While runs a kernel can work too Increases the tree depth, improves the overall result kernel execution call cpu control can work here! kernel execution gpu ready event 51

107 Simultaneous / simulating While runs a kernel can work too Increases the tree depth, improves the overall result kernel execution call cpu control can work here! kernel execution gpu ready event 51

108 Simultaneous / simulating While runs a kernel can work too Increases the tree depth, improves the overall result kernel execution call cpu control can work here! kernel execution gpu ready event 51

109 Simultaneous / simulating While runs a kernel can work too Increases the tree depth, improves the overall result kernel execution call cpu control can work here! kernel execution gpu ready event 51

110 Simultaneous / simulating While runs a kernel can work too Increases the tree depth, improves the overall result Processed by kernel execution call cpu control can work here! kernel execution gpu ready event 51

111 Simultaneous / simulating While runs a kernel can work too Increases the tree depth, improves the overall result Processed by kernel execution call cpu control can work here! kernel execution gpu ready event 51

112 Simultaneous / simulating While runs a kernel can work too Increases the tree depth, improves the overall result kernel execution call Processed by Processed by cpu control can work here! kernel execution gpu ready event 51

113 Simultaneous / simulating - results 14 Points Average point difference (score) 1 vs 128 s Game step Depth Average tree depth Game step 52

114 MCTS Possible Scalability Limitations Analysis 53

115 MPI Parallel Scheme Process number 0 controls the game N processes init Other machine i.e. core i7, Fedora Other machine i.e. Phenom, Ubuntu Receive the opponent s move Input data Send the current state of the game to all processes Root process id = 0 n-1 processes broadcast data Think simulate Network simulate simulate All simulations are independent Accumulate results Output data collect data (reduce) Choose the best move and send it to the opponent 54

116 Multi- results First Conclusions Simulations/second No communication bottleneck ~20mln sim/s Average Point Difference ,376 threads Improvement gets worse No of s (112 block x 64 Threads) 55

117 Scalability analysis Findings: Weak scaling of the algorithm - problem s complexity affects the scalability Exploitation/exploration ratio - higher exploitation needed for more trees No communication bottleneck Much more efficient than the version 56

118 Sampling with replacement Probability of having exactly x distinct simulations after n simulations, where m is the total number of possible combinations P(m,n, x) = m n 1 x n x m + n 1 n n D(m,n) x P(m,n, x) x 1 D(m,n) - expected number of distinct samples Small problem size : increased number of repeated samples 57

119 Sampling with replacement If the state-space is small, the impact of the parallelism will be diminished Low problem complexity The problem is simple itself (i.e. small instances of SameGame, TicTacToe) Ending phases of games/puzzles (few steps ahead) Problem s size decreases Depth Game step 58

120 Scalability analysis 50 score difference (1p vs 2p) 256 s 3,670,016 threads and 2048 threads vs sequential MCTS Score Point difference Parallel - Sequential TSUBAME 2.0 TESLA s 0 Losing if below Game step 59

121 Exploration/exploitation in parallel MCTS Trees High exploitation SUM Trees High exploration SUM 60

122 Problems/proposed solutions Problem Solution - my contribution architecture Block parallelism, fast random sequences generation (not presented here) Latency, slow iterations Hybrid - processing Low efficient leaf parallel part of block parallelism Variance based error estimation for decision making Scalability unknown MPI and Implementation and analysis 61

123 How universal and important is the MCTS block-parallel algorithm? MCTS has many applications already New ones are appearing The architecture is likely to follow the trend in the future Programming s may become easier, rather not harder 62

124 Current challenges Further investigation on large scale tree search New applications of MCTS, i.e. TSP New AI and optimization algorithms for Multiple threads per in block parallelism GTC 2010 presentation: Playing Zero-Sum Games on the, NVIDIA, Tic tac toe, Sudoku, 63

125 64

126 Publications Kamil Rocki, Reiji Suda, "Parallel Minimax Tree Searching on ", PPAM 2009 Eighth International Conference on Parallel Processing and Applied Mathematics, Wroclaw, Poland, 15 Sep. (13-16 Sep.), Kamil Rocki, Reiji Suda, "Massively Parallel Monte Carlo Tree Search", 2010 VECPAR Conference, Berkeley, CA (USA) June 22-25, 2010 Kamil Rocki, Reiji Suda, Improving the parallel Monte Carlo Tree Search performance by the standard deviation based error estimation, 3rd International Conference on Machine Learning and Computing (ICMLC 2011) Singapore, February 26-28, 2011 Kamil Rocki, Reiji Suda, "MPI- Monte Carlo Tree Search", IEEE 2011 International Conference on Information and Computer Applications (ICICA), Dubai, UAE, March Mar 2011 Kamil Rocki, "Large-Scale Parallel Monte Carlo Tree Search on ", PhD Forum, 25th IEEE IPDPS, May 16-20, 2011, Anchorage, USA Kamil Rocki, Reiji Suda, "Parallel Monte Carlo Tree Search on ", 11th Scandinavian Conference on Artificial Intelligence, Norwegian University of Science and Technology, Trondheim, Norway May 24th - 26th, 2011 (nomination for the best paper, automatically submitted to a journal, in progress) Kamil Rocki, Reiji Suda, "Parallel Monte Carlo Tree Search Scalability Discussion", 24th Australasian Joint Conference on Artificial Intelligence, 5-8, December, 2011, Perth, Australia 65