Fast methods for N-body problems

Transcription

1 CME342- Parallel Methods in Numerical Analysis Fast methods for N-body problems May 21, 2014 Lecture 13

2 Outline Motivation: N-body problem is O(N^2) Data structures: quad tree and oct tree (ADT?) The Barnes-Hut algorithm: An O(N log N) approximate algorithm for the N-body problem The Fast Multipole method: An O(N ) approximate algorithm for the N-body problem Parallelization issues

3 Particle simulations Particle simulations are used in: Computational Fluid Dynamics: vortex methods Molecular Dynamics, plasma simulations: electrostatic force Astrophysics: gravitational force Computer graphics: radiosity computation t = 0! while t < t_final! for i = 1 to n n = number of particles! compute f(i) = force on particle i! for i = 1 to n! move particle i under force f(i) for time dt using F=ma! compute interesting properties of particles (energy, etc.)! t = t + dt! end while!

4 Applications Computational Fluid Dynamics: vortex methods using FMM (Koumoutsakos)

5 Spray modeling Applications

6 Applications From a computer simulation of the formation of large scale structures in the Universe. The image shows 287,865,861 individual bodies (Warren and Salmon)

7 Particle simulations t = 0! while t < t_final! for i = 1 to n n = number of particles! compute f(i) = force on particle i! for i = 1 to n! move particle i under force f(i) for time dt using F=ma! compute interesting properties of particles (energy, etc.)! t = t + dt! end while! force = F external + F neighbor + F farfield O( N) O( N) O( N 2 ) Using a classical approach, this will require O(N^2) operations, but with a clever divide and conquer approach we can solve the problem in O(N logn) or O(N) operations

8 Particles simulations force = F external + F neighbor + F farfield p O(N) φ (z) = M log(z z ) + β ( j) O(N) c j=1 (z z c ) j External/ Body forces: external magnetic or electric field Can be computed independently Nearest neighbor force: Van der Waals Only requires interaction with the nearest neighbor Far field force: all to all interaction Expensive to compute, the force depends on all other particles

9 Fast hierarchical methods Use of adaptive quadtree or octree to subdivide the space Barnes-Hut algorithm: can solve the problem in O(N logn) not very accurate use the center of mass and total mass Fast Multipole method : can solve the problem in O(N) accurate use more information than BH more difficult to implement

10 Particle simulations Using a classical approach, they require O(N^2) operations, but with a clever divide and conquer approach we can solve the problem in O(NlogN) or O(N) operations Consider computing the force on earth due to all celestial bodies, we can reduce the number of particles in the force sum approximating all stars in a certain galaxy by a macroparticle located at its center of mass with the same total mass D = size of box containing Andromeda, r = distance of CM to Earth Require that D/r be small enough

11 Particle simulations Newton did this at the end of 17th century. What is new is to use this idea recursively. As long as D1/r1 is small enough, stars inside smaller box can be replaced by their CM to compute the force. Boxes nest in boxes recursively

12 Quad tree Data structure to subdivide the plane Nodes can contain coordinates of center of box, side length Eventually also coordinates of CM, total mass, etc. In a complete quad tree, each non leaf node has 4 children

13 Oct tree Data structure to subdivide the space In a complete oct tree, each non leaf node has 8 children

14 Adaptive quad tree Most algorithms begin by constructing a tree to hold all the particles Adaptive Quad (Oct) Tree only subdivides space where particles are located Children nodes enumerated counterclockwise from SE corner, empty ones excluded

15 Procedure QuadTreeBuild! QuadTree = {emtpy}! for j = 1 to N Adaptive quad tree loop over all N particles! QuadTreeInsert(j, root) insert particle j in QuadTree! endfor! At this point, the QuadTree may have some empty leaves,! whose siblings are not empty! Traverse the QuadTree eliminating empty leaves via, say Breadth First Search!! Procedure QuadTreeInsert(j, n)! Try to insert particle j at node n in QuadTree! By construction, each leaf will contain either 1 or 0 particles! if the subtree rooted at n contains more than 1 particle n is an internal node! determine which child c of node n contains particle j! QuadTreeInsert(j, c)! else if the subtree rooted at n contains 1 particle n is a leaf! add n s 4 children to the QuadTree! move the particle already in n into the child containing it! let c be the child of n containing j! QuadTreeInsert(j, c)! else the subtree rooted at n is an empty leaf! store particle j in node n! end! Cost <= N * maximum cost of QuadTreeInsert = O(N * maximum depth of QuadTree) Uniform distribution => depth of QuadTree = O(log N), so Cost = O( N log N) Arbitrary distribution => depth of Quad Tree = O(b) = O(# bits in particle coords), so Cost = O(bN)

16 Parallel ADT search library Used in SUmb for arbitrary geometric searches in distributed grids Application program only provides its local grid information and an MPI communicator to the library. Interpolation data and/or interpolation coefficients are returned; the coefficients are interpolation weights, the corresponding element ID and type and the processor ID in the group of the MPI-communicator. All processes which belong to the communicator should call the APIs even if no local information is present/needed. An arbitrary number of ADT s can be stored in the data structures Support for coupling any number of single-discipline simulation codes. Designed to avoid memory bottlenecks Target platform Blue Gene/L for the full simulation. Limited amount of memory per processor. Even surface grids cannot be gathered on one processor anymore (as was done for wall distance searches and sliding mesh interfaces.)

17 Parallel ADT search library Building of the local trees Create the bounding box of every local element; this bounding box can be represented by a point in 6D. Build the 6-dimensional ADT as shown in the figure. Use a logical rather than a geometric split Figure from Aftosmis VKI LS notes Advantage: degenerate trees are avoided and the amount of memory is known a priori. Disadvantage: a sort operation is needed every time a leaf is split building the tree is more complicated.

18 Parallel ADT search library Building of a global tree The root leaves of every processor are gathered on all processors The geometric range of every local ADT is known on all processors. Possible target ADTs can be determined for every point to be searched. Any processor is able to search trees that may reside in different processors transparently. Does this pose potential CPU scalability bottlenecks? Work in progress.

19 Parallel ADT search library Searching a global tree Determine, for every point to be searched, the local target trees and determine the number of local trees to be searched for the current set of points. Make this number available to all processors Every processor can determine the number of points it has to search in the local tree. Determine the number of search rounds to avoid a possible memory bottleneck. Determine the sending and receiving processors for every search round. Loop over the search rounds MPI_Alltoall to request data from processors. Send coordinates to target processors. Perform the interpolation in the local tree to overlap between communication and computation. Loop over the number of processors I need to interpolate data in my local tree Receive the coordinates. Interpolate the data in the local tree. Send interpolation data back to the requesting processor. Loop over the number of processors to which I send coordinates. Receive the interpolation data.

20 Parallel ADT search library Searching a local tree (containment search) Loop over the number of points to be searched. Traverse the tree and store the leaves which contain the point. At the base of the tree this gives a limited number of bounding boxes that contain the point. Loop over the corresponding volume elements. Determine if the element contains the point. Usually a Newton algorithm is needed, except for tetrahedra. If the element contains the point, store the interpolation weights and perform the interpolation of the variables (if requested)

21 Parallel ADT search library Applications. SUmb, sliding interpolation. SUmb, wall distance search. CHIMPS, volume interpolation between an arbitrary number of application codes. OOGA, volume interpolation for overset meshes. NASA Ames solution interpolation between mismatched meshes (Shuttle investigation.) Others? Future improvements. Efficiency of minimum distance search, especially wall distance search, should be improved.

22 Barnes-Hut algorithm A Hierarchical O(n log n) force calculation algorithm, J. Barnes and P. Hut, Nature, v. 324 (1986), many later papers Good for low accuracy calculations: RMS error = (Σ k approx f(k) - true f(k) 2 / true f(k) 2 /N) 1/2 ~ 1% 1) Build the QuadTree using QuadTreeBuild! already described, cost = O( N log N) or O(b N)! 2) For each node = subsquare in the QuadTree, compute the! CM and total mass (TM) of all the particles it contains! post order traversal of QuadTree, cost = O(N log N) or O(b N)! 3) For each particle, traverse the QuadTree to compute the force on it,! using the CM and TM of distant subsquares! core of algorithm! cost depends on accuracy desired but still O(N log N) or O(bN)!

23 Compute CM and total mass of each node Compute the CM = Center of Mass and TM = Total Mass of all the particles in each node of the QuadTree ( TM, CM ) = Compute_Mass( root ) function ( TM, CM ) = Compute_Mass( n ) compute the CM and TM of node n if n contains 1 particle the TM and CM are identical to the particle s mass and location store (TM, CM) at n return (TM, CM) else post order traversal : process parent after all children for all children c(j) of n j = 1,2,3,4 ( TM(j), CM(j) ) = Compute_Mass( c(j) ) endfor TM = TM(1) + TM(2) + TM(3) + TM(4) the total mass is the sum of the children s masses CM = ( TM(1)*CM(1) + TM(2)*CM(2) + TM(3)*CM(3) + TM(4)*CM(4) ) / TM the CM is the mass-weighted sum of the children s centers of mass store ( TM, CM ) at n return ( TM, CM ) end if Cost = O(# nodes in QuadTree) = O( N log N ) or O(b N)

24 Compute force on each particle For each node = square, we can approximate force on particles outside the node due to particles inside node by using the node s CM and TM This will be accurate enough if the node is far away enough from the particle For each particle, use as few nodes as possible to compute force, subject to accuracy constraint Need criterion to decide if a node is far enough from a particle D = side length of node r = distance from particle to CM of node q = user supplied error tolerance < 1 Use CM and TM to approximate force of node on box if D/r < q

25 Fast Multipole Method A fast algorithm for particle simulation, L. Greengard and V. Rokhlin, J. Comp. Phys. V. 73, 1987, many later papers Differences from Barnes-Hut FMM computes the potential at every particle, not just the force FMM uses more information in each box than just the CM and TM, so it is both more accurate and more expensive To compensate, FMM accesses a fixed set of boxes at every level, independent of D/r BH uses fixed information (CM and TM) in every box, but # boxes increases with accuracy. FMM uses a fixed # boxes, but the amount of information per box increases with accuracy.

26 Fast Multipole method FMM uses two kinds of expansions Outer expansions represent potential outside a node due to particles inside another one, analogous to (CM,TM) Inner expansions represent potential inside node due to particles outside; Computing this for every leaf node is the computational goal of FMM Force on particle at (x,y,z) due to one at origin = -m1 m2(x,y,z)/r2 Instead of force, consider φ(x,y,z)=-1/r Force =-grad φ(x,y,z)=-(d φ /dx, d φ /dy, d φ /dz) FMM will compute a compact expression for φ(x,y,z), which can be evaluated and/or differentiated at any point

27 Fast Multipole method Use the potential instead of the force FMM uses two kinds of expansions Outer expansions represent potential outside node due to particles inside, analogous to (CM,TM) Inner expansions represent potential inside a node due to particles outside; Computing these for every leaf node is the computational goal of FMM

28 Fast Multipole method For simplicity, we will discuss the 2D version. In 3D, can use spherical harmonics or other expansions Force on particle at (x,y) due to one at origin = F=-(x,y)/r2= -z / z 2 where z = x + iy (complex number) Potential φ(x,y)=-1/r =log z equivalent to gravity between infinite parallel wires instead of point masses We can write a 2D multipole expansion of φ that is accurate when r is large: φ(z) = M log z + j=1 (Outer expansion) α( j) z j N j i α( j) = z m i j i=1

29 Error in multipole expansion error( r ) = O( {max k z k / z }p+1 ) bounded by geometric sum If z >c*max z k, then error( r ) = O(1/cp+1) Suppose all the particles z k lie inside a D- by-d square centered at the origin Suppose z is outside a 3D-by-3D square centered at the origin c = 1.5*D/(D/sqrt(2)) ~ 2.12 Each extra term in expansion increases accuracy about 1 bit 24 terms enough for single,53 terms for double φ(z) = M log z + p j=1 α( j) z j

30 Outer expansion p φ (z) = M log(z z ) + α 1 ( j) (z z 1 ) j Outer(n) = (M, α 1, α 2,, α r, z n ) Stores data for evaluating potential φ(z) outside node n due to particles inside n Will be computed for each node in QuadTree Cost of evaluating φ(z) is O( p ), independent of the number of particles inside n Outer expansion needed to compute the potential away from n due to particles inside n. We also want to compute potential inside n due to particles away from n j =1

31 Inner expansion Inner(n) = ( β 1, β 2,, β r, z n ) Stores data for evaluating potential φ(z) inside node n due to particles outside n φ (z) = c p i=0 β(i) (z z c ) i Computing this for every leaf node is the computational goal of FMM

32 Converting Outer Expansions We want to compute Outer(n) cheaply from Outer( c ) for all children of n How to combine outer expansions around different points? First step: make them expansions around same point n 1 is a subsquare of n 2, z k = center(n k ) for k=1,2 Outer(n 1 ) expansion accurate outside blue dashed square, so also accurate outside black dashed square So there is an Outer(n 2 ) expansion with different a k and center z 2 which represents the same potential as Outer(n 1 ) outside the black dashed box

33 Converting outer expansions Given Solve for M2 p φ (z) = M log(z z ) + α 1 ( j) (z z 1 ) j and α 2 in φ (z) φ 1 2 (z) = 2 Get M2 = M 1 and each α 2 is a linear combination of the α 1 multiply r-vector of α 1 values by a fixed r-by-r matrix to get α 2 ( M 2, α 12,, α r2 ) = Outer_shift( Outer(n 1 ), z 2 ) = desired Outer( n 2 ) j =1 p M log(z z ) + α 2 ( j) 2 (z z 2 ) j j=1

34 Converting inner expansions How to combine inner expansions around different points? First step: make them expansions around same point n 1 is a subsquare of n 2, z k = center(n k ) for k=1,2 Inner(n 2 ) expansion accurate inside black dashed square, so also accurate inside blue dashed square So there is an Inner(n 1 ) expansion with different β k and center z 1 which represents the same potential as Inside(n 2 ) inside the blue dashed box

35 Converting Inner Expansions Given Solve for β 2 such that p φ (z) = 1 p i=0 β 2 (i) = (z z 2 ) i i=0 β(i) (z z 1 ) i p i=0 β 1 (i) (z z 1 ) i Each β 2 is a linear combination of the β 1 multiply p-vector of α 1 values by a fixed p-by-p matrix to get β 2 (β 12,, β r2 ) = Inner_shift( Inner(n 1 ), z 2 ) = desired Inner( n 2 )

36 Converting Outer to Inner We need to show how to convert Outer(n3) to Inner(n4) when n4 is far enough from n3 Inner(n 4 ) = ( β 0, β 1,, β r, z 4 ) Outer(n 3 )=(M, α 1, α 2,, α r, z 3 ) p M log(z z ) + α 3 ( j) 3 3 j=1 (z z 3 ) j = p i=1 β 4 (i) (z z 4 ) i Once again is a weighted sum Inner(n4)=Convert(Outer(n3),center(n4))

37 Top Level Description of FMM 1) Build the QuadTree! 2) Call Build_Outer(root), to compute outer expansions! of each node n in the QuadTree! Traverse QuadTree from bottom to top,! combining outer expansions of children! to get out outer expansion of parent! 3) Call Build_ Inner(root), to compute inner expansions! of each node n in the QuadTree!! Traverse QuadTree from top to bottom,! converting outer to inner expansions! and combining them! 4) For each leaf node n, add contributions of nearest particles! directly into Inner(n)! final Inner(n) is desired output: expansion for potential at! each point due to all particles!

38 Compute Outer(n) for each node Compute Outer(n) for each node of the QuadTree! outer = Build_Outer( root )!! function ( M, a 1,,a r, z n ) = Build_Outer( n ) compute outer expansion of node n! if n if a leaf it contains 1 (or a few) particles! compute and return Outer(n) = ( M, a 1,,a r, z n ) directly from! its definition as a sum! else post order traversal : process parent after all children! Outer(n) = 0! for all children c(k) of n k = 1,2,3,4! Outer( c(k) ) = Build_Outer( c(k) )! Outer(n) = Outer(n) +! Outer_shift( Outer(c(k)), center(n))! just add component by component! endfor! return Outer(n)! end if! Cost = O(# nodes in QuadTree) = O( N ) same as for Barnes-Hut

39 Compute Inner from other expansions We will use Inner_shift and Convert to build each Inner(n) by combing expansions from other nodes Which other nodes? As few as necessary to compute the potential accurately Inner_shift(Inner(parent(n), center(n)) will account for potential from particles far enough away from parent (red nodes below) Convert(Outer(i), center(n)) will account for potential from particles in boxes at same level in Interaction Set

40 Interaction set Interaction Set = { nodes i that are children of a neighbor of parent(n), such that i is not itself a neighbor of n} For each i in Interaction Set, Outer(i) is available, so that Convert(Outer(i),center(n)) gives contribution to Inner(n) due to particles in i Number of i in Interaction Set is at most = 27 in 2D Number of i in Interaction Set is at most = 189 in 3D

41 Compute Inner(n) for each node Compute Inner(n) for each node of the QuadTree! outer = Build_ Inner( root )!!! function ( b 1,,b r, z n ) = Build_ Inner( n ) compute inner expansion of node n! p = parent(n) p=nil if n = root! Inner(n) = Inner_shift( Inner(p), center(n) ) Inner(n) = 0 if p = root! for all i in Interaction_Set(n) Interaction_Set(root) is empty! Inner(n) = Inner(n) + Convert( Outer(i), center(n) )! add component by component! end for! for all children c of n complete preorder traversal of QuadTree! Build_Inner( c )! end for! Cost = O(# nodes in QuadTree) = O( N )

42 Compute nearest neighbour contribution Inner(n) includes the potential of all the particles except those in the nearest neighbour and particles inside n itself. This is a small number of particles (otherwise we would have further refined the tree), so we can use a direct sum!

43 Parallelization issues for N-body Barnes-Hut, FMM and related algorithms have similar computational structure: 1) Build the QuadTree 2) Traverse QuadTree from leaves to root and build outer expansions 3) Traverse QuadTree from root to leaves and build any inner expansions 4) Traverse QuadTree to accumulate forces for each particle Parallelization scheme: Assign regions of space to each processor Regions may have different shapes, but to get load balance Each region will have about N/p particles

44 Parallelization issues for N-body Each processor will store a part of the Quadtree containing all particles (=leaves) in its region, as well as their ancestors in the Quadtree Top of tree stored by all processors, lower nodes may also be shared Each processor will also store adjoining parts of Quadtree needed to compute forces for particles it owns Subset of Quadtree needed by a processor called the Locally Essential Tree (LET) Given the LET, all force accumulations (step 4)) are done in parallel, without communication

45 Orthogonal Recursive Bisection Recursively split region along axes into regions containing equal numbers of particles Works well for 2D, not 3D Warren and Salmon, Supercomputing 92 Partitioning for 16 procs

46 Hashed Oct Tree Instead of partitioning space, work on quadtree Estimate work for each node, call total work W Arrange nodes of QuadTree in some linear order Assign contiguous blocks of nodes with work W/p to processors Works well in 3D In shared memory environment these are called cost zones Warren and Salmon, Supercomputing 93

47 Hashed Oct Tree Arrange nodes of QuadTree in some linear order Morton ordering Assign unique key to each node in QuadTree, then compute hash(key) to get integers that can be linearly ordered If (x,y) are coordinates of center of node, interleave bits to get key Put 1 at left as sentinel Nodes at root of tree have shorter keys