Towards Load Balanced Distributed Transactional Memory

Transcription

1 Towards Load Balanced Distributed Transactional Memory Gokarna Sharma and Costas Busch Louisiana State University Euro-Par 12, August 31, 2012

2 Distributed Transactional Memory (DTM) Transactions run on network nodes They ask for shared objects distributed over the network for either read or write The reads and writes on shared objects are supported through three operations: Publish Lookup Move

3 Suppose the object ξ is at node and is a requesting node Requesting node Predecessor node ξ Data-flow model: transactions are immobile and the objects are mobile

4 Lookup operation Read-only copy ξ Main copy ξ Replicates the object to the requesting node

5 Lookup operation Read-only copy ξ Read-only copy ξ Main copy ξ Replicates the object to the requesting nodes

6 Move operation Main copy ξ Invalidated ξ Relocates the object explicitly to the requesting node

7 Move operation Main copy ξ Invalidated ξ Invalidated ξ - Relocates the object explicitly to the requesting node - Invalidates also the read-only copies (if available)

8 General routing: choose paths from sources to destinations u 2 u 1 v 3 v 2 u 3 v 1 Routing in DTM: source node of the predecessor request in the total order is the destination of a successor request

9 Edge congestion Node congestion C edge C node maximum number of paths that use any edge maximum number of paths that use any node

10 Stretch = Length of chosen path Length of shortest path stretch u shortest path v chosen path

11 Oblivious Routing Each request path choice is independent of other request path choices

12 Problem Statement Given a d-dimensional mesh and a finite set of operations R ={r 0,r 1,,r l } on an object ξ Design a DTM algorithm that: Minimizes congestion C = max e {i : p i e} on any edge e Minimizes total communication cost A(R) = σ i=1 for all the operations l pi

13 Related Work Protocol Stretch Network Kind Runs on Arrow [DISC 98] Relay [OPODIS 09] Combine [SSS 10] Ballistic [DISC 05] Spiral [IPDPS 12] O(S ST )=O(D) General Spanning tree O(S ST )=O(D) General Spanning tree O(S OT )=O(D) General Overlay tree O(log D) Constant-doubling dimension Hierarchical directory with independent sets O(log 2 n log D) General Hierarchical directory with sparse covers D is the diameter of the network kind S * is the stretch of the tree used

14 Limitations and Motivations These protocols only minimize stretch and they cannot control congestion Congestion can also be a major bottleneck may affect the overall performance of the algorithm A natural question is whether stretch and congestion can be controlled simultaneously Congestion and stretch can not be minimized simultaneously in arbitrary networks

15 Our Contributions MultiBend DTM algorithm for mesh networks For 2-dimensional mesh, MultiBend has both stretch and (edge) congestion O(log n) For d-dimensional mesh, MultiBend has stretch O(d log n) and (edge) congestion O(d 2 log n) For fixed d, stretch is within O(log log n) factor and congestion is within O(1) factor far from optimal

16 In the Remaining Model General Approach Analogy to a Distributed Queue Hierarchical decomposition for MultiBend MultiBend Analysis Stretch Congestion Discussion

17 Model Mesh network G = (V,E) of n reliable nodes One shared object Nodes receive-compute-send atomically Nodes are uniquely identified Node u can send to node v if it knows v One node executes one request at a time

18 General Approach

19 Hierarchical clustering Network graph

20 Hierarchical clustering Alternative representation as a hierarchy tree with leader nodes

21 At the lowest level (level 0) every node is a cluster Directories at each level cluster, downward pointer if object locality known

22 A Publish operation root Owner node ξ ξ Assume that is the creator of which invokes the Publish operation Nodes know their parent in the hierarchy

23 Send request to the leader root

24 Continue up phase root Sets downward pointer while going up

25 Continue up phase root Sets downward pointer while going up

26 Root node found, stop up phase root

27 root Predecessor node ξ A successful Publish operation

28 Supporting a Move operation root Requesting node Predecessor node ξ Initially, nodes point downward to object owner (predecessor node) due to Publish operation Nodes know their parent in the hierarchy

29 Send request to leader node of the cluster upward in hierarchy root

30 Continue up phase until downward pointer found root Sets downward path while going up

31 Continue up phase root Sets downward path while going up

32 Continue up phase root Sets downward path while going up

33 Downward pointer found, start down phase root Discards path while going down

34 Continue down phase root Discards path while going down

35 Continue down phase root Discards path while going down

36 Predecessor reached, object is moved from node to node root Lookup is similar without change in the directory structure and only a read-only copy of the object is sent

37 Distributed Queue Analogy

38 Distributed Queue root u head tail u

39 Distributed Queue root u v head tail v u

40 Distributed Queue root u v w head tail v u w

41 Distributed Queue root v w head tail v u w

42 Distributed Queue root w head tail v u w

43 Results on Mesh Networks

44 Type-1 Mesh Decomposition 2-dimensional mesh

45 Type-1 Mesh Decomposition

46 Type-1 Mesh Decomposition

47 Type-2 Mesh Decomposition

48 Type-2 Mesh Decomposition

49 Decomposition for 2 3 x2 3 2-dimensional mesh (i+1,2) (i+1,1) (i,2) (i,1) Hierarchy levels

50 MultiBend Hierarchy Find a predecessor node via multi-bend paths for each leaf node u by visiting leaders of all the clusters that contain u from level 0 to the root level root u p(u) v p(v)

51 MultiBend Hierarchy (2) The hierarchy guarantees: (1) For any two nodes u,v, their multi-bend paths p(u) and p(v) meet at level min{h, log(dist(u,v))+2} root (2) length(p i (u)) is at most 2 i+3 u p(u) v p(v)

52 (Canonical) downward Paths root root u p(u) v p(v) u p(u) p(v) is a (canonical) downward path

53 Load Balancing Through a leader election procedure Every time we access the leader of a sub-mesh, we replace it with another leader chosen uniformly at random among its nodes The directory is updated appropriately by updating parent and child leaders Locking may needed in concurrent executions The update cost is low in comparison to the cost of serving requests This step is necessary to control congestion With fixed leader, edge congestion can be O(l), the number of requests If congestion requirement can be relaxed by a factor of ρ, the leader change is needed after every ρ requests

54 Analysis of MultiBend

55 Analysis on (move) Stretch Assume a sequential execution R of l+1 Move requests, where r 0 is an initial Publish request. A*(R) max 1 k h (S k -1) 2 k-1 A(R) σ h k=1 (Sk 1) 2 k+3 h... k Level r 0... r 0... r 0 r 0 r 0 r 1.. r 1 r 1 r 1 r 2 r 2 r 2.. r 2 r 2 r 2... r l-1 r l-1 r l-1 r 2.. r l... r l r l r l Thus, x u v y w request C(R)/C*(R) = σ h k=1 (Sk 1) 2 k+3 / max 1 k h (S k -1) 2 k-1 = 16 h max 1 k h (S k -1) 2 k-1 / max 1 k h (S k -1) 2 k-1 = O(log n)

56 Analysis on (Edge) Congestion A sub-path uses edge e with probability 2/m l P : set of paths from M 1 to M 2 or vice-versa C (e): Congestion caused by P on e M 2 E[C (e)] 2 P /m l B P /out(m 1 ) e M 1 out(m 1 ) 4m l C* B ==> E[C (e)] 8C* m l Assume M 1 is a type-1 submesh

57 Analysis on (Edge) Congestion (2) As M 1 at level (i,2) is always completely contained in M 2 at level (i+1,2) log n +2 levels E[C(e)] 8C*(log n + 2) Considering type-2 submeshes exactly one type-2 submesh between every two type-1 submeshes the type-2 submeshes may not be proper subset of type-1 submeshes 4 possible type-2 submesh choices (path may bend at most 2 times) Increases the load by a factor of 4 Thus, using standard Chernoff bound, C = O(C* log n), w.h.p.

58 d-dimensional Mesh

59 Extensions to d-dimensional mesh 2-dimesional decomposition can be directly generalized to a d-dimensional mesh Problem: the congestion become O(2 d log n) Another decomposition is used to control congestion in O(C* d 2 log n) and stretch O(d log n) We set appropriate λ and shift the type-1 submeshes by (j-1)λ nodes in each dimension to get type-j submeshes

60 Extensions to d-dimensional mesh (2) 3-dimensional mesh decomposition. Only 2 of the 3 dimensions are shown O(d)-types of submeshes on each level O(d) sub-levels in every level with type-1 submesh first and type-o(d) submesh last in the order

61 Summary First load balanced distributed directory protocol with both stretch and congestion O(log n) in 2-dimensional mesh In d-dimensional mesh, stretch O(d log n) and (edge) congestion O(d 2 log n) MultiBend is starvation-free and provides linearizability. Future work: extend the results to dynamic networks make it fault-tolerant

62 Thank you for your attention!