SCALE OUT AND CONQUER: ARCHITECTURAL DECISIONS BEHIND DISTRIBUTED IN-MEMORY SYSTEMS VLADIMIR OZEROV YAKOV ZHDANOV

Transcription

1 SCALE OUT AND CONQUER: ARCHITECTURAL DECISIONS BEHIND DISTRIBUTED IN-MEMORY SYSTEMS VLADIMIR OZEROV YAKOV ZHDANOV

2 WHO? Yakov Zhdanov: - GridGain s Product Development VP - With GridGain since Apache Ignite committer and PMC - Passion for performance & scalability - Finding ways to make product better - St. Petersburg, Russia

3 WHY IN-MEMORY?

4 PLAN 1. Data partitioning and affinity functions examples

5 PLAN 1. Data partitioning and affinity functions examples 2. Data affinity colocation

6 PLAN 1. Data partitioning and affinity functions examples 2. Data affinity colocation 3. Synchronization in distributed systems

7 PLAN Data partitioning and affinity functions examples Data affinity colocation Synchronization in distributed systems Multithreading: local architecture

8 PLAN Data partitioning and affinity functions examples Data affinity colocation Synchronization in distributed systems Multithreading: local architecture

9 WHERE? On which node of the cluster does the key reside?

10 AFFINITY Partitio n Nod e

11 AFFINITY

12 AFFINITY Ke y Partitio n Nod e

13 AFFINITY

14 NAIVE AFFINITY

15 NAIVE AFFINITY

16 NAIVE AFFINITY

17 NAIVE AFFINITY

18 NAIVE AFFINITY

19 NAIVE AFFINITY

20 NAIVE AFFINITY Problem: partition to node mapping depends on nodes count. NODE = F (PARTITION, NODES_COUNT);

21 AFFINITY: BETTER ALGORITHMS Consistent hashing [1] Rendezvous hashing (or highest random weight - HRW) [2] [1] [2]

22 RENDEZVOUS AFFINITY WEIGHT = W(PARTITION, NODE);

23 RENDEZVOUS AFFINITY

24 RENDEZVOUS AFFINITY

25 RENDEZVOUS AFFINITY

26 RENDEZVOUS AFFINITY

27 RENDEZVOUS AFFINITY

28 RENDEZVOUS AFFINITY: EVEN DISTRIBUTION?

29 PLAN Data partitioning and affinity functions examples Data affinity colocation Synchronization in distributed systems Multithreading: local architecture

30 TRANSACTIONS: NO COLOCATION 1: class Customer { 2: long id; 3: City city; 4: }

31 TRANSACTIONS: NO COLOCATION

32 TRANSACTIONS: NO COLOCATION 2 (2 nodes)

33 TRANSACTIONS: NO COLOCATION 2 (2 nodes) 2 (primary + backup)

34 TRANSACTIONS: NO COLOCATION 2 (2 nodes) 2 (primary + backup) 2 (two-phase commit)

35 TRANSACTIONS: NO COLOCATION (2 nodes) (primary + backup) (two-phase commit) (request-response)

36 TRANSACTIONS: NO COLOCATION (2 nodes) (primary + backup) (two-phase commit) (request-response) 16 Messages

37 TRANSACTIONS: NO COLOCATION

38 TRANSACTIONS: WITH COLOCATION 1: class Customer { 2: long id; 3: 5: City city; 6: }

39 TRANSACTIONS: WITH COLOCATION

40 TRANSACTIONS: WITH COLOCATION 1 (1 node)

41 TRANSACTIONS: WITH COLOCATION 1 (1 node) 2 (primary + backup)

42 TRANSACTIONS: WITH COLOCATION 1 (1 node) 2 (primary + backup) (one-phase commit)

43 TRANSACTIONS: WITH COLOCATION 1 (1 node) 2 (primary + backup) (one-phase commit) 1 (request-response)

44 TRANSACTIONS: WITH COLOCATION 1 (1 node) 2 (primary + backup) (one-phase commit) 1 (request-response) 4 Messages

45 TRANSACTIONS: COLOCATION VS NO COLOCATION 4 Messages VS 16 Messages

46 SQL Let s run a query on our data

47 SQL No colocation: FULL SCAN

48 SQL No colocation: FULL SCAN

49 SQL No colocation: FULL SCAN 1/3x Latency

50 SQL No colocation: FULL SCAN 1/3x Latency 3x Capacity

51 SQL

52 SQL 1 node

53 SQL 1 node N nodes

54 SQL What about complexity? log 1_000_000 20

55 SQL What about complexity? log 1_000_000 vs log 333_333 log 333_333 log 333_

56 SQL: INDEXED

57 SQL No colocation: INDEXED QUERY Same latency! Same capacity!

58 SQL: INDEX AND COLOCATION Colocation: INDEXED QUERY

59 SQL Colocation: INDEXED QUERY

60 SQL: INDEX AND COLOCATION Colocation: INDEXED QUERY Same latency But 3x capacity!

61 SQL: EVEN DISTRIBUTION WITH COLOCATION?

62 SQL: JOINS IN DISTRIBUTED ENVIRONMENT

63 SQL: JOINS WITH COLOCATION

64 SQL: JOINS WITH REPLICATION

65 PLAN Data partitioning and affinity functions examples Data affinity colocation Synchronization in distributed systems Multithreading: local architecture

66 SYNCHRONIZATION: LOCAL COUNTER 1: AtomicLong ctr; 2: 3: long getnext() { 4: return ctr.incrementandget(); 5: }

67 SYNCHRONIZATION: LOCAL (RE-INVENTING A BICYCLE) 1: 2: 3: 4: 5: 6: 7: 8: 9: 10: 11: 12: 13: AtomicLong ctr; ThreadLocal<Long> localctr; long getnext() { long res = localctr.get(); if (res % 1000 == 0) res = ctr.getandadd(1000); localctr.set(++res); return res; }

68 SYNCHRONIZATION: LOCAL

69 SYNCHRONIZATION: DISTRIBUTED

70 SYNCHRONIZATION: COUNTER IN THE CLUSTER Local implementation: millions ops/sec Distributed implementation: thousands ops/sec

71 SYNCHRONIZATION: COUNTER IN THE CLUSTER Proper requirements: Unique Monotonously growing 8 bytes

72 SYNCHRONIZATION: COUNTER IN THE CLUSTER Requirements: unique, monotonous, 8 bytes.

73 SYNCHRONIZATION: COUNTER IN THE CLUSTER Requirements: unique, monotonous, 8 bytes.

74 SYNCHRONIZATION: COUNTER IN THE CLUSTER Requirements: unique, monotonous, 8 bytes.

75 SYNCHRONIZATION: COUNTER IN THE CLUSTER Requirements: unique, monotonous, 8 bytes. See also: org.apache.ignite.lang.igniteuuid

76 SYNCHRONIZATION AS FRICTION FOR A CAR

77 SYNCHRONIZATION: DATA TO CODE

78 SYNCHRONIZATION: DATA TO CODE 1: Account acc = cache.get(acckey); 3: 3: acc.add(100); 4: 5: cache.put(acckey, acc);

79 SYNCHRONIZATION: DATA TO CODE 1: Account acc = cache.get(acckey); 3: 3: acc.add(100); 4: 5: cache.put(acckey, acc);

80 SYNCHRONIZATION: CODE TO DATA

81 SYNCHRONIZATION: CODE TO DATA 1: cache.invoke(acckey, (entry) -> { 1: Account acc = entry.getvalue(); 3: 3: acc.add(100); 4: 5: entry.setvalue(acc); 6: });

82 SYNCHRONIZATION: CODE TO DATA 1: cache.invoke(acckey, (entry) -> { 1: Account acc = entry.getvalue(); 3: 3: acc.add(100); 4: 5: entry.setvalue(acc); 6: });

83 SYNCHRONIZATION: DATA TO CODE What if we have a bug?!

84 SYNCHRONIZATION: CODE TO DATA What if we have a bug?!

85 SYNCHRONIZATION: CODE TO DATA What if we have a bug?! `

86 PLAN Data partitioning and affinity functions examples Data affinity colocation Synchronization in distributed systems Multithreading: local architecture

87 LOCAL TASKS DISTRIBUTION

88 LOCAL TASKS DISTRIBUTION

89 LOCAL TASKS DISTRIBUTION

90 LOCAL TASKS DISTRIBUTION

91 LOCAL TASKS DISTRIBUTION

92 LOCAL TASKS DISTRIBUTION: THREAD PER PARTITION

93 LOCAL TASKS DISTRIBUTION: THREAD PER PARTITION

94 LOCAL TASKS DISTRIBUTION: THREAD PER PARTITION

95 LOCAL TASKS DISTRIBUTION: THREAD PER PARTITION

96 LESSONS LEARNED 1) Data partitioning: balance and stability

97 LESSONS LEARNED 1) Data partitioning: balance and stability 2) Colocation: balance and efficiency

98 LESSONS LEARNED 1) Data partitioning: balance and stability 2) Colocation: balance and efficiency 3) Data model: should be adopted accordingly

99 LESSONS LEARNED 1) 2) 3) 4) Data partitioning: balance and stability Colocation: balance and efficiency Data model: should be adopted accordingly Synchronization: delicate and only when really needed

100 LESSONS LEARNED 1) Data partitioning: balance and stability 2) Colocation: balance and efficiency 3) Data model: should be adopted accordingly 4) Synchronization: delicate and only when really needed 5) Thread per partition: can improve simple operations, but also may slow down complex ones

101 CONTACTS

102 QUESTIONS? ANY QUESTIONS?