Scalable Wikipedia with Erlang

Transcription

1 Scalable Wikipedia with Erlang Thorsten Schütt, Florian Schintke, Alexander Reinefeld Zuse Institute Berlin (ZIB) onscale solutions Thorsten Schütt 1

2 Scaling Web 2.0 Hosting Thorsten Schütt 2

3 1. Step Clients Single Server Webserver DB Server Thorsten Schütt 3

4 2. Step Clients Single Webserver Single DB server Thorsten Schütt 4

5 3. Step Clients Load Balancer n Webservers... Single DB server Thorsten Schütt 5

6 4. Step Clients Load Balancer n Webservers... Master DB server Slave DB servers Thorsten Schütt 6

7 5. Step Clients Load Balancer n Webservers... DB partitioning DB Directory DB clusters n DB servers memcached... Thorsten Schütt 7

8 What did others do? SimpleDB BigTable Dynamo MySQL Cluster MapReduce Google FS Thorsten Schütt 8

9 OurApproach: P2P in thedata Center Clients Key/Value Store (simple DBMS) ACID Transactions Replication P2P Overlay Thorsten Schütt 9

10 ScalableWikipediawithErlang Chord# Transactional Layer Load Balancing Wikipedia Thorsten Schütt 10

11 Chord # = Chord \Hash Thorsten Schütt 11

12 Chord # A dictionaryhas 3 ops: insert(key, value) delete(key) lookup(key) Chord # implements a distributeddictionary dictionary Key Value node A Clarke 2007 node B Allen 2006 Bachman 1973 Each node only Thompson 1983 Knuth 1974 node D node C stores part of the data Codd 1981 Thorsten Schütt 12

13 Chord # Key Space isan arbitraryset witha total order, e.g. strings Wirth Yao Backus Codd Dijkstr a Rivest Flyod Every nodeisassigneda random key Ritchie Gray Nodes form logicalring overthekeyspace Perlis Karp Nygaar d Knuth Naur Minsky Milner Thorsten Schütt 13

14 Distributed Dictionary Items are stored on their successor, i.e. the first node encountered in clockwise direction Thomson on Wirth Allen on Backus Bachman on Backus Clarke on Codd Rivest Wirth Yao Backus Codd (Clarke, 2007) (Bachman, 1973) Ritchie (Thompson, 1983) (Allen, 2006) Thorsten Schütt 14

15 Distributed Dictionary Each node puts the successor and log 2 Nexponentially spaced fingers in its routing table Backus With each routing step using greedy routing, the distance between the currrent node Wirth Yao Codd Dijkstra and the destination is halved. Rivest Flyod => Yields O(log 2 N)hops. Ritchie Gray Perlis Karp Nygaard Knuth Naur Minsky Milner Thorsten Schütt 15

16 Summary: Chord# DHTs/Chord Chord# DHT is a fully decentralized data structure DHTs self-organize as nodes join, leave, and fail All operations only require local knowledge Onlyonehop for calculating a routing pointer, Chord needs log(n) hops. max. log (N)hops, while Chord guarantees this only with high probability. Can adapt to imbalances in the query load; Chord can t. Supports range queries. Thorsten Schütt 16

17 ScalableWikipediawithErlang Chord# Transactional Layer Load Balancing Wikipedia ACID Thorsten Schütt 17

18 Transactions on DHTs are Challenging high churn rate nodes may leave, join, or crash at any time changing responsibilities crash stop fault model no perfect failure detector never know whether a node crashed or just slow network Thorsten Schütt 18

19 Transactions + Replicas START debit (a, 100); deposit (b, 100); COMMIT START debit (a 1, 100); debit (a 2, 100); debit (a 3, 100); deposit (b 1, 100); deposit (b 2, 100); deposit (b 3, 100); COMMIT Thorsten Schütt 19

20 Adapted PaxosCommit Optimistic CC with fallback option Fast Read Operation just 1 round reads a majority (of the last versions) of all replicates Write Operation 3 rounds nonblocking(fallback) succeeds when > f/2 nodes alive Leader GetTP RegisterTP Prepare Ack Prepared Commit Items InitTM Transaction Manager RegisterRTM TMs andtps Prepared 1. Step 2. Step 3. Step 4. Step 5. Step 6. Step Thorsten Schütt 20

21 Summary Transactions Consistent way to update several items and their replicas Mitigates some of the Overlay Oddities Node Failures Asynchronous programming model Thorsten Schütt 21

22 ScalableWikipediawithErlang Chord# Transactional Layer Load Balancing Wikipedia Thorsten Schütt 22

23 Load-Balancing 1. Pick 2 randomnodes( Naur, Wirth ) 2. IF Load( Wirth ) >> Load( Naur ) 1. Naur leaves system 2. Naur joins as Tarjan Rivest Wirth Yao Backus Codd Overloaded Dijkstr a Flyod Loadcanbeanymetric Ritchie Gray Perlis Karp Nygaar d Knuth D. Karger, M. Ruhl. Simple Efficient Load Balancing Algorithms for Peer-to-Peer Systems. IPTPS Naur Minsky Milner Thorsten Schütt 23

24 Load-Balancing 1. Pick 2 randomnodes( Naur, Wirth ) 2. IF Load( Wirth ) >> Load( Naur ) 1. Naur leaves system 2. Naur joins as Tarjan Tarjan Wirth Yao Backus Codd Dijkstr a Flyod Loadcanbeanymetric Rivest Gray Ritchie Karp Perlis Knuth D. Karger, M. Ruhl. Simple Efficient Load Balancing Algorithms for Peer-to-Peer Systems. IPTPS Nygaa rd Naur Minsk y Milner Thorsten Schütt 24

25 Multi Data-Center Scenario Multi-Data-Center Scenarios Optimize for Latency Increase Availability Prefix Articles with Language Replicas Number E.g. de:main Page 5 replicas 2 in Germany 1 in UK 1 in USA 1 in Asia 0de:Main Page, 1de:Main Page, 2de:Main Page, Thorsten Schütt 25

26 ScalableWikipediawithErlang Chord# Transactional Layer Load Balancing Wikipedia Thorsten Schütt 26

27 Wikipedia Top 10 Web sites 1. Yahoo! 2. Google 3. YouTube 4. Windows Live 5. MSN 6. Myspace 7. Wikipedia 8. Facebook 9. Blogger.com 10. Yahoo! カテゴリ Wikipedia is the top1 openweb Site source code is open architecture is open dumps available Source: alexa.com Thorsten Schütt 27

28 Wikipedia Top 10 Web sites 1. Yahoo! 2. Google 3. YouTube 4. Windows Live 5. MSN 6. Myspace 7. Wikipedia 8. Facebook 9. Blogger.com 10. Yahoo! カテゴリ requests/sec 95% are answered by squid proxies 2,000 req./sec hit the backend Thorsten Schütt 28

29 The Wikipedia System Architecture other search servers NFS web servers Thorsten Schütt 29

30 ErlangImplementation Thorsten Schütt 30

31 Overlay Implemented in Erlang Chord# Load-Balancing Transaction Framework OTP behaviours Supervisor gen_server Distributed Erlang Security Problems Scalability Problems -> OwnTransport-Layer on top oftcp Thorsten Schütt 31

32 Data Model Wikipedia SQL DB Chord# Key-Value Store CREATE TABLE /*$wgdbprefix*/page ( page_id int unsigned NOT NULL auto_increment, page_namespace int NOT NULL,... Map Relations to Key-Value Pairs (Title, List of Versions) (CategoryName, List of Titles) (Title, List of Titles) //Backlinks Thorsten Schütt 32

33 Data Model void updatepage(string title, int oldversion, string newtext) { //new transaction Transaction t = new Transaction(); //read old version Page p = t.read(title); //check for concurrent update if(p.currentversion!= oldversion) t.abort(); else{ //write new text t.write(p.add(newtext)); //update categories foreach(category c in p) t.write(t.read(c.name).add(title)); //commit t.commit(); } } Thorsten Schütt 33

34 Wiki Database: Chord# Mapping Wiki -> Key-Value Store Renderer: Java Tomcat Plog4u Jinterface Interface to Erlang Thorsten Schütt 34

35 IEEE ScaleChallenge 2008 Live Demos Bavarian Simple English Browsing Editing with full History Category-Pages Deployments Planet-Lab 20 nodes Cluster 320 nodes in Berlin Thorsten Schütt 35

36 Summary DHT + Transactions = Scalable, Reliable, Efficient Key/Value Store Previously, P2P was mainly used for file sharing (read only). We support consistent, distributed write operations. Numerous applications Internet databases, transactional online-services, Thorsten Schütt 36

37 Team Thorsten Schütt, Florian Schintke, Monika Moser, Stefan Plantikow, Alexander Reinefeld, Nico Kruber, Christian von Prollius, Seif Haridi (SICS), Ali Ghodsi (SICS), Tallat Shafaat (SICS) Thorsten Schütt 37

38 Detailed Descriptions Wiki S. Plantikow, A. Reinefeld, F. Schintke. Transactions for Distributed Wikis on Structured Overlays. DSOM, October Transactions M. Moser, S. Haridi. Atomic Commitment in Transactional DHTs. 1st CoreGRID Symposium, August DHT T. Shafaat, M. Moser, A. Ghodsi, S. Haridi, T. Schütt, A. Reinefeld. Key-Based Consistency and Availability in Structured Overlay Networks. Infoscale, June T. Schütt, F. Schintke, A. Reinefeld. A Structured Overlay for Multi-dimensional Range Queries. Euro-Par, August T. Schütt, F. Schintke, A. Reinefeld. Structured Overlay without Consistent Hashing: Empirical Results. GP2PC, May Thorsten Schütt 38

39 Questions Thorsten Schütt 39