Cluster Computing Architecture. Intel Labs

Transcription

1 Intel Labs

2 Legal Notices INFORMATION IN THIS DOCUMENT IS PROVIDED IN CONNECTION WITH INTEL PRODUCTS. NO LICENSE, EXPRESS OR IMPLIED, BY ESTOPPEL OR OTHERWISE, TO ANY INTELLECTUAL PROPERTY RIGHTS IS GRANTED BY THIS DOCUMENT. EXCEPT AS PROVIDED IN INTEL S TERMS AND CONDITIONS OF SALE FOR SUCH PRODUCTS, INTEL ASSUMES NO LIABILITY WHATSOEVER, AND INTEL DISCLAIMS ANY EXPRESS OR IMPLIED WARRANTY, RELATING TO SALE AND/OR USE OF INTEL PRODUCTS INCLUDING LIABILITY OR WARRANTIES RELATING TO FITNESS FOR A PARTICULAR PURPOSE, MERCHANTABILITY, OR INFRINGEMENT OF ANY PATENT, COPYRIGHT OR OTHER INTELLECTUAL PROPERTY RIGHT. INTEL PRODUCTS ARE NOT INTENDED FOR USE IN MEDICAL, LIFE SAVING, OR LIFE SUSTAINING APPLICATIONS. Intel may make changes to specifications and product descriptions at any time, without notice. All products, dates, and figures specified are preliminary based on current expectations, and are subject to change without notice. Intel, processors, chipsets, and desktop boards may contain design defects or errors known as errata, which may cause the product to deviate from published specifications. Current characterized errata are available on request. Code names featured are used internally within Intel to identify products that are in development and not yet publicly announced for release. Customers, licensees and other third parties are not authorized by Intel to use code names in advertising, promotion or marketing of any product or services and any such use of Intel's internal code names is at the sole risk of the user Performance tests and ratings are measured using specific computer systems and/or components and reflect the approximate performance of Intel products as measured by those tests. Any difference in system hardware or software design or configuration may affect actual performance. Intel, Intel Inside, and the Intel logo are trademarks of Intel Corporation in the United States and other countries. *Other names and brands may be claimed as the property of others. Copyright 2012 Intel Corporation. 2

3 Peta-scale graphs: The end-to-end challenge Full Internet Map [Lumeta] Social Graph [Facebook] GraphLab is indeed promising But we struggled with feeding it and other practicalities Set out to study potential approaches 3

4 #1: Get a budget #2: Hire ninja coders with ninja resources! 4

5 How do we construct the graph? How do we store it? Query it? Analyze it? it? Many of these challenges are solved for small problems... but what about Internet scale? 5

6 Viz Analysis Interactive Query Transforms Storage Construction 6

7 Viz Analysis Transforms Distributed Graph Databases GBASE [CMU] Construction *[Neo4j] 7

8 Graphviz [ATT] Multiscale layout Analysis Transforms Distributed Graph Databases GBASE [CMU] Construction *[Neo4j] 8

9 Graphviz [ATT] [CMU] Multiscale layout Peta-scale graph mining on [Stanford] Distributed Graph Databases GBASE [CMU] Delete nodes of degree 10 Construction *[Neo4j] 9

10 Graphviz [ATT] [CMU] Multiscale layout Peta-scale graph mining on [Stanford] Distributed Graph Databases GBASE [CMU] Delete nodes of degree 10 *[Neo4j] You. 10

11 The Data Spectrum Unstructured Data Semi-structured Data Structured Data JSON Images Text OEM XML <?xml version= 1.0?> <quiz> <question> Are we really separated by six degrees of separation? </question> <answer> According to Facebook, it s more like </answer>... </quiz> Spreadsheets Graph Databases EDI Relational DBs Speech 11

12 Building Graphs for Practical Apps Raw Data Preprocessing Graph Formation Add Network Information What words are most associated with what topics? XML Docs Extract Topics & Words Bipartite (Topics, Words) Count Word Frequency What does context tell me about the type (person, place, thing) of this noun? News Feeds Extract Noun Phrases and Contexts Bipartite (NP, Context) Count NP Frequency & Initialize type Distribution What are the highest ranked pages? Web Pages Extract Page URLs and Links Directed Graph N/A 12

13 And, in practice and at scale we must: Raw Data Preprocessing Graph Formation Add Network Information Finalize for Parallel Computation Minimize the use of system resources, like memory, storage, etc. Ensure GL s computational effort is load balanced for power-law graphs Do our best to ensure the graph we generated is the one we intended to but the application programmer shouldn t be responsible for this domain expertise! 13

14 14

15 GraphBuilder makes it easy. Fills a hole in the ecosystem Written in Java for easy use in Hadoop MapReduce and applications Offloads domain expertise Raw Data To GraphLab Parsing Tokenization or Feature Extraction Edge List Generation E, V Data Tabulation Graph Normalization Graph Checks & Transforms Graph Partitioning & Serialization App-Specific Code GraphBuilder Library 15

16 Graph Building is MapReduce-able SHUFFLE 16

17 People Interests Kushal Diana Nilesh Danny Ted Frank Ivy Jay 17

18 Okay, we have a graph. Now what? 18

19 Graph Normalization We can save memory if we normalize it (e.g., reduced Wikipedia PageRank graph by 70%) But, seems to call for a global lookup in a framework that prefers independent subproblems A simple, scalable solution is to shard ordered lists: Dictionary Shard 1 Unconverted Converted Edge List Edge List M1 (Aaron,0) (AMD,4) (Brad,1) (CMU,2) (Dan,5) (Dave,3) (IBM,6) (Intel,7) (Aaron,0) (AMD,4) (Brad,1) (CMU,2) Shard 2 (Dan,5) (Dave,3) (IBM,6) (Intel,7) (Aaron,IBM) (Brad,Intel) (Dan,AMD) (Dave,CMU) (Source Sorted) M2 (AMD,5) (CMU,3) (IBM,0) (Intel,1) (Dest Sorted) (5,4) (3,2) (0,6) (1,7) 19

20 Graph Checks & Transforms Would like the ability to: Optionally filter duplicate and/or self edges Transform a directed graph into an undirected graph The library provides: Functions to perform self- and duplicate-edge removal Directionality transformation Solutions are based on a distributed hashing algorithm M1 A C B D H(A, B) H(C, D) Detector M2 B D A C H(A, B) H(C, D) Detector Steering function 20

21 Graph Checks & Transforms Would like the ability to: Optionally filter duplicate and/or self edges Transform a directed graph into an undirected graph The library provides: Functions to perform self- and duplicate-edge removal Directionality transformation Solutions are based on a distributed hashing algorithm M1 C D H(A, B) H(C, D) A B Detector B A M2 D C H(A, B) H(C, D) Detector Steering function 21

22 Graph Checks & Transforms Would like the ability to: Optionally filter duplicate and/or self edges Transform a directed graph into an undirected graph The library provides: Functions to perform self- and duplicate-edge removal Directionality transformation Solutions are based on a distributed hashing algorithm M1 C D H(A, B) H(C, D) Detector A B M2 D C H(A, B) H(C, D) Detector Steering function 22

23 Graph Checks & Transforms Would like the ability to: Optionally filter duplicate and/or self edges Transform a directed graph into an undirected graph The library provides: Functions to perform self- and duplicate-edge removal Directionality transformation Solutions are based on a distributed hashing algorithm M1 H(A, B) H(C, D) Detector A B M2 H(A, B) H(C, D) C D Detector D C Steering function 23

24 Graph Partitioning Objectives Minimize communications by minimizing the number of machines v spans Maximize the edges placed on each machine, subject an imbalance factor S A D 1 C B S 2 S S 24

25 Graph Partitioning Objectives Minimize communications by minimizing the number of machines v spans Maximize the edges placed on each machine, subject an imbalance factor A D C B 25

26 Partitioning Strategies Random edge assignment (to systems) Greedy uses global history used to place edges Oblivious implements a local version of the Greedy strategy 26

27 Oblivious Algorithm K J Machine 2 H I D A B C Machine G 1 E F

28 Machine 1 s Shard D A B A B G F E C E F D C Partition 1 Partition 2 CASE 1: Both end points have never been seen before Randomly assign

29 Machine 1 s Shard D A B A C E G F D Partition 1 Partition 2 A B F E D C CASE 2: Both end points have been seen before on the same partition Assign to a partition which contains both endpoints

30 Machine 1 s Shard D A B E C E G F C Partition 1 Partition 2 A B F E D C CASE 3: Both end points have been seen before but on different partitions Assign to any partition that contains an endpoint

31 Machine 1 s Shard D A B E B G D F C E F Partition 1 Partition 2 A B F E D C CASE 3: Both end points have been seen before but on different partitions Assign to any partition that contains an endpoint C

32 Machine 1 s Shard D A B E G C E F G Partition1 Partition 2 A B F E D F C CASE 4: Only one end point has been seen before Assign to a partition that contains the endpoint D C

33 Replication Results Twitter Graph: 41M vertices, 1.4B edges *Gonzalez et al., PowerGraph: Distributed Graph-Parallel Computation on Natural Graphs, submitted to OSDI 12 33

34 Graph Serialization { } Partitioning JSON Encoding Edge Lists src_id : 34, dest_id : 45 e-data : 30 { } Vertex Lists ver_id : 34, v-data : 56, mirror : [1,2,3], owner : 1 Self-describing data format JSON JSON with compression Extensible Easy to extend to alternative frameworks like Giraph May port to Graph Databases 34

35 Build it! 35

36 Framework Adapter JSON GraphBuilder Stack Graph Partitioning User Defined GraphBuilder app GraphLab app Data Normalization Built-in Parsers Tabulations Graph Transformation GraphBuilder MapReduce Distributed Graph Computation (GLv2) User defined parser User defined functions HDFS 36

37 Prototype Overview Hardware: 8 node cluster 1U Dual CPU (Intel SNB) Amazon build ZT systems 64 GB Memory, Four SATA Hard Drives Intel 10G Adapter and Switch Software: Apache Hadoop GraphLab v2.1 GraphBuilder beta 37

38 Our Wikipedia Graphs LDA Top 1% of vertices are adjacent to 49% of the edges! PageRank Graph V E LDA 4.9M 478M 2.23 PageRank 9.7M 107M

39 Overall Time (Seconds) Scaling Experiment Finalization Partitioning 400 Pre-processing/ Graph Formation x 2x 1x Dataset Size 39

40 How did we do at partitioning? Random Oblivious 7.5 replications/vertex 5.7 replications/vertex For PageRank, the stats were 4.1 and 3.5, respectively. 40

41 What s next for GraphBuilder? Improve partitioning quality and performance Explore better iterative MapReduce models Consider additional library functions Push to peta-scale graphs Prepare for open sourcing! 41

42 The Broader Perspective Collaborate with the Big Data ISTC to develop largescale Graph DBs for GL2 and GraphBuilder Bring new technologies to large-scale ML systems, such as fast persistent memory-storage Explore new scale-out platform architectures for Big Data and Big Learning Brown Big Data ISTC Five focus areas: Databases & Analytics Math & Algorithms Visualization Architecture Streaming U Tenn Knoxville UW Seattle MIT UC Santa Barbara PSU Stanford 42

43