EYWA: a Distributed Graph Engine in the Huawei MIND Platform. Yinglong Xia Huawei Research America 02/09/2017

Transcription

1 EYWA: a Distributed Graph Engine in the Huawei MIND Platform Yinglong Xia Huawei Research America 02/09/2017

2 About Huawei employees countries revenue 2012 Labs regional HQ R&D centers YoY growth 2

3 ICT Enterprise Business Increase user experience for HW s consumers Maintain large scale systems and predict/ diagnose faults Powerful back-end datacenter for cloud services 3

4 Glance at Huawei MIND Platform Support internal business Product AI as a Service Incubate AI competence Knowledge base services Analytic models Basic algorithms Graph Engine Data Processing 4

5 Graphs in AI: Intelligent Service Use Case: AI assists Technical Services Private Knowledge Graph 5 Human-Machine Collaboration Inference & Reasoning

6 Graphs in AI: Knowledge based Search Use Case: AI helps stuff productivity Domain specific questions Private Knowledge Graph 6 Raw data & Evidence Models & Rules Business procedures Search results

7 Graphs in AI: Private Knowledge Graph Use Case: Domain knowledge persistency Unstructured/ Semistructured content External domain data Domain ontology Enterprise & User data 7 Models, Dataset, etc.

8 Reasoning w/ Uncertainty: Graphical Model Use Case: AI for (network) automatic DevOps System diagnosis or root causality analysis are usually conducted under uncertainty Uncertainty Inference 8 Extended Jena reasoner and rules engine

9 Graph Acceleration Use Case: Real-time interactive exploration Acceleration is critical for some analytics in streaming mode and/or interactive mode Intelligent Search & knowledge inference Machine Learning & Deep Learning Graph-based reasoning Large scale optimization 9

10 EYWA: Graph Engine for MIND Visualization Dynamic Graph Vis Property Vis Large Graph Vis Analytics Community Centrality Bayes Net Anomaly detection Label propagation Matching Ego Feature Max Flow Graph engines Streaming Graph Graphical Model Hyper Graph Basic Engine Incremental Update Data Management Structure Management Property Management Metadata Management Permission Control Infrastructure Single Machine Cluster GPU Server Cloud 10

11 Trade-off Between Analytics and Query Physical edge-sets Flow direction

12 Scheduler/Prefetch of Relaxed BSP Flow direction load shard load PSW local computation queue updates send updates recv updates update/storepsw load shard local computation queue updates update/storepsw speedup due to overlapping partially included due to preloading send updates recv updates Dash box includes all tasks within an iteration 12 load PSW preload next shared task type: disk IO CPU network IO

13 Support Very Large Scale Graphs Communication for distributed computing Load current (sub)shard and local PSW; if additional space available, preload other (sub)shard logic mem mem mem sub-shard disk disk local-shards input graph sharding results logic disk server-1 server-2 a logic single server consists of two physical servers 13

14 How to Query & Analyze the Graphs Business users Web-based explorer Interactive visualization front-end (Graph) technical users > Bi-lingual Cypher (w/ additions for active query) Gremlin (LDBC s effort on query language) (Non-graph) technical users RESTful APIs Service on Cloud 14

15 Experiments Disk read bandwidth over time by GraphChi and EdgesSet. Edge-Set showed up to 2x aggregate bandwidth and more constant IO usage. 15 Performance improvement of SSSP against GraphChi including data ingestion Execution time breakdown on Pagerank running Twitter-2010 Execution time breakdown on Pagerank running Twitter-2010

16 Copyright 2012 Huawei Technologies Co., Ltd. All Rights Reserved. The information in this document may contain predictive statements including, without limitation, statements regarding the future financial and operating results, future product portfolio, new technology, etc. There are a number of factors that could cause actual results and developments to differ materially from those expressed or implied in the predictive statements. Therefore, such information is provided for reference purpose only and constitutes neither an offer nor an acceptance. Huawei may change the information at any time without notice.