Efficient and Scalable Friend Recommendations

Size: px
Start display at page:

Download "Efficient and Scalable Friend Recommendations"

Transcription

1 Efficient and Scalable Friend Recommendations Comparing Traditional and Graph-Processing Approaches Nicholas Tietz Software Engineer at GraphSQL January 13, 2014

2 1 Introduction 2 Problem Description 3 RDBMS Approach 4 NoSQL Approach 5 GraphSQL Approach 6 Questions Nicholas Tietz (GraphSQL) Efficient and Scalable Friend Recommendations January 13, / 32

3 Introduction I m Nicholas Tietz B.S. in Mathematics and Computer Science Software Engineer at GraphSQL (1+ years) We re GraphSQL Founding team from Teradata, Twitter, Google, IBM, etc. Founded 1.5 years ago Working on the fastest, most scalable graph platform (We re hiring!) Nicholas Tietz (GraphSQL) Efficient and Scalable Friend Recommendations January 13, / 32

4 Background - Graphs Graph: a collection of edges and vertices (a network) Big graphs contain: over 100 million vertices billions of edges Graphs provide clear insights into: Recommendations Fraud detection Resource optimization Churn analysis Difficult to process traditionally Sparse offerings, improving Nicholas Tietz (GraphSQL) Efficient and Scalable Friend Recommendations January 13, / 32

5 1 Introduction 2 Problem Description 3 RDBMS Approach 4 NoSQL Approach 5 GraphSQL Approach 6 Questions Nicholas Tietz (GraphSQL) Efficient and Scalable Friend Recommendations January 13, / 32

6 The Problem Providing friend recommendations Goal: Provide friend recommendations to all users Must be fast and scalable Motivation: Many social services Keeps users engaged Drives business Extremely hard problem to solve: Worked on at LinkedIn, Facebook, Twitter, etc. Lots of money spent solving Lots of servers used Nicholas Tietz (GraphSQL) Efficient and Scalable Friend Recommendations January 13, / 32

7 Requirements Providing friend recommendations Provide 10 recommendations to each user Must be fast sub-second required under 0.1 seconds ideal Must support real-time updates New users added constantly Cannot do in a batch Must scale well Needs to support hundreds of millions of users Must be good Require reasonably high acceptance rate Cannot just return random users Friends-of-friends Nicholas Tietz (GraphSQL) Efficient and Scalable Friend Recommendations January 13, / 32

8 Naive Algorithm We ll use a simple friend-of-friends algorithm 1 : 1 Retrieve your friends of friends. 2 Rank by number of common neighbors. 3 Select the top 10 scores. 1 We use a much more complicated algorithm in production Nicholas Tietz (GraphSQL) Efficient and Scalable Friend Recommendations January 13, / 32

9 1 Introduction 2 Problem Description 3 RDBMS Approach 4 NoSQL Approach 5 GraphSQL Approach 6 Questions Nicholas Tietz (GraphSQL) Efficient and Scalable Friend Recommendations January 13, / 32

10 RDBMS - Schema CREATE TABLE friends ( user_id INTEGER, friend_id INTEGER ); (Assumption: if (a, b) friends, then (b, a) friends.) Nicholas Tietz (GraphSQL) Efficient and Scalable Friend Recommendations January 13, / 32

11 RDBMS - Query Query for naive algorithm. WITH my_friends AS ( SELECT friend_id FROM friends f WHERE f.user_id = ) SELECT fof.friend_id AS recommended_id, count(*) AS common_friends FROM friends fof WHERE fof.user_id IN (SELECT * FROM my_friends) AND fof.friend_id!= AND fof.friend_id NOT IN (SELECT * FROM my_friends) GROUP BY recommended_id ORDER BY common_friends DESC LIMIT 10; And the real algorithm used is much more complicated! Nicholas Tietz (GraphSQL) Efficient and Scalable Friend Recommendations January 13, / 32

12 RDBMS - Problems This approach has a few problems: Will not scale Difficult-to-optimize multiway self-joins Requires many thousands of index lookups Will not feel responsive to users Effective way to DOS your own DB Nicholas Tietz (GraphSQL) Efficient and Scalable Friend Recommendations January 13, / 32

13 1 Introduction 2 Problem Description 3 RDBMS Approach 4 NoSQL Approach 5 GraphSQL Approach 6 Questions Nicholas Tietz (GraphSQL) Efficient and Scalable Friend Recommendations January 13, / 32

14 NoSQL Approach Replace RDBMS with HBase Python app server to: retrieve friend lists from HBase perform join logic and recommendations (Optional) Batch mode in Hadoop Does not solve the problem: Still need to do many lookups Not natural programming model for problem Difficult to deal with hub nodes Nicholas Tietz (GraphSQL) Efficient and Scalable Friend Recommendations January 13, / 32

15 1 Introduction 2 Problem Description 3 RDBMS Approach 4 NoSQL Approach 5 GraphSQL Approach 6 Questions Nicholas Tietz (GraphSQL) Efficient and Scalable Friend Recommendations January 13, / 32

16 GraphSQL - Design Overview Store data in a native graph format Users are vertices Friendships (or contacts) are edges REST server built into our stack supports this use case Can modify the graph Can call your functions Perform recommendations via quick graph-based computations (Optional) Batch pre-compute recommendations Nicholas Tietz (GraphSQL) Efficient and Scalable Friend Recommendations January 13, / 32

17 GraphSQL - Algorithm in Graph Model Nicholas Tietz (GraphSQL) Efficient and Scalable Friend Recommendations January 13, / 32

18 GraphSQL - Algorithm in Graph Model Nicholas Tietz (GraphSQL) Efficient and Scalable Friend Recommendations January 13, / 32

19 GraphSQL - Algorithm in Graph Model Nicholas Tietz (GraphSQL) Efficient and Scalable Friend Recommendations January 13, / 32

20 GraphSQL - Algorithm in Graph Model Nicholas Tietz (GraphSQL) Efficient and Scalable Friend Recommendations January 13, / 32

21 GraphSQL - Algorithm in Graph Model Nicholas Tietz (GraphSQL) Efficient and Scalable Friend Recommendations January 13, / 32

22 GraphSQL - Algorithm in Graph Model Nicholas Tietz (GraphSQL) Efficient and Scalable Friend Recommendations January 13, / 32

23 GraphSQL - Graph Programming Model Analogous to Pregel + MapReduce Iteration-based Activation-based or whole-graph modes Each iteration has: 2 EdgeMap Called on each outgoing edge for each active vertex VertexReduce Called for each vertex which received messages Very efficient for many problems Graph problems fit naturally Database join problems fit easily 2 Other specialized functions are available. Nicholas Tietz (GraphSQL) Efficient and Scalable Friend Recommendations January 13, / 32

24 GraphSQL - Pseudocode (cont.) EdgeMap: def edge_map(from_vertex, to_vertex): if iteration == 1 or iteration == 2 and to_vertex.value == 0: emit(to_vertex.id, from_vertex.value) Reduce: def reduce(vertex, messages): score = sum(messages) if iteration == 1: set_value(vertex.id, score) else if iteration == 2: result_heap.add((vertex.id, score)) Nicholas Tietz (GraphSQL) Efficient and Scalable Friend Recommendations January 13, / 32

25 GraphSQL - Implementation Process Implementing is easy on our platform: Only need to write one class defining EdgeMap, Reduce, etc. REST API already exists Similar experience to Hadoop Current: no public API or SDK, more on this later Nicholas Tietz (GraphSQL) Efficient and Scalable Friend Recommendations January 13, / 32

26 GraphSQL - Industry Experience We have this deployed for two companies. Requires fewer servers than the NoSQL approach Faster end-to-end response times Allows more sophisticated recommendation algorithms Easier to write and maintain Nicholas Tietz (GraphSQL) Efficient and Scalable Friend Recommendations January 13, / 32

27 Moral of the Story I don t do friend recommendations, why do I care? Two reasons: 1 Networks are everywhere, and you have one in your data. Use the right tools for the right jobs. 2 Joins are everywhere. They are expensive to do, but with a graph platform you can have pre-computed always-up-to-date joins. Nicholas Tietz (GraphSQL) Efficient and Scalable Friend Recommendations January 13, / 32

28 Shameless Plug We re hiring! Software engineer (data / log analysis, RDBMS, dashboard / data visualization) Systems software engineer (file systems, database storage, distributed systems) POC Software engineer (algorithms background, work with customers) We re looking for companies to work with! Develop proof-of-concept for you Helps us improve our offering Contact us for more information! Nicholas Tietz (GraphSQL) Efficient and Scalable Friend Recommendations January 13, / 32

29 1 Introduction 2 Problem Description 3 RDBMS Approach 4 NoSQL Approach 5 GraphSQL Approach 6 Questions Nicholas Tietz (GraphSQL) Efficient and Scalable Friend Recommendations January 13, / 32

30 Questions? graphsql.com Nicholas Tietz (GraphSQL) Efficient and Scalable Friend Recommendations January 13, / 32

31 Appendix A: NoSQL Pseudocode def get_recommendations(user_id): friends = hbase.get_row(user_id).as_set candidates = {} for f1 in friends: f1_friends = hbase.get_row(f1).as_set for f2 in friend_friends: if f2 in candidates: candidates[f2] += 1 else candidates[f2] = 0 for f1 in friends: candidates.remove(f1) candidates.remove(user_id) return get_top_10(candidates) Nicholas Tietz (GraphSQL) Efficient and Scalable Friend Recommendations January 13, / 32

32 Appendix B: Hub Nodes (NoSQL) A hub node has high degree Dangerous to traverse from Difficult to join on No obvious way to avoid expanding hub nodes (in NoSQL) Storing degree information shifts the problem How do you safely apply graph updates that change degree? Nicholas Tietz (GraphSQL) Efficient and Scalable Friend Recommendations January 13, / 32

Webinar Series TMIP VISION

Webinar Series TMIP VISION Webinar Series TMIP VISION TMIP provides technical support and promotes knowledge and information exchange in the transportation planning and modeling community. Today s Goals To Consider: Parallel Processing

More information

Apache Giraph: Facebook-scale graph processing infrastructure. 3/31/2014 Avery Ching, Facebook GDM

Apache Giraph: Facebook-scale graph processing infrastructure. 3/31/2014 Avery Ching, Facebook GDM Apache Giraph: Facebook-scale graph processing infrastructure 3/31/2014 Avery Ching, Facebook GDM Motivation Apache Giraph Inspired by Google s Pregel but runs on Hadoop Think like a vertex Maximum value

More information

One Trillion Edges. Graph processing at Facebook scale

One Trillion Edges. Graph processing at Facebook scale One Trillion Edges Graph processing at Facebook scale Introduction Platform improvements Compute model extensions Experimental results Operational experience How Facebook improved Apache Giraph Facebook's

More information

Distributed Graph Storage. Veronika Molnár, UZH

Distributed Graph Storage. Veronika Molnár, UZH Distributed Graph Storage Veronika Molnár, UZH Overview Graphs and Social Networks Criteria for Graph Processing Systems Current Systems Storage Computation Large scale systems Comparison / Best systems

More information

Syncsort DMX-h. Simplifying Big Data Integration. Goals of the Modern Data Architecture SOLUTION SHEET

Syncsort DMX-h. Simplifying Big Data Integration. Goals of the Modern Data Architecture SOLUTION SHEET SOLUTION SHEET Syncsort DMX-h Simplifying Big Data Integration Goals of the Modern Data Architecture Data warehouses and mainframes are mainstays of traditional data architectures and still play a vital

More information

The Evolution of Big Data Platforms and Data Science

The Evolution of Big Data Platforms and Data Science IBM Analytics The Evolution of Big Data Platforms and Data Science ECC Conference 2016 Brandon MacKenzie June 13, 2016 2016 IBM Corporation Hello, I m Brandon MacKenzie. I work at IBM. Data Science - Offering

More information

Big Data. Big Data Analyst. Big Data Engineer. Big Data Architect

Big Data. Big Data Analyst. Big Data Engineer. Big Data Architect Big Data Big Data Analyst INTRODUCTION TO BIG DATA ANALYTICS ANALYTICS PROCESSING TECHNIQUES DATA TRANSFORMATION & BATCH PROCESSING REAL TIME (STREAM) DATA PROCESSING Big Data Engineer BIG DATA FOUNDATION

More information

Data Informatics. Seon Ho Kim, Ph.D.

Data Informatics. Seon Ho Kim, Ph.D. Data Informatics Seon Ho Kim, Ph.D. seonkim@usc.edu HBase HBase is.. A distributed data store that can scale horizontally to 1,000s of commodity servers and petabytes of indexed storage. Designed to operate

More information

Graph Algorithms using Map-Reduce. Graphs are ubiquitous in modern society. Some examples: The hyperlink structure of the web

Graph Algorithms using Map-Reduce. Graphs are ubiquitous in modern society. Some examples: The hyperlink structure of the web Graph Algorithms using Map-Reduce Graphs are ubiquitous in modern society. Some examples: The hyperlink structure of the web Graph Algorithms using Map-Reduce Graphs are ubiquitous in modern society. Some

More information

Embedded Technosolutions

Embedded Technosolutions Hadoop Big Data An Important technology in IT Sector Hadoop - Big Data Oerie 90% of the worlds data was generated in the last few years. Due to the advent of new technologies, devices, and communication

More information

Big Data with Hadoop Ecosystem

Big Data with Hadoop Ecosystem Diógenes Pires Big Data with Hadoop Ecosystem Hands-on (HBase, MySql and Hive + Power BI) Internet Live http://www.internetlivestats.com/ Introduction Business Intelligence Business Intelligence Process

More information

MODERN BIG DATA DESIGN PATTERNS CASE DRIVEN DESINGS

MODERN BIG DATA DESIGN PATTERNS CASE DRIVEN DESINGS MODERN BIG DATA DESIGN PATTERNS CASE DRIVEN DESINGS SUJEE MANIYAM FOUNDER / PRINCIPAL @ ELEPHANT SCALE www.elephantscale.com sujee@elephantscale.com HI, I M SUJEE MANIYAM Founder / Principal @ ElephantScale

More information

Practice and Applications of Data Management CMPSCI 345. Lecture 18: Big Data, Hadoop, and MapReduce

Practice and Applications of Data Management CMPSCI 345. Lecture 18: Big Data, Hadoop, and MapReduce Practice and Applications of Data Management CMPSCI 345 Lecture 18: Big Data, Hadoop, and MapReduce Why Big Data, Hadoop, M-R? } What is the connec,on with the things we learned? } What about SQL? } What

More information

Intro to Neo4j and Graph Databases

Intro to Neo4j and Graph Databases Intro to Neo4j and Graph Databases David Montag Neo Technology! david@neotechnology.com Early Adopters of Graph Technology Survival of the Fittest Evolution of Web Search Pre-1999 WWW Indexing 1999-2012

More information

BIG DATA ANALYTICS A PRACTICAL GUIDE

BIG DATA ANALYTICS A PRACTICAL GUIDE BIG DATA ANALYTICS A PRACTICAL GUIDE STEP 1: GETTING YOUR DATA PLATFORM IN ORDER Big Data Analytics A Practical Guide / Step 1: Getting your Data Platform in Order 1 INTRODUCTION Everybody keeps extolling

More information

Lecture 25 Overview. Last Lecture Query optimisation/query execution strategies

Lecture 25 Overview. Last Lecture Query optimisation/query execution strategies Lecture 25 Overview Last Lecture Query optimisation/query execution strategies This Lecture Non-relational data models Source: web pages, textbook chapters 20-22 Next Lecture Revision COSC344 Lecture 25

More information

Topics. Big Data Analytics What is and Why Hadoop? Comparison to other technologies Hadoop architecture Hadoop ecosystem Hadoop usage examples

Topics. Big Data Analytics What is and Why Hadoop? Comparison to other technologies Hadoop architecture Hadoop ecosystem Hadoop usage examples Hadoop Introduction 1 Topics Big Data Analytics What is and Why Hadoop? Comparison to other technologies Hadoop architecture Hadoop ecosystem Hadoop usage examples 2 Big Data Analytics What is Big Data?

More information

Bring Context To Your Machine Data With Hadoop, RDBMS & Splunk

Bring Context To Your Machine Data With Hadoop, RDBMS & Splunk Bring Context To Your Machine Data With Hadoop, RDBMS & Splunk Raanan Dagan and Rohit Pujari September 25, 2017 Washington, DC Forward-Looking Statements During the course of this presentation, we may

More information

JAVASCRIPT CHARTING. Scaling for the Enterprise with Metric Insights Copyright Metric insights, Inc.

JAVASCRIPT CHARTING. Scaling for the Enterprise with Metric Insights Copyright Metric insights, Inc. JAVASCRIPT CHARTING Scaling for the Enterprise with Metric Insights 2013 Copyright Metric insights, Inc. A REVOLUTION IS HAPPENING... 3! Challenges... 3! Borrowing From The Enterprise BI Stack... 4! Visualization

More information

Review - Relational Model Concepts

Review - Relational Model Concepts Lecture 25 Overview Last Lecture Query optimisation/query execution strategies This Lecture Non-relational data models Source: web pages, textbook chapters 20-22 Next Lecture Revision Review - Relational

More information

S2Graph : A large-scale graph database

S2Graph : A large-scale graph database daumkakao S2Graph : A large-scale graph database with Hbase Doyoung Yoon x Taejin Chin DaumKakao A Mobile Lifestyle Platform 1. KakaoTalk a. Mobile Messenger replacing SMS b. KaTalkHe is being used as

More information

RAMCloud. Scalable High-Performance Storage Entirely in DRAM. by John Ousterhout et al. Stanford University. presented by Slavik Derevyanko

RAMCloud. Scalable High-Performance Storage Entirely in DRAM. by John Ousterhout et al. Stanford University. presented by Slavik Derevyanko RAMCloud Scalable High-Performance Storage Entirely in DRAM 2009 by John Ousterhout et al. Stanford University presented by Slavik Derevyanko Outline RAMCloud project overview Motivation for RAMCloud storage:

More information

6 TIPS FOR IMPROVING YOUR WEB PRESENCE

6 TIPS FOR IMPROVING YOUR WEB PRESENCE 6 TIPS FOR IMPROVING YOUR WEB PRESENCE 6 TIPS FOR IMPROVING YOUR WEB PRESENCE We all want to get noticed on the web. If you are running a business you want to be on the first page in Google via organic

More information

CIB Session 12th NoSQL Databases Structures

CIB Session 12th NoSQL Databases Structures CIB Session 12th NoSQL Databases Structures By: Shahab Safaee & Morteza Zahedi Software Engineering PhD Email: safaee.shx@gmail.com, morteza.zahedi.a@gmail.com cibtrc.ir cibtrc cibtrc 2 Agenda What is

More information

This Event Is About Your Internet Presence.

This Event Is About Your Internet Presence. This Event Is About Your Internet Presence. For Financial Services Representatives, The Internet Isn t About Selling, It Is About Attracting People Who May Eventually Buy - - And Your Website Is The Hub

More information

An Introduction to Big Data Formats

An Introduction to Big Data Formats Introduction to Big Data Formats 1 An Introduction to Big Data Formats Understanding Avro, Parquet, and ORC WHITE PAPER Introduction to Big Data Formats 2 TABLE OF TABLE OF CONTENTS CONTENTS INTRODUCTION

More information

A NoSQL Introduction for Relational Database Developers. Andrew Karcher Las Vegas SQL Saturday September 12th, 2015

A NoSQL Introduction for Relational Database Developers. Andrew Karcher Las Vegas SQL Saturday September 12th, 2015 A NoSQL Introduction for Relational Database Developers Andrew Karcher Las Vegas SQL Saturday September 12th, 2015 About Me http://www.andrewkarcher.com Twitter: @akarcher LinkedIn, Twitter Email: akarcher@gmail.com

More information

Promoting Your Small Business with and Social Media

Promoting Your Small Business with  and Social Media How To Guide: Promoting Your Small Business with Email and Social Media Connect with Constant Contact. Everywhere. v1.0 06.27.2016 Market Your Email Socially Did you know that social media and email work

More information

MapReduce and Friends

MapReduce and Friends MapReduce and Friends Craig C. Douglas University of Wyoming with thanks to Mookwon Seo Why was it invented? MapReduce is a mergesort for large distributed memory computers. It was the basis for a web

More information

GraphCEP Real-Time Data Analytics Using Parallel Complex Event and Graph Processing

GraphCEP Real-Time Data Analytics Using Parallel Complex Event and Graph Processing Institute of Parallel and Distributed Systems () Universitätsstraße 38 D-70569 Stuttgart GraphCEP Real-Time Data Analytics Using Parallel Complex Event and Graph Processing Ruben Mayer, Christian Mayer,

More information

SQT03 Big Data and Hadoop with Azure HDInsight Andrew Brust. Senior Director, Technical Product Marketing and Evangelism

SQT03 Big Data and Hadoop with Azure HDInsight Andrew Brust. Senior Director, Technical Product Marketing and Evangelism Big Data and Hadoop with Azure HDInsight Andrew Brust Senior Director, Technical Product Marketing and Evangelism Datameer Level: Intermediate Meet Andrew Senior Director, Technical Product Marketing and

More information

CMO Briefing Google+:

CMO Briefing Google+: www.bootcampdigital.com CMO Briefing Google+: How Google s New Social Network Can Impact Your Business Facts Google+ had over 30 million users in the first month and was the fastest growing social network

More information

Introduction to NoSQL by William McKnight

Introduction to NoSQL by William McKnight Introduction to NoSQL by William McKnight All rights reserved. Reproduction in whole or part prohibited except by written permission. Product and company names mentioned herein may be trademarks of their

More information

ECE 7650 Scalable and Secure Internet Services and Architecture ---- A Systems Perspective

ECE 7650 Scalable and Secure Internet Services and Architecture ---- A Systems Perspective ECE 60 Scalable and Secure Internet Services and Architecture ---- A Systems Perspective Part II: Data Center Software Architecture: Topic 3: Programming Models Pregel: A System for Large-Scale Graph Processing

More information

Prototyping Data Intensive Apps: TrendingTopics.org

Prototyping Data Intensive Apps: TrendingTopics.org Prototyping Data Intensive Apps: TrendingTopics.org Pete Skomoroch Research Scientist at LinkedIn Consultant at Data Wrangling @peteskomoroch 09/29/09 1 Talk Outline TrendingTopics Overview Wikipedia Page

More information

Hadoop 2.x Core: YARN, Tez, and Spark. Hortonworks Inc All Rights Reserved

Hadoop 2.x Core: YARN, Tez, and Spark. Hortonworks Inc All Rights Reserved Hadoop 2.x Core: YARN, Tez, and Spark YARN Hadoop Machine Types top-of-rack switches core switch client machines have client-side software used to access a cluster to process data master nodes run Hadoop

More information

Big Data Hadoop Developer Course Content. Big Data Hadoop Developer - The Complete Course Course Duration: 45 Hours

Big Data Hadoop Developer Course Content. Big Data Hadoop Developer - The Complete Course Course Duration: 45 Hours Big Data Hadoop Developer Course Content Who is the target audience? Big Data Hadoop Developer - The Complete Course Course Duration: 45 Hours Complete beginners who want to learn Big Data Hadoop Professionals

More information

Increase Value from Big Data with Real-Time Data Integration and Streaming Analytics

Increase Value from Big Data with Real-Time Data Integration and Streaming Analytics Increase Value from Big Data with Real-Time Data Integration and Streaming Analytics Cy Erbay Senior Director Striim Executive Summary Striim is Uniquely Qualified to Solve the Challenges of Real-Time

More information

Databricks, an Introduction

Databricks, an Introduction Databricks, an Introduction Chuck Connell, Insight Digital Innovation Insight Presentation Speaker Bio Senior Data Architect at Insight Digital Innovation Focus on Azure big data services HDInsight/Hadoop,

More information

HBase vs Neo4j. Technical overview. Name: Vladan Jovičić CR09 Advanced Scalable Data (Fall, 2017) Ecolé Normale Superiuere de Lyon

HBase vs Neo4j. Technical overview. Name: Vladan Jovičić CR09 Advanced Scalable Data (Fall, 2017) Ecolé Normale Superiuere de Lyon HBase vs Neo4j Technical overview Name: Vladan Jovičić CR09 Advanced Scalable Data (Fall, 2017) Ecolé Normale Superiuere de Lyon 12th October 2017 1 Contents 1 Introduction 3 2 Overview of HBase and Neo4j

More information

DATABASE DESIGN II - 1DL400

DATABASE DESIGN II - 1DL400 DATABASE DESIGN II - 1DL400 Fall 2016 A second course in database systems http://www.it.uu.se/research/group/udbl/kurser/dbii_ht16 Kjell Orsborn Uppsala Database Laboratory Department of Information Technology,

More information

Real-Time Deep-Link Analytics for Big Graphs. Challenges and Solutions

Real-Time Deep-Link Analytics for Big Graphs. Challenges and Solutions Real-Time Deep-Link Analytics for Big Graphs Challenges and Solutions Victor Lee, Sr. Product Manager BigGraph Meetup October 11, 2017 Welcome to We meet to network, share, discuss, and invent together

More information

relational Key-value Graph Object Document

relational Key-value Graph Object Document NoSQL Databases Earlier We have spent most of our time with the relational DB model so far. There are other models: Key-value: a hash table Graph: stores graph-like structures efficiently Object: good

More information

Scalable Web Programming. CS193S - Jan Jannink - 2/25/10

Scalable Web Programming. CS193S - Jan Jannink - 2/25/10 Scalable Web Programming CS193S - Jan Jannink - 2/25/10 Weekly Syllabus 1.Scalability: (Jan.) 2.Agile Practices 3.Ecology/Mashups 4.Browser/Client 7.Analytics 8.Cloud/Map-Reduce 9.Published APIs: (Mar.)*

More information

An Introduction to Apache Spark

An Introduction to Apache Spark An Introduction to Apache Spark 1 History Developed in 2009 at UC Berkeley AMPLab. Open sourced in 2010. Spark becomes one of the largest big-data projects with more 400 contributors in 50+ organizations

More information

Microsoft Azure Databricks for data engineering. Building production data pipelines with Apache Spark in the cloud

Microsoft Azure Databricks for data engineering. Building production data pipelines with Apache Spark in the cloud Microsoft Azure Databricks for data engineering Building production data pipelines with Apache Spark in the cloud Azure Databricks As companies continue to set their sights on making data-driven decisions

More information

Intro Cassandra. Adelaide Big Data Meetup.

Intro Cassandra. Adelaide Big Data Meetup. Intro Cassandra Adelaide Big Data Meetup instaclustr.com @Instaclustr Who am I and what do I do? Alex Lourie Worked at Red Hat, Datastax and now Instaclustr We currently manage x10s nodes for various customers,

More information

G(B)enchmark GraphBench: Towards a Universal Graph Benchmark. Khaled Ammar M. Tamer Özsu

G(B)enchmark GraphBench: Towards a Universal Graph Benchmark. Khaled Ammar M. Tamer Özsu G(B)enchmark GraphBench: Towards a Universal Graph Benchmark Khaled Ammar M. Tamer Özsu Bioinformatics Software Engineering Social Network Gene Co-expression Protein Structure Program Flow Big Graphs o

More information

Microsoft Big Data and Hadoop

Microsoft Big Data and Hadoop Microsoft Big Data and Hadoop Lara Rubbelke @sqlgal Cindy Gross @sqlcindy 2 The world of data is changing The 4Vs of Big Data http://nosql.mypopescu.com/post/9621746531/a-definition-of-big-data 3 Common

More information

Overview Computer Networking Lecture 16: Delivering Content: Peer to Peer and CDNs Peter Steenkiste

Overview Computer Networking Lecture 16: Delivering Content: Peer to Peer and CDNs Peter Steenkiste Overview 5-44 5-44 Computer Networking 5-64 Lecture 6: Delivering Content: Peer to Peer and CDNs Peter Steenkiste Web Consistent hashing Peer-to-peer Motivation Architectures Discussion CDN Video Fall

More information

Hadoop An Overview. - Socrates CCDH

Hadoop An Overview. - Socrates CCDH Hadoop An Overview - Socrates CCDH What is Big Data? Volume Not Gigabyte. Terabyte, Petabyte, Exabyte, Zettabyte - Due to handheld gadgets,and HD format images and videos - In total data, 90% of them collected

More information

BIG DATA COURSE CONTENT

BIG DATA COURSE CONTENT BIG DATA COURSE CONTENT [I] Get Started with Big Data Microsoft Professional Orientation: Big Data Duration: 12 hrs Course Content: Introduction Course Introduction Data Fundamentals Introduction to Data

More information

Big Data Technology Ecosystem. Mark Burnette Pentaho Director Sales Engineering, Hitachi Vantara

Big Data Technology Ecosystem. Mark Burnette Pentaho Director Sales Engineering, Hitachi Vantara Big Data Technology Ecosystem Mark Burnette Pentaho Director Sales Engineering, Hitachi Vantara Agenda End-to-End Data Delivery Platform Ecosystem of Data Technologies Mapping an End-to-End Solution Case

More information

HTML presentation, positioning and designing responsive web applications.

HTML presentation, positioning and designing responsive web applications. Hi I am Rodolfo. I put to life to MEAN Stack development and Serverless applications in Amazon and Google Cloud. My passion revolves around helping clients solve very complex problems using cool technologies

More information

Big Data Analytics. Rasoul Karimi

Big Data Analytics. Rasoul Karimi Big Data Analytics Rasoul Karimi Information Systems and Machine Learning Lab (ISMLL) Institute of Computer Science University of Hildesheim, Germany Big Data Analytics Big Data Analytics 1 / 1 Outline

More information

AllegroGraph for Flexibility in the Enterprise and on the Web. Jans Aasman Franz Inc

AllegroGraph for Flexibility in the Enterprise and on the Web. Jans Aasman Franz Inc AllegroGraph for Flexibility in the Enterprise and on the Web Jans Aasman Franz Inc ja@franz.com What is a triple store (1 (2 3) (4 5) (6 7) (8 9) (10 11) (12 13) (14 15)(16 17) (18 19 20 21 22 23 24

More information

Apache Giraph. for applications in Machine Learning & Recommendation Systems. Maria Novartis

Apache Giraph. for applications in Machine Learning & Recommendation Systems. Maria Novartis Apache Giraph for applications in Machine Learning & Recommendation Systems Maria Stylianou @marsty5 Novartis Züri Machine Learning Meetup #5 June 16, 2014 Apache Giraph for applications in Machine Learning

More information

The Technology of the Business Data Lake. Appendix

The Technology of the Business Data Lake. Appendix The Technology of the Business Data Lake Appendix Pivotal data products Term Greenplum Database GemFire Pivotal HD Spring XD Pivotal Data Dispatch Pivotal Analytics Description A massively parallel platform

More information

Big Data com Hadoop. VIII Sessão - SQL Bahia. Impala, Hive e Spark. Diógenes Pires 03/03/2018

Big Data com Hadoop. VIII Sessão - SQL Bahia. Impala, Hive e Spark. Diógenes Pires 03/03/2018 Big Data com Hadoop Impala, Hive e Spark VIII Sessão - SQL Bahia 03/03/2018 Diógenes Pires Connect with PASS Sign up for a free membership today at: pass.org #sqlpass Internet Live http://www.internetlivestats.com/

More information

A Review Paper on Big data & Hadoop

A Review Paper on Big data & Hadoop A Review Paper on Big data & Hadoop Rupali Jagadale MCA Department, Modern College of Engg. Modern College of Engginering Pune,India rupalijagadale02@gmail.com Pratibha Adkar MCA Department, Modern College

More information

Container 2.0. Container: check! But what about persistent data, big data or fast data?!

Container 2.0. Container: check! But what about persistent data, big data or fast data?! @unterstein @joerg_schad @dcos @jaxdevops Container 2.0 Container: check! But what about persistent data, big data or fast data?! 1 Jörg Schad Distributed Systems Engineer @joerg_schad Johannes Unterstein

More information

Text transcript of show #280. August 18, Microsoft Research: Trinity is a Graph Database and a Distributed Parallel Platform for Graph Data

Text transcript of show #280. August 18, Microsoft Research: Trinity is a Graph Database and a Distributed Parallel Platform for Graph Data Hanselminutes is a weekly audio talk show with noted web developer and technologist Scott Hanselman and hosted by Carl Franklin. Scott discusses utilities and tools, gives practical how-to advice, and

More information

A New Parallel Algorithm for Connected Components in Dynamic Graphs. Robert McColl Oded Green David Bader

A New Parallel Algorithm for Connected Components in Dynamic Graphs. Robert McColl Oded Green David Bader A New Parallel Algorithm for Connected Components in Dynamic Graphs Robert McColl Oded Green David Bader Overview The Problem Target Datasets Prior Work Parent-Neighbor Subgraph Results Conclusions Problem

More information

Figure 1: A directed graph.

Figure 1: A directed graph. 1 Graphs A graph is a data structure that expresses relationships between objects. The objects are called nodes and the relationships are called edges. For example, social networks can be represented as

More information

Jure Leskovec Including joint work with Y. Perez, R. Sosič, A. Banarjee, M. Raison, R. Puttagunta, P. Shah

Jure Leskovec Including joint work with Y. Perez, R. Sosič, A. Banarjee, M. Raison, R. Puttagunta, P. Shah Jure Leskovec (@jure) Including joint work with Y. Perez, R. Sosič, A. Banarjee, M. Raison, R. Puttagunta, P. Shah 2 My research group at Stanford: Mining and modeling large social and information networks

More information

A Glimpse of the Hadoop Echosystem

A Glimpse of the Hadoop Echosystem A Glimpse of the Hadoop Echosystem 1 Hadoop Echosystem A cluster is shared among several users in an organization Different services HDFS and MapReduce provide the lower layers of the infrastructures Other

More information

Turning NoSQL data into Graph Playing with Apache Giraph and Apache Gora

Turning NoSQL data into Graph Playing with Apache Giraph and Apache Gora Turning NoSQL data into Graph Playing with Apache Giraph and Apache Gora Team Renato Marroquín! PhD student: Interested in: Information retrieval. Distributed and scalable data management. Apache Gora:

More information

CISC 7610 Lecture 2b The beginnings of NoSQL

CISC 7610 Lecture 2b The beginnings of NoSQL CISC 7610 Lecture 2b The beginnings of NoSQL Topics: Big Data Google s infrastructure Hadoop: open google infrastructure Scaling through sharding CAP theorem Amazon s Dynamo 5 V s of big data Everyone

More information

Getting to know. by Michelle Darling August 2013

Getting to know. by Michelle Darling August 2013 Getting to know by Michelle Darling mdarlingcmt@gmail.com August 2013 Agenda: What is Cassandra? Installation, CQL3 Data Modelling Summary Only 15 min to cover these, so please hold questions til the end,

More information

Analyzing Flight Data

Analyzing Flight Data IBM Analytics Analyzing Flight Data Jeff Carlson Rich Tarro July 21, 2016 2016 IBM Corporation Agenda Spark Overview a quick review Introduction to Graph Processing and Spark GraphX GraphX Overview Demo

More information

PROFESSIONAL. NoSQL. Shashank Tiwari WILEY. John Wiley & Sons, Inc.

PROFESSIONAL. NoSQL. Shashank Tiwari WILEY. John Wiley & Sons, Inc. PROFESSIONAL NoSQL Shashank Tiwari WILEY John Wiley & Sons, Inc. Examining CONTENTS INTRODUCTION xvil CHAPTER 1: NOSQL: WHAT IT IS AND WHY YOU NEED IT 3 Definition and Introduction 4 Context and a Bit

More information

Maximizing Fraud Prevention Through Disruptive Architectures Delivering speed at scale.

Maximizing Fraud Prevention Through Disruptive Architectures Delivering speed at scale. Maximizing Fraud Prevention Through Disruptive Architectures Delivering speed at scale. January 2016 Credit Card Fraud prevention is among the most time-sensitive and high-value of IT tasks. The databases

More information

Developing Enterprise Cloud Solutions with Azure

Developing Enterprise Cloud Solutions with Azure Developing Enterprise Cloud Solutions with Azure Java Focused 5 Day Course AUDIENCE FORMAT Developers and Software Architects Instructor-led with hands-on labs LEVEL 300 COURSE DESCRIPTION This course

More information

Big Data Hadoop Stack

Big Data Hadoop Stack Big Data Hadoop Stack Lecture #1 Hadoop Beginnings What is Hadoop? Apache Hadoop is an open source software framework for storage and large scale processing of data-sets on clusters of commodity hardware

More information

Accelerate your SAS analytics to take the gold

Accelerate your SAS analytics to take the gold Accelerate your SAS analytics to take the gold A White Paper by Fuzzy Logix Whatever the nature of your business s analytics environment we are sure you are under increasing pressure to deliver more: more

More information

Acquiring Big Data to Realize Business Value

Acquiring Big Data to Realize Business Value Acquiring Big Data to Realize Business Value Agenda What is Big Data? Common Big Data technologies Use Case Examples Oracle Products in the Big Data space In Summary: Big Data Takeaways

More information

Information Retrieval (IR) Introduction to Information Retrieval. Lecture Overview. Why do we need IR? Basics of an IR system.

Information Retrieval (IR) Introduction to Information Retrieval. Lecture Overview. Why do we need IR? Basics of an IR system. Introduction to Information Retrieval Ethan Phelps-Goodman Some slides taken from http://www.cs.utexas.edu/users/mooney/ir-course/ Information Retrieval (IR) The indexing and retrieval of textual documents.

More information

Stages of Data Processing

Stages of Data Processing Data processing can be understood as the conversion of raw data into a meaningful and desired form. Basically, producing information that can be understood by the end user. So then, the question arises,

More information

Data Science and Open Source Software. Iraklis Varlamis Assistant Professor Harokopio University of Athens

Data Science and Open Source Software. Iraklis Varlamis Assistant Professor Harokopio University of Athens Data Science and Open Source Software Iraklis Varlamis Assistant Professor Harokopio University of Athens varlamis@hua.gr What is data science? 2 Why data science is important? More data (volume, variety,...)

More information

HBase: Overview. HBase is a distributed column-oriented data store built on top of HDFS

HBase: Overview. HBase is a distributed column-oriented data store built on top of HDFS HBase 1 HBase: Overview HBase is a distributed column-oriented data store built on top of HDFS HBase is an Apache open source project whose goal is to provide storage for the Hadoop Distributed Computing

More information

Data Processing on Large Clusters. By: Stephen Cardina

Data Processing on Large Clusters. By: Stephen Cardina Data Processing on Large Clusters By: Stephen Cardina Introduction MapReduce is used on clusters to get data that you are specifically looking for. MapReduce was made back in 2004 by Google in order to

More information

Let's Play... Try to name the databases described on the following slides...

Let's Play... Try to name the databases described on the following slides... Database Software Let's Play... Try to name the databases described on the following slides... "World's most popular" Free relational database system (RDBMS) that... the "M" in "LAMP" and "XAMP" stacks

More information

Big Data Infrastructure CS 489/698 Big Data Infrastructure (Winter 2017)

Big Data Infrastructure CS 489/698 Big Data Infrastructure (Winter 2017) Big Data Infrastructure CS 489/698 Big Data Infrastructure (Winter 2017) Week 5: Analyzing Graphs (2/2) February 2, 2017 Jimmy Lin David R. Cheriton School of Computer Science University of Waterloo These

More information

Introduction to the Active Everywhere Database

Introduction to the Active Everywhere Database Introduction to the Active Everywhere Database INTRODUCTION For almost half a century, the relational database management system (RDBMS) has been the dominant model for database management. This more than

More information

CERTIFICATE IN SOFTWARE DEVELOPMENT LIFE CYCLE IN BIG DATA AND BUSINESS INTELLIGENCE (SDLC-BD & BI)

CERTIFICATE IN SOFTWARE DEVELOPMENT LIFE CYCLE IN BIG DATA AND BUSINESS INTELLIGENCE (SDLC-BD & BI) CERTIFICATE IN SOFTWARE DEVELOPMENT LIFE CYCLE IN BIG DATA AND BUSINESS INTELLIGENCE (SDLC-BD & BI) The Certificate in Software Development Life Cycle in BIGDATA, Business Intelligence and Tableau program

More information

Fast Innovation requires Fast IT

Fast Innovation requires Fast IT Fast Innovation requires Fast IT Cisco Data Virtualization Puneet Kumar Bhugra Business Solutions Manager 1 Challenge In Data, Big Data & Analytics Siloed, Multiple Sources Business Outcomes Business Opportunity:

More information

Lecture Map-Reduce. Algorithms. By Marina Barsky Winter 2017, University of Toronto

Lecture Map-Reduce. Algorithms. By Marina Barsky Winter 2017, University of Toronto Lecture 04.02 Map-Reduce Algorithms By Marina Barsky Winter 2017, University of Toronto Example 1: Language Model Statistical machine translation: Need to count number of times every 5-word sequence occurs

More information

Repurposing Your Podcast. 3 Places Your Podcast Must Be To Maximize Your Reach (And How To Use Each Effectively)

Repurposing Your Podcast. 3 Places Your Podcast Must Be To Maximize Your Reach (And How To Use Each Effectively) Repurposing Your Podcast 3 Places Your Podcast Must Be To Maximize Your Reach (And How To Use Each Effectively) What You ll Learn What 3 Channels (Besides itunes and Stitcher) Your Podcast Should Be On

More information

Data 101 Which DB, When. Joe Yong Azure SQL Data Warehouse, Program Management Microsoft Corp.

Data 101 Which DB, When. Joe Yong Azure SQL Data Warehouse, Program Management Microsoft Corp. Data 101 Which DB, When Joe Yong (joeyong@microsoft.com) Azure SQL Data Warehouse, Program Management Microsoft Corp. The world is changing AI increased by 300% in 2017 Data will grow to 44 ZB in 2020

More information

Developing with Google App Engine

Developing with Google App Engine Developing with Google App Engine Dan Morrill, Developer Advocate Dan Morrill Google App Engine Slide 1 Developing with Google App Engine Introduction Dan Morrill Google App Engine Slide 2 Google App Engine

More information

Session 7: Oracle R Enterprise OAAgraph Package

Session 7: Oracle R Enterprise OAAgraph Package Session 7: Oracle R Enterprise 1.5.1 OAAgraph Package Oracle Spatial and Graph PGX Graph Algorithms Oracle R Technologies Mark Hornick Director, Oracle Advanced Analytics and Machine Learning July 2017

More information

BIG DATA TECHNOLOGIES: WHAT EVERY MANAGER NEEDS TO KNOW ANALYTICS AND FINANCIAL INNOVATION CONFERENCE JUNE 26-29,

BIG DATA TECHNOLOGIES: WHAT EVERY MANAGER NEEDS TO KNOW ANALYTICS AND FINANCIAL INNOVATION CONFERENCE JUNE 26-29, BIG DATA TECHNOLOGIES: WHAT EVERY MANAGER NEEDS TO KNOW ANALYTICS AND FINANCIAL INNOVATION CONFERENCE JUNE 26-29, 2016 1 OBJECTIVES ANALYTICS AND FINANCIAL INNOVATION CONFERENCE JUNE 26-29, 2016 2 WHAT

More information

Introduction to Graph Databases

Introduction to Graph Databases Introduction to Graph Databases David Montag @dmontag #neo4j 1 Agenda NOSQL overview Graph Database 101 A look at Neo4j The red pill 2 Why you should listen Forrester says: The market for graph databases

More information

Research challenges in data-intensive computing The Stratosphere Project Apache Flink

Research challenges in data-intensive computing The Stratosphere Project Apache Flink Research challenges in data-intensive computing The Stratosphere Project Apache Flink Seif Haridi KTH/SICS haridi@kth.se e2e-clouds.org Presented by: Seif Haridi May 2014 Research Areas Data-intensive

More information

The Hadoop Ecosystem. EECS 4415 Big Data Systems. Tilemachos Pechlivanoglou

The Hadoop Ecosystem. EECS 4415 Big Data Systems. Tilemachos Pechlivanoglou The Hadoop Ecosystem EECS 4415 Big Data Systems Tilemachos Pechlivanoglou tipech@eecs.yorku.ca A lot of tools designed to work with Hadoop 2 HDFS, MapReduce Hadoop Distributed File System Core Hadoop component

More information

Exploring the Structure of Data at Scale. Rudy Agovic, PhD CEO & Chief Data Scientist at Reliancy January 16, 2019

Exploring the Structure of Data at Scale. Rudy Agovic, PhD CEO & Chief Data Scientist at Reliancy January 16, 2019 Exploring the Structure of Data at Scale Rudy Agovic, PhD CEO & Chief Data Scientist at Reliancy January 16, 2019 Outline Why exploration of large datasets matters Challenges in working with large data

More information

Scaling Up HBase. Duen Horng (Polo) Chau Assistant Professor Associate Director, MS Analytics Georgia Tech. CSE6242 / CX4242: Data & Visual Analytics

Scaling Up HBase. Duen Horng (Polo) Chau Assistant Professor Associate Director, MS Analytics Georgia Tech. CSE6242 / CX4242: Data & Visual Analytics http://poloclub.gatech.edu/cse6242 CSE6242 / CX4242: Data & Visual Analytics Scaling Up HBase Duen Horng (Polo) Chau Assistant Professor Associate Director, MS Analytics Georgia Tech Partly based on materials

More information

Moving from RELATIONAL TO NoSQL: Relational to NoSQL:

Moving from RELATIONAL TO NoSQL: Relational to NoSQL: Moving from RELATIONAL TOtoNoSQL: Relational NoSQL: GETTING STARTED SQL SERVER HOW TOFROM GET STARTED Moving from Relational to NoSQL: How to Get Started Why the shift to NoSQL? NoSQL has become a foundation

More information

Real-time Fraud Detection with Innovative Big Graph Feature. Gaurav Deshpande, VP Marketing, TigerGraph; Mingxi Wu, VP Engineering, TigerGraph

Real-time Fraud Detection with Innovative Big Graph Feature. Gaurav Deshpande, VP Marketing, TigerGraph; Mingxi Wu, VP Engineering, TigerGraph Real-time Fraud Detection with Innovative Big Graph Feature Gaurav Deshpande, VP Marketing, TigerGraph; Mingxi Wu, VP Engineering, TigerGraph Speaking Today Gaurav Deshpande VP Marketing, TigerGraph gaurav@tigergraph.com

More information

Understanding the SAP HANA Difference. Amit Satoor, SAP Data Management

Understanding the SAP HANA Difference. Amit Satoor, SAP Data Management Understanding the SAP HANA Difference Amit Satoor, SAP Data Management Webinar Logistics Got Flash? http://get.adobe.com/flashplayer to download. The future holds many transformational opportunities Capitalize

More information