BIG DATA TECHNOLOGIES: WHAT EVERY MANAGER NEEDS TO KNOW ANALYTICS AND FINANCIAL INNOVATION CONFERENCE JUNE 26-29,

Transcription

1 BIG DATA TECHNOLOGIES: WHAT EVERY MANAGER NEEDS TO KNOW ANALYTICS AND FINANCIAL INNOVATION CONFERENCE JUNE 26-29,

2 OBJECTIVES ANALYTICS AND FINANCIAL INNOVATION CONFERENCE JUNE 26-29,

3 WHAT IS BIG DATA? A term used for data sets that are too large for traditional data processing applications to handle in a timely manner ANALYTICS AND FINANCIAL INNOVATION CONFERENCE JUNE 26-29,

4 WHAT IS BIG DATA? IT S A TERM, NOT A TECHNOLOGY NoSQL ANALYTICS AND FINANCIAL INNOVATION CONFERENCE JUNE 26-29,

5 TRADITIONAL VS BIG DATA columns Customer Id Name Account Balance 100 Bill Smith Mary Evans John Doe 1.99 rows Example SQL code: INSERT INTO USERS (`user_name`,`user_password`) VALUES ('andrea123','mypassword') SELECT Customer_ID, Name FROM CUSTOMERS WHERE customer_id=100 AND Account_Balance => You can t add a record that does not fit ANALYTICS AND FINANCIAL INNOVATION CONFERENCE JUNE 26-29,

6 TRADITIONAL VS BIG DATA Client Node 1 Client Node 2 Client Node 3 Client Node 4 Client Node 5. Master Information Repository & Data Nodes / Subsets Reduce() Reduce() Reduce() Real Time Processing Map() Map() Map() Map() Map() Map() Real Time Processing New real-time Internet Crawled Inforamation Updated, known Information Sources Manually added / historical information 3 rd party participants profilesi.e. Global distribution network / resellers New Data / Time Sensitive Data structure and format are often unknown in advance ANALYTICS AND FINANCIAL INNOVATION CONFERENCE JUNE 26-29,

7 TWO UNIVERSES NEW REALITY Structured Data RDBMS -SQL Un-Structured Data NoSQL ANALYTICS AND FINANCIAL INNOVATION CONFERENCE JUNE 26-29,

8 NoSQL is not a replacement for SQL it s an alternative SQL NoSQL Data Storage Flexibility Relational Model; rows and columns; limited associations, more expensive data storage Each record conforms to a strict schema; each new row must contain data for each column in a table; schema is decided before data entry and hard to change; easier to learn All different types of data storage, each with a different model; data relationships are often determined after load Schemas are dynamic, information can be added on the fly; harder to find experts Scalability Vertical, no parallel processing Horizontal, parallel processing ACID compliance Best Uses Compliant, can have data validation; designed for fault tolerance Enforces data integrity, useful in repeatable high volume transactions Varies between technologies, validation processes, size of data corpus, quality of inputs, fault tolerance = extra csot Need behavior, correlation analysis, have variable transactions of large volumes; preferred choice for mobile and web development, behavior analysis 8

9 THE FOUR V S OF BIG DATA TECHNOLOGIES Volume (Scale of Data; collecting data at high speeds) 6B cell phones 20TB of Facebook texts 471M Visa debit cards 92M suspected fraud 100 Car sensors per car 179B GB health care data No SQL Variety (Different Formats of Data) Music downloads Web searches Cell phone logs GIS Documents, profiles Blog posts Structured transactions Video languages Biometric Sensor feeds Velocity (Analysis, and Output; Acting on data in a timely manner) 1.5 TB of daily stock trade info 19B network connections 30B pieces social media posts daily 400M daily tweets 500 million held desk calls (US) 100M weather sensor inputs 500 million help desk calls Veracity (Quality, Integrity, and Validity) Whole populations of data = greater validation capability 1 in 3 leaders don t trust the information they use to make decisions 30% of marketing data cannot be validated Quality of trustworthiness of data input, mapping and output JUNE 26-29,

10 NOSQL INFRASTRUCTURE EXAMPLES ANALYTICS AND FINANCIAL INNOVATION CONFERENCE JUNE 26-29,

11 HADOOP ANALYTICS AND FINANCIAL INNOVATION CONFERENCE JUNE 26-29,

12 Hadoop Cluster Master Node Job Tracker Sharding HDFS = Hadoop Distributed File System Sharding 1,4 2,4 3,6 Data node replication across nodes JUNE 26-29,

13 NOSQL DATABASE OPTIONS DECIDE THE PROBLEM, THEN PICK THE TOOL Data Base Type Description Examples Key Value Column Based Document Databases Graph All of the data consists within an indexed key and value. EXTREMELY simple, FAST. There is a key and there is the rest of the data (the values) and that s it. There are no alternate keys and no foreign keys and no broad text searching capabilities against the values. Stores data tables as sections of columns of data, but not rows. It s a key value pair only with multiple values and keys in a columnar format (two dimensional arrays whereby each key (i.e. row / record) has one or more key / value pairs attached to it). Best for very large blobs of data attached to a columnar key reference. Processes and stores unstructured data and structured data as documents, each document has a unique key used to reference or retrieve the entire document. Best for the largest variety of data. Graphical input with interconnected elements with a large, undetermined number of relations between them Web session tracking, multi-player on-line gaming to manage each player session, shopping carts for online buyers Redis MemcacheDB Voldemort Flight data, banking transaction analysis, realtime credit scoring Casandra Hbase Chats, tweets, blogs, s, documents, spreadsheets, user comments combined with structured or semi structured data. Behavior analysis, election predictions, shipping estimates, event logs, etc. MongoDB CouchDB Social relations, public transport links, road map, network toplogies OrientDB Neo4J ANALYTICS AND FINANCIAL INNOVATION CONFERENCE JUNE 26-29,

14 WHAT IS MAP REDUCE AND WHY DO I NEED TO KNOW? 14 JUNE 26-29,

15 PUTTING IT ALL TOGETHER! 15

16 QUESTIONS? ANALYTICS AND FINANCIAL INNOVATION CONFERENCE JUNE 26-29,