Indiana Oracle Users Group January meeting January 27 th, Big Data

Transcription

1 Indiana Oracle Users Group January meeting January 27 th, 2017 Big Data

2 Agenda Big Data Hype RDBMS Vs. Big Data Data explosion Why do we need big data? How to approach? Big data process cycle Machine Learning Oracle Big Data Solutions Big Data and Cloud Use cases Expanding DBA/Sys Admin/Developer role into Big Data space 2

3 About me Ashokkumar Sivasankaran (Ashok) Database Architect, Ensono Twenty eight years in IT services OCE RAC Expert & OCP Database Administrator 7.3 to 11g AWS Solution Architect - Associate ITIL V3 Foundation Certified Member IOUG, COUG and INOUG 3

4 Snapshot of Ensono Client-first philosophy Hybrid IT services mainframe to cloud and everything in between Expertise managing missioncritical infrastructure Agile development to drive innovation Global capabilities Revenue of $265 million 4

5 Ensono s Presence Around the World Headquarters Chicago, IL Data Centers Downers Grove, IL Elk Grove Village, IL Conway, AR Little Rock, AR London, UK Slough, UK Delivery Centers Onshore IL and AR Offshore UK, Poland, Germany, India 5

6 The Power in Numbers NEARLY 20,000 SERVERS UNDER MANAGEMENT 17,700 DATABASES MANAGED 80,000 MAINFRAME MIPS UNDER MANAGEMENT 10 OVER 1.4 MILLION PETABYTES OF STORAGE MANAGED TECHNOLOGY COMPONENTS MONITORED DATA CENTER AVAILABILITY 100% ACHIEVED OVER LAST 5 YEARS 100% SECURITY AUDITS PASSED BY CLIENTS 30 GIGABITS PER SECOND OF INTERNET BANDWIDTH CAPACITY 6

7 Exceeding Expectations for Over 47 Years Ranked #1 in Customer Satisfaction by The Black Book of Outsourcing (2010) Received Uptime Institute Award for 100% Data Center availability ( ) Recognized by Sunday Times as Top 100 Companies to Work For ( ) Launched cloud platform(2007) Ranked in Top 3 for Technology Innovation by InformationWeek (2011) Ranked in Gartner Magic Quadrant, Data Center Outsourcing, North America ( ) Ranked in Gartner Magic Quadrant, Cloud Managed Hosting, Europe ( ) 7

8 Big Data Hype Source: Gartner, 2013 Hype Cycle for Emerging Technologies 8

9 Big Data Hype 9

10 Big Data Hype Last 15 years we heard about Big data, Hadoop, Data analytics, Data Visualization, Data Science Is it still in hype or matured? Answer is Yes and No Many companies are using Big data extensively and growing their business Social media, Non-profit organizations, Sports, Research, Medical, Space, Political campaigns Some are in planning, evaluation stages Many are still new to Big Data So, potential growth opportunities 10

11 Timeline 2003 Google File system paper released ( Hadoop is born 2006 Yahoo deploys 300 nodes Hadoop cluster 2007 Yahoo runs two clusters of 1000 nodes 2008 Hadoop wins Terabyte sort benchmark 2009 Hadoop sorts a petabyte 2010 Yahoo 4,000 nodes handles 70 petabytes 11

12 RDBMS Vs Big Data RDBMS Relational Database Management System(RDBMS), invented by E. F. Codd at IBM in Structured data stored in rows and columns in normalized form Improves data access and reduces data duplication. i.e., employee master table and monthly salary table Major RDBMS are: Oracle, Sqlserver, MySql, DB2 and Sybase It will continue to manage OLTP needs Big data Superset of all types of data including data managed in RDBMS All kinds of digital data (rdbms data, text, video, images, sensor data) Mainly unstructured Very large volume Big data processing is an alternate to expensive supercomputing model to some extent 12

13 Big Data 13

14 Data Explosion 1 Bit = Binary Digit 8 Bits = 1 Byte 1024 Bytes = 1 Kilobyte 1024 Kilobytes = 1 Megabyte 1024 Megabytes = 1 Gigabyte 1024 Gigabytes = 1 Terabyte 1024 Terabytes = 1 Petabyte 1024 Petabytes = 1 Exabyte 1024 Exabytes = 1 Zettabyte 1024 Zettabytes = 1 Yottabyte 1024 Yottabytes = 1 Brontobyte 1024 Brontobytes = 1 Geopbyte Source: Infragistics 14

15 Data explosion reasons Social Networks Mobile IOT devices Sensors Clickstream Research Existing data growth, capturing detailed attributes 15

16 Why do we need Big Data? Better understanding about customer preference, market trends Predict, plan and take proactive actions Converting challenges and changes to growth opportunities Thinking ahead of the curve, product issue postings at social network(i.e., about a new cell phone) Out of box thinking from pattern, correlation, outliers Nowcasting vs Forecasting Nowcasting methods based on social media content (Google, Twitter) have been developed to estimate hidden quantities such as the 'mood' of a population or the presence of a flu epidemic. 16

17 Big Data Process Cycle 17

18 Big Data Landscape Source: matturck.com 18

19 How to approach Big Data Project? Start with questions What we want to know? How do we want it? What will be the benefit? Develop Big Data strategy Connect Stakeholders Identify data, data source, extracting, storing, analyzing and visualizing processes Identify right tools Architect a pilot project to meet your requirement Don t allow scope creep Validate results at each stage 19

20 Hadoop Apache Hadoop is an open-source software framework written in Java for distributed storage and distributed processing of very large data sets on computer clusters built from commodity hardware. All the modules in Hadoop are designed with a fundamental assumption that hardware failures (of individual machines, or racks of machines) are commonplace and thus should be automatically handled in software by the framework. Source: Wikipedia Hadoop is for offline batch processing and not for online and transaction processing(oltp) Hadoop consists of two primary components: HDFS Distributed file system for storage MapReduce Parallel processing framework 20

21 Hadoop Architecture 21

22 NoSql Databases NoSql Non Sql, Non Relational, Not only Sql databases Used to store Key-value type data, documents, graph, wide column data MongoDb (document, Native JSON support,..) Oracle NoSql AWS DnyamoDB Redis (Stores all data in memory, good for real-time analytics) Cassendra (developed at Facebook, key-value store column based) CouchDB ( JSON access over HTTP, best for web access) Hbase (integrated with Hadoop architecture, Tabular/columnar storage, non-relational, distributed database, best for complex computing jobs) 22

23 Hadoop Distributions, other tools and Software Hadoop Distributions o Amazon Web Services(EMR Elastic Map Reduce) o Cloudera o Hortonworks o MapR Other tools ZooKeeper Coordination framework Hive SQL like interface for querying Pig High level scripting language for batch processing and ETL 23

24 Hadoop Distributions, other tools and Software Flume Distributed service for collecting, aggregating, and moving large data Sqoop Bulk data transfer Mahout Data Mining utility using machine learning Hue Web console for Hadoop Oozie Management workflow and job scheduling Storm Distributed computation framework Kaffka open source message broker Impala Cloudera Impala is massively parallel processing SQL query engine for data stored in Hadoop cluster YARN Resource management system 24

25 Implementation Select software according to data type and processing requirement Use commodity low cost servers or virtual servers or cloud infrastructure Dynamic distribution and parallelism Efficient Map-Reduce, aggregation Parallel, in-memory, minimize network communication Fault tolerance and automatic recovery Moving computation to the data(moving data is expensive) Data processed in chunks in multiple nodes(distributed computing) High availability and scalability 25

26 BIG data workflow Support Area Resource Skills required Developing and Architecting Solution Big Data Architect Understanding Business need for Big data solution, data source, moving data, processing methods, infrastructure need and developing cost effective flexible solution Hardware, Storage and OS support Hardware, Storage, OS and VM Admin Linux admin experience to build physical or virtual linux servers to create Big data cluster Network and Security Network and Security Admin Providing secured network for the Linux servers Resource and Availability Monitoring Monitoring Admin Monitor servers and key processes Backup and restore Backup Admin Backup and restore identified file systems Big data admin(hadoop or other software) Scheduling Data collection, processing and sending data jobs Hadoop/No Sql(MongoDB/CouchDB/..) Admin Job scheduler/middleware Admin Administering Big data cluster Operation, Development and Middleware support Data collection and processing Developer Strong Java /Python programming and Shell scripting experience Analytics and Data visualization Data Analyst/ Data scientist Analyzing, pattern discovery and predicting 26

27 Data Science 27

28 Machine Learning and Deep Learning Machine learning is a type of artificial intelligence (AI) that provides computers with the ability to learn without being explicitly programmed. Machine learning focuses on the development of computer programs that can teach themselves to grow and change when exposed to new data. Source: Techtarget Supervised learning (classification/labeling i.e., spam filter) Unsupervised learning (clustering/association i.e., pattern identification) Deep Learning is a branch of machine learning. Many Deep Leaning algorithms are applied to unsupervised learning Neural networks, Tensorflow, MXNet 28

29 Machine Learning Source: Gartner, 2016 Hype Cycle for Emerging Technologies Note: Machine Learning is at the peak, IOT, Big Data disappeared 29

30 Data Analytics and Visualization Data analytics is the science of examining data using quantitative and qualitative techniques, finding relationship, pattern, outliers and predict. Data visualization is the presentation of data in pictorial format. It helps to understand the data, pattern, impacts and make decision. 30

31 R programming R is a programming language and software environment for statistical computing and graphics supported by the R Foundation for Statistical Computing. > table(awr_data$wait_class) Commit System I/O User I/O > pie(table(awr_data$wait_class)) 31

32 Oracle Big Data Solutions Oracle Big Data Cloud Service ( Oracle Big Data Appliance ( Oracle NoSql Database ( Oracle Big Data Sql ( Oracle Advanced Analytics ( Oracle Data integrator for Big data ( Oracle Data Mining ( Oracle provides Oracle Big Data Lite virtual machine for evaluation and educational purposes ( 32

33 Big Data and Cloud Big data processing handles large volume and complex data Data flow and processing time may change rapidly We want to spend less on IT projects Easy to add or remove resources in cloud on demand Easy to scale up or down resources in cloud according to data flow Cloud offers better cost savings solutions We can get comprehensive big data processing clusters and visualization services in cloud 33

34 Big Data Use Cases Sentiment Analysis 360 degree of customer view Internet of Things (IoT) Real Time Analytics Fraud deduction Clickstream Analysis Data warehouse optimization Case studies: 34

35 Expanding DBA/Sys Admin/Developer role into Big Data space You know about data, data access, data move, security, application processing, constraints You know about compute resources - cpu, memory, i/o, storage, network Now expand your role by Learning Hadoop Architecture Learning Python or Java or R programming Understand Cloud Services (ideal infrastructure for Big data processing) Understand Big data process cycle Identify preferred area based on your interest and strength Online material, Youtube, MOOC courses(edx, Coursera), Vendors resources are available to learn Practice using Cloud resources, Virtual Boxes, Online lab 35

36 Questions? 36

37 Thank you! 37