BIG DATA TESTING: A UNIFIED VIEW
|
|
- Bertha Black
- 5 years ago
- Views:
Transcription
1 BIG DATA TESTING: A UNIFIED VIEW BY NAM THAI ECU, Computer Science Department, March 16, 2016
2 2/30 PRESENTATION CONTENT 1. Overview of Big Data A. 5 V s of Big Data B. Data generation C. Data acquisition D. Data pre-processing E. Data analysis F. Apache Hadoop 2. Testing Big Data A. Database Testing B. Application Testing C. Performance Testing D. Traditional Testing vs Big Data Testing
3 3/30 WHAT? WHY? Comparative Definition: Data sets whose size is beyond the ability of typical database software tools to capture, store, manage and analyze -McKinsey Global Institute Report Moore s law (generation / storage) Industry demand (science, business, etc..) Traditional RDBMS not enough Barrack Obama (200$) million Y. Demchenko; C. Laat; P. Membrey; 2014
4 4/30 HOW? Increasing data storage capability Physical data (cost) 228$ ->.88$GB Virtualized data (size) NoSQL data management Distributed computing networks (parallel processing/cloud computing) Improved network latency (speed) Advances in data analysis ( machine learning ) Google invents MapReduce Y. Demchenko; C. Laat; P. Membrey; 2014 Images:
5 5/30 WHAT IS BIG DATA? Attributive Definition: Big Data Technologies describe a new generation of technologies and architectures, designed to economically extract value from very large volumes of a wide variety of data, by enabling highvelocity capture, discovery and/or analysis -International Data Company
6 6/30 THE 4 V S OF BIG DATA: Volume: Terabytes and Petabytes of storage Database functionality to handle TB & PB Velocity: TB/sec. data transfer rates Variety: Data Types: text, video, images, speech etc... Data Source: Many sources, from varying distances, at varying speeds Value: Analysis: data analysis applications Veracity: data must be correct Data cleansing: removing noise and correcting errors Demchenko, Y.; de Laat, C.; Membrey, P., "Defining architecture components of the Big Data Ecosystem,"
7 7/30 LAYERED VIEW OF BIG DATA Application Layer Data analysis Query and clustering Infrastructure Layer Network storage resources Network computation resources Data classification Computing Layer Programming models Data management: NoSQL, Files System Data Integration Y. Demchenko; C. Laat; P. Membrey; 2014
8 8/30 BIG DATA LIFE CYCLE 1. Data Generation Attributes Sources II. Data Acquisition Collection Transmission Pre-processing III. Data Storage File systems Database technologies Programming models IV. Data Analysis Analysis techniques Analysis paradigms Y. Demchenko; C. Laat; P. Membrey; 2014
9 I. Data Generation II. Data Acquisition III. Data Storage IV. Data Analysis 9/30 1. DATA GENERATION The Data in Big Data Volume: Petabytes & Exabytes Velocity: PB/sec or real-time Variety: text, image, video, logs, reports Value: source of data Domain specificity
10 I. Data Generation II. Data Acquisition III. Data Storage IV. Data Analysis 10/30 1. DATA GENERATION CONT. Business Data: Stock market, internet purchases, business to business Billions of transactions per day Networking Data: Internet: Google 30 Pb/day Data is already available! Amazon RedShift Petabyte sized data warehouse Social Networking: Facebook 30Pb/day Internet of things: 30 million networked sensors Scientific Data: Astronomy: 20 TB of images a night High-Energy Physics: LHC 2Pb/second
11 I. Data Generation II. Data Acquisition III. Data Storage IV. Data Analysis 11/30 II. DATA ACQUISITION: DATA COLLECTION Two categories Pull-Based approach Push-Based approach Common Collection methods: Sensors: Physical to Digital Pull-Based approach Log File: Record activity of software systems Push-Based approach Web Crawler: Collecting URLs for search engines Pull-Based approach
12 I. Data Generation II. Data Acquisition III. Data Storage IV. Data Analysis 12/30 II. DATA ACQUISITION: DATA TRANSMISSION Transfer collected data into storage infrastructure Transfer via IP Back Bone Region or Internet scale High Capacity Transfer Data Center Transmission Data Center Network Architecture Consists of racks of servers Connected by internal network Transportation Protocols Governs data transmission within data center
13 I. Data Generation II. Data Acquisition III. Data Storage IV. Data Analysis 13/30 II. DATA ACQUISITION: PRE-PROCESSING Data quality is critical Reduce noise and redundancy Increase consistency Integration Combining data into a unified view Distributed sources Data needs to be standardized Cleansing Search for inaccurate, incomplete, irrelevant data Requires data rules Amend or remove bad data Redundancy Elimination Reduce transmission overhead Prevent wasted storage space, inconsistency Data corruption can destroy databases
14 I. Data Generation II. Data Acquisition III. Data Storage IV. Data Analysis 14/30 III. DATA STORAGE: STORAGE INFRASTRUCTURE Store collected data into format for analysis Physical Storage Random access memory (RAM) Magnetic disk (HDD) Storage class memory (SDD) Optical / Tape storage (Big Data Obsolete) Network Infrastructure Direct Attached Storage (DAS) Network Attached Storage (NAS) Storage Area Network (SAN) Attributes Persistent and reliable Infrastructure must be able to scale up and down to meet application demand SAN networks allows virtualization Virtualization allows multiple networks to function as a single storage device
15 I. Data Generation II. Data Acquisition III. Data Storage IV. Data Analysis 15/30 III. DATA STORAGE: DATA MANAGEMENT FRAMEWORK File Systems Google File System (GFS) Scalable distributed file system Fault tolerance High performance for large number of clients Distributed over clusters of commodity servers Hadoop File System (HDFS) Open-source based on GFS Database Technologies NoSQL systems Schema free Easy replication (for distribution) Support huge amounts of data Simple API
16 I. Data Generation II. Data Acquisition III. Data Storage IV. Data Analysis 16/30 III. DATA STORAGE: PROGRAMMING MODELS Programming models provide application logics that allow large data processing Generic programming model Stream programming model Batch programming model Generic Process Model MapReduce invented by google MapReduce most widely used in big data ecosystem Allows distributed processing Can be integrated with SQL MapReduce consists of three main phases: Map() - Data objects are mapped based analysis constraints Shuffle() Consolidates mapped objects into classes Reduce() Aggregate all shuffled objects into one
17 I. Data Generation II. Data Acquisition III. Data Storage IV. Data Analysis 17/30 IV. DATA ANALYSIS Data Analysis methods are domain specific! Goals of Data analysis Extrapolate and interpret data Check legitimacy of data Assist decision making Predict future trends Provide recommendations Types of Data Analysis Descriptive analytics Uses historical data to describe a trend or occurrence Usually translated to graphical visualizations Associated with business intelligence Predictive analytics Uses data to predict future trends or probabilities Utilizes data mining to calculate predictions Statistical techniques used to interpret data Prescriptive analytics Uses data to diagnose and infer information to assist decision making
18 I. Data Generation II. Data Acquisition III. Data Storage IV. Data Analysis 18/30 IV. DATA ANALYSIS: ANALYSIS PARADIGMS Stream Processing Model (Real-time) Analyze as soon as possible Data value relies on data freshness High processing speed Little raw data is stored Batch Processing Model Analysis of large batches of data MapReduce is most common batchprocessing model Processing is scheduled near the data location
19 I. Data Generation II. Data Acquisition III. Data Storage IV. Data Analysis 19/30 IV. DATA ANALYSIS: DATA ANALYSIS TECHNIQUES Data Mining Computational process of discovering patterns in data sets Data mining is used in: Artificial intelligence, machine learning, pattern recognition, statistics etc. Types of data mining algorithms Classification Clustering Data Visualization Information graphics and visualization Graphical data representation is easy to understand Due to volume and variety of data, visualization is needed Visualization can assist Algorithm design Software development Regression Statistical learning Association analysis
20 20/30 APACHE HADOOP Leading Big Data Industry and Academia Open-Sourced Companies using Hadoop: Amazon, LinkedIn, IBM, Microsoft and Intel Adapted from Google s MapReduce Scalability & Flexibility Clusters and servers can be added/removed without interrupting system Because of Java base, is compatible on all platforms Fault tolerance Does not rely on hardware for FT Hadoop library designed to detect and handle failures on application level Images:
21 21/30 BIG DATA TESTING I. Database Testing II. Application Testing III. Performance Testing IV. Traditional Testing vs Big Data Testing
22 22/30 DATABASE TESTING: VERIFICATION OF DATABASE Testing is Domain specific Testers must know how to cover integrity constraints Testers must cater data to trigger integrity constraints Every integrity constraint must be tested Traditional verification: Manually filling tables with data copied from other sources Random data (Only good for performance and load testing) Implementing custom, domain specific, data generators Big Database Functionality validation requires: AUTOMATION Generating formatted and unformatted data Parse & interpret integrity constraints Generate data to trigger data rules Validate the correctness of generated data Correctness of data structure Ability to trigger integrity constraints Sneed, H.M.; Erdoes, K., "Testing big data (Assuring the quality of large databases),"
23 23/30 BIG DATA TESTING Goal: Verify Big Data application Will examine big data testing Hadoop System Three main steps to Big Data testing 1. Data staging validation II. MapReduce Validation III. Output Validation
24 24/30 BIG DATA TESTING STEPS Step I: Data Staging Validation Pre-Hadoop Phase Validate data pulled from data sources Compare data to data pushed into system Verify the data is loaded correctly into HDFS Step II: MapReduce Validation Ensuring that the Map Reduce process works correctly Verify correct assess to all nodes Data aggregation/segregation rules are implemented on the data Map function key value pairs are generated Validating the data after Reduce process Step III: Output Validation Analyzing data output of Hadoop Confirm that transformation rules are correct Check data integrity and destination loading Detect corruption by comparing output with HDFS
25 Sneed, H.M.; Erdoes, K., "Testing big data (Assuring the quality of large databases)," 25/30 PERFORMANCE TESTING A form of non-functional testing which simulate load conditions Detect bottlenecks and performance issues Provide benchmark data on system Central Characteristics Response time (Faster response time) Resource use (Efficient resource use) Stability (Reliable Stability) A. Alexandrov, C. Brücke, and V. Markl, Issues in Big Data Testing and Benchmarking,
26 26/30 PERFORMANCE TESTING Performance testing focuses on improving the 4 V s Performance Test Types Concurrent test Tests the concurrent usage of a specific block of the Big Data system Determines any problems with many concurrent users Load testing Testing the performance of Big Data system in different load levels to determine levels of performance. Provides reliability and stability metrics Focuses on user transactions with the system Stress testing Examine system performance in the most extreme conditions of concurrent users and user transactions Provides metrics on peak loads and failure conditions Reveals weaknesses in system Capacity testing Determine maximum resource loads available to the system Provides metrics on physical limitations of the system Provides metrics on maximum concurrent users and maximum simultaneous transactions A. Alexandrov, C. Brücke, and V. Markl, 2015
27 27/30 TRADITIONAL VS BIG DATA TESTING Attribute Traditional Database Testing Big Data Testing Data Structured Testing is well defined & established Can use manual sampling of data Structured and unstructured data Testing requires analysis of big data system domain Requires automation Infrastructure Does not require test environment Requires test environment due to large data sizes Validation Tools Excel based or UI based automation tools Does not require domain knowledge or extensive training No defined universal tools Tools require skills and training Requires knowledge of specific big data systems A. Alexandrov, C. Brücke, and V. Markl,
28 28/30 BIG DATA TESTING CHALLENGES Automation* Much of the testing approaches need to be automated Due to size, speed, and complexity of data Automation cannot handle unexpected problems in testing process Generating Data Create very large realistic data sets Interpreting data rules from data Requires machine learning and artificial intelligence Cross-Platform Testing Tools Testing applications are application/domain specific Standardized Testing Framework Testing frameworks for big data in their infancy Variety of big data systems and system components creates problems
29 29/30 CONCLUSIONS Big Data Attributes Volume Velocity Variety Value Big Data Life Cycle 1. Generation 2. Acquisition 3. Storage 4. Analysis Big Data Application Google s MapReduce Apache Hadoop Big Data Testing Database testing Application testin Performance Testing Traditional vs. Big Data Testing Big Data Testing Challenges
30 ` Thank You!
Nowcasting. D B M G Data Base and Data Mining Group of Politecnico di Torino. Big Data: Hype or Hallelujah? Big data hype?
Big data hype? Big Data: Hype or Hallelujah? Data Base and Data Mining Group of 2 Google Flu trends On the Internet February 2010 detected flu outbreak two weeks ahead of CDC data Nowcasting http://www.internetlivestats.com/
More informationTopics. Big Data Analytics What is and Why Hadoop? Comparison to other technologies Hadoop architecture Hadoop ecosystem Hadoop usage examples
Hadoop Introduction 1 Topics Big Data Analytics What is and Why Hadoop? Comparison to other technologies Hadoop architecture Hadoop ecosystem Hadoop usage examples 2 Big Data Analytics What is Big Data?
More informationBased on Big Data: Hype or Hallelujah? by Elena Baralis
Based on Big Data: Hype or Hallelujah? by Elena Baralis http://dbdmg.polito.it/wordpress/wp-content/uploads/2010/12/bigdata_2015_2x.pdf 1 3 February 2010 Google detected flu outbreak two weeks ahead of
More informationData Management Glossary
Data Management Glossary A Access path: The route through a system by which data is found, accessed and retrieved Agile methodology: An approach to software development which takes incremental, iterative
More informationEmbedded Technosolutions
Hadoop Big Data An Important technology in IT Sector Hadoop - Big Data Oerie 90% of the worlds data was generated in the last few years. Due to the advent of new technologies, devices, and communication
More informationBig Data with Hadoop Ecosystem
Diógenes Pires Big Data with Hadoop Ecosystem Hands-on (HBase, MySql and Hive + Power BI) Internet Live http://www.internetlivestats.com/ Introduction Business Intelligence Business Intelligence Process
More informationData Clustering on the Parallel Hadoop MapReduce Model. Dimitrios Verraros
Data Clustering on the Parallel Hadoop MapReduce Model Dimitrios Verraros Overview The purpose of this thesis is to implement and benchmark the performance of a parallel K- means clustering algorithm on
More informationChapter 3. Foundations of Business Intelligence: Databases and Information Management
Chapter 3 Foundations of Business Intelligence: Databases and Information Management THE DATA HIERARCHY TRADITIONAL FILE PROCESSING Organizing Data in a Traditional File Environment Problems with the traditional
More informationWhat is the maximum file size you have dealt so far? Movies/Files/Streaming video that you have used? What have you observed?
Simple to start What is the maximum file size you have dealt so far? Movies/Files/Streaming video that you have used? What have you observed? What is the maximum download speed you get? Simple computation
More informationData Platforms and Pattern Mining
Morteza Zihayat Data Platforms and Pattern Mining IBM Corporation About Myself IBM Software Group Big Data Scientist 4Platform Computing, IBM (2014 Now) PhD Candidate (2011 Now) 4Lassonde School of Engineering,
More informationThe amount of data increases every day Some numbers ( 2012):
1 The amount of data increases every day Some numbers ( 2012): Data processed by Google every day: 100+ PB Data processed by Facebook every day: 10+ PB To analyze them, systems that scale with respect
More informationHadoop/MapReduce Computing Paradigm
Hadoop/Reduce Computing Paradigm 1 Large-Scale Data Analytics Reduce computing paradigm (E.g., Hadoop) vs. Traditional database systems vs. Database Many enterprises are turning to Hadoop Especially applications
More information2/26/2017. The amount of data increases every day Some numbers ( 2012):
The amount of data increases every day Some numbers ( 2012): Data processed by Google every day: 100+ PB Data processed by Facebook every day: 10+ PB To analyze them, systems that scale with respect to
More informationBig Data Analytics. Izabela Moise, Evangelos Pournaras, Dirk Helbing
Big Data Analytics Izabela Moise, Evangelos Pournaras, Dirk Helbing Izabela Moise, Evangelos Pournaras, Dirk Helbing 1 Big Data "The world is crazy. But at least it s getting regular analysis." Izabela
More informationChapter 6 VIDEO CASES
Chapter 6 Foundations of Business Intelligence: Databases and Information Management VIDEO CASES Case 1a: City of Dubuque Uses Cloud Computing and Sensors to Build a Smarter, Sustainable City Case 1b:
More informationBig Data com Hadoop. VIII Sessão - SQL Bahia. Impala, Hive e Spark. Diógenes Pires 03/03/2018
Big Data com Hadoop Impala, Hive e Spark VIII Sessão - SQL Bahia 03/03/2018 Diógenes Pires Connect with PASS Sign up for a free membership today at: pass.org #sqlpass Internet Live http://www.internetlivestats.com/
More informationMicrosoft Big Data and Hadoop
Microsoft Big Data and Hadoop Lara Rubbelke @sqlgal Cindy Gross @sqlcindy 2 The world of data is changing The 4Vs of Big Data http://nosql.mypopescu.com/post/9621746531/a-definition-of-big-data 3 Common
More informationA Review Paper on Big data & Hadoop
A Review Paper on Big data & Hadoop Rupali Jagadale MCA Department, Modern College of Engg. Modern College of Engginering Pune,India rupalijagadale02@gmail.com Pratibha Adkar MCA Department, Modern College
More informationCloud Analytics and Business Intelligence on AWS
Cloud Analytics and Business Intelligence on AWS Enterprise Applications Virtual Desktops Sharing & Collaboration Platform Services Analytics Hadoop Real-time Streaming Data Machine Learning Data Warehouse
More informationBig Data Technology Ecosystem. Mark Burnette Pentaho Director Sales Engineering, Hitachi Vantara
Big Data Technology Ecosystem Mark Burnette Pentaho Director Sales Engineering, Hitachi Vantara Agenda End-to-End Data Delivery Platform Ecosystem of Data Technologies Mapping an End-to-End Solution Case
More informationCloud Computing Techniques for Big Data and Hadoop Implementation
Cloud Computing Techniques for Big Data and Hadoop Implementation Nikhil Gupta (Author) Ms. komal Saxena(Guide) Research scholar Assistant Professor AIIT, Amity university AIIT, Amity university NOIDA-UP
More informationVOLTDB + HP VERTICA. page
VOLTDB + HP VERTICA ARCHITECTURE FOR FAST AND BIG DATA ARCHITECTURE FOR FAST + BIG DATA FAST DATA Fast Serve Analytics BIG DATA BI Reporting Fast Operational Database Streaming Analytics Columnar Analytics
More informationTESTING BIG DATA WORLD RIGA. by Konstantin Pletenev OCTOBER, 2017, TAPOST GROW CONFIDENTLY
RIGA TESTING BIG DATA WORLD by Konstantin Pletenev OCTOBER, 2017, TAPOST GROW CONFIDENTLY BIG DATA IS NOT ABOUT THE DATA THE REVOLUTION IS NOT THAT THERE S MORE DATA AVAILABLE THE REVOLUTION IS THAT WE
More informationBig Data Programming: an Introduction. Spring 2015, X. Zhang Fordham Univ.
Big Data Programming: an Introduction Spring 2015, X. Zhang Fordham Univ. Outline What the course is about? scope Introduction to big data programming Opportunity and challenge of big data Origin of Hadoop
More informationHadoop An Overview. - Socrates CCDH
Hadoop An Overview - Socrates CCDH What is Big Data? Volume Not Gigabyte. Terabyte, Petabyte, Exabyte, Zettabyte - Due to handheld gadgets,and HD format images and videos - In total data, 90% of them collected
More informationMassive Online Analysis - Storm,Spark
Massive Online Analysis - Storm,Spark presentation by R. Kishore Kumar Research Scholar Department of Computer Science & Engineering Indian Institute of Technology, Kharagpur Kharagpur-721302, India (R
More informationThe Hadoop Ecosystem. EECS 4415 Big Data Systems. Tilemachos Pechlivanoglou
The Hadoop Ecosystem EECS 4415 Big Data Systems Tilemachos Pechlivanoglou tipech@eecs.yorku.ca A lot of tools designed to work with Hadoop 2 HDFS, MapReduce Hadoop Distributed File System Core Hadoop component
More informationDATA MINING II - 1DL460
DATA MINING II - 1DL460 Spring 2016 A second course in data mining http://www.it.uu.se/edu/course/homepage/infoutv2/vt16 Kjell Orsborn Uppsala Database Laboratory Department of Information Technology,
More informationModern Database Concepts
Modern Database Concepts Introduction to the world of Big Data Doc. RNDr. Irena Holubova, Ph.D. holubova@ksi.mff.cuni.cz What is Big Data? buzzword? bubble? gold rush? revolution? Big data is like teenage
More informationCSE6331: Cloud Computing
CSE6331: Cloud Computing Leonidas Fegaras University of Texas at Arlington c 2019 by Leonidas Fegaras Cloud Computing Fundamentals Based on: J. Freire s class notes on Big Data http://vgc.poly.edu/~juliana/courses/bigdata2016/
More informationEXTRACT DATA IN LARGE DATABASE WITH HADOOP
International Journal of Advances in Engineering & Scientific Research (IJAESR) ISSN: 2349 3607 (Online), ISSN: 2349 4824 (Print) Download Full paper from : http://www.arseam.com/content/volume-1-issue-7-nov-2014-0
More informationIntroduction to Data Mining and Data Analytics
1/28/2016 MIST.7060 Data Analytics 1 Introduction to Data Mining and Data Analytics What Are Data Mining and Data Analytics? Data mining is the process of discovering hidden patterns in data, where Patterns
More informationHadoop 2.x Core: YARN, Tez, and Spark. Hortonworks Inc All Rights Reserved
Hadoop 2.x Core: YARN, Tez, and Spark YARN Hadoop Machine Types top-of-rack switches core switch client machines have client-side software used to access a cluster to process data master nodes run Hadoop
More informationLOG FILE ANALYSIS USING HADOOP AND ITS ECOSYSTEMS
LOG FILE ANALYSIS USING HADOOP AND ITS ECOSYSTEMS Vandita Jain 1, Prof. Tripti Saxena 2, Dr. Vineet Richhariya 3 1 M.Tech(CSE)*,LNCT, Bhopal(M.P.)(India) 2 Prof. Dept. of CSE, LNCT, Bhopal(M.P.)(India)
More informationdata-based banking customer analytics
icare: A framework for big data-based banking customer analytics Authors: N.Sun, J.G. Morris, J. Xu, X.Zhu, M. Xie Presented By: Hardik Sahi Overview 1. 2. 3. 4. 5. 6. Why Big Data? Traditional versus
More informationHierarchy of knowledge BIG DATA 9/7/2017. Architecture
BIG DATA Architecture Hierarchy of knowledge Data: Element (fact, figure, etc.) which is basic information that can be to be based on decisions, reasoning, research and which is treated by the human or
More informationStrategic Briefing Paper Big Data
Strategic Briefing Paper Big Data The promise of Big Data is improved competitiveness, reduced cost and minimized risk by taking better decisions. This requires affordable solution architectures which
More informationFlash Storage Complementing a Data Lake for Real-Time Insight
Flash Storage Complementing a Data Lake for Real-Time Insight Dr. Sanhita Sarkar Global Director, Analytics Software Development August 7, 2018 Agenda 1 2 3 4 5 Delivering insight along the entire spectrum
More informationCloud Computing 2. CSCI 4850/5850 High-Performance Computing Spring 2018
Cloud Computing 2 CSCI 4850/5850 High-Performance Computing Spring 2018 Tae-Hyuk (Ted) Ahn Department of Computer Science Program of Bioinformatics and Computational Biology Saint Louis University Learning
More information5 Fundamental Strategies for Building a Data-centered Data Center
5 Fundamental Strategies for Building a Data-centered Data Center June 3, 2014 Ken Krupa, Chief Field Architect Gary Vidal, Solutions Specialist Last generation Reference Data Unstructured OLTP Warehouse
More informationIntroduction to Hadoop. Owen O Malley Yahoo!, Grid Team
Introduction to Hadoop Owen O Malley Yahoo!, Grid Team owen@yahoo-inc.com Who Am I? Yahoo! Architect on Hadoop Map/Reduce Design, review, and implement features in Hadoop Working on Hadoop full time since
More informationNext-Generation Cloud Platform
Next-Generation Cloud Platform Jangwoo Kim Jun 24, 2013 E-mail: jangwoo@postech.ac.kr High Performance Computing Lab Department of Computer Science & Engineering Pohang University of Science and Technology
More informationBIG DATA AND HADOOP ON THE ZFS STORAGE APPLIANCE
BIG DATA AND HADOOP ON THE ZFS STORAGE APPLIANCE BRETT WENINGER, MANAGING DIRECTOR 10/21/2014 ADURANT APPROACH TO BIG DATA Align to Un/Semi-structured Data Instead of Big Scale out will become Big Greatest
More informationWhere We Are. Review: Parallel DBMS. Parallel DBMS. Introduction to Data Management CSE 344
Where We Are Introduction to Data Management CSE 344 Lecture 22: MapReduce We are talking about parallel query processing There exist two main types of engines: Parallel DBMSs (last lecture + quick review)
More informationHadoop, Yarn and Beyond
Hadoop, Yarn and Beyond 1 B. R A M A M U R T H Y Overview We learned about Hadoop1.x or the core. Just like Java evolved, Java core, Java 1.X, Java 2.. So on, software and systems evolve, naturally.. Lets
More information2014 年 3 月 13 日星期四. From Big Data to Big Value Infrastructure Needs and Huawei Best Practice
2014 年 3 月 13 日星期四 From Big Data to Big Value Infrastructure Needs and Huawei Best Practice Data-driven insight Making better, more informed decisions, faster Raw Data Capture Store Process Insight 1 Data
More informationBig Data on AWS. Peter-Mark Verwoerd Solutions Architect
Big Data on AWS Peter-Mark Verwoerd Solutions Architect What to get out of this talk Non-technical: Big Data processing stages: ingest, store, process, visualize Hot vs. Cold data Low latency processing
More informationCSE 124: Networked Services Lecture-17
Fall 2010 CSE 124: Networked Services Lecture-17 Instructor: B. S. Manoj, Ph.D http://cseweb.ucsd.edu/classes/fa10/cse124 11/30/2010 CSE 124 Networked Services Fall 2010 1 Updates PlanetLab experiments
More informationOverview of Data Services and Streaming Data Solution with Azure
Overview of Data Services and Streaming Data Solution with Azure Tara Mason Senior Consultant tmason@impactmakers.com Platform as a Service Offerings SQL Server On Premises vs. Azure SQL Server SQL Server
More informationHigh Performance Computing on MapReduce Programming Framework
International Journal of Private Cloud Computing Environment and Management Vol. 2, No. 1, (2015), pp. 27-32 http://dx.doi.org/10.21742/ijpccem.2015.2.1.04 High Performance Computing on MapReduce Programming
More informationA brief history on Hadoop
Hadoop Basics A brief history on Hadoop 2003 - Google launches project Nutch to handle billions of searches and indexing millions of web pages. Oct 2003 - Google releases papers with GFS (Google File System)
More informationA Review Approach for Big Data and Hadoop Technology
International Journal of Modern Trends in Engineering and Research www.ijmter.com e-issn No.:2349-9745, Date: 2-4 July, 2015 A Review Approach for Big Data and Hadoop Technology Prof. Ghanshyam Dhomse
More informationCISC 7610 Lecture 2b The beginnings of NoSQL
CISC 7610 Lecture 2b The beginnings of NoSQL Topics: Big Data Google s infrastructure Hadoop: open google infrastructure Scaling through sharding CAP theorem Amazon s Dynamo 5 V s of big data Everyone
More informationBig Data - Some Words BIG DATA 8/31/2017. Introduction
BIG DATA Introduction Big Data - Some Words Connectivity Social Medias Share information Interactivity People Business Data Data mining Text mining Business Intelligence 1 What is Big Data Big Data means
More informationScalable Tools - Part I Introduction to Scalable Tools
Scalable Tools - Part I Introduction to Scalable Tools Adisak Sukul, Ph.D., Lecturer, Department of Computer Science, adisak@iastate.edu http://web.cs.iastate.edu/~adisak/mbds2018/ Scalable Tools session
More informationHadoop File System S L I D E S M O D I F I E D F R O M P R E S E N T A T I O N B Y B. R A M A M U R T H Y 11/15/2017
Hadoop File System 1 S L I D E S M O D I F I E D F R O M P R E S E N T A T I O N B Y B. R A M A M U R T H Y Moving Computation is Cheaper than Moving Data Motivation: Big Data! What is BigData? - Google
More informationIntroduction to Big Data
Università degli Studi di Roma Tor Vergata Dipartimento di Ingegneria Civile e Ingegneria Informatica Introduction to Big Data Corso di Sistemi e Architetture per Big Data A.A. 2016/17 Valeria Cardellini
More informationBringing Data to Life
Bringing Data to Life Data management and Visualization Techniques Benika Hall Rob Harrison Corporate Model Risk March 16, 2018 Introduction Benika Hall Analytic Consultant Wells Fargo - Corporate Model
More informationBig Data and Cloud Computing
Big Data and Cloud Computing Presented at Faculty of Computer Science University of Murcia Presenter: Muhammad Fahim, PhD Department of Computer Eng. Istanbul S. Zaim University, Istanbul, Turkey About
More informationDecentralized Distributed Storage System for Big Data
Decentralized Distributed Storage System for Big Presenter: Wei Xie -Intensive Scalable Computing Laboratory(DISCL) Computer Science Department Texas Tech University Outline Trends in Big and Cloud Storage
More informationBig Trend in Business Intelligence: Data Mining over Big Data Web Transaction Data. Fall 2012
Big Trend in Business Intelligence: Data Mining over Big Data Web Transaction Data Fall 2012 Data Warehousing and OLAP Introduction Decision Support Technology On Line Analytical Processing Star Schema
More informationINTRODUCTION TO BIG DATA, DATA MINING, AND MACHINE LEARNING
CS 7265 BIG DATA ANALYTICS INTRODUCTION TO BIG DATA, DATA MINING, AND MACHINE LEARNING * Some contents are adapted from Dr. Hung Huang and Dr. Chengkai Li at UT Arlington Mingon Kang, PhD Computer Science,
More informationBIG DATA TECHNOLOGIES: WHAT EVERY MANAGER NEEDS TO KNOW ANALYTICS AND FINANCIAL INNOVATION CONFERENCE JUNE 26-29,
BIG DATA TECHNOLOGIES: WHAT EVERY MANAGER NEEDS TO KNOW ANALYTICS AND FINANCIAL INNOVATION CONFERENCE JUNE 26-29, 2016 1 OBJECTIVES ANALYTICS AND FINANCIAL INNOVATION CONFERENCE JUNE 26-29, 2016 2 WHAT
More informationChallenges for Data Driven Systems
Challenges for Data Driven Systems Eiko Yoneki University of Cambridge Computer Laboratory Data Centric Systems and Networking Emergence of Big Data Shift of Communication Paradigm From end-to-end to data
More informationInternational Journal of Advance Engineering and Research Development. A Study: Hadoop Framework
Scientific Journal of Impact Factor (SJIF): e-issn (O): 2348- International Journal of Advance Engineering and Research Development Volume 3, Issue 2, February -2016 A Study: Hadoop Framework Devateja
More informationAcquiring Big Data to Realize Business Value
Acquiring Big Data to Realize Business Value Agenda What is Big Data? Common Big Data technologies Use Case Examples Oracle Products in the Big Data space In Summary: Big Data Takeaways
More informationHiTune. Dataflow-Based Performance Analysis for Big Data Cloud
HiTune Dataflow-Based Performance Analysis for Big Data Cloud Jinquan (Jason) Dai, Jie Huang, Shengsheng Huang, Bo Huang, Yan Liu Intel Asia-Pacific Research and Development Ltd Shanghai, China, 200241
More informationBig data easily, efficiently, affordably. UniConnect 2.1
Connecting Data. Delivering Intelligence Big data easily, efficiently, affordably UniConnect 2.1 The UniConnect platform is designed to unify data in a highly scalable and seamless manner, by building
More informationOracle Database 11g for Data Warehousing & Big Data: Strategy, Roadmap Jean-Pierre Dijcks, Hermann Baer Oracle Redwood City, CA, USA
Oracle Database 11g for Data Warehousing & Big Data: Strategy, Roadmap Jean-Pierre Dijcks, Hermann Baer Oracle Redwood City, CA, USA Keywords: Big Data, Oracle Big Data Appliance, Hadoop, NoSQL, Oracle
More informationApache Spark is a fast and general-purpose engine for large-scale data processing Spark aims at achieving the following goals in the Big data context
1 Apache Spark is a fast and general-purpose engine for large-scale data processing Spark aims at achieving the following goals in the Big data context Generality: diverse workloads, operators, job sizes
More informationBig Data Hadoop Stack
Big Data Hadoop Stack Lecture #1 Hadoop Beginnings What is Hadoop? Apache Hadoop is an open source software framework for storage and large scale processing of data-sets on clusters of commodity hardware
More informationUNIFY DATA AT MEMORY SPEED. Haoyuan (HY) Li, Alluxio Inc. VAULT Conference 2017
UNIFY DATA AT MEMORY SPEED Haoyuan (HY) Li, CEO @ Alluxio Inc. VAULT Conference 2017 March 2017 HISTORY Started at UC Berkeley AMPLab In Summer 2012 Originally named as Tachyon Rebranded to Alluxio in
More informationThe Future of High Performance Computing
The Future of High Performance Computing Randal E. Bryant Carnegie Mellon University http://www.cs.cmu.edu/~bryant Comparing Two Large-Scale Systems Oakridge Titan Google Data Center 2 Monolithic supercomputer
More informationAgenda. AWS Database Services Traditional vs AWS Data services model Amazon RDS Redshift DynamoDB ElastiCache
Databases on AWS 2017 Amazon Web Services, Inc. and its affiliates. All rights served. May not be copied, modified, or distributed in whole or in part without the express consent of Amazon Web Services,
More informationCloud Computing & Visualization
Cloud Computing & Visualization Workflows Distributed Computation with Spark Data Warehousing with Redshift Visualization with Tableau #FIUSCIS School of Computing & Information Sciences, Florida International
More informationDistributed File Systems II
Distributed File Systems II To do q Very-large scale: Google FS, Hadoop FS, BigTable q Next time: Naming things GFS A radically new environment NFS, etc. Independence Small Scale Variety of workloads Cooperation
More informationProcessing Unstructured Data. Dinesh Priyankara Founder/Principal Architect dinesql Pvt Ltd.
Processing Unstructured Data Dinesh Priyankara Founder/Principal Architect dinesql Pvt Ltd. http://dinesql.com / Dinesh Priyankara @dinesh_priya Founder/Principal Architect dinesql Pvt Ltd. Microsoft Most
More informationLarge-Scale Data Engineering. Overview and Introduction
Large-Scale Data Engineering Overview and Introduction Administration Blackboard Page Announcements, also via email (pardon html formatting) Practical enrollment, Turning in assignments, Check Grades Contact:
More informationBig Data Architect.
Big Data Architect www.austech.edu.au WHAT IS BIG DATA ARCHITECT? A big data architecture is designed to handle the ingestion, processing, and analysis of data that is too large or complex for traditional
More informationHadoop محبوبه دادخواه کارگاه ساالنه آزمایشگاه فناوری وب زمستان 1391
Hadoop محبوبه دادخواه کارگاه ساالنه آزمایشگاه فناوری وب زمستان 1391 Outline Big Data Big Data Examples Challenges with traditional storage NoSQL Hadoop HDFS MapReduce Architecture 2 Big Data In information
More informationIntroduction to Big-Data
Introduction to Big-Data Ms.N.D.Sonwane 1, Mr.S.P.Taley 2 1 Assistant Professor, Computer Science & Engineering, DBACER, Maharashtra, India 2 Assistant Professor, Information Technology, DBACER, Maharashtra,
More informationMassively Parallel Processing. Big Data Really Fast. A Proven In-Memory Analytical Processing Platform for Big Data
Big Data Really Fast A Proven In-Memory Analytical Processing Platform for Big Data 2 Executive Summary / Overview: Big Data can be a big headache for organizations that have outgrown the practicality
More informationDatabases 2 (VU) ( / )
Databases 2 (VU) (706.711 / 707.030) MapReduce (Part 3) Mark Kröll ISDS, TU Graz Nov. 27, 2017 Mark Kröll (ISDS, TU Graz) MapReduce Nov. 27, 2017 1 / 42 Outline 1 Problems Suited for Map-Reduce 2 MapReduce:
More informationSQT03 Big Data and Hadoop with Azure HDInsight Andrew Brust. Senior Director, Technical Product Marketing and Evangelism
Big Data and Hadoop with Azure HDInsight Andrew Brust Senior Director, Technical Product Marketing and Evangelism Datameer Level: Intermediate Meet Andrew Senior Director, Technical Product Marketing and
More informationParallel Programming Principle and Practice. Lecture 10 Big Data Processing with MapReduce
Parallel Programming Principle and Practice Lecture 10 Big Data Processing with MapReduce Outline MapReduce Programming Model MapReduce Examples Hadoop 2 Incredible Things That Happen Every Minute On The
More informationBIG DATA & HADOOP: A Survey
Available Online at www.ijcsmc.com International Journal of Computer Science and Mobile Computing A Monthly Journal of Computer Science and Information Technology ISSN 2320 088X IMPACT FACTOR: 6.017 IJCSMC,
More informationIntroduction to Hadoop and MapReduce
Introduction to Hadoop and MapReduce Antonino Virgillito THE CONTRACTOR IS ACTING UNDER A FRAMEWORK CONTRACT CONCLUDED WITH THE COMMISSION Large-scale Computation Traditional solutions for computing large
More informationEvolving To The Big Data Warehouse
Evolving To The Big Data Warehouse Kevin Lancaster 1 Copyright Director, 2012, Oracle and/or its Engineered affiliates. All rights Insert Systems, Information Protection Policy Oracle Classification from
More informationCloud Computing and Hadoop Distributed File System. UCSB CS170, Spring 2018
Cloud Computing and Hadoop Distributed File System UCSB CS70, Spring 08 Cluster Computing Motivations Large-scale data processing on clusters Scan 000 TB on node @ 00 MB/s = days Scan on 000-node cluster
More informationChapter 6. Foundations of Business Intelligence: Databases and Information Management VIDEO CASES
Chapter 6 Foundations of Business Intelligence: Databases and Information Management VIDEO CASES Case 1a: City of Dubuque Uses Cloud Computing and Sensors to Build a Smarter, Sustainable City Case 1b:
More informationSpark, Shark and Spark Streaming Introduction
Spark, Shark and Spark Streaming Introduction Tushar Kale tusharkale@in.ibm.com June 2015 This Talk Introduction to Shark, Spark and Spark Streaming Architecture Deployment Methodology Performance References
More informationMOHA: Many-Task Computing Framework on Hadoop
Apache: Big Data North America 2017 @ Miami MOHA: Many-Task Computing Framework on Hadoop Soonwook Hwang Korea Institute of Science and Technology Information May 18, 2017 Table of Contents Introduction
More informationLecture 11 Hadoop & Spark
Lecture 11 Hadoop & Spark Dr. Wilson Rivera ICOM 6025: High Performance Computing Electrical and Computer Engineering Department University of Puerto Rico Outline Distributed File Systems Hadoop Ecosystem
More informationPart I What are Databases?
Part I 1 Overview & Motivation 2 Architectures 3 Areas of Application 4 History Saake Database Concepts Last Edited: April 2019 1 1 Educational Objective for Today... Motivation for using database systems
More informationIntroduction to MapReduce (cont.)
Introduction to MapReduce (cont.) Rafael Ferreira da Silva rafsilva@isi.edu http://rafaelsilva.com USC INF 553 Foundations and Applications of Data Mining (Fall 2018) 2 MapReduce: Summary USC INF 553 Foundations
More informationTITLE: PRE-REQUISITE THEORY. 1. Introduction to Hadoop. 2. Cluster. Implement sort algorithm and run it using HADOOP
TITLE: Implement sort algorithm and run it using HADOOP PRE-REQUISITE Preliminary knowledge of clusters and overview of Hadoop and its basic functionality. THEORY 1. Introduction to Hadoop The Apache Hadoop
More informationMellanox InfiniBand Solutions Accelerate Oracle s Data Center and Cloud Solutions
Mellanox InfiniBand Solutions Accelerate Oracle s Data Center and Cloud Solutions Providing Superior Server and Storage Performance, Efficiency and Return on Investment As Announced and Demonstrated at
More informationBig Data is Better on Bare Metal
WHITE PAPER Big Data is Better on Bare Metal Make big data performance a priority EXECUTIVE SUMMARY Today, businesses create and capture unprecedented amounts of data from multiple sources in structured
More information<Insert Picture Here> Introduction to Big Data Technology
Introduction to Big Data Technology The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into
More informationBig Data The end of Data Warehousing?
Big Data The end of Data Warehousing? Hermann Bär Oracle USA Redwood Shores, CA Schlüsselworte Big data, data warehousing, advanced analytics, Hadoop, unstructured data Introduction If there was an Unwort
More information