Data Analysis Using MapReduce in Hadoop Environment
|
|
- Kevin Banks
- 5 years ago
- Views:
Transcription
1 Data Analysis Using MapReduce in Hadoop Environment Muhammad Khairul Rijal Muhammad*, Saiful Adli Ismail, Mohd Nazri Kama, Othman Mohd Yusop, Azri Azmi Advanced Informatics School (UTM AIS), Universiti Teknologi Malaysia (UTM), Jalan Sultan Yahya Petra, Kuala Lumpur, Malaysia. Abstract Data is growing at every moment which makes it almost impossible to process all of it. More than a decade ago, researchers from big company in IT industry such as Google and Yahoo introduce Hadoop and MapReduce as a solution for big volume of data processing. Now, by running on a free and open platform both method had gain popularity among the industry and academicians in the academic world. Thus, this paper will use MapReduce method in Hadoop environment and apply it on a sample dataset as a way to analyse the given data. The purpose of study is to gain understanding of using MapReduce in Hadoop environment. Keywords: Hadoop, Data Analysis, MapReduce, HDFS 1. INTRODUCTION Since the computer being made, data has become essential part in life. The growth of data is unstoppable and by 2020, it is said the amount of data will reach 44 zettabytes or equivalent to 44 trillion gigabytes. Data comes in three types: structured, unstructured and semi-structured and it comes from various of source such as documents, computers, sensor and many more. Companies such as Google, Yahoo, Amazon, ebay and so on deal with huge volume of data every day. The big problem is to handle such a huge volume of data for human benefit. Increasingly, data is seeing as an asset. Data that being collected normally has enormous potential and support decision making for people and organization. Traditional way of managing data collection is database management system or called DBMS. DBMS serves as an interface between databases and application or end users. In DBMS, most of the databases are already organized and structured. In real world, data that is still growing and coming from various sources are mostly unstructured and require big investment for IT infrastructure. * Corresponding author. address mkrijal@gmail.com 12
2 In 2002, Doug Cutting started a project call Nutch with a purpose to make a better web search engine. Subsequently, Google released a Google File System and MapReduce paper in Project Nutch evolved to be Hadoop and MapReduce becomes the programming model of Hadoop. Now, Hadoop is famous for being an open source software framework with scalable distributed computing that can provide fast and reliable analysis of structured and unstructured data [1]. MapReduce is a programming model that has been ingrained into Hadoop. It is considered to be the most popular computing paradigm for parallel, batch style and analysis of large amount of data [2]. In this paper, section 2 will explain in detail about the relationship between MapReduce and Hadoop. While section 3 will discuss the experiment and result of data analysis based on map reduce programming model using wordcount process. The last section will provide the conclusion of this paper. 2. RELATED WORK Hadoop, formerly called Apache Hadoop is an open source framework that is based on Java programming. It is a project sponsored by Apache Software Foundation [3]. Hadoop can handle thousand terabytes of data and allow applications to run on systems with thousands of commodities hardware node. One of the advantage of Hadoop is that they are not relying on hardware to deliver high availability but have a library that can detect and handle failures at the application layers which subsequently enable highly-available service to be delivered on top of cluster of computers, each of which may prone to failures [4]. The core component of Hadoop is HDFS and MapReduce [5]. HDFS or Hadoop Distributed File System is a storage and processing system designed for very large amount of distributed unstructured data. HDFS is a reliable place to store data and provide fast and scalable access to information [6]. Furthermore, HDFS is fault tolerance and a costefficient storage for big data. MapReduce is a framework for processing parallelizable problems across large datasets using many computers (nodes), collectively referred to as a cluster (if all nodes are on the same local network and use similar hardware) or a grid (if the nodes are shared across geographically and administratively distributed systems, and use more heterogenous hardware[7]. MapReduce can take advantage of locality of data, processing it on or near the storage assets in order to reduce the distance over which it must be transmitted [8]. According to Czajkowski et. al, MapReduce can handle multiple process simultaneously even at the scale of petabytes [9]. Both HDFS and MapReduce will be discussed in detail below. 13
3 3. HDFS An HDFS cluster is comprised of a NameNode, which manages the cluster metadata, and DataNodes that store the data. Files and directories are represented on the NameNode by inodes. Inodes record attributes like permissions, modification and access times, or namespace and disk space quotas. The file content is split into large blocks (typically 128 megabytes), and each block of the file is independently replicated at multiple DataNodes. The blocks are stored on the local file system on the DataNodes [10]. Below is the architecture of HDFS. Figure 1. HDFS Architecture [10] Both components of HDFS are software which run on Linux operating system and cheap commodity hardware [5]. Technically, HDFS can be implemented at low cost. By using Java as their base, HDFS is generally wide accepted among users because of the free and open platform. HDFS is fault tolerant. During the process, the Namenode actively monitors the number of replicas of a block. It does not directly send requests to DataNodes. The Namenode send instruction to DataNode which commands containing replicate blocks to other nodes, remove local block replicas, re-register and send an immediate block report, or shut down the node. 14
4 4. MAPREDUCE MapReduce gained its popularity when Google used it successfully [2]. MapReduce is a programming model and is preferred among industry and academic world because of the fault tolerance, simplicity and scalability. The primary objective of MapReduce is to split input data set into independent chunks that are processed in a completely parallel manner [16]. Initially, a distributed file system (DFS) partitions data in multiple machines and data is represented as (key, value) pairs. On a single master machine, map function is called to map the job which takes a set of data and converts it into another set of data and later the reduce function takes the output from a map as input and combines those data tuples into a smaller set of tuples. A pair of map and reduce functions may be executed once or numerous times as it depends on the characteristics of an application [17]. Figure 2 shows Hadoop MapReduce architecture. Figure 2. Hadoop MapReduce Architecture[18] Based on the above figure, there is two main processing stages. The first stage is Map phase and the second stage is Reduce phase. The actual MapReduce process happens in task tracker which is tagged as M1. In between map and reduce stages, there will be an intermediate process. During the intermediate process, the operations such as shuffle and sorting of the mapper output data will be executed. The intermediate data is going to get stored in local file system [18]. Below is the diagram that illustrate the scenario. 15
5 Figure 3. MapReduce Word count Process[19] The next section will discuss the experiment of wordcount process as MapReduce programming model using Java programming language and Hadoop environment on sample dataset. 5. EXPERIMENT The experiment required Java IDE Eclipse, Hortonworks Sandbox version 2.1, Jar file that containing library for Hadoop and dataset sample. Java IDE Eclipse is used for writing the source code for wordcount program using Java programming language. Hortonworks Sandbox version 2.1 acts as the Hadoop environment for the experiment. It can be downloaded from Sandbox download site. The sandbox requires computer with at least 8 GB of RAM and can be run in virtual environment such as Virtual Box[20]. A Java library for Hadoop need to be downloaded to make source code functioning. Lastly, a sample dataset containing estimation for Malaysia monthly average temperature from 2041 until To summarize it, a simple application of MapReduce called wordcount will be developed in Java IDE Eclipse and tested in Hortonworks Sandbox to illustrate the application of MapReduce in Hadoop. The dataset contains data in numeric value with 12 column representing months and 10 rows representing years. The data is saved in text file format and taken from National Hydraulic Research Institute of Malaysia (NAHRIM) website. The purpose of using this dataset is to analyse the estimation of monthly average temperature in Malaysia from 2041 to Table below shown the contain of dataset. Figure 4. Malaysian Montly Average Temperature from 2041 to
6 6. MAP AND REDUCE Firstly, a new Java project need to be created in Java IDE Eclipse. The project must contain a package and at least one class to run the coding. Inside the class, the first step is to import the library to execute MapReduce. There is two set of import as shown in the list below. 1) Java Class: Import Java function such as IOException and StringTokenizer that can be apply on input output data and string input. Figure 5. Import Java Classes 2) Hadoop class: Import Hadoop library that contains function that enable features for MapReduce to be apply. Figure 6. Import Hadoop Classes The coding section will be divided into three parts: Mapper, Reduce and Configuration. Set of codes are as below. 1) Mapper: A function that split each line of data in dataset and then map the data inside the line separately. Figure 7. Mapper function Reduce: A function that takes the result of Mapper. Firstly, by shuffling the map data and then count the occurrence to reduce it. 17
7 Figure 8. Reducer function Configuration: A function that call all classes that have been imported to run. Figure 9. Configuration function 7. HORTONWORKS SANDBOX For this experiment, Hortonworks Sandbox version 2.1 will be used as Hadoop environment. The Sandbox version 2.1 uses Linux Centos as their operating system. Figure 10 is the list of component inside Hortonworks Sandbox version 2.1: Figure10. Hortonworks Sandbox 2.1 Component The first step is to create a folder for input file. Type in the url browser and then login as hue. Go to File Browser and create a new folder. Then, upload the text file containing the sample data as shown in the figure below. 18
8 Figure 11. File uploaded in the File Browser The Java project that has been created need to be saved in Java Archive format (JAR). As the experiment would be run in Hadoop environment, the JAR file need to be uploaded into Hortonworks Sandbox version 2.1. The Java runtime saved for the project must be similar to the one inside Sandbox to avoid compatibility issue. To execute the JAR file, use the terminal and type command as shown below. Figure 12 Command to execute MapReduce JAR file The successful execution will appear as in figure 13. Figure 13 MapReduce process 8. RESULT The output of the MapReduce program can be seen inside the output folder that has been created. Open back File Browser in Hue and search the folder called MR_Output. Figure 14 shows the output result. 19
9 Figure 14 Output of MapReduce program Based on the result, all the data have been sort in pairs from the lowest to the highest. At the top of the list is the lowest recorded monthly average temperature which happen only once while at the bottom is the highest recorded monthly average temperature which also happen once. The estimation shows the monthly average temperature will be recorded between 26.9 c and 31.5 c and the most likely to occur is 30.9 c. 9. CONCLUSION Based on this research it can be concluded that MapReduce simplified the concept of filtering and analyzing data. Traditional method such as DBMS use the query language which performance will suffer when there is a lot of querying need to be done at the same time. The wordcount program did illustrate the concept of MapReduce though it has been over simplified if compare to real world. Nevertheless, the introduction of MapReduce is crucial as it innovates the data processing techniques especially when there is a need for fast response albeit in the large volume of data. 20
10 MapReduce run well in Hadoop environment as they are built in the same Java language. Hadoop can speed-up the processing of large amount of data through distributed process and thus enable the fast response. The advantage of fault-tolerance, free and open framework while also offering big data management will tempt industry and academicians in using it for the purpose of advancement. ACKNOWLEDGEMENT The authors would like to thanks Advanced Informatics School (AIS), Universiti Teknologi Malaysia Kuala Lumpur for the support in R&D. REFERENCES [1] What is Apache Hadoop? Hortonworks, 17-May [Online]. Available: [Accessed: 02-Jun-2017]. [2] S. Maitrey and C. K. Jha, MapReduce: Simplified Data Analysis of Big Data, Procedia Comput. Sci., vol. 57, pp , Jan [3] What is Hadoop? - Definition from WhatIs.com, SearchCloudComputing. [Online]. Available: [Accessed: 02-Jun-2017]. [4] Welcome to Apache TM Hadoop! [Online]. Available: [Accessed: 02-Jun-2017]. [5] K. Dwivedi and S. K. Dubey, Analytical review on Hadoop Distributed file system, in th International Conference - Confluence The Next Generation Information Technology Summit (Confluence), 2014, pp [6] F. Chang et al., Bigtable: A Distributed Storage System for Structured Data, ACM Trans Comput Syst, vol. 26, no. 2, p. 4:1 4:26, Jun [7] V. K. Jain, Big Data and Hadoop. Khanna Publishing, [8] Define: MapReduce and Text Parsing. [Online]. Available: [Accessed: 02-Jun-2017]. [9] Sorting Petabytes with MapReduce - The Next Episode, Research Blog.. [10] Apache Hadoop HDFS, Hortonworks, 21-Mar [Online]. Available: [Accessed: 02-Jun-2017]. [11] What is Hadoop MapReduce? Webopedia Definition. [Online]. Available: [Accessed: 02-Jun- 2017]. [12] Y. Kim and K. Shim, Parallel Top-K Similarity Join Algorithms Using MapReduce, in 2012 IEEE 28th International Conference on Data Engineering, 2012, pp [13] S. User, Hadoop Map Reduce Architecture and Example. [Online]. Available: [Accessed: 02-Jun-2017]. [14] map_reduce_tutorial.pdf.. [15] Sandbox Deployment and Install Guide, Hortonworks, 24-Apr [Online]. Available: [Accessed: 02-Jun-2017]. [16] CodHoop: A system for optimizing big data processing (PDF Download Available), ResearchGate. [Online]. Available: stem_for_optimizing_big_data_processing. [Accessed: 21-Jun-2017]. 21
11 [17] J. Dean and S. Ghemawat, MapReduce: Simplified Data Processing on Large Clusters, Commun ACM, vol. 51, no. 1, pp , Jan [18] S. User, Hadoop Map Reduce Architecture and Example. [Online]. Available: [Accessed: 02-Jun-2017]. [19] map_reduce_tutorial.pdf.. [20] Sandbox Deployment and Install Guide, Hortonworks, 24-Apr [Online]. Available: [Accessed: 02-Jun-2017]. 22
A brief history on Hadoop
Hadoop Basics A brief history on Hadoop 2003 - Google launches project Nutch to handle billions of searches and indexing millions of web pages. Oct 2003 - Google releases papers with GFS (Google File System)
More informationData Clustering on the Parallel Hadoop MapReduce Model. Dimitrios Verraros
Data Clustering on the Parallel Hadoop MapReduce Model Dimitrios Verraros Overview The purpose of this thesis is to implement and benchmark the performance of a parallel K- means clustering algorithm on
More informationIntroduction to Hadoop. Owen O Malley Yahoo!, Grid Team
Introduction to Hadoop Owen O Malley Yahoo!, Grid Team owen@yahoo-inc.com Who Am I? Yahoo! Architect on Hadoop Map/Reduce Design, review, and implement features in Hadoop Working on Hadoop full time since
More informationTopics. Big Data Analytics What is and Why Hadoop? Comparison to other technologies Hadoop architecture Hadoop ecosystem Hadoop usage examples
Hadoop Introduction 1 Topics Big Data Analytics What is and Why Hadoop? Comparison to other technologies Hadoop architecture Hadoop ecosystem Hadoop usage examples 2 Big Data Analytics What is Big Data?
More informationHADOOP FRAMEWORK FOR BIG DATA
HADOOP FRAMEWORK FOR BIG DATA Mr K. Srinivas Babu 1,Dr K. Rameshwaraiah 2 1 Research Scholar S V University, Tirupathi 2 Professor and Head NNRESGI, Hyderabad Abstract - Data has to be stored for further
More informationHadoop/MapReduce Computing Paradigm
Hadoop/Reduce Computing Paradigm 1 Large-Scale Data Analytics Reduce computing paradigm (E.g., Hadoop) vs. Traditional database systems vs. Database Many enterprises are turning to Hadoop Especially applications
More informationGlobal Journal of Engineering Science and Research Management
A FUNDAMENTAL CONCEPT OF MAPREDUCE WITH MASSIVE FILES DATASET IN BIG DATA USING HADOOP PSEUDO-DISTRIBUTION MODE K. Srikanth*, P. Venkateswarlu, Ashok Suragala * Department of Information Technology, JNTUK-UCEV
More informationIntroduction to MapReduce
Basics of Cloud Computing Lecture 4 Introduction to MapReduce Satish Srirama Some material adapted from slides by Jimmy Lin, Christophe Bisciglia, Aaron Kimball, & Sierra Michels-Slettvet, Google Distributed
More informationBig Data Analytics. Izabela Moise, Evangelos Pournaras, Dirk Helbing
Big Data Analytics Izabela Moise, Evangelos Pournaras, Dirk Helbing Izabela Moise, Evangelos Pournaras, Dirk Helbing 1 Big Data "The world is crazy. But at least it s getting regular analysis." Izabela
More informationHadoop File System S L I D E S M O D I F I E D F R O M P R E S E N T A T I O N B Y B. R A M A M U R T H Y 11/15/2017
Hadoop File System 1 S L I D E S M O D I F I E D F R O M P R E S E N T A T I O N B Y B. R A M A M U R T H Y Moving Computation is Cheaper than Moving Data Motivation: Big Data! What is BigData? - Google
More informationIntroduction to Hadoop and MapReduce
Introduction to Hadoop and MapReduce Antonino Virgillito THE CONTRACTOR IS ACTING UNDER A FRAMEWORK CONTRACT CONCLUDED WITH THE COMMISSION Large-scale Computation Traditional solutions for computing large
More informationMap Reduce & Hadoop Recommended Text:
Map Reduce & Hadoop Recommended Text: Hadoop: The Definitive Guide Tom White O Reilly 2010 VMware Inc. All rights reserved Big Data! Large datasets are becoming more common The New York Stock Exchange
More informationPLATFORM AND SOFTWARE AS A SERVICE THE MAPREDUCE PROGRAMMING MODEL AND IMPLEMENTATIONS
PLATFORM AND SOFTWARE AS A SERVICE THE MAPREDUCE PROGRAMMING MODEL AND IMPLEMENTATIONS By HAI JIN, SHADI IBRAHIM, LI QI, HAIJUN CAO, SONG WU and XUANHUA SHI Prepared by: Dr. Faramarz Safi Islamic Azad
More informationKeywords Hadoop, Map Reduce, K-Means, Data Analysis, Storage, Clusters.
Volume 6, Issue 3, March 2016 ISSN: 2277 128X International Journal of Advanced Research in Computer Science and Software Engineering Research Paper Available online at: www.ijarcsse.com Special Issue
More informationInternational Journal of Advance Engineering and Research Development. A Study: Hadoop Framework
Scientific Journal of Impact Factor (SJIF): e-issn (O): 2348- International Journal of Advance Engineering and Research Development Volume 3, Issue 2, February -2016 A Study: Hadoop Framework Devateja
More informationA Review Approach for Big Data and Hadoop Technology
International Journal of Modern Trends in Engineering and Research www.ijmter.com e-issn No.:2349-9745, Date: 2-4 July, 2015 A Review Approach for Big Data and Hadoop Technology Prof. Ghanshyam Dhomse
More informationTITLE: PRE-REQUISITE THEORY. 1. Introduction to Hadoop. 2. Cluster. Implement sort algorithm and run it using HADOOP
TITLE: Implement sort algorithm and run it using HADOOP PRE-REQUISITE Preliminary knowledge of clusters and overview of Hadoop and its basic functionality. THEORY 1. Introduction to Hadoop The Apache Hadoop
More informationHigh Performance Computing on MapReduce Programming Framework
International Journal of Private Cloud Computing Environment and Management Vol. 2, No. 1, (2015), pp. 27-32 http://dx.doi.org/10.21742/ijpccem.2015.2.1.04 High Performance Computing on MapReduce Programming
More informationThe MapReduce Framework
The MapReduce Framework In Partial fulfilment of the requirements for course CMPT 816 Presented by: Ahmed Abdel Moamen Agents Lab Overview MapReduce was firstly introduced by Google on 2004. MapReduce
More informationMI-PDB, MIE-PDB: Advanced Database Systems
MI-PDB, MIE-PDB: Advanced Database Systems http://www.ksi.mff.cuni.cz/~svoboda/courses/2015-2-mie-pdb/ Lecture 10: MapReduce, Hadoop 26. 4. 2016 Lecturer: Martin Svoboda svoboda@ksi.mff.cuni.cz Author:
More informationWhat is the maximum file size you have dealt so far? Movies/Files/Streaming video that you have used? What have you observed?
Simple to start What is the maximum file size you have dealt so far? Movies/Files/Streaming video that you have used? What have you observed? What is the maximum download speed you get? Simple computation
More informationIntroduction to Map Reduce
Introduction to Map Reduce 1 Map Reduce: Motivation We realized that most of our computations involved applying a map operation to each logical record in our input in order to compute a set of intermediate
More informationDepartment of Computer Science San Marcos, TX Report Number TXSTATE-CS-TR Clustering in the Cloud. Xuan Wang
Department of Computer Science San Marcos, TX 78666 Report Number TXSTATE-CS-TR-2010-24 Clustering in the Cloud Xuan Wang 2010-05-05 !"#$%&'()*+()+%,&+!"-#. + /+!"#$%&'()*+0"*-'(%,1$+0.23%(-)+%-+42.--3+52367&.#8&+9'21&:-';
More informationThe amount of data increases every day Some numbers ( 2012):
1 The amount of data increases every day Some numbers ( 2012): Data processed by Google every day: 100+ PB Data processed by Facebook every day: 10+ PB To analyze them, systems that scale with respect
More information2/26/2017. The amount of data increases every day Some numbers ( 2012):
The amount of data increases every day Some numbers ( 2012): Data processed by Google every day: 100+ PB Data processed by Facebook every day: 10+ PB To analyze them, systems that scale with respect to
More informationEmbedded Technosolutions
Hadoop Big Data An Important technology in IT Sector Hadoop - Big Data Oerie 90% of the worlds data was generated in the last few years. Due to the advent of new technologies, devices, and communication
More information50 Must Read Hadoop Interview Questions & Answers
50 Must Read Hadoop Interview Questions & Answers Whizlabs Dec 29th, 2017 Big Data Are you planning to land a job with big data and data analytics? Are you worried about cracking the Hadoop job interview?
More informationIntroduction to MapReduce. Instructor: Dr. Weikuan Yu Computer Sci. & Software Eng.
Introduction to MapReduce Instructor: Dr. Weikuan Yu Computer Sci. & Software Eng. Before MapReduce Large scale data processing was difficult! Managing hundreds or thousands of processors Managing parallelization
More informationBigData and Map Reduce VITMAC03
BigData and Map Reduce VITMAC03 1 Motivation Process lots of data Google processed about 24 petabytes of data per day in 2009. A single machine cannot serve all the data You need a distributed system to
More informationHadoop An Overview. - Socrates CCDH
Hadoop An Overview - Socrates CCDH What is Big Data? Volume Not Gigabyte. Terabyte, Petabyte, Exabyte, Zettabyte - Due to handheld gadgets,and HD format images and videos - In total data, 90% of them collected
More informationMapReduce. U of Toronto, 2014
MapReduce U of Toronto, 2014 http://www.google.org/flutrends/ca/ (2012) Average Searches Per Day: 5,134,000,000 2 Motivation Process lots of data Google processed about 24 petabytes of data per day in
More informationParallel HITS Algorithm Implemented Using HADOOP GIRAPH Framework to resolve Big Data Problem
I J C T A, 9(41) 2016, pp. 1235-1239 International Science Press Parallel HITS Algorithm Implemented Using HADOOP GIRAPH Framework to resolve Big Data Problem Hema Dubey *, Nilay Khare *, Alind Khare **
More informationTP1-2: Analyzing Hadoop Logs
TP1-2: Analyzing Hadoop Logs Shadi Ibrahim January 26th, 2017 MapReduce has emerged as a leading programming model for data-intensive computing. It was originally proposed by Google to simplify development
More informationChapter 5. The MapReduce Programming Model and Implementation
Chapter 5. The MapReduce Programming Model and Implementation - Traditional computing: data-to-computing (send data to computing) * Data stored in separate repository * Data brought into system for computing
More informationThe Analysis Research of Hierarchical Storage System Based on Hadoop Framework Yan LIU 1, a, Tianjian ZHENG 1, Mingjiang LI 1, Jinpeng YUAN 1
International Conference on Intelligent Systems Research and Mechatronics Engineering (ISRME 2015) The Analysis Research of Hierarchical Storage System Based on Hadoop Framework Yan LIU 1, a, Tianjian
More informationEnhanced Hadoop with Search and MapReduce Concurrency Optimization
Volume 114 No. 12 2017, 323-331 ISSN: 1311-8080 (printed version); ISSN: 1314-3395 (on-line version) url: http://www.ijpam.eu ijpam.eu Enhanced Hadoop with Search and MapReduce Concurrency Optimization
More informationBig Data Programming: an Introduction. Spring 2015, X. Zhang Fordham Univ.
Big Data Programming: an Introduction Spring 2015, X. Zhang Fordham Univ. Outline What the course is about? scope Introduction to big data programming Opportunity and challenge of big data Origin of Hadoop
More informationIntroduction to MapReduce
Basics of Cloud Computing Lecture 4 Introduction to MapReduce Satish Srirama Some material adapted from slides by Jimmy Lin, Christophe Bisciglia, Aaron Kimball, & Sierra Michels-Slettvet, Google Distributed
More informationClustering Lecture 8: MapReduce
Clustering Lecture 8: MapReduce Jing Gao SUNY Buffalo 1 Divide and Conquer Work Partition w 1 w 2 w 3 worker worker worker r 1 r 2 r 3 Result Combine 4 Distributed Grep Very big data Split data Split data
More informationA SURVEY ON SCHEDULING IN HADOOP FOR BIGDATA PROCESSING
Journal homepage: www.mjret.in ISSN:2348-6953 A SURVEY ON SCHEDULING IN HADOOP FOR BIGDATA PROCESSING Bhavsar Nikhil, Bhavsar Riddhikesh,Patil Balu,Tad Mukesh Department of Computer Engineering JSPM s
More informationCloud Computing and Hadoop Distributed File System. UCSB CS170, Spring 2018
Cloud Computing and Hadoop Distributed File System UCSB CS70, Spring 08 Cluster Computing Motivations Large-scale data processing on clusters Scan 000 TB on node @ 00 MB/s = days Scan on 000-node cluster
More informationA Survey on Big Data
A Survey on Big Data D.Prudhvi 1, D.Jaswitha 2, B. Mounika 3, Monika Bagal 4 1 2 3 4 B.Tech Final Year, CSE, Dadi Institute of Engineering & Technology,Andhra Pradesh,INDIA ---------------------------------------------------------------------***---------------------------------------------------------------------
More informationHuge Data Analysis and Processing Platform based on Hadoop Yuanbin LI1, a, Rong CHEN2
2nd International Conference on Materials Science, Machinery and Energy Engineering (MSMEE 2017) Huge Data Analysis and Processing Platform based on Hadoop Yuanbin LI1, a, Rong CHEN2 1 Information Engineering
More informationScalable Web Programming. CS193S - Jan Jannink - 2/25/10
Scalable Web Programming CS193S - Jan Jannink - 2/25/10 Weekly Syllabus 1.Scalability: (Jan.) 2.Agile Practices 3.Ecology/Mashups 4.Browser/Client 7.Analytics 8.Cloud/Map-Reduce 9.Published APIs: (Mar.)*
More informationWhere We Are. Review: Parallel DBMS. Parallel DBMS. Introduction to Data Management CSE 344
Where We Are Introduction to Data Management CSE 344 Lecture 22: MapReduce We are talking about parallel query processing There exist two main types of engines: Parallel DBMSs (last lecture + quick review)
More informationHDFS: Hadoop Distributed File System. CIS 612 Sunnie Chung
HDFS: Hadoop Distributed File System CIS 612 Sunnie Chung What is Big Data?? Bulk Amount Unstructured Introduction Lots of Applications which need to handle huge amount of data (in terms of 500+ TB per
More informationCloud Computing 2. CSCI 4850/5850 High-Performance Computing Spring 2018
Cloud Computing 2 CSCI 4850/5850 High-Performance Computing Spring 2018 Tae-Hyuk (Ted) Ahn Department of Computer Science Program of Bioinformatics and Computational Biology Saint Louis University Learning
More informationParallel Programming Principle and Practice. Lecture 10 Big Data Processing with MapReduce
Parallel Programming Principle and Practice Lecture 10 Big Data Processing with MapReduce Outline MapReduce Programming Model MapReduce Examples Hadoop 2 Incredible Things That Happen Every Minute On The
More informationCS370 Operating Systems
CS370 Operating Systems Colorado State University Yashwant K Malaiya Fall 2017 Lecture 26 File Systems Slides based on Text by Silberschatz, Galvin, Gagne Various sources 1 1 FAQ Cylinders: all the platters?
More informationLecture 11 Hadoop & Spark
Lecture 11 Hadoop & Spark Dr. Wilson Rivera ICOM 6025: High Performance Computing Electrical and Computer Engineering Department University of Puerto Rico Outline Distributed File Systems Hadoop Ecosystem
More informationΕΠΛ 602:Foundations of Internet Technologies. Cloud Computing
ΕΠΛ 602:Foundations of Internet Technologies Cloud Computing 1 Outline Bigtable(data component of cloud) Web search basedonch13of thewebdatabook 2 What is Cloud Computing? ACloudis an infrastructure, transparent
More informationMapReduce Simplified Data Processing on Large Clusters
MapReduce Simplified Data Processing on Large Clusters Amir H. Payberah amir@sics.se Amirkabir University of Technology (Tehran Polytechnic) Amir H. Payberah (Tehran Polytechnic) MapReduce 1393/8/5 1 /
More informationDistributed Systems CS6421
Distributed Systems CS6421 Intro to Distributed Systems and the Cloud Prof. Tim Wood v I teach: Software Engineering, Operating Systems, Sr. Design I like: distributed systems, networks, building cool
More informationDistributed Systems. 05r. Case study: Google Cluster Architecture. Paul Krzyzanowski. Rutgers University. Fall 2016
Distributed Systems 05r. Case study: Google Cluster Architecture Paul Krzyzanowski Rutgers University Fall 2016 1 A note about relevancy This describes the Google search cluster architecture in the mid
More informationCS427 Multicore Architecture and Parallel Computing
CS427 Multicore Architecture and Parallel Computing Lecture 9 MapReduce Prof. Li Jiang 2014/11/19 1 What is MapReduce Origin from Google, [OSDI 04] A simple programming model Functional model For large-scale
More informationCS 345A Data Mining. MapReduce
CS 345A Data Mining MapReduce Single-node architecture CPU Machine Learning, Statistics Memory Classical Data Mining Disk Commodity Clusters Web data sets can be very large Tens to hundreds of terabytes
More informationMap Reduce. Yerevan.
Map Reduce Erasmus+ @ Yerevan dacosta@irit.fr Divide and conquer at PaaS 100 % // Typical problem Iterate over a large number of records Extract something of interest from each Shuffle and sort intermediate
More informationBatch Inherence of Map Reduce Framework
Available Online at www.ijcsmc.com International Journal of Computer Science and Mobile Computing A Monthly Journal of Computer Science and Information Technology IJCSMC, Vol. 4, Issue. 6, June 2015, pg.287
More informationCLIENT DATA NODE NAME NODE
Volume 6, Issue 12, December 2016 ISSN: 2277 128X International Journal of Advanced Research in Computer Science and Software Engineering Research Paper Available online at: www.ijarcsse.com Efficiency
More informationEXTRACT DATA IN LARGE DATABASE WITH HADOOP
International Journal of Advances in Engineering & Scientific Research (IJAESR) ISSN: 2349 3607 (Online), ISSN: 2349 4824 (Print) Download Full paper from : http://www.arseam.com/content/volume-1-issue-7-nov-2014-0
More informationCloud Computing. Hwajung Lee. Key Reference: Prof. Jong-Moon Chung s Lecture Notes at Yonsei University
Cloud Computing Hwajung Lee Key Reference: Prof. Jong-Moon Chung s Lecture Notes at Yonsei University Cloud Computing Cloud Introduction Cloud Service Model Big Data Hadoop MapReduce HDFS (Hadoop Distributed
More informationDistributed File Systems II
Distributed File Systems II To do q Very-large scale: Google FS, Hadoop FS, BigTable q Next time: Naming things GFS A radically new environment NFS, etc. Independence Small Scale Variety of workloads Cooperation
More informationSystems Infrastructure for Data Science. Web Science Group Uni Freiburg WS 2013/14
Systems Infrastructure for Data Science Web Science Group Uni Freiburg WS 2013/14 MapReduce & Hadoop The new world of Big Data (programming model) Overview of this Lecture Module Background Cluster File
More informationIntroduction to HDFS and MapReduce
Introduction to HDFS and MapReduce Who Am I - Ryan Tabora - Data Developer at Think Big Analytics - Big Data Consulting - Experience working with Hadoop, HBase, Hive, Solr, Cassandra, etc. 2 Who Am I -
More informationBig Data Analytics by Using Hadoop
Governors State University OPUS Open Portal to University Scholarship All Capstone Projects Student Capstone Projects Spring 2015 Big Data Analytics by Using Hadoop Chaitanya Arava Governors State University
More informationMap Reduce Group Meeting
Map Reduce Group Meeting Yasmine Badr 10/07/2014 A lot of material in this presenta0on has been adopted from the original MapReduce paper in OSDI 2004 What is Map Reduce? Programming paradigm/model for
More informationCS 61C: Great Ideas in Computer Architecture. MapReduce
CS 61C: Great Ideas in Computer Architecture MapReduce Guest Lecturer: Justin Hsia 3/06/2013 Spring 2013 Lecture #18 1 Review of Last Lecture Performance latency and throughput Warehouse Scale Computing
More informationProcessing Large / Big Data through MapR and Pig
Processing Large / Big Data through MapR and Pig Arvind Kumar-Senior ERP Solution Architect / Manager Suhas Pande- Solution Architect (IT and Security) Abstract - We live in the data age. It s not easy
More informationCPSC 426/526. Cloud Computing. Ennan Zhai. Computer Science Department Yale University
CPSC 426/526 Cloud Computing Ennan Zhai Computer Science Department Yale University Recall: Lec-7 In the lec-7, I talked about: - P2P vs Enterprise control - Firewall - NATs - Software defined network
More informationHadoop and Map-reduce computing
Hadoop and Map-reduce computing 1 Introduction This activity contains a great deal of background information and detailed instructions so that you can refer to it later for further activities and homework.
More informationImproving the MapReduce Big Data Processing Framework
Improving the MapReduce Big Data Processing Framework Gistau, Reza Akbarinia, Patrick Valduriez INRIA & LIRMM, Montpellier, France In collaboration with Divyakant Agrawal, UCSB Esther Pacitti, UM2, LIRMM
More informationBig Data with Hadoop Ecosystem
Diógenes Pires Big Data with Hadoop Ecosystem Hands-on (HBase, MySql and Hive + Power BI) Internet Live http://www.internetlivestats.com/ Introduction Business Intelligence Business Intelligence Process
More informationSurvey Paper on Traditional Hadoop and Pipelined Map Reduce
International Journal of Computational Engineering Research Vol, 03 Issue, 12 Survey Paper on Traditional Hadoop and Pipelined Map Reduce Dhole Poonam B 1, Gunjal Baisa L 2 1 M.E.ComputerAVCOE, Sangamner,
More informationIndexing Strategies of MapReduce for Information Retrieval in Big Data
International Journal of Advances in Computer Science and Technology (IJACST), Vol.5, No.3, Pages : 01-06 (2016) Indexing Strategies of MapReduce for Information Retrieval in Big Data Mazen Farid, Rohaya
More informationData Informatics. Seon Ho Kim, Ph.D.
Data Informatics Seon Ho Kim, Ph.D. seonkim@usc.edu HBase HBase is.. A distributed data store that can scale horizontally to 1,000s of commodity servers and petabytes of indexed storage. Designed to operate
More informationHadoop Map Reduce 10/17/2018 1
Hadoop Map Reduce 10/17/2018 1 MapReduce 2-in-1 A programming paradigm A query execution engine A kind of functional programming We focus on the MapReduce execution engine of Hadoop through YARN 10/17/2018
More informationEvaluation of Apache Hadoop for parallel data analysis with ROOT
Evaluation of Apache Hadoop for parallel data analysis with ROOT S Lehrack, G Duckeck, J Ebke Ludwigs-Maximilians-University Munich, Chair of elementary particle physics, Am Coulombwall 1, D-85748 Garching,
More informationA Study of Comparatively Analysis for HDFS and Google File System towards to Handle Big Data
A Study of Comparatively Analysis for HDFS and Google File System towards to Handle Big Data Rajesh R Savaliya 1, Dr. Akash Saxena 2 1Research Scholor, Rai University, Vill. Saroda, Tal. Dholka Dist. Ahmedabad,
More informationMitigating Data Skew Using Map Reduce Application
Ms. Archana P.M Mitigating Data Skew Using Map Reduce Application Mr. Malathesh S.H 4 th sem, M.Tech (C.S.E) Associate Professor C.S.E Dept. M.S.E.C, V.T.U Bangalore, India archanaanil062@gmail.com M.S.E.C,
More informationImproved MapReduce k-means Clustering Algorithm with Combiner
2014 UKSim-AMSS 16th International Conference on Computer Modelling and Simulation Improved MapReduce k-means Clustering Algorithm with Combiner Prajesh P Anchalia Department Of Computer Science and Engineering
More informationBigData and MapReduce with Hadoop
BigData and MapReduce with Hadoop Ivan Tomašić 1, Roman Trobec 1, Aleksandra Rashkovska 1, Matjaž Depolli 1, Peter Mežnar 2, Andrej Lipej 2 1 Jožef Stefan Institute, Jamova 39, 1000 Ljubljana 2 TURBOINŠTITUT
More informationBig Data and Object Storage
Big Data and Object Storage or where to store the cold and small data? Sven Bauernfeind Computacenter AG & Co. ohg, Consultancy Germany 28.02.2018 Munich Volume, Variety & Velocity + Analytics Velocity
More informationDatabases 2 (VU) ( / )
Databases 2 (VU) (706.711 / 707.030) MapReduce (Part 3) Mark Kröll ISDS, TU Graz Nov. 27, 2017 Mark Kröll (ISDS, TU Graz) MapReduce Nov. 27, 2017 1 / 42 Outline 1 Problems Suited for Map-Reduce 2 MapReduce:
More informationMap-Reduce. Marco Mura 2010 March, 31th
Map-Reduce Marco Mura (mura@di.unipi.it) 2010 March, 31th This paper is a note from the 2009-2010 course Strumenti di programmazione per sistemi paralleli e distribuiti and it s based by the lessons of
More informationScaling Up 1 CSE 6242 / CX Duen Horng (Polo) Chau Georgia Tech. Hadoop, Pig
CSE 6242 / CX 4242 Scaling Up 1 Hadoop, Pig Duen Horng (Polo) Chau Georgia Tech Some lectures are partly based on materials by Professors Guy Lebanon, Jeffrey Heer, John Stasko, Christos Faloutsos, Le
More informationCS / Cloud Computing. Recitation 3 September 9 th & 11 th, 2014
CS15-319 / 15-619 Cloud Computing Recitation 3 September 9 th & 11 th, 2014 Overview Last Week s Reflection --Project 1.1, Quiz 1, Unit 1 This Week s Schedule --Unit2 (module 3 & 4), Project 1.2 Questions
More informationData Platforms and Pattern Mining
Morteza Zihayat Data Platforms and Pattern Mining IBM Corporation About Myself IBM Software Group Big Data Scientist 4Platform Computing, IBM (2014 Now) PhD Candidate (2011 Now) 4Lassonde School of Engineering,
More informationOverview. Why MapReduce? What is MapReduce? The Hadoop Distributed File System Cloudera, Inc.
MapReduce and HDFS This presentation includes course content University of Washington Redistributed under the Creative Commons Attribution 3.0 license. All other contents: Overview Why MapReduce? What
More informationA BigData Tour HDFS, Ceph and MapReduce
A BigData Tour HDFS, Ceph and MapReduce These slides are possible thanks to these sources Jonathan Drusi - SCInet Toronto Hadoop Tutorial, Amir Payberah - Course in Data Intensive Computing SICS; Yahoo!
More informationImplementation of Aggregation of Map and Reduce Function for Performance Improvisation
2016 IJSRSET Volume 2 Issue 5 Print ISSN: 2395-1990 Online ISSN : 2394-4099 Themed Section: Engineering and Technology Implementation of Aggregation of Map and Reduce Function for Performance Improvisation
More informationIntroduction to MapReduce (cont.)
Introduction to MapReduce (cont.) Rafael Ferreira da Silva rafsilva@isi.edu http://rafaelsilva.com USC INF 553 Foundations and Applications of Data Mining (Fall 2018) 2 MapReduce: Summary USC INF 553 Foundations
More informationAn Improved Performance Evaluation on Large-Scale Data using MapReduce Technique
Available Online at www.ijcsmc.com International Journal of Computer Science and Mobile Computing A Monthly Journal of Computer Science and Information Technology ISSN 2320 088X IMPACT FACTOR: 6.017 IJCSMC,
More informationBESIII Physical Analysis on Hadoop Platform
BESIII Physical Analysis on Hadoop Platform Jing HUO 12, Dongsong ZANG 12, Xiaofeng LEI 12, Qiang LI 12, Gongxing SUN 1 1 Institute of High Energy Physics, Beijing, China 2 University of Chinese Academy
More informationDistributed Filesystem
Distributed Filesystem 1 How do we get data to the workers? NAS Compute Nodes SAN 2 Distributing Code! Don t move data to workers move workers to the data! - Store data on the local disks of nodes in the
More informationSystems Infrastructure for Data Science. Web Science Group Uni Freiburg WS 2012/13
Systems Infrastructure for Data Science Web Science Group Uni Freiburg WS 2012/13 MapReduce & Hadoop The new world of Big Data (programming model) Overview of this Lecture Module Background Google MapReduce
More informationUNIT-IV HDFS. Ms. Selva Mary. G
UNIT-IV HDFS HDFS ARCHITECTURE Dataset partition across a number of separate machines Hadoop Distributed File system The Design of HDFS HDFS is a file system designed for storing very large files with
More informationDistributed Face Recognition Using Hadoop
Distributed Face Recognition Using Hadoop A. Thorat, V. Malhotra, S. Narvekar and A. Joshi Dept. of Computer Engineering and IT College of Engineering, Pune {abhishekthorat02@gmail.com, vinayak.malhotra20@gmail.com,
More informationAn Indian Journal FULL PAPER ABSTRACT KEYWORDS. Trade Science Inc. The study on magnanimous data-storage system based on cloud computing
[Type text] [Type text] [Type text] ISSN : 0974-7435 Volume 10 Issue 11 BioTechnology 2014 An Indian Journal FULL PAPER BTAIJ, 10(11), 2014 [5368-5376] The study on magnanimous data-storage system based
More informationPlumbing the Web. Narayanan Shivakumar. Google Distinguished Entrepreneur & Director
Plumbing the Web Narayanan Shivakumar Google Distinguished Entrepreneur & Director Google Developer Day 2007 powered by 3 Copyright 2007, Google Inc Developers and Google AJAX Search API Google Code Project
More informationMapReduce, Hadoop and Spark. Bompotas Agorakis
MapReduce, Hadoop and Spark Bompotas Agorakis Big Data Processing Most of the computations are conceptually straightforward on a single machine but the volume of data is HUGE Need to use many (1.000s)
More information