MapReduce Algorithm Design
|
|
- Justin Conley
- 5 years ago
- Views:
Transcription
1 MapReduce Algorithm Design
2 Contents Combiner and in mapper combining Complex keys and values Secondary Sorting
3 Combiner and in mapper combining Purpose Carry out local aggregation before shuffle and sort phase Reduce the communication volume between map and reduce stages
4 Combiner example (word count) Use reducer as combiner Integer addition is both associative and commutative
5 MapReduce with Combiner
6 MapReduce with combiner MapReduce with combiner map: (k1, v1) > [(k2, v2)] combine: (k2, [v2]) > [(k2, v2)] reduce: (k2, [v2]) > [(k3, v3)] The combiner input and output key value types must match the mapper output key value type
7 Combiner is an optimization, not a requirement Combiner is optional A particular implementation of MapReduce framework may choose to execute the combine method many times or none Calling the combine method zero, one, or many times should produce the same output from the reducer The correctness of the MapReduce program should not rely on the assumption that the combiner is always carried out
8 In method combining
9 Use Java Map to implement AssociativeArray A Map is an object that maps keys to values A map cannot contain duplicate keys Each key can map to at most one value The Java platform contains three general purpose Map implementations HashMap, TreeMap, and LinkedHashMap Basic operations boolean containskey(object key) get(object key) put(k key, V value) keyset() Returns a Set view of the keys contained in this map
10 In mapper combining
11 Demo and Some online resource Demo on in method and in mapper combining Online resource in mapper combiningdesign pattern for mapreduce programming codingjunkie.net Blog by Bill Bejeck, who provides a lot of source code on MapReduce in Hadoop
12 Complex keys and values Both keys and values can be complex data structures Pairs Stripes Serialization and deserialization for complex structures After map stage, structures need to be serialized to be written to storage Complex keys and values need to be de serialized at the reducer side
13 Motivation An example to compute the mean of value
14 With combiner
15 Revised mapper
16 In mapper combining
17 A running example Build word co occurrence matrices for large corpora Build the word co occurrence matrix for all the works by Shakespeare Co occurrence within a specific context A sentence A paragraph A document A certain window of m words
18 Pairs For each map task For each unique pair (a, b), emit [pair (a, b), count] Use combiner or in mapper combining to reduce the volume of intermediate pairs Reducers sum up counts associated with these pairs
19 Pairs approach
20 Stripes Idea: group together pairs into an associative array (a, b) 1 (a, c) 2 (a, d) 5 (a, e) 3 (a, f) 2 a { b: 1, c: 2, d: 5, e: 3, f: 2 } Mappers emit [word, associate array] Reducers perform element wise sum of associative arrays + a { b: 1, d: 5, e: 3 } a { b: 1, c: 2, d: 2, f: 2 } a { b: 2, c: 2, d: 7, e: 3, f: 2 }
21 Stripes approach
22 Relative frequency What proportion of the time does B appear in the context A? Whenever there is a co occurrence of (A,*), for what percentage will * be B The total count of co occurrence of (A,*) is called marginal f ( B A) count( A, B) count( A,*) B' count( A, B) count( A, B') (A, *) 32 (A, B 1 ) 3 (A, B 2 ) 12 (A, B 3 ) 7 (A, B 4 ) 1 (A, B 5 ) 4 (A, B 6 ) 5 Reducer holds this value in memory (A, B 1 ) 3 / 32 (A, B 2 ) 12 / 32 (A, B 3 ) 7 / 32 (A, B 4 ) 1 / 32 (A, B 5 ) 4 / 32 (A, B 6 ) 5 / 32
23 Relative frequency with stripes It is easy One pass to compute (A, *) Another pass to directly compute f(b A) May have the scalability issue for really large data The final associative array holds all the neighbors and their co occurrence counts with the word A
24 Relative frequency with pairs Must emit extra (A, *) for every B (B can be any word) in mapper Must make sure all pairs of (A, *) and (A, B) get sent to same reducer (use partitioner) Must make sure (A, *) comes first (define sort order) Must hold state in reducer across different key value pairs This pattern is called order inversion
25 Secondary Sorting A motivating example The readings of m sensors are recorded over the time (t1, m1, r80521) (t1, m2, r14209) (t1, m3, r76042) (t2, m1, r21823) (t2, m2, r66508) (t2, m3, r98347) (m1, t1, r80521) (m1, t2, r21823) (m2, t1, r14209) (m2, t2, r66508) (m3, t1, r76042) (m3, t2, r98347)
26 First approach (t1, m1, r80521) (t1, m2, r14209) (t1, m3, r76042) (t2, m1, r21823) (t2, m2, r66508) (t2, m3, r98347) (m1, (t1, r80521)) (m2, (t1, r14209)) (m3, (t1, r76042)) (m1, (t2, r21823)) (m2, (t2, r66508)) (m3, (t2, r98347)) However, Hadoop MapReduce sorts intermediate pairs by key Values can be arbitrarily ordered E.g., (m1, [(t100, r23456), (t2, r21823),,(t234, r34870)]) Buffer values in memory, then sort Is there issue with this approach?
27 Second approach Value to key conversion Move part of the value into the intermediate key to form a composite key Composite key: (m, t) Intermediate pair: ((m, t), r) Let execution framework do the sorting First sort by the sensor id, i.e., m (the left element in the key) Then sort by the timestamp, i.e., t (the right element in the key) Implement the custom partitioner All pairs with the same sensor shuffled to the same reducer ((m1, t1), r80521) ((m1, t2), r21823) ((m1, t3), r149625)
Developing MapReduce Programs
Cloud Computing Developing MapReduce Programs Dell Zhang Birkbeck, University of London 2017/18 MapReduce Algorithm Design MapReduce: Recap Programmers must specify two functions: map (k, v) * Takes
More informationBig Data Infrastructure CS 489/698 Big Data Infrastructure (Winter 2016)
Big Data Infrastructure CS 489/698 Big Data Infrastructure (Winter 2016) Week 2: MapReduce Algorithm Design (2/2) January 14, 2016 Jimmy Lin David R. Cheriton School of Computer Science University of Waterloo
More informationMapReduce Design Patterns
MapReduce Design Patterns MapReduce Restrictions Any algorithm that needs to be implemented using MapReduce must be expressed in terms of a small number of rigidly defined components that must fit together
More informationLecture 7: MapReduce design patterns! Claudia Hauff (Web Information Systems)!
Big Data Processing, 2014/15 Lecture 7: MapReduce design patterns!! Claudia Hauff (Web Information Systems)! ti2736b-ewi@tudelft.nl 1 Course content Introduction Data streams 1 & 2 The MapReduce paradigm
More informationTI2736-B Big Data Processing. Claudia Hauff
TI2736-B Big Data Processing Claudia Hauff ti2736b-ewi@tudelft.nl Intro Streams Streams Map Reduce HDFS Pig Pig Design Patterns Hadoop Ctd. Graphs Giraph Spark Zoo Keeper Spark Learning objectives Implement
More informationBig Data Infrastructure CS 489/698 Big Data Infrastructure (Winter 2017)
Big Data Infrastructure CS 489/698 Big Data Infrastructure (Winter 2017) Week 2: MapReduce Algorithm Design (2/2) January 12, 2017 Jimmy Lin David R. Cheriton School of Computer Science University of Waterloo
More informationData-Intensive Distributed Computing
Data-Intensive Distributed Computing CS 451/651 431/631 (Winter 2018) Part 1: MapReduce Algorithm Design (4/4) January 16, 2018 Jimmy Lin David R. Cheriton School of Computer Science University of Waterloo
More informationAnurag Sharma (IIT Bombay) 1 / 13
0 Map Reduce Algorithm Design Anurag Sharma (IIT Bombay) 1 / 13 Relational Joins Anurag Sharma Fundamental Research Group IIT Bombay Anurag Sharma (IIT Bombay) 1 / 13 Secondary Sorting Required if we need
More informationLaboratory Session: MapReduce
Laboratory Session: MapReduce Algorithm Design in MapReduce Pietro Michiardi Eurecom Pietro Michiardi (Eurecom) Laboratory Session: MapReduce 1 / 63 Preliminaries Preliminaries Pietro Michiardi (Eurecom)
More informationMapReduce Algorithms
Large-scale data processing on the Cloud Lecture 3 MapReduce Algorithms Satish Srirama Some material adapted from slides by Jimmy Lin, 2008 (licensed under Creation Commons Attribution 3.0 License) Outline
More informationAlgorithms for MapReduce. Combiners Partition and Sort Pairs vs Stripes
Algorithms for MapReduce 1 Assignment 1 released Due 16:00 on 20 October Correctness is not enough! Most marks are for efficiency. 2 Combining, Sorting, and Partitioning... and algorithms exploiting these
More informationMapReduce Patterns, Algorithms, and Use Cases
MapReduce Patterns, Algorithms, and Use Cases In this article I digested a number of MapReduce patterns and algorithms to give a systematic view of the different techniques that can be found on the web
More informationMapReduce Algorithm Design
MapReduce Algorithm Design Bu eğitim sunumları İstanbul Kalkınma Ajansı nın 2016 yılı Yenilikçi ve Yaratıcı İstanbul Mali Destek Programı kapsamında yürütülmekte olan TR10/16/YNY/0036 no lu İstanbul Big
More informationBasic MapReduce Algorithm Design
Chapter 3 Basic MapReduce Algorithm Design This is a post-production manuscript of: Jimmy Lin and Chris Dyer. Data-Intensive Text Processing with MapReduce. Morgan & Claypool Publishers, 2010. This version
More informationData Partitioning and MapReduce
Data Partitioning and MapReduce Krzysztof Dembczyński Intelligent Decision Support Systems Laboratory (IDSS) Poznań University of Technology, Poland Intelligent Decision Support Systems Master studies,
More informationChapter 4. Distributed Algorithms based on MapReduce. - Applications
Chapter 4 Distributed Algorithms based on MapReduce - Applications 1 Acknowledgements MapReduce Algorithms - Understanding Data Joins: http://codingjunkie.net/mapreduce-reduce-joins/ Joins with Map Reduce:
More informationCONTAİNERS COLLECTİONS
CONTAİNERS Some programs create too many objects and deal with them. In such a program, it is not feasible to declare a separate variable to hold reference to each of these objects. The proper way of keeping
More informationIntroduction to MapReduce
732A54 Big Data Analytics Introduction to MapReduce Christoph Kessler IDA, Linköping University Towards Parallel Processing of Big-Data Big Data too large to be read+processed in reasonable time by 1 server
More information27/04/2012. Objectives. Collection. Collections Framework. "Collection" Interface. Collection algorithm. Legacy collection
Objectives Collection Collections Framework Concrete collections Collection algorithm By Võ Văn Hải Faculty of Information Technologies Summer 2012 Legacy collection 1 2 2/27 Collections Framework "Collection"
More informationBig Data Infrastructure CS 489/698 Big Data Infrastructure (Winter 2017)
Big Data Infrastructure CS 489/698 Big Data Infrastructure (Winter 07) Week 4: Analyzing Text (/) January 6, 07 Jimmy Lin David R. Cheriton School of Computer Science University of Waterloo These slides
More informationKillTest *KIJGT 3WCNKV[ $GVVGT 5GTXKEG Q&A NZZV ]]] QORRZKYZ IUS =K ULLKX LXKK [VJGZK YKX\OIK LUX UTK _KGX
KillTest Q&A Exam : CCD-410 Title : Cloudera Certified Developer for Apache Hadoop (CCDH) Version : DEMO 1 / 4 1.When is the earliest point at which the reduce method of a given Reducer can be called?
More informationMapReduce Patterns. MCSN - N. Tonellotto - Distributed Enabling Platforms
MapReduce Patterns 1 Intermediate Data Written locally Transferred from mappers to reducers over network Issue - Performance bottleneck Solution - Use combiners - Use In-Mapper Combining 2 Original Word
More informationVendor: Cloudera. Exam Code: CCD-410. Exam Name: Cloudera Certified Developer for Apache Hadoop. Version: Demo
Vendor: Cloudera Exam Code: CCD-410 Exam Name: Cloudera Certified Developer for Apache Hadoop Version: Demo QUESTION 1 When is the earliest point at which the reduce method of a given Reducer can be called?
More informationHadoop Map Reduce 10/17/2018 1
Hadoop Map Reduce 10/17/2018 1 MapReduce 2-in-1 A programming paradigm A query execution engine A kind of functional programming We focus on the MapReduce execution engine of Hadoop through YARN 10/17/2018
More information15/03/2018. Counters
Counters 2 1 Hadoop provides a set of basic, built-in, counters to store some statistics about jobs, mappers, reducers E.g., number of input and output records E.g., number of transmitted bytes Ad-hoc,
More informationMap Reduce. Yerevan.
Map Reduce Erasmus+ @ Yerevan dacosta@irit.fr Divide and conquer at PaaS 100 % // Typical problem Iterate over a large number of records Extract something of interest from each Shuffle and sort intermediate
More informationApril Final Quiz COSC MapReduce Programming a) Explain briefly the main ideas and components of the MapReduce programming model.
1. MapReduce Programming a) Explain briefly the main ideas and components of the MapReduce programming model. MapReduce is a framework for processing big data which processes data in two phases, a Map
More informationWe are ready to serve Latest Testing Trends, Are you ready to learn?? New Batches Info
We are ready to serve Latest Testing Trends, Are you ready to learn?? New Batches Info START DATE : TIMINGS : DURATION : TYPE OF BATCH : FEE : FACULTY NAME : LAB TIMINGS : PH NO: 9963799240, 040-40025423
More informationData-Intensive Distributed Computing
Data-Intensive Distributed Computing CS 45/65 43/63 (Winter 08) Part 3: Analyzing Text (/) January 30, 08 Jimmy Lin David R. Cheriton School of Computer Science University of Waterloo These slides are
More informationInforma)on Retrieval and Map- Reduce Implementa)ons. Mohammad Amir Sharif PhD Student Center for Advanced Computer Studies
Informa)on Retrieval and Map- Reduce Implementa)ons Mohammad Amir Sharif PhD Student Center for Advanced Computer Studies mas4108@louisiana.edu Map-Reduce: Why? Need to process 100TB datasets On 1 node:
More informationCMSC 723: Computational Linguistics I Session #12 MapReduce and Data Intensive NLP. University of Maryland. Wednesday, November 18, 2009
CMSC 723: Computational Linguistics I Session #12 MapReduce and Data Intensive NLP Jimmy Lin and Nitin Madnani University of Maryland Wednesday, November 18, 2009 Three Pillars of Statistical NLP Algorithms
More informationMap-Reduce and Adwords Problem
Map-Reduce and Adwords Problem Map-Reduce and Adwords Problem Miłosz Kadziński Institute of Computing Science Poznan University of Technology, Poland www.cs.put.poznan.pl/mkadzinski/wpi Big Data (1) Big
More information17. Java Collections. Organizing Data. Generic List in Java: java.util.list. Type Parameters ( Parameteric Polymorphism ) Data Structures that we know
Organizing Data Data Structures that we know 17 Java Collections Generic Types, Iterators, Java Collections, Iterators Today: Arrays Fixed-size sequences Strings Sequences of characters Linked Lists (up
More informationDatabases 2 (VU) ( / )
Databases 2 (VU) (706.711 / 707.030) MapReduce (Part 3) Mark Kröll ISDS, TU Graz Nov. 27, 2017 Mark Kröll (ISDS, TU Graz) MapReduce Nov. 27, 2017 1 / 42 Outline 1 Problems Suited for Map-Reduce 2 MapReduce:
More informationChapter 3. Distributed Algorithms based on MapReduce
Chapter 3 Distributed Algorithms based on MapReduce 1 Acknowledgements Hadoop: The Definitive Guide. Tome White. O Reilly. Hadoop in Action. Chuck Lam, Manning Publications. MapReduce: Simplified Data
More informationParallel Programming Principle and Practice. Lecture 10 Big Data Processing with MapReduce
Parallel Programming Principle and Practice Lecture 10 Big Data Processing with MapReduce Outline MapReduce Programming Model MapReduce Examples Hadoop 2 Incredible Things That Happen Every Minute On The
More information[3-5] Consider a Combiner that tracks local counts of followers and emits only the local top10 users with their number of followers.
Quiz 6 We design of a MapReduce application that generates the top-10 most popular Twitter users (based on the number of followers) of a month based on the number of followers. Suppose that you have the
More informationSets and Maps. Sets Maps The Comparator Interface Sets and Maps in Java Collections API. Review for Exam Reading:
Sets and Maps Sets Maps The Comparator Interface Sets and Maps in Java Collections API TreeSet TreeMap Review for Exam Reading: 13.1-13.6 Sets A set is a collection of elements with no duplicates We had
More informationHash Table. Ric Glassey
Hash Table Ric Glassey glassey@kth.se Overview Hash Table Aim: Describe the map abstract data type with efficient insertion, deletion and search operations Motivation: List data structures are divided
More informationHadoop. copyright 2011 Trainologic LTD
Hadoop Hadoop is a framework for processing large amounts of data in a distributed manner. It can scale up to thousands of machines. It provides high-availability. Provides map-reduce functionality. Hides
More informationitpass4sure Helps you pass the actual test with valid and latest training material.
itpass4sure http://www.itpass4sure.com/ Helps you pass the actual test with valid and latest training material. Exam : CCD-410 Title : Cloudera Certified Developer for Apache Hadoop (CCDH) Vendor : Cloudera
More informationSets and Maps. Part of the Collections Framework
Sets and Maps Part of the Collections Framework The Set interface A Set is unordered and has no duplicates Operations are exactly those for Collection int size( ); boolean isempty( ); boolean contains(object
More informationApache Hive for Oracle DBAs. Luís Marques
Apache Hive for Oracle DBAs Luís Marques About me Oracle ACE Alumnus Long time open source supporter Founder of Redglue (www.redglue.eu) works for @redgluept as Lead Data Architect @drune After this talk,
More informationProgramming Models MapReduce
Programming Models MapReduce Majd Sakr, Garth Gibson, Greg Ganger, Raja Sambasivan 15-719/18-847b Advanced Cloud Computing Fall 2013 Sep 23, 2013 1 MapReduce In a Nutshell MapReduce incorporates two phases
More informationData Analytics Framework and Methodology for WhatsApp Chats
Data Analytics Framework and Methodology for WhatsApp Chats Transliteration of Thanglish and Short WhatsApp Messages P. Sudhandradevi Department of Computer Applications Bharathiar University Coimbatore,
More informationThe MapReduce Abstraction
The MapReduce Abstraction Parallel Computing at Google Leverages multiple technologies to simplify large-scale parallel computations Proprietary computing clusters Map/Reduce software library Lots of other
More informationPrincipal Software Engineer Red Hat Emerging Technology June 24, 2015
USING APACHE SPARK FOR ANALYTICS IN THE CLOUD William C. Benton Principal Software Engineer Red Hat Emerging Technology June 24, 2015 ABOUT ME Distributed systems and data science in Red Hat's Emerging
More information11-1. Collections. CSE 143 Java. Java 2 Collection Interfaces. Goals for Next Several Lectures
Collections CSE 143 Java Collections Most programs need to store and access collections of data Collections are worth studying because... They are widely useful in programming They provide examples of
More informationIntroducing Hashing. Chapter 21. Copyright 2012 by Pearson Education, Inc. All rights reserved
Introducing Hashing Chapter 21 Contents What Is Hashing? Hash Functions Computing Hash Codes Compressing a Hash Code into an Index for the Hash Table A demo of hashing (after) ARRAY insert hash index =
More informationBig Data and Scripting map reduce in Hadoop
Big Data and Scripting map reduce in Hadoop 1, 2, connecting to last session set up a local map reduce distribution enable execution of map reduce implementations using local file system only all tasks
More informationDistributed Systems. CS422/522 Lecture17 17 November 2014
Distributed Systems CS422/522 Lecture17 17 November 2014 Lecture Outline Introduction Hadoop Chord What s a distributed system? What s a distributed system? A distributed system is a collection of loosely
More informationLecture Map-Reduce. Algorithms. By Marina Barsky Winter 2017, University of Toronto
Lecture 04.02 Map-Reduce Algorithms By Marina Barsky Winter 2017, University of Toronto Example 1: Language Model Statistical machine translation: Need to count number of times every 5-word sequence occurs
More informationAbstract Data Types Spring 2018 Exam Prep 5: February 12, 2018
CS 61B Abstract Data Types Spring 2018 Exam Prep 5: February 12, 2018 1 Assorted ADTs A list is an ordered collection, or sequence. 1 interface List { 2 boolean add(e element); 3 void add(int index,
More informationCS 2102 Exam 2 B-Term 2013
NAME: CS 2102 Exam 2 B-Term 2013 Question 1: (25) Question 2: (55) Question 3: (20) TOTAL: (100) 1 1. A method that computes n m is correctly coded recursively as follows: class Power{ Power(){ // computes
More informationMap Reduce.
Map Reduce dacosta@irit.fr Divide and conquer at PaaS Second Third Fourth 100 % // Fifth Sixth Seventh Cliquez pour 2 Typical problem Second Extract something of interest from each MAP Third Shuffle and
More informationCME 323: Distributed Algorithms and Optimization Instructor: Reza Zadeh HW#3 - Due at the beginning of class May 18th.
CME 323: Distributed Algorithms and Optimization Instructor: Reza Zadeh (rezab@stanford.edu) HW#3 - Due at the beginning of class May 18th. 1. Download the following materials: Slides: http://stanford.edu/~rezab/dao/slides/itas_workshop.pdf
More informationBatch Processing Basic architecture
Batch Processing Basic architecture in big data systems COS 518: Distributed Systems Lecture 10 Andrew Or, Mike Freedman 2 1 2 64GB RAM 32 cores 64GB RAM 32 cores 64GB RAM 32 cores 64GB RAM 32 cores 3
More information15/03/2018. Combiner
Combiner 2 1 Standard MapReduce applications The (key,value) pairs emitted by the Mappers are sent to the Reducers through the network Some pre-aggregations could be performed to limit the amount of network
More informationMitigating Data Skew Using Map Reduce Application
Ms. Archana P.M Mitigating Data Skew Using Map Reduce Application Mr. Malathesh S.H 4 th sem, M.Tech (C.S.E) Associate Professor C.S.E Dept. M.S.E.C, V.T.U Bangalore, India archanaanil062@gmail.com M.S.E.C,
More informationIntroduction to MapReduce (cont.)
Introduction to MapReduce (cont.) Rafael Ferreira da Silva rafsilva@isi.edu http://rafaelsilva.com USC INF 553 Foundations and Applications of Data Mining (Fall 2018) 2 MapReduce: Summary USC INF 553 Foundations
More informationFall 2017 Mentoring 7: October 9, Abstract Data Types
CSM 61B Sequences & ADTs Fall 2017 Mentoring 7: October 9, 2017 1 Abstract Data Types A list is an ordered sequence of items: like an array, but without worrying about the length or size. interface List
More informationGraph Algorithms using Map-Reduce. Graphs are ubiquitous in modern society. Some examples: The hyperlink structure of the web
Graph Algorithms using Map-Reduce Graphs are ubiquitous in modern society. Some examples: The hyperlink structure of the web Graph Algorithms using Map-Reduce Graphs are ubiquitous in modern society. Some
More informationHadoop and Map-reduce computing
Hadoop and Map-reduce computing 1 Introduction This activity contains a great deal of background information and detailed instructions so that you can refer to it later for further activities and homework.
More informationAn Introduction to The Beam Model
An Introduction to The Beam Model Apache Beam (incubating) Slides by Tyler Akidau & Frances Perry, April 2016 Agenda 1 Infinite, Out-of-order Data Sets 2 The Evolution of the Beam Model 3 What, Where,
More informationCS S-17 Recursion IV 1. ArrayLists give some extra functionality to arrays (automatic resizing, code for inserting, etc)
CS112-2012S-17 Recursion IV 1 17-0: HashMap Arrays allow us to store elements in a list, using ints to reference locations ArrayLists give some extra functionality to arrays (automatic resizing, code for
More informationPart A: MapReduce. Introduction Model Implementation issues
Part A: Massive Parallelism li with MapReduce Introduction Model Implementation issues Acknowledgements Map-Reduce The material is largely based on material from the Stanford cources CS246, CS345A and
More informationSpring 2017 EXTERNAL SORTING (CH. 13 IN THE COW BOOK) 2/7/17 CS 564: Database Management Systems; (c) Jignesh M. Patel,
Spring 2017 EXTERNAL SORTING (CH. 13 IN THE COW BOOK) 2/7/17 CS 564: Database Management Systems; (c) Jignesh M. Patel, 2013 1 Motivation for External Sort Often have a large (size greater than the available
More informationParallel Dijkstra s Algorithm
CSCI4180 Tutorial-6 Parallel Dijkstra s Algorithm ZHANG, Mi mzhang@cse.cuhk.edu.hk Nov. 5, 2015 Definition Model the Twitter network as a directed graph. Each user is represented as a node with a unique
More informationEfficiency. Efficiency: Indexing. Indexing. Efficiency Techniques. Inverted Index. Inverted Index (COSC 488)
Efficiency Efficiency: Indexing (COSC 488) Nazli Goharian nazli@cs.georgetown.edu Difficult to analyze sequential IR algorithms: data and query dependency (query selectivity). O(q(cf max )) -- high estimate-
More informationJava Collections Framework reloaded
Java Collections Framework reloaded October 1, 2004 Java Collections - 2004-10-01 p. 1/23 Outline Interfaces Implementations Ordering Java 1.5 Java Collections - 2004-10-01 p. 2/23 Components Interfaces:
More informationCSC 321: Data Structures. Fall 2016
CSC 321: Data Structures Fall 2016 Balanced and other trees balanced BSTs: AVL trees, red-black trees TreeSet & TreeMap implementations heaps priority queue implementation heap sort 1 Balancing trees recall:
More informationCSC 321: Data Structures. Fall 2017
CSC 321: Data Structures Fall 2017 Balanced and other trees balanced BSTs: AVL trees, red-black trees TreeSet & TreeMap implementations heaps priority queue implementation heap sort 1 Balancing trees recall:
More informationPortable stateful big data processing in Apache Beam
Portable stateful big data processing in Apache Beam Kenneth Knowles Apache Beam PMC Software Engineer @ Google klk@google.com / @KennKnowles https://s.apache.org/ffsf-2017-beam-state Flink Forward San
More informationPSON: A Parallelized SON Algorithm with MapReduce for Mining Frequent Sets
2011 Fourth International Symposium on Parallel Architectures, Algorithms and Programming PSON: A Parallelized SON Algorithm with MapReduce for Mining Frequent Sets Tao Xiao Chunfeng Yuan Yihua Huang Department
More informationGraphics Pipeline 2D Geometric Transformations
Graphics Pipeline 2D Geometric Transformations CS 4620 Lecture 8 1 Plane projection in drawing Albrecht Dürer 2 Plane projection in drawing source unknown 3 Rasterizing triangles Summary 1 evaluation of
More informationFall 2018: Introduction to Data Science GIRI NARASIMHAN, SCIS, FIU
Fall 2018: Introduction to Data Science GIRI NARASIMHAN, SCIS, FIU !2 MapReduce Overview! Sometimes a single computer cannot process data or takes too long traditional serial programming is not always
More informationCS / Cloud Computing. Recitation 3 September 9 th & 11 th, 2014
CS15-319 / 15-619 Cloud Computing Recitation 3 September 9 th & 11 th, 2014 Overview Last Week s Reflection --Project 1.1, Quiz 1, Unit 1 This Week s Schedule --Unit2 (module 3 & 4), Project 1.2 Questions
More informationShark: Hive (SQL) on Spark
Shark: Hive (SQL) on Spark Reynold Xin UC Berkeley AMP Camp Aug 21, 2012 UC BERKELEY SELECT page_name, SUM(page_views) views FROM wikistats GROUP BY page_name ORDER BY views DESC LIMIT 10; Stage 0: Map-Shuffle-Reduce
More informationEnabling Cost-effective Data Processing with Smart SSD
Enabling Cost-effective Data Processing with Smart SSD Yangwook Kang, UC Santa Cruz Yang-suk Kee, Samsung Semiconductor Ethan L. Miller, UC Santa Cruz Chanik Park, Samsung Electronics Efficient Use of
More informationToday's Agenda. > To give a practical introduction to data structures. > To look specifically at Lists, Sets, and Maps
Today's Agenda > To give a practical introduction to data structures > To look specifically at Lists, Sets, and Maps > To talk briefly about Generics in Java > To talk about interfaces in Java Data Structures
More informationClustering Lecture 8: MapReduce
Clustering Lecture 8: MapReduce Jing Gao SUNY Buffalo 1 Divide and Conquer Work Partition w 1 w 2 w 3 worker worker worker r 1 r 2 r 3 Result Combine 4 Distributed Grep Very big data Split data Split data
More informationInformation Retrieval Processing with MapReduce
Information Retrieval Processing with MapReduce Based on Jimmy Lin s Tutorial at the 32nd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2009) This
More informationThe MapReduce Framework
The MapReduce Framework In Partial fulfilment of the requirements for course CMPT 816 Presented by: Ahmed Abdel Moamen Agents Lab Overview MapReduce was firstly introduced by Google on 2004. MapReduce
More informationCS 314 Midterm 2 Fall 2012
Points off 1 2 3 4 5 Total off Net Score CS 314 Midterm 2 Fall 2012 Your Name_ Your UTEID Circle yours TA s name: John Zihao Instructions: 1. There are 5 questions on this test. 2. You have 2 hours to
More information1B1a Arrays. Arrays. Indexing. Naming arrays. Why? Using indexing. 1B1a Lecture Slides. Copyright 2003, Graham Roberts 1
Ba Arrays Arrays A normal variable holds value: An array variable holds a collection of values: 4 Naming arrays An array has a single name, so the elements are numbered or indexed. 0 3 4 5 Numbering starts
More informationMap/Reduce. Large Scale Duplicate Detection. Prof. Felix Naumann, Arvid Heise
Map/Reduce Large Scale Duplicate Detection Prof. Felix Naumann, Arvid Heise Agenda 2 Big Data Word Count Example Hadoop Distributed File System Hadoop Map/Reduce Advanced Map/Reduce Stratosphere Agenda
More informationApp Engine MapReduce. Mike Aizatsky 11 May Hashtags: #io2011 #AppEngine Feedback:
App Engine MapReduce Mike Aizatsky 11 May 2011 Hashtags: #io2011 #AppEngine Feedback: http://goo.gl/snv2i Agenda MapReduce Computational Model Mapper library Announcement Technical bits: Files API User-space
More informationCSC630/CSC730 Parallel & Distributed Computing
CSC630/CSC730 Parallel & Distributed Computing Analytical Modeling of Parallel Programs Chapter 5 1 Contents Sources of Parallel Overhead Performance Metrics Granularity and Data Mapping Scalability 2
More informationUniversity of Maryland. Tuesday, March 23, 2010
Data-Intensive Information Processing Applications Session #7 MapReduce and databases Jimmy Lin University of Maryland Tuesday, March 23, 2010 This work is licensed under a Creative Commons Attribution-Noncommercial-Share
More informationIntroduction to Data Management CSE 344
Introduction to Data Management CSE 344 Lecture 26: Parallel Databases and MapReduce CSE 344 - Winter 2013 1 HW8 MapReduce (Hadoop) w/ declarative language (Pig) Cluster will run in Amazon s cloud (AWS)
More informationModel Solutions. COMP 103: Test April, 2013
Family Name:............................. Other Names:............................. ID Number:............................... Signature.................................. Instructions Time allowed: 40 minutes
More informationGraphCEP Real-Time Data Analytics Using Parallel Complex Event and Graph Processing
Institute of Parallel and Distributed Systems () Universitätsstraße 38 D-70569 Stuttgart GraphCEP Real-Time Data Analytics Using Parallel Complex Event and Graph Processing Ruben Mayer, Christian Mayer,
More informationCollective Communication Patterns for Iterative MapReduce
Collective Communication Patterns for Iterative MapReduce CONTENTS 1 Introduction... 4 2 Background... 6 2.1 Collective Communication... 6 2.2 MapReduce... 7 2.3 Iterative MapReduce... 8 3 MapReduce-MergeBroadcast...
More information1. Stratified sampling is advantageous when sampling each stratum independently.
Quiz 1. 1. Stratified sampling is advantageous when sampling each stratum independently. 2. All outliers within a dataset are invalid observations. 3. Consider a dataset comprising a set of (single value)
More informationVendor: Hortonworks. Exam Code: HDPCD. Exam Name: Hortonworks Data Platform Certified Developer. Version: Demo
Vendor: Hortonworks Exam Code: HDPCD Exam Name: Hortonworks Data Platform Certified Developer Version: Demo QUESTION 1 Workflows expressed in Oozie can contain: A. Sequences of MapReduce and Pig. These
More informationCS 378 Big Data Programming. Lecture 4 Summariza9on Pa:erns
CS 378 Big Data Programming Lecture 4 Summariza9on Pa:erns Review Assignment 1 Ques9ons? Using maven Using AWS Simple Debugging Counters controller syslog Custom counters context.getcounter(group, counter).increment(1l);
More informationLarge-Scale GPU programming
Large-Scale GPU programming Tim Kaldewey Research Staff Member Database Technologies IBM Almaden Research Center tkaldew@us.ibm.com Assistant Adjunct Professor Computer and Information Science Dept. University
More informationCloud Computing CS
Cloud Computing CS 15-319 Programming Models- Part III Lecture 6, Feb 1, 2012 Majd F. Sakr and Mohammad Hammoud 1 Today Last session Programming Models- Part II Today s session Programming Models Part
More informationMongoDB DI Dr. Angelika Kusel
MongoDB DI Dr. Angelika Kusel 1 Motivation Problem Data is partitioned over large scale clusters Clusters change the rules for processing Good news Lots of machines to spread the computation over Bad news
More informationMapReduce: Recap. Juliana Freire & Cláudio Silva. Some slides borrowed from Jimmy Lin, Jeff Ullman, Jerome Simeon, and Jure Leskovec
MapReduce: Recap Some slides borrowed from Jimmy Lin, Jeff Ullman, Jerome Simeon, and Jure Leskovec MapReduce: Recap Sequentially read a lot of data Why? Map: extract something we care about map (k, v)
More information