Programming Languages and Techniques (CIS120e)

Transcription

1 Programming Languages and Techniques (CIS120e) Lecture 11 Oct 1, 2010 MapReduce

2 FuncBons as Data In the past couple of lectures, we ve seen a number of ways in which funcbons can be treated as data in OCaml (passed as arguments to funcbons, etc.) Present- day programming pracbce offers many more examples iterators ( cursors for walking over data structures) event listeners (in GUIs) etc. One parbcularly well- known example: MapReduce CIS120e / Fall

3 MapReduce Natural framework for efficient distributed computabon over massive data sets Developed by Google engineers about 10 years ago based on funcbonal programming ideas Now available in many variabons CIS120e / Fall

4 from Wikipedia MapReduce ImplementaBons The Google MapReduce framework is implemented in C++ with interfaces in Python and Java." The Hadoop project is a free open source Java MapReduce implementation." Twister is an open source Java MapReduce implementation that supports iterative MapReduce computations efficiently." Greenplum is a commercial MapReduce implementation, with support for Python, Perl, SQL and other languages.[14]" Aster Data Systems ncluster In-Database MapReduce supports Java, C, C++, Perl, and Python algorithms integrated into ANSI SQL.[15]" GridGain is a free open source Java MapReduce implementation." Phoenix is a shared-memory implementation of MapReduce implemented in C." Plasma MapReduce is an open source MapReduce implementation in Ocaml with its own distributed filesystem, PlasmaFS." FileMap is an open version of the framework that operates on files using existing file-processing tools rather than tuples." MapReduce has also been implemented for the Cell Broadband Engine, also in C. [2]" Mars: MapReduce has been implemented on NVIDIA GPUs (Graphics Processors) using CUDA [3]." Qt Concurrent is a simplified version of the framework, implemented in C++, used for distributing a task between multiple processor cores [16]." CouchDB uses a MapReduce framework for defining views over distributed documents and is implemented in Erlang." Skynet is an open source Ruby implementation of Googleʼs MapReduce framework" mincemeat.py is a lightweight, open source Python implementation of Google's MapReduce framework." Disco is an open source MapReduce implementation by Nokia. Its core is written in Erlang and jobs are normally written in Python." Misco is an open source MapReduce designed for mobile devices and is implemented in Python." Qizmt is an open source MapReduce framework from MySpace written in C#." The open-source Hive framework from Facebook (which provides an SQL-like language over files, layered on the open-source Hadoop MapReduce engine.)" The Holumbus Framework: Distributed computing with MapReduce in Haskell Holumbus-MapReduce" BashReduce: MapReduce written as a Bash script written by Erik Frey of Last.fm" Sector/Sphere which is implemented in C++." MapReduce for Go" MongoDB is a scalable, high-performance, open source, schema-free, document-oriented database. Written in C++ that features MapReduce" mapreduce provides R-like implementation that demonstrates the simplicity of the mapreduce pattern in a functional programming language.[4]" RHIPE integrates the R statistics language environment with Hadoop.[17] and makes it possible to code map-reduce algorithms in R." Parallel::MapReduce is a CPAN module providing experimental MapReduce functionality for Perl." Many companies have their own private MapReduce implementations. CIS120e / Fall

5 MapReduce ApplicaBons distributed searching distributed sorbng web link- graph reversal term- vector per host web access log stats inverted index construcbon document clustering machine learning stabsbcal machine translabon... CIS120e / Fall

6 MapReduce Steps (i) iterabon over the input; (ii) computabon of key/value pairs from each piece of input; (iii) grouping of all intermediate values by key; (iv) iterabon over the resulbng groups; (v) reducbon of each group. CIS120e / Fall

7 Example type page = string list! type web = page list! In: let myweb : web = [! ["foo"; "bar"; "baz"];! ["foo"; "baz"; "quux"];! ["foo"; "bar"]! ]! Out: bar -> 2! baz -> 2! foo -> 3! quux -> 1! CIS120e / Fall

8 Demo CIS120e / Fall