Process Big Data in MATLAB Using MapReduce

Transcription

1 Process Big Data in MATLAB Using MapReduce This example shows how to use the datastore and mapreduce functions to process a large amount of file-based data. The MapReduce algorithm is a mainstay of many modern big data applications. This example operates on a single computer, but the code can scale up to use Hadoop. The data set used here is a collection of records for USA domestic airline flights between 1987 and If you have experimented with big data before, you may already be familiar with this data set. The full data set can be downloaded from A small subset of the data set is also included with MATLAB to allow you to run this and other examples without downloading the entire data set. Introduction to Datastore Creating a datastore allows you to access a collection of data in a chunk-based manner. A datastore can process arbitrarily large amounts of data, and the data can even be spread across multiple files. You can create a datastore for a collection of tabular text files (demonstrated next), image files, custom binary files, or a Hadoop Distributed File System (HDFS ). Create a datastore for a collection of tabular text files and preview the contents. ds = datastore( airlinesmall.csv ); dspreview = preview(ds); dspreview(:,10:15) ans = Flightnum TailNum ActualElapsedTime CRSElapsedTime Airtime ArrDelay 1503 NA NA NA NA NA NA NA NA NA NA NA NA NA NA NA NA 11

2 The datastore automatically parses the input data and makes a best guess as to the type of data in each column. In this case, use the TreatAsMissing Name-Value pair argument to have datastore replace the missing values correctly. For numeric variables (such as AirTime ), datastore replaces every NA string with a NaN value, which is the IEEE arithmetic representation for Not-a-Number. ds = datastore( airlinesmall.csv, TreatAsMissing, NA ); ds.selectedformats{strcmp(ds.selectedvariablenames, TailNum )} = %s ; ds.selectedformats{strcmp(ds.selectedvariablenames, CancellationCode )} = %s ; dspreview = preview(ds); dspreview(:,{ AirTime, TaxiIn, TailNum, CancellationCode }) ans = AirTime TaxiIn TailNum CancellationCode

3 Scan for Rows of Interest datastore objects contain an internal pointer to keep track of which chunk of data the read function returns next. Use the hasdata and read functions to step through the entire data set, and filter the data set to only the rows of interest. In this case, the rows of interest are flights on United Airlines ( UA ) departing from Boston ( BOS ). subset = []; while hasdata(ds) t = read(ds); t = t(strcmp(t.uniquecarrier, UA ) & strcmp(t.origin, BOS ), :); subset = vertcat(subset, t); subset(1:10,[9,10,15:17]) ans = UniqueCarrier FlightNum ArrDelay DepDelay Origin UA BOS UA BOS UA BOS UA BOS UA BOS UA BOS UA BOS UA BOS UA BOS UA BOS

4 Introduction to Mapreduce MapReduce is an algorithmic technique to divide and conquer big data problems. In MATLAB, mapreduce requires three input arguments: 1. A datastore to read data from 2. A mapper function that is given a subset of the data to operate on. The output of the map func tion is a partial calculation. mapreduce calls the mapper function one time for each chunk in the datastore, with each call operating indepently. 3. A reducer function that is given the aggregate outputs from the mapper function. The reducer function finishes the computation begun by the mapper function, and outputs the final answer. This is an over-simplification to some extent, since the output of a call to the mapper function can be shuffled and combined in interesting ways before being passed to the reducer function. This will be examined later in this example. Use Mapreduce to Perform a Computation A simple use of mapreduce is to find the longest flight time in the entire airline data set. To do this: 1. The mapper function computes the maximum of each chunk from the datastore. 2. The reducer function then computes the maximum value among all of the maxima computed by the calls to the mapper function. First, reset the datastore and filter the variables to the one column of interest. reset(ds); ds.selectedvariablenames = { ActualElapsedTime }; Write the mapper function, maxtimemapper.m. It takes three input arguments: 1. The input data, which is a table obtained by applying the read function to the datastore. 2. A collection of configuration and contextual information, info. This can be ignored in most cases, as it is here. 3. An intermediate data storage object, which records the results of the calculations from the mapper function. Use the add function to add Key/Value pairs to this intermediate output. In this exam ple, the name of the key ( MaxElapsedTime ) is arbitrary. Save the following mapper function (maxtimemapper.m) in your current folder. type maxtimemapper function maxtimemapper(data, ~, intermkvstore) maxelaspedtime = max(data{:,:});

5 add(intermkvstore, MaxElaspedTime,maxElaspedTime); Next, write the reducer function. It also takes three input arguments: 1. A set of input keys. Keys will be discussed further below, but they can be ignored in some simple problems, as they are here. 2. An intermediate data input object that mapreduce passes to the reducer function. This data is in the form of Key/Value pairs, and you use the hasnext and getnext functions to iterate through the values for each key. 3. A final output data storage object. Use the add and addmulti functions to directly add Key/Value pairs to the output. Save the following reducer function (maxtimereducer.m) in your current folder. type maxtimereducer function maxtimereducer(~, intermvalsiter, outkvstore) maxelaspedtime = -inf; while hasnext(intermvalsiter) maxelaspedtime = max(maxelaspedtime, getnext(intermvalsiter)); add(outkvstore, MaxElaspedTime, maxelaspedtime); Once the mapper and reducer functions are written and saved in your current folder, you can call mapreduce using the datastore, mapper function, and reducer function. If you have Parallel Computing Toolbox, MATLAB will automatically start a pool and parallelize execution. Use the readall function to display the results of the MapReduce algorithm. result readall(result) Starting parallel pool (parpool) using the local profile... connected to 12 workers. Parallel mapreduce execution on the parallel pool:

6 ******************************** * MAPREDUCE PROGRESS * ******************************** Map 0% Reduce 0% Map 50% Reduce 0% Map 100% Reduce 0% Map 100% Reduce 100% ans = Key Value MaxElaspedTime [1650] Use of Keys in Mapreduce The use of keys is an important and powerful feature of mapreduce. Each call to the mapper function adds intermediate results to one or more named buckets, called keys. The number of calls to the mapper function by mapreduce corresponds to the number of chunks in the datastore. If the mapper function adds values to multiple keys, this leads to multiple calls to the reducer function, with each call working on only one key s intermediate values. The mapreduce function automatically manages this data movement between the map and reduce phases of the algorithm. This flexibility is useful in many contexts. The next section uses keys in a relatively obvious way for illustrative purposes. Calculating Group-wise Metrics with Mapreduce The behavior of the mapper function in this application is more complex. For every flight carrier found in the input data, use the add function to add a vector of values. This vector is a count of the number of flights for that carrier on each day in the 21+ years of data. The carrier code is the key for this vector of values. This ensures that all of the data for each carrier will be grouped together when mapreduce passes it to the reducer function. Save the following mapper function (countflightsmapper.m) in your current folder.

7 type countflightsmapper function countflightsmapper(data, ~, intermkvstore) daynumber = days((datetime(data.year, data.month, data.dayofmonth) - datetime(1987,10,1)))+1; dayssinceepoch = days(datetime(2008,12,31) - datetime(1987,10,1))+1; [airlinename, ~, airlineindex] = unique(data.uniquecarrier, stable ); for i = 1:numel(airlineName) daytotals = accumarray(daynumber(airlineindex==i), 1, [dayssinceepoc h, 1]); add(intermkvstore, airlinename{i}, daytotals); The reducer function is less complex. It simply iterates over the intermediate values and adds the vectors together. At completion, it outputs the values in this aggregate vector. Note that the reduce function does not need to sort or examine the intermediatekeysin values; each call to the reduce function by mapreduce only passes the values for one airline carrier. Save the following reducer function (countflightsreducer.m) in your current folder. type countflightsreducer function countflightsreducer(intermkeysin, intermvalsiter, outkvstore) %countflightsreducer Reducer function for mapreduce to count flights dayssinceepoch = days(datetime(2008,12,31) - datetime(1987,10,1))+1; dayarray = zeros(dayssinceepoch, 1); while hasnext(intermvalsiter) dayarray = dayarray + getnext(intermvalsiter); add(outkvstore, intermkeysin, dayarray);

8 Reset the datastore and select the variables of interest. Once the mapper and reducer functions are written and saved in your current folder, you can call mapreduce using the datastore, mapper function, and reducer function. reset(ds); ds.selectedvariablenames = { Year, Month, DayofMonth, UniqueCarrier }; result result = readall(result); Parallel mapreduce execution on the parallel pool: ******************************** * MAPREDUCE PROGRESS * ******************************** Map 0% Reduce 0% Map 50% Reduce 0% Map 100% Reduce 0% Map 100% Reduce 100% In case this example was run with only the sample data set, load the results of the mapreduce algorithm run on the entire data set. load airlineresults Visualizing the Results Using only the top 7 carriers, apply a filter to the data to smooth out the effects of week travel. This would otherwise clutter the visualization. lines = result.value; lines = horzcat(lines{:}); [~,sortorder] = sort(sum(lines), desc ); lines = lines(:,sortorder(1:7)); result = result(sortorder(1:7),:);

9 lines(lines==0) = nan; for carrier=1:size(lines,2) lines(:,carrier) = filter(repmat(1/7, [7 1]), 1, lines(:,carrier)); Plot the data. figure( Position,[ ]); plot(datetime(1987,10,1):caldays(1):datetime(2008,12,31),lines) title ( Domestic airline flights per day per carrier ) xlabel( Date ) ylabel( Flights per day (7-day moving average) ) leg(result.key, Location, South ) The plot shows the emergence of Southwest Airlines (WN) during this time period.

10 Learn More Implementing MapReduce with MATLAB 5:17 Using MapReduce with Hadoop 2:08 42:23 Tackling Big Data with MATLAB mathworks.com 2016 The MathWorks, Inc. MATLAB and Simulink are registered trademarks of The MathWorks, Inc. See mathworks.com/trademarks for a list of additional trademarks. Other product or brand names may be trademarks or registered trademarks of their respective holders v00 04/16