Lab: MapReduce Service on BlueMix

Transcription

1 Lab: MapReduce Service on BlueMix 2/15/2014 Minkyong Kim Rehan Altaf Introduction In this lab, we will show how easy it is to use the BigInsights MapReduce Service on BlueMix and connect it to a BlueMix application, to create compelling data driven apps. The scenario we use is that of a financial analyst, who wants to analyze 30 years of Nasdaq stock data to pick stocks with particular characteristics for her clients. The analyst may want to pick the most liquid stocks (i.e. stocks that have been traded the most in 30 years) or identify highly volatile stocks for day trading. A scalable Hadoop based solution can be powerful in this scenario to crunch numbers for the analyst on demand. In this scenario, BlueMix can be used to quickly build a data driven app that uses the BigInsights service to crunch numbers on historical Nasdaq data and then visualize that data with a rails app. We will demonstrate how to provision a BigInsights cluster on BlueMix and how to get a BlueMix app to submit jobs on to that cluster. How to build this yourself Prerequisites: VNC or SSH/putty A basic familiarity with JavaScript. Preferred but not required.

2 Setting up We will deploy an application (STEP 1) and then bind it to a BlueMix Hadoop service (STEP 2). Here is the list of steps for the lab experience: STEP 1. Deploy Application BlueMix provides CLI tool that allows you to deploy application. 1. SSH or VNC into the IP Address assigned to you by the instructors 2. Open a shell: xterm (If you are using VNC) 3. cf login a api.ng.bluemix.net a. Login with your BlueMix ID IBM ID: devpulse??@us.ibm.com PASSWORD: jstart Go to the application directory: cd ~/mapreduce/apps/nasdaqanalyticsapplication/ 5. cf push [YOURAPP] YOURAPP name should be nasdaq?? where?? is the last two letters of your log in ID, devpulse??.

3 6. cf set-env [YOURAPP] DATABASE_URL sqlite3://db/production.sqlite3 STEP 2. Provision/Bind MRaaS instance from BlueMix ACE UI We will log into the BlueMix UI to provision an instance of the BigInsights MapReduce service and then bind it to the application. 2a. Log into ACE UI 1. Open a browser: Disable the security in Chrome. From an xterm, start chrome from the command line with -disable-web-security option. $ google-chrome -disable-web-security & 2. From the brower, go to the 3. Login with your BlueMix ID IBM ID: devpulse??@us.ibm.com PASSWORD: jstart Navigate to your application

4

5 2b. Provision/Bind BigInsights-MapReduce 1. Create and bind a Hadoop service to our app. To do that, click Add Service. 2. Choose BigInsights-MapReduce 3. To add the service to your application, choose your application from the dropdown menu: 4. From the command line, restart your app cf restart [YOURAPP]

6 STEP 3. VCAP_SERVICES variable Once the service is bound to the application BlueMix UI allows you to look at the vcap_services variable, which contains the information regarding the services that bound to the application. 3a. Fetch Rest API Navigate to the VCAP_SERVICES variable. The application extracts VCAP_SERVICES variable 3b. Fetch Cluster URL We make a curl request with the cluster id to get back the BigInsights IP address curl -s -H 'api-key: $PASSWORD' (Note that the values for $PASSWORD, $USERID and $CLUSTERID are included in the VCAP_SERVICES.)

7 STEP 4. Log into BigInsights UI In the above example, the BigInsights console is running at We will go to the BigInsights UI and look at the data we have pre-loaded and also look at the jobs that we will use in our application. 4a. Verify preloaded data

8 4b. Verify Hadoop Apps (Hive/Pig) STEP 5. Run application Now that we have the BigInsights cluster deployed and we have updated the application, we can redeploy it on BlueMix and run a dynamic hadoop job. App architecture: The application is built with HTML5 and two opensource javascript visualization libraries (D3.js and JQPlot), and it is deployed onbluemix.

9 5a. Walk through the UI This is the Nasdaq application UI, on the left side we have few different visualization of the Nasdaq data. First few links show you visualization of data that has already been analyzed. These are a few examples of how an analyst can visualize the analyzed data. First one shows you the 30-year trading volume of all Nasdaq stocks. Next one shows you the Trading volume by volatility and dividends.

10 The third one shows the Day Trading candidates. 5b. Run dynamic Hadoop job From the Recession Analysis page, you can submit a dynamic Hadoop job. You can select any 10 stocks and run a job to analyze the volatility of these stocks during the recession years. When you submit, it will call the 2 Hadoop applications and analyze the data and return the analyzed data. Once the data is analyzed we use a heat graph to display the results. Brighter color means higher volatility. Conclusion To summarize, we have shown how you can use the BigInsights MapReduce service provided by BlueMix to quickly develop application that analyze big data. We walked through publishing our application, binding it to BlueMix MapReduce service and then running Hadoop jobs on BigInsights from our application to analyze Nasdaq data. We then visualize the analyzed data using D3 graphs.