BlueMix Hands-On Workshop Lab E - Using the Blu Big SQL application uemix MapReduce Service to build an IBM Version : 3.00 Last modification date : 05/ /11/2014 Owner : IBM Ecosystem Development
Table of Contents Lab E: Using the BlueMix MapReduce Service to build an IBM Big SQL application... 5 1. Creating a BlueMix MapReduce Service instance...5 2. Uploading the social media data to the BlueMix MapReduce Service instance...5 3. Importing the data into BigSheets...7 4. Exporting the workbook data...8 5. Launch Eclipse to bring up the sample Big SQL application...9 6. Define the BlueMix Server...10 7. Push the BigInsights sample application from Eclipse...11 8. Running the application...13 9. Validating the data import using the BigSQL console...16 Copyright IBM Corporation 2013-2014. All rights reserved. 3
Lab E: Using the BlueMix MapReduce Service to build an IBM Big SQL application Lab Objectives: This lab will guide you to understand and use the sample application that is provided for the BlueMix MapReduce Service. You will learn how to work with the BlueMix MapReduce cloud offering. As a Hadoop developer, you can get started quickly by using the BlueMix MapReduce Service (that is powered by IBM InfoSphere BigInsights ) to create big data applications. There are many advantages of developing this type of application on BlueMix. You can avoid the overhead of acquiring and provisioning your own hardware cluster. You can create an application by combining multiple BlueMix services You can scale yourhardware environment as your application needs increase. You will use InfoSphere BigInsights BigSheets component that comes with BlueMix Map reduce Service to learn how to explore social media data. This data is a look at the global coverage of the IBM Watson brand. Then, you will learn how to query the data by using IBM Big SQL, which is the InfoSphere BigInsights SQL interface. You will then deploy a BlueMix Java application that reads the data. The result of this exercise is an application that you create for your users to dynamically run queries on that data. Lab Duration : 30 minutes 1. Creating a BlueMix MapReduce Service instance In this section you'll create the instance of the BlueMix Map Reduce Service that will be used throughout teh rest of this lab. Step 1 Step 2 Step 3 Step 4 Step 5 Step 6 Step 7 Step 8 In your browser go to the BlueMix URL https://ace.ng.bluemex.net and login if necessary Make sure you're in the Dashboard tab (if not click on the Dashboard link at the top of the page to take you there) Scroll down to the Services section and click on Add a service Scroll down to the Big Data category and click on the icon for the MapReduce service Click on CREATE AND ADD TO APP For Add to: select [Do Not Associate] in the pulldown. Leave the default name. Click Create Your new service should now appear in the Services section 2. Uploading the social media data to the BlueMix MapReduce Service instance In this section you'll upload social media data about Watson to the Service that you just created instance of the BlueMix Map Reduce Step 1 In the Services section of your BlueMix Dashboard click on the hexagon for the service you just created to see more information about it Copyright IBM Corporation 2013-2014. All rights reserved. 5
Figure 1 MapReduce service icon Step 2 You see the URL and the login information that you need to open the services underlying InfoSphere BigInsights Console. Step 3 Step 4 Write down the Username assigned to this instance (note: it is case sensitive). You'll need it later on in the lab. Click on Launch the Console to bring up the console in another tab Figure 2 Launching the console Step 5 Step 6 Step 7 From the console click the Files tab In the DFS navigator, expand the user directory and select the directory with the same user name that was in the service details Click the Upload icon. Copyright IBM Corporation 2013-2014. All rights reserved. 6
Figure 3 Upload icon Step 8 Click Browse and select the file \BlueMixLabs\LabE\blogs-data.txt where \BlueMixLabs is the root folder of the files provided to you by the instructor. Click OK. 3. Importing the data into BigSheets Now that you have the sample data uploaded to HDFS, you can import the data into BigSheets and create a workbook that contains that data. Step 1 From the console click the BigSheets tab and click New workbook. Figure 4 Create BigSheets Workbook Step 2 Step 3 Name the workbookwatsonblogdata. In the distributed file system browser, select the file blogs-data.txt that you uploaded in the previous step. In the preview pane on the right, select a new reader to map the data into a spreadsheet format. (Currently the reader is Line Reader.) Click the edit icon next to 'Line Reader' and select JSON Array from the drop-down list. Figure 5 Changing the reader Step 4 Click the green check mark to change the reader. Click Fit Columns in the preview pane to Copyright IBM Corporation 2013-2014. All rights reserved. 7
make the tabular data appear more compact. Figure 6 Fit Columns to width Step 5 Step 6 Step 7 Step 8 Click the green check at the bottom to save the workbook. From the BigSheets workbook page, click Build new workbook to create a workbook with the same data. Because you do not need all of the columns to complete your design, drop some of the columns. Click the arrow in any column heading, and select Organize columns Click the x next to the following columns to remove them: Crawled Inserted IsAdult PostSize Step 9 Step 10 Step 11 Click the check at the bottom to confirm removal. Click Save. Then, click Save and Exit,. Change the name of the workbook towatsonblogsdatav2 and click Save. Click Run to run the workbook. Wait for the run to complete. 4. Exporting the workbook data In this section you ll export the data from your updated workbook so that it can be loaded into BigSQL later and then accessed using standard SQL queries Step 1 In the menu bar of the WatsonBlogDataV2 workbook, click Export data. Figure 7 Export Data menu item Step 2 Step 3 Step 4 In the drop-down window, select the TSV type in the Format Type field Specify Export to File. Click Browse to select a destination directory in the HDFS file system. Select your path /user/<username and then type the name for the new file,watsonblogs Click OK. Copyright IBM Corporation 2013-2014. All rights reserved. 8
Figure 8 Naming the exported file Step 5 Make sure that the Include Headers check box is cleared, because you want to export the data only. Click OK. A message dialog shows that the workbook is successfully exported, Click OK. Figure 9 Don't include headers 5. Launch Eclipse to bring up the sample Big SQL application In this section you ll start Eclipse and load the workspace that contains the sample Big SQL application you'll use to interact with your BlueMix MapReduce service Step 1 Step 2 Open Windows Explorer and launch the program \BlueMixLabs\eclipse\eclipse.exe where \BlueMixLabs is the root folder of the files provided to you by the instructor. Select the workspace \BlueMixLabs\workspaces\LabE when prompted and click OK Copyright IBM Corporation 2013-2014. All rights reserved. 9
Step 3 The sample BigInsightsSample should appear in the Project Explorer Figure 10 BigInsightsSample project in Eclipse Step 4 Feel free to explore the code. BigSQLJdbcClient.java is the interface to Big SQL.BigInsightsServlet.java handles all the URL requests 6. Define the BlueMix Server In this section you'll define the BlueMix server so you can automatically push your Eclipse code out to BlueMix when needed Step 1 Click on the Servers tab at the bottom of the Eclipse UI as shown in Figure 24. Figure 11 Eclipse Servers tab Step 2 Click the link labeled...click this link to create a new server. Step 3 Expand the IBM category and select IBM BlueMix. Click Next. Copyright IBM Corporation 2013-2014. All rights reserved. 10
Figure 12 Select IBM BlueMix as server type Step 4 In the next dialog enter your BlueMix credentials and click on Validate Account Figure 13 Validate your BlueMix account Step 5 Verify that the dialog reports that your account is valid and then click Finish. Step 6 Your server definition should now appear in the Servers tab Figure 14 Your server definition 7. Push the BigInsights sample application from Eclipse In this section push the sample appication to BlueMix, bind it to your existing MapReduce service and start it from within Eclipse. Step 1 In Eclipse go to the Servers tab and select your server, right click and select Add and Remove from the context menu Copyright IBM Corporation 2013-2014. All rights reserved. 11
Figure 15 Add project to server Step 2 Select the BigInsightsSample application and click Add. Figure 16 Add project to BlueMix Step 3 Step 4 Click Finish. Click Next in the resulting dialog Append a unique string to the Subdomain (eg your first initial + last name ) since this has to be unique systemwide. See Figure 17 for an example. Click Next Figure 17 Specify unique Subdomain for app Step 5 Select your previously create MapReduce service. Click Next Copyright IBM Corporation 2013-2014. All rights reserved. 12
Figure 18 Add existing service Step 6 Step 7 Click Finish to push and start your application. Messages should start appearing on the console. Wait for the message that says The server defaultserver is ready to run a smarter planet. Figure 19 Console messages 8. Running the application Now that the application has been deployed and started, you can run it from inside Eclipse Step 1 From the Eclipse menu select Window>Web Browser and then select one of your installed Web browsers (eg Firefox) Copyright IBM Corporation 2013-2014. All rights reserved. 13
Figure 20 Select an external browser Step 2 In the Project Explorer select your project, right click and then select Run As->Run on Server from the context menu. Figure 21 Launch application Step 3 Step 4 Click on Finish in the resulting dialog..the browser you selected in Step 1 should launch with the application The application will load the file you exported in BigSheets into BigSQL. You have to provide the name of the file and the name of the new table in BigSQL that will host it's contents. Type in Copyright IBM Corporation 2013-2014. All rights reserved. 14
WatsonData.tsv as the TSV File andmy_demotable as the BigSQL table name Figure 22 Exported Data File and BigSQL Table Name Step 5 Clock Load. Verify that the returned message indicates a successful load. Note: if it fails contact your instructor Figure 23 Successful Load Step 6 Click Fetch data and verify that the first 5 rows of data are returned from BigSQL Copyright IBM Corporation 2013-2014. All rights reserved. 15
Lab E: Build a BI Sample Application using MapReduce Service (Big Data) in BlueMix Figure 24 Successful fetch 9. Validating the data import using the BigSQL console In this section you ll the data import performed by the application by looking directly at the BigSQL console Step 1 In your browser go to the BlueMix URL https://ace.ng.bluemex.net and login if necessary Step 2 Make sure you're in the Dashboard tab (if not click on the Dashboard link at the top of the page to take you there). Step 3 In the Services section of your BlueMix Dashboard click on the hexagon for the Map Reduce service you've been using in this lab Step 4 You see the URL and the login information that you need to open the services underlying InfoSphere BigInsights Console. Step 5 Click on Launch the Console to bring up the console in another tab Step 6 In the console on the right side of the screen under Quick Links click on Run Big SQL Queries Figure 25 Launch Big SQL console Copyright IBM Corporation 2013-2014. All rights reserved. 16
Step 7 Step 8 Step 9 Step 10 You'll need your BigSQL database name to proceed.go to tab in your browser where you have your BlueMix environment running. Go to the Dashboard area and click on the square icon for you BigInsightsSample application Under App Runtime click on the round icon for the Liberty Runtime. Scroll down to Environment Variables and note the BigSQL database name as shown in Figure 26. Figure 26 Big SQL database name Step 11 Step 12 Step 13 Go back to the Big SQL console and enter the following command (replacing [db name] with the actual database name you noted in the previous step. use [db_name] Click Run and verify that you get a successful status message. Figure 27 Status message after use command Step 14 Replace the use command with the following command and click Run again select * from my_demotable Step 15 Verify that the data you loaded via the applications is returned Copyright IBM Corporation 2013-2014. All rights reserved. 17
Lab E: Build a BI Sample Application using MapReduce Service (Big Data) in BlueMix Figure 28 Query results Congratulations! You ve successfully completed Lab E where you explored the BlueMix MapReduce service that is powered by IBM BigInsights Copyright IBM Corporation 2013-2014. All rights reserved. 18