Hadoop User Platform management and examples

Size: px

Start display at page:

Download "Hadoop User Platform management and examples"

Colin Carroll
5 years ago
Views:

1 P a g e 1

2 Revision History Version Date Prepared By Summary of Changes 1.0 July 19, 2017 Ray Cheung Initial release 1.1 Jul 24, 2018 Ray Cheung Minor change P a g e 2

3 Table of Contents 1. Introduction How to Login Mapreduce example: Wordcount Pig example: Pig test data Hive / Impala example: SQL P a g e 3

4 1. Introduction This document is shown a general information and guidelines on how to use the Hadoop environment with examples. It is also shown the login method and related information. The examples are included: a. MapReduce b. Pig c. Hive/Impala For more information on Pilot HPC Platform, please refer this section "High Performance Computing Support Services" ( ) in the ITS website. For creating your Pilot HPC Platform account, please complete the online form ( ). P a g e 4

5 2. How to Login Access via web browers. Login using netid and password as same as pilot HPC cluster. 3. Mapreduce example: Wordcount Click File Browser and create directory WordcountTest. P a g e 5

Go to directory /example/mapreduce select the file hadoop- mapreduce-examples-2.6.0- cdh5.8.2.jar Click Actions > Copy.

6 Go to directory /example/mapreduce select the file hadoop- mapreduce-examples cdh5.8.2.jar Click Actions > Copy. Select directory /user/{username}/wordcountt est/hadoop-mapreduceexamples cdh5.8.2.jar and click copy. P a g e 6

7 Query Editors > Job Designer. New action > Mapreduce. P a g e 7

8 Name: mapwordcount Description: mapworcount Jar path: /user/{username}/wordcountt est/hadoop-mapreduceexamples cdh5.8.2.jar. Update the properties with information shown in the table on the right side. mapreduce.input.fileinputformat.in putdir mapreduce.output.fileoutputforma t.outputdir mapreduce.job.map.class ${inputdir} ${outputdir} org.apache.hadoop.examples.word Count$TokenizerMapper P a g e 8

9 mapreduce.job.reduce.class org.apache.hadoop.examples.word Count$IntSumReducer mapreduce.job.combine.class org.apache.hadoop.examples.word Count$IntSumReducer mapreduce.job.output.key.class org.apache.hadoop.io.text mapreduce.job.output.value.class org.apache.hadoop.io.intwritable mapred.mapper.new-api true mapred.reducer.new-api true mapreduce.job.reduces 3 mapreduce.job.maps 6 mapreduce.cluster.mapmemory.m 1024 b mapreduce.cluster.reducememory mb mapreduce.job.queuename Queue1 Click Save and submit. InputDir:./WordcountTest/text.txt (under home directory) OutputDir:./WordcountTest/m apwordcountout1 (Create new directory by program) Get the result Click file brower Go to path /user/{username}/wordcountt est/mapwordcountout1 Click part-r / part-r-0001 / part-r to view the result P a g e 9

10 P a g e 10

command records = load './PigTestData/student.

11 4. Pig example: Pig test data Go to File Browser Create directory PigTestData. Copy student.txt file from /example/pig to directory /user/{username}/pigtestd ata Click Query editors > PIG Insert follow command records = load './PigTestData/student.txt' using PigStorage(',') as(classno:chararray, studno:chararray, score_chinese:int, score_math:int); dump records; store records into './PigTestData/student_out' using PigStorage(':'); P a g e 11

$view the result Go to File browser and click the dir /user/{username}/pigtestd ata/student_out P a g e$

12 Enter Script name and click Save Click Submit to run the test on the left panel of the Pig editor To view the result Go to File browser and click the dir /user/{username}/pigtestd ata/student_out P a g e 12

Click part-m-00000 to view the result. 5.

13 Click part-m to view the result. 5. Hive / Impala example: SQL Click Query Editors > Hive or Impala. Click Saved Queries and select one of sample queries. P a g e 13

14 Select one of the queue for execution. Click button on the left panel for execution Result will be shown. To export result Click the Get result icon. Then select one of the format to export the result. P a g e 14

15 Save SQL Click the save icon on the right top corner > Save as. Enter the query name and description then click save. P a g e 15

Oracle. Oracle Big Data 2017 Implementation Essentials. 1z Version: Demo. [ Total Questions: 10] Web:

Oracle. Oracle Big Data 2017 Implementation Essentials. 1z Version: Demo. [ Total Questions: 10] Web: Oracle 1z0-449 Oracle Big Data 2017 Implementation Essentials Version: Demo [ Total Questions: 10] Web: www.myexamcollection.com Email: support@myexamcollection.com IMPORTANT NOTICE Feedback We have developed