Instructor : Dr. Sunnie Chung. Independent Study Spring Pentaho. 1 P a g e

Size: px

Start display at page:

Download "Instructor : Dr. Sunnie Chung. Independent Study Spring Pentaho. 1 P a g e"

Leo Kennedy
5 years ago
Views:

ABSTRACT Pentaho Business Analytics from different data source, Analytics from csv/sql,create Star Schema Fact & Dimension Tables, kettle transformation for

1 ABSTRACT Pentaho Business Analytics from different data source, Analytics from csv/sql,create Star Schema Fact & Dimension Tables, kettle transformation for big data integration, MongoDB kettle Transformation, HBase kettle Transformation Instructor : Dr. Sunnie Chung Independent Study Spring Pentaho 1 P a g e

2 Table of Contents Installation Pentaho Business Analytics... 3 Analytics from Default data source... 8 CSV data source Population 10 Analytics - Population SQL Data Source - Population Analytics - Population Database Pentaho Architecture Big Data Kettle Transformation Data integration Kettle Transformation MongoDB & JSON Data Research on Kettle Transformation HBase References P a g e

Installation Pentaho Business Analytics Download Pentaho: It s a 30 days trial. http://www.pentaho.

3 Installation Pentaho Business Analytics Download Pentaho: It s a 30 days trial. For analytics download pentaho Business Analytics How to install? 3 P a g e

4 4 P a g e

5 5 P a g e

6 6 P a g e

7 7 P a g e

8 Different Analytics from Default data source. Country State wise Analytics 8 P a g e

9 9 P a g e

Country, State, City population year wise from 1990 to 2014.

10 Country, State, City population year wise from 1990 to Create Population Data source from csv file (8 MB) To get analytics login as Business User. Create another data source for this csv file. For that login as admin and create a new data source. 10 P a g e

11 Browse: Preview: 11 P a g e

12 Next will prompt all fields with datatype : Data Source is created for the new analytics. 12 P a g e

13 Data Analytics from Population data source: Now as a user we need to login as Business User as below. 13 P a g e

14 Create New - >Analysis Report Select Data Source. for i.e. Population is our data source 14 P a g e

15 Click OK. Suppose we want the analytics for county,year wise population. In original data there are multiple jurisdiction for same county and we want to aggregate it such that it will show us one value for each year and county instead multiple. Pentaho will do that automatically. We 15 P a g e

16 just need to give measures and rows as below by drag the field from the left columns to the layout area. 1) Drag County We can compare those county which are aggregated. 16 P a g e

17 2) For each county we want population.suppose for 2014, so drag that field into the measures underneath the Measures. 3) Switch to the chart format. 4) Comparison population 2014 vs P a g e

18 Washington has the largest population for both years. 5) Sunburst chart for the same data. 6) Pie chart 18 P a g e

19 7) Area chart : Sometime we need more aggregation over the data.so here we can do that using filter data. 19 P a g e

20 Aggregation with Condition 8) POP 2014 is greater than for each county Result : 20 P a g e

21 So we can see there are only 4 county having population greater than 14,000,00. 9) The county having population between and P a g e

22 Equivalent table form for above graph is: SQL Data Source There could be so many data sources in pentaho like H2,Generic Database, Hadoop Hive, Hypersonic,Impala, MS-Sql server, MonetDB, PostgreSQL etc. For Example SQL Server Population county wide jurisdiction wise From SQL Table: 1) From single table: -> No separate Fact Table & Dimension Table. 22 P a g e

23 23 P a g e

24 24 P a g e

25 25 P a g e

26 Analytics from SQL Data Source: Generate Report by New fact-dimension data source recently created : 26 P a g e

27 27 P a g e

28 Same way there could be other data sources from Hadoop platform too. 28 P a g e

29 Possible Pentaho Architecture: 29 P a g e

The Pentaho Stack consists of 4 elements: Presentation Layer The data in Pentaho can be viewed either from a browser, portal, office, and e-mail or web services.

30 The Pentaho Stack consists of 4 elements: Presentation Layer The data in Pentaho can be viewed either from a browser, portal, office, and or web services. The data available in the presentation layer is through reporting, analysis, dashboards and process management. Business Intelligence Platform It basically talks about security and repository. Data and Application Integration It is the integration layer of ETL. Third Party Applications Here the source database can be anything Various Layers in Pentaho Architecture 30 P a g e

Data Layer It can be used to connect to any data sources. Server Layer it is the middle layer in Pentaho BI Architecture. The application runs on server and serves as middle layer.

31 Data Layer It can be used to connect to any data sources. Server Layer it is the middle layer in Pentaho BI Architecture. The application runs on server and serves as middle layer. They are BA Server, Pentaho User Console and Pentaho Data Integration Server. We can deploy our report and dashboards and make it available to the end user. User console is for the user role mapping, security and for configuring which report should be shown to which user. The Pentaho Data Integration server runs the jobs and transformations. Client Layer There are two types mainly: Thin Client Runs on server. Examples include, Community Dashboard Editor and Pentaho Analyzer. Thick Client Runs as a Standalone. Examples include, Schema workbench, Report Designer and Data Integration. The aggregation designer is for the purpose of having consolidated data for reporting. The metadata editor is for mapping physical database and business logic. Report designer is for designing reports. 31 P a g e

Some useful contents from following Some research on white papers : http://www.slideshare.

32 Some useful contents from following Some research on white papers : 1) Pentaho Stack 32 P a g e

33 2) Pentaho ETL 33 P a g e

34 34 P a g e

35 35 P a g e

36 36 P a g e

37 37 P a g e

38 3) Data Exploration Using Pentaho 38 P a g e

39 39 P a g e

40 40 P a g e

41 4) Exploring Big Data with pentaho 41 P a g e

42 42 P a g e

43 43 P a g e

44 44 P a g e

45 Big data kettle Transformation :- Data Integration Install Data integration Linux platform Major steps: 45 P a g e

46 46 P a g e

47 47 P a g e

48 48 P a g e

49 49 P a g e

50 50 P a g e

51 51 P a g e

52 To MongoDB Transformation - From Text file 1) Take a text file input 52 P a g e

53 53 P a g e

54 54 P a g e

55 55 P a g e

56 Preview of total fields 56 P a g e

57 Add MongoDB Output in current Transformtion : 57 P a g e

58 58 P a g e

59 59 P a g e

60 Run the transformation 60 P a g e

61 61 P a g e

62 finish transformation. the database in mongo is created by this transformation with data. 62 P a g e

63 2) Same way I did for population csv (15 MB) around rows Because I want to generate from population data. 63 P a g e

64 3) For Report Design I want to blend this population database into small only required amount of data so I will create another transformation. 64 P a g e

65 Give input as population database. Now I want only those records having state name is Alabama. Preview: 68 rows out of rows. 65 P a g e

66 Give this output to another JSON source for report: 66 P a g e

67 I want only few field to compare for the report. 67 P a g e

68 save as 68 P a g e

Now I will use this saved transformation as a data source to my

Same way I cannot just transform the data but can also do merging of

69 Now I will use this saved transformation as a data source to my report and generate report. Same way I cannot just transform the data but can also do merging of more than one sources to single source and use that for my report generation. I can do the same for HBase, Hive, and Cassandra and so on. The following screen shows how large aggregation and merging would be applied to the transformation. 69 P a g e

Few Research for Hbase : According to pentaho documentation, I found there are several prerequisites that I need to install to work with Hbase. Key requirement is Pentaho Data Integration tool.

Steps to Establish Hbase Data Source Reporting on HBase Data Reporting on HDFS File Data Reporting on Data in Hadoop How to create a report that sources data from HBase.

70 Few Research for Hbase : According to pentaho documentation, I found there are several prerequisites that I need to install to work with Hbase. Key requirement is Pentaho Data Integration tool. So, I will switch a little to Data Integration. Following are the things which are in more detail stepwise. Steps to Establish Hbase Data Source Reporting on HBase Data Reporting on HDFS File Data Reporting on Data in Hadoop How to create a report that sources data from HBase. In order to report on HBase data in a parameterized fashion, you will need to perform the following steps: Create a PDI Transformation that sources the picklist of distinct IP Address to be used for a report parameter Create a PDI Transformation that sources a set of weblog data for a selected IP Address. This will be the primary data source for the report Create a Report that uses the PDI transformations for parameter list and report data. Prerequisites In order follow along with this how-to guide you will need the following: Hadoop 70 P a g e

71 Pentaho Data Integration HBase Pentaho Report Designer Loading Data into HBase guide completed Sample Files There are no sample files for this guide. The Loading Data into HBase Guide must be completed prior to starting this guide as it loads the sample HBase table data. Step-By-Step Instructions Setup Start Hadoop if it is not already running. Start HBase if it is not already running. Create a HBase Parameter Picklist PDI Transformation In this task you will create a PDI transformation to get a list of distinct IP Addresses from HBase. This transformation will later be used as the data source for a report parameter. 1. Start PDI on your desktop. Once it is running choose 'File' -> 'New' -> 'Transformation' from the menu system or click on the 'New file' icon on the toolbar and choose the 'Transformation' option. 2. Add a HBase Input Step: You are going to read data from a HBase table, so expand the 'Big Data' section of the Design palette and drag a 'HBase Input' node onto the transformation canvas. Your transformation should look like: 3. Edit the HBase Input Step: Double-click on the 'HBase Input' node to edit its properties. Do the following: 71 P a g e

a. Zookeeper host(s) and Zookeeper port: Enter your Zookeeper connection information. For local single node clusters use 'localhost' and port '2181'. b.

72 a. Zookeeper host(s) and Zookeeper port: Enter your Zookeeper connection information. For local single node clusters use 'localhost' and port '2181'. b. Create a Key Only Mapping: Switch to the 'Create/Edit mappings' tab and enter the following. i. HBase table name: Select 'weblogs' ii. Mapping name: Enter 'key_only' iii. Alias: Enter 'key' iv. Key: Select 'Y' v. Type: Select 'String' vi. Click Save Mapping When you are done your window should look like this: c. Configure Query: Switch to the 'Configure Query' tab and do the following. i. HBase table name: Click the 'Get mapped table names' button, then select 'weblogs'. ii. iii. Mapping name: Click the 'Get mappings for the specified table' button, then select 'key_only'. Click the 'Get Key/Fields Info' button. When you are done your window should look like: 72 P a g e

73 Click 'OK' to close the window. 4. Add a Split Fields Step: You need to split the key field currently formatted as client_ip year into two separate fields, so expand the 'Transform' section of the Design palette and drag a 'Split Fields' node onto the transformation canvas. Your transformation 73 P a g e

74 should look like: 5. Connect the Input and Split Fields steps: Hover the mouse over the 'HBase Input' node and a tooltip will appear. Click on the output connector (the green arrow pointing to the right) and drag a connector arrow to the 'Split Fields' node. Your canvas should look like this: 6. Edit the Split Fields Step: Double-click on the 'Split Fields' node to edit its properties. Do the following: a. Field to split: Select 'key'. 74 P a g e

75 b. Delimiter: Enter ' ' c. Fields: New field Type Client_ip String Year Integer d. When you are done your window should look like: Click 'OK' to close the window. 7. Add a Sort Rows Step: You need to sort the HBase data, so expand the 'Transform' section of the Design palette and drag a 'Sort rows' node onto the transformation canvas. 75 P a g e

Your transformation should look like: 8.

to the 'Sort rows' node. Your canvas should look like this: 9.

76 Your transformation should look like: 8. Connect the Split Fields and Sort steps: Hover the mouse over the 'Split Fields' node and a tooltip will appear. Click on the output connector (the green arrow pointing to the right) and drag a connector arrow to the 'Sort rows' node. Your canvas should look like this: 9. Edit the Sort Step: Double-click on the 'Sort rows' node to edit its properties. Enter this information: a. Check 'Only pass unique rows? (verifies keys only)' b. Fields: Add 'client_ip' sorted in ascending order. When you are done your window should look like this: 76 P a g e

drag a 'Dummy (do nothing)' node onto the transformation canvas. Your transformation should look like: 11.

77 Click 'OK' to close the window. 10. Add a Dummy Step: You need a component for the report to select it's data from, so expand the 'Flow' section of the Design palette and drag a 'Dummy (do nothing)' node onto the transformation canvas. Your transformation should look like: 11. Connect the Sort and Dummy steps: Hover the mouse over the 'Sort rows' node and a tooltip will appear. Click on the output connector (the green arrow pointing to the right) and drag a connector arrow to the 'Dummy (do nothing)' node. Your canvas should look like 77 P a g e

Save the transformation as 'hbase_ip_list.ktr' into a folder of your choice.

78 this: 12. Edit the Dummy Step: Double-click on the 'Dummy (do nothing)' node to edit its properties. Set the Step name to 'Output'. When you are done your window should look like: Click 'OK' to close the window. 13. Save the Transformation: Choose 'File' -> 'Save as...' from the menu system. Save the transformation as 'hbase_ip_list.ktr' into a folder of your choice. Create a HBase Query PDI Transformation In this task you will create a PDI transformation to get the pageviews per month for a particular IP address from HBase. This transformation will be used as the primary data source for the report. 1. Start PDI on your desktop. Once it is running choose 'File' -> 'New' -> 'Transformation' from the menu system or click on the 'New file' icon on the toolbar and choose the 'Transformation' option. 78 P a g e

2. Add a Parameter: Right click on the transformation and select 'Transformation settings'.

Default Value: '127.0.0.1' When you are done your window should look like: Click 'OK' to close the window.

of the Design palette and drag a 'HBase Input' node onto the transformation canvas.

79 2. Add a Parameter: Right click on the transformation and select 'Transformation settings'. Then do the following: a. Switch to the 'Parameters' tab. b. Parameter: 'paramipaddress' c. Default Value: ' ' When you are done your window should look like: Click 'OK' to close the window. 3. Add a HBase Input Step: You are going to read data from a HBase table, so expand the 'Big Data' section of the Design palette and drag a 'HBase Input' node onto the transformation canvas. Your transformation should look like: 4. Edit the HBase Input Step: Double-click on the 'HBase Input' node to edit its properties. Do the following: 79 P a g e

d. Click the 'Get Key/Fields Info' button. When you are done your window should look like: Click 'OK' to close the window. 5.

80 a. Zookeeper host(s) and Zookeeper port: Enter your Zookeeper connection information. For local single node clusters use 'localhost' and port '2181'. b. HBase table name: Click the 'Get mapped table names' button, then select 'weblogs'. c. Mapping name: Click the 'Get mappings for the specified table' button, then select 'pageviews'. d. Click the 'Get Key/Fields Info' button. When you are done your window should look like: Click 'OK' to close the window. 5. Add a Split Fields Step: You need to split the key field currently formatted as client_ip year into two separate fields, so expand the 'Transform' section of the Design palette and drag a 'Split Fields' node onto the transformation canvas. Your transformation 80 P a g e

81 should look like: 6. Connect the Input and Split Fields steps: Hover the mouse over the 'HBase Input' node and a tooltip will appear. Click on the output connector (the green arrow pointing to the right) and drag a connector arrow to the 'Split Fields' node. Your canvas should look like this: 7. Edit the Split Fields Step: Double-click on the 'Split Fields' node to edit its properties. Do the following: a. Field to split: Select 'key'. b. Delimiter: Enter ' ' 81 P a g e

82 c. Fields: New field Type Client_ip String Year Integer d. When you are done your window should look like: Click 'OK' to close the window. 8. Add a Get Variables Step: You need to add the parameter you created earlier to your stream, so expand the 'Job' section of the Design palette and drag a 'Get Variables' node 82 P a g e

onto the transformation canvas. Your transformation should look like: 9.

'Get Variables' node. Your canvas should look like this: 10.

83 onto the transformation canvas. Your transformation should look like: 9. Connect the Split Fields and Get Variables steps: Hover the mouse over the 'Split Fields' node and a tooltip will appear. Click on the output connector (the green arrow pointing to the right) and drag a connector arrow to the 'Get Variables' node. Your canvas should look like this: 10. Edit the Get Variables Step: Double-click on the 'Get Variables' node to edit its properties. Enter this information: a. Name: Enter 'selectedip' b. Variable: Enter '${paramipaddress}' c. Type: Select 'String'. When you are done your window should look like this: 83 P a g e

of the Design palette and drag a 'Filter rows' node onto the transformation canvas. Your transformation should look like: 12.

Click on the output connector (the green arrow pointing to the right) and drag a connector arrow to the 'Filter rows' node.

84 Click 'OK' to close the window. 11. Add a Filter rows Step: You want to filter for only the rows that match the selected IP Address, so expand the 'Flow' section of the Design palette and drag a 'Filter rows' node onto the transformation canvas. Your transformation should look like: 12. Connect the Get Variables and Filter Rows steps: Hover the mouse over the 'Get Variables' node and a tooltip will appear. Click on the output connector (the green arrow pointing to the right) and drag a connector arrow to the 'Filter rows' node. Your canvas should look like this: 13. Edit the Filter Rows Step: Double-click on the 'Filter Rows' node to edit its properties. Do the following: a. Click the <field> box to the left of the = and select 'client_ip' 84 P a g e

b. Click the <field> box to the right of the = and select 'selectedip' When you are done your window should

Add a Sort Rows Step: You want the rows in sorted order by year, so expand the 'Transform' section of the

Your transformation should look like: 15.

85 b. Click the <field> box to the right of the = and select 'selectedip' When you are done your window should look like this: Click 'OK' to close the window. 14. Add a Sort Rows Step: You want the rows in sorted order by year, so expand the 'Transform' section of the Design palette and drag a 'Sort rows' node onto the transformation canvas. Your transformation should look like: 15. Connect the Filter Rows and Sort Rows steps: Hover the mouse over the 'Filter rows' node and a tooltip will appear. Click on the output connector (the green arrow pointing to the right) and drag a connector arrow to the 'Sort rows' node. When you release the mouse 85 P a g e

and a tooltip appears select the 'Result is TRUE' option.

Edit the Sort Rows Step: Double-click on the 'Sort rows' node to

like this: Click 'OK' to close the window. 17.

86 and a tooltip appears select the 'Result is TRUE' option. Your canvas should look like this: 16. Edit the Sort Rows Step: Double-click on the 'Sort rows' node to edit its properties. Do the following: a. Fieldname: Enter 'year' b. Ascending: Select 'Y' When you are done your window should look like this: Click 'OK' to close the window. 17. Add two Dummy Steps: Expand the 'Flow' section of the Design palette and drag a 'Dummy (do nothing)' node onto the transformation canvas. Repeat to add a second 86 P a g e

'Dummy (do nothing)' step. Your transformation should look like: 18.

Click on the output connector (the green arrow pointing to the right) and drag a connector arrow to the first 'Dummy (do nothing)' node.

Edit the Dummy Step: Double-click on the 'Dummy (do nothing)' node you just connected to edit its properties. Change the Step name to 'Output'.

87 'Dummy (do nothing)' step. Your transformation should look like: 18. Connect the Sort Rows and Dummy steps: Hover the mouse over the 'Sort rows' node and a tooltip will appear. Click on the output connector (the green arrow pointing to the right) and drag a connector arrow to the first 'Dummy (do nothing)' node. Your canvas should look like this: 19. Edit the Dummy Step: Double-click on the 'Dummy (do nothing)' node you just connected to edit its properties. Change the Step name to 'Output'. 20. Connect the Filter Rows and Dummy steps: Hover the mouse over the 'Filter rows' node and a tooltip will appear. Click on the output connector (the green arrow pointing to the right) and drag a connector arrow to the second 'Dummy (do nothing)' node. When you release the mouse and a tooltip appears select the 'Result is FALSE' option. Your canvas 87 P a g e

should look like this: 21. Save the Transformation: Choose 'File' -> 'Save as...' from the menu system. Save the transformation as 'hbase_to_report.ktr' into a folder of your choice.

88 should look like this: 21. Save the Transformation: Choose 'File' -> 'Save as...' from the menu system. Save the transformation as 'hbase_to_report.ktr' into a folder of your choice. Create a Report In this task you will create a report that uses PDI transformations that extract data from HBase as the source. 1. Start Report Designer on your desktop. Once it is running choose 'File' -> 'Report Wizard' -> from the menu system. 2. Select a Template: Report Wizard will automatically layout and do some basic formatting of your report, so select a template of your choice in the select box. When you are done your screen should look like: 88 P a g e

Click 'Next' to go to the next screen. 3.

Choose Type 'Pentaho Data Integration' b.

89 Click 'Next' to go to the next screen. 3. Create a Data Source: You need to create the Hive query to select data for this report, so click the plus button and do the following: a. Choose Type 'Pentaho Data Integration' b. Create a New Query: Click the button to create a new query and do the following: 89 P a g e

Create a Second Query: Click the button to create a new query and do the following: i. Name: Enter 'IP List' ii. iii. File: Select the hbase_ip_list.

90 i. Name: Enter 'Page Views' ii. iii. File: Select the hbase_to_report.ktr transformation you just created. Steps: Select 'Output' When you are done your window should look like: c. Create a Second Query: Click the button to create a new query and do the following: i. Name: Enter 'IP List' ii. iii. File: Select the hbase_ip_list.ktr transformation you just created. Steps: Select 'Output' When you are done your window should look like: Click 'OK' to close the window. d. Highlight the 'Page Views' query in the Report Design Wizard window. Your Report Design Wizard window should now look like: 90 P a g e

do the following: a. Selected Items: Add the following in this order i. Year ii. Jan iii.

91 Click 'Next' to go to the next screen. 4. Set up Report Layout: You need to tell Report Design Wizard how to lay out your report, so do the following: a. Selected Items: Add the following in this order i. Year ii. Jan iii. Feb iv. Mar v. Apr vi. May vii. Jun viii. Jul ix. Aug x. Sep xi. Oct xii. Nov xiii. Dec When you are done your window should look like: 91 P a g e

select 'Data' -> 'Add Parameter' and do the following: a. Select the 'Pentaho Data Integration' data source. b. Name: Enter 'paramipaddress'. c.

92 Click 'Finish' to complete to the design wizard. 5. Create a Parameter: You need to create a parameter so you will be able to select a specific IP address when running the report, so in the menu system select 'Data' -> 'Add Parameter' and do the following: a. Select the 'Pentaho Data Integration' data source. b. Name: Enter 'paramipaddress'. c. Label: Enter 'IP Address' d. Value Type: Select 'String' e. Check 'Mandatory' f. Display Type: 'Drop Down' g. Query: Select 'IP List' h. Value: Select 'client_ip' i. Display Name: Select 'client_ip' When you are done your window should look like: 92 P a g e

Views query, so in the 'Data' pane expand 'Pentaho Data Integration', right click on

Click on the 'Edit Parameter' button. b. Under 'Transformation Parameter' click the.

93 Click 'OK' to close the window. 6. Add Parameter to Query: You need to add the parameter you just created to your Page Views query, so in the 'Data' pane expand 'Pentaho Data Integration', right click on 'Page Views' and select 'Edit Query'. Then do the following: a. Click on the 'Edit Parameter' button. b. Under 'Transformation Parameter' click the. c. DataRow Column: Select 'paramipaddress' d. Transformation Parameter: Select 'paramipaddress' When you are done your window should look like: 93 P a g e

Save the report as 'hbase_report.prpt' into a folder of your choice. 8.

menu system. A 'Print Preview' window will open.

94 Click 'OK' to close the window. 7. Save the Report: Choose 'File' -> 'Save as...' from the menu system. Save the report as 'hbase_report.prpt' into a folder of your choice. 8. Preview the Report: Choose 'File' -> 'Preview' -> 'Print Preview' from the menu system. A 'Print Preview' window will open. Select an 'IP Address' of your choice from the drop down. After a few seconds the report results will appear: Do any desired formatting to the report. 94 P a g e

95 References P a g e

Pentaho 30 for 30 QUICK START EVALUTATION. Aakash Shah

Pentaho 30 for 30 QUICK START EVALUTATION Aakash Shah You just downloaded a comprehensive data integration and business analytics platform and you have 30 days to evaluate Pentaho for your company This