Customizing Incremental ETL for Operational Reporting

Transcription

1 Customizing Incremental ETL for Operational Reporting for HP Project and Portfolio Management Center version 9.12 Content Pack 1 About this Document... 2 Operational Reporting ETL Architecture... 2 ETL Batch Job... 2 PPM Center Tables Published for Change Data Capture... 3 COMMON ETL Module... 6 TM ETL Module... 7 RM ETL Module... 8 FM ETL Module... 8 PM ETL Module... 9 Guidelines for Custom Code CDC Operations Adding PPM Center Tables for Incremental Change Creating a New ETL Module... 12

2 About this Document This paper provides information about how to customize the incremental ETL process for the Operational Reporting solution for PPM Center. It is written for PPM Center administrators, configurators, and DBAs. Readers are assumed to be knowledgeable about PPM Center and SAP BusinessObjects Enterprise. The Operational Reporting Administrator s Guide provides information about the incremental ETL job (PPM_ETL_BATCH_JOB) that covers the day-to-day updates for PPM Center data tables. This paper describes how you can bring data that are not part of the default ETL process into the Operational Reporting system from the PPM Center database. Operational Reporting ETL Architecture The incremental ETL tool (extract, transform and load for Operational Reporting in PPM Center is designed using Oracle s Change Data Capture (CDC) and job infrastructure. It uses synchronous CDC. Tables are published in the PPM Center database and subscribed in multiple modules in PPM Center databases. The five PPM Center modules that participate in the incremental ETL process for Operational Reporting are designed to handle specific tables in the PPM Center database based on functionality. For example, the TM module loads and converts data for tables related to HP Time Management. The five functional modules are: CM (Common) TM (Time Management) RM (Resource Management) FM (Financial Management) PM (Project Management) ETL Batch Job RPT_ETL_BATCH_JOB is a scheduled job that calls RPT_ETL_BATCH_JOB_CHAIN, which is illustrated in Figure 1. This ETL job chain has seven blocks, with one block for each of the five ETL modules, one for the ETL_START_PROGRAM, and one for ETL_END_PROGRAM. Incremental ETL for each functional module in PPM Center is called from each block of the ETL job chain. The Oracle job scheduler schedules the ETL job chain, which is illustrated in Figure 1. Each block calls a function from the RPT_ETL_JOB_UTIL package. In the event of a failure at any point along the chain, the function throws an exception and the process jumps to the ETL_END block, instead of the next block in the job chain. 2

3 Figure 1. ETL job chain PPM Center Tables Published for Change Data Capture The framework uses the Oracle publish and subscribe model to publish a list of PPM Center tables for change. Each module subscribes to a list of tables. (The subscription is created during installation or upgrade.) To load changes in the Operational Reporting database, each module calls the Oracle API using the subscription name, and then calls a module-specific API to process changes within that specific window or time frame. For example, the CM (common) module calls the RPT_INCREMENTAL_ETL.DO_COMMON_INCREMENTAL_ETL function. The RPT_ETL_JOB_UTIL.RUN_ETL function maintains the CDC window and also the transition COMMIT or ROLLBACK. 3

4 Figure 2. Each module executes generic code to copy the change capture data (CDC) from the PPM Center database CDC table to the Operational Reporting database CDC table using a database link. During Operational Reporting deployment or upgrade, this CDC table is created in the Operational Reporting database. It (?) executes another generic method to merge those changes in the PPM Center tables into the Operational Reporting database. After the changes are copied and merged, PPM Center tables in the Operational Reporting schema are the same as the tables in the PPM Center schema and each module executes module-specific fact calculation logic. (See Figure 3 below) Figure 3. 4

5 The following PPM Center tables are published for change data capture (CDC): FM_APPROVED_BUDGETS FM_FINANCIAL_LINE_CELLS FM_FINANCIAL_SUMMARY FM_FORECAST_ACTUALS FM_FORECAST_ACTUAL_LINES FM_USER_DATA ITG_ACTIVITIES_NLS ITG_AUDIT_EVENTS ITG_SETTINGS_ATTRIBUTES ITG_SETTINGS_CONTAINERS ITG_SETTINGS_CONTAINER_CHLDRN ITG_SETTINGS_CONT_ATTRS KCRT_CONTACTS KCRT_FG_MASTER_PROJ_REF KCRT_FG_PFM_ASSET KCRT_FG_PFM_PROJECT KCRT_FG_PFM_PROPOSAL KCRT_FG_PROJ_ISSUE KCRT_FG_PROJ_RISK KCRT_FG_PROJ_SCOPE_CHANGE KCRT_REQUESTS KCRT_REQUEST_DETAILS KCRT_REQUEST_TYPES_NLS KCRT_REQ_HEADER_DETAILS KCRT_STATUSES_NLS KCST_CURRENCIES_NLS KDLV_PACKAGES KDRV_CALENDARS KDRV_CALENDAR_EXCEPTIONS KDRV_CALENDAR_SETUPS KDRV_WORKING_DAYS KNTA_LOOKUPS_NLS KNTA_REGIONS_NLS KNTA_SECURITY_GROUPS_NLS KNTA_USERS KPMO_BUSINESS_OBJECTIVES_NLS KRSC_ORG_UNITS_NLS KRSC_ORG_UNIT_MEMBERS KTMG_CHARGE_CODES_NLS KTMG_PERIODS KTMG_PERIOD_TYPES_NLS KTMG_POLICIES KWFL_WORKFLOWS_NLS KWFL_WORKFLOW_INSTANCES 5

6 KWFL_WORKFLOW_INSTANCE_STEPS KWFL_WORKFLOW_STEPS_NLS PFM_LIFECYCLE_PARENT_ENTITY PGM_PROGRAMS PGM_PROGRAM_CONTENT PM_PROGRAM_BUSINESS_OBJECTIVES PM_PROJECTSPM_PROJECT_ROLLUP PM_PROJECT_TYPES PM_WORK_PLANS PPM_FISCAL_PERIODS_NLS RSC_POSITIONS RSC_POSITION_FORECAST RSC_RESOURCES RSC_RESOURCE_ASSIGNMENTS RSC_RESOURCE_POOLS RSC_RESOURCE_POOL_MANAGERS RSC_ROLES_NLS RSC_RP_DISTRIBUTION_ENTRIES RSC_STAFFING_PROFILES RSC_STAFF_PROF_ALLOCATION TM_ACTUALS TM_ACTUALS_EFFORT TM_LINE_CHARGE_CODES TM_TIME_SHEETS TM_TIME_SHEET_LINES WP_MILESTONES WP_TASKS WP_TASK_ACTUALS WP_TASK_COSTS WP_TASK_INFO WP_TASK_SCHEDULE WP_USER_DATA WP_WORK_UNITS The different modules subscribe to different tables to get the changes for their areas of functionality. COMMON ETL Module The CM module loads all common tables and transforms requests, fiscal periods, and custom parameter data to fact. The CM module subscribes to the following tables. (This list of tables is defined in the RPT_PPM_OBJECTS.COMMON_CDC_TABLES object.) KCRT_CONTACTS KCRT_FG_PFM_ASSET KCRT_FG_PFM_PROJECT KCRT_FG_PFM_PROPOSAL KCRT_REQUESTS KCRT_REQUEST_TYPES_NLS KCRT_REQ_HEADER_DETAILS 6

7 KCRT_REQUEST_DETAILS KCRT_STATUSES_NLS KCST_CURRENCIES_NLS KDRV_CALENDAR_SETUPS KDRV_CALENDARS KDRV_WORKING_DAYS KNTA_LOOKUPS_NLS KNTA_REGIONS_NLS KNTA_USERS KPMO_BUSINESS_OBJECTIVES_NLS KRSC_ORG_UNIT_MEMBERS KRSC_ORG_UNITS_NLS FM_FINANCIAL_SUMMARY RSC_STAFFING_PROFILES KWFL_WORKFLOW_INSTANCES KWFL_WORKFLOW_INSTANCE_STEPS KWFL_WORKFLOW_STEPS_NLS KWFL_WORKFLOWS_NLS PFM_LIFECYCLE_PARENT_ENTITY PGM_PROGRAM_CONTENT PGM_PROGRAMS PM_PROGRAM_BUSINESS_OBJECTIVES PM_PROJECTS PM_PROJECT_TYPES PM_PROJECT_ROLLUP PM_WORK_PLANS WP_TASKS WP_TASK_ACTUALS PPM_FISCAL_PERIODS_NLS RSC_RESOURCES RSC_RESOURCE_POOL_MANAGERS RSC_RESOURCE_POOLS RSC_ROLES_NLS RSC_RP_DISTRIBUTION_ENTRIES TM_TIME_SHEETS TM_TIME_SHEET_LINES Fact transformation logic for the Common ETL is defined in the following packages and is called from the RPT_INCREMENTAL_ETL.DO_COMMON_INCREMENTAL_ETL function: RPT_POPULATE_DIMENSION.RPT_POPULATE_DIM_FISC_PERIODS(); rpt_request_update_dimension.do_req_custom_params_incrm_etl('h'); rpt_request_update_dimension.do_req_custom_params_incrm_etl('d'); rpt_request_update_dimension.do_request_incremental_etl(start_year_boundry); TM ETL Module The TM module works with tables for HP Time Management. The TM module subscribes to the following tables. (This list is of tables is defined in the RPT_PPM_OBJECTS.TM_CDC_TABLES object.) 7

8 ITG_AUDIT_EVENTS KTMG_PERIODS KTMG_POLICIES TM_ACTUALS TM_ACTUALS_EFFORT TM_TIME_SHEETS TM_TIME_SHEET_LINES WP_TASKS WP_TASK_INFO TM_LINE_CHARGE_CODES KTMG_CHARGE_CODES_NLS KTMG_PERIOD_TYPES_NLS KDLV_PACKAGES ITG_ACTIVITIES_NLS Fact transformation logic for HP Time Management ETL is defined in the following packages/functions and is called from the RPT_INCREMENTAL_ETL.DO_TM_INCREMENTAL_ETL function: rpt_tm_refresh.rpt_tm_refresh_all RPT_RM_UPDATE_EFFORT_FACT.DO_INCREMENTAL_ACTUALS; RM ETL Module The RM module works with tables for HP Resource Management. The RM module subscribes to the following tables. (This list is of tables is defined in the RPT_PPM_OBJECTS.RM_CDC_TABLES object. RSC_POSITIONS RSC_RESOURCE_ASSIGNMENTS RSC_STAFF_PROF_ALLOCATION RSC_POSITION_FORECAST RSC_RESOURCE_POOLS RSC_STAFFING_PROFILES KNTA_REGIONS_NLS KDRV_WORKING_DAYS KDRV_CALENDAR_SETUPS RSC_RP_DISTRIBUTION_ENTRIES RSC_RESOURCES KNTA_USERS KDRV_CALENDAR_EXCEPTIONS KNTA_SECURITY_GROUPS_NLS Fact transformation logic for the HP Resource Management ETL is defined in the following packages/ functions and is called from the RPT_INCREMENTAL_ETL.DO_RM_INCREMENTAL_ETL function: rpt_rm_update_facts.do_incremental_capacity(start_year_boundry, end_year_boundry); rpt_rm_update_facts.do_incremental_demand(start_year_boundry, end_year_boundry); FM ETL Module The FM module works with tables for HP Financial Management. The FM module subscribes to the following tables. (This list is of tables is defined in the RPT_PPM_OBJECTS.FM_CDC_TABLES object.) PM_PROGRAM_BUSINESS_OBJECTIVES PM_PROJECTS 8

9 KPMO_BUSINESS_OBJECTIVES_NLS PFM_LIFECYCLE_PARENT_ENTITY KCRT_FG_PFM_PROPOSAL KCRT_FG_PFM_PROJECT KCRT_FG_PFM_ASSET KCST_CURRENCIES_NLS FM_FORECAST_ACTUAL_LINES FM_FINANCIAL_LINE_CELLS FM_APPROVED_BUDGETS FM_USER_DATA FM_FORECAST_ACTUALS FM_FINANCIAL_SUMMARY PGM_PROGRAM_CONTENT PGM_PROGRAMS KRSC_ORG_UNITS_NLS ITG_SETTINGS_ATTRIBUTES ITG_SETTINGS_CONT_ATTRS ITG_SETTINGS_CONTAINER_CHLDRN ITG_SETTINGS_CONTAINERS Fact transformation logic for the HP Financial Management ETL is defined in the frpt_fm_update_facts.do_fm_full_etl; function, and is called from the RPT_INCREMENTAL_ETL.DO_FM_INCREMENTAL_ETL function. PM ETL Module The PM module works with tables for HP Project Management. The PM module subscribes to the following tables. (This list is of tables is defined in the RPT_PPM_OBJECTS.PM_CDC_TABLES object.) KCRT_FG_PFM_PROJECT KCRT_FG_PROJ_ISSUE KCRT_FG_PROJ_RISK KCRT_FG_PROJ_SCOPE_CHANGE KCRT_FG_MASTER_PROJ_REF WP_TASK_SCHEDULE WP_TASK_ACTUALS WP_WORK_UNITS WP_TASK_COSTS WP_TASK_INFO WP_TASKS RSC_RESOURCES KNTA_USERS PM_WORK_PLANS PM_PROJECTS WP_MILESTONES WP_USER_DATA KNTA_LOOKUPS_NLS KCRT_REQ_HEADER_DETAILS KCRT_REQUEST_DETAILS 9

10 Fact transformation logic for the HP Project Management ETL is defined in following packages/ functions, and is called from the RPT_INCREMENTAL_ETL.DO_PM_INCREMENTAL_ETL function: RPT_PM_UPDATE_FACTS.update_task_dim; RPT_PM_UPDATE_FACTS.RPT_PM_UPDATE_TASK_COST_FCT; RPT_PM_UPDATE_FACTS.UPDATE_ASSIGNMENT_FACTS; RPT_PM_UPDATE_FACTS.update_aggr_effort_fact; RPT_PM_UPDATE_FACTS.UPDATE_PROJECT_MGR; Note: For a complete list of data tables and views created in PPM Center for the Operational Reporting schema, see the Data Model Guide. 10

11 Guidelines for Custom Code You can add custom code to the functions called during ETL. You can leverage custom code using the CDC infrastructure, logging mechanisms, scheduled job, and so on. The custom code that you add to functions must comply with the following restrictions: Place all custom code into a separate package. Custom code must not update/modify any existing fact transformation logic. Custom code must select the right PPM Center module. For example, if the code uses tables included in the common list, then it would be called from the common module. Custom code must not call any commit or rollback. Transaction is managed in the upper layer. Custom code must throw an exception for any failure. The upper layer can then break the job chain execution. Custom code must not use DBMS_OUTPUT.PUT_LINE for logging, but instead use the following rpt_event_util functions: Use rpt_event_util.log_begin_func_event to mark the start of the function. Use rpt_event_util.log_info_event to log debug information. Use rpt_event_util.log_end_func_event to mark the end of the function. Use rpt_event_util.log_error_event to log error information. Custom code can include events in the CDC table in its logic. A CDC table exists for each table in a module. For example, for the KNTA_USERS table, the CM module contains a KNTA_USERS_CDC_CM_W table for CDC changes. Similarly, the RM module contains a KNTA_USERS_CDC_RM_W table exists for the RM module. CDC Operations The following operation types occur in CDC: I (insert) UU (before update) UN (after update) D (delete) For any update in PPM Center, CDC adds UU(with old values) and UN (with new values). For any data created in PPM Center, CDC may add insert and multiple update events. However, in the Operational Reporting schema, the infrastructure only copies the last event, and if last event happens to be an update, then it brings in data of the UN operation type. For the RM module, CDC copies data of the UU and UN operation types. These CDC data are deleted before the next copy and merge are performed during next incremental ETL run. 11

12 Adding PPM Center Tables for Incremental Change Custom code can use additional PPM Center tables for incremental change. There is no straightforward way to implement it, but it can be done in different ways. The approach described here requires minimal change to existing code. Suppose, that custom code must bring in changes for the following portfolio hierarchy tables (for HP Portfolio Management): KPFM_PORTFOLIO_LIFE_CYCLE PFM_PORTFOLIOS PFM_PORTFOLIO_CONTENTS PFM_PORTFOLIO_MANAGERS This requires a new ETL module. Creating a New ETL Module In this approach, you build custom code using the API that the framework provides. Because this involves minimal interaction with the existing code base, the risk of overriding changes during an upgrade is very small. To create a new ETL module with the example codes provided: 1 Unzip the example codes package and, and open the rpt_ex_etl_objects.pls file in a text editor. 2 Update the following parameters: ETL_OBJECTS: Add one or more PPM Center data table names to participate in the ETL for the new module. ETL_CHANGE_SET_NAME: Add a change set name for Oracle Change Data Capture. All ETL_OBJECTS tables are to be published under this change set. ETL_SUBS_NAME: Add a subscription name for Oracle Change Data Capture. Changes in published tables can be used to this subscription. ETL_VIEW_PREFIX: Add a two-letter preview name for the Oracle Change Data Capture subscription. This is used to uniquely identify views in the subscription. 3 If any table name that you specified for the ETL_OBJECTS parameter is more than 15 characters long, open the latest rpt_constants.plb output file from your reporting schema, and then update the two functions below (see examples in rpt_constants.plb): GET_SHORTNAME: This function must return the shortened form (fewer than 15 characters) of the table name. GET_FULLNAME: This function must return the full form of the table name. 4 If any of the tables that you specified for the ETL_OBJECTS parameter are missing a primary key or have composite primary keys, you must open the latest rpt_sync_objects.plb file for your reporting schema and update the following function (see examples in rpt_sync_objects.plb): GET_PRIMARY_KEYS(table_name IN VARCHAR2): This function must return the primary key for the table. 5 Open the rpt_fct_ex_table.sql file and update the fact table structure. The rpt_fct_ex_table.sql file creates the RPT_FCT_EXTEND table. (You can either reuse the RPT_FCT_EXTEND table, or create a new fact table to use.) 6 Open rpt_ex_populate_facts.pls file and update logic for initial load data in populate_facts function. The populate_facts function does not contain any code, you may need to add code to populate the fact table. 12

13 7 Open rpt_ex_update_facts.pls file and update logic for incremental data changes in update_facts function. The update_facts function does not contain any code, you may need to add code to update fact tale with delta changes. To apply the above changes in PPM Center and Operational Reporting databases, execute the scripts described in the following table. Script update_ppm_schema.sql update_reporting_schema.sql run_ex_inc_etl.sql Description Top-level script. Run this script in PPM database using schema username and password. It requires two parameters: PPM Schema name, and Index table space name for PPM schema. This script publishes new tables, creates subscription, and index for newly created CDC tables. Top-level script. To apply changes in the Operational Reporting database and load initial data, run this script in the Operational Reporting database using the schema username and password. The script requires the following two parameters: PPM Center schema name Data tablespace name for the Operational Reporting database schema Run this script to bring incremental changes from PPM Center to the Operational Reporting database. The script copies and merges changes, and then executes code to calculate fact. 13

14 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. The only warranties for HP products and services are set forth in the express warranty statements accompanying such products and services. Nothing herein should be construed as constituting an additional warranty. HP shall not be liable for technical or editorial errors or omissions contained herein. Business Objects and the Business Objects logo, BusinessObjects, Web Intelligence, and other Business Objects products and services mentioned herein as well as their respective logos are trademarks or registered trademarks of Business Objects Software Ltd. Business Objects is an SAP company. Adobe is a trademark of Adobe Systems Incorporated. Oracle and Javaô are registered trademarks of Oracle Corporation and/or its affiliates. March 2012