Framework Components Our ETL parameter framework will include primarily two components.

Transcription

1 An ETL Parameter Framewrk t Deal with all srts f Parametrizatin Needs We spke abut different etl framewrks in ur prir articles. Here in this article let's talk abut an ETL framewrk t deal with parameters we nrmally use in different ETL jbs and different use cases. Using parameterizatin in the ETL cde increases cde reusability, cde maintainability and is critical t the quality f the cde and reduces the develpment cycle time. Framewrk Cmpnents Our ETL parameter framewrk will include primarily tw cmpnents. 1. A Relatinal Table: - T stre the parameter details and parameter values. 2. Reusable Mapplet: - Mapplet t lg the parameter details and values int the relatinal table. 1. Relatinal Table A relatin table will be used t stre the parameter details with the belw structure. This will stre the parameter name, value and the ther infrmatin relevant t identify the cntext f the parameter, like flder name, wrkflw name and sessin name. ETL_PARM_ID: A unique sequence number. FOLDER_NAME: Flder name, in which the parameter is used. WRKFLW_NAME: Wrkflw name, in which the parameter is used. SESSN_NAME: Sessin name, in which the parameter is used. PARM_NAME: Name f the parameter PARM_VAL: Value f the parameter. ETL_CRT_DATE: Recrd creates timestamp. ETL_UPD_DATE: Recrd update timestamp. Nte: Yu can add the repsitry name t the table, if the framewrk is planned t use fr wrkflws running in multiple repsitries. Nte: All parameters shuld be stred int the parameter table with its initial value t start with. 2. Reusable Mapplet An mapplet t capture and lad the parameter values int the database table. This mapplet takes tw input values and gives all the data elements required in the parameter table mentined abve.

2 Mapplet Input: Parameter name, parameter value. Mapplet Output: All the data elements required t be stred in the parameter table mentined abve. This utput can be cnnected t the target table t stre the infrmatin in the relatinal table. Framewrk Implementatin in a Wrkflw This framewrk can be implemented fr bth dynamically changing parameters as well as rarely changing r static parameters. Dynamically Changing Parameters Typical example f dynamically changing parameter is "ETL Run Timestamp" which is used fr incremental data extractin lgic. Lets see hw incremental data extractin is implemented using this parameter framewrk. Create a mapping variable with MAX aggregatin. This variable will hld the parameter value. Nte : Reset the mapping variable in the wrkflw using the pre-sessin variable assignment. Set the mapping variable using the SETVARIABLE functin in an expressin as shwn in belw image. This will update the mapping variable t the greatest ETL_UPD_DATE value, which will finally be stred int the parameter table using the mapplet.

3 Adjust the surce filter t pull incremental data. Incremental data is pulled frm the surce based n ETL_UPD_DATE as shwn in belw image. The abve mapping cnfiguratin will make sure the crrect parameter is used and will set the crrect parameter value, which is t be stred int the parameter table. Add an additinal mapping pipeline as shwn in belw image t stre the parameter value in the parameter table. This pipeline will update the current value in the parameter table t the latest value. The mapplet used will make sure the crrect parameter and parameter value is updated in the parameter table.

4 Nte : Set the target lad rder f the new pipeline t the last ne in the mapping. Surce qualifier f this pipeline will generate ne recrd using "select 'x' frm dual" SQL. Belw shwn is the cmplete mapping design. Static r Rarely Changing Parameters Parameters, which might need ccasinal changes r static parameter can be stred in the parameter table and can be retrieved in the Infrmatica mapping using a LkUp transfrmatin. Any changes require fr the parameter value shuld be ne time updated dne utside f the ETL prcess. Belw shwn is the lkup transfrmatin, which can be used t retrieve parameter value. Yu just need t pass in the input parameters t the lkup and get the parameter value frm the parameter table.

5 Nte: The static parameter value shuld already be saved in the parameter table with its static value, befre it can be used in a mapping. Hw Parameter Data is Stred in the Parameter table As discussed, the parameter framewrk supprt bth static and dynamic parameters. Let's cnsider a sample data fr the explanatin. ETL_PARM_ID FOLDER_NAME WRKFLW_NAME SESSN_NAME PARM_NAME PARM_VAL 1 ALL ALL ALL YR_BEGIN DW_SALES ALL ALL REGION_NAME USA 3 DW_SALES wf_load_cust_dim s_load_cust_dim LST_RUN_TS 01-JAN- 10-OCT Parameter IDs 1 and 2 are static parameters. First parameter is defined t use acrss all flders, wrkflw and sessins. Secnd parameter is still a static ne, but specific t all wrkflws and sessins in the flder DW_SALES. Third parameter is a dynamic parameter specific t the sessin s_load_cust_dim, which is running in DW_SALES flder.

6 Better than Infrmatica Parameters and Variables Since the parameter framewrk stres the values utside Infrmatica envirnment, yu get much mre flexibility with it. Prevents any accidentally parameter value changes, which might happen t mapping variables during cde migratin. Centralized strage fr all parameter values rather than the string it in different parameter files r mapping variables. Easy t update r change the parameter value, unlike it is with mapping variables. When using it with incremental data extractin lgic, it is t update the parameter value t reprcess same data set and enable restartability. Dynamic changing parameters can be handled in the framewrk. Mapping variables can have nly MAX r MIN peratins t handle dynamically changing parameters. Parameter framewrk can handle bth static and dynamic parameters. Mre secure than string the parameters in a parameter file.