A New Approach of Extraction Transformation Loading Using Pipelining

Transcription

1 A New Approach of Extraction Transformation Loading Using Pipelining Dr. Rajender Singh Chhillar* (Professor, CS Department, M.D.U) Barjesh Kochar (Head(MCA,IT),GNIM) Abstract Companies have lots of valuable data which they need for the future use. They have their data in different formats lying on the various heterogeneous systems. To accumulate data at one place to make useful and strategic decisions from a data warehouse they need data to be in a uniform format. To solve this problem, companies use extract, transform and load (ETL) software, which includes extracting data from its source, cleaning it up and transform it into desired database formant and load it into the various data marts for further use. Such software s take enormous time for the purpose. In this research paper we just try to define a new ETL model which speeds up the ETL process from the other models which already exist. Keywords Data Warehouse, Data Mart, pipelining, nested pipelining. Introduction In today s real time business, world demand to process the large data volumes at the lightning speed. In this fast changing competitive environment data volumes are increasing exponentially and increasing demand on the data ware house to deliver instantly available information. Additionally, data ware house should generate the consistent accurate result. This data came from many sources like Mainframe, ERP applications, CRM tools, an excel spread sheets, a flat file etc. All type of data coming from these sources must be transformed into single source which can be stored in the large repository called data warehouse. ETL is the process that involves:- 1.1 Extract The first part of an ETL process is to extract the data from various source systems. Most data warehousing projects consolidate data from different source systems. These systems may also use a different data organization / format. Common data source formats are relational databases and flat files, but may include nonrelational database structures such as IMS or other data structures such as VSAM or ISAM. In extraction process there are two operations are performed: a. Data profiling. b. Data cleaning. In data profiling we examine the data that we require is existing in the data source and collect the information about that data. After this, the concept of data cleaning came into existence in which we perform the following tasks: a. Filling up the missing values. b. Smooth noisy data. c. Identify or remove outliners. d. Resolve in consistency. e. Resolve redundancy cause by integration. 1.2 Transformation The transform phase applies a set of instructions or functions to the extracted data so as to convert different data formats into single format which can be loaded. Following are the transformations types ehich may be required: a. Selecting only certain columns to load (Which is needed in decision or selecting null columns not to load) b. Translating coded values (e.g., if the source system stores M for male and F for female, but the warehouse stores 1 for male and 2 for female) 1

2 c. Encoding free-form values (e.g., mapping "Male", M " and "Mr" onto 1) d. Deriving a new calculated value (e.g., Age=CurrentDate-DOB) e. Joining together data from multiple sources (e.g., lookup, merge, etc.) f. Summarizing multiple rows of data (e.g., total sales for each Month) g. Generating surrogate key values h. Transposing or pivoting (turning multiple columns into multiple rows or vice versa) i. Splitting a column into multiple columns (e.g., putting a commaseparated list specified as a string in one column as individual values in different columns) In transformation phase we divide it into following parts: a. Smoothing (remove noisy data) b. Aggregation( summarization, data cube construction) c. Generalization( hierarchy climbing) d. Attribute construction(new attribute constructed from the given ones) 1.3 Load The load phase loads the data into the data warehouse, depending on the requirements of the organization, this process ranges widely. Some old information may be replaced with the new changes done in the data to load in the warehouse. This phase directly interact with the database,the constraints and the triggers as well, which helps in improving the over all quality of data and the ETL performance. 2. Challenges This process have various hurdles or we can say that while doing this process various challenges can be there,dealing with which may be cause of worry. Following are the some of the challenges: a. Source systems are very diverse and disparate. b. Need to deal with source systems with multiple platforms and different operating systems. c. Many source systems are older legacy applications running obsolete data base technologies. d. Historical data change values are not preserved in source operations. e. This process takes enormous time. Another development in ETL software is the implementation of parallel processing. This has enabled a number of methods to improve overall performance of ETL processes when dealing with large volumes of data. There are 3 main types of parallelisms as implemented in ETL applications: a. Data: By splitting a single sequential file into smaller data files to provide parallel access. b. Pipeline: Allowing the simultaneous running of several components on the same data stream.. c. Component: The simultaneous running of multiple processes on different data streams in the same job. All three types of parallelism are usually combined in a single job.[2][3] ETL Tools ETL process is the one whose software can be maintained in any languages. There are the various tools which are developed by many software companies to make this process more reliable and advanced. Following are the various tools which are there in the market: a. Informatica(Informatica Corporation) b. Oracle Warehouse Builder(Oracle Corporation) c. Transformation manager(etl Solutions) d. DB2 Universal Enterprise Edition(IBM) e. Cognos Decision Stream(IBM Cognos) f. SQL server integration system(microsoft) g. Ab Initio(Ab Initio Software Corporation) h. Data Stage(IBM)[5] Problem with ETL design? Time for ETL process: The major problem with the earlier ETL model is the 2

3 time taken by the process. Many models came in market to reduce this time span. Previously we have discussed what is ETL process and the challenges face by this process. Every design/model came in the market with the main aim of reducing time taken to extract, transform and load into warehouse. Solution: A new design introduced with the concept of nested pipelining To perform Extraction transformation and loading for Data warehouse, we are using Piplining. Pipelining is technique where the complete process is being divided into various segments and all these segments can work simultaneously. In our model we had divided our ETL process into 9 segments as follows Segment1 Fmv which will fill missing values Segment2 Smd which will Smooth noisy data. Segment3 Ro which will Remove outliners. Segment4 Rc which will Remove Inconsistency. Segment5 Sm Smoothing. Segment6 Agg Aggregation. Segment7 AC Attribute Construction Segment8 Gen Generalization Segment9 Load which will load the data Data is divided into different modules. During the first clockcycle Module1 enters into segment1 and during 2 nd clockcycle first module moves to second segment and module2 enters into segment1. During 3 rd clockcycle module1 enters into segment3, module2 enters into segment2 and module3 enters into segment1. So in this way all segments work together on a no of data modules simultaneously as shown in the diagram. 3

4 Clockpulse Seg1 Seg2 Seg3 Seg4 Seg5 Seg6 Seg7 Seg8 Seg9 Fmv Smd Ro Rc Sm Agg AttConst Gen Load 1 M1 2 M2 M1 3 M3 M2 M1 4 M4 M3 M2 M1 5 M5 M4 M3 M2 M1 6 M6 M5 M4 M3 M2 M1 7 M7 M6 M5 M4 M3 M2 M1 8 M8 M7 M6 M5 M4 M3 M2 M1 9 M9 M8 M7 M6 M5 M4 M3 M2 M1 10 M9 M8 M7 M6 M5 M4 M3 M2 11 M9 M8 M7 M6 M5 M4 M3 12 M9 M8 M7 M6 M5 M4 13 M9 M8 M7 M6 M5 14 M9 M8 M7 M6 15 M9 M8 M7 16 M9 M8 17 M9 If we perform ETL process in this way then we will be able to save a lot of clock cycles and will be able to speed up the process. The following calculation will show you the speeding. Speeding up ETL process of n. Under this condition, the speedup becomes S= tn/t p If we assume the time takes to process a task is same in the pipeline and non pipeline circuits, we will have t n = kt p. It includes, this assumption, the speed reduces to S=N*t n (k+n-1)t p Where, N = no. Of tasks t n= Time to execute a task in case of NonPipeline k = no. of segments. Time to execute a task in case of Pipeline t p = As the no of tasks increases, n became much larger than k-1, and k+n-1 approches the value S = kt p /t p We will do it with the help of an example, let the time it takes to process a sub operation in each segment be equal to t p = 20ns. Assume that the, pipeline has k= 9 segments and execute n= 100 tasks in sequence. The pipeline system will take (k+n-1)t p = (3+99)*20 = 2040 ns to complete the task. Assuming that t n = kt p = 9*20 = 180 ns, a non pipeline system requires nkt p = 100*180 = ns to complete 100 tasks. The speed ratio is equal to 18000/2040 = 8.82.So we had speed up the process aproximate 9 times. Conclusion: Extraction Transformation and loading process may involve a number of other things also but in this paper we had shown that if (assuming) ETL process 4

5 has only these 9 task to do then we can speed up the whole process to great extent using pipelining.this model can have as many no. of segments as you want for ETL process. Future Scope: In future such type of more models can revolutionize the Information management. These type of models helps in reducing the time frame which helps the company to save lots of money and time. References: [1] Floriana Esposito, A Comparative analysis of methods for pruning Decision trees, IEEE Transaction on pattern analysis and pattern matching vol 19, [2] [3] [4] J.A. Blakeley, N. Coburn, and P.-A. Larson. Updating derived relations: etecting irrelevant and autonomously computable updates. ACM Transactions on Database Systems, 14(3):369{400, September 1989 [5]P. Vassiliadis, C. Quix, Y. Vassiliou, M. Jarke. Data Warehouse Process Management.cInformation Systems, vol. 26, no.3, pp , June 2001 [6]T. Stöhr, R. Müller, E. Rahm. An Integrative and Uniform Model for Metadata Management in Data Warehousing Environments. In Proceedings of the International Workshop on Design and Management of Data Warehouses (DMDW 99), pp , Heidelberg, Germany, [7]Inmon, W.H., Building the Data Warehouse. John Wiley, 1992 [8] M. Jarke, M. Lenzerini, Y. Vassiliou, P. Vassiliadis (eds.). Fundamentals of Data Warehouses. 2nd Edition, Springer-Verlag, Germany, 2003 [9] [10] 5