DWH Performance Tuning For Better Reporting

Transcription

1 DWH Performance Tuning For Better Sandeep Bhargava Reearch Scholar Naveen Hemrajani Aociate Profeor Dineh Goyal Aociate Profeor Subhah Gander IT Profeional ABSTRACT: The concept of data warehoue deal in huge amount of data and lot of analytically querie run on DWH, which cover bae data in term of thouand of gigabyte, to unveil the hidden pattern of buine. So repone time of query i exponential proportional (metaphorically) to involved bae data. So we can ay THUMB RULE a "MORE BASE DATE MORE ACCURATE RESULTS". But it will degrade the performance if not taken care properly. Alo we, a human, hate to wait due to natural phenomenon encoded in our DNA. Lot of work ha been done by many literate around the globe on DWH performance tuning by propoing many framework related with variou focu data quality, Metadata management etc... In thi paper, an effort ha been made to dicu about the real indutry problem and how we improve the performance of data warehoue by minimize the exiting CPU cycle wiely uing metadata driven approach. Keyword DWH- data warehoue, BI- Buine Intelligence 1. INTRODUCTION There i lot of hardware computation power i being involved to make DWH align with requirement of buine. The current growth of emi conductor indutry and world-cla competition made computational power relatively cheaper but till hardware cot ha a ignificant contribution on cot etimation / infra implementation. Performance of DWH can be conidered at below mentioned tage. Fetch data from ource ytem Data proceing through Layer Feeding data in to DWH Time involved in Fetching data from DWH for reporting Here our focu will be more on the report repone time a dicued above and the propoed olution will help to make that better. So for better reporting many factor are there which can contribute in poitive and negative ene depending upon how factor ha been conidered or implemented? 1.1 Factor Affecting Layer Repone Time Tool Selection Phyical Deign Normalized v. de-normalized Relational v. dimenional Hybrid frequency Concurrent Uer Amount of data Indexing Statitic All above-mentioned factor can impact reporting repone time but in thi paper we will be explaining more on how to keep tatitic up to date with help of available metadata for better reporting?[1] 1.2 What i Statitic? In DWH tatic are miletone directive that help databae optimizer / databae engine to come up with THE COST EFFECTIVE execution plan. So what information i there in tatitic? In tatitic below information can be tored in DWH, which will make ure pre availability of required data to come up with effective execution plan. For example: Number of null Number of Unique value Average row correction per value change Number of interval Number of row updated. Mode Frequency etc With thi available precompiled information optimizer / databae engine will ue thi information rather than calculating or etimating the ame at run time which will increae the repone time unnecearily. How tatitic can be defined &do we need to define every Time? Statitic can be defined by developer / DBA / Architect to make ure mooth execution of querie on the box. It can be defined on Column or combination of column level and No, Statitic hould be defined only one time and thi i a phyical attribute of a databae, which can be defined on a column or combination of column but it require a frequent refreh on the bai of data changing frequency and amount of change thi ha to be refrehed on a level[5]. Where thi information will be aved & how thi will be ued? Sytem will tore thi information in ytem and depending upon the databae oftware ued it can be tring field or even a clob or blob object and when query will be 1

2 ubmitted to databae then optimizer / databae engine will ue thi pre compiled information from ytem for cot effective execution plan.[7] 1.3 What i Metadata? Metadata, a very confuing term in word of data warehoue.mot individual with ome level of involvement with data warehoue, from a technical or buine perpective know of the term meta data. When aked to define what i meta data mot of thee individual can reiterate the common definition Data about data [6] So What i data about data? Why thi i o important? How thi piece play it role? Suppoe in a databae oftware like oracle, ql erver etc.. we have 30 databae and each databae contain 100 and lot many column. Now every column ha it aociated attribute but how databae will know about that below. What i the length of a particular column? What all are with Databae A? So to anwer thee quetion not only databae but every ingle available tool which ha concept of backend databae ha it own area dedicated to that application or databae only for technical pecification of the attribute and in phyical term thi area called a y_dba in oracle, dbc in teradata and o on. I It Neceary? No, having tatitic i not neceary but depending on the filter and join condition in querie it will help to avoid full can and over conumption of CPU cycle.[2] But if tatitic are there on the databae then it i eential to have it up to date otherwie it will provide fale information to optimizer / databae engine which will turn a a feed in to bad execution plan. e.g. if tatitic i not up to date? Static ay there are 10, 00,000 record in a. But actually it ha increaed by 10 million o thi type of inconitency will lead in to cot and CPU conuming execution plan. 2. TRADITIONAL METHODOLOGY FOR STATISTICS REFRESH. At many ite, to addre thi refreh proce, they have a weekly / fort nightly / monthly job to run on box which will imply go and refreh all available tatitic on the box regardle of the functional data load frequency.[3] On thi traditional olution we did ome analyi and below are ome more decription for the ame. 2.1 What wa done in invetigation? During analyi we tarted to plot the variou graph and trend to identify the nature of watage. And we figured out that abence of intelligence in weekly tat collection proce i cauing thi watage in CPU cycle. Evident Trait 1) Every time tat were getting refrehed on databae level 2) No eparation for tatic data 3) No pecial conideration on heavily floating fact 4) Require a large chuck of CPU once in a week, reduce other proce lice 5) No Aging information available 6) No auditing about wa, what i and what hould be? Source Sytem within DWH within DWH with In DWH reporting emantic layer Cutomer Facing Source1 dump only Buine Logic () Semantic Layer State weekly proce without any intelligence DATA WAREHOUSE FLOW WITHOUT STATS INTEGRATION Fig. 1 Available Traditional Methodology 2

3 What alerted u for olution? Vilferdo Pareto, an Itilian economit, oberved 80% of the land in Italy wa owned by 20% of the population; he developed the principle by oberving that 20% of the pea pod in hi garden contained 80% of the pea. Later on ame tudy wa generalized by Joeph M. Juran, buine management conultant, and named after Vilferdo Pareto a Pareto Principle Pareto Principle tate that 80% effect come from 20% caue. If we allow the ame extenion in BI life cycle then we will ee that more then 20 % CPU cycle were conumed by approx 80% ytem management tak and if we were running low on CPU cycle then aving every bit will be an add on to pocket, which indirectly convert in aving. So baically what we ave i alo a kind of earning, quoted by Warren Edward Buffett, an American invetor. On the imilar line when we tart analyzing ytem then we found that out of 100 % CPU cycle aigned to ytem management, approx 80 % were getting conumed in tat management. Upon further invetigation we found that out of aigned 80%, approximately more then 50% wa kind of *watage becaue of unneceary repetition of proce even though it i not required.[4]. International Journal of Computer Application ( ) 3. PROPOSED SOLUTION 3.1 Flow Chart Step 1: A thi proce i o tightly integrated with at level o to initiate with we will pa Databae name and name to the proce. Step 2: In thi Step proce will check the exitence of the databae object in databae to make ure thi doe exit in databae a a. If i there in databae then it will go on tep 3 otherwie will break the proce and come out. Step 3: After getting confirmation on exitence with help of tep 2, proce will make ure that ha data. If i populated then it will go on next level otherwie will break the proce and come out. Step 4: After getting confirmation on data population with help of tep 3, proce will make ure that tat are there on the. If tat are there then it will proceed to next tep otherwie will break the proce and come out. Step 5: Now a proce know with help of previou tep about tat o now quetion i to know weather data load i incremental or will be truncate and load. Now becaue thi cannot be identified uing metadata information o end uer a an input to the proce will pa thi information. If i truncate and load then proce will refreh the whole tat on the but if not then imply proceed to next tep. Step 6: In thi tep proce will make ure that data of bae ha been modified by how much percentage. Ideally if data change i more the 9% then go with a refreh otherwie leave it alone. Step 7: In thi tep proce will check the age of exiting tat and if it i more then 30 day then will refreh the tat otherwie imply come out of the loop. Fig. 2. Flow Chart of Propoed Solution 3

4 3.2 Proce Flow Model Source ytem within DWH within DWH with In DWH reporting emantic layer Cutomer Facing Source1 dump only STATS PROCESS Buin e Logic () STATS PROCESS Semantic Layer DATA WAREHOUSE FLOW WITH STATS INTEGRATION Fig. 3 Propoed Model Uing STATS 4. RESULTS The above mentioned graph i howing CPU utilization of four different uer for a application. Thee uer were reponible to operate on tage load, reporting load, reporting need etc. for a ample data of a firm M/ Adect Technologie, Jaipur for two conecutive month in year Thi proce wa implemented for uer 3 alone. And the trend of CPU utilization for uer 3 (green) i upporting the fact proce i working. Becaue in three and half pike it ha more requirement for CPU a compare to lat three and half pike. The above graph prove that the technique introduced in thi paper ha reduced the CPU utilization dratically and the ame may be implemented practically for further better uage of technology Cot Benefit Analyi Let u aume the cot of 1 CPU Cycle i 1 unit then in firt week cot occurred i approximately 4900 unit which continue till firt four week, thi reult i obtained without uing tat, while next four week the cot i reduced to almot le than half when STATS approach wa applied & new cot i jut 2400 unit Solution Piloting To verify the reult of thi approach pilot teting wa done on Teradata databae o terminology i inclined toward that databae oftware but thi framework can be generalized acro any platform. Fig. 4 Typical CPU conumption chart for an application in a DWH 4

5 5. CONCLUSION Today Data Ware Houing i the back bone of mot of MNC and large cale organization. But mot of them have not looked into databae adminitration cot epecially over hardware. The data entry & updation cot. The above work i an aim for optimizing the cot of data warehouing by tuning the performance over OLAP. Intead of updating all attribute and row of a for even a ingle value of a entity we have propoed to a methodology to reduce the effort on the machine ide. In thi effort i made by reducing the number of CPU Cycle to be ued for editing the value by uing STATS. 5.1 Intended Audience Thi paper involve very advanced concept of data ware houing. Alo while writing thi paper Tier 3 and Tier 4 population of IT indutry wa kept in mind a per below lit. Senior Developer Project Manager Technical Lead DBA Solution Architect 6. ACKNOWLEDGMENTS I would like to take thi opportunity to thank & expre my pecial gratitude to Mr Subhah Gander and M.Ruchi Dave (Program Coordinator) and alo all my friend and clamate for their upport and help me. Lat but not the leat I wih to expre my gratitude to god almighty for hi abundant bleing without which thi work would not have been ucceful. 7. REFERENCES [1] Data Warehoue Performance Management Technique, Andrew Hold worth, Oracle Service, Advanced Technologie, Data Warehouing Practice. 2/9/96 [2] Dicover Teradata Meta Data Service [3] Performance Tuning Mechanim for Data Warehoue: Query International Journal of Computer Application ( ) Volume 2 No.2, May 2010 [4] Teradata RDBMS performance optimization, NCR Corporation [5] Oracle Databae Performance Tuning Guide10g Releae(10.2)Part Number B [6] Metadata implementation with Ab Initio EME, Article by Mike Green on 13 May 2009 [7] le.pdf 5