Best Practice: Optimizing the cube build process in SAS 9.2 Mary Simmons, SAS Institute, Cary, NC Michelle Wilkie, SAS Institute, Cary, NC

Transcription

1 Paper Best Practice: Optimizing the cube build prcess in SAS 9.2 Mary Simmns, SAS Institute, Cary, NC Michelle Wilkie, SAS Institute, Cary, NC ABSTRACT The lading and prcessing f yur data int a cube needs t be cntrlled and ptimized t get the best perfrmance ut f yur system and meet apprpriate time windw cnstraints that may be set by IT r the business user. The SAS OLAP Server 9.2 shws several perfrmance imprvements and changes in the cube building prcess cmpared t that in SAS This paper will intrduce factrs that affect the cube building prcess; ptins that are available t cntrl and ptimize the cube building prcess; best practices fr cube building; lgging ptins and understanding pssible prblems that can ccur, in relatin t these imprvements in SAS 9.2. INTRODUCTION SAS OLAP Server (On-Line Analytical Prcessing) prvides users with the ability t build and view cubes. Cubes allw users t gain knwledge f the data by slicing, dicing and analyzing it in a summarized, multidimensinal apprach. This helps surface interactins within the data s that users can make infrmed decisins abut their business. It is essential fr these decisins, therefre, t make sure that the data is served t the user in a timely and relevant fashin. The SAS OLAP Server supprts Multidimensinal OLAP (MOLAP), Relatinal OLAP (ROLAP) and Hybrid OLAP (HOLAP) structured cubes. A SAS MOLAP structure is a prprietary data stre, in which data is feed frm a warehuse int the cube building prcess; fr summarizatin, defining data relatinships via metadata, and cmputatin f analytical variables. T build such a cube is simple; hwever, it is imprtant t understand the cube building prcess s that: the apprpriate SAS System ptins and PROC OLAP ptins are utilized crrectly the system utilizatin f memry, I/O and strage is ptimized the cube can be built successfully and in the time windw necessary Getting the OLAP cube int the business users hands as fast as pssible is the ultimate gal f ptimizing the cube build prcess. The majr fcus f this paper will be n changes made in the SAS 9.2 cube build prcess; this will be illustrated using PROC OLAP. SAS 9.2 PROC OLAP In SAS 9.2, the underlying OLAP cube structure changed significantly, autmatically prviding several benefits including: Better perfrmance fr the cube building prcess Lifting the Hierarchy member limitatin Significantly faster member metadata creatin Updateable cubes (prviding incremental updates) Better query perfrmance allwing faster respnse times fr reprting applicatins surfacing OLAP data In additin t these benefits, it may be pssible t further ptimize yur cube build prcess. T d s, it is imprtant t understand the underlying cmpnents f the cube, the phases f the PROC OLAP prcess, the ptins that apply t each phase and t understand the needs f yur envirnment s that yu can make the mst f the resurces that are available t yu. 1

2 Cube Cmpnents and Build Phases In Figure 1, yu will see the OLAP cube brken dwn int its cmpnents. Cubes cntain a Metadata Registratin, navigatinal files that cntain internal cube member metadata (separate frm the Metadata Registratin), as well as the NWAY and aggregatin tables and indices. Figure 1 Cmpnents f an OLAP cube The cube build prcess can be brken dwn int 2 majr phases that are clsely related t the cube cmpnents. Phase I is when the data summarizatin begins and the navigatin files are created. In Phase II the NWAY, Aggregatin tables and related indices are built. Phase 1 Phase 2 PROC OLAP PHASE I Phase I cmprises the internal metadata preparatin which includes metadata summarizatin and srting f member infmatin, and the creatin f the internal cube metadata. Cube metadata includes infrmatin abut: Figure 2 Phases f the OLAP cube build prcess lcatin f the data the cube structure members and their relatinship with each ther fr navigatin purpses, frmats, member prperties calculated members 2

3 The first phase ccurs immediately upn executin and cmpletes just prir t the NWAY message relating t number f recrds. The lg shwn in Figure 3 is that f a cube build using a star schema input surce, the first phase is highlighted with the red bx. Ntes abut Empty captin, are indicatrs that ragged and/r unbalanced hierarchies exist in this cube. These messages ccur when members in levels are identified that match the EMPTY_NUM r EMPTY_CHAR ptin. In the example belw, the EMPTY_CHAR= ptin was used n the PROC OLAP statement. Figure 3 PROC OLAP lg with first phase f the cube build prcess identified PROC OLAP The secnd phase f build prcess is the creatin f the NWAY and aggregatin tables (and their related indices, if requested). The indices make data retrieval faster and mre efficient, chsing t nt build the indices will result in slwer query perfrmance and is nt recmmended. This secnd phase can be identified in lg shwn in Figure 4 with a red bx and always starts just prir t the nte Number f NWAY recrds and cncludes with a message which indicates the successful cube build. (See Lgging Sectin fr further granularity in identifying the start f Phase II) Figure 4 PROC OLAP lg with the secnd phase f the cube build prcess identified 3

4 SAS OPTIONS Belw are specific SAS ptins that help ptimize and cntrl the cube build prcess at each phase. Fr mre detail n each ptin please refer t the SAS 9.2 OLAP Server: Users Guide. Optin Type Descriptin Relevance MEMSIZE SYSTEM Specifies the limit n the ttal amunt f virtual memry that can be used by a SAS sessin. REALMEMSIZE SYSTEM Specifies the amunt f real (physical) memry SAS can expect t allcate. SUMSIZE SYSTEM Specifies the limit n the memry allcatin fr the Summarizatin Prcess SORTSIZE SYSTEM Specifies the limit n the memry allcatin fr the Srt Prcess UTILLOCn SYSTEM Specifies ne r mre file system lcatins in which applicatins can stre utility files. (WORK will be used if UTILLOC is nt set) SPDEUTILLOC SYSTEM Specifies ne r mre file system lcatins in which temprarily utility files can be written. (UTILLOC will be used if SPDEUTILLOC is nt set) INDEXSORTSIZE PROC OLAP INDEXSORTSIZE indicates the max amunt f memry per thread f executin yu want t make available during NWAY and aggregatin index creatin. MAXTHREADS PROC OLAP Specifies the maximum number f threads that are used t asynchrnusly create aggregatin indexes. Setting MAXTHREADS t high can lead t perfrmance degradatins. ASYNCINDEXLIMIT PROC OLAP The ASYNCINDEXLIMIT ptin is new t SAS 9.2. It gives the user the cntrl ver hw many indexes fr ne table can be built at a time. PHASE I and PHASE I and PHASE I and PHASE I and PHASE I and ADDNWAYINDEX PROC OLAP Ability t create NWAY indexes after the initial PROC OLAP cube build prcess, in which NOINDEX was used fr the NWAY. INDEX NOINDEX PROC OLAP Specifies whether r nt t create aggregatins with indexes. CONCURRENT PROC OLAP Specifies the maximum number f aggregatins t create in parallel. It des nt impact the NWAY which is always built first. COMPRESS PROC OLAP Cmpress the resulting NWAY and aggregatin tables, reducing the size f the cube. Table 1 Relevant Optins t cntrl cubes build prcess 4

5 RECOMMENDATIONS Input Data: A star schema is the ptimal data surce fr building a MOLAP cube. A star schema is cmprised f dimensin tables and a fact table which allws SAS t read and translate the infrmatin faster. Additinal features are available nly with a star schema such as multiple language supprt and mre flexibility during cube update. Avid where pssible using a data set view as an input data surce as this requires additinal system resurces fr prcessing the view and can cause cntentin with the resurces that are required fr the cube building prcess. Memry Optins: MEMSIZE: REALMEMSIZE: SUMSIZE: Memsize is an indicatr f the ttal memry yu want t allw fr this SAS prcess. Setting this value t 0 (r MAX) is useful fr debugging r testing purpses but highly discuraged in a prductin envirnment. Yu can use the fullstimer ptin in SAS cmbined with MEMSIZE=MAX t determine hw much yur cube build will actually need and then set yur MEMSIZE slightly larger fr yur prductin builds. If yu are building cubes cncurrently yu shuld adjust the MEMSIZE smaller t avid cntentin fr system resurces. REALMEMSIZE shuld be a percentage f MEMSIZE but this will be dependent n the size f yur data. Prper specificatin f SUMSIZE= can imprve prcedure perfrmance by restricting the swapping f memry that is cntrlled by the perating envirnment. Generally, the value f the SUMSIZE= system ptin shuld be less than the physical memry available t yur prcess. If the prcedure yu are using needs mre memry than yu specify, the system creates a temprary utility file. If the value f SUMSIZE is greater than the values f the MEMSIZE ptin and the REALMEMSIZE ptin, SAS uses the values f the MEMSIZE ptin and REALMEMSIZE ptin. Temprary wrk space: UTILLOC: During the summarizatin phase, if bth UTILLOC1 and UTILLOC2 are specified, they will each be used fr tw distinctly different functins f the summarizatin prcess. At a minimum bth UTILLOC1 and UTILLOC2 shuld be big enugh t hld the size f the input data (give a little mre than just the exact size f the input table fr each lcatin) and UTILLOC lcatins shuld be f equal size. Fr Phase II, SPDEUTILLOC (r UTILLOC) shuld be set t allw fr a separate lcatin fr the temprary utility files needed in this phase. It is strngly recmmended that ALL temprary wrkspace lcatins be lcated n physically separate disks frm that f the input data t avid disk cntentin. NWAY Creatin: ASYNCINDEXLIMIT: This is mst effective fr the NWAY table which will cntain indexes fr all hierarchies in the cube. The default behavir at SAS was unlimited. This meant that all indices were being created at ne time which has lead at times t thread cntentin when indexing very large tables (such as the NWAY). It is recmmended nt t use this functin when dealing with small data and lw cardinality as it will nt add t perfrmance and may have an adverse interactin with the cube build prcess. ADDNWAYINDEX: It was added fr cubes that cmpleted successfully but the NWAY index build failed. At SAS 9.2, if this ccurred yu wuld als want t cnsider bumping up yur MEMSIZE and using the ASYNCINDEXLIMIT ptin described abve. 5

6 General Aggregatin Index Creatin: INDEXSORTSIZE/MAXTHREADS: Typically MAXTHREADS shuld be set t a number less than the number f CPUs available. The default is calculated ff f the CPUCOUNT ptin which is an indicatr f the number f available CPUs (it is nt a hard and fast limit). On UNIX machines at SAS 9.2, CPUCOUNT defaults the min value f 4 and ACTUAL. Example: INDEXSORTSIZE=4096. If MAXTHREADS was set t 4, yu wuld be telling the subsystem that each thread has 1G f memry it can use. The INDEXSORTSIZE applies t a certain subset f the memry usage during index creatin. Additinal memry is needed fr ther functinality at that time s it is imprtant t keep that in mind when setting this ptin. PROBLEMS AND RESOLUTIONS Wrk Space (Utility) Prblems: a. If yu get a message mentining Utility write failed r disk full cndidtin check bth UTILLOC lcatins. If ne lcatin is full but the ther is nt, this is an indicatin that the size fr bth UTILLOC lcatins shuld be increased. b. If yu receive the errr Utility File read failed make sure that nlargefiles is nt set n the file systems as n file will be created greater than 2GB. This is specific t UNIX perating systems. Yu can validate if this ptin is set by checking /etc/mnttab. c. If ne f the UTILLOC lcatins has been set t small expect a system errr (nt a SAS errr) in PHASE I during the data summarizatin step ERROR: Cmplex utility space issue r yu may see the perating system CPU usage drp t ~1% and hang. Increase the UTILLOC lcatin as per the recmmendatin mentined abve in PHASE I Recmmendatins. PHASE I (data summarizatin) Prblems: a. Errrs with the terms classificatin, summarizatin r categry are usually indicatrs that SUMSIZE needs t be increased. (and / r MEMSIZE and REALMEMSIZE). Example: ERROR: Out f memry. ERROR: A prblem was encuntered during data summarizatin phase (8). NOTE: The initial memry limit fr classificatin tables was K bytes. Actual memry acquired was K bytes. b. If yu are encuntering a Critical memry shrtage r Memry exhausted message during the data summarizatin step MEMSIZE and REALMEMSIZE need t be adjusted. T make sure that yu can track memry usage during the summarizatin step use %let syssumtrace=3; Refer t the lgging sectin belw fr mre details. (NWAY and Aggregatin creatin) Prblems: a. If an errr ccurs after the message IN PROGRESS: NWAY index creatin and it cntains the keywrd index/es and NWAY aggregatin investigate using ASYNCINDEXLIMIT, INDEXSORTSIZE and tuning MEMSIZE t cntrl a successful NWAY creatin. b. If yu are adding aggregatins t an existing cube fr imprving query perfrmance and receive the belw errr messages r a message cntaining the wrds N memry can be allcated, index r aggregatin and aggregate ; resurce cntentin may be the issue. Cnsider the interactin f PROC OLAP ptins MAXTHREADS=, CONCURRENT= and INDEXSORTSIZE= and reduce memry utilizatin. ERROR: Insufficient memry ERROR: A failure ccurred as a result f a prblem in creating cube aggregatins ERROR: New aggregatins cannt be created fr the existing cube. All AGGREGATION statements will be ignred 6

7 INCREMENTAL UPDATE With the release f SAS 9.2 OLAP Server, yu can nw add new data t yur cubes withut rebuilding the entire cube via the incremental update prcess. An incremental update typically invlves adding cell data and members t an existing cube as shwn in Figure 5. Figure 5 Incremental Update Prcess fr an OLAP Cube and the additin f new data There are several decisins at cube creatin that can impact the cube build prcess fr incremental updates. These were discussed at length in Paper Avid Grwing Pains: New Cube Update Features Yu Shuld Knw Abut (Weinberger, Tierney). In the fllwing sectins, tw key decisins will be discussed and the impact these decisins can make n Phase I and Phase II f the cube build prcess: D all dimensins need updating (typically nly sme dimensins such as time will need updating)? What infrmatin is being updated (is it metadata infrmatin, actual data values r bth)? DO ALL DIMENSIONS NEED UPDATING? We wuld recmmend that yu take time, prir t the initial build f yur cube, t review yur cube s dimensins. Identify any dimensins that cntain fixed number f distinct values, these dimensins can be pre-laded and the dimensin can be marked NONUPDATEABLE. Fr example, a relatively static dimensin wuld be Gegraphy, while a dimensin such as time r custmer wuld require regular additins f new members. Benefits Smaller ftprint: The first is the smaller disk ftprint. All aggregatins that include this dimensin will take up less space when this ptin is used. NOTE: By default, dimensins are UPDATEABLE. Table 2 illustrates the effect f the NONUPDATEABLE ptin specified n the PROC OLAP statement, n the verall disk ftprint fr that cube. With NONUPDATEABLE ptin used n PROC OLAP statement instead f the DIMENSION statement all dimensins will be stred in the nn-updateable frmat. 7

8 Cube Name All dimensins UPDATEABLE (MB) All dimensins NON- UPDATEABLE (MB) Cube % Cube % Cube % Cube % % Smaller Cube % Table 2 Cube size cmparisns with NONUPDATEABLE ptin Update perfrmance bst: The secnd benefit t using the NONUPDATEABLE ptin is a perfrmance bst during cube update. Dimensins that are updateable will have their dimensin tables scanned fr every update. Nn-updateable dimensin tables will never be re-scanned fr ptential new members. WHAT INFORMATION IS BEING UPDATED? As mentined abve, a cube cmprises f metadata, navigatin files and aggregatins. Typically when updating a cube yu will either be: updating the navigatin files which cntain member infrmatin (UPDATE_DIMENSION ptin) r the aggregatins which cntain the summarized data values (ADD_DATA ptin) r bth. ADD_DATA The ADD_DATA ptin n the PROC OLAP statement enables yu t add new members and data t a cube. Yu can update a cube in-place r create generatins f the cube (see SAS 9.2 OLAP Server: Users Guide fr mre details). Here is an example f the ADD_DATA ptin used with PROC OLAP: PROC OLAP data=mylib.newdata cube=cubea add_data utcube=cubeb utschema=testschema; metasvr hst="myhst" prt=8561 repsitry=myrepsitry lap_schema=prdschema; run; The ADD_DATA ptin requires that either UPDATE_IN_PLACE, OUTCUBE=, r OUTSCHEMA= als be specified. UPDATE_DIMENSION The UPDATE_DIMENSION ptin n the DIMENSION statement gives yu the flexibility n the updateable dimensins t cntrl whether r nt the dimensin table is scanned. By default if the dimensin is updateable the table will be scanned during each update event (default: UPDATE_DIMENSION=MEMBERS) The values fr the UPDATE_DIMENSION statement are as fllws: MEMBERS: This means that the dimensin table currently assciated with the dimensin shuld be read and prcessed fr new members f every hierarchy in that dimensin. When specified n a star schema cube withut the PROC OLAP ADD_DATA ptin, the MEMBERS ptin prvides a way t update the cube with new members withut having t add data t the cube. MEMBERS_AND_PROPERTIES: This means that the dimensin table currently assciated with the dimensin shuld be read and prcessed fr new members PLUS member prperties fr existing members shuld be changed with the new values in 8

9 OFF: the dimensin table. When specified n a star schema cube withut the PROC OLAP ADD_DATA ptin, the MEMBERS_AND_PROPERTIES ptin prvides a way t update the cube with new members and prperties withut having t add data t the cube. This means that the dimensin table currently assciated with the dimensin shuld NOT be read at all. It applies nly t cubes laded frm a star schema and never t a cube laded frm a single detail table. If the ADD_DATA ptin is used n the PROC, and an updateable dimensin s table shuld nt be prcessed, then this ptin must be used t avid a table scan. If the ADD_DATA ptin is nt used n the PROC, then OFF is the DEFAULT ptin fr all dimensins in the cube. INCREMENTAL UPDATE - PHASE I Remember that the tw steps f PHASE I cube build prcess is metadata summarizatin and metadata creatin. In incremental update the prcess is similar metadata summarizatin f the new member infrmatin ccurs and then the existing member metadata is updated. There are instances, hwever, where neither f these steps is perfrmed when updating a cube. If the cube was riginally created using the NONUPDATEABLE ptin n the PROC OLAP statement then PHASE I will nt be needed. Als, PHASE I is nt perfrmed fr specific dimensins in an incremental update when the DIMENSION statement has either NONUPDATEABLE r UPDATE_DIMENSION=OFF ptin specified. INCREMENTAL UPDATE If the ptin ADD_DATA is specified then the new data is read. An NWAY and aggregatin tables are created fr that new data and is added as a slice f the cube as depicted in Figure 5. LOGGING OPTIONS There are many different lgging ptins available thrugh SAS, fr general lgging infrmatin refer t the Paper The SAS OLAP Server: Understanding and Slving Cmmn Prblems (Budlng). The fllwing table describes additinal lgging ptins that ur custmers have fund particularly useful in gaining a better understanding f the build prcess and diagnsing issues when they ccur. 9

10 Lgging Optin and Descriptin Usage Sample utput SYSSUMTRACE Turns n additinal lgging fr PHASE I f the PROC OLAP prcess. Shwing summarizatin memry usage. %let syssumtrace=3; IN PROGRESS: WRITE NWAY TO DISK (TSTLVL TIMING)Begin time fr writing nway t disk: 14H:45M:1S NOTE: The initial memry limit fr classificatin tables was K bytes. Actual memry acquired was 2368K bytes. NOTE: The utility file buffer size selected was 16K bytes. PROC OPTIONS (MEMORY) Get infrmatin abut MEMORY settings and hw they gt their values frm the cmmand line, shipped default, cnfig file etc. PROC OPTIONS grup=memory value; run; Optin Value Infrmatin Fr SAS Optin MEMSIZE Optin Value: Optin Scpe: SAS Sessin Hw ptin value set: SAS Sessin Startup Cmmand Line PROC OPTIONS (PERFORMANCE) Get infrmatin abut PERFORMANCE settings and hw they gt their values. PROC OPTIONS grup=performance value; run; Optin Value Infrmatin Fr SAS Optin SPDEUTILLOC Optin Value: Optin Scpe: Default Hw ptin value set: Shipped Default TEST_LEVEL=2 Gives finer grained lgging f the cube build prcess. The actual beginning f Phase II is clearly demarcated, by the test _level message: IN PROGRESS: NWAY creatin This message ccurs well befre the first NOTE in the lg abut the NWAY. PLEASE NOTE! TEST_LEVEL is an undcumented ptin f the PROC OLAP statement PROC OLAP CUBE=name TEST_LEVEL=2 <additinal PROC OLAP statement ptins>; General TEST_LEVEL lg frmat PHASE I Cnnect with metadata server Initialize cube build Per hierarchy get captins create member metadata elapsed times N WAYcreatin begins NWAY write t disk NWAY index build Aggregatin builds begin (aka subaggregatin) Cube files disked Metadata server updated with cube infrmatin Elapsed times See Figure 6 fr Sample Output 10

11 Figure 6 Sample utput f TEST_LEVEL=2 ptin CONCLUSION By understanding the cmpnents f a MOLAP cube, metadata and data, the cube build prcess can be explained using tw Phase apprach. PHASE I is assciated t prcessing the metadata, and the prcessing f the data. Being aware f the apprpriate SAS ptins and PROC OLAP ptins and hw they can be applied can: Ensure a successful build especially when assigning memry cnsumptin and disk space Optimizatin the build prcesses, maximizing the delivery f the data while minimizing the impact t yur system In additin, recmmendatins have been made s that yu can design, architect and recgnize prblems with yur build prcess, allwing yu t make decisins relating t: Input data Temprary wrk space NWAY creatin Aggregatin creatin Ultimately, this paper will aid in the success f building yur cube therefre, prviding yur business user infrmatin in a timely manner. Allwing the business user t fcus n the right business decisin they need t make fr yur cmpany. 11

12 ACKNOWLEDGMENTS Thank yu t all wh helped ut with testing and cntributing t the cntent f this paper. RECOMMENDED READING SAS 9.2 OLAP Server: User s Guide Paper Avid Grwing Pains: New Cube Update Features Yu Shuld Knw Abut (Weinberger, Tierney) Paper The SAS OLAP Server: Understanding and Slving Cmmn Prblems (Budlng) CONTACT INFORMATION Yur cmments and questins are valued and encuraged. Cntact the authr at: Mary Simmns SAS Institute Inc. Campus Drive Cary, NC, mary.simmns@sas.cm Web: Michelle Wilkie SAS Institute Inc. Campus Drive Cary, NC, michelle.wilkie@sas.cm Web: SAS and all ther SAS Institute Inc. prduct r service names are registered trademarks r trademarks f SAS Institute Inc. in the USA and ther cuntries. indicates USA registratin. Other brand and prduct names are trademarks f their respective cmpanies. 12