Nicholas Dritsas Principal Program Manager Microsoft Corporation Microsoft Corporation. All rights reserved

Size: px

Start display at page:

Christian Harrington
5 years ago
Views:

1 Nicholas Dritsas Principal Program Manager Microsoft Corporation

2 Who is SQL Customer Advisory Team (SQL CAT) Overview of large AS projects Lessons Learned People and Infrastructure Performance Improving Processing Performance Improving Query Performance Performance MDX Tips Scale up vs. Scale Out

3 Works with largest, most complex SQL Server projects worldwide US: NASDAQ, USDA, Verizon, Raymond James Europe: LSE, Barclay s Capital APAC: NUL, KT, Western Digital, JR East Drives enterprise requirements back into SQL Server Shares best practices with SQL Server community SQL CAT Blog: SQL ISV PM Blog: You will see us at PASS & TechED

4 Hilton Hotels Forecasting and budgeting application for their 2,400 + hotels. Real time OLAP, scale out AS and RS Danske Supermarket 2TB AS DB, 500GB largest cube size, 800 users, 10 concurrent queries Retail Sales Analysis Sales reporting and analysis for multiple business. 30+ terabyte relational warehouse, scale up AS, use of RS and ProClarity Inventory Tracking and Analysis Purchase and inventory tracking and analysis. Teradata backend with ROLAP AS, RS, and ProClarity Financial Institution Account Management Complex design with linked cubes and parent child dimensions 40M Member customer dimension

5 MDX Expert, MDX Expert, MDX Expert MDX Syntax, Profiler analysis (subcube usage etc.) Analysis Services AS2000 expertise!= AS2005 expertise SQL Server Business Domain Expert The usual infrastructure guru with knowledge of: 64-bit NUMA SAN / DISKS Performance Monitor Tools and analysis

SQL Server Profiler Performance Monitor ASCMD.exe (http://msdn2.microsoft.com/en-us/library/ms365187.

6 SQL Server Profiler Performance Monitor ASCMD.exe ( Command line execution tool for batch processing Custom Aggregation Tool Available as a sample with SP2 / on CodePlex Browse aggregations Create custom aggregations based on the query log or completely from scratch Stress testing tools Based on Visual Studio for Testers (Okracoke) Web tests for Reporting Services Query tests for MDX ASLoadSim just posted to Codeplex

7 64-bit is critical on large implementations because of the extra memory it can access If the warehouse is not memory constrained, 64-bit might actually be slower Older 64-bit chips run at much slower clock speeds than current 32-bit chips Newer dual core chips run much faster than older 64-bit chips This is not a SQL issue it is true of any app that moves from 32-bit to 64-bit

8 To take full advantage of new AS2005 features, redesign rather than migrate from AS2000 to AS2005 AS2005 is a very different product and cubes may need redesign to take advantage of new features Calculations are often handled differently and may need to be rewritten (especially to use the Scope statement) Many customers are going from 32-bit AS2000 to 64-bit AS2005

9 Who is SQL Customer Advisory Team (SQL CAT) Overview of large AS projects Lessons Learned People and Infrastructure Performance Improving Processing Performance Improving Query Performance Performant MDX Tips Scale up vs. Scale Out

10 Improve the queries that are used for extracting data from the source system Use query binding to optimize partition queries Use INT for keys whenever possible Use SP2!! Many processing improvements were included in SP2 Don t let UI default for parallel processing Go into advanced processing tab and change it Parallel tasks: X number of processors Also set max number of data source connections Utilize partitions to limit processing scope Avoid too many aggregations

11 Use incremental updates whenever possible For best performance use ASCMD.EXE and XMLA Use <Parallel> </Parallel> to group processing tasks together until Server is using maximum resources Proper use of <Transaction> </Transaction> ProcessFact and ProcessIndex separately instead of ProcessFull Different CPU usage pattern

12 Watch for long running queries that may block processing Lock waits counter for AS Ensure sufficient memory to avoid temp file Temp file rows (or bytes) written/sec Avoid requesting too many operations in parallel Quota blocked counter for AS Memory grants pending for SQL engine

13 Define natural hierarchies Most common cause of query performance problems Create cascading attribute relationships and define natural hierarchies Remove redundant relationships whenever possible Partition cube data Reduce cube space touched by a query Increase query parallelism Partitioning strategy must match query pattern Define custom aggregations when needed Capture a workload using the Query Log Use the Aggregation Manager sample and have it read the query log Avoid too many aggregations

14 Write performant MDX Read Nicholas Dritsas blog on Read Mosha Pasumansky blog on Make sure you have MDX expert on project Keep cube space as small as possible Only include measure groups that are needed Use multiple measure groups when measures are not queried together Calculated Members Optimize according to following MDX tips Some can be done using SCOPE in MDX Use NON_EMPTY_BEHAVIOR whenever possible Warm the cache

15 Monitor for competing processes, such as processing events and long running queries Monitor for insufficient memory cached data being pushed out of memory Cleaner Memory Nonshrinkable KB counter in AS Use SQL Server Profiler to determine if bottleneck is in storage engine or query execution engine Add all duration values (in the duration column) for the Query End event for each subquery generated by the MDX query to determine total storage engine time Subtract from total execution time to get total query execution engine time Reduce cube space before calculations Total cells calculated counter in AS

17 How many partitions should you have? In general more smaller partitions are better than a few large partitions until you get over 2,000 (dependent on your hardware) More CPUs and more I/O threads as you increase the number of partitions Partitioning Tips (dependent on hardware) <= 20M records per partition Have seen acceptable performance up to 55M records Partitions of 250MB or smaller where possible Have seen acceptable performance up to 2GB Fewer than 2000 partitions

18 Large Parent-Child Dimensions Redesign as denormalized table to improve performance Linked measure groups on remote servers Linked measure groups on the same server perform well Linked dimensions Additional querying overhead Referenced dimensions Materialize to improve query performance Real Time ROLAP Use separate partition to improve query performance

19 Who is SQL Customer Advisory Team (SQL CAT) Overview of large AS projects Lessons Learned People and Infrastructure Performance Improving Processing Performance Improving Query Performance Performant MDX Tips Scale up vs. Scale Out

20 Avoid assigning values like 0, Null, N/A, - to cells that would remain empty otherwise Avoid redundant Sum/Aggregate calculations in situations where default/normal cell value aggregation would do Avoid run-time checks that result in slow execution path Case and IIF frequently used in this manner Use SCOPE to reduce calculation space instead, or write such that one condition is written null Prefer using static literal hierarchy and member references Use Measures.Sales instead of Dimensions(0).Sales, Avoid using StrToSet, StrToMember, StrToValue Perform string manipulation in AS stored procedures

Use Non_Empty_Behavior optimization hint Use instead of writing calculation expressions of the form Aggregate(NonEmptyCrossjoin(Descendants(, Leaves) ).

21 Use Non_Empty_Behavior optimization hint Use instead of writing calculation expressions of the form Aggregate(NonEmptyCrossjoin(Descendants(, Leaves) ). Use an explicit measure name Use instead of Measures.CurrentMember whenever possible. With calculations like expr1 * expr2, ensure the expression sweeping the largest area/volume in the cube space is on the left side Replace simple Measure1 + Measure2 calculations with computed columns in the DSV or in the SQL data source Instead of writing expressions like Sum(Customer.City.Members, Customer.Population.MemberValue), consider defining a separate measure group on the City table, with a Sum measure on the Population column

22 Use Exists Function Exists function should be used where ever possible instead of filter on member properties. Use Non Empty keyword and IsEmpty function to reduce cube space Use Minus over Filter for a single member When filtering out a single member from the set use minus over filter function Avoid: filter({set},.currentmember <> "UNKN") Use : ( {set} minus {&[UNKN] member}) Avoid Linked Member Use multiple measure groups in single cube

23 Scale up to 64-bit, multiple processors, SAN, Scale up tips Add more memory (4GB per core recommended) More CPUs can help parallelize processing and complex queries Partition your cubes effectively (see Partitioning Tips slide) If the project reaches the physical limits of the hardware, scale out becomes the next logical solution

and multiple query servers Use multiple query servers to load balance queries Use dedicated processing server

24 Can create multiple instances of AS on the same server Can be difficult to manage Requires the use of Windows Resource Manager Frequently better to scale out to multiple smaller servers Create a separate processing server and multiple query servers Use multiple query servers to load balance queries Use dedicated processing server Use synchronize function in AS or copy directories using Robocopy (or similar) for best performance with large databases

25 References BLOGS: PROJECT REAL-Business Intelligence in Practice Analysis Services Performance Guide TechNet: Analysis Services for IT Professionals

. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.

26 . Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

6234A - Implementing and Maintaining Microsoft SQL Server 2008 Analysis Services

6234A - Implementing and Maintaining Microsoft SQL Server 2008 Analysis Services Course Number: 6234A Course Length: 3 Days Course Overview This instructor-led course teaches students how to implement