IDU0010 ERP,CRM ja DW süsteemid Loeng 5 DW concepts. Enn Õunapuu

Transcription

1 IDU0010 ERP,CRM ja DW süsteemid Loeng 5 DW concepts Enn Õunapuu enn.ounapuu@ttu.ee

2 Content Oveall approach Dimensional model Tabular model

3 Overall approach Data modeling is a discipline that has been practiced for many years by BI professionals with one common goal: organizing disparate data into an analytic model that effectively and efficiently supports the reporting and analysis needs of the business.

4 Business Intelligence Customer Inventory Marketing Data Mart OLAP Credit Sales ETL tools Data Warehouse Finance Data Mart BI Operation Distribution Data Mart Reports External Pivot Table

5

6 Star Schema A Star schema is optimized for data analysis as is typically required in a Decision Support System. A star schema has a fact table surrounded by dimension tables

7 Data Mart A data mart is a repository of data gathered from operational data and other sources that is designed to serve a particular community of knowledge workers. The data may derive from an enterprise-wide database or data warehouse or be more specialized. The emphasis of a data mart is on meeting the specific demands of a particular group of knowledge users in terms of analysis, content, presentation, and ease-of-use. Users of a data mart can expect to have data presented in terms that are familiar.

8 On-Line Analytical Processing (OLAP) Literally, On-Line Analytical Processing. Designates a category of applications and technologies that allow the collection, storage, manipulation and reproduction of multidimensional data, with the goal of analysis. Example:

9 Pivot Table A pivot table is a great reporting tool that allows for slicing and dicing data.

10 BI Semantic Model Third-Party BI Applications Reporting Services Reports Excel Workbooks PowerPivot Applications SharePoint Dashboards & Scorecards BI Applications Relational Multidimensional Data Model DAX MDX BI Semantic Model Business Logic VertiPaq Realtime Data Access Files Odata Feeds Data Sources

11 Scalability & Performance VertiPaq engine in memory, column oriented store High performance via brute force memory scans No tuning, indexes, aggregates required VertiPaq is optimized for latest x86 and x64 chipsets Designed to exploit cheap memory on latest server h/w Inherently multi-threaded and scales linearly with number of cores Scales from desktops to highest end servers State-of-the-art compression algorithms reduce data volumes by 10x or more Partitioning & paging to support large models

12 How Should I Build my Model? Depends on the application needs for each layer Data model Business logic Data access & storage Two Visual Studio (BIDS) project types in Denali Multidimensional project with MDX and MOLAP/ROLAP Tabular project with DAX and VertiPaq/DirectQuery Project types could change post-denali VertiPaq in multidimensional projects, MDX scripts in tabular projects Based on customer feedback

13 How Should I Build my Model? Depends on the application needs for each layer Data model Business logic Data access & storage Two Visual Studio (BIDS) project types in Denali Multidimensional project with MDX and MOLAP/ROLAP Tabular project with DAX and VertiPaq/DirectQuery Project types could change post-denali VertiPaq in multidimensional projects, MDX scripts in tabular projects Based on customer feedback

14 Data Model Tabular Familiar model, easier to build, faster time to solution Advanced concepts (parentchild, many-to-many) not available natively in the model need calculations to simulate these Easy to wrap a model over a raw database or warehouse for reporting & analytics Multidimensional Sophisticated model, higher learning curve Advanced concepts baked into the model and optimized (parent-child, many-to-many, attribute relationships, key vs. name, etc.) Ideally suited for OLAP type apps (e.g. planning, budgeting, forecasting) that need the power of the multidimensional model

15 Business Logic DAX Based on Excel formulas and relational concepts easy to get started Complex solutions require steeper learning curve row/filter context, Calculate, etc. Calculated columns enable new scenarios, however no named sets or calc members MDX Based on understanding of multidimensional concepts higher initial learning curve Complex solutions require steeper learning curve CurrentMember, overwrite semantics, etc. Ideally suited for apps that need the power of multidimensional calculations scopes, assignments, calc members

16 Data Access and Storage VertiPaq In-memory column store typical 10x compression Brute force memory scans high performance by default no tuning required Basic paging support data volume mostly limited to physical memory DirectQuery Passes through DAX queries & calculations fully exploits backend database capabilities No support for MDX queries no support for data sources other than SQL Server (in Denali) MOLAP Disk based store typical 3x compression Disk scans with in-memory subcube caching aggregation tuning required Extensive paging support data volumes can scale to multiple terabytes ROLAP Passes through fact table requests not recommended for large dimension tables Supports most relational data sources no support for aggregations except SQL Server indexed views

17 Analysis Services Architecture Internet Explorer SharePoint BI Development Studio Project Juneau Excel Services PowerPivot for Excel xlsx Reporting Services Analysis Services Excel PowerPivot for SharePoint (Analysis Services) xlsx BI Semantic Model Third Party Apps Personal BI Team BI Organizational BI

18 Multidimensional modelling Multidimensional modeling, introduced with SQL Server 7.0 OLAP Services and continuing through SQL Server 2012 Analysis Services, enables BI professionals to create sophisticated multidimensional cubes using traditional online analytical processing (OLAP)..

19 Extraction, Transformation, and Loading (ETL) The process of data consolidation is often called Extraction, Transformation, and Loading (ETL) The ETL process extracts data from the various source systems Data is then transformed to make it consistent and improve data quality The consolidated, consistent, and cleaned data is then loaded into a data repository Developing the ETL process often consumes 80% of the development time

20 The Users of Business Intelligence

21 Strategic, Tactical & Functional Benefits of Business Intelligence

22 The Approaches to Consuming Business Intelligence Scorecards Customized high-level views with limited analytic capabilities Reports Standardized reports aimed at a large audience, with no or limited analytic capabilities Analytics Applications Applications designed to allow complex data analysis Custom Applications Embed BI data within an application

23 The Components of a Data Warehouse There are several items that make up a data warehouse Cubes Measures Key Performance Indicators Dimensions Attributes Hierarchies

24 Cubes Cubes are the structures in which data is stored Users access data in the cubes by navigating through various dimensions

25

26 Perfomance One of the primary ways to optimize query performance is the use of aggregations. An aggregation is a precalculated summary of data that is used to enhance query performance for multidimensional models. When you query a multidimensional model, the Analysis Services query processor decomposes the query into requests for the OLAP storage engine. For each request, the storage engine first attempts to retrieve data from the storage engine cache in memory. If no data is available in the cache, it attempts to retrieve data from an aggregation. If no aggregation is present, it retrieves the data from a measure group s partitions.

27 Measures Measures are what you want to see They are almost always numeric They are often additive Dollar sales, unit sales, profit, expenses, and more Some measures are not additive Date of last shipment Inventory counts and number of unique customers

28 Key Performance Indicators Key Performance Indicators (KPIs) are typically a special type of measure A KPI might be Customer Retention, which is a calculation of customer churn A KPI may be Customer satisfaction derived from one or more measures (ratings in a survey or product returns + number of repeat customers). KPIs are often what are shown on scorecards KPIs often contain not just the number, but also a target number Used to evaluate the health of the value

29 Dimensions Dimensions are how you want to see the data You usually want to see data by time, geography, product, account, employee, Dimensions are made up of attributes and may or may not include hierarchies Year Semester Quarter Month Day Product Category Product Subcategory - Product

30 Attributes Attributes are individual values that make up dimensions A Time dimension may have a Month attribute, a Year attribute, and so forth A Geography dimension may have a Country attribute, a Region attribute, a City attribute, and so on A Product dimension may have a Part Number attribute, a size attribute, a color attribute, a manufacturer attribute, and more

31 Hierarchies You can put attributes into a hierarchical structure to assist user analysis One of the most common functions in BI is to drill down to a more detailed level For example, Time hierarchy might be to go from Year to Quarter to Month to Day Another Time hierarchy might go from Year to Month to Week to Day to Hour

32 Tabular model Tabular modeling, introduced with PowerPivot for Microsoft Excel 2010, provides self-service data modeling capabilities to business and data analysts. The tabular modeling experience is more accessible to these users, many who have spent years working with data in desktop productivity tools like Excel and Microsoft Access. In SQL Server 2012, tabular modeling has been extended to enable BI professionals to create tabular models in Analysis Services or to import a tabular model from PowerPivot into Analysis Services. Note that a PowerPivot model cannot be imported into an Analysis Services multidimensional model.

33 Tabular model perfomance When a user queries a tabular model, the engine performs memory scans to retrieve the data and to calculate aggregations on the fly without the need of disk I/O processing. This approach can provide very high query performance without requiring special tuning and aggregation management. The best and easiest way to optimize query performance for tabular models is to maximize available memory. From a scalability perspective, data volume is mostly limited by physical memory. It is highly recommended that you provide sufficient memory to contain all of the data in your tabular model. In scenarios where memory is constrained, the inmemory engine also provides basic paging support according to physical memory. In addition, there are serverside configuration settings that allow IT to more finely manage the memory available to tabular models.

34 Tabular model example Now consider the same scenario in a tabular model. In the tabular model, there is no concept of dimensions and measure groups. Instead, data is organized into tables that have relationships to each other. Assume that sales organization data and sales data are each in their own respective tables with a relationship based on the individual sales rep. With this design, when you refresh your sales organization table, it automatically updates any impacted calculated columns, relationships, and user hierarchies. This means that the sales data automatically reflects the updated sales region rollups without the need to reprocess the sales data. This flexibility can provide significant benefits when you have rapidly changing dimensions and you need the data to reflect the latest updates.

35

36

37

38 Summary The ETL process extracts data from source systems, transforms it and then loads it to a data warehouse or a data mart. Using reports and dashboards, BI looks at data as a collection of measures and KPIs viewed by dimensions.

39 Questions?