Data Modelling for DW & Cubes

Transcription

1 Data Modelling for DW & Cubes Alex Whittles PurpleFrogSystems.com

2 SQLSaturday #467 Sponsors

3 Alex Whittles SQL Relay ExCo SQLRelay.co.uk SQL Bits Committee SQLBits.com Birmingham SQL UG SQLMidlands.com Birmingham Azure UG AzureBirmingham.uk MSc in Business Intelligence, CEng, CITP, FBCS, FIOEE, MIET, MIOD Run BI Consultancy Business Intelligence Consultancy Data Modelling Data Warehousing OLAP Cubes ETL Systems Reporting Systems Managed Service

4 Data Modelling for Cubes - Agenda Why remodel data? Scenarios: Simple Transactional Data Semi Additive Data State/Event Based Data

5 SSAS Options Multi-Dimensional (Cubes) Tabular (PowerPivot)

6 Getting Data into a Cube DSV Source System OLAP Cube Use the Data Source View in SSAS to pull data directly in from the source system

7 Getting Data into a Cube DSV Data Warehouse OLAP Cube Source Systems Use a data warehouse to remodel data, and store history

8 What are we going to cover? Simple Transactional Data Bank Transactions Purchases Semi-Additive Data Bank Balances Stock Levels State Orientated Data Project Management Loan Application Status Insurance Claim Status Property Occupancy

9 Simple Transactional Data Bank Transactions Purchases Payments Website Hits Event Count Durations Date Account Transaction 4/5/ /5/ /5/ /5/ SUM: Fully Additive Load Directly into Fact table

10 Create Transactional Data Durations VisitID Name Entry Exit 1 Joe Bloggs 18/8/12 10:05 18/8/12 10:15 2 Dave Angel 18/8/12 10:10 18/8/12 10:30 VisitID Name Entry Exit Visit Duration 1 Joe Bloggs 18/8/12 10:05 18/8/12 10: Dave Angel 18/8/12 10:10 18/8/12 10:30 20

11 Create Transactional Data Durations VisitID Name Type Time 1 Joe Bloggs Entry 18/8/12 10:05 2 Dave Angel Entry 18/8/12 10:10 3 Joe Bloggs Exit 18/8/12 10:15 4 Dave Angel Exit 18/8/12 10:30? VisitID Name Entry Exit Visit Duration 1 Joe Bloggs 18/8/12 10:05 18/8/12 10: Dave Angel 18/8/12 10:10 18/8/12 10:30 20

12 What about orphans? VisitID Name Type Time 1 Joe Bloggs Exit 18/8/12 08:30 2 Joe Bloggs Entry 18/8/12 10:05 3 Dave Angel Entry 18/8/12 10:10 4 Joe Bloggs Exit 18/8/12 10:15 5 Dave Angel Exit 18/8/12 10:30 6 Joe Bloggs Entry 18/8/12 11:00 VisitID Name Entry Exit Visit Duration 1 Joe Bloggs? 18/8/12 08:30? 2 Joe Bloggs 18/8/12 10:05 18/8/12 10: Dave Angel 18/8/12 10:10 18/8/12 10: Joe Bloggs 18/8/12 11:00??

13 Demo

14 Semi-Additive Measures Date Transaction Balance 1/5/ /5/ /5/ /5/ SUM LastNonEmpty 150

15 Semi-Additive Measures LastNonEmpty Not available in Standard Snapshot every record, every day/month Can replicate functionality in MDX [Slow] Or Convert into transactions

16 Semi-Additive Measures Snapshot every record, every day/month Day 1 Day 2 Day 3 Day 4 Day 5 Product A Product B Product C LastNonEmpty Without a complete dataset on every date, LastNonEmpty will give incorrect results

17 Semi-Additive Measures Or, convert into transactions Day 1 Day 2 Day 3 Day 4 Day 5 Product A Product B Product C Day 1 Day 2 Day 3 Day 4 Day 5 Product A Product B Product C Sum To Date

18 Turn transactions back into Balance Expensive in SQL, but cheap in MDX Sum all transactions to date: CREATE MEMBER CURRENTCUBE.[Measures].[Stock Level] AS SUM({NULL:[Date].[Calendar].CurrentMember}, [Measures].[Stock Movement] );

19 Modelling Status Project Management Loan Application Insurance Claim Property Occupancy Manufacturing Stage

20 Modelling Status Project Management Loan Application Insurance Claim Property Occupancy Manufacturing Stage How many current projects Projects Started Project Completed Change in active projects

21 Project Management Simple Option Project ID Name Start Date End Date 1 Project A 3/4/2013 7/5/ Project B 8/4/ /4/ Project C 17/4/ /5/2013

22 Project Management Simple Option Project State Start Date Project Manager End Date

23 Project Management Simple Option Project count for Day X AGGREGATE( {NULL:LINKMEMBER([DATE].[Calendar].CURRENTMEMBER,[START DATE].[Calendar])} * {LINKMEMBER([DATE].[Calendar].CURRENTMEMBER, [END DATE].[Calendar]):NULL}, [Measures].[Project COUNT]) Not so simple!

24 Project Management Simple Option Years of hell with MDX Why? Fact table should represent an event, not an item

25 Project Management Better Option - Turn an Item into an Event Project ID Name Event Date Project Count 1 Project A Start 3/4/ Project A End 7/5/ Project B Start 8/4/ Project B End 23/4/ Project C Start 17/4/ Project C End 15/5/ Project A Project B Project C

26 Project Management Better Option Project Project Event Type Project Event Manager Date

27 Project Management Project Count on Date X AGGREGATE({NULL:[DATE].[Calendar].CURRENTMEMBER},[Measures].[Project Count]) Projects Starting ([Project Event Type].[Event].[Start], [Measures].[Project Count]) Projects Ending ([Project Event Type].[Event].[End], [Measures].[Project Count])

28 Demo

29 More than one way. Scenario 1 Event is duration VisitID Name Entry Exit Visit Duration 1 Joe Bloggs 18/8/12 10:05 18/8/12 10: Dave Angel 18/8/12 10:10 18/8/12 10:30 20 Scenario 2 Event is start or end Project ID Name Event Date Project Count 1 Project A Start 3/4/ Project A End 7/5/2013-1

30 Modelling Status Project Management Loan Application Insurance Claim Property Occupancy Manufacturing Stage

31 Modelling a Status Project Management Loan Application Insurance Claim Property Occupancy Manufacturing Stage Application Received Application Review Credit Check Offer Acceptance

32 Modelling a Status What do we want to Report on? Number of applications in a given stage Throughput Backlogs

33 Modelling a Status Customer Date Amount Status Joe Bloggs 1/8/ Applied Joe Bloggs 2/8/ Applied Joe Bloggs 3/8/ Reviewed Joe Bloggs 4/8/ Credit Checked Joe Bloggs 5/8/ Offer Sent Joe Bloggs 6/8/ Offer Accepted Loan Application Progress

34 The Status Pipeline in/out flow Customer Date Amount Status Joe Bloggs 1/8/ Applied Joe Bloggs 2/8/ Applied Joe Bloggs 3/8/ Reviewed Joe Bloggs 4/8/ Credit Checked Joe Bloggs 5/8/ Offer Sent Joe Bloggs 6/8/ Offer Accepted 1/8/2012 2/8/2012 3/8/2012 4/8/2012 5/8/2012 6/8/2012 Applied Reviewed Credit Checked Offer Sent Offer Accepted 1500

35 Turn into bucket in/out flow Customer Date Amount Status Joe Bloggs 1/8/ Applied Joe Bloggs 2/8/ Applied Joe Bloggs 2/8/ Applied Joe Bloggs 3/8/ Applied Joe Bloggs 3/8/ Reviewed Joe Bloggs 4/8/ Reviewed Joe Bloggs 4/8/ Credit Checked Joe Bloggs 5/8/ Credit Checked Joe Bloggs 5/8/ Offer Sent Joe Bloggs 6/8/ Offer Sent Joe Bloggs 6/8/ Offer Accepted

36 Turn into bucket in/out flow Customer Date Amount Status Joe Bloggs 1/8/ Applied Joe SUM Bloggs TO DATE 2/8/ Applied Joe Bloggs 2/8/ Applied Joe Bloggs 3/8/ Applied Joe Bloggs 3/8/ Reviewed Joe SUM Bloggs TO DATE 4/8/ Reviewed Joe Bloggs 4/8/ Credit Checked Joe Bloggs 5/8/ Credit Checked Joe Bloggs 5/8/ Offer Sent Joe Bloggs 6/8/ Offer Sent Joe Bloggs 6/8/ Offer Accepted SUM TO DATE 1500

37 Turning Status into Transactions ETL Intensive Generates a lot of fact records Avoid singleton query approach Use T-SQL Merge

38 Demo Find current status Source Data Back out old record Create new record Staging Table Merge Fact Table

39 Demo

40 Summary Put the effort into the data model, not the MDX Iterate and evolve the data model until it s right Facts should be EVENTS Dimensions should be ITEMS Think MDX, not SQL Prototype cube Write MDX Redesign Data Model Write simplified MDX

41 Data Modelling for Analysis &