Big Data 13. Data Warehousing

Transcription

1 Ghislain Fourny Big Data 13. Data Warehousing fotoreactor / 123RF Stock Photo

2 2 The road to analytics Aurelio Scetta / 123RF Stock Photo

3 3 Another history of data management (T. Hofmann) 1970s 2000s Age of Transactions Age of Business Intelligence 2000s - Age of Big Data

4 4 Paradigms vs. OLTP OLAP

5 5 OnLine Transaction Processing Consistent and Reliable Record-Keeping

6 OnLine Transaction Processing Transactions and results on small portions of data 6

7 OnLine Transaction Processing Lots of transactions on small portions of data 7

8 8 OnLine Transaction Processing Normalized Data

9 9 OnLine Analytical Processing Data-based Decision Support arturaliev / 123RF Stock Photo

10 10 OLAP is Big Large portions of the data Possibly many joins Few long heavy queries

11 11 OLAP Examples Web analytics Sales analytics Management support Statistical analysis (census) Scientific databases (e.g., bio-informatics)

12 OLTP vs. OLAP Detailed Individual Records Historical Summarized Consolidated Data Aurelio Scetta / 123RF Stock Photo vs. OLTP OLAP 12

13 OLTP OLAP 13 OLTP vs. OLAP vs. Lots of writes Lots of reads

14 OLTP OLAP 14 OLTP vs. OLAP Small sets of records vs. Analysis over big chunks

15 OLTP OLAP 15 OLTP vs. OLAP fully interactive (< 1s) vs. Slow interactive

16 OLTP vs. OLAP Redundancy Redundancy Redundancy Consistency OLTP OLAP 16

17 17 OLAP Aurelio Scetta / 123RF Stock Photo

18 A data warehouse... is a subject-oriented integrated time-variant nonvolatile collection of data in support of management's decision-making process 18

19 19 Subject-oriented customers sales products events

20 Integrated 20

21 21 Time-variant Time in data warehouses is paramount (not so in OLTP systems)

22 Time-variant Y-9 Y-8 Y-7 Y-6 Y-5 Y-4 Y-3 Y-2 Y Often past 5-10 years 22

23 Non-volatile Load. Access. Period. no updates Milosh Kojadinovich / 123RF Stock Photo 23

24 24 Architecture ERP Analyze CRM ETL Report OLTP Files Mine

25 OLAP: Redundancy Materialized views (denormalized) 25

26 1st Normal Form (tabular) The Key 26

27 2nd Normal Form (not joined) The Whole Key 27

28 3rd Normal Form Nothing But The Key 28

29 Why materialize? Operational data sources are too heterogeneous 29

30 OLAP: Special-purpose indices 30

31 OLAP: Derived data 31

32 32 Querying OLAP 6 Slow interactive Category 1 Category 2 Category 3 Category 4 Series 1 Series 2 Series 3 vs. Continuous monitoring/tracking 1-10s hours

33 Summary of differences OLTP OLAP Source Original (operational) Derived (consolidated) Purpose Business tasks Decision support Interface Snapshot Multidimensional views Writing short and fast, by end user period refreshes, by batch jobs Queries Simple, small results Complex and aggregating Design Many normalized tables Few denormalized cubes Precision ACID Sampling, confidence intervals Freshness Serializability Reproducibility Speed Very fast Often slow Optimization Inter-query Intra-query Space Small, archiving old data Large, less space efficient Backup Very important Re-ETL 33

34 Data Model 34

35 35 Data Cubes Data is stored in multidimensional hypercubes

36 36 Data Cubes Year

37 37 Data Cubes Country

38 Data Cubes Product 38

39 39 Fact 2016 CH Server

40 Dimensions Which currency? Where What? Who? When? etc. 40

41 41 Fact table Where? Germany 2016 Peter 1,000$ Germany 2015 Mary 15,000$ Switzerland 2016 Mary 1,500$ Switzerland 2015 Peter 3,000$ Australia 2015 Peter 6,000$ China 2015 Mary 1,000$

42 42 Aggregation Where? Germany 2016 Peter 1,000$ Germany 2015 Mary 15,000$ Switzerland 2016 Mary 1,500$ Switzerland 2015 Peter 3,000$ Australia 2015 Peter 6,000$ China 2015 Mary 1,000$

43 Aggregation 43

44 44 Aggregation Where? Germany 2016 Peter 1,000$ Germany 2015 Mary 15,000$ Switzerland 2016 Mary 1,500$ Switzerland 2015 Peter 3,000$ Australia 2015 Peter 6,000$ China 2015 Mary 1,000$

45 45 Aggregation 2016 Peter 1,000$ 2015 Mary 16,000$ 2016 Mary 1,500$ 2015 Peter 9,000$

46 Slicing 46

47 Slicers and Dicers Slicers Dicers 47

48 Slicers and Dicers Usually between 1 and 3 dicers, often 2 Slicers Dicers 48

49 49 Slicers and Dicers Slicers Servers World USD

50 50 Slicers and Dicers Dicers Slicers Servers World USD Peter 1,000,000$ 1,500,000$ 1,400,000$ Mary 2,000,000$ 2,300,000$ 2,200,000$

51 51 Products: the big three Essbase Cognos Analysis Services

52 ETLing 52

53 OLAP: Derived data 53

54 54 OLAP: Derived data ETL

55 55 ETL Extract Transform Load

56 56 Extract Triggers Gateways Incremental updates Log extraction

57 57 Transform Herr Derivation Mister Value transformation Cleaning Filter, split, merge, join

58 58 Load Integrity constraints Sorting Build indices Partition

59 Considerations When? Granularity Infrastructure 59

60 Implementation 60

61 61 Two flavors of OLAP ROLAP MOLAP

62 62 Fact table (ROLAP) Dim1 Dim2 Dim3 Dim4 Dim5 Value

63 63 Star Schema Dim1 Dim2 Dim3 Dim4 Dim5 Value

64 64 Snow-flake schema Dim1 Dim2 Dim3 Dim4 Dim5 Value Normalize More

65 Querying 65

66 66 Querying cubes Tables: SQL Cubes: MDX

67 67 MDX stands for... Multi-Dimensional expressions

68 68 Measures Amount of licenses Revenues Taxes paid...

69 69 Dimensions Quarter Salesperson Product Country

70 70 In short... A cube is a list of dimensions indexing a list of measures

71 71 Hierarchies Dimension values are organized in hierarchies. [Location] [Geo] [Economy] i.e., slice and aggregate by geographic region, etc i.e., slice and aggregate by economic partnership, etc

72 Members Members correspond to levels in a hierarchy. [Ocenia] [Africa] [Geo] [Europe] [Asia] [America] [Switzerland] [China] [Canada] [ZH] [India] [USA] [BE]... [Brazil] [Germany]

73 73 Identifying a member [Location].[Geo].[Europe].[Switzerland].[ZH].[Zurich]

74 Tuples A list of members ([Location].[Geo].[Europe].[Switzerland].[ZH].[Zurich], [Salesmen].[People].[John], [Time].[Year].[2016].[Q4]) Associated with a dimensionality (list of hierarchies) ([Location].[Geo], [Salesmen].[People] [Time].[Year]) 74

75 Sets A set of tuples with same dimensionality { ([Location].[Geo].[Europe].[Switzerland].[ZH].[Zurich], [Salesmen].[People].[John], [Time].[Year].[2016].[Q4]), ([Location].[Geo].[Europe].[Switzerland].[BE].[Bärn], [Salesmen].[People].[Mary], [Time].[Year].[2016].[Q4]), ([Location].[Geo].[Europe].[Germany].[Berlin], [Salesmen].[People].[John], [Time].[Year].[2016].[Q3]) } 75

76 76 MDX statements: dicing SELECT [Measures].Members ON COLUMNS, [Location].[Geo].Members ON ROWS FROM [Sales]

77 77 MDX statements: slicing SELECT [Measures].Members ON COLUMNS, [Location].[Geo].Members ON ROWS FROM [Sales] WHERE [Products].[Line].[Laptops].[MBP]

78 Syntax 78

79 XBRL Architecture Linkbase (.xml) Schema (.xsd) Instance (.xml) Discoverable Taxonomy Set 79

80 80 Technologies XML XML Schema XML Names XML Link

81 Fact Dimension Value What? Assets Who? Coca Cola When? Dec 31, 2011 Of what? USD <us-gaap:assets contextref="fi2012q4" decimals="-6" id="fact fd4d06e63b4f8f6874c6e5be74" unitref="usd"> </us-gaap:assets> 81

82 82 Context <xbrli:context id="fi2011q4"> <xbrli:entity> <xbrli:identifier scheme=" </xbrli:identifier> </xbrli:entity> <xbrli:period> <xbrli:instant> </xbrli:instant> </xbrli:period> </xbrli:context> December

83 Unit <xbrli:unit id="usd"> <xbrli:measure>iso4217:usd</xbrli:measure> </xbrli:unit> 83

84 84 Concept (XML Schema) <xs:element id='us-gaap_assets' name='assets' nillable='true' substitutiongroup='xbrli:item' type='xbrli:monetaryitemtype' xbrli:balance='debit' xbrli:periodtype='instant' />

85 Graphs 85

86 DAGs 86

87 Trees 87

88 88 Node: locator <loc xlink:href=" xlink:label="loc_usgaap_assets_102d7a4d204ed45ac0deda6bbc78f38 6" xlink:type="locator" />

89 89 Node: resource <link:label id="lab_ko_netchangeinoperatingassetsandliabilitiesdisclosureabstrac t_a6469a522e35cbf ee_label_en-us" xlink:label="lab_ko_netchangeinoperatingassetsandliabilitiesdisclosur eabstract_a6469a522e35cbf ee" xlink:role=" xlink:type="resource" xml:lang="en-us"> NET CHANGE IN OPERATING ASSETS AND LIABILITIES DISCLOSURE [Abstract] </link:label>

90 90 Edge <presentationarc order="10" preferredlabel=" xlink:arcrole=" xlink:from="loc_usgaap_assetsabstract_2f55ecb2bf7c1a62009cda6bbc757094" xlink:to="loc_usgaap_assets_102d7a4d204ed45ac0deda6bbc78f386" xlink:type="arc" />

91 Summary 91

92 92 Architecture ERP Analyze CRM ETL Report OLTP Files Mine