COMP9318: Data Warehousing and Data Mining. L2: Data Warehousing and OLAP

Transcription

1 COMP9318: Data Warehousig ad Data Miig L2: Data Warehousig ad OLAP 1

2 Why ad What are Data Warehouses? 2

3 Data Aalysis Problems The same data foud i may differet systems Example: customer data across differet departmets The same cocept is defied differetly Heterogeeous sources Relatioal DBMS, OLie Trasactio Processig (OLTP) Ustructured data i files (e.g., MS Excel) ad documets (e.g., MS Word) 3

4 Data Aalysis Problems (Cot d) Data is suited for operatioal systems Accoutig,billig,etc. Do ot support aalysis across busiess fuctios Data quality is bad Missig data, imprecise data, differet use of systems Data are volatile Data deleted i operatioal systems (6moths) Data chage over time o historical iformatio 4

5 Solutio: Data Warehouse Defied i may differet ways, but ot rigorously. A decisio support database that is maitaied separately from the orgaizatio s operatioal database Support iformatio processig by providig a solid platform of cosolidated, historical data for aalysis. A data warehouse is a subject-orieted, itegrated, time-variat, ad ovolatile collectio of data i support of maagemet s decisio-makig process. W. H. Imo Data warehousig: The process of costructig ad usig data warehouses 5

6 Data Warehouse Subject-Orieted Orgaized aroud major subjects, such as customer, product, sales. Focusig o the modelig ad aalysis of data for decisio makers, ot o daily operatios or trasactio processig. Provide a simple ad cocise view aroud particular subject issues by excludig data that are ot useful i the decisio support process. 6

7 Data Warehouse Itegrated Costructed by itegratig multiple, heterogeeous data sources relatioal databases, flat files, o-lie trasactio records Data cleaig ad data itegratio techiques are applied. Esure cosistecy i amig covetios, ecodig structures, attribute measures, etc. amog differet data sources E.g., Hotel price: currecy, tax, breakfast covered, etc. Whe data is moved to the warehouse, it is coverted. 7

8 Data Warehouse Time Variat The time horizo for the data warehouse is sigificatly loger tha that of operatioal systems. Operatioal database: curret value data. Data warehouse data: provide iformatio from a historical perspective (e.g., past 5-10 years) Every key structure i the data warehouse Cotais a elemet of time, explicitly or implicitly But the key of operatioal data may or may ot cotai time elemet. 8

9 Data Warehouse No-Volatile 1. A physically separate store of data trasformed from the operatioal eviromet. 2. Operatioal update of data does ot occur i the data warehouse eviromet. Does ot require trasactio processig, recovery, ad cocurrecy cotrol mechaisms Requires oly two operatios i data accessig: iitial loadig of data ad access of data. 9

10 Data Warehouse Architecture Extract data from operatioal data sources clea, trasform Bulk load/refresh warehouse is offlie OLAP-server provides multidimesioal view Multidimesioal-olap (Essbase, oracle express) Relatioal-olap (Redbrick, Iformix, Sybase, SQL server) 10

11 Data Warehouse Architecture All subjects, itegrated Advaced aalysis Fuctio-orieted systems Subject-orieted systems 11

12 Why Separate Data Warehouse? High performace for both systems DBMS tued for OLTP: access methods, idexig, cocurrecy cotrol, recovery Warehouse tued for OLAP: complex OLAP queries, multidimesioal view, cosolidatio. Differet fuctios ad differet data: missig data: Decisio support requires historical data which operatioal DBs do ot typically maitai data cosolidatio: DS requires cosolidatio (aggregatio, summarizatio) of data from heterogeeous sources data quality: differet sources typically use icosistet data represetatios, codes ad formats which have to be recociled 12

13 Why OLAP Servers? Differet workload: OLTP (o-lie trasactio processig) Major task of traditioal relatioal DBMS Day-to-day operatios: purchasig, ivetory, bakig, maufacturig, payroll, registratio, accoutig, etc. OLAP (o-lie aalytical processig) Major task of data warehouse system Data aalysis ad decisio makig Queries hard/ifeasible for OLTP, e.g., Which week we have the largest sales? Does the sales of dairy products icrease over time? Geerate a spread sheet of total sales by state ad by year. Difficult to represet these queries by usig SQL ç Why? 13

14 OLTP vs. OLAP OLTP OLAP users clerk, IT professioal kowledge worker fuctio day to day operatios decisio support DB desig applicatio-orieted subject-orieted data curret, up-to-date detailed, flat relatioal isolated usage repetitive ad-hoc access read/write lots of scas idex/hash o prim. key uit of work short, simple trasactio complex query # records accessed tes millios #users thousads hudreds DB size 100MB-GB 100GB-TB historical, summarized, multidimesioal itegrated, cosolidated metric trasactio throughput query throughput, respose 14

15 Comparisos Databases Data Warehouses Purpose May purposes; Flexible ad geeral Oe purpose: Data aalysis Coceptual Model ER Multidimesioal Logical Model (Normalized) Relatioal Model (Deormalized) Star schema / Data cube/cuboids Physical Model Relatioal Tables ROLAP: Relatioal tables MOLAP: Multidimesioal arrays Query Laguage Query Processig SQL (hard for aalytical queries) B+-tree/hash idexes, Multiple joi optimizatio, Materialized MDX (easier for aalytical queries) Bitmap/Joi idexes, Star joi, Materialized data cube 15

16 The Multidimesioal Model 16

17 The Multidimesioal Model A data warehouse is based o a multidimesioal data model which views data i the form of a data cube, which is a multidimesioal geeralizatio of 2D spread sheet. Key cocepts: Facts: the subject it models Typically trasactios i this course; other types icludes sapshots, etc. Measures: umbers that ca be aggregated Dimesios: cotext of the measure Hierarchies: Provide cotexts of differet graularities (aka. grais) Goals for dimesioal modelig: Surroud facts with as much relevat cotext (dimesios) as possible ç Why? 17

18 Supermarket Example Subject: aalyze total sales ad profits Fact: Each Sales Trasactio Measure: Dollars_Sold, Amout_Sold, Cost Calculated Measure: Profit Dimesios: Store Product Time 18

19 Visualizig the Cubes A valid istace of the model is a data cube total Sales city product p1 p2 p3 p4 NY $ $925 LA $468 $ SD $296 - $240 - SF $652 - $540 $745 city product Cocepts: cell, fact (=o-empty cell), measure, dimesios Q: How to geeralize it to 3D? 19

20 3D Cube ad Hierarchies Cocepts: hierarchy (a tree of dimesio values), level Sales of book176 i NY i Ja ca be foud i this cell city DIMENSIONS PRODUCT LOCATION TIME NY Book176 ALL ALL ALL category regio year product product coutry quarter state moth week Ja Feb Mar moth city store day 20

21 Hierarchies Cocepts: hierarchy (a tree of dimesio values), level Which desig is better? Why? ALL Food Beverage Book Book176 ALL category product ALL category subcategory brad product 21

22 The (city, moth) Cuboid Sales of ALL_PROD i NY i Ja city NY Book176 DIMENSIONS PRODUCT LOCATION TIME ALL ALL ALL category regio year product product coutry quarter state moth week Ja Feb Mar moth city store day 22

23 All the Cuboids Assume: o other o-all levels o all dimesios. TV PC VCR Product Date 1Qtr 2Qtr 3Qtr 4Qtr U.S.A Caada Mexico Coutry 23

24 All the Cuboids /2 Assume: o other o-all levels o all dimesios. Product all TV PC VCR Date 1Qtr 2Qtr 3Qtr 4Qtr all Total aual sales of TV i U.S.A. U.S.A Caada Mexico Coutry all 24

25 Lattice of the cuboids Base cuboid product, quarter, coutry 3-dim cuboid product,quarter quarter, coutry product, store 2-dim cuboid quarter product coutry 1-dim cuboid 0-dim cuboid -dim cube ca be represeted as (D 1, D 2,, D d ), where D i is the set of allowed values o the i-th dimesio. if D i = L i (a particular level), the Di = all descedat dimesio values of L i. ALL ca be omitted ad hece reduces the effective dimesioality dy ( i + 1) A complete cube of d-dimesios cosists of i=1 cuboids, where i is the umber of levels (excludig ALL) o i-th dimesio. They collectively form a lattice. 25

26 Properties of Operatios All operatios are closed uder the multidimesioal model i.e., both iput ad output of a operatio is a cube So that they ca be composed Q: What s the aalogy i the Relatioal Model? 26

27 Commo OLAP Operatios Roll-up: move up the hierarchy Sales of book176 i NY i Q1 here city product NY Book176 Q: what should be its value? DIMENSIONS PRODUCT LOCATION TIME ALL ALL ALL category regio year product coutry quarter state moth week Q1 Q2 moth Q3 COMP9318: Data Warehousig ad Data Miig city store day 27

28 city Book176 productny quarter DIMENSIONS PRODUCT LOCATION TIME ALL ALL ALL product city NY Book176 category regio year product coutry quarter state moth week city day Ja Feb Mar moth store 28

29 Commo OLAP Operatios Drill-dow: move dow the hierarchy more fie-graied aggregatio 29

30 city Slice ad Dice Queries Slice ad Dice: select ad project o oe or more dimesio values product moth Product = Book176 The output cube has smaller dimesioality tha the iput cube 30

31 Pivotig city NY Book176 Pivotig: aggregate o selected dimesios usually 2 dims (crosstabulatio) product Sales (of all products) i NY i Q1 =sum(??? ) Q1 Q2 moth Q3 pivot o (city, moth) City NY Q1 Q2 Q3 moth 31

32 A Reflective Pause Let s review the defiitio of data cubes agai. Key message: Disetagle the object from its represetatio or implemetatio 32

33 Modelig Exercise 1: Mothly Phoe Service Billig Theme: aalyze the icome/reveue of Telstra 33

34 Solutio FACT MEASURE DIMENSIONS 34

35 35

36 The Logical Model 36

37 Logical Models Two mai approaches: Usig relatioal DB techology: Star schema, Sowflake schema, Fact costellatio Usig multidimesioal techology: Just as multidimesioal data cube 37

38 Uiversal Schema è Star Schema May data warehouses adopt a star schema to represet the multidimesioal model Each dimesio is represeted by a dimesio-table LOCATION (locatio_key, store, street_address, city, state, coutry, regio) dimesio tables are ot ormalized Trasactios are described through a fact-table each tuple cosists of a poiter to each of the dimesio-tables (foreig-key) ad a list of measures (e.g. sales $$$) The uiversal schema for supermarket Store City State Prod Brad Category $Sold #Sold Cost S136 Syd NSW 76Ha Nestle Biscuit S173 Melb Vic 76Ha Nestle Biscuit

39 The Star Schema TIME time_key day day_of_the_week moth quarter year SALES time_key product_key PRODUCT product_key product_ame category brad color supplier_ame locatio_key LOCATION uits_sold locatio_key store amout street_address city state coutry Thik why: regio (1) Deormalized oce from the uiversal schema (2) Cotrolled redudacy 39

40 Typical Models for Data Warehouses Modelig data warehouses: dimesios & measures Star schema: A fact table i the middle coected to a set of dimesio tables Sowflake schema: A refiemet of star schema where some dimesioal hierarchy is ormalized ito a set of smaller dimesio tables, formig a shape similar to sowflake Fact costellatios: Multiple fact tables share dimesio tables, viewed as a collectio of stars, therefore called galaxy schema or fact costellatio 40

41 Example of Star Schema time time_key day day_of_the_week moth quarter year brach brach_key brach_ame brach_type Measures Sales Fact Table time_key time_key item_key brach_key locatio_key uits_sold dollars_sold avg_sales item item_key item_ame brad type supplier_type locatio locatio_key street city state_or_provice coutry 41

42 Example of Sowflake Schema time time_key day day_of_the_week moth quarter year Sales Fact Table time_key item_key item item_key item_ame brad type supplier_key supplier supplier_key supplier_type brach brach_key brach_ame brach_type Measures brach_key locatio_key uits_sold dollars_sold avg_sales locatio locatio_key street city_key city city_key city state_or_provice coutry 42

43 Example of Fact Costellatio time time_key day day_of_the_week moth quarter year Sales Fact Table time_key item_key item_key brach_key item item_key item_ame brad type supplier_type Shippig Fact Table time_key item_key Shipper_key from_locatio brach brach_key brach_ame brach_type Measures locatio_key uits_sold dollars_sold avg_sales locatio locatio_key street city provice_or_state coutry to_locatio dollars_cost uits_shipped shipper shipper_key shipper_ame locatio_key shipper_type 43

44 Advatages of Star Schema Facts ad dimesios are clearly depicted dimesio tables are relatively static, data is loaded (apped mostly) ito fact table(s) easy to comprehed (ad write queries) Fid total sales per product-category i our stores i Europe SELECT PRODUCT.category, SUM(SALES.amout) FROM SALES, PRODUCT,LOCATION WHERE SALES.product_key = PRODUCT.product_key AND SALES.locatio_key = LOCATION.locatio_key AND LOCATION.regio= Europe GROUP BY PRODUCT.category Operatios: Slice (Loc.Regio.Europe) + Pivot (Prod.category) 44

45 Query Laguage 45

46 Query Laguage Two approaches: Usig relatioal DB techology: SQL (with extesios such as CUBE/PIVOT/UNPIVOT) Usig multidimesioal techology: MDX SELECT PRODUCT.category, SUM(SALES.amout) FROM SALES, PRODUCT,LOCATION WHERE SALES.product_key = PRODUCT.product_key AND SALES.locatio_key = LOCATION.locatio_key AND LOCATION.regio= Europe GROUP BY PRODUCT.category SELECT {[PRODUCT].[category]} o ROWS, {[MEASURES].[amout]} o COLUMNS FROM [SALES] WHERE ([LOCATION].[regio].[Europe]) Operatios: Slice (Loc.Regio.Europe) + Pivot (Prod.category, Measures.amt) 46

47 Physical Model + Query Processig Techiques 47

48 Physical Model + Query Processig Techiques Two mai approaches: Usig relatioal DB techology: ROLAP Usig multidimesioal techology: MOLAP Hybrid: HOLAP Base cuboid: ROLAP Other cuboids: MOLAP 48

49 Q1: Selectio o low-cardiality attributes TIME time_key day day_of_the_week moth quarter year measures { SALES time_key customer_key locatio_key uits_sold amout Igorig the fial GROUP BY for ow Omittig the Product dimesio S regio= Europe JOIN JOIN CUSTOMER customer_key customer_ame regio type LOCATION locatio_key store street_address city state coutry regio 49

50 Idexig OLAP Data: Bitmap Idex (1) BI o dimesio tables Idex o a attribute (colum) with low distict values Each distict values, v, is associated with a -bit vector ( = #rows) Chap 4.2 i [JPT10] The i-th bit is set if the i-th row of the table has the value v for the idexed colum Multiple BIs ca be efficietly combied to eable optimized sca of the table Custom Cust Regio Type C1 Asia Retail C2 Europe Dealer C3 Asia Dealer C4 America Retail C5 Europe Dealer BI o Customer.Regio v bitmap Asia Europe America

51 Idexig OLAP Data: Bitmap Idex /2 (1) Bitmap joi idex (BI o Fact Table Joied with Dimesio tables) Coceptually, perform a joi, map each dimesio value to the bitmap of correspodig fact table rows. -- ORACLE SYNTAX CREATE BITMAP INDEX sales_cust_regio_bjix ON sales(customer.cust_regio) FROM sales, customer WHERE sales.cust_id = customers.cust_id; 51

52 Idexig OLAP Data: Bitmap Idex /3 Sales time customer loc Sale 101 C C C C C C C C Customer Cust Regio Type C1 Asia Retail C2 Europe Dealer C3 Asia Dealer C4 America Retail C5 Europe Dealer BI o Sales(Customer.Regio) v bitmap Asia Europe America

53 Q2: Selectio o high-cardiality attributes TIME time_key day day_of_the_week moth quarter year SALES time_key customer_key JOIN CUSTOMER customer_key customer_ame regio type measures { locatio_key uits_sold amout JOIN S city= Kigsford LOCATION locatio_key store street_address city state coutry regio 53

54 Chap 4.3 i [JPT10] Idexig OLAP Data: Joi Idices Joi idex relates the values of the dimesios of a star schema to rows i the fact table. a joi idex o city maitais for each distict city a list of ROW-IDs of the tuples recordig the sales i the city Locatio city = Sydey city = Radwick city = Kigsford city = Coogee R102 R117 R118 Sales Joi idices ca spa multiple dimesios OR R124 1 ca be implemeted as bitmapidexes (per dimesio) use bit-op for multiple-jois Kigsford R102,R117,R118,R124 54

55 Q3: Arbitrary selectios o Dimesios TIME time_key day day_of_the_week moth quarter year SALES time_key product_key S regio = Asia JOIN Customer customer_key customer_ame regio type S isschoolholidy(day) measures { locatio_key uits_sold amout JOIN S city =~ /\S+ford/ LOCATION locatio_key store street_address city state coutry regio 55

56 Chap 4.4 i [JPT10] Star Query ad Star Joi (Cot.) Time Customer Loc Usually oly part of the dim tables because of the selectio predicates time 1 2 thousads of tuples X Cust X loc time cust loc millios of tuples Sales time cust loc sold Sales!" σ 1 (Time)!" σ 2 (Cust)!" σ 3 (Loc) è Sales!" (σ 1 (Time) X σ 2 (Cust) X σ 3 (Loc)) 56

57 Q4: Coarse-grai Aggregatios Fid total sales per customer type i our stores i Europe Joi-idex will prue ¾ of the data (uiform sales), but the remaiig ¼ is still large (several millios trasactios) Idex is uclustered High-level aggregatios are expesive!!!!! regio LOCATON ÞLog Query Respose Times ÞPre-computatio is ecessary ÞPre-computatio is most beeficial coutry state city store 57

58 Cuboids = GROUP BYs Multidimesioal aggregatio = selectio o correspodig cuboid GB (type, city) ( Sales!" σ 1 (Time)!" σ 2 (Cust)!" σ 3 (Loc)) ) σ 1 selects some Years, σ 2 selects some Brads, σ 3 selects some Cities, GB (type, city) ( σ (Cuboid(Year, Type, City)) ) Materialize some/all of the cuboids A complex decisio ivolvig cuboid sizes, query workload, ad physical orgaizatio 58

59 Two Issues How to store the materialized cuboids? How to compute the cuboids efficietly? 59

60 CUBE BY i ROLAP Store Sales Product ALL ALL Group-bys here: (store,product) (store) (product) () Need to write 4 queries!!! Compute them idepedetly Store Product_key sum(amout) ALL ALL ALL ALL 1937 ALL ALL ALL ALL ALL ALL 5120 SELECT LOCATION.store, SALES.product_key, SUM (amout) FROM SALES, LOCATION WHERE SALES.locatio_key=LOCATION.locatio_key CUBE BY SALES.product_key, LOCATION.store 60

61 Top-dow Approach Model depedecies amog the aggregates: product,quarter quarter Cuboid AB product,store,quarter store,quarter product oe A B M product, store store most detailed view ca be computed from view (product,store,quarter) by summig-up all quarterly sales Cuboid A A B M Cuboid B 1 (all)??? 2 (all)??? A B M (all) 1??? (all) 2??? (all) 3??? (all) 4??? 61

62 Bottom-Up Approach (BUC) BUC (Beyer & Ramakrisha, SIGMOD 99) Ideas Compute the cube from bottom up Divide-ad-coquer A simpler recursive versio: BUC-SR A B

63 Uderstadig Recursio /1 Powerful computig/problem-solvig techiques Examples Factorial: f() = 1, if = 1 f() = f(-1) *, if 1 Quick sort: Sort([x]) = [x] Sort([x1,, pivot, x]) = sort[ys] ++ sort[zs]), where ys = [ x x i xi, x pivot ] zs = [ x x <- xi, x > pivot ] f(0) = 0! =??? List comprehesio i Haskell or pytho 63

64 Uderstadig Recursio /2 Let C(, m) be the umber of ways to select m balls from umbered balls Show that C(, m) = C(-1, m-1) + C(-1, m) etire solutio space Example: m = 3, = 5 Cosider ay ball i the 5 balls, e.g., d?1?2?3 d?1?2?1 (o d)?2 (o d)?3 (o d)?i i { a b c d e } 64

65 Key Poits Sub-problems eed to be smaller, so that a simple/trivial boudary case ca be reached Divide-ad-coquer There may be multiple ways the etire solutio space ca be divided ito disjoit sub-spaces, each of which ca be coquered recursively. 65

66 Geometric Ituitio /1 Reduce Cube(i 2D) to Cube(i 1D) a1 a2 [a1] [a2] [*] b1 b2 b3 M11 M12 M13 [Step 1] M21 M22 M23 [Step 1] [Step 2] [Step 2] [Step 2] [Step 3] b1 b2 b3 M11 M12 M13 [Step 1] M21 M22 M23 [Step 1] [Step 2] [Step 2] [Step 2] [Step 3] 66

67 Geometric Ituitio /2 Reduce Cube(i 3D) to Cube(i 2D) A C B 67

68 Geometric Ituitio /3 A C B Reduce Cube(i 3D) to Cube(i 2D) A B C 68

69 You eed to fix the icomplete output part of the algorithm! BUC-SR (Simple Recursio)* BUC-SR(data, dims) If (dims is empty) Else Output (sum(data)) Boudary case: data is essetially a list of measure values Dims = [dim1, rest_of_dims] For each distict value v of dim1 slice_v = slice of data o dim1 = v BUC-SR(slice_v, rest_of_dims) data = Project(data, rest_of_dims) BUC-SR(data, rest_of_dims) Geeral case: 1)Slice o dim1. Call BUC-SR recursively for each slice 2)Project out dim1, ad call BUC-SR o it recursively 69

70 Iput Iteral Output Output Example [{r1-r4}, B] B M A B M B M * * * * [{r5}, B] B M A B M B M [{r1-r5}, AB] 1 50 * 50 2 * 50 r1 r2 r3 r4 r5 A B M [{r1 -r5 }, B] B M B M * 150 A B M * 1 80 * 2 30 * 3 40 * *

71 Try a 3D-Cube by Yourself [{r1-r5}, ABC] r1 r2 r3 r4 r5 6/3/18 A B C M

72 MOLAP (Sparse) array-based multidimesioal storage egie Pros: small size (esp. for dese cubes) fast i idexig ad query processig Cos: scalability coversio from relatioal data 72

73 Multidimesioal Array Step 2: A ijective map from cell to offset f(time, item) = 4*time + item time item dollars_sold Q1 Q2 Q3 Q4 home etertaimet 605 home etertaimet 680 home etertaimet 812 home etertaimet 927 Q1 computer 825 Q2 computer 952 Q3 computer 1023 Step 1 Mappigs time value Q1 0 Q2 1 Q3 2 Q4 3 time item dollars_s old offset Q4 computer Q1 phoe 14 Q2 phoe 31 Q3 phoe 30 Q4 phoe 38 Q1 security 400 Q2 security 512 Q3 security 501 Q4 security 580 item value home etertai met 0 computer 1 phoe 2 security

74 Step 3 : If sparse Multidimesioal Array Step 3: If dese, oly eed to store sorted slots offset dollars_sold Dese MD array Thik: how to decode a slot? Multidimesioal array is typically sparse Use sparse array (i.e., offset + value) Could use chuk to further reduce the space Space usage: (d+1)**4 vs 2**4 HOLAP: Store all o-base cuboid i MD array Assig a value for ALL