Data Warehouse: Introduction

Transcription

1 Data Warehuse: Intrductin Data warehuse Intrductin Database and data mining grup, Plitecnic di Trin Plitecnic di Trin Database and data mining grup, Plitecnic di Trin Decisin supprt systems Huge peratinal databases are available in mst cmpanies these databases may prvide a large wealth f useful infrmatin Decisin supprt systems prvide means fr in depth analysis f a cmpany s business faster and better decisins Cpyright All rights reserved INTRODUCTION - 1 Plitecnic di Trin Cpyright All rights reserved INTRODUCTION - 2 Plitecnic di Trin Database and data mining grup, Plitecnic di Trin Strategic decisin supprt Demand evlutin analysis and frecast Critical business areas identificatin Budgeting and management transparency reprting, practices against frauds and mney laundering Identificatin and implementatin f winning strategies cst reductin and prfit increase Business Intelligence Database and data mining grup, Plitecnic di Trin BI prvides supprt t strategic decisin supprt in cmpanies Objective: transfrming cmpany data int actinable infrmatin at different detail levels fr analysis applicatins Users may have hetergeneus needs BI requires an apprpriate hardware and sftware infrastructure Cpyright All rights reserved INTRODUCTION - 3 Plitecnic di Trin Cpyright All rights reserved INTRODUCTION - 4 Plitecnic di Trin Applicatins Database and data mining grup, Plitecnic di Trin Manifacturing cmpanies: rder management, client supprt Distributin: user prfile, stck management Financial services: buyer behavir (credit cards) Insurance: claim analysis, fraud detectin Telecmmunicatin: call analysis, churning, fraud detectin Public service: usage analysis Health: service analysis and evaluatin... and many mre... Lan Amunt Eample Database and data mining grup, Plitecnic di Trin Bank clients with a lan : bad clients wing peridic payments t the bank after due date : gd clients respecting peridic payment due date Incme Cpyright All rights reserved INTRODUCTION - 5 Plitecnic di Trin Cpyright All rights reserved INTRODUCTION - 6 Plitecnic di Trin Plitecnic di Trin Pag. 1

2 Data Warehuse: Intrductin Database and data mining grup, Plitecnic di Trin Database and data mining grup, Plitecnic di Trin Eample Eample Lan Amunt k Incme Lan Amunt Incme If Incme < k then bad client Cpyright All rights reserved INTRODUCTION - 7 Plitecnic di Trin Cpyright All rights reserved INTRODUCTION - 8 Plitecnic di Trin Data management Database and data mining grup, Plitecnic di Trin Traditinal DBMS usage, characterized by snapsht f current data state detailed data, relatinal representatin well-knwn, structured and repetitive peratins read/write access t few recrds shrt transactins islatin, reliability and integrity (ACID) are critical database size 100MB-GB Data analysis Database and data mining grup, Plitecnic di Trin Data prcessing fr decisin supprt, characterized by histrical data cnslidated and integrated data ad hc applicatins read access t millins f recrd cmple queries data cnsistency befre and after peridical lads database size 100GB-TB Cpyright All rights reserved INTRODUCTION - 9 Plitecnic di Trin Cpyright All rights reserved INTRODUCTION - 10 Plitecnic di Trin Data warehuse Database and data mining grup, Plitecnic di Trin Database devted t decisin supprt, which is kept separate frm cmpany peratinal databases Data which is integrated time dependent, nn vlatile devted t a specific subject used fr decisin supprt in a cmpany W. H. Inmn, Building the data warehuse, 1992 Why separate data? Database and data mining grup, Plitecnic di Trin Perfrmance cmple queries reduce perfrmance f peratinal transactin management different access methds at the physical level Data managent missing infrmatin (e.g., histry) data cnslidatin data quality (incnsistency prblems) Cpyright All rights reserved INTRODUCTION - 11 Plitecnic di Trin Cpyright All rights reserved INTRODUCTION - 12 Plitecnic di Trin Plitecnic di Trin Pag. 2

3 shp shp Data Warehuse: Intrductin Eternal data surces Database and data mining grup, Plitecnic di Trin Data warehuse: architecture Metadata DW management Back end tls Data warehuse Data marts OLAP servers Analysis tls Data Analysis Database and data mining grup, Plitecnic di Trin Data warehuse and data mart Cmpany warehuse: it cntains all the infrmatin n the cmpany business etensive functinal mdelling prcess design and implementatin require a lng time Data mart: departimental infrmatin subset fcused n a given subject tw architectures dependent, feeded by the primary data warehuse independent, feeded directly by the surces faster implementatin requires careful design, t avid subsequent datamart integratin prblems Cpyright All rights reserved INTRODUCTION - 13 Plitecnic di Trin Cpyright All rights reserved INTRODUCTION - 14 Plitecnic di Trin Back-end tls Database and data mining grup, Plitecnic di Trin Feed the data warehuse (ETL = Etractin Transfrmatin Lading) data estractin frm eternal surces data cleaning (errrs, missing r duplicated data) frmat trasfrmatins and cnversins data lading and peridical refresh Database and data mining grup, Plitecnic di Trin Servers fr Data Warehuses ROLAP (Relatinal OLAP) server etended relatinal DBMS cmpact representatin fr sparse data SQL etensins fr aggregate cmputatin specialized access methds which implement efficient OLAP data access MOLAP (Multidimensinal OLAP) server data represented in prprietary (multidimensinal) matri frmat sparse data require cmpressin special OLAP primitives HOLAP (Hybrid OLAP) server Cpyright All rights reserved INTRODUCTION - 15 Plitecnic di Trin Cpyright All rights reserved INTRODUCTION - 16 Plitecnic di Trin Database and data mining grup, Plitecnic di Trin Database and data mining grup, Plitecnic di Trin Multidimensinal representatin Data warehuse fr tracking sales in a supermarket chain Multidimensinal representatin Measures n which analysis is perfrmed: cells at dimensin intersectin prduct Data is represented as an (hyper)cube with three r mre dimensins prduct Cpyright All rights reserved INTRODUCTION - 17 Plitecnic di Trin Cpyright All rights reserved INTRODUCTION - 18 Plitecnic di Trin Plitecnic di Trin Pag. 3

4 Data Warehuse: Intrductin Database and data mining grup, Plitecnic di Trin Relatinal representatin: star mdel (Numerical) measures stred in the fact table attribute dmain is numeric Dimensins describe the cntet f each measure in the fact table characterized by many descriptive attributes Eample Database and data mining grup, Plitecnic di Trin Data warehuse fr tracking sales in a supermarket chain Shp Sale Prduct Time Cpyright All rights reserved INTRODUCTION - 19 Plitecnic di Trin Cpyright All rights reserved INTRODUCTION - 20 Plitecnic di Trin Data warehuse size Database and data mining grup, Plitecnic di Trin Time dimensin: 2 years 365 days Shp dimensin: 300 shps Prduct dimensin: prducts, f which sld every day in every shp Number f rws in the fact table: = 657 millins Size f the fact table 21GB Data analysis tls Database and data mining grup, Plitecnic di Trin OLAP analysis: cmple aggregate functin cmputatin supprt t different types f aggregate functins (e.g., mving average, tp ten) Data analysis by means f data mining techniques varius analysis types significant algrithmic cntributin Cpyright All rights reserved INTRODUCTION - 21 Plitecnic di Trin Cpyright All rights reserved INTRODUCTION - 22 Plitecnic di Trin Data analysis tls Database and data mining grup, Plitecnic di Trin Presentatin separate activity: data returned by a query may be rendered by means f different presentatin tls Mtivatin search Data eplratin by means f prgressive, incremental refinements (e.g., drill dwn) Database and data mining grup, Plitecnic di Trin Types f data mining activities Classificatin and regressin: predictive mdel generatin requires a previusly labelled data set Assciatin rules: etractin f data crrelatins Clustering: data partined in hmgeneus grups requires the ntin f distance between tw elements Cpyright All rights reserved INTRODUCTION - 23 Plitecnic di Trin Cpyright All rights reserved INTRODUCTION - 24 Plitecnic di Trin Plitecnic di Trin Pag. 4

5 Data Warehuse: Intrductin high Eample: classificatin Database and data mining grup, Plitecnic di Trin Age Car type Risk categry 40 SW lw 65 sprt high 20 utility high Age < sprt high 50 utility lw Car type = sprt Database and data mining grup, Plitecnic di Trin Eample: assciatin rules Given a cllectin f cunter transactins in a supermarket (receipts) Assciatin rules diapers beer 2% f transactins cntains bth elements 30% f transactins cntaining diapers als cntains beer high lw Decisin tree Cpyright All rights reserved INTRODUCTION - 25 Plitecnic di Trin Cpyright All rights reserved INTRODUCTION - 26 Plitecnic di Trin Tetbks Database and data mining grup, Plitecnic di Trin Useful links Database and data mining grup, Plitecnic di Trin Data warehusing Glfarelli, Rizzi, Data warehuse: teria e pratica della prgettazine, McGraw-Hill 2006 Data mining Han, Kamber, Data mining: cncepts and techniques, Mrgan Kaufmann 2006 General n DW OLAP Cuncil Data mining Cpyright All rights reserved INTRODUCTION - 27 Plitecnic di Trin Cpyright All rights reserved INTRODUCTION - 28 Plitecnic di Trin Plitecnic di Trin Pag. 5