Decision Support Systems

Transcription

1 Decision Support Systems 2011/2012 Week 3. Lecture 5

2 Previous Class: Data Pre- Processing Data quality: accuracy, completeness, consistency, 4meliness, believability, interpretability Data cleaning: handling missing and noisy values Data integra4on: Remove redundancies Data transforma4on: Normaliza4on Data reduc4on: Dimensionality reduc4on

3 Today Data reduc4on & discre4za4on (conclusion) Data Warehousing: Introduc4on Data cubes Online analy4cal processing (OLAP)

4 Previously, on DSS

5 Data ReducGon (revisited) Obtain reduced representa4on of data that produces similar mining results Data reduc4on strategies: 1. Dimensionality reduc4on: Data is encoded so as to reduce dimension (wavelet transforms, PCA, feature selec4on) 2. Numerosity reduc4on: Data is replaced by smaller representa4ons, either parametric (e.g., log- linear models) or non- parametric (e.g., clustering, sampling) 3. ANribute selec4on: Where irrelevant or redundant anributes are removed

6 1. Dimensionality ReducGon Transforms (wavelets, Fourier) use specific basis to represent data PCA builds a basis from the data (eigenvalues of covariance matrix) Transformed data can be truncated Data New Basis (Fourier, Wavelet, PC) Trunca4on

7 2. Numerosity ReducGon Reduce data volume using smaller forms of data representa4on Two main classes of methods Parametric methods: Assume the data fits a model Es4mate and store model parameters Discard the data (except possible outliers) Non- parametric methods: Do not assume models

8 Parametric Data ReducGon Regression and Log- Linear models Linear regression: Data modeled to fit a straight line OVen uses the least- square method to fit the line Mul4ple regression: Data modeled as a linear func4on of mul4dimensional feature vector Log- linear model: Approximates data as a (parameterized) probability distribu4on Also useful for dimensionality reduc4on and smoothing

9 Histogram Analysis Already met histograms as a tool for data visualiza4on Histogram approach to numerosity reduc4on: Divide data into buckets Store mean for each bucket Par44oning rules: Equal- width (equal bucket range) Equal- frequency (equal number of points)

10 Clustering Clustering approach to numerosity reduc4on: Par44on data set into clusters based on similarity Store cluster representa4on (e.g., centroid and diameter) Can be very effec4ve if data is clustered; not if data is smeared There are many choices of clustering defini4ons and clustering algorithms Cluster analysis will be covered later on

11 Sampling Sampling: obtaining a small sample to represent the whole data set Complexity of mining algorithms poten4ally sub- linear in the size of the data Key principle: Choose a representa4ve subset of the data Sampling approaches: Simple random sampling (may have very poor performance if data is skewed) Stra4fied sampling (data divided into strata, each sampled separately) Cluster sampling (data is cluster and then clusters are sampled)

12 3. AOribute Subset SelecGon (aka Feature SelecGon) Remove redundant anributes/dimensions With N anributes, 2 N possible subsets (exhaus4ve search imprac4cal) Common techniques: Stepwise forward selec4on: depart from empty set, add one feature at a 4me Stepwise backward elimina4on: depart from full set, remove one feature at a 4me Combina4on of forward selec4on and backward elimina4on Heuris4cs to guide selec4on/elimina4on: Sta4s4cal tests Informa4on gain

13 Data Cube AggregaGon Data in a database is aggregated at different granulari4es Mul4dimensional aggregated informa4on is stored a data cube dark pastel white all skirt dress shirt pants all all small medium large dark pastel white TOTAL skirt dress shirt pants TOTAL

14 Data DiscreGzaGon

15 DiscreGzaGon Discre4za4on: Divide range of con4nuous anribute into intervals Interval labels used as actual data values Reduces and simplifies original data Discre4za4on techniques can: Be supervised (uses class informa4on) or unsupervised Rely on spligng (top- down) or merging (bonom- up) Discre4za4on can be performed recursively on an anribute 15

16 Data DiscreGzaGon Methods Binning: already discussed for smoothing Top- down split, unsupervised Histogram analysis: already discussed in data visualiza4on Top- down split, unsupervised Cluster analysis: Values par44oned in clusters from data distribu4on Unsupervised, top- down split or bonom- up merge Entropy- based analysis: Values par44oned from informa4on measures Supervised, top- down split Correla4on analysis: Correla4on informa4on guides merging with neighbor values Unsupervised, bonom- up merge

17 Concept Hierarchies Concept hierarchies organize concepts hierarchically Usually associated dimensions in a data warehouse Concept hierarchies facilitate viewing data at mul4ple granulari4es Concept hierarchy forma4on: Recursively reduce the data by collec4ng and replacing low level concepts by higher level concepts Concept hierarchies can be explicitly specified by domain experts or automa4cally generated for both numeric and nominal data

18 Concept Hierarchy GeneraGon (Nominal Data) Specifica4on of a par4al/total ordering of anributes E.g., street < city < state < country Specifica4on of a hierarchy for a set of values by explicit data grouping E.g., {Urbana, Champaign, Chicago} < Illinois Specifica4on of an par4al/total order on a par4al set of anributes E.g., street < city, but not others Automa4c genera4on of hierarchies by analysis of dis4nct values E.g., {street, city, state, country}

19 Example Automa4c genera4on of hierarchies by analysis of dis4nct values ANribute with most dis4nct values is placed at the lowest level of the hierarchy 15 dis4nct values Country Careful: There are excep4ons! 365 dis4nct values 3567 dis4nct values Province or State City 674,339 dis4nct values Street

20 Data Warehousing

21 IntroducGon

22 What is a Data Warehouse? Defined in different ways, not rigorously: A decision support database maintained separately from the organiza4on s opera4onal database Support informa4on processing by providing a solid plaoorm of consolidated, historical data for analysis A data warehouse is a subject- oriented, integrated, 6me- variant, and non - vola6le collec6on of data in support of management s decision- making process Data warehousing: The process of construc4ng and using data warehouses W. H. Inmon

23 Subject- Oriented Organized around major subjects, such as customer, product, sales Focusing on the modeling and analysis of data for decision makers, not on daily opera4ons or transac4on processing Provide a simple and concise view around par4cular subject issues by excluding data that are not useful in the decision support process

24 Integrated Constructed by integra4ng mul4ple heterogeneous data sources E.g., rela4onal databases, flat files, on- line transac4on records Construc4on requires data cleaning and data integra4on techniques: Ensure consistency across different data sources in naming conven4ons, encoding structures, anribute measures, etc. When data is consolidated in the warehouse, it is converted

25 Time Variant Time horizon for data warehouses is longer than that of opera4onal DB systems: Opera4onal databases: Current values of data Data warehouses: Informa4on from a historical perspec4ve Every key structure in the data warehouse contains elements of 4me, explicitly or implicitly Keys in opera4onal data may or may not contain 4me elements

26 Non- volagle A store of data transformed that is physically separated from the opera4onal environment Opera4onal update of data does not occur in the data warehouse environment: Does not require transac4on processing, recovery, and concurrency control mechanisms Requires only two opera4ons in data accessing: ini4al loading of data and access of data

27 OLTP OLTP vs. OLAP OLAP Users Clerk, IT professional Knowledge worker FuncGon Day to day opera4ons Decision support DB design Applica4on- oriented Subject- oriented Data Current, up- to- date, detailed, flat rela4onal, isolated Usage Repe44ve Ad- hoc Access Read/write, index/hash on primary keys Historical, summarized, mul4dimensional, integrated Scan Unit of work Short, simple transac4ons Complex queries Records accessed Tens Millions Users Thousands Hundreds DB size 10Mb- Gb 100Gb- Tb Metric Transac4on throughput Query throughput, response

28 Why a Separate Data Warehouse? High performance for both systems DBMS tuned for OLTP: Access methods, indexing, concurrency control, recovery Warehouse tuned for OLAP: Complex OLAP queries, mul4dimensional view, consolida4on Different func4ons and data: Missing data: DS requires historical data that opera4onal DBs typically do not maintain Data consolida4on: DS requires consolida4on (aggrega4on, summariza4on) of data from heterogeneous sources Data quality: Different sources typically use inconsistent data representa4ons, codes and formats must be reconciled

29 Basic Concepts

30 Data Warehouse: A MulG- Gered Architecture = = = Metadata Monitor & Integrator OLAP Server Other sources Opera4onal DBs Extract Transform Load Refresh Data Warehouse Serve Analysis Query Reports Data mining Data Marts Data Sources Data Storage OLAP Engine Front- end Tools

31 Data Warehouse Models Enterprise warehouse: Collects all of the informa4on about subjects spanning the en4re organiza4on Data mart: A subset of corporate- wide data that is of value to a specific groups of users. Its scope is confined to specific selected groups, such as marke4ng data mart Independent vs dependent (directly from warehouse) data mart Virtual warehouse: A set of views over opera4onal databases Only some of the possible summary views may be materialized

32 ExtracGon, TransformaGon, and Loading (ETL) Data extrac4on: Get data from mul4ple heterogeneous external sources Data cleaning: Detect errors in the data and rec4fy them when possible Data transforma4on: Convert data from legacy or host format to warehouse format Load: Sort, summarize, consolidate, compute views, check integrity, and build indices and par44ons Refresh: Propagate the updates from the data sources to the warehouse

33 Metadata Repository Metadata is the data defining warehouse objects: Descrip4on of the structure of the data warehouse (schemas, views, dimensions, hierarchies, derived data defini4ons, etc.) Opera4onal metadata (data lineage, currency of data, monitoring informa4on Algorithms used for summariza4on Mapping from opera4onal environment to the data warehouse Data related to system performance (warehouse schema, view and derived data defini4ons) Business data (business terms and defini4ons, ownership of data, charging policies)

34 Data Cubes and OLAP

35 From Tables to Data Cubes A data warehouse is based on a mul4dimensional data model which views data in the form of a data cube A data cube allows data to be modeled and viewed in mul4ple dimensions A data cube contains: Dimension tables Fact table containing measures and keys to each of the related dimension tables An n- D base cube is called a base cuboid. The top- most 0- D cuboid, which holds the highest- level of summariza4on, is called the apex cuboid. The lagce of cuboids forms a data cube

36 Example (2- D view) Time (quarter) Home entertainment LocaGon = Vancouver Item (type) Computer Phone Security Q Q Q Q Cross TabulaGon (Dimensions tabulated across one another)

37 Example (3- D view) LocaGon = Chicago LocaGon = New York LocaGon = Toronto LocaGon = Vancouver Item Item Item Item Time Home ent. Comp. Phone Sec. Home ent. Comp. Phone Sec. Home ent. Comp. Phone Sec. Home ent. Comp. Phone Sec. Q Q Q Q

38 Example 0- D (appex) cuboid 1- D cuboids 2- D cuboids 3- D cuboids 4- D (base) cuboids

39 A Concept Hierarchy (LocaGon) all all Region Europe North America Country Germany Spain Canada Mexico City Frankfurt Vancouver Toronto Office L. Chan M. Wind

40 MulGdimensional Data Sales volume as a func4on of product, month, and region Dimensions: Product, Loca4on, Time Hierarchical summariza4on paths: Type Category Item Country State City Street Year Quarter Month Day Week

41 Typical OLAP OperaGons Roll up (drill- up): Summarize data by climbing up hierarchy or by dimension reduc4on Drill down (roll down): Go from less detailed to more detailed data or introduce new dimensions (reverse of roll- up) Slice: Project Dice: Select Pivot: Reorient the cube Other opera4ons: Drill across: Involves summariza4on/aggrega4on across mul4ple fact tables Drill through: Involves going through the bonom level of the cube to its back- end rela4onal tables

42 Example: Roll- up Drill- down Slice Dice Pivot

43 A Starnet Query Model

44 Browsing a Data Cube Visualiza4on OLAP capabili4es Interac4ve manipula4on

45 Finally

46 Summary Data Reduc4on Numerosity Reduc4on: Reduce data volume using smaller forms of data representa4on Data discre4za4on: Reduces and simplifies original data Data warehousing: A mul4- dimensional model of a data warehouse Data cube: Dimensions & measures OLAP opera4ons: drilling, rolling, slicing, dicing and pivo4ng

47 Next Class (Data Warehouse) Architecture