Data Preparation for Analytics

Transcription

1 Data Preparation for Analytics Dr. Gerhard Svolba SAS-Austria Vienna

2 Agenda Data Preparation for Analytics General Thoughts Data Structures for Analytics Case Study: Building an efficient one-row-persubject data mart Data Management with the SAS Language and with SAS Tools Closing Thoughts

3 Some Words on Data Preparation Is for techies Is just coding Is boring Consumes up to 80 % of the project Was not in the focus of data mining literature so far Is something that SAS can excellently do Is vital to the quality of the project

4 The Analysis Process: From Raw Data to Actionable Results Data Sources Data Preparation Analytic Modeling Results and Actions Different Data Sources Relational Models, Star Schemes Merges, Denormalisation Derived Variables Transpositions, Aggregations Modeling, Parameter Estimation, Tuning, Predictions, Classifications, Clustering Usage of Results Profiling Interpretations Data Availability + Adequate Preparation Clever + Modeling = Good Results

5 Four Dimensions for Analytic Data Preparation Business and Process Knowledge Analytical Knowledge Analytic Data Preparaton Efficient and Clever SAS Coding Documentation and Maintainance

6 Four Dimensions for Analytic Data Preparation Business and Process Knowledge Analytical Knowledge Analytic Data Preparaton Efficient and Clever SAS Coding Documentation and Maintainance

7 Challenging a Business Question Can you calculate the probability that a customer will cancel the usage of product X? Questions: Cancellation or decline in usage? How do we treat usage of more advanced products? Include non-volontary cancellation? How quickly can retention measures take place? Do you want to have probabilites per customer or a list of the high risk customers or risk classes in general?

8 Business, Statistican and IT: The Optimal Triangle!? I have formulated my business question. Are there any reasons, not to be back with results in 2 days? Analytic Data Preparaton I would like to have an analytic database with a high number of attributes and an environment to perform both, data management and analysis. Simon Retention Manager Name me the list of attributes and derived variables that you will use in your final model and which I have to deliver periodically. Daniele Quantitative Expert Elias IT Expert

9 Four Dimensions for Analytic Data Preparation Business and Process Knowledge Analytical Knowledge Analytic Data Preparaton Efficient and Clever SAS Coding Documentation and Maintainance

10 Business Questions Analytical Models Event prediction (Churn, Fraud, Delinquency, Response, ) Value prediction (Purchase Size, Claim Amount, ) Clustering (Segmentation, ) Market Basket Analysis (Association Analysis, ) Time Series Forecasting

11 Analysis Subjects and Multiple Observations Analysis subjects are entities that are being analyzed and the analysis results are interpreted in their context. Multiple observations per analysis subject Repeated measurements over time Multiple observations because of hierarchical relationships

12 Main Types of Data Marts One-Row-per- Subject Data Mart Multiple-Row-per- Subject Data Mart Longitudinal Data Mart

13 Data Mart Structures Data Mart Structure for the Analysis Structure of the source data: Multiple observations per analysis subject exist? NO YES One-Row-per-Subject Data Mart Data Mart with one row per subject is created. Information of multiple observations has to be aggregated or transposed per analysis subject Multiple-Row-per-Subject Data Mart (Key-Value Table) Data Mart with one row per multiple observations is created. Variables at the analysis subject level are duplicated for each repetition

14 The One-Row-Per-Subject Data Mart Required by many statistical methods Regression Analysis, Neural Networks, Decision Trees, Survival analysis, Cluster analysis, Most prominent data mart structure in data mining Event prediction (Churn, Fraud, Delinquency, Response, ) Value prediction (Purchase Size, Claim Amount, ) Segmentation (Clustering, )

15 The One-Row-Per-Subject Paradigm

16 Transposing Data to One-Row-Per-Subject PROC TRANSPOSE DATA = long OUT = wide(drop = _name_) PREFIX = weight; BY id ; VAR weight; ID time; RUN;

17 Clever Aggregations Interval Data Static Aggregation Correlation of Values Course over Time Concentration of Values Categorical Data Frequency Counts Concatenated Frequencies Total and Distinct Counts

18 Correlation of Values How do the monthly values per subject correlate with the overall mean per month?

19 Measures for the Course over Time PROC REG DATA = longitud NOPRINT OUTEST=Est_LongTerm(KEEP = CustID month RENAME = (month=longterm)); MODEL usage = month; BY CustID; RUN; PROC REG DATA = longitud NOPRINT OUTEST=Est_ShortTerm(KEEP = CustID month RENAME = (month=shortterm)); MODEL usage = month; BY CustID; WHERE month in (5 6); RUN;

20 Concentration of Values Concentration = proportion of the sum of the top 50 % sub-hierarchies / the total sum over all sub hierarchies

21 Categorical Variables: Frequency Counts Source Data Absolute and Relative Frequencies Counts and Distinct Counts

22 Categorical Variables: Concatenated Frequencies Account_ Cumulative Cumulative RowPct Frequency Percent Frequency Percent _100_0_ _0_0_ _0_0_ _0_0_ _0_100_ _0_0_ _0_50_ _0_50_

23 Hierarchies: Aggregating Up, Copying Down

24 Main Types of Data Marts One-Row-per- Subject Data Mart Multiple-Row-per- Subject Data Mart Longitudinal Data Mart

25 The standard form for longitudinal data The inter-leaved longitudinal data mart

26 The cross-sectional dimension data mart

27 Longitudinal Data Marts Aggregating data at the right level Transactional Data Finest Granularity Most Appropriate Aggregation Level Defining cross sectional groups Alignment of time values Creation of event indicators and input variables

28 Four Dimensions for Analytic Data Preparation Business and Process Knowledge Analytical Knowledge Analytic Data Preparaton Efficient and Clever SAS Coding Documentation and Maintainance

29 Case Study: Business Question Predict customers that have a high probability to leave the company Derive target variable Cancellation YES/NO from the monthly value segment history (Entry 8. LOST ) Create a one-row-per-subject data mart for data mining analysis

30 Case Study: Data and Data Model Customer data: demographic and customer baseline data Account data: information customer accounts Leasing data: data on leasing information Call Center data: data on Call center contacts Score data: data of value segment scores

31 Considerations for Predictive Modeling Only consider customers that are still active at this point in time Snapshot Date: June 30th, 2003 Do not use input data after this point in time Define Cancellation YES/NO based on values in the target window Observation Window Offset Target Window April 2003 May 2003 June 2003 July 2003 August 2003

32 Using Data from the CALLCENTER Table %let snapdate = 30JUN2003 d; PROC FREQ DATA = callcenter NOPRINT; TABLE CustID / OUT = CallCenterComplaints (DROP = Percent RENAME = (Count = Complaints)); WHERE Category = 'Complaint' and datepart(date) <= &snapdate; RUN;

33 Using Data from the SCORES-Table %let snapdate = 30JUN2003 d; DATA ScoreFuture(RENAME = (ValueSegment = FutureValueSegment)) ScoreActual ScoreLastMonth(RENAME = (ValueSegment = LastValueSegment)); SET Scores; DATE = INTNX('MONTH',Date,0,'END'); DROP Date; IF Date = &snapdate THEN OUTPUT ScoreActual; ELSE IF Date = INTNX('MONTH',&snapdate,-1) THEN OUTPUT ScoreLastMonth; ELSE IF Date = INTNX('MONTH',&snapdate,2) THEN OUTPUT ScoreFuture; RUN; Observation Window Offset Target Window April 2003 May 2003 June 2003 July 2003 August 2003

34 DATA CustomerMart; ATTRIB /* Customer Baseline */ CustID FORMAT = 8. LABEL = "Customer ID" Birthdate FORMAT = DATE9. LABEL = "Date of Birth" Alter FORMAT = 8. LABEL = "Age (years)" Gender FORMAT = $6. LABEL = "Gender" MaritalStatus FORMAT = $10. LABEL = "Marital Status" Title FORMAT = $10. LABEL = "Academic Title" HasTitle FORMAT = 8. LABEL = "Has Title? 0/1" Branch FORMAT = $5. LABEL = "Branch Name"; MERGE Customer (IN = InCustomer) AccountSum (IN = InAccounts) AccountTypes LeasingSum (IN = InLeasing) CallCenterContacts (IN = InCallCenter) CallCenterComplaints ScoreFuture ScoreActual ScoreLastMonth; BY CustID; IF InCustomer;

35 /* Customer Baseline */ HasTitle = (Title ne ""); Alter = (&Snapdate-Birthdate)/365.25; CustomerMonths = (&Snapdate- CustomerSince)/(365.25/12); /* Accounts */ HasAccounts = InAccounts; LoanPct = Loan / BalanceSum * 100; SavingAccountPct = SavingAccount / BalanceSum * 100; FundsPct = Funds / BalanceSum * 100; /* Leasing */ HasLeasing = InLeasing; /* Call Center */ HasCallCenter = InCallCenter; ComplaintPct = Complaints / Calls *100; /* Value Segment */ Cancel = (FutureValueSegment = '8. LOST'); ChangeValueSegment = (ValueSegment = LastValueSegment); RUN;

36 Screenshots of the Resulting Data Mart

37 The Power of the SAS Language for Data Preparation

38 SAS Data Step vs. SQL Transposing a table PROC TRANSPOSE DATA = sampsio.assocs(obs=21) OUT = assoc_tp (DROP = _name_); BY customer; ID Product; RUN;

39 Changing between longitdutinal data structures Standard Form Interleaved

40 Selection of the first and last line per customer CustID Month Value FirstValue LastValue

41 Creating a sequential number and summing items CustID Date Points PurchNo Cum_Poi

42 Copyingomitteddata m w m 26 1 Value Month Gender Age CustID 32 9 m m m w w w m m m 26 1 Value Month Gender Age CustID

43 Shift down a column for k positions Date Value Value_ PrevDay

44 Analytic Data Preparation in SAS Enterprise Miner

45 Analytic Data Preparation in SAS Enterprise Miner Input Data Source Node Sample Node Data Partition Node Metadata Node Filter Node Transform Variables Node Impute Node SAS Code Node Principal Components Node

46 Working with Multiple-Row-per-Subject Data in SAS Enterprise Miner Time Series Node Association Node Merge Node

47 Extending SAS Enterprise Miner with Data Preparation for Analytics Nodes Extension Nodes Correlation TrendRegression Concentration CategoryCount

48 Analytic Data Preparation in SAS Enterprise Miner - Summary Documentation of the data preparation process as a whole in the process flow diagram Definition of variables metadata that are used in the analysis Automatic Creation of Dummy-Variables for categorical data Powerful data transformations in the filter node, transform variables node and impute node. Possibility to perform association analysis and time-series-analysis Creation of score code from all nodes in SAS Enterprise Miner as SAS datastep code

49 Analytic Data Preparation in SAS Forecast Studio

50 Analytic Data Preparation in SAS Forecast Studio Convert transactional data into time series data Treatment of missing values Alignment of date values in intervals Aggregation of data on various levels Handling of events Allows users to create event definitions, assign events to selected series in the project and delete events Users can specify event duration, shape, and recurrence options. Pre-defined common events and holidays are available for inclusion in the forecasting models

51 Four Dimensions for Analytic Data Preparation Business and Process Knowledge Analytical Knowledge Analytic Data Preparaton Efficient and Clever SAS Coding Documentation and Maintainance

52 SAS Data Integration Studio

53 SAS Data Integration Studio

54 SAS Data Integration Studio General Features Comprehensive transformation library Graphical user interface, drag & drop, wizard driven Multi-developer support: Check-in/check-out, change management Impact analysis Documentation of the data management process Data Mining Model Scoring Register Model in SAS metadata repository Apply SAS Enterprise Miner model to new data source Creates the target table definition in the metadata

55 SAS DI Studio Analytic Features Data Mining Model Scoring Register Model in SAS metadata repository Apply SAS Enterprise Miner model to new data source Creates the target table definition in the metadata Forecast Analysis Transformation Allows to perform time series analysis Based on HPF (high performance forecasting) procedure Allows to integrate the forecast step into the data flow process in the metadata

56 Summary Analytic Data Preparation is a discipline, not a incommodious necessity Analytic Data Preparation is more than just coding The One-Row-Per-Subject Paradigm Central in data mining and predictive modeling Do not stop with simple transpose, summing or averaging Clever aggregations can be the key success factor Predictive Modeling and Historic Data: Which data are you allowed to use for modeling? SAS combines powerful data management functionality and market leading analytics in one integrated package SAS Tools like SAS DI-Studio, SAS Enterprise Miner and SAS Forecast Studio assist you in analytic data preparation

57 The commercial break

58 "It is exciting to see a book completely devoted to data preparation in SAS." Jin Li Statistician Capital One Financial Services This is a must read for anyone, who prepares data for data mining and analytics. Not only you receive ideas and suggestions for important derived variables for your datamart, you also get a lot of insight on the business rationale behind the scenes. The author does a great job in explaining you step by step the world of data preparation for data mining and shows a lot of example code and macros. Christine Hallwirth MHE & Partners Gerhard Svolba's book "Data Preparation for Analytics" is an "owner's manual" for data miners, business analysts and all who prefer to be in charge of and responsible for their own datasets. This book is for those who are not afraid of data, who understand data, and for whom rolling up their sleeves and getting their hands into data is an integral part of analytics and predictive modeling. Lessia Shajenko (American Bank)

59 Recommended Reading Data Preparation for Analytics by Gerhard Svolba SAS-Press (#60502) Business Rationale Concepts Coding Examples

60 Downloads Data Preparation for Analytics

61 Questions and Contact Dr. Gerhard Svolba