VISUALIZATION TECHNIQUES UTILIZING THE SENSITIVITY ANALYSIS OF MODELS

Transcription

1 VISUALIZATION TECHNIQUES UTILIZING THE SENSITIVITY ANALYSIS OF MODELS Ivo Kondapaneni, Pavel Kordík, Pavel Slavík Department of Computer Science and Engineering, Faculty of Eletrical Engineering, Czech Technical University in Prague, Czech Republic Presenting author: Pavel Kordík

2 Overview Motivation Data mining models Visualization based on sensitivity analysis Regression problems Classification problems Definition of interesting plots Genetic search for 2D and 3D plots 2

3 Motivation Data mining extracting new, potentially useful information from data DM Models are automatically generated Are models always credible? Are models comprehensible? How to extract information from models? Visualization 3

4 Data mining models Often black-box models generated from data E.g. Neural networks Input variables What is inside? Data mining black box model Output variable (s) 4

5 Inductive model Estimates output from inputs Generated automatically Evolved by niching GA Grows from minimal form Contains hybrid units Several training methods Ensemble of models input variables P L C first layer P C P G second layer 3 inputs max P P C third layer interlayer connection 4 inputs max L output layer output variable 5

6 Example: Housing data Input variables CRIM ZN INDUS NOX RM AGE DIS RAD TAX PTRATIO B LSTA Per capita crime rate by town Proportion of owner-occupied units built prior to 1940 Weighted distances to five Boston employment centres MEDV Output variable Median value of owner-occupied homes in $1000's 6

7 Housing data records Input variables CRIM ZN INDUS NOX RM AGE DIS RAD TAX PTRATIO B LSTA MEDV Output variable 7

8 Housing data inductive model Input variables CRIM ZN INDUS NOX RM AGE DIS RAD TAX PTRATIO B LSTA Niching genetic algorithm evolves units in first layer sigmoid sigmoid Error: 0.13 Error: 0.21 MEDV=1/(1-exp(-5.724*CRIM )) MEDV=1/(1-exp(-5.861*AGE )) MEDV Output variable 8

9 Housing data inductive model Input variables CRIM ZN INDUS NOX RM AGE DIS RAD TAX PTRATIO B LSTA sigmoid sigmoid sigmoid linear Error: 0.13 Error: 0.21 Error: 0.24 Error: 0.26 polyno mial Niching genetic algorithm Error: 0.10 evolves units in second layer MEDV=0.747*(1/(1-exp(-5.724*CRIM ))) *(1/(1-exp(-5.861*AGE ))) MEDV Output variable 9

10 Housing data inductive model Input variables CRIM ZN INDUS NOX RM AGE DIS RAD TAX PTRATIO B LSTA sigmoid sigmoid sigmoid linear polyno mial polyno mial expo nential linear Constructed model has very low validation error! Error: 0.08 MEDV Output variable 10

11 Housing data inductive model Input variables CRIM ZN INDUS NOX RM AGE DIS RAD TAX PTRATIO B LSTA MEDV=(exp((0.038* 3.451*(1/(1-exp(-5.724*CRIM )))*(1/(1- S S S L exp(2.413*dis-2.581)))*(1/(1-exp(2.413*dis-2.581)))+0.429*(1/(1- exp(-5.861*age )))+0.024*(1/(1-exp(2.413*DIS ))) P 0.038*0.350*(1/(1-exp(-3.613*RAD-0.088))) *( 0.747*(1/(1-exp(-5.724*CRIM )))+0.582*(1/(1-exp( *AGE+ P 2.111)))*(1/(1-exp(-5.861*AGE+ L 2.111)))+0.016) *(1/(1-exp(-5.724*CRIM ))) *INDUS *LSTA )*0.860)*13.072) E Error: 0.08 Math equation is not comprehensible MEDV any more we have to treat it as a black box model! Output variable 11

12 Visualization based on sensitivity analysis constant x 1 y k x 1 = x 3 = const. moving x 2 y k constant x 3 ModGMDH GAME min x 2 max moving moving x 1 x 2 y m y m x 3 = const. max x 1 constant x 3 ModGMDH GAME min x 2 max 12

13 Sensitivity analysis of inductive model of MEDV House no. 189 House no. 164 Credible output? What will happen with the value of house when criminality in the area decreases/increases? 13

14 Ensemble of inductive models Random initialization y k x1 x2 y k-1 Developing on the same training set y k-1 x3 x1 x2 ModGMDH GAME y k Training affect just well defined areas of input space y k+1 min i = x 2 max x3 x1 x2 x3 ModGMDH GAME ModGMDH GAME y k+1 Each model - unique architecture, similar complexity similar transfer functions Similar behavior for well defined areas Different behavior under-defined defined areas 14

15 Credibility of models: Artificial data set Advantages: No need of the training data set, Modeling method success considered, Inputs importance considered. Credibility: the criterion is a dispersion of models` responses. 15

16 Example: Models of hot water consumption 16

17 Cold water consumption, increasing humidity 17

18 Models on Housing data Single model Ensemble of 10 models Before After 18

19 Classification problems Data: Setoza class Virginica class Versicolor class Blue GAME model (Iris Setoza class) output = 1 decision boundary output = 0 Petal width Petal length 19

20 Credibility of classifiers GAME model 1 GAME model 2 GAME model 3 GAME models (1*2*3) Iris Setoza * * = Iris Virginica * * = Iris Versicolor * * = 20

21 Overlapping models X ensemble 21

22 Random behavior filtered out Before After 22

23 Problem how to find information in n-dim n space? Multidimensional space of input variables What we are looking for? Interesting relationship of IO variables Regions of high sensitivity Credible models (compromise response) Can we automate the search? 23

24 When a plot is interesting for us? x istart x isize x i 24

25 Definition of interesting plot Minimal volume of the envelope p min Maximal sensitivity of the output to the change of x i input variable y size max Maximal size of the area x isize max 25

26 Multiobjective optimization Interestingness: Unknown variables: x 1,x 2,..., x i-1,x i+1, x n x istart, x isize We will use Niching genetic algorithm Chromosome: x 1 x 2... x i-1 x i+1 x n x istart x isize 26

27 Niching GA on simple data Training data Chromosome: x start x size fitness = x size * 1/p * y size Very simple problem Search space is 2D, can be visualized x start GAME ensemble x size fitness 27

28 Niching GA locates also local optima Three subpopulations (niches) of individuals survived 28

29 Automated retrieval of plots showing interesting behavior Genetic Algorithm Genetic algorithm with special fitness function is used to adjust all other inputs (dimensions) Best so far individual found (generation 0 17) 29

30 Housing data interesting plot retrieved Low fitness High fitness Before After 30

31 Conclusion Credible regression Credible classification Automated retrieval 31

32 Future work I Automated knowledge extraction from data KNOWLEDGE EXTRACTION and INFORMATION VISUALIZATION KNOWLEDGE INPUT DATA AUTOMATED DATA PREPROCESSING AUTOMATED DATA MINING FAKE INTERFACE GAME ENGINE 32

33 Future work II FAKE GAME framework DATA WAREHOUSING DATA INTEGRATION Classes boundaries, relationship of variables Classification, Prediction, Identification and Regression DATA CLEANING INPUT DATA DATA INSPECTION AUTOMATED DATA PREPROCESSING MO DEL MO DEL MO DEL GAME ENGINE MO DEL MO DEL MO DEL Credibility estimation Math equations DATA COLLECTION Interesting behaviour PROBLEM IDENTIFICATION FAKE INTERFACE Feature ranking 33

34 Future work III Just released as open source project Automated data preprocessing Automated model building, validation Optimization methods Visualization see and join us: 34