IV. MODELS FROM DATA: Data mining

Size: px

Start display at page:

Download "IV. MODELS FROM DATA: Data mining"

Julia Carter
5 years ago
Views:

1 IV. MODELS FROM DATA: Data mining 1 Outline A) THEORETICAL BACKGROUND 1. Knowledge discovery in data bases (KDD) 2. Data mining Data Patterns Data mining algorithms B) PRACTICAL IMPLEMENTATIONS 3. Applications: Equations Decision trees Rules 2 1

Methods Classic predictive Methods - Data

, 1991: KDD is the non-trivial process of

data? Data mining (DM) central step in the

computational techniques to actually find

2 Methods Classic predictive Methods - Data models Hypothesis Model Knowledge Model - Hypothesis discovery in data Data New Knowledge Marko Debeljak Knowledge discovery in data bases (KDD) KDD-data mining Frawley et al., 1991: KDD is the non-trivial process of identifying valid, novel, potentially useful, and ultimately understandable patterns in data, How to find patters in data? Data mining (DM) central step in the KDD process concerned with applying computational techniques to actually find patterns in the data (15-25% of the effort of the overall KDD process). - step 1: preparing data for DM (data preprocessing) - step 3: evaluating the discovered patterns (results of DM) 4 2

3 Knowledge discovery in data bases (KDD) When the patterns can be treated as knowledge? Frawley et al., (1991): A pattern that is interesting (according to a user- imposed interest measure) and certain enough (again according to the user s criteria) is called knowledge. Condition 1: Discovered patterns should be valid on new data with some degree of certainty (typically prescribed by the user). Condition 2: The patterns should potentially lead to some useful actions (according to user defined utility criteria). 5 Knowledge discovery in data bases (KDD) What can KDD and data mining contribute to environmental sciences (ES) (e.g. agronomy, forestry, ecology, )? Environmental sciences deal with complex unpredictable natural systems (e.g. arable, forest and water ecosystems) in order to get answers on complex questions. The amount of collecting environmental data is increasing exponentially. KDD was purposefully designed to cope with such complex questions about complex systems like: - Understanding the domain/system studied (e.g., gene flow, seed bank, life cycle, ) - Predicting future values of system variables of interest (e.g., the rate of out-crossing with GM plants at location x at time y, seedbank dynamics, ) 6 3

4 What is data mining? Data mining focuses on the discovery of previously unknown knowledge and integrates machine learning. Machine learning focuses on descriptions and prediction, based on known properties learned from the training empirical data (examples) using computer algorithms. Learning from examples is called inductive learning. If the goal of inductive learning is to obtain a model that predicts the value of the target variable from learning examples, then it is called predictive or supervised learning. Data mining (DM) What is data mining from a technical perspective? Data Mining, is the process of automatically searching large volumes of data for patterns using algorithms. Data Mining Machine learning Data Mining is the application of Machine Learning techniques to data analysis problems. The most relevant notions of data mining: 1. Data 2. Patterns 3. Data mining algorithms 8 4

This image cannot currentl y be display ed. This image cannot currentl y be display ed. Data mining (DM) - data Data stored in one flat table. Each example represented by a fixed number of attributes.

RELATIONAL data mining PROPOSITIONAL data mining Loss of information due to aggregation Objects Properties of objects Distance (m) Wind direction ( 0 ) Wind speed (m/s) Out-crossing rate (%) 10 123 3

5 This image cannot currentl y be display ed. This image cannot currentl y be display ed. Data mining (DM) - data Data stored in one flat table. Each example represented by a fixed number of attributes. Data stored in original tables or relations. No loss of information, due to aggregation. RELATIONAL data mining PROPOSITIONAL data mining Loss of information due to aggregation Objects Properties of objects Distance (m) Wind direction ( 0 ) Wind speed (m/s) Out-crossing rate (%) Marko Debeljak WP2 contributions to SIGMEA Final Meeting, Cambridge, UK, April, Data Data are not stored at all but they continuously flow through an algorithm. Each example can be propositional or relational. DATA STREAM mining Marko Debeljak WP2 contributions to SIGMEA Final Meeting, Cambridge, UK, April,

6 Data mining (DM) - pattern 2. What is a pattern? A pattern is defined as: A statement (expression) in a given language, that describes relationships among the facts (attributes, elements) in a subset of the given data and is (in some sense) simpler than the enumeration of all facts in the subset (Frawley et al. 1991, Fayyad et al. 1996). =model Classes of patterns considered in data mining: A. equations, B. decision trees, relational decision trees C. association, classification, and regression rules. 11 Data mining (DM) - pattern A. Equations To predict the value of a target (dependent) variable as a linear or nonlinear combination of the input (independent) variables: - Algebraic equations To predict the behavior of dynamic systems, which change their rates over time: - Difference equations - Differential equations 12 6

Data mining (DM) - pattern B. Decision trees To predict the value of one or several target dependent variables from the values of other independent variables by a decision tree.

7 Data mining (DM) - pattern B. Decision trees To predict the value of one or several target dependent variables from the values of other independent variables by a decision tree. Decision tree has a hierarchical structure, where: - each internal node contains a test on an independent variable(s), - each branch corresponds to an outcome of the test (critical values of independent variable(s)), - each leaf gives a prediction for the value of the dependent (predicted) variable(s). 13 Data mining (DM) - pattern The decision tree is called: A classification tree: class value in the leaf is discrete (a finite set of nominal values): e.g., (yes, no), (spec. A, spec. B, ) A regression tree: class value in leaf is a constant (infinite set of values): e.g.,120, 220, 312, A model tree: leaf contains linear model predicting the class value (piecewise linear function): out-crossing rate= 12.3 distance wind speed wind direction 14 7

8 Data mining (DM) - pattern 3. Rules To perform association analysis between attributes discovered by association rules. The rule denotes patterns of the form: IF Conjunction of conditions THEN Conclusion. -For classification rules, the conclusion assigns one of the possible discrete values to the class (a finite set of nominal values): e.g., (yes, no), (spec. A, spec. B, spec. D) -For predictive rules, the conclusion gives a prediction for the value of the target (class) variable (infinite set of values): e.g., 120, 220, 312, 15 Data mining (DM) - algorithm 3. What is data mining algorithm? Algorithm in general: -A procedure (a finite set of well-defined instructions) for accomplishing some task which will terminate in a defined endstate. Data mining algorithm: - A computational process defined by a Turing machine (Gurevich et al. 2000) for finding patterns in data 16 8

9 Data mining (DM) - algorithm What kind of possible algorithms do we use for discovering patterns? Selection of the algorithm depends on the problem at hand: 1. Equations = Linear and multiple regressions, equation discovery 2. Decision trees = Top/down induction of decision trees 3. Rules = Rule induction 17 Data mining (DM) - algorithm 1. Linear and multiple regression Bivariate linear regression: Predicted variable (C-class (ML) may be contusions or discontinues) can be expressed as a linear function of one attribute (A): C = α+ β A Multiple regression: Predicted variable (C-class (ML) may be contusions or discontinues) can be expressed as a linear function of a multi-dimensional attribute vector (AI): C = Σ n i=1 β i A i 18 9

10 Data mining (DM) - algorithm 2. Top/down induction of decision trees The decision tree is induced by Top-Down Induction of Decision Trees (TDIDT) algorithm (Quinlan, 1986) Tree construction proceeds recursively starting with the entire set of training examples (entire table). At each step, an attribute is selected as the root of the (sub) tree and the current training set is split into subsets according to the values of the selected attribute. 19 Data mining (DM) - algorithm 3. Rule induction A rule that correctly classifies some examples is constructed first. The positive examples covered by the rule from the training set are removed and the process is repeated until no more examples remain

11 Data mining (DM) - Statistics Data mining vs. Statistics Common to both approaches: Reasoning FROM properties of a data sample TO properties of a population. 21 Data mining (DM) Machine learning - Statistics Statistics Hypothesis testing when certain theoretical expectations about the data distribution, independence, random sampling, sample size, etc. are satisfied. Main approach: best fitting all the available data. Data mining Automated construction of understandable patterns, and structured models. Main approach: structuring the data space, heuristic search for decision trees, rules, covering (parts of) the data space

12 DATA MINING CASE STUDIES 23 Applications ecological modeling Propositional and relational supervised data mining: - Simple data mining - Data mining of time series - Spatial data mining 1. Equations: Algebraic equations Differential equations 2. Single and multi target decision trees: Classification trees Regression trees Model trees (single target only) 12

13 Applications ecological modeling POPULATION DYNAMICS HABITAT MODELLING GENE FLOW MODELLING RISK MODELLING Algebraic equations: SOIL WATER FLOW Problem: Prediction of drainage water Type of pattern: Algebraic equation Algorithm: CIPER 13

PCQE Database Experimental site La Jaillière Western France Owned by ARVALIS Shallow silt clay soils 11 fields are observed Field size about 0.

14 PCQE Database Experimental site La Jaillière Western France Owned by ARVALIS Shallow silt clay soils 11 fields are observed Field size about ha Introduction Domain & Problem Related work Methodology Data sources Experimental design Results Discussion Further work PCQE Database (continued) Agricultural practices Fertilization Irrigation Phytochemical protection Harvesting Tillage Slope Water flow Drainage Runoff 25 campaigns ( ) The campaign is defined as the period starting from 01 SEP and finishing on 31 AUG, following year Introduction Domain & Problem Related work Methodology Data sources Experimental design Results Discussion Further work 14

15 DRAINAGE predictive model - CIPER Polynomial equations induced on data for a whole campaign CIPER algorithm Evaluation Leave one out approach Fields Test field Std. Dev. RMSE RRSE Corr. coeff. (r) All T % All T % All T % All T % All T % All T % All T % All T % All T % T3 T % T6 T % Introduction Domain & Problem Related work Methodology Data sources Experimental design Results Discussion Further work Predictive models Model (All/T4) Drainage = * RainfallA1 * Temp * DrainageN * CDCoef 2 * RainfallA1 2 * Slope * Runoff * DrainageN1 * Temp 2 * RainfallA * Runoff 2 * DrainageN1 * Slope * Temp 2 * RainfallA * RainfallA * Runoff * Slope * Runoff * Slope 2 * RainfallA * RainfallA1 * Slope * Slope 3 * CDCoef 3 * Runoff * Slope Introduction Domain & Problem Related work Methodology Data sources Experimental design Results Discussion Further work 15

16 Predictive models Introduction Domain & Problem Related work Methodology Data sources Experimental design Results Discussion Further work Constraint algebraic equations: GENE FLOW 16

17 Constraint algebraic equations: GENE FLOW Problem: Prediction of gene flow Type of pattern: Constraint algebraic equation Algorithm: Lagramge Constraint algebraic equations: GENE FLOW Experimental design: Federal Biological Research Centre, BBA, D Field design 2000 transgenic field / donor non-transgenic field / recipient access paths 2 m 100 m 220 m 96 points 6 sampling point 3 system o f coordinates a for the sampling points 60 cobs 2500 kernels n o p q 3 a 1 m N b % of outcrossing l k Donors Direction of drilling- GMO corns c d 4.5 m 3 m 7.5 m 13.5 m 2 m j h g f e 25,5m 49,5 m Receptors - NT corns 17

18 Constraint algebraic equations: GENE FLOW Constraint algebraic equations: GENE FLOW 18

19 e cannot currently be displayed. Differential equations: COMMUNITY STRUCTURE Differential equations: COMMUNITY STRUCTURE Problem: Time dependent ecosystem processes Type of pattern: Differential equations Algorithm: LAgramge 19

20 Differential equations: COMMUNITY STRUCTURE Differential equations: COMMUNITY STRUCTURE Data: 1995 to

21 Differential equations: COMMUNITY STRUCTURE Phosphorus Water inflow out-flow Respiration Growth Differential equations: COMMUNITY STRUCTURE Phytoplankton Growth Respiration Sedimentation Grazing 42 21

22 Differential equations: COMMUNITY STRUCTURE Zooplankton Feeds on phytoplankton Respiration Mortality Differential equations: COMMUNITY STRUCTURE 22

23 EQUATIONS Algebraic - CIPER Constraint algebraic- Lagramge Differential- Lagramge SUITABLE for predictions, UNSUITABLE for interpretation Decision trees: HABITAT MODELS 23

24 24 Problem: Classification of habitats (suitable, unsuitable) Type of pattern: Classification decision trees Algorithm: J4.8 Decision trees: HABITAT MODELS Observed locations of BBs Decision trees: HABITAT MODELS

25 Decision trees: HABITAT MODELS The training dataset Positive examples: - Locations of bear sightings (Hunting Association; telemetry) - Females only - Using home-range (HR) areas instead of raw locations - Narrower HR for optimal habitat, wider for maximal Negative examples: - Sampled from the unsuitable part of the study area - Stratified random sampling - A different land cover types equally accounted for Decision trees: HABITAT MODELS Propositional dataset 1,73,26,0,0,1,88,0,2,70,7,20,1,0,0,1,0,60,0,0,0,0,0,2,0,0,0,4123,0,0,0,0,63,211,11,11,11,83,213,213,0,0, ,62,37,0,0,2,88,0,2,70,7,20,1,0,0,1,0,60,0,0,0,0,0,2,1,53,0,3640,0,0,0,-1347,63,211,11,11,11,83,213,213,11,89, ,0,99,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,6,82,0,10404,0,2074,-309,48,0,0,11,11,11,83,83,83,0,20, ,0,100,0,0,1,76,0,16,71,0,12,0,0,0,0,0,0,0,0,0,0,0,1,6,82,0,7500,0,1661,-319,-942,0,0,11,11,11,0,0,0,0,20, ,8,91,0,0,1,52,0,59,41,0,0,0,0,0,0,0,4,0,0,0,0,5,1,6,82,0,6500,0,1505,-166,879,9,57,11,11,11,281,281,281,0,20, ,3,0,86,9,0,75,0,33,67,0,0,0,0,0,0,1,2,0,0,0,0,0,1,2,54,0,0,0,465,-66,-191,4,225,11,31,31,41,72,272,60,619, ,34,65,0,0,2,51,9,76,9,5,1,4,1,0,1,0,29,0,0,0,0,0,1,2,54,0,3000,0,841,-111,-264,34,220,11,41,41,151,141,112,60,619, ,100,0,0,0,3,52,0,86,6,3,5,9,6,7,38,40,0,0,0,0,0,0,1,17,64,0,8062,0,932,-603,-71,100,337,11,41,41,171,232,202,4,24, Present: 1 Absent: 0 25

26 The model for maximal habitat The model for optimal habitat 26

27 Decision trees: HABITAT MODELS Map of maximal habitat (39% SLO territory) Decision trees: HABITAT MODELS Map of optimal habitat (13% SLO territory) 54 27

Multi target predictions: COMMUNITY STRUCTURE Decision trees: COMMUNITY STRUCTURE Problem: Temporal prediction of height and cover of crop and

28 Multi target predictions: COMMUNITY STRUCTURE Decision trees: COMMUNITY STRUCTURE Problem: Temporal prediction of height and cover of crop and weeds Type of pattern: a) Multi target regression trees / time series b) Constraint multi target regression clustering trees Algorithm: CLUS 28

Data 130 sites, monitoring every 7 to 14 days for 5 months (2665 samples: 1322 conventional, 1333, HT OSR observations) Each sample (observation) described by 65 attributes Original data collected by

29 Data 130 sites, monitoring every 7 to 14 days for 5 months (2665 samples: 1322 conventional, 1333, HT OSR observations) Each sample (observation) described by 65 attributes Original data collected by the Centre for Ecology and Hydrology, Rothamsted Research and SCRI within the Farm Scale Evaluation Program (2000, 2001, 2002) Results scenario A: Multiple target regression tree Target: Avg Crop Covers, Avg Weed Covers Excluded attributes: / Constraints: MinimalInstances = 64.0; MaxSize = 15 Predictive power: Corr.Coef.: , RMSE: , RRMSE: ,

30 Results scenario B: Constraint predictive clustering trees for time series Syntactic constraint Results scenario B: Constraint predictive clustering trees for time series Target: Avg Weed Covers (Time Series) Scenario 3.9 Constraints: Syntactic, MinInstances = 32 Predictive power: TSRMSExval: 4.98 TSRMSEtrain: 4.86 ICVtrain:

31 Results scenario B: Constraint predictive clustering trees for time series Results scenario B: Constraint predictive clustering trees for time series 31

Relational data mining: GENE FLOW MODELLING Marko Debeljak WP2 contributions to SIGMEA Final Meeting, Cambridge, UK, 16-19 April, 2007 Relational data mining: GENE

32 Relational data mining: GENE FLOW MODELLING Marko Debeljak WP2 contributions to SIGMEA Final Meeting, Cambridge, UK, April, 2007 Relational data mining: GENE FLOW MODELLING Problem: Classification of fields to above or below 0.9% of outcrossing Type of pattern: Relational classification decision tree Algorithm: TILDE 32

Spatial temporal relations 2004: 40 GM fields 7 non-gm fields 181 sampling points 2005: 17 GM fields 4 non-gm fields 127 sampling points 2006: 43 GM fields 4 non-gm fields 112 sampling points Marko

33 Spatial temporal relations 2004: 40 GM fields 7 non-gm fields 181 sampling points 2005: 17 GM fields 4 non-gm fields 127 sampling points 2006: 43 GM fields 4 non-gm fields 112 sampling points Marko Debeljak WP2 contributions to SIGMEA Final Meeting, Cambridge, UK, April, 2007 Relational data mining: GENE FLOW MODELLING Data scattered over several tables or relations: A table storing general information on each field (e.g., area) A table storing the cultivation techniques for each field and each year A table storing the relations (e.g., distance) between fields Marko Debeljak WP2 contributions to SIGMEA Final Meeting, Cambridge, UK, April,

34 Relational data mining: GENE FLOW MODELLING Marko Debeljak WP2 contributions to SIGMEA Final Meeting, Cambridge, UK, April, 2007 Relational data mining: GENE FLOW MODELLING Relation database system PostGIS Marko Debeljak WP2 contributions to SIGMEA Final Meeting, Cambridge, UK, April,

35 Relational data mining building model Relation data analysis: Algorithm Tilde (Blockeel and De Raedt, 1998; De Raedt et al., 2001) => upgrade of algorithm C.4.5 (Quinlan, 1993) for classification decision trees The algorithm is included in the ACE-ilProlog data mining system Marko Debeljak WP2 contributions to SIGMEA Final Meeting, Cambridge, UK, April, 2007 Relational data mining results Threshold 0.01% Threshold 0.45% Threshold 0.9% Marko Debeljak WP2 contributions to SIGMEA Final Meeting, Cambridge, UK, April,

36 Multi-target regression model: RISK MODELLING Decision trees: RISK MODELLING Problem: Prediction of soil exposed to disturbances Type of pattern: Multi target regression trees (resistance, resilience) Algorithm: CLUS 36

Multi-target regression model: RISK MODELLING The dataset: soil samples taken at 26 locations throughout SCO The dataset: The flat table of data: 26 of 18 data entries The dataset: Multi-target

37 Multi-target regression model: RISK MODELLING The dataset: soil samples taken at 26 locations throughout SCO The dataset: The flat table of data: 26 of 18 data entries The dataset: Multi-target regression model: RISK MODELLING Physical properties: soil texture: sand, silt, clay Chemical properties: ph, C, N, SOM (soil organic matter) FAO soil classification: Order and Suborder Physical resilience: resistance to compression: 1/Cc, recovery from compression: Ce/Cc, overburden stress: e.g., recovery from overburden stress after two day cycles: eg2dc Biological resilience: heat, copper 37

38 Multi-target regression model: RISK MODELLING Multi-target regression model: RISK MODELLING Macaulay Institute (Aberdeen): soils data attributes and maps: Approximately soil profiles held in database Descriptions of over soil horizons 38

39 Application 77 Application 78 39

40 Application 79 Decision trees Single or multiple decision trees Classification, regression, model trees Propositional and relational data Temporal and spatial data SUITABLE for predictions, SUITABLE for interpretation 40

41 Conclusions What can data mining do for you? Knowledge discovered by analyzing data with DM techniques can help: Understand the domain studied Make predictions/classifications Support decision processes in environmental management Conclusions What data mining cannot do for you? The law of information conservation (garbage-ingarbage-out) The knowledge we are seeking to discover has to come from the combination of data and background knowledge If we have very little data of very low quality and no background knowledge no form of data analysis will help 41

42 Conclusions Side-effects? Discovering problems with the data during analysis Missing values Erroneous values Inappropriately measured variables Identifying new opportunities New problems to be addressed Recommendations on what data to collect and how DATA MINING data preprocessing DATA FORMAT File extension.arff This a plain text format, files should be edited by editors such as Notepad, TextPad, WordPad (that do not add extra formatting information) The file consists of NameOfDataset List of AttName AttType AttType can be numeric or nominal list of categorical values, e.g., {red, green, blue} (in a separate line), followed by the actual data in comma separated value (.csv) format 84 42

43 DATA MINING data preprocessing DATA outlook {sunny, overcast, temperature humidity windy {TRUE, play {yes, sunny,85,85,false,no sunny,80,90,true,no overcast,83,86,false,yes 85 DATA MINING data preprocessing Excel Attributes (variables) in columns and cases in lines Use decimal POINT and not decimal COMMA for numbers Save excel sheet as CSV file 86 43

44 DATA MINING data preprocessing TextPad, Notepad Open CSV file Delete on the beginning of lines and save (just save, don t change the format) Change all ; to, The numbers must have decimal dot (.) and not a decimal comma (,) Save file as CSV file (don't change the format) 87 DATA MINING Hands-on exercises 2. Data mining 88 44

DATA MINING data preprocessing WEKA http://www.cs.waikato.ac.nz/ml/weka/index.

tree? Performance of the classification tree: Classification accuracy: (Correctly classified examples) /(all examples) 1 0.

45 DATA MINING data preprocessing WEKA Open CVS file in WEKA Select algorithm and attributes Perform data mining 89 How to select the best classification tree? Performance of the classification tree: Classification accuracy: (Correctly classified examples) /(all examples) True positive rate False positive rate Confusion matrix is a matrix showing actual and predicted classifications 90 45

$) where instances are fractured Or (0,0) a split on a nominal attribute and one or more of the values do not occur in the subset of instances at the node$

46 How to select the best classification tree? Classification trees: J48 Interpretable size: -Pruned or unpruned - Minimal number of objects per leaf The number of instances classified in this leaf The number of instances INCORRECTLY classified in this leaf It could appear: (13) no incorrectly classified instances Or (3.5/0.5) due to missing values (?) where instances are fractured Or (0,0) a split on a nominal attribute and one or more of the values do not occur in the subset of instances at the node in question 91 How to select the best regression / model tree? The performance of the regression / model tree: 92 46

from the leaf's linear model for the instances that reach the leaf, expressed as a percentage of the global standard deviation of the class attribute (i.e. the standard deviation of the class attribute computed from all the training data).

47 How to select the best regression / model tree? The interpretable size: -Pruned or unpruned - Minimal number of objects per leaf The number of instances that REACH this leaf Root of the mean squared error (RMSE) of the predictions from the leaf's linear model for the instances that reach the leaf, expressed as a percentage of the global standard deviation of the class attribute (i.e. the standard deviation of the class attribute computed from all the training data). Sum is not 100%. The smaller this value, the better. 93 Accuracy and error Avoid over fitting the data by tree pruning. Pruned trees are: - Less accurate (percentage of correct classifications) of training data - More accurate when classifying unseen data 94 47

48 How to prune optimally? Pre-pruning: stop growing the tree, e.g. when data split not statistically significant or too few examples are in a split (minimum number of objects in leaf) Post-pruning: grow full tree, then post-prune (confidence factor-classification trees) 95 Optimal accuracy 10-fold cross-validation is a standard classifier evaluation method used in machine learning: - Break data into 10 sets of size n/10. - Train on 9 datasets and test on 1. - Repeat 10 times and takes a mean accuracy

IV. MODELS FROM DATA. MODELLING METHODS- Data mining. Data mining. Outline A) THEORETICAL BACKGROUND B) PRACTICAL IMPLEMENTATIONS

IV. MODELS FROM DATA. MODELLING METHODS- Data mining. Data mining. Outline A) THEORETICAL BACKGROUND B) PRACTICAL IMPLEMENTATIONS IV. MODELS FROM DATA Data mining 1 MODELLING METHODS- Data mining data Data mining 2 Outline A) THEORETICAL BACKGROUND 1. Knowledge discovery in data bases (KDD) 2. Data mining Data Patterns Data mining