IV. MODELS FROM DATA. MODELLING METHODS- Data mining. Data mining. Outline A) THEORETICAL BACKGROUND B) PRACTICAL IMPLEMENTATIONS

Transcription

1 IV. MODELS FROM DATA Data mining 1 MODELLING METHODS- Data mining data Data mining 2 Outline A) THEORETICAL BACKGROUND 1. Knowledge discovery in data bases (KDD) 2. Data mining Data Patterns Data mining algorithms B) PRACTICAL IMPLEMENTATIONS 3. Applications: Equations Decision trees Rules 3 1

2 Knowledge discovery in data bases (KDD) What is KDD? Frawley et al., 1991: KDD is the non-trivial process of identifying valid, novel, potentially useful, and ultimately understandable patterns in data, How to find patters in data? Data mining (DM) central step in the KDD process concerned with applying computational techniques to actually find patterns in the data (15-25% of the effort of the overall KDD process). - step 1: preparing data for DM (data preprocessing) - step 3: evaluating the discovered patterns (results of DM) 4 Knowledge discovery in data bases (KDD) When the patterns can be treated as knowledge? Frawley et al., (1991): A pattern that is interesting (according to a user- imposed interest measure) and certain enough (again according to the user s criteria) is called knowledge. Condition 1: Discovered patterns should be valid on new data with some degree of certainty (typically prescribed by the user). Condition 2: The patterns should potentially lead to some useful actions (according to user defined utility criteria). 5 Knowledge discovery in data bases (KDD) What may KDD contribute to environmental sciences (ES) (e.g. agronomy, forestry, ecology, )? ES deal with complex unpredictable natural systems (e.g. arable, forest and water ecosystems) in order to get answers on complex questions. The amount of collected environmental data is increasing exponentially. KDD was purposively designed to cope with such complex questions about complex systems like: - understanding the domain/system studied (e.g., gene flow, seed bank, life cycle, ) - predicting future values of system variables of interest (e.g., rate of out-crossing with GM plants at location x at time y, seedbank dynamics, ) 6 2

3 Data mining (DM) What is data mining? Data Mining, is the process of automatically searching large volumes of data for patterns using algorithms. Data Mining Machine learning Data Mining is the application of Machine Learning techniques to data analysis problems. The most relevant notions of data mining: 1. Data 2. Patterns 3. Data mining algorithms 7 Data mining (DM) - data 1. What is data? According to Fayyad et al. (1996): Data is a set of facts, e.g., cases in a database. Data in DM is given in a single flat table: - rows: objects or records (examples in ML) - columns: properties of objects (attributes, features in ML) which is then used as input to a data mining algorithm. Objects Properties of objects Distance (m) Wind direction ( 0 ) Wind speed (m/s) Out-crossing rate (%) Data mining (DM) - pattern 2. What is a pattern? A pattern is defined as: A statement (expression) in a given language, that describes (relationships among) the facts in a subset of the given data and is (in some sense) simpler than the enumeration of all facts in the subset (Frawley et al. 1991, Fayyad et al. 1996). Classes of patterns considered in DM (depend on the data mining task at hand): 1. equations, 2. decision trees 3. association, classification, and regression rules 9 3

4 Data mining (DM) - pattern 1.Equations To predict the value of a target (dependent) variable as a linear or non linear combination of the input (independent) variables. Linear equations involving: - two variables: straight lines in a two dimensional space - three variables: planes in a three-dimensional space - more variables: hyper-plains in multidimensional spaces Nonlinear equations involving: - two variables: curves in a two dimensional space - three variables: surfaces in a three-dimensional space - more variables: hyper-surfaces in multidimensional spaces 10 Data mining (DM) - pattern 2. Decision trees To predict the value of one or several target dependent variables (class) from the values of other independent variables (attributes) by decision tree. Decision tree is a hierarchical structure, where: - each internal node contains a test on an attribute, - each branch corresponds to an outcome of the test, - each leaf gives a prediction for the value of the class variable. 11 Data mining (DM) - pattern Decision tree is called: A classification tree: class value in leaf is discrete (finite set of nominal values): e.g., (yes, no), (spec. A, spec. B, ) A regression tree: class value in leaf is a constant (infinite set of values): e.g.,120, 220, 312, A model tree: leaf contains linear model predicting the class value (piece-wise linear function): out-crossing rate= 12.3 distance wind speed wind direction 12 4

5 Data mining (DM) - pattern 3. Rules To perform association analysis between attributes discovered by association rules. The rule denotes patterns of the form: IF Conjunction of conditions THEN Conclusion. For classification rules, the conclusion assigns one of the possible discrete values to the class (finite set of nominal values): e.g., (yes, no), (spec. A, spec. B, spec. D) For predictive rules, the conclusion gives a prediction for the value of the target (class) variable (infinite set of values): e.g.,120, 220, 312, 13 Data mining (DM) - algorithm 3. What is data mining algorithm? Algorithm in general: -a procedure (a finite set of well-defined instructions) for accomplishing some task which will terminate in a defined endstat. Data mining algorithm: - a computational process defined by a Turing machine (Gurevich et al. 2000) for finding patterns in data 14 Data mining (DM) - algorithm What kind of possible algorithms do we use for discovering patterns? It depends on the goals: 1. Equations = Linear and multiple regressions 2. Decision trees = Top/down induction of decision trees 3. Rules = Rule induction 15 5

6 Data mining (DM) - algorithm 1. Linear and multiple regression Bivariate linear regression: predicted variable (C-class (ML) my be contusions or discontinues) can be expressed as a linear function of one attribute (A): C = α+ β A Multiple regression: predicted variable (C-class (ML) my be contusions or discontinues) can be expressed as a linear function of a multi-dimensional attribute vector (A i ): C = Σ n i =1 β i A i 16 Data mining (DM) - algorithm 2. Top/down induction of decision trees Decision tree is induced by Top-Down Induction of Decision Trees (TDIDT) algorithm (Quinlan, 1986) Tree construction proceeds recursively starting with the entire set of training examples (entire table). At each step, an attribute is selected as the root of the (sub)tree and the current training set is split into subsets according to the values of the selected attribute. 17 Data mining (DM) - algorithm 3. Rule induction A rule that correctly classifies some examples is constructed first. The positive examples covered by the rule from the training set are removed and the process is repeated until no more examples remain. 18 6

7 Data mining (DM) - Statistics Data mining vs. statistics Common to both approaches: Reasoning FROM properties of a data sample TO properties of a population. 19 Data mining (DM) Machine learning - Statistics Statistics Hypothesis testing when certain theoretical expectations about the data distribution, independence, random sampling, sample size, etc. are satisfied. Main approach: best fitting all the available data. Data mining Automated construction of understandable patterns, and structured models. Main approach: structuring the data space, heuristic search for decision trees, rules, covering (parts of) the data space. 20 DATA MINING CASE STUDIES 21 7

8 Practical implementations Each class of described patterns is illustrated with examples of applications: 1. Equations: Algebraic equations Differential equations 2. Decision trees: Classification trees Regression trees Model trees 3. Predictive rules 22 Applications Difference equations Algebraic equations: CIPER 23 Applications Algebraic equations Materials and methods Measured radial increments: - 8 trees - 69 years old Hydrological conditions (HMS Lendava; monthly data on minimal, average and maximum values) - Ledava River levels - groundwater levels Management data (thinning; m 3 /y removed from the stand; Forestry Unit Lendava) Dataset Meteorological conditions (monthly data, HMS Lendava): -Time of solar radiation (h), - precipitation (mm), - ET (mm) - Number of days with white frost - Number of days with snow - T: max, aver, min - Cumulative T>0ºC, >5ºC, and >10ºC - Number of days with: -mint>0ºc - mint<-10ºc -mint<-4ºc - mint>25ºc - maxt>10ºc - maxt>25ºc Monthly data + aggregated data (AMJ, MJJ, JJA, MJJA etc.) Σ: 333 attributes; 35 years 24 8

9 Applications Algebraic equations 52 different combinations of attributes were tested. Σ: 124 models Experiment RRSE eq. elements jnj3_2m 0, jnj3_3s 0, jnj3_1s 0, jnj3_4m 0, jnj2_2 0, jly_4xl 0, Applications Algebraic equations Model jnj3_2m: RadialGrowthIncrement = minl8-10^ maxl8-10^ t-sun4-7^ t-sun8-10^ e-05 t-sun8-10^ d-wf-4-7^ e-05 minl4-7^1 t-sun4-7^ minl4-7^1 t-sun8-10^ e-05 minl8-10^1 t-sun8-10^ e-05 maxl8-10^1 t-sun4-7^ t-sun4-7^1 d-wf-4-7^ t-sun8-10^1 d-wf-4-7^ Relative Root Squared Error = Correlation between average measured (r-aver8) and modeled increments: linear regression: R 2 = Applications Algebraic equations Model jnj3_2m 27 9

10 Applications Algebraic equations Algebraic equations: Lagramge 28 Applications Algebraic equations Data source: Federal Biological Research Centre (BBA), Braunschweig, Germany (2000, 2001) Slovenian Agricultural Institute (KIS), Slovenia (2006) Plants involved: BBA: - transgenic maize (var. Acrobat, glufosinate tolerant line) donor - non-transgenic maize field (var. Anjou ) - receptor KIS: - yellowkernel variety of maize (hybrid Bc462, simulating a transgenic maize variety) donor - white kernel variety of maize (variety Bc38W, simulating a non-gm variety) - receptor 29 Applications Algebraic equations Experiment design: 96 points Field design 2000 transgenic field / donor 2 m 100 m 220 m 6 non-transgenic field / recipient access paths sampling point 3 system of coordinates a for the sampling points 60 cobs 2500 kernels n o p q a 1 m N b % of outcrossing l k Donors Direction of drilling- GMO corns c d 4.5 m 3 m 7.5 m 13.5 m 2 m j h g f e 25,5m 49,5 m Receptors - NT corns 30 10

11 Selected attributes: % of outcrossing Cardinal direction of the sampling point from the center of donor field Visual angle between the sampling point and the donor field Distance between the point to the center of donor filed The shortest distance between the sampling point and the donor field % of appropriate wind direction (exposure time) Length of the wind ventilation route Wind velocity Applications Algebraic equations: outcrossing rate 31 Applications Algebraic equations 32 Applications Algebraic equations 33 11

12 Applications Algebraic equations 34 Applications Algebraic equations 35 Applications Algebraic equations 36 12

13 Applications Differential equations Differential equations: Lagramge 37 Applications Differential equations 38 Applications Differential equations 39 13

14 Applications Differential equations Phosphorus water in-flow out-flow respiration growth 40 Applications Differential equations Phytoplankton growth respiration sedimentation grazing 41 Applications Differential equations Zooplankton Feeds on phytoplankton respiration mortality 42 14

15 Applications Differential equations Applications Classification trees: habitat models Classification trees: J Applications Classification trees: habitat models Observed locations of BBs

16 Applications Classification trees: habitat models The training dataset Positive examples: - Locations of bear sightings (Hunting association; telemetry) - Females only - Using home-range (HR) areas instead of raw locations - Narrower HR for optimal habitat, wider for maximal Negative examples: - Sampled from the unsuitable part of the study area - Stratified random sampling - Different land cover types equally accounted for 46 Applications Classification trees: habitat models Dataset 1,73,26,0,0,1,88,0,2,70,7,20,1,0,0,1,0,60,0,0,0,0,0,2,0,0,0,4123,0,0,0,0,63,211,11,11,11,83,213,213,0,0, ,62,37,0,0,2,88,0,2,70,7,20,1,0,0,1,0,60,0,0,0,0,0,2,1,53,0,3640,0,0,0,-1347,63,211,11,11,11,83,213,213,11,89, ,0,99,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,6,82,0,10404,0,2074,-309,48,0,0,11,11,11,83,83,83,0,20, ,0,100,0,0,1,76,0,16,71,0,12,0,0,0,0,0,0,0,0,0,0,0,1,6,82,0,7500,0,1661,-319,-942,0,0,11,11,11,0,0,0,0,20, ,8,91,0,0,1,52,0,59,41,0,0,0,0,0,0,0,4,0,0,0,0,5,1,6,82,0,6500,0,1505,-166,879,9,57,11,11,11,281,281,281,0,20, ,3,0,86,9,0,75,0,33,67,0,0,0,0,0,0,1,2,0,0,0,0,0,1,2,54,0,0,0,465,-66,-191,4,225,11,31,31,41,72,272,60,619, ,34,65,0,0,2,51,9,76,9,5,1,4,1,0,1,0,29,0,0,0,0,0,1,2,54,0,3000,0,841,-111,-264,34,220,11,41,41,151,141,112,60,619, ,100,0,0,0,3,52,0,86,6,3,5,9,6,7,38,40,0,0,0,0,0,0,1,17,64,0,8062,0,932,-603,-71,100,337,11,41,41,171,232,202,4,24, Present: 1 Absent: 0 47 Applications Classification trees: habitat models The model for optimal habitat 48 16

17 Applications Classification trees: habitat models The model for maximal habitat 49 Applications Classification trees: habitat models Map of optimal habitat (13% SLO territory) 50 Applications Classification trees: habitat models Map of maximal habitat (39% SLO territory) 51 17

18 Applications Multi-target classification : outcrossing rate Multi-target classification model (Clus): Modelling pollen dispersal of genetically modified oilseed rape within the field Marko Debeljak, Claire Lavigne, Sašo Džeroski, Damjan Demšar In: 90th ESA Annual Meeting [jointly with the]ix International Congress of Ecology, August 7-12,2005, Montréal, Canada. Abstracts. [S.l.]: ESA, 2005, p Applications Multi-target classification : outcrossing rate Experiment design: 90m Filed for receptors (90 90m) 3 3m grid = 841 nodes Donors: MF transgenic oilseed rape B004.oxy (10 10m) Field planted with MF oilseed rape B seeds of MS oilseed rape FU58B004 planted per node 90m % MS outcrossing % MF outcrossing 53 Applications Multi-target classification : outcrossing rate Selected attributes for modelling: Rate of outcrossing of MS and MF receptor plants [rate per 1000] Cardinal direction of the sampling point from the center of donor field [rad] Visual angle between the sampling point and the donor field [rad] Distance between the point to the center of donor filed [m] The shortest distance between the sampling point and the donor field [m] 54 18

19 Applications Multi-target classification : outcrossing rate Number of examples: 817 Correlation coefficient: MF: MS: MF MS 55 Applications Multi-target regression model: soil resilience 56 Applications Multi-target regression trees: soil resilience The dataset: soil samples taken on 26 location throughout SCO The dataset: The flat table of data:26 by 18 data entries 57 19

20 Applications Multi-target regression trees: soil resilience The dataset: physical properties: soil texture: sand, silt, clay chemical properties: ph, C, N, SOM (soil organic matter) FAO soil classification: Order and Suborder physical resilience: resistance to compression: 1/Cc, recovery from compression: Ce/Cc, overburden stress: eg, recovery from overburden stress after two days cycles: eg2dc biological resilience: heat, copper 58 Applications Multi-target regression trees: soil resilience Different scenarios and multi-target regression models have been constructed: A model predicting the resistance and resilience of soils to copper perturbation. 59 Applications Multi-target regression trees: soil resilience The increasing importance of mapping soil functions to advice on land use and environmental management -to make a map of soil resilience for Scotland. The models = filters for existing GIS datasets about physical and chemical properties of Scottish soils

21 Applications Multi-target regression trees: soil resilience Macaulay Institute (Aberdeen): soils data attributes and maps: Approximately soil profiles held in database Descriptions of over soil horizons 61 Application 62 Application 63 21

22 Application 64 USING RELATIONAL DECISION TREES TO MODEL FLEXIBLE CO-EXISTENCE MEASURES IN A MULTI-FIELD SETTING Marko Debeljak 1, Aneta Trajanov 1, Daniela Stojanova 1, Florence Leprince 2,Sašo Džeroski 1 1: Jožef Stefan Institute, Ljubljana, Slovenia 2:ARVALIS-Institut du végétal, Montardon, France Introduction Initial questions: To what extent will GM maze grown on Geens genetically interfere with the maize on Yelows? Will this interference be small enough to allow co-existence? N DKC6041 YG Semis du 26/04 8 ha Pr34N44 YG Semis du 20/04 24 rgs Pr33A ha Pr33A46 Semis du 21/ m Marko Debeljak WP2 contributions to SIGMEA Final Meeting, Cambridge, UK, April,

23 Relational data preprocessing GIS Marko Debeljak WP2 contributions to SIGMEA Final Meeting, Cambridge, UK, April, 2007 Relational data mining results Out-crossing rate: Threshold 0.9 % Marko Debeljak WP2 contributions to SIGMEA Final Meeting, Cambridge, UK, April,

24 Data 130 sites, monitoring every 7 to 14 days for 5 month (2665 samples: 1322 conventional, 1333, HT OSR observations) Each sample (observation) described with 65 attributes Original data collected by Centre for Ecology and Hydrology, Rothamsted Research and SCRI within Farm Scale Evaluation Program (2000, 2001, 2002) Marko Debeljak ISEM '09, Quebec, Canada, 6-9 October 2009 Results scenario B: Multiple target regression tree Target: Avg Crop Covers, Avg Weed Covers Excluded attributes: / Constraints: MinimalInstances = 64.0; MaxSize = 15 Predictive power: Corr.Coef.: , RMSE: , RRMSE: , Marko Debeljak ISEM '09, Quebec, Canada, 6-9 October 2009 Results scenario D: Constraint predictive clustering trees for time series including TS clusters for crop (CLUS) syntactic constraint Marko Debeljak ISEM '09, Quebec, Canada, 6-9 October

25 Results scenario D: Constraint predictive clustering trees for time series including TS clusters for crop (CLUS) Target: Avg Weed Covers (Time Series) Scenario 3.9 Constraints: Syntactic, MinInstances = 32 Predictive power: TSRMSExval: 4.98 TSRMSEtrain: 4.86 ICVtrain: Marko Debeljak ISEM '09, Quebec, Canada, 6-9 October 2009 Results scenario D: Constraint predictive clustering trees for time series including TS clusters for crop (CLUS) Marko Debeljak ISEM '09, Quebec, Canada, 6-9 October 2009 Results scenario D: Constraint predictive clustering trees for time series including TS clusters for crop (CLUS) Marko Debeljak ISEM '09, Quebec, Canada, 6-9 October

26 Applications Rules 76 Applications Rules 77 Applications Rules The simulations were run with the first GENESYS version (published 2001, evaluated 2005, studied in sensitivity analyses 2004, 2005) Only one field plan was used: - maximising pollen and seed dispersal 78 26

27 Applications Rules Large-risk field pattern 79 Applications Rules Variables describing simulations - simulation number - genetic variables - for each field (1 to 35), the cultivation techniques of year -3, -2, -1, 0 - for each field (1 to 35) the number of years since the last GM oilseed rape crop - the number of years since the last non-gm oilseed rape crop - proportion of GM seeds in non-gm oilseed rape of field 14 at year 0 -TOTAL NUMBER OF VARIABLES: Applications Rules Run of experiment simulation started with an empty seed bank lasted 25 years, but only the last 4 years were kept in the files for data mining TOTAL NUMBER of simulations on the field pattern without the borders:

28 Applications Rules Non aggregated data: CUBIST Use 60% of data for training Each rule must cover >=1% of cases Maximum of 10 rules Rule 1: [29499 cases, mean , range 0 to , est err ] if SowingDateY0F14 > 258 then PropGMinField14 = Rule 2: [12726 cases, mean , range e-07 to , est err ] if SowingDateY0F14 > 258 SowingDateY0F14 <= 277 then PropGMinField14 = YearsSinceLastGMcrop_F14 82 Applications Rules Rule 4: [22830 cases, mean , range 0 to , est err ] if SowingDateY0F14 <= 258 YearsSinceLastGMcrop_F14 > 2 then PropGMinField14 = SowingDateY0F SowingDensityY0F14 83 Applications Rules Rule 10: [1911 cases, mean , est err ] if TillageSoilBedPrepY0F14 in {0, 2} SowingDateY0F14 <= 258 SowingDensityY0F14 <= 55 YearsSinceLastGMcrop_F14 <= 2 then PropGMinField14 = YearsSinceLastGMcrop_F SowingDateY0F SowingDensityY0F SowingDateY-1F HarvestLossY-1F HarvestLossY-2F SowingDensityY-2F HarvestLossY-1F EfficHerb2Y-3F27-7e-05 2cuttingY-3F23 + 7e-05 2cuttingY-2F SowingDateY-1F HarvestLossY0F9 + 5e-05 1cuttingY-3F SowingDensityY-2F18-6e-05 2cuttingY-2F24-6e-05 2cuttingY-2F15 + 6e-05 2cuttingY0F32-6e-05 2cuttingY-2F SowingDateY0F11-4e-05 1cuttingY0F SowingDensityY-1F EfficHerb2Y-3F HarvestLossY-1F22-5e-05 2cuttingY-2F EfficHerb2GMvolunY0F EfficHerb1GMvolunY-2F fficherb2gmvoluny0f

29 Applications Rules Non aggregated data: CUBIST Options: Use 60% of data for training Each rule must cover >=1% of cases Maximum of 10 rules Target attribute `PropGMinField14' Evaluation on training data (60000 cases): Average error Relative error 0.47 Correlation coefficient 0.77 Evaluation on test data (40000 cases): Average error Relative error 0.49 Correlation coefficient Conclusions What can data mining do for you? Knowledge discovered by analyzing data with DM techniques can help: Understand the domain studied Make predictions/classifications Support decision processes in environmental management 86 Conclusions What data mining cannot do for you? The law of information conservation (garbage-in-garbage-out) The knowledge we are seeking to discover has to come from the combination of data and background knowledge If we have very little data of very low quality and no background knowledge no form of data analysis will help 87 29

30 Conclusions Side-effects? Discovering problems with the data during analysis missing values erroneous values inappropriately measured variables Identifying new opportunities new problems to be addressed recommendations on what data to collect and how 88 DATA MINING Hands-on exercises 1. Data preprocessing 89 DATA MINING data preprocessing DATA FORMAT File extension.arff This a plain text format, files should be edited by editors such as Notepad, TextPad, WordPad (that do not add extra formatting information) File consists of NameOfDataset List of AttName AttType AttType can be numeric or nominal list of categorical values, e.g., {red, green, blue} (in a separate line), followed by the actual data in comma separated value (.csv) format 90 30

31 DATA MINING data preprocessing DATA outlook {sunny, overcast, temperature humidity windy {TRUE, play {yes, sunny,85,85,false,no sunny,80,90,true,no overcast,83,86,false,yes 91 DATA MINING data preprocessing Excel Attributes (variables) in columns and cases in lines Use decimal POINT and not decimal COMMA for numbers Save excel sheet as CSV file 92 DATA MINING data preprocessing TextPad, Notepad Open CSV file Delete on the beginning of lines and save (just save, don t change the format) Change all ; to, Numbers must have decimal dot (.) and not decimal comma (,) Save file as CSV file (don't change format) 93 31

32 DATA MINING Hands-on exercises 2. Data minig 94 DATA MINING data preprocessing WEKA Open CVS file in WEKA Select algorithm and attributes Perform data mining 95 How to select the best classification tree? Performance of the classification tree: classification accuracy: (correctly classified examples)/(all examples) True positive rate False positive rate Confusion matrix is a matrix showing actual and predicted classifications 96 32

33 How to select the best classification tree? Classification trees: J48 Interpretable size: -pruned or unpruned - minimal number of objects per leaf the number of instances CORRECTLY classified into this leaf the number of instances INCORRECTLY classified into this leaf It could appear: (13) no incorrectly classified instances or (3.5/0.5) due to missing values (?) where instances are fractured or (0,0) a split on a nominal attribute and one or more of the values do not occur in the subset of instances at the node in question 97 How to select the best regression / model tree? Performance of the regression / model tree: 98 How to select the best regression / model tree? The interpretable size: -pruned or unpruned - minimal number of objects pre leaf The number of instances that REACH this leaf Root of the mean squared error (RMSE) of the predictions from the leaf's linear model for the instances that reach the leaf, expressed as a percentage of the global standard deviation of the class attribute (i.e. the standard deviation of the class attribute computed from all the training data). Sum is not 100%. The smaller this value, the better

34 Accuracy and error Avoid overfitting the data by tree pruning. Pruned trees are: - less accurate (percentage of correct classifications) on training data - more accurate when classifying unseen data 100 How to prune optimally? Pre-pruning: stop growing the tree e.g., when data split not statistically significant or too few examples are in a split (minimum number of objects in leaf) Post-pruning: grow full tree, then post-prune (confidence factor-classification trees) 101 Optimal accuracy 10-fold cross-validation is a standard classifier evaluation method used in machine learning: - Break data into 10 sets of size n/10. - Train on 9 datasets and test on 1. - Repeat 10 times and take a mean accuracy