Machine Learning & Statistical Models

Transcription

1 Astroinformatics

2 Machine Learning & Statistical Models Neural Networks Feed Forward Hybrid Decision Analysis Decision Trees Random Decision Forests Evolving Trees Minimum Spanning Trees Perceptron Multi Layer Perceptron Radial Basis Functions Recurrent / Feedback Fuzzy Sets Genetic Algorithms K-NN Principal Component Analysis Support Vector Machine Soft Computing Statistical Models Competitive Networks Hopfield Networks Adaptive Reasoning Theory Bayesian Networks Hidden Markov Models Mixture of Gaussians Principal Probabilistic Surface Maximum Likelihood 2 Negentropy 2

3 Fuzzy reasoning There are two assumptions that are essential for the use of formal set theory: For any element and a set belonging to some universe, the element is either a member of the set or else it is a member of the complement of that set An element cannot belong to both a set and also to its complement Both these assumptions are violated in Lofti Zadeh s fuzzy set theory Zadeh s main contention is that, although probability theory is appropriate for measuring randomness of information, it is inappropriate for measuring the meaning of the information Zadeh proposes possibility theory as a measure of vagueness, just like probability theory measures randomness 3

4 Fuzzy Logic Lofti Zadeh has coined the term Fuzzy Set in 1965 and opened a new field of research and applications A Fuzzy Set is a class with different degrees of membership. Almost all real world classes are fuzzy! Examples of fuzzy sets include: { Tall people }, { Nice day }, { Round object } Fuzzy sets. Information and Control. 1965; 8: If a person s height is 1.88 meters is he considered tall? What if we also know that he is an NBA player? Height, cm Crisp value Fuzzy Extremely tall Very tall tall 4

5 A crisp definition of Fuzzy Logic does not exist, however It generalizes bivalent Aristotelian (Crisp) logic: Aristotele s modus ponens IF <Antecedent == True> THEN <Do Consequent> IF (X is a prime number) THEN (Send TCP packet) Generalized modus ponens IF a region is green and highly textured AND the region is somewhat below a sky region THEN the region contains trees with high confidence 5

6 In other words Fuzzy sets are an extension of classical (or crisp) sets The notation of fuzzy set can be describes as follows: Let S be a set and s a member of that set. A fuzzy subset F of S is defined by a membership function mf(s) that measures the degree to which s belongs to F For example: S to be the set of positive integers and F to be the fuzzy subset of S called small integers. Now, various integer values can have a possibility distribution defining their fuzzy membership in the set of small integers: mf(1)=1.0, mf(3)=0.9, mf(50)=0.001 In the traditional logic of crisp set, the confidence of an element being in a set must be either 1 or 0 6

7 An example of FL: Speed Representation of the speed with Representation of the speed classical sets with fuzzy sets 7

8 An example of FL: temperature Fuzzy Logic (for Temperature ) Extremely Hot Hot Quite Hot Quite Cold Cold Extremely Cold 8

9 An example of FL: washing machine 9

10 An example of FL: temperature 10

11 Fuzzification / Defuzzification Usually input to a fuzzy system are crisp values which have to be converted to fuzzy sets The mapping from a crisp value to a fuzzy set is called fuzzification The inverse process is referred to as defuzzification (seen before) Rule base contains the fuzzy rules Database contains the membership functions 11

12 Fuzzy logic inference Fuzzifier converts a crisp input into a vector of fuzzy membership values. The membership functions reflects the designer's knowledge provides smooth transition between fuzzy sets are simple to calculate Typical shapes of the membership function are Gaussian, trapezoidal antecedent and triangular. Defuzzifier Converts the output fuzzy numbers into a unique (crisp) number consequent Method: Add all weighted curves and find the center of mass 12

13 MF examples Gaussian xc fgmf x;, c e f smf Sigmoid 1 x, a, c e ax c 1 Triangular Trapezoidal f x a c x x; a, b, c max min,, 0 b a c b f x a d x x; a, b, c, d max min,1,,0 b a d c 13

14 Linguistic Variables Operate on the Membership Function (Linguistic Variable) 1. Expansive ( Less, Very Little ) 2. Restrictive ( Very, Extremely ) 3. Reinforcing/Weakening ( Really, Relatively ) Very Little 4 x A x x c A Less x Very x 2 Extremely x 4 14

15 Fuzzy Sets Membership values on [0,1] Law of Excluded Middle and Non-Contradiction do not necessarily hold: A A A A Fuzzy Membership Function Flexible operators (T-Norm), Union (S-Norm) and Negation Crisp vs Fuzzy Sets Crisp Sets True/False {0,1} Law of Excluded Middle and Non- Contradiction hold: A A A A Crisp Membership Function Intersection (AND), Union (OR), and Negation (NOT) are fixed 15

16 Generalized Mean: h Aggregation Operators a, a, 1 2, a n a 1 a 2 a n n 1 0 and a 0, i,1 i n i 1, h 0, h 1, h, h min Harmonic Mean Geometric Mean Al gebraic Mean, h max Generalized Mean Drastic T-Norm Product Zadehian min Zadehian max Bounded Sum Drastic S-Norm Geometric Harmonic Algebraic (Mean) 16

17 T-norms: intersection logic Aggregation Operators b, if a 1 D, T a b a, if b 1 0, otherwise Drastic T a, b ab Product, min, T a b a b Z Zadehian S-Norms: union logic (the dual of T-norm) Z Zadehian, max, S a b a b S a, b a b a b BS Bounded Sum Drastic b, if a 0 SD a, b a, if b 0 1, otherwise Generalized Mean Drastic T-Norm Product Zadehian min Zadehian max Bounded Sum Drastic S-Norm Geometric Harmonic Algebraic (Mean) 17

18 Fuzzy Logic vs Vagueness Fuzziness=Unsharp Boundaries I will be back sometime I will be back in a few minutes Vague Fuzzy Vagueness=Insufficient Specificity 18

19 Example: find an image threshold f smf Membership Value 1 x, a, c e ax c 1 Gray Level 19

20 Fuzzy Relations A relation among crisp sets is a subset of the Cartesian product A fuzzy relation is a fuzzy set defined on the Cartesian product R ( X, Y ) {( x, y ), R ( x, y ) ( x, y ) X Y } Example of Aggregation The output variable of a fuzzy system has three fuzzy values: L, M and H In a given instant, four rules are fired producing fuzzy sets L1, M2, M3 and H4, respectively maximum The final fuzzy set obtained is a linguistic term which may be used as it is, or most often defuzzified in order to produce a numeric value Center of area (COA) (or center of gravity or centroid) Mean of maxima (MOM) 20

21 1. Fuzzification of the input Fuzzy Inference steps 2. Evaluation of activation degree or weight of each rule. This includes sub-steps 2.1 and 2.2: 2.1 computing the membership degrees of each input variable to the fuzzy sets in the antecedent of each rule (a t-norm is applied to the fuzzified input variable and the antecedent fuzzy sets) 2.2 if and operator joins input labels, a t-norm is applied to the membership degrees obtained in step 2.1, in order to achieve the weight of the rule 21

22 Fuzzy Inference steps 3. Implication and composition (we assume either minimum or product implication) 22

23 Fuzzy Inference steps 4. Rule aggregation (we assume three rules) 5. Defuzzification 23

24 Example: Tourist prediction system The fuzzy system predicts the number of tourists visiting a resort 2 input variables: Temperature (in degrees) and Sunshine (a % of maximum sunshine) 1 output variable: Tourists, estimated amount of tourists (% of resort s capacity) Rule 1: if (Temperature is Hot) or (Sunshine is Sunny) then (Tourists is High) Rule 2: if (Temperature is Warm) and (Sunshine is Partially Sunny) then (Tourists is Medium) Rule 3:if (Temperature is Cold) or (Sunshine is Cloudy) then (Tourists is Low) Let us refer to a slightly cold, partially sunny day: Temperature = 19 degrees Sunshine = 60% How to fuzzify such day conditions? 24

25 Example: Tourist prediction system Temperature = 19 degrees Sunshine = 60% µ Cold (19) = 0.33 µ Cloudy (60) = 0 µ Warm (19) = 0.67 µ PartSunny (60) = 0.8 µ Hot (19) = 0 µ Sunny (60) =

26 Example: Tourist prediction system 26

27 Example: Tourist prediction system µ Rule1 = 0.2 µ Rule2 = 0.67 µ Rule3 =

28 Example: Tourist prediction system 28

29 Example: Tourist prediction system Step 4. COG defuzzification 29

30 Types of fuzzy rules Mamdani fuzzy rule if X 1 is A i1 and... and X m is A im then Y is B i The rule consequent is a fuzzy set Main advantage: high interpretability Drawbacks: low accuracy high computational cost Mamdani & Assilian International Journal of Man-Machine Studies, 7(1):

31 Types of fuzzy rules Takagi-Sugeno-Kang (TSK) fuzzy rule if X 1 is A i1 and... and X m is A im then Y is a i 0 + a i1 X a im X m where a i 0, a i 1,, a i m are real numbers The rule consequent is a function usually linear of the input variables. Thus, each rule can be considered as a local linear model The system s output is the weighted average of each rule s output Main advantage: greater accuracy than Mamdani systems Drawback: low interpretability high computational cost Takagi & Sugeno IEEE Trans. Systems, Man, and Cybernetics 15,

32 TSK applied to Tourist example Let us convert the system into a TSK system by replacing the three linguistic consequents Low, Medium and High with calculated corresponding linear functions of the input variables The three rule consequents thus become: Rule 1: if then Tourists = f 1 (T, S ) = 2T S - 40 Rule 2: if then Tourists = f 2 (T, S ) = T + S - 23 Rule 3: if then Tourists = f 3 (T, S ) = 0.5T + 0.3S 32

33 Fuzzy Logic vs Statistics Walking in the desert, close to being dehydrated, you find two bottles of water: The first contains deadly poison with a probability of 0.1 The second has a 0.9 fuzzy membership value Safe drinks Which one will you choose to drink from??? #1 #2 33

34 Fuzzy Logic vs Statistics Walking in the desert, close to being dehydrated, you find two bottles of water: The first contains deadly poison with a probability of 0.1 The second has a 0.9 fuzzy membership value Safe drinks Which one will you choose to drink from??? #1 Fuzzy is said to measure possibility rather than probability. Difference All probable things are possible. All possible things may be not probable. Contrapositive All impossible things are improbable Not all improbable things are impossible So, make your choice #2 34

35 Fuzzy Logic vs Statistics Walking in the desert, close to being dehydrated, you find two bottles of water: The first contains deadly poison with a probability of 0.1 The second has a 0.9 fuzzy membership value Safe drinks Which one will you choose to drink from??? #1 Fuzzy is said to measure possibility rather than probability. Difference All probable things are possible. All possible things may be not probable. Contrapositive All impossible things are improbable Not all improbable things are impossible the second is better!!! #2 35

36 Fuzzy Logic Adaptive Networks We have only considered membership functions that have been fixed, and somewhat arbitrarily chosen. Also, we have only applied fuzzy inference to modeling systems whose rule structure is essentially predetermined by the user s interpretation of the characteristics of the variables in the model A new modeling scenario We have a collection of input/output data that we would like to use for modeling We do not necessarily have a predetermined model structure based on characteristics of variables in our system Rather than choosing the parameters associated with a given membership function arbitrarily, these parameters could be chosen so as to tailor the membership functions to the input/output data we introduce Adaptive Networks 36

37 Fuzzy Logic Adaptive Networks An adaptive network is a multi-layer feedforward network in which each node performs a particular function (node function) on incoming signals A square node (adaptive node) has parameters; A circle node (fixed node) has no parameters; The links only indicate the flow direction of signals between nodes; no weights are associated with the links In order to achieve a desired input-output mapping, the parameters are updated according to given training data and a gradient-descent learning procedure 37

38 ANFIS ANFIS (Adaptive Network based Fuzzy Inference System) refers to a class of adaptive networks which are functionally equivalent to fuzzy inference systems. Assume that a fuzzy inference system has two inputs X and Y and one output f. Suppose that the rule base contains two fuzzy if-then rules of TSK type: Rule1 : if X is A 1 and Y is B 1 then f 1 = p 1 x + q 1 y + r 1 Rule 2 : if X is A 2 and Y is B 2 then f 2 = p 2 x + q 2 y + r 2 Fuzzy reasoning ANFIS architecture 38

39 A Fuzzy based ML model Fuzzy C-Means (FCM) is a method of clustering which allows one piece of data to belong to two or more clusters. This method (developed by Dunn in 1973 and improved by Bezdek in 1981) is frequently used in pattern recognition. Similar to the k-means clustering algorithm, the FCM algorithm attempts to create C cluster centers in the space, and then measures the distance of each point to the center of each cluster. Unlike the k-means clustering algorithm, the fuzzy C-means computes the degree to which each point belongs to a cluster. This allows points farther away from the center to be in the cluster to a lesser degree than those points close to the center 39

40 A Fuzzy based ML model Fuzzy C-Means (FCM) is a method of clustering which allows one piece of data to belong to two or more clusters. This method (developed by Dunn in 1973 and improved by Bezdek in 1981) is frequently used in pattern recognition. Input: Unlabeled data set n is the number of data points inx x k p Main Output A c-partition of X, which is Common Additional Output p X x1, x2,, xn { } is the number of features in each vector c n V v, v,, v Set of vectors { } 1 2 v is called cluster center i c matrix U p 40

41 A Fuzzy based ML model EXAMPLE X n 188 p 2 Rows of U (Membership Functions) U and c 4 V 41

42 A Fuzzy based ML model FCM OBJECTIVE FUNCTION Optimization of an objective function or performance index c n m 2 min Jm( UV, ) uikdik ( UV, ) i1 k1 Constraint Distance Degree of Fuzzification c u ik i Dik xk vi A m 1 1, k x T x, x x Ax A A-norm A 42

43 A Fuzzy based ML model FCM Tips Initial Choices Number of clusters Maximum number of iterations (Typ.: 100) Weighting exponent (Fuzziness degree) m=1: crisp m=2: Typical 1cn Termination measure 1-norm t t t1 Termination threshold (Typ. 0.01) Guess Initial Cluster Centers E V Initialization is done on Iterates become Termination criterion V 0 0 1,0 c,0 m V ( v, v ) U 0 T cp U V U U t 1 t t t Ut 1 43

44 A Fuzzy based ML model Fuzzy C-means Clustering

45 A Fuzzy based ML model Fuzzy C-means Clustering

46 A Fuzzy based ML model Fuzzy C-means Clustering

47 A Fuzzy based ML model Fuzzy C-means Clustering

48 A Fuzzy based ML model Fuzzy C-means Clustering

49 A Fuzzy based ML model Fuzzy C-means Clustering

50 A Fuzzy based ML model Fuzzy C-means Clustering

51 A Fuzzy based ML model FCM Pro & Cons Advantages Unsupervised Always converges Disadvantages Long computational time Sensitivity to the initial guess (speed, local minima) Sensitivity to noise One expects low (but even no) membership degree for outliers (noisy points) 51

52 A Fuzzy based ML model FCM Outliers, a potential problem! FCM on FCM on X11 u1,6 0.5 u2,6 0.5 X12 u u u u2, , , ,6 0.5 x 12 x 6 x 6 x 12 is an outlier but has the same membership degrees as x 6 52

53 Fuzzy C-Means procedure FCM-CLUSTERING (x) returns prototypes and partition matrix input : data x = {x1, x2,..., xk} local: fuzzification parameter: m threshold: The pseudo-algorithm The flow-chart norm:. INITIALIZE-PARTITION-MATRIX t 0 repeat for i=1:c do N m uik (t) xk v k1 i (t) compute prototypes N m uik (t) k 1 for i = 1:c do for k = 1:N do update partition matrix 1 uik (t 1) 2/(m1) c xk vi (t) j1 xk v j(t) update partition matrix t t + 1 until U(t+1)-U(t) return U, V 53

54 Example of FCM in Astrophysics Measurements for each image, such as ellipticity, average transmission over image, and the image gradients, were computed. In total, 14 measurements made and used as the parameters to define each image. Using a data set consisting of ~5000 stars and ~4000 galaxies, they compared the results of FCM to two neural network approaches (MLP and SOM) over various subsets of the data. Although unsatisfying results, the fuzzy classifier has own advantages. Its output can be used directly as an estimate of the classification reliability. In fact, classification memberships lie within the range [0,1], and those classifications around 0.5 are considered very unreliable. These objects can then be sent for more sensitive processing, whether it be a different classification system, or a human observer. Mahonen & Frantti. "Fuzzy classier for star-galaxy separation." The Astrophysical Journal (2000):