Grouping Objects by Linear Pattern

Transcription

1 Grouping Objects by Linear Pattern Ruben Zamar Department of Statistics University of British Columbia Vancouver, CANADA

2 This talk is based on work with several collaborators: 1 Van Aelst, Wang, Zamar and Zhu (2006) Linear Grouping Using Orthogonal Regression, CSDA Garcia-Escudero, Gordaliza, San Martin, Van Aelst and Zamar (2008) Robust linear clustering, JRSS, Series B and Yan, Welch and Zamar (2010) A likelihood approach to linear clustering, submitted Special thanks to Justin Harrington for the R package LGA that implements some of the the methods presented here.

3 Outline 2 Grouping by linear patterns Linear grouping algorithm (LGA) The number of groups (GAP) Applications Dealing with outliers (RLGA) Model based approach

4 Clustering Algorithms 3 Traditional clustering algorithms are effective when clusters are distinct homogenous groups What about other interesting patterns?

5 Young and Old Trees 4

6 SNP Genotyping Data 5

7 Allometry Data 6

8 Our Goal 7 To find groups clustered around hyperplanes of different dimensions l i, where 0 l i d 1 i = 1, 2,..., g

9 Example d = 3 and g = 3 8 l 1 = 1 l 2 = 0 l 3 = 2 cluster around a line. cluster around a point. cluster around plane.

10 Linear Grouping Algorithm (LGA) 9 Goal: to find k groups around hyperplanes of dimension d 1 To find clusters around k lines in R 2. To find clusters around k planes in R 3....

11 Some References 10 Murtagh and Raftery (1984) Gawrysiak et al. (2000) Spath (1982,1985) Desarbo, Oliver and Rangaswamy (1989) Wedel and Kistemaker (1989) Kamgar-Parsi, Kamgar-Parsi and Wechsler (1990) Gawrysiak, Okoniewski and Rybinski (2000) Specified response variable.

12 Unsupervised Learning 11 Clustering is most often used for unsupervised learning. Unsupervised learning is characterized by the absence of response variables. Different linear groups may involve different subsets of variables.

13 Simple Example 12

14 Response Variable = Y 13

15 Response Variable = Z 14

16 Orthogonal Residuals 15

17 Hyperplane 16 H (α, β) = {z : α z = β, α = 1}

18 Data 17 Let z 1, z 2,..., z n be n points in R d z = 1 n zi (Sample Mean) S = 1 n (zi z)(z i z) (Sample Covariance)

19 Orthogonal Regression 18 J (α, β) = (α z i β) 2 ( ) ˆα, ˆβ minimizes J (α, β) ˆα = normalized first eigenvector of S ˆβ = ˆα z

20 LGA Algorithm 19 INPUT: d-dimensional data points z 1, z 2,..., z n and the number k of groups OUTPUT: The best partition of the dataset into k groups centered around hyperplanes of dimension d 1

21 LGA Step-by-Step 20 1) Initialization: Initial hyperplanes are defined by the exact fitting of k sub-samples of size d 2) Forming k groups: Each data point is assigned to its closest hyperplane using Euclidean distances. 3) Computing k Hyperplanes: New hyperplanes are computed applying orthogonal regression to each group. Steps 2) and 3) are repeated several times

22 The Number of Groups 21 The number of groups k is an input for lga k may be suggested by subject matter knowledge Finding k may be the an important research goal

23 Graphical Approach 22

24 The GAP Statistic 23 Tibshirani, Walther and Hastie (2001) proposed the GAP statistic to determine the number of clusters in a data set. GAP compares the pooled within-cluster sum of squares around the cluster centers with its expectation under a null reference distribution. The null distribution is obtained by generating uniformly distributed points on the hyper-rectangle aligned with the principal components of the data. The (modified) GAP statistic for linear grouping is obtained by replacing distance to the center by distance to the hyperplane.

25 The GAP Statistic (continued) 24 GAP (k) = [ 1 B B b=1 log (SSR k (b)) ] log (SSR k ) ˆk = smallest k such that GAP (k) GAP (k + 1) s k+1 s k+1 = S k (1/B) S k+1 = Standard Deviation of log (SSR k+1 (b))

26 Simple Example 25

27 Simple Example 26

28 Simple Example 27

29 Simple Example 28

30 Application to Allometry 29 Figure 6: Olfactory Bulb vs. Brain Weight (log-scale) for some mammal species: Insectivores (i), Carnivores (c), Prosimians (p), Apes (a), Monkeys (m), Human (h) and Horse (o).

31 Application to Allometry 30 Biologists study the relation between sizes of organs for different species. The (log-transformed) sizes are linearly related. Linear associations differ across species because of different living habits, environment, food sources, etc. Grouping by different linear patterns is necessary. Biologists make manual assignments based on experience (Jerison 1973).

32 Application to Allometry 31

33 Application to Allometry 32

34 Application to Allometry 33 Jerison (1973) I II III insectivores, carnivores, horses, prosimians (primitive primates) anthropoids (monkeys, apes, human) LGA with k=3 I II III insectivores, carnivores, horses, prosimians and apes monkeys and human LGA & GAP I II insectivores, carnivores, horses, prosimians monkeys, apes and human

35 Application to Sport Data 34 Performance of 871 players in the 94/95 Hockey League Variables PTS P/M PIM PP Description # of Goal Scored + # of Assists Plus/Minus Rating + team scored, - oponent team scored Total penalty time (minutes) Total number of power-play goals scored

36 Application to Sport Data 35 We applied LGA with k=3 The results:

37 Sharp Shooters - Team Players 36

38 Simple Example 37

39 Dealing with Outliers 38 Use trimming to allow a fraction of points not following any linear structure The resulting procedure is called Robust LGA (RLGA) RLGA is computed by the function rlga() in the lga package

40 RLGA Step-by-Step 39 1) Initialization: k hyperplanes are defined by exact fitting k random sub-samples of size d 2) Trimming and Forming the Groups: For 1 i n let r i (1), r i (2),..., r i (k) be the orthogonal distances from point i to each of the k hyplerplanes Let r i = min{r i (1), r i (2),..., r i (k)} The n(1 α) points with smallest r i are assigned to their closest hyperplanes 3) Computing k Hyperplanes: New hyperplanes are computed applying orthogonal regression to each group. Steps 2) - 3) are repeated several times

41 Simple Example - Continued 40

42 Simple Example - Continued 41

43 Computer Vision Data 42