Spectral Clustering with Two Views

Transcription

1 Spectral Clustering with Two Views Virginia R. de Sa Departent of Cognitive Science, 55 University of California, San Diego 95 Gilan Dr. La Jolla, CA Abstract In this paper we develop an algorith for spectral clustering in the ulti-view setting where there are two independent subsets of diensions, each of which could be used for clustering (or classification). The canonical exaples of this are siultaneous input fro two sensory odalitites, where input fro each sensory odality is considered a view, as well as web pages where the text on the page is considered one view and text on links to the page another view. Our spectral clustering algorith creates a bipartite graph and is based on the iniizing-disagreeent idea. We show a siple artifically generated proble to illustrate when we expect it to perfor well and then apply it to a web page clustering proble. We show that it perfors better than clustering in the joint space and clustering in the individual spaces when soe patterns have both views and others have just one view. Spectral clustering is a very successful idea for clustering patterns. The idea is to for a pairwise affinity atrix A between all pairs of patterns, noralize it, and copute eigenvectors of this noralized affinity atrix (graph Laplacian)L. It can be shown that the second eigenvector of the noralized graph Laplacian is a relaxation of a binary vector solution that iniizes the noralized cut on a graph (Shi & Malik, 998; J.Shi & Malik, ; Meila & Shi, ; Ng et al., ). Spectral clustering has the advantage of perforing well with non-gaussian clusters as well as being easily ipleentable. It is also non-iterative with no local inia. The Ng,Jordan,Weiss(Ng et al., Appearing in Proceedings of the Workshop on Learning with Multiple Views, nd ICML, Bonn, Gerany, 5. Copyright 5 by the author(s)/owner(s). ) (NJW) generalization to ulticlass clustering (which we will build on) is suarized below for data patterns x i to be clustered in to k clusters. For the affinity atrix A(i, j) = exp( x i x j /σ ) Set the diagonal entries A(i, i) = Copute the noralized graph Laplacian as L = D.5 AD.5 where D is a diagonal atrix with D(i, i) = j A(i, j) Copute top k eigenvectors of L and place as us in a atrix X For Y fro X by noralizing the rows of X Run keans to cluster the row vectors of Y pattern x i is assigned to cluster α iff row i of Y is assigned to cluster α In this paper we develop an algorith for spectral clustering in the ulti-view setting where there are two independent subsets of diensions, each of which could be used for clustering (or classification). The canonical exaples of this are ulti-sensory input fro two odalities where input fro each sensory odality is considered a view as well as web pages where the text on the page is considered one view and text on links to the page another view. Also coputer vision applications with ultiple conditionally independent sensor or feature vectors can be viewed in this way.. Algorith Developent Our spectral ulti-view algorith is based on ideas originally developed for the (non-spectral) Miniizing- Disagreeent algorith (de Sa, 994a; de Sa & Ballard, 998). The idea behind the Miniizing- Disagreeent (M-D) algorith is that two (or ore) - -

2 networks receiving data fro different views, but with no explicit supervisory label, should cluster the data in each view so as to iniize the disagreeent between the clusterings. The Miniizing-Disagreeent algorith was described intuitively using the following diagra shown in Figure. In the figure iagine that there are two classes of objects, with densities given by the thick curve and the thin curve and that this arginal density is the sae in each one-diensional view. The scatter plots on the left of the figure show two possible scenarios for how the views ay be related. In the top case, the views are conditionally independent. Given that a thick/dark object is present, the particular pattern in each view is independent. On the right, the sae data is represented in a different forat. In this case the values in view are represented along one line and the values in view along another line. Lines are joined between a pair if those values occurred together. The iniizing disagreeent algorith wants to find a cut fro top to botto that crosses the fewest lines within the pattern space (subject to soe kind of balance constraint to prevent trivial solutions with epty or near epty clusters). Disagreeent is iniized for the dashed line shown. Here we transfor this intuitive idea for -D views to a general algorith on a weighted bipartite graph. The difficulty in transforing this intuitive idea into a general algorith for a M-D spectral algorith is that in describing it as aking a cut fro top to botto, we assue that we have a neighborhood relationship within each top set and botto set, that is not explicitly represented. That is we assue that points drawn in a line next to each other are siilar points in the sae view. Treating the points as nodes in a graph and applying a graph cut algorith, would lose that inforation. One solution would be to siply connect co-occurring values and also join nearest neighbors (or join neighbors according to a siilarity easure) in each view. This, however, raises the tricky issue of how to encode the relative strengths of the pairing weights with the within-view affinity weights. Instead, our solution is to draw reduced weight cooccurrence relationships between neighbors of an observed pair of patterns (weighted by a uniodal function such as a Gaussian). We call our algorith sm-d Each input in each view is represented by a node in the graph. The strength of the weight between two nodes in different views depends on the nuber of ulti-view patterns (which we can think of as co-occuring pairs of patterns) that are sufficiently close (in both views) (with a fall off in weight as the distances grow). This representation has the seantics that we believe there is noise in the actual patterns that occur or alternatively that we wish to consider the pairings siultaneously at ultiple scales. More specifically, let us define x (v) i as view v of the ith pattern. We will construct a graph node for each view of each pattern and define n (i,v) to represent the node for view v of the ith pattern. Now consider the pattern x () = [ ] (where throughout this paper denotes the transpose operator) and the pattern x () = [ ] + ɛ. These two patterns should probably be considered identical for sall ɛ. This eans that x () the co-occurring pattern for x () should probably also be linked with x (). The Gaussian weighting allows us to do this in a sooth way for increasing ɛ. To copute the total weight between node n (i,) and n (j,) we su over all observed pattern cooccurrences (k= to p): the product of (the (Gaussian weighted) distance between x () i (the pattern represented by n (i,) ) and x () k and the sae sae ter for the relationship between the x () j and x (). That is e (x () x () i k ) e (x k () x () j k ) w ij = σ p = [A v A v ] ij () σ () where A v is the affinity atrix for the view patterns and A v the affinity atrix for just the view pat- (x () x () ) i j σ terns. A v (i, j) = e. Note that the product between the Gaussian weighted distances within each view is just the Gaussian weighted noralized distance between the two concatenated patterns (when considered as ulti-view patterns). Then we take the p p atrix of w s and put it in a large p p atrix of the for A sm D = [ ] p p W W p p where p p represents a p p atrix of zeros (and we will drop the subscript fro here on for clarity). This atrix could then be considered an affinity atrix (for a bipartite graph) and given to the spectral clustering algorith of (Ng et al., ). However note that the next step is to copute eigenvectors of the atrix D.5 A sm D D.5 where D is a diagonal atrix with D(i, i) = j A sm D(i, j) (row sus of A sm D ) which is equal - -

3 to (where D row (D ) is the diagonal atrix with diagonal entries equal to the row (un) sus of W ) [ ] [ ] [ ] D.5 row W D.5 D.5 W row D.5 but that atrix has the sae eigenvectors as the atrix [ D.5 roww D W ] Drow.5 D.5 W DrowW D.5 which has conjoined eigenvectors of each of the blocks DrowW.5 D W Drow.5 and D.5 W DrowW D.5 and these parts can be found efficiently together by coputing the SVD of the atrix L W = DrowW.5 D.5. This trick is used in the co-clustering literature (Dhillon, ; Zha et al., ), but there the affinity subatrix W is derived siply fro the ter docuent atrix (or equivalent) not derived as a product of affinity atrices fro different views. The final clustering/segentation is obtained fro the top eigenvectors. There are several slightly different ways to cluster the values of this eigenvector. We use the prescription of Ng, Jordan and Weiss fro the first page where Y is obtained as follows. Av =exp(-distatview/(*sigsq)); Av =exp(-distatview/(*sigsq)); W=Av*Av; Dtop=(su(W )); Dbot=(su(W)); Lw=diag(Dtop.^(-.5))*W*diag(Dbot.^(-.5)); [U,S,V]=svds(Lw) X=[U(:,:nuclusts);V(:,:nuclusts)]; Xsq=X.*X; divat=repat(sqrt(su(xsq ) ),,nuclusts); Y=X./divat; Note that coputing the SVD of the atrix L W = DrowW.5 D.5, gives two sets of eigenvectors, those of L W L W and those of L W L W. The algorith above concatenates these to for the atrix Y (as one would get if perforing spectral clustering on the large atrix A sm D ). This thus provides clusters for each view of each pattern. To get a cluster for the ulti-view pattern, when both views are approxiately equally reliable, the top p rows of the Y atrix can be averaged with the botto p rows before the k-eans step. If one view is significantly ore reliable than the other, one can just use the Y entries corresponding to the ore reliable view (The eigenvectors of L W L W reveal the clustering for the view segents and the eigenvectors of L W L W for the view segents).. It is possible to cobine these ideas and use ultiple views, each (or one) of which is a co-clustering For coparison, we consider the patterns to be in the joint space given by the inputs in the two views. We call this algorith JOINT In this case, we can siply use the standard spectral clustering algorith to deterine clusters. Note that in this case A JOINT (i, j) = e (x i x j ) σ = e (x () i = A v (i, j) σ x () ) + x () x () ) j i j σ σ A v (i, j) σ σ Thus the affinity atrix for clustering in the joint space can be obtained by a coponentwise product (Hadaard product or.* in Matlab) of the affinity atrices for the individual odalities. [As shown above, a person who ignored the ulti-view structure of the data would use one σ for all diensions, however to give this algorith the best chance we allowed the use of different σ and σ.] In other words, we actually used A JOINT (i, j) = A v (i, j) A v (i, j) We also copare our algorith to one where the affinity atrices of the two individual odalities are added. This idea is entioned in (Joachis, 3) for the sei-supervised case. We call this algorith SUM. case A SUM (i, j) = A v (i, j) + A v (i, j).. Theoretical Coparison of Algoriths As discussed in (Ng et al., ), the siplest case for spectral clustering algoriths, is when the affinity atrix is block diagonal. One can easily see that the following stateents are true. Stateent : For consistent block diagonal A v and A v, all 3 algoriths preserve block diagonal for. Stateent : If the affinity atrix in one view is block diagonal but rando in the other then only JOINT results in a block diagonal affinity atrix. When is the sm-d algorith better than the JOINT algorith? Figure shows a siple exaple that shows that clustering in the joint space and M-D style algoriths are not identical. The datapoints are nubered for the purposes of discussion. Consider in particular the ebership of the circled datapoint (4). The sm-d algorith would cluster it with datapoints, and 3. The JOINT algorith is uch ore likely (over a wider range of paraeters and noise levels) to cluster datapoint 4 with datapoints 5,6,7 and 8. To quantify this effect, we constructed an affinity atrix for each view fro the exaple in Figure and - -

4 Marginal densities for two classes in each of two views 7 8 Conditionally Independent View View View Highly Correlated View View Figure. A Consider two classes of patterns with two -D views. The top of the figure represents the density for the two pattern classes (bold and unbold) in View. Assue the arginal densities in are siilar. An exaple scatterplot is shown on the left of the figure. On the right,the sae data is presented in a different forat. Here lines are joined between co-occurring patterns in the two iaginary -D views/odalities (as shown at top). The M-D algorith wants to find a partition that crosses the fewest lines. Two cases are shown for when the views are conditionally independent or highly correlated. In the conditionally independent case, there is a clear non-trivial optial cut. In the correlated case,there are any equally good cuts and the M-D algorith will not perfor well in this case. ran spectral clustering algoriths on noisy versions of these affinity atrices for varying levels of noise and varying cross-cluster strength. A v = A v = The cross-cluster strength relates to the relative spacing between the two clusters with respect to the σ paraeter in the spectral clustering algorith. The results are robust over a broad range of noise levels ( 9 to ). For =, all three algoriths cor- Figure. A siple exaple that would give a different solutions clustered in the joint space JOINT, than if the sm-d algorith was used. rectly cluster nodes -4 and 5-8. However for.5 the JOINT ethod breaks down and groups one of nodes 4 or 5 with the wrong cluster. The SUM algorith breaks down for.8 and the sm- D algorith continues to group appropriately until =.9. Figure 3 explains these results graphically as well as showing the actual (pre-noise) atrices coputed W sm D,A SUM, and A JOINT. 3. Clustering results with the course webpage dataset This dataset consists of two views of web pages. The first view consists of text on the web page and the second view consists of text on the links to the web page (Blu & Mitchell, 998). We use the six class (course, departent, faculty, project, staff, student) version in (Bickel & Scheffer, 4) consisting of tfidf (ter frequency inverse docuent frequency - where a docuent is stored as a vector of weighted words. Tfidf weights words ore if they occur ore in a docuent and downweights words that occur often in the full dataset) vectors without steing. Patterns were noralized within each view so that squared distances reflected the coonly used cosine siilarity easure. We use the average entropy error etric of (Bickel & Scheffer, 4) E = k i ( j p ij log (p ij )) i= where p ij is the proportion of cluster i that is fro ixture coponent j, i is the nuber of patterns in class i and is the total nuber of patterns. On this dataset, with this error easure, perfect agreeent would result in E =, everybody in the sae - 3 -

5 class would give E =.9 (and equal size clusters with probability easureent equal to the base class probabilities also gives E =.). We first copared the algoriths on the full dataset. To do this we first searched for good σ and σ fro clustering in the individual views. We found that (with the proper noralization), the joint ethod worked slightly better(e=.64) than the su (E=.7) and -d version (E=.66) (standard error estiates are provided later when 9% of the data is used). For coparison, Bickel and Scheffer report easures on the sae error easure (with 6 clusters) of approxiately.73 (ulti-view) and.3 (single view) for their ixture-of-ultinoials EM algorith and approxiately.97 (ulti-view) and.7 (single view) for their spherical k-means algorith (Bickel & Scheffer, 4). As entioned, when coputing the SVD of the atrix L W = DrowW.5 D.5, one gets two sets of eigenvectors, those of L W L W and those of L W L W and for equally reliable views, the Y atrices can be averaged before the k-eans step. For this dataset however, view is significantly ore reliable than view and we obtain iproved perforance by siply using the eigenvectors fro view. The ain advantage of our algorith is that it can allow us to cobine sources of inforation with different nubers of views. To see this, reeber that the affinity subatrix W is in ters of how siilar pairs are to co-occurring pairs. Thus a single view pattern x () i fro view does not contribute to the library of paired occurrences but can still be related to patterns x () j in view according to how siilar the pair (x () i, x () j ) is to the set of co-occurring patterns. Thus we can construct a full bipartite affinity atrix between patterns fro view and those fro view using equation where p sus over only the paired patterns. This results in a atrix ultiplication of the for A v A v where this tie A v is (p + ) p diensional and A v is p (p + n) diensional where there are p cooccurring (ulti-view) patterns and patterns with only view and n patterns with only view (see Figure 4). Note that the botto right quadrant of the resulting W atrix coputes the affinity between an unpaired view pattern and an unpaired view pattern according to the su of the affinities between this pair (x () p+i, x() p+j ) and each of the set of observed pairs {(x (), x() ),...(x() p, x () p )}. The affinity between two pairs of patterns is the product between the affinity between each view of each pattern. In this case we use the eigenvectors of L W L W to find the clusters for both the paired and view data and ust use the eigenvectors of L W L W to find the clusters for the data that only has view. For coparison, we consider two other alternatives for clustering data that consists of soe ulti-view patterns and soe single view patterns. Alternative A using JOINT: cluster only the p patterns consisting of x () i and x () j concatenated in the joint space. Spectral clustering will give clusters for these patterns. To report clusters for the + n unpaired patterns, report the cluster of the nearest sae view paired pattern of the pattern. Alternative B: cluster the patterns fro each view separately. In this case the pairing inforation is lost. Results for different values of p are reported in Tables thru 3. Table shows that there is a very slight but significant perforance advantage for the ulti-view patterns using Alternative A when 84 (9%) of the patterns have both views, but that Alternatives B and our sm-d ethod perfor significantly better on the patterns that only have values for view and our sm-d ethod perfors significantly better than both alternatives for patterns that only have values for view. When only 58 (5% ) of the patterns are provided with two views, the sm-d algorith perfors significantly better in all categories. Table 3 shows how the sm-d algorith varies for different nubers of paired patterns. (The slight iproveent in clustering perforance (with increased variance) for the paired view data in the 5% paired case is likely due to an increased chance of not including inappropriate pairs in the paired dataset. Perforance decreases with non independent sources of inforation have been observed with the non-spectral M-D algorith. If leaving out soe data vectors increases the independence between views, we would expect iproved perforance.) Perforance for the single view data is seen to decrease gradually with less paired training data. One value of an algorith that can train with ultiview data and report data for single-view data would be when the single-view data arrive at a later tie. We are working on using the Nystro approxiation (Charless Fowlkes & Malik, 4) for such out of saple estiates. This would allow us to train with paired data and provide cluster labels for later unpaired data. estiated fro their graph - 4 -

6 Table 3. Average Entropy for sm-d for varying aount of two-view data. (See Table for an explanation of ters) 84 (9%) 6 (7%) 58 (5%) 694 (3%) 3 (%) both views.68 ±.3.66 ±.6.64 ±..68 ±..76 ±.3 View only.63 ±..66 ±..66 ±.6.67 ±..73 ±. only.83 ±..9 ±..95 ±.6.97 ±.. ±. Table. Average Entropy where 84 (9%) of the Patterns have both views. Alt. is an abbreviation for Alternative. All values are given ± standard error of the ean over runs. The both view line refers to the error for patterns that had two views, View only refers to errors on patterns that consisted of only view and only refers to errors on patterns that consisted of only. All errors are using the average entropy error easure Alt. A Alt B sm-d both views.66 ±.3.68 ±..68 ±.3 View only.83 ±..64 ±..63 ±. only.95 ±..4 ±.3.83 ±. Table. Average Entropy where 58 (5%) of the Patterns have both views. (See Table for an explanation of ters) Alt. A Alt. B sm-d both views.67 ±..69 ±..64 ±. View only.9 ±..68 ±.6.66 ±.6 only.4 ±.6.4 ±.3.95 ±.6 4. Discussion We have shown that spectral clustering is copetitive in the webpage doain and have introduced a novel ulti-view spectral clustering algorith. While it perfors slightly worse than properly noralized joint spectral clustering in the full webpage doain, the difference is sall and the sm-d algorith has the ajor advantage that it allows single view patterns to benefit fro the paired dataset. This allows one to incorporate all available inforation to for the best clusters when there is lots of single-view data to be clustered. The spectral Miniizing-Disagreeent algorith was otivated by the earlier Miniizing-Disagreeent algorith(de Sa, 994a; de Sa & Ballard, 998) and we believe that of the different ways of spectral clustering with ultiple views, sm-d best incorporates the idea of iniizing the disagreeent of the outputs of two classifiers (clusterers). In the appendix we reproduce an arguent fro (de Sa, 994b; de Sa & Ballard, 998) that otivates, in the -D case, the iniizingdisagreeent approach as an approxiation to iniizing isclassifications. The spectral ipleentation of the Miniizing- Disagreeent idea shares any of the advantages and disadvantages of other spectral techniques. It does not work as well for ulti-class classifications as for binary. It is quick to ipleent and run (with sparse atrices) and has a guaranteed global optiu which is related by a relaxation to the desired optiu. Putting the algorith in the fraework of graph partitioning should allow easier coparison and cobination with results fro clustering in the joint space. Also it should be straightforward to odify the algorith to incorporate soe labeled data so that the algorith can be used in a sei-supervised way. We are currently exploring these avenues. Appendix: Miniizing Disagreeent as an Approxiation to Miniizing Misclassifications The M-D algorith to iniize the disagreeent corresponds to the LVQ. algorith(kohonen, 99) except that the label for each view s pattern is the hypothesized output of the other view. To understand how aking use of this label, through iniizing the disagreeent between the two outputs, relates to the true goal of iniizing isclassifications in each view, consider the conditionally independent (within a class) version of the -view exaple illustrated in Figure 5. In the supervised case (Figure 5A) the availability of the actual labels allows sapling of the actual arginal distributions. For each view, the nuber of isclassifications can be iniized by setting the boundaries for each view at the crossing points of their arginal distributions. However in the self-supervised syste, the labels are not available. Instead we are given the output of the - 5 -

7 other view. Consider the syste fro the point of view of view. Its patterns are labeled according to the outputs of view. This labels the patterns in Class A as shown in Figure 5B. Thus fro the actual Class A patterns, the second view sees the labeled distributions shown. Letting a be the fraction of Class A patterns that are isclassified by view, the resulting distributions of the real Class A patterns seen by view are ( a)p (C A )p(x C A ) and (a)p (C A )p(x C A ). Siilarly Figure 5C shows s view of the patterns fro class B (given View s current border). Letting b be the fraction of Class B patterns isclassified by view, the distributions are given by ( b)p (C B )p(x C B ) and (b)p (C B )p(x C B ). Cobining the effects on both classes results in the labeled distributions shown in Figure 5D. The apparent Class A distribution is given by ( a)p (C A )p(x C A ) + (b)p (C B )p(x C B ) and the apparent Class B distribution by (a)p (C A )p(x C A )+( b)p (C B )p(x C B ). The crossing point of these two distributions occurs at the value of x for which ( a)p (C A )p(x C A ) = ( b)p (C B )p(x C B ). Coparing this with the crossing point of the actual distributions that occurs at x satisfying P (C A )p(x C A ) = P (C B )p(x C B ) reveals that if the proportion of Class A patterns isclassified by view is the sae as the proportion of Class B patterns isclassified by view (i.e. a = b) the crossing points of the distributions will be identical. This is true even though the approxiated distributions will be discrepant for all cases where there are any isclassified patterns (a > OR b > ). If a b, the crossing point will be close. Siultaneously the second view is labeling the patterns to the first view. At each iteration of the algorith both borders ove according to the saples fro the apparent arginal distributions. Acknowledgents Many thanks to Ulf Brefeld, Tobias Scheffer, Steffen Bickel, and Anju Gupta for kindly sending their processed datasets and to Marina Meila and Deepak Vera for providing a great library of spectral clustering code at deepak/spectral/library.tgz. Finally, war thanks to Patrick Gallagher, Jochen Triesch, Serge Belongie and three anonyous reviewers for helpful coents. This work is supported by NSF CAREER grant References Bickel, S., & Scheffer, T. (4). Multi-view clustering. Proceedings of the IEEE International Conference on Data Mining. Blu, A., & Mitchell, T. (998). Cobining labeled and unlabeled data with co-training. Proceedings of the Eleventh Annual Conference on Coputational Learning Theory (COLT-98) (pp. 9 ). Madison, WI. Charless Fowlkes, Serge Belongie, F. C., & Malik, J. (4). Spectral grouping using the nystro ethod. IEEE Transactions Pattern Analysis and Machine Intelligence, 6. de Sa, V. R. (994a). Learning classification with unlabeled data. Advances in Neural Inforation Processing Systes 6 (pp. 9). Morgan Kaufann. de Sa, V. R. (994b). Unsupervised classification learning fro cross-odal environental structure. Doctoral dissertation, Departent of Coputer Science, University of Rochester. also available as TR 536 (Noveber 994). de Sa, V. R., & Ballard, D. H. (998). Category learning through ultiodality sensing. Neural Coputation,, Dhillon, I. S. (). Co-clustering docuents and words using bipartite spectral graph partitioning. KDD. San Francisco, CA. Joachis, T. (3). Transductive learning via spectral graph partitioning. Proceedings of the th International Conference on Machine Learning (ICML 3). J.Shi, & Malik, J. Noralized cuts and iage segentation. IEEE Transactions on Pattern Analysis and Machine Intelligence, Kohonen, T. (99). Iproved versions of learning vector quantization. IJCNN International Joint Conference on Neural Networks (pp. I 545 I 55). Meila, M., & Shi, J. (). Learning segentation by rando walks. Advances in Neural Inforation Processing Systes 3. Ng, A. Y., Jordan, M. I., & Weiss, Y. (). On spectral clustering: Analysis and an algorith. Advances in Neural Inforation Processing Systes 4. Shi, J., & Malik, J. (998). Motion segentation and tracking using noralized cuts. Proceedings of IEEE Conference on Coputer Vision and Pattern Recognition. Zha, H., Ding, C., & Gu, M. (). Bipartite graph partitioning and data clustering. CIKM

8 p p n a) p p View + + W sm D = b) W sm D 3= 4 W A SUM = sm D = 5 6 b ɛ ɛ ɛ ɛ ɛ ɛ A SUM = A SUM = A JOINT = ɛ ɛ ɛ ɛ ɛ ɛ ɛ ɛ ɛ ɛ c) 7 8 e e ɛ ɛ e e ɛ ɛ e e 3 4 ɛ ɛ A A JOINT = ɛ ɛ 5 6 JOINT = e e e e ɛ ɛ e e ɛ ɛ e e ɛ ɛ e e ɛ ɛ Figure 3. The resulting graphs (and atrices) resulting fro the three algoriths a) sm-d b)sum c) JOINT. applied to the atrices A and A above. The light lines correspond to weights of and and the dark lines correspond to weights of and +. In a) the solid lines correspond to co-occurrence lines and the dashed lines, inferred relationships. In c) the faint dotted lines only arise due to noise. The e s in the atrix result only fro the noise and would be different sall nubers at each spot). Each algorith tries to find the sallest noralized cut in its graph Figure 4. A graphical view of the atrix ultiplication required to copute W when there are p patterns with both views, patterns with only view and n patterns with only view. P( A - P(CA)p(x CA) B B - P(CB)p(x CB) A A C a) ( b) B B - (-b)p(cb)p(x CB) - (b)p(cb)p(x CB) B B c) - P(CA)p(x CA) - P(CB)p(x CB) A A B B ( a) A A + b B a D - (a)p(ca)p(x CA) - (-a)p(ca)p(x CA) A A b) a + ( b) A A B B - (a)p(ca)p(x CA)+(-b)P(CB)p(x CB) - (-a)p(ca)p(x CA)+(b)P(CB)p(x CB) B B d) ( a) A A Figure 5. An exaple joint and arginal distribution for a conditionally independent exaple proble. (For better visualization the joint distribution is expanded vertically twice as uch as the arginal distributions.) The darker gray represents patterns labeled A, while the lighter gray are labeled B. (A) shows the labeling for the supervised case. (B) shows the labeling of Class A patterns as seen by view given the view border shown. a represents the fraction of the Class A patterns that are isclassified by view. (C) shows the labeling of Class B patterns as seen by view given the sae view border. b represents the fraction of the Class B patterns that are isclassified by view. (D) shows the total pattern distributions seen by view given the labels deterined by view. These distributions can be considered as the labeled distributions on which view is perforing a for of supervised learning. (However it is ore coplicated as view s border is concurrently influenced by the current position of view s border). See text for ore details