Semi-Supervised Learning with Sparse Distributed Representations

Transcription

1 Semi-Supervised Learning with Sparse Distributed Representations David Zieger CS 229 Fina Project 1 Introduction For many machine earning appications, abeed data may be very difficut or costy to obtain. For instance in the case of speech anaysis, the average annotation time for a one hour teephone conversation transcript is 400 hours.[7] To circumvent this probem, one can use semi-supervised earning agorithms which utiize unabeed data to improve performance on a supervised earning task. Since unabeed data is typicay much easier to obtain, this can be an attractive approach. In this paper, we combine semi-supervised earning with two types of sparse distributed representations: oca and goba. In a oca sparse distributed representation we attempt to find oca sparse features, whie in a goba sparse distributed representation we attempt to represent our test set using a sparse combination of the training set. In both cases, sparsity turns out to have desirabe properties which we can everage in semi-supervised earning. 2 Loca Sparse Distributed Representations Sparse coding is very we suited toward the extraction oca sparse features. The goa of sparse coding is to represent input vectors as a sparse approximate weighted inear combination of basis vectors. That is, for input vector y R k, y j b j s j = Bs (1) where b 1,,b n R k and s R n is a sparse vector of coefficients. Unike simiar methods such as PCA, the basis set B founded in sparse coding can be overcompete (n >k), and can represent noninear features of x. To find the optima B and s we sove the foowing optimization probem as formuated by [3]: minimize b,a x (i) j i a (i) j b j β a (i) 1 (2) s.t b j 2 1, j 1,,s (3) 3 The Structura Sef-Taught Learning Agorithm Traditionay, semi-supervised agorithms have assumed that the unabeed data and the abeed data are drawn from the same distribution. A recent semi-supervised framework proposed by Raina et a[4] caed sef-taught earning abandons this assumption, and ony requires that the unabeed data be within the same probem domain, or modaity. The approach taken by the authors in [4] is to earn a set of bases on the unabeed data via sparse coding, and then represent the abeed data as a sparse inear combination of the earned bases. This aows them to map their abeed data into a higher eve represenation, which they can then perform supervised earning on. The agorithm is summarized beow. This agorithm fas Agorithm 1 Sef-taught Learning via Sparse Coding 1: Input: Labeed training set T = (x (1),y (1) ),, (x (m),y (m) ). Unabeed data x (1) u,,x (k) u. 2: Using unabeed data x (i) u, sove the optimization probem (1) to obtain bases B. Compute features for the cassification task to obtain a new abeed training set ˆT = {(â(x (i) ),y (i) )} m i=1, where â(x (i),y (i) ) = argmin a (i) x (i) sum j a (i) j b j β a (i) 1. Learn a cassifier C by appying a supervised earning agorithm to the abeed training set ˆT. 3: Ouput: Learned cassifier for the cassification task. under the more genera case of sef-taught earning

2 agorithms which we ca structura sef-taught earning. In structura sef-taught earning, we attempt to extract an abstract underying structure from the unabeed data, use this underying structure to project our abeed data into a higher eve represenation, and then perform supervised earning on the abeed data. This approach to sef-taught earning opens up two important questions which we wi attempt to address: 1. How does the choice of our structura mapping function affect performance on our supervised earning task (in this case, sparse coding)? 2. How does the simiarity between the abeed and unabeed data affect performance on our supervised earning task? Whie there has been good theoretica work done recenty in semi-supervised earning[1] and transfer earning[2], none of these approaches are appicabe to the case of sef-taught earning, where we do not assume anything about the distribution of the unabeed data. We define the compatibiity function χ : D u D f [0, 1] reating the compatabiity between unabeed distribution D u, abeed distribution D, and structura mapping function f, as χ(d u,d,f) = 1 E x D [χ(x u,x,f)]. In the case of sparse coding, χ(x u,x,a) = 1 E[x Bâ(x )] (4) Essentiay, we measure the compatibiity between an unabeed distribution, a abeed distribution, and a mapping function, by the reconstruction error with respect to the the optima basis found over the unabeed data. 4 MNIST Experiments For our experiments with sef-taught earning, we used the MNIST handwritten digits dataset for abeed data. Unabeed data was taken from three sources: natura images, computer font characters, and MNIST. The number of bases was kept fixed at 512 and a images were resized to pixes. We used two cassifiers: a inear and gaussian svm and compared the performance between the raw pixe intensities and the sparse codes. The resuts generay support our hypothesis that a higher compatibiity function yieds better resuts on the sparse coding cassification reative to the raw pixe intensities (figure 1). The resuts are more pro- Figure 2: Sampe bases earned from natura images nounced for the inear cassifier. This coud be that the gaussian kerne is better suited towards raw pixe intensities, and we note that to aow for a fair comparison between the raw and sparse coding features, we did not use the sparse coding kerne derived in [4]. 5 Goba Sparse Distributed Representations A different approach which we ca a goba sparse distributed representation is to represent features from a test exampe using a sparse combination of features from the training set. Unike oca sparse distributed representations, it turns out that within the context of a goba sparse distributed representation, the seection of features becomes competey irreevant. Recent research has shown that for face recognition, if the representation is sparse enough, random inear projected features perform just as we or better than feature seection for via Eigenfaces, Lapacianfaces, or Fisherfaces [6]. Buiding off of the pattern recognition agorithm used in [6] we devise a simpe semisupervised earning agorithm which utiizes a bootstrapping approach to expand our training set. We assume that images within the same category ie on a ow-dimensiona inear subspace. The k individua subspaces can be represented by matrices A 1,A 2,,A k where each coumn in A i is a training sampe from cass i. Let y be an image from the test set, and A =[A 1 A 2 A k ]. Then ideay, y = Ax 0 R m (5) 2

3 !"#$%&#&'()$*$ +$%&#&'(,-$."."/(0&*(0.1& 2$3(4+."&$-5 06(4+."&$-5 2$3( ( $*:-$#(;<$/&= I?BCID C?? I?BGID ICBEID >??? I?BEID IGBCED IEB?GD!"#$%&#&'()$*$ +$%&#&'(,-$."."/(0&*(0.1& 2$3(4+."&$-5 06(4+."&$-5 2$3( ( ;0, C?? I?BGID IGBC>D ICBEID >??? I?BEID IFBGC ICBAI!"#$%&#&'()$*$ +$%&#&'(,-$."."/(0&*(0.1& 2$3(4+."&$-5 06(4+."&$-5 2$3( ( @GBCHD C?? I?BGID I?BI>D ICBEID IHBEID >??? I?BEID I>BI?D IFB?ID +,-./ +,-0-0 +,-0.1 +,-23. +,-233 +,-232 Figure 1: MNIST Resuts where x 0 = [0,, 0,α i,1,α i,2,,α i,ni, 0,, 0, ] T R n is a coefficient vector whose entries are mosty zero except those associated with category i. Thus, a new test image shoud be predominanty represented using training images from the same subject, and this representation wi be naturay sparse if the number of categories k is reasonaby arge. Let R R dxm be a transform matrix with d m. Mutipying both sides of (5) by R yieds ỹ = Ry = RAx 0 = Ãx 0. Recent deveopments in compressed sensing have shown that the desired x 0 is the soution to the 1 minimization probem: min x 1 s.t. ỹ = Ãx (6) Furthermore, if x has p n nonzeros, then d 2p og(n/d) (7) random measurements are sufficient for sparse recovery with high probabiity[6]. Thus even with randomy seected facia features, if the dimension of the image is sufficienty arge then we shoud sti be abe to cassify the test image. Let δ i (x) R n be a vector whose ony nonzero entries are the entries in x associated with category i. The cassification agorithm is then given by Agorithm 2 [6]. Note that the minimization probem in step 3 is sighty different than in (6) to mode noise and error in the data. 6 Semi-Supervised Learning in Goba Sparse Distributed Representations We define the Sparsity Concentration Index as in [6]: SCI(x) = k k 1 max i δ i (x) 1 / x 1 1 [0, 1] (8) The SCI provides a measure of how concentrated the coefficients are on a specific category. When SCI = 1, the test image is represented using ony images from a singe subject, and when SCI = 0, the coefficients are spread eveny over a categories. We aso define ˆr to be the ratio between the smaest and second smaest residua. The residuas cacuated in Agorithm 1 measures how we the the sparse representation approximates the test image, whie the SCI and ˆr give us a measure of how good the representation is in terms of ocaization. Agorithm 2 Recognition via Sparse Representation 1: Input: a matrix of training images A R m n for k subjects, a inear feature transform R R d m, a test image y R m, and an error toerance ɛ. 2: Compute features ỹ = Ry and à = RA and normaize ỹ and coumns of à to unit ength. 3: Sove: min x 1 s.t. ỹ Ãx 2 ɛ 4: Compute residuas r i (y) = ỹ Ãδ i(x) 2 for i = 1,,k 5: Output: identity(y) = argmin i r i (y) After running Agorithm 3 on our unabeed data, we then run Agorithm 2 to perform the supervised earning task. Aternativey, we can run Agorithm 3 on the test set, augmenting our training set with images 3

4 Agorithm 3 Bootstrapping a Goba Sparse Representation Cassifier 1: Input: a matrix of training images A R m n for k subjects, a inear feature transform R R d m, a matrix of unabeed images Y R m q, and an error toerance ɛ. 2: oop 3: numlabeed =0 4: for a y Y do 5: Compute features ỹ = Ry and à = RA and normaize ỹ and coumns of à to unit ength 6: Sove: min x 1 s.t. ỹ Ãx 2 ɛ 7: Compute residuas r i (y) = ỹ Ãδ i(x) 2 for i =1,,k 8: if SCI(x) τ and ˆr σ then 9: j = identity(y) = argmin i r i (y) 10: Append y to A j and remove y from Y 11: Update: A =[A 1 A 2 A k ] 12: numlabeed+ =1 13: end if 14: end for 15: if numlabeed =0then 16: break 17: end if 18: end oop 19: Output: Augmented training matrix A R m p, p n Origina image Random projection feature Figure 4: Random Feature Exampe from the test set, and then run Agorithm 2 on the remaining test images. This agorithm has two potentia drawbacks. The first is that if τ and σ are set impropery, then we might add miscassified images to our training set, which woud then cause compounding errors in ater cassifications, thus obscuring the categories in our training set. In our experiments this probem did not arise, but this is an inherent weakness in any bootstrapping agorithm. However, so ong as the conditions in (7) are satisfied, the probabiity of a sparse incorrect reconstruction occurring has very ow probabiity. Thus by setting τ and σ propery, the probabiity of a miscassified images being added to the training set is very ow. The other concern is raised by the theory underying (7), which is that by increasing our training set whie keeping the dimensionaity and number of categories fixed, we reduce the probabiity that we can successfuy reconstruct the test image since we are potentiay decreasing the sparsity of x. For category i with n i exampes, and R R m n if n i + support(ɛ) > m/3 (9) then we shoud not expect to perfecty recover x and ɛ [5]. However, this is generay not a probem since so ong as the dimension of R is arge enough. If one suspects that this might be a probem, one coud stop the agorithm once the sparsity started getting cose to the bound in (4). 7 Experiments with Face Recognition Figure 3: Top:Residuas, Bottom: x obtained from (6) Experiments were conducted the Extended Yae B database. The database consists of 2, 414 cropped and normaized fronta-face images of 38 individuas under various aboratory controed ighting conditions. The feature space dimension chosen was 504. We first randomy spit haf the data into a training 4

5 Figure 5: Training set augmented with unabeed set Figure 6: Training set augmented with test set set and haf into a test set. Then a portion of the training set was randomy spit off and randomy permuted into a third unabeed training set. We then ran our bootstrapping agorithm on this unabeed set to augment our training set, and then tested our augmented training set with the separate test set (figure 5). The agorithm does demonstrate significant improvement on the test set, athough the amount it improves performance seems to decrease asymptoticay as the initia training set size is increased. We aso ran a third experiment where we used the entire training set and ran the bootstrapping agorithm on the test set, augmenting the training set with the test set. We again get a performance improvement, athough the effect is sma since we start out with a arge training set (figure 6). In a of the bootstrapping experiments, τ and σ were set conservativey to ensure that no unabeed data was added to the training set and miscassified. In a the above experiments, post-experiment anaysis showed that the unabeed data which was added to the training set was a correcty cassified. 8 Future Work and Concusions We provided a way which aows one to quantitativey determine how the compatibiity between the unabeed and abeed data wi affect performance on the supervised earning task for a given structura mapping function. To make this resut more robust, it woud be nice to show this compatibiity with nonimage data, and to experiment with different mapping functions such as PCA. Aso, it woud be interesting to see at what point of incompatibiity the unabeed data stopped heping performance on the supervised task. recent deveopment, with many attractive properties. In particuar, the abiity to cassify using random features. The bootstrapping agorithm shown here is fairy simpistic but surprisingy robust and hepfu in boosting performance. To expore this agorithm further, it woud be usefu to perform experiments where the parameters were set too ow, causing miscassifications to enter our training set, and see how performance degrades as a resut. 9 Acknowedgements Thanks to Hongak Lee for heping me with my sparse coding impementation. References [1] M. Bacan and A. Bum. A pac-stye mode for earning from abeed and unabeed data, [2] J. Baxter. Theoretica modes of earning to earn, [3] Hongak Lee, Aexis Batte, Rajat Raina, and Andrew Y. Ng. Efficient sparse coding agorithms. In Neura Information Processing Systems, [4] Rajat Raina, Aexis Batte, Hongak Lee, Benjamin Packer, and Andrew Y. Ng. Sef-taught earning: transfer earning from unabeed data. In ICML 07: Proceedings of the 24th internationa conference on Machine earning, pages , New York, NY, USA, ACM. [5] John Wright, Arvind Ganesh, Aen Yang, and Yi Ma. Robust face recognition via sparse representation, [6] Aen Y. Yang, John Wright, Yi Ma, and S. Shankar Sastry. Feature seection in face recognition: A sparse representation perspective. Technica Report UCB/EECS , EECS Department, University of Caifornia, Berkeey, Aug [7] X. Zhu. Semi-supervised earning iterature survey. Technica Report TR 1530, Computer Sciences Department, University of Wisconsin - Madison, June The goba sparse distributed representation is a fairy 5