Transductive Regression Piloted by Inter-Manifold Relations

Transcription

1 Huan Wang IE, The Chnese Unversty of Hong Kong, Hong Kong Shucheng Yan Thomas Huang ECE, Unversty of Illnos at Urbana Champagn, USA Janzhuang Lu Xaoou Tang IE, The Chnese Unversty of Hong Kong, Hong Kong Abstract In ths paper, we present a novel semsupervsed regresson algorthm workng on multclass data that may le on multple manfolds. Unlke conventonal manfold regresson algorthms that do not consder the class dstncton of samples, our method ntroduces the class nformaton to the regresson process and tres to explot the smlar confguratons shared by the label dstrbuton of mult-class data. To utlze the correlatons among data from dfferent classes, we develop a cross-manfold label propagaton process and employ labels from dfferent classes to enhance the regresson performance. The nterclass relatons are coded by a set of ntermanfold graphs and a regularzaton tem s ntroduced to mpose nter-class smoothness on the possble solutons. In addton, the algorthm s further extended wth the kernel trck for predctng labels of the out-ofsample data even wthout class nformaton. Experments on both syntheszed data and real world problems valdate the effectveness of the proposed framework for semsupervsed regresson.. Introducton Large scale and hgh dmensonal data are ubqutous n real-world applcatons, yet the processng and Appearng n Proceedngs of the 24 th Internatonal Conference on Machne Learnng, Corvalls, OR, Copyrght 2007 by the author(s)/owner(s). analyss of these data are often dffcult due to the curse of dmensonalty as well as the hgh computatonal cost nvolved. Usually, these hgh dmensonal data le approxmately on an underlyng compact low dmensonal manfold, whch may turn the problem tractable. Substantve works have been devoted to unvelng the ntrnsc structure of the manfold data, among whch the popular ones nclude ISOMAP (Tenenbaum et al., 2000), LLE (Rowes & Saul, 2000), Laplacan Egenmap (Belkn & Nyog, 2003), and MVU (Wenberger et al., 2004). Besdes these unsupervsed algorthms that purely explore the manfold structure of the data, researchers have also well utlzed the latent manfold structure nformaton from both labeled and unlabeled samples to enhance learnng algorthms wth lmted number of labeled samples. Great success has been acheved n varous areas, such as classfcaton (Krshnapuram et al., 2005) (Belkn et al., 2005), manfold algnment (Ham et al., 2005) and regresson (Belkn et al., 2004) (Zhu & Goldberg, 2005). Recently ncreasng attenton has been drawn to the semsupervsed regresson problem by consderng the manfold structure. (Chapelle et al., 999) proposed a transductve algorthm mnmzng the leave-one-out error of the rdge regresson on the jont set composed from both labeled and unlabeled data. To explot the manfold structure, (Belkn et al., 2004) adds a graph Laplacan regularzaton tem to the regresson objectve, whch mposes extra condton on the smoothness along the data manfold and t has proved to be qute useful n applcatons. (Cortes & Mohr, 2007) frst roughly transduces the functon values from the labeled data to the unlabeled ones utlzng local neghborhood relatons, and then optmzes the global objectve that best fts the

2 Transductve Regresson Ploted by Inter-Manfold Relatons labels of the tranng ponts as well as the estmated labels provded by the frst step None of these state-of-the-art regresson algorthms make use of the class nformaton to gude the regresson. In real-world applcatons, however, multclass samples are ubqutous and samples from dfferent classes can be regarded as lyng on multple manfolds. For example, n age estmaton, the personal agng process vares dfferently for dfferent genders, but there stll exsts consstency between the agng processes of male and female. In the pose estmaton from mages of multple persons, each person can be regarded as one class and the mages vary smlarly wth poses from dfferent persons. Moreover, we may also encounter the mult-modalty samples whose representatons may not be consstent due to the lack of correspondence, whle the nner manfold confguraton for the label dstrbuton across dfferent modaltes may stll be smlar. Besdes, n the functon learnng stage of the semsupervsed regresson framework, the class or modalty nformaton s usually easy to obtan, thus t s desrable to utlze ths nformaton to further boost the regresson accuracy. To fully explot the relatons among data manfolds, we develop n ths paper a semsupervsed regresson algorthm called TRIM (Transductve Regresson ploted by Inter-Manfold Relatons). TRIM s based on the assumpton that the data of dfferent classes share smlar confguraton for label dstrbutons. In our proposed algorthm, manfolds of dfferent classes are algned accordng to the landmark connectons constructed from both the sample label dstance and manfold structural smlarty, and then the labels are transduced across dfferent manfolds based on the algnment output. Besdes the ntra-manfold smoothness condton wthn each class, our method ntroduces an nter-manfold regularzaton tem and employs the transduced labels from varous data manfolds to plot the trend of the functon values over all the samples. In addton, the functon to be learnt can be approxmated by the functonals lyng on the Reproducng Kernel Hlbert Space (RKHS), and the regresson on the nduced RKHS drectly leads to an effcent soluton for the label predcton of the out-of-sample data even wthout correspondng class nformaton. 2. Background 2.. Intutve Motvatons Consder the problem of adult heght estmaton based on the parents heghts. Dramatc dfference exsts between male and female adults, whle the general confguraton of the heght dstrbuton wthn each gender Fgure. Demonstraton of the nter-class relatons. Labeled samples are marked by + and the nter-manfold relatons are ndcated by the dashed lnes. The manfolds are algned and the sample labels are propagated across manfolds to plot the regresson. may stll be smlar: the taller the parents are, the hgher the chldren are supposed to be. Whle once we mx up the data from both genders, the heght dstrbuton may become complcated. Fgure dsplays a toy example for the obstacles encountered n the multclass regresson problem. We have two classes of samples and the functon values wthn each class manfold share smlar confguratons, whle the number of samples and labels may vary across classes. As s shown n the upper part of Fgure, t s dffcult to get a satsfactory regresson f we put all the samples together due to the complex structures produced by the manfold ntersectons. On the other hand, f the regresson s carred out separately on dfferent classes, the accuracy could stll be low due to the lack of suffcent labeled samples for each class. Also, for the ncomng out-of-sample data, the predcton cannot be done wthout class nformaton. Moreover, we may face some regresson problems wth mult-modalty samples, e.g., the estmaton of human ages from both photos and sketch mages. It s meanngless to compose data of dfferent modaltes nto one manfold snce the semantcs of the features from dfferent modaltes are essentally dfferent, whle t s stll possble that the ntra-modalty label confguratons are smlar. In ths paper we focus on the mult-class regresson problem, but our framework can also be easly extended to gve predctons for the mult-modalty data Related Work Mult-vew learnng (Blum & Mtchell, 998) technque can also be appled to the regresson problem wth mult-class data. Denote the labeled and unlabeled nstances as xl X l X and xu X u X respectvely, where X l, X u, X are the labeled, unlabeled and whole sample sets. State-of-the-art semsupervsed mult-vew learnng frameworks employ multple regresson functons from the Hlber space Hv so that the estmaton error on the tranng set and dsagreement among the functons on the unlabeled data

3 are mnmzed (Brefeld et al., 2006),.e., f v M l v= = mn f v H v= M l ( +λ x X l c(f v (x l ),y(x l )) + γ f v 2 ) M l v,v 2= x X u c(f v (x u ) f v2 (x u )), () where y(x) s the sample label, f v M l v= are M l multple learners and c( ) s a cost functon. We would lke to hghlght beforehand some propertes of our framework compared to the mult-vew learnng algorthms:. There exsts a clear correspondence between the mult-vew data from the same nstance n the mult-vew learnng framework, whle our algorthm does not requre the correspondence among dfferent manfolds. The data of dfferent modaltes or classes may be obtaned from dfferent nstances n our confguraton, thus t s much more challengng to gve an accurate regresson. 2. The class nformaton s utlzed n two ways: a) Sample relatons wthn each class are coded by ntra-manfold graphs and a correspondng regularzaton tem s ntroduced to ensure the wthnclass smoothness separately. b) A set of nter-manfold graphs are constructed from the cross manfold label propagaton on the algned manfolds and an nter-manfold regularzaton tem s proposed to fully explot the nformaton conveyed among dfferent classes. 3. The class nformaton s used n the functon learnng phase but no class attrbutes are requred for the out-of-sample data n the predcton stage. 3. Problem Formulaton 3.. Notatons Assume that the whole sample set X conssts of N samples from M classes, denoted as X,X 2,...,X M. For each class X k, N k = l k + u k samples are gven,.e., X k = {(x k,y k ),...,(x k l k,yk l k),(xk l k +,y k l k + ),...,(x k l k +u k,y k l k +u k)}, where Yl k = [y,y k 2,...,y k k l ] T represents the functon k values wth respect to the gven labeled samples and Yu k = [y k l k +,...,y k l k +u ] T corresponds to the functon k values of the remanng unlabeled data to be estmated n the k th class. In ths paper, the label refers to a real value to be regressed. Let G k = (V k,e k ) denote the ntra-class graph wth vertex set V k and edge set E k constructed wthn the data of the k th class. Here we focus on the undrected graph and t s easy to generalze our algorthm for drected graphs. The edges n E k reflect the neghborhood relatons along the manfold data, whch can be defned n terms of k-nearest neghbors or an ǫ-ball dstance crteron n the sample feature space F. One choce for those non-negatve weghts on the correspondng edges s to use the heat kernel (Belkn & Nyog, 2003) or the nverse of feature dstances (Cortes & Mohr, 2007),.e., w j =e Φ k (x ) Φ k (x j ) 2 t or w j = Φ k (x ) Φ k (x j ), where t R s the parameter for the heat kernel and Φ k ( ) s a feature mappng from X to the normed feature vector space F for the sample of the k th class. For samples from dfferent modaltes/manfolds, the feature mappngs may be dfferent. The other one s to solve a least-square problem to mnmze the reconstructon error and get the weghts w j : w j = arg mn w j x j s.t. j (Rowes & Saul, 2000). w j x j 2, w j =,w j 0 To encode the mutual relatons among dfferent sample classes, we also ntroduce the nter-manfold graph, denoted as a trplet G kkj = (V k,v kj,e kkj ). The nter-manfold graph G kkj s a bpartte graph wth one vertex set from the k th class and the other from the kj th. The constructon of wll be dscussed n Gkkj the followng subsectons Regularzaton along Data Manfold Now we are gven the sample data of multple classes, denoted as X, ncludng both labeled and unlabeled samples X l and X u. The manfold structures wthn each class data are encoded by the ntra-class graph. (Belkn et al., 2004) ntroduced a manfold regularzaton tem for the semsupervsed regresson framework,.e., the graph Laplacan, whch s expected to mpose smoothness condtons along the manfold on the possble solutons. The fnal formulaton seeks a balance between the fttng tem and the smoothness regular-

4 zaton,.e., Transductve Regresson Ploted by Inter-Manfold Relatons f = arg mn (f y ) 2 + γf t L p f, (2) l where p N. When p =, the regularzaton tem s the graph Laplacan and for p = 2, the regularzaton turns out to be the 2-norm of the reconstructon error when the weghts w j are normalzed,.e., f T L T Lf = w.r.t. j f j w j f j 2 w j =,w j Cross Manfold Label Propagaton In our confguraton, there does not exst a clear correspondence among the manfolds of dfferent classes or modaltes, and even the representatons for dfferent modaltes can be dstnct. Thus t s rather dffcult to construct sample relatons drectly from the smlarty n the sample space X. Alternatvely, the functon labels stll convey some correspondence nformaton, whch may be utlzed to gude the nter-manfold relatons. Moreover, the manfold structure also contans some ndcatons about the correspondence. Thus we frst seek a pont-to-pont correspondence for the labeled data combnng the ndcatons provded by both the sample labels and the manfold structures. Ths s done under two assumptons:. Samples wth smlar labels le generally n smlar relatve postons on the correspondng manfold. 2. Correspondng sample sets tend to share smlar graph structures on the respectve manfold Renforced Landmark Correspondence Frst we search for a set of stable landmarks to gude the manfold algnment. Specfcally, we use the ǫ- ball dstance crteron on the sample labels to ntalze the nter-manfold graphs. To gve a robust correspondence, we renforce the nter-manfold connectons by teratvely mplementng W kkj W k W kkj W kj. (3) Smlar to the smlarty propagaton process on the drected graph (Blondel et al., 2004), (3) renforces the smlarty score of sample pars by the smlarty of ts neghbor pars,.e., W = W W W AB A AB B ab 2 aa 2 ab 2 bb 2 + W W W A AB B aa 3 ab 3 4 bb W W W A AB B aa 4 ab 4 3 bb 2 3 a 6 a 2 a 3 Fgure 2. Demonstraton of renforced landmark correspondence. The smlarty score of a and b 2 s renforced by the scores of neghbor pars. The assumpton here s that two nodes tend to be smlar f they have smlar neghbors. (4) utlzes the ntra-manfold structure nformaton to renforce the nter-manfold smlarty and thus can generate a more robust correspondence. The accuracy of the landmark correspondence s crtcal for our algorthm. To ensure a robust performance, only the correspondences wth the top 20% largest smlarty scores are selected as the landmarks and t s common that some classes may mss certan sample labels, so plenty of labeled samples reman unmatched. a Manfold Algnment To propagate the sample labels to the unlabeled ponts across manfolds, we stretch all the manfolds wth respect to the landmark ponts obtaned from the prevous step and ths can be realzed by the semsupervsed manfold algnment (Ham et al., 2005). In the manfold algnment process, we seek an embeddng that mnmzes the correspondence error on the landmark ponts and at the same tme keeps the ntramanfold structures,.e., and f k M C(f k M k k = = arg mn( ) = k f kt ), (5) D k k f C(f k M k =) = k k j w kkj j +γ a 4 a 5 f k x k b b 2 f kj x k j j 2 b 3 b 4 M (f kt L p k f k ) + βf T L a f, (6) k = where D k s the dagonal degree matrx, L k s the graph Laplacan matrx for the k th class and p N. To ensure the nter-class adjacency, we add a global compactness regularzaton tem βf T L a f to the cost functon C, where L a s the Laplacan Matrx of W a wth { f x and x j are of dfferent classes w a j = 0 o.w. w kkj j m,n w k m wkkj mn w kj nj. (4) The labels are propagated across manfolds on the derved algned manfolds usng the nearest neghbor ap-

5 Algorthm Procedure to construct nter-manfold connectons : Inter-manfold graph ntalzaton { f y k w kkj j = 0 o.w. 2: Correspondence Renforcement for Iter = : N Iter y kj j 2 < ǫ W kkj = W k W kkj W kj end 3: Landmark Selecton: Select the sample pars wth the top 20% largest smlarty scores as correspondence. Set the correspondng elements n W kkj as and others 0. 4: Manfold Algnment usng the Inter-manfold Graphs W kkj. 5: Fnd the correspondng ponts from dfferent classes for the unmatched labeled samples usng the nearest neghbor approach n the algned space and update the nter-manfold graphs W kkj. proach,.e., we connect the labeled samples wth the nearest ponts from other classes on the algned space. The derved nter-manfold graphs are concatenated to form W r = O W 2... W M W 2 O... W 2M W M W M2... O, (7) whch s then symmetrzed and employed n the followng Inter-Manfold regularzaton tem Inter-Manfold Regularzaton We rearrange the samples to place the data from the same class together and put the labeled ponts frst for each class, that s, X = {x,x 2,...,x l,x l +,...,x l +u,x2,x 2 2,...,x 2 l 2,x2 l 2 +,...,x 2 l 2 +u 2,...,xM,x M 2,...,x M l M,xM l M +,...,xm l M +u M }. Denote the correspondng functon values as f, whch s a concatenaton of f k = [f k,...,f k ] T from all x k x k l k +u k the classes. Our regresson objectve s defned as: f = arg mn f l k k +β k f k x y k k 2 x k X l (N k ) 2 (fk ) T L p k fk + λ N 2 ft L r f, (8) where L r s the Laplacan matrx of the symmetrzed W r. The mnmzaton of the objectve s acheved when where f = R k l k (Sk l ksk ) T Y k l (9) S k l =( I k lk l O ) k l k u k, ( O S k k = Nk P I Nk N O k P N k N k M k = N k k =k+ are label and class selecton matrces respectvely and R = k + β k l k (Sk l ksk ) T (S k l ksk ) (N k ) 2 SkT L p k Sk + λ N 2 Lr. 4. Regresson on Reproducng Kernel Hlbert Space (RKHS) A seres of algorthms such as SVM, Rdge regresson and LapRLS (Belkn et al., 2005) employ dfferent regularzaton tems and emprcal cost measures to the objectve and solve the optmzaton problem n an approprately chosen Reproducng Kernel Hlbert Space (RKHS). One mert for the regresson on RKHS s the ablty to predct out-of-sample labels. Let K denote a Mercer kernel: X X R and H K denote the nduced RKHS of functons X R wth norm K. The regresson wth nter-manfold regularzaton on the RKHS s defned as f = arg mn f H K l k f k x y k k 2 + γ f 2 K k x k X l +β k (N k ) 2 (fk ) T L p k fk + λ N 2 ft L r f. (0) Smlar to the Tkhonov regularzaton, we add an RKHS norm penalzaton tem to the TRIM algorthm as a smoothness condton. )

6 MAE: MAE:23.87 MAE: (a) (b) (c) (d) Fgure 3. Regresson on the nonlnear Two Moons Dataset. (a) Orgnal Functon Value Dstrbuton. (b) Tradtonal Graph Laplacan Regularzed Regresson (separate regressors for dfferent classes). (c) Two Class TRIM. (d) Two Class TRIM on RKHS. Note the dfference n the area ndcated by the rectangle. For the mult-class data under the same modalty, we have the followng theorem: Theorem-. The soluton to the mnmzaton problem (0) admts an expanson f(x) = N= P k (lk +u k ) = α K(x,x) () Theorem- s a specal verson of the Generalzed Representer Theorem (Schlkopf et al., 200) and the proof s omtted here. It says that the mnmzer of (0) can be expressed n terms of the lnear expanson of K(x,x) on both labeled and unlabeled data over all the sample classes. Thus the mnmzaton over Hlbert space bols down to mnmzng the coeffcent vector α = [α,...,α l,...,α l +u,...,α M,...,α M l,...,α M M l M +u ] T M over R N and the mnmzer s gven by: where J = k α = J k + β k l k (Sk l ksk ) T Y k l, (2) l k (Sk l ksk ) T (S k l ksk )K + γi (N k ) 2 SkT L p k Sk K + λ N 2 Lr K. and K s the N N Gram matrx of labeled and unlabeled ponts over all the sample classes. For the out-of-sample data, the estmated labels can be obtaned usng: y new = N= P k (lk +u k ) = α K(x,x new ). (3) Note here n ths framework the class nformaton for the ncomng sample s not requred n the predcton stage. 5. Experments We performed experments on two synthetc datasets and one real-world regresson problem of human age estmaton. Comparsons are made wth the tradtonal graph Laplacan regularzed semsupervsed regresson (Belkn & Nyog, 2003). We also evaluate the generalzaton performance of the mult-class regresson on RKHS. In the experments, the ntra-manfold graphs are constructed usng 0 nearest neghbors and the nter-manfold graphs are constructed by followng the procedure descrbed n Secton 3. For all the confguratons, the parameter β for the ntra-manfold graph s emprcally set as 0.00 and the λ for the nter-manfold regularzaton s set as 0.. For the kernelzed algorthms, the coeffcent for the RKHS norm γ = 0.00 and the gaussan kernel K(x,y) = exp{ x y 2 /δo} 2 wth parameters δ o = 2 /2.5 δ s appled, where δ s the standard devaton of the sample data. We use the Mean Absolute Error (MAE) crteron to measure the regresson accuracy and t s defned as an average of the absolute errors between the estmated labels and the ground truth labels. 5.. Synthetc Data: Nonlnear Two Moons The nonlnear two moons dataset s shown n Fgure 3. The colors n the fgure are assocated wth the functon values and the varaton of those sample labels along the manfold s not unform. The labeled samples are marked by + and ther dstrbutons are qute dfferent across classes. As we can see, the sample labels for the class lyng on the upper part of the fgure are not enough to gve an accurate gudance for the regresson on the nonlnear label dstrbuton. The tradtonal graph Laplacan regularzed regresson algorthm does not make use of the nformaton conveyed by the nter-class smlarty and the predcton result s not satsfactory, whle n our algorthm the sample labels from dfferent classes can be utlzed to gude the regresson and thus estmaton accuracy s much hgher.

7 MAE: MAE:44. MAE:07.64 (a) (b) (c) (d) Fgure 4. Regresson on Cyclone Dataset: (a) Orgnal Functon Values. (b) Tradtonal Graph Laplacan Regularzed Regresson (separate regressors for dfferent classes). (c) Three Class TRIM. (d) Three Class TRIM on RKHS Synthetc Data: Three-class Cyclones One mert of our algorthm s that the regresson accuracy may be boosted as the class number ncreases. Ths s because the cross manfold nformaton that could be utlzed grows rapdly as the class number ncreases. The Cyclone dataset conssts of three classes of samples and the class dstrbuton s demonstrated n Fgure 5. The label dstrbutons among dfferent classes are qute smlar, whle the labeled samples scatter dfferently from class to class. As we may observe from Fgure 4, wthout the nter-class regularzaton, the regresson for certan class may fal due to the lack of suffcent labeled samples whle our algorthm stll gves a satsfyng performance Human Age Estmaton In ths experment we consder as a regresson problem the age estmaton from facal mages. In real applcatons, we often cannot obtan enough age labels though we may easly get plenty of facal mages. Those unlabeled data can be used to gude our regresson n the semsupervsed framework. One challenge for ths estmaton s caused by the gender dfference. Although the male and female may share some smlar confguratons n the age dstrbuton, just as demonstrated n Fgure, mxng the two may complcate the regresson problem. On the other hand, the gender nformaton s usually easy to obtan and t s thus desrable to use both the age labels and gender nformaton to gude the regresson. The agng data set used n ths experment s the Yamaha database, whch contans 8000 Japanese facal mages of 600 persons wth ages rangng from 0 to 93. Each person has 5 mages and the Yamaha database s dvded nto two parts wth 4000 mages from 800 males and another 4000 mages for 800 females. The ages dstrbute evenly from 0 to 69 wth 3500 male mages and 3500 female mages, and the rest are dstrbuted n the age gap from 7 to 93. We randomly sampled 000 photos from the male and female Fgure 5. Class Dstrbuton of the Cyclone Dataset subset respectvely and thus altogether 2000 mages are used n our experments. Before the regresson, the nput mage data s preprocessed wth Prncpal Component Analyss and the frst 20 dmensons are used for data projecton. To fully evaluate the regresson performance for both close set samples and those out-of-sample data, we desgn two expermental confguratons for ths dataset. Confguraton : Close Set Evaluaton. In ths confguraton, we do not have the out-of-sample data and the close set performance of TRIM s evaluated n comparson wth the tradtonal graph Laplacan regularzed regresson (Belkn et al., 2004). The close set contans altogether 2000 samples wth 000 mages from males and 000 mages from females respectvely. We vary the number of randomly selected labeled samples and examne the performance of dfferent regresson algorthms. The comparson results between TRIM and the tradtonal sngle class Laplacan Regularzed regresson are shown n Fgure 6. We can observe that our algorthm generally gves a hgher regresson accuracy and the performance mprovement s remarkable especally when the number of labeled samples s small. As the number of sample labels ncreases, the dfference between the two algorthms becomes smaller. Ths may be caused by the fact that as the labels are sparse, the label gudance s far less enough and thus the class nformaton and nter-manfold relatons wll have a greater nfluence on the regresson accuracy whle when the labels are abundant enough to gude the regresson, the class nformaton as well as those nter-manfold connectons becomes less mportant. Confguraton 2: Open Set Evaluaton. Now we examne the out-of-sample predcton performance for

8 Fgure 6. TRIM vs tradtonal graph Laplacan regularzed regresson for the close set evaluaton on YAMAHA database. (a) (b) Fgure 7. Open set evaluaton for the kernelzed regresson on the YAMAHA database. (a) Regresson on the tranng set. (b) Regresson on out-of-sample data. the kernelzed TRIM compared wth kernelzed verson of the tradtonal graph Laplacan regularzed regresson (Belkn et al., 2005). In ths confguraton, the sample set s dvded nto two subsets: one for the regresson functon tranng and the other for the evaluaton of agng estmaton performance on those outof-sample data. The tranng set contans randomly selected 800 male and 800 female mages and the remanng ones are used for out-of-sample evaluaton. In the testng phase we do not nput any gender nformaton. As s demonstrated n Fgure 7, our algorthm acheves a lower MAE n both tranng and testng sets. Another observaton smlar as n the case for close set confguraton s that the regresson accuracy mprovement grows as the sample labels become sparser. 6. Conclusons In ths paper, we have presented a novel algorthm dedcated for the regresson problem on mult-class data. Manfolds constructed from dfferent classes are regularzed separately and to utlze the nter-manfold relatons, we developed an effcent cross manfold label propagaton method and the labels from dfferent classes can thus be employed to plot the regresson. Moreover, the regresson functon s further extended by the kernel trck to predct the labels of the out-of-sample data wthout class nformaton. Both syntheszed experments and real world applcatons demonstrated the superorty of our proposed framework over the state-of-the-art semsupervsed regresson algorthms. To the best of our knowledge, ths s the frst work to dscuss the semsupervsed regresson problem on mult-class data. 7. Acknowledgement The work descrbed n ths paper was supported by the Research Grants Councl of the Hong Kong SAR (Project No. CUHK 44306) and DTO Contract NBCHC06060 of USA. References Belkn, M., Matveeva, I., & Nyog, P. (2004). Regularzaton and sem-supervsed learnng on large graphs. COLT. Belkn, M., & Nyog, P. (2003). Laplacan egenmaps for dmensonalty reducton and data representaton. Neural Comput. Belkn, M., Nyog, P., & Sndhwan, V. (2005). On manfold regularzaton. AISTATS. Blondel, V. D., Gajardo, A., Heymans, M., Senellart, P., & Dooren, P. V. (2004). A measure of smlarty between graph vertces: Applcatons to synonym extracton and web searchng. SIAM Rev. Blum, A., & Mtchell, T. (998). Combnng labeled and unlabeled data wth co-tranng. COLT. Brefeld, U., Gartner, T., Scheffer, T., & Wrobel, S. (2006). Effcent co-regularsed least squares regresson. ICML. Chapelle, O., Vapnk, V., & Weston, J. (999). Transductve nference for estmatng values of functons. NIPS. Cortes, C., & Mohr, M. (2007). On transductve regresson. In Nps. Ham, Lee, & Saul (2005). Semsupervsed algnment of manfolds. AISTATS. Krshnapuram, B., DavdWllams, Xue, Y., Hartemnk, A., & Carn, L. (2005). On sem-supervsed classfcaton. NIPS. Rowes, S. T., & Saul, L. K. (2000). Nonlnear dmensonalty reducton by locally lnear embeddng. Scence. Schlkopf, B., Herbrch, R., & Smola, A. J. (200). A generalzed representer theorem. COLT/EuroCOLT. Tenenbaum, J. B., de Slva, V., & Langford, J. C. (2000). A global geometrc framework for nonlnear dmensonalty reducton. Scence. Wenberger, K. Q., Sha, F., & Saul, L. K. (2004). Learnng a kernel matrx for nonlnear dmensonalty reducton. ICML. Zhu, X., & Goldberg, A. B. (2005). Sem-supervsed regresson wth order preferences. Computer Scences TR.