Linear Ensembles of Word Embedding Models

Transcription

1 Linea Ensembles of Wod Embedding Models Avo Muomägi Univesity of Tatu Tatu, Estonia Kaiit Sits Univesity of Tatu Tatu, Estonia Sven Lau Univesity of Tatu Tatu, Estonia Abstact This pape exploes linea methods fo combining seveal wod embedding models into an ensemble. We constuct the combined models using an iteative method based on eithe odinay least squaes egession o the solution to the othogonal Pocustes poblem. We evaluate the poposed appoaches on Estonian a mophologically complex language, fo which the available copoa fo taining wod embeddings ae elatively small. We compae both combined models with each othe and with the input wod embedding models using synonym and analogy tests. The esults show that while using the odinay least squaes egession pefoms pooly in ou expeiments, using othogonal Pocustes to combine seveal wod embedding models into an ensemble model leads to 7-10% elative impovements ove the mean esult of the initial models in synonym tests and 19-47% in analogy tests. 1 Intoduction Wod embeddings dense low-dimensional vecto epesentations of wods have become vey popula in ecent yeas in the field of natual language pocessing (NLP). Vaious methods have been poposed to tain wod embeddings fom unannoted text copoa (Mikolov et al., 2013b; Pennington et al., 2014; Al-Rfou et al., 2013; Tuian et al., 2010; Levy and Goldbeg, 2014), most well-known of them being pehaps Wod2Vec (Mikolov et al., 2013b). Embedding leaning systems essentially tain a model fom a copus of text and the wod embeddings ae the model paametes. These systems contain a andomized component and so the tained models ae not diectly compaable, even when they have been tained on exactly the same data. This andom behaviou povides an oppotunity to combine seveal embedding models into an ensemble which, hopefully, esults in a bette set of wod embeddings. Although model ensembles have been often used in vaious NLP systems to impove the oveall accuacy, the idea of combining seveal wod embedding models into an ensemble has not been exploed befoe. The main contibution of this pape is to show that wod embeddings can benefit fom ensemble leaning, too. We study two methods fo combining wod embedding models into an ensemble. Both methods use a simple linea tansfomation. Fist of them is based on the standad odinay least squaes solution (OLS) fo linea egession, the second uses the solution to the othogonal Pocustes poblem (OPP) (Schönemann, 1966), which essentially also solves the OLS but adds the othogonality constaint that keeps the angles between vectos and thei distances unchanged. Thee ae seveal easons why using an ensemble of wod embedding models could be useful. Fist is the typical ensemble leaning agument the ensemble simply is bette because it enables to cancel out andom noise of individual models and einfoce the useful pattens expessed by seveal input models. Secondly, wod embedding systems equie a lot of taining data to lean eliable wod epesentations. While thee is a lot of textual data available fo English, thee ae many smalle languages fo which even obtaining enough plain unannotated text fo taining eliable embeddings is a poblem. Thus, an ensemble appoach that would enable to use the available data moe effectively would be beneficial. Accoding to ou knowledge, this is the fist wok that attempts to leveage the data by combining seveal wod embedding models into a new impoved model. Linea methods fo combin- 96 Poceedings of the 21st Nodic Confeence of Computational Linguistics, pages , Gothenbug, Sweden, May c 2017 Linköping Univesity Electonic Pess

2 ing two embedding models fo some task-specific pupose have been used peviously. Mikolov et al. (2013a) optimized the linea egession with stochastic gadient descent to lean linea tansfomations between the embeddings in two languages fo machine tanslation. Mogadala and Rettinge (2016) used OPP to tanslate embeddings between two languages to pefom coss-lingual document classification. Hamilton et al. (2016) aligned a seies of embedding models with OPP to detect changes in wod meanings ove time. The same poblem was addessed by Kulkani et al. (2015) who aligned the embedding models using piecewise linea egession based on a set of neaest neighboing wods fo each wod. Recently, Yin and Schütze (2016) expeimented with seveal methods to lean meta-embeddings by combining diffeent wod embedding sets. Ou wok diffes fom theis in two impotant aspects. Fist, in thei wok each initial model is tained with a diffeent wod embedding system and on a diffeent data set, while we popose to combine the models tained with the same system and on the same dataset, albeit using diffeent andom initialisation. Secondly, although the 1toN model poposed in (Yin and Schütze, 2016) is vey simila to the linea models studied in this pape, it doesn t involve the othogonality constaint included in the OPP method, which in ou expeiments, as shown late, poves to be cucial. We conduct expeiments on Estonian and constuct ensembles fom ten diffeent embedding models tained with Wod2Vec. We compae the initial and combined models in synonym and analogy tests and find that the ensemble embeddings combined with othogonal Pocustes method indeed pefom significantly bette in both tests, leading to a elative impovement of 7-10% ove the mean esult of the initial models in synonym tests and 19-47% in analogy tests. 2 Combining wod embedding models A wod embedding model is a matix W R V d, whee V is the numbe of wods in the model lexicon and d is the dimensionality of the vectos. Each ow in the matix W is the continuous epesentation of a wod in a vecto space. Given embedding models W 1,...,W we want to combine them into a taget model Y. We define a linea objective function that is the sum of linea egession optimization goals: J = i=1 Y W i P i 2, (1) whee P 1,...,P ae tansfomation matices that tanslate W 1,...,W, espectively, into the common vecto space containing Y. We use an iteative algoithm to find matices P 1,...,P and Y. Duing each iteation the algoithm pefoms two steps: 1. Solve linea egession poblems with espect to the cuent taget model Y, which esults in updated values fo matices P 1,...P ; 2. Update Y to be the mean of the tanslations of all models: Y = 1 i=1 W i P i. (2) This pocedue is continued until the change in the aveage nomalised esidual eo, computed as 1 i=0 Y W i P i V d, (3) will become smalle than a pedefined theshold value. We expeiment with two diffeent methods fo computing the tanslation matices P 1,...,P. The fist is based on the standad least squaes solution to the linea egession poblem, the second method is known as solution to the Othogonal Pocustes poblem (Schönemann, 1966). 2.1 Solution with the odinay least squaes (SOLS) The analytical solution fo a linea egession poblem Y = PW fo finding the tansfomation matix P, given the input data matix W and the esult Y is: P = (W T W) 1 W T Y (4) We can use this fomula to update all matices P i at each iteation. The poblem with this appoach is that because Y is also unknown and will be updated epeatedly in the second step of the iteative algoithm, the OLS might lead to solutions whee both W i P i and Y ae optimized towads 0 which is not a useful solution. In ode to counteact this effect we escale Y at the stat of each iteation. This is done by scaling the elements of Y so that the vaiance of each column of Y would be equal to 1. 97

3 SOLS SOPP Dim Eo # Ite Eo # Ite Table 1: Final eos and the numbe of iteations until convegence fo both SOLS and SOPP. The fist column shows the embedding size Solution to the Othogonal Pocustes poblem (SOPP) Othogonal Pocustes is a linea egession poblem of tansfoming the input matix W to the output matix Y using an othogonal tansfomation matix P (Schönemann, 1966). The othogonality constaint is specified as PP T = P T P = I The solution to the Othogonal Pocustes can be computed analytically using singula value decomposition (SVD). Fist compute: S = W T Y Then diagonalize using SVD: Finally compute: S T S = V D S V T SS T = UD S U T P = UV T This has to be done fo each P i duing each iteation. This appoach is vey simila to SOLS. The only diffeence is the additional othogonality constaint that gives a potential advantage to this method as in the tanslated wod embeddings W i P i the lengths of the vectos and the angles between the vectos ae peseved. Additionally, we no longe need to woy about the tivial solution whee P 1,...,P and Y all convege towads 0. 3 Expeiments We tested both methods on a numbe of Wod2Vec models (Mikolov et al., 2013b) tained on the Estonian Refeence Copus. 1 Estonian Refeence 1 Dim SOLS SOPP Mean W Avg Table 2: Aveage mean anks of the synonym test, smalle values ae bette. The best esult in each ow is in bold. All diffeences ae statistically significant: with p < fo all cases. Copus is the lagest text copus available fo Estonian. Its size is appoximately 240M wod tokens, which may seem like a lot but compaed to fo instance English Gigawod copus, which is often used to tain wod embeddings fo English wods and which contains moe than 4B wods, it is quite small. All models wee tained using a window size 10 and the skip-gam achitectue. We expeimented with models of 6 diffeent embedding sizes: 50, 100, 150, 200, 250 and 300. Fo each dimensionality we had 10 models available. The numbe of distinct wods in each model is Duing taining the iteative algoithm was un until the convegence theshold th = was eached. The numbe of iteations needed fo convegence fo both methods and fo models with diffeent embedding size ae given in Table 1. It can be seen that the convegence with SOPP took significantly fewe iteations than with SOLS. This diffeence is pobably due to two aspects: 1) SOPP has the additional othogonality constaint which educes the space of feasible solutions; 2) although SOLS uses the exact analytical solutions fo the least squaes poblem, the final solution fo Y does not move diectly to the diection pointed to by the analytical solutions due to the vaiance escaling. 4 Results We evaluate the goodness of the combined models using synonym and analogy tests. 4.1 Synonym anks One of the common ways to evaluate wod embeddings is to use elatedness datasets to measue the coelation between the human and model judge- 98

4 Figue 1: Histogam of the synonym anks of the 100 dimensional vectos. Dak left columns show the ank fequencies of the SOPP model, light ight columns pesent the ank fequencies of one of the initial models. ments (Schnabel et al., 2015). In those datasets, thee ae wod pais and each pai is human annotated with a elatedness scoe. The evaluation is then pefomed by coelating the cosine similaities between wod pais with the elatedness scoes. As thee ae no annotated elatedness datasets fo Estonian, we opted to use a synonym test instead. We ely on the assumption that the elatedness between a pai of synonyms is high and thus we expect the cosine similaity between the synonymous wods to be high as well. We obtained the synonyms fom the Estonian synonym dictionay. 2 We queied each wod in ou vocabulay and when the exact match fo this wod was found then we looked at the fist synonym offeed by the dictionay. If this synonym was pesent in ou vocabulay then the synonym pai was stoed. In this manne we obtained a total of 7579 synonym pais. We odeed those pais accoding to the fequency of the fist wod in the pai and chose the 1000 most fequent wods with thei synonyms fo the synonym test. Fo each fist wod in the synonym pai, we computed its cosine similaity with evey othe wod in the vocabulay, odeed those similaities in the descending ode and found the ank of the second wod of the synonym pai in this esulting list. Then we computed the mean ank ove all 1000 synonym pais. We pefomed these steps on both types of combined models Y SOLS and Y SOPP and also on all input models W i. Finally we also computed the mean of the mean anks of all 10 input models. The esults as shown in Table 2 eveal that the 2 The Institute of the Estonian Language, eki.ee/dict/sys/ synonym similaities tend to be anked lowe in the combined model obtained with SOLS when compaed to the input models. SOPP, on the othe hand, poduces a combined model whee the synonym similaities ae anked highe than in initial models. This means that the SOPP combined models pull the synonymous wods close togethe than they wee in the initial models. The diffeences in mean anks wee tested using paied Wilcoxon signed-ank test at 95% confidence level and the diffeences wee statistically significant with p-value being less than in all cases. In oveall, the SOPP anks ae on aveage 10% lowe than the mean anks of the initial models. The absolute impovement on aveage between SOPP and mean of W is Although we assumed that the automatically extacted synonym pais should be anked closely togethe, looking at the aveage mean anks in Table 2 eveals that it is not necessaily the case the aveage ank of the best-pefoming SOPP model is ove 31K. In ode to undestand those esults bette we looked at the ank histogam of the SOPP model and one of the initial models, shown on Figue 1. Although the fist bin coveing the ank ange fom 1 to 10 contains the most wods fo both models and the numbe of synonym pais falling to futhe ank bins deceases the cuve is not steep and close to 100 wods (87 in case of SOPP and 94 in case of the initial model) belong to the last bin counting anks highe than Looking at the fathest synonym pais evealed that one wod in these pais is typically polysemous and its sense in the synonym pai is a elatively aely used sense of this wod, while thee ae othe moe common senses of this wod with a 99

5 Dim SOLS SOPP Mean W Min W Max W SOLS SOPP Mean W Min W Max W Avg Table 3: Hit@1 and Hit@10 accuacies of the analogy test. SOLS and SOPP columns show the accuacies of the combined models. Mean W, Min W and Max W show the mean, minimum and maximum accuacies of the initial models W i, espectively. The best accuacy among the combined models and the mean of the initial models is given in bold. The last ow shows the aveage accuacies ove all embedding sizes. completely diffeent meaning. We give some examples of such synonym pais: kaks (two) - puudulik (insufficient): the sense of this pai is the insufficient gade in high school, while the most common sense of the wod kaks is the numbe two; ida (east) - ost (loan wod fom Geman also meaning east): the most common sense of the wod ost is puchase; ubla (ouble) - kull (bank note in slang): the most common sense of the wod kull is hawk. 4.2 Analogy tests Analogy tests ae anothe common intinsic method fo evaluating wod embeddings (Mikolov et al., 2013c). A famous and typical example of an analogy question is a man is to a king like a woman is to a?. The coect answe to this question is queen. Fo an analogy tuple a : b,x : y (a is to b as x is to y) the following is expected in an embedding space to hold: w b w a + w x w y, whee the vectos w ae wod embeddings. Fo the above example with man, king, woman and queen this would be computed as: w king w man + w woman w queen Given the vecto epesentations fo the thee wods in the analogy question w a, w b and w x the goal is to maximize (Mikolov et al., 2013b) cos(w y,w b w a + w x ) (5) ove all wods y in the vocabulay. We used an Estonian analogy data set with 259 wod quatets. Each quatet contains two pais of wods. The wod pais in the data set belong into thee diffeent goups whee the two pais contain eithe: a positive and a compaative adjective fom, e.g. pime : pimedam, jõukas : jõukam (in English dak : dake, wealthy : wealthie); the nominative singula and plual foms of a noun, e.g. vajadus : vajadused, võistlus : võistlused (in English need : needs, competition : competitions); The lemma and the 3d peson past fom of a veb, e.g. aitama : aitas, katsuma : katsus (in English help : helped, touch : touched). We evaluate the esults of the analogy test using pediction accuacy. A pediction is consideed coect if and only if the vecto w y that maximizes (5) epesents the wod expected by the test case. We call this accuacy Hit@1. Hit@1 can be quite a noisy measuement as thee could be seveal wod vectos in a vey close ange to each othe competing fo the highest ank. Theefoe, we also compute Hit@10, which consides the pediction coect if the wod expected by the test case is among the ten closest wods. As a common pactice, the question wods epesented by the vectos w a, w b and w x wee excluded fom the set of possible pedictions. The Hit@1 and Hit@10 esults in Table 3 show simila dynamics: combining models with SOPP is much bette than SOLS in all cases. The SOPP 100

6 Figue 2: Mean squaed distances descibing the scatteing of the tanslated wod embeddings aound the combined model embedding fo evey wod in the vocabulay. The wods in the hoizontal axis ae odeed by the fequency with most fequent wods plotted fist. Figue 3: Mean squaed distances descibing the scatteing of the tanslated wod embeddings aound the combined model embedding fo a andom sample of 1000 wods. The wods in the hoizontal axis ae odeed by the fequency with most fequent wods plotted fist. combined model is bette than the mean of the initial models in all six cases. Futhemoe, it is consistently above the maximum of the best initial models. The aveage accuacy of SOPP is bette than the aveage of the mean accuacies of initial models by 41%, elatively (7.7% in absolute) in tems of Hit@1 and 27% elatively (10.5% in absolute) in tems of Hit@10. 5 Analysis In ode to gain moe undestanding how the wods ae located in the combined model space in compaison to the initial models we pefomed two additional analyses. Fist, we computed the distibution of mean squaed eos of the wods to see how the tanslated embeddings scatte aound the wod embedding of the combined model. Secondly, we looked at how both of the methods affect the paiwise similaities of wods. 5.1 Distibution of mean squaed distances We computed the squaed Euclidean distance fo each wod in vocabulay between the combined model Y and all the input embedding models. The distance e i j fo a jth wod and the ith input model is: d i j = Y j T i j 2, whee T i = W i P i is the ith tanslated embedding model. Then we found the mean squaed distance fo the jth wod by calculating: d j = 1 d i j i=0 These distances ae plotted on Figue 2. The wods on the hoizontal axis ae odeed by thei fequency the most fequent wods coming fist. We show these esults fo models with 100 dimensions but the esults with othe embedding sizes wee simila. Notice that the distances fo less fequent wods ae similaly small fo both SOLS and SOPP methods. Howeve, the distibution of distances fo fequent wods is quite diffeent while the distances go up with both methods, the fequent wods ae much moe scatteed when using the SOPP appoach. Figue 3 shows the mean squaed distances of a andom sample of 1000 wods. These plots eveal anothe diffeence between the SOLS and SOPP methods. While fo SOPP, the distances tend to decease monotonically with the incease in wod fequency ank, with SOLS the distances fist incease and only then they stat to get smalle. Ou vocabulay also includes punctuation maks and function wods, which ae among the most fequent tokens and which occu in many diffeent contexts. Thus, the individual models have a lot of feedom to position them in the wod embedding space. The SOLS combined model is able to bing those wods moe close to each othe 101

7 Figue 4: Cosine similaities of 1000 andomly chosen wod pais odeed by thei similaity in the combined model Y. Red, blue and geen bands epesent the maximum, mean and minimum similaities in the initial models, espectively. in the aligned space, while SOPP has less feedom to do that because of the othogonality constaint. When looking at the wods with lagest distances unde SOPP in the 1000 wod andom sample then we see that the wod with the highest mean squaed distance efes to the pope name of a well-known Estonian politician who has been pobably mentioned often and in vaious contexts in the taining copus. Othe wods with a lage distance in this sample include fo instance a name of a month and a few quantifying modifies. 5.2 Wod pai similaities In this analysis we looked at how the cosine similaities between pais of wods change in the combined model compaed to thei similaities in the input embedding models. Fo that, we chose a total of 1000 wod pais andomly fom the vocabulay. Fo each pai we calculated the following values: cosine similaity unde the combined model; maximum and minimum cosine similaity in the initial models W i ; mean cosine similaity ove the initial models W i. The esults ae plotted in Figue 4. These esults ae obtained using the wod embeddings with size 100, using diffeent embedding sizes evealed the same pattens. In figues, the wod pais ae odeed on the hoizontal axis in the ascending ode of thei similaities in the combined model Y. The plots eveal that 1) the wods that ae simila in initial models W i ae even moe simila in the combined model Y ; and 2) distant wods in initial models become even moe distant in the combined model. Although these tends ae visible in cases of both SOLS and SOPP, this behaviou of the combined models to bing moe simila wods close togethe and place less simila wods fathe away is moe emphasized in the combined model obtained with SOLS. In Figue 4, the ed, geen and blue bands, epesenting the maximum, mean and minimum similaities of the initial models, espectively, ae wide on the SOLS plot. This indicates that SOPP peseves moe the oiginal ode of wod pais in tems of thei similaities. Howeve, some of this diffeence may be explained by the fact that SOPP has an oveall smalle effect on the similaity compaed to SOLS, which is due to the popety of SOPP to peseve the angles and distances between the vectos duing the tansfomation. 6 Discussion and futue wok Fom the two linea methods used to combine the models, SOPP was pefoming consistently bette in both synonym and analogy tests. Although, as shown in Figues 2 and 3, the wod embeddings of the aligned initial models wee moe closely clus- 102

8 teed aound the embeddings of the SOLS combined model, this seemingly bette fit is obtained at the cost of distoting the elations between the individual wod embeddings. Thus, we have povided evidence that adding the othogonality constaint to the linea tansfomation objective is impotant to etain the quality of the tanslated wod embeddings. This obsevation is elevant both in the context of poducing model ensembles as well as in othe contexts whee tanslating one embedding space to anothe could be elevant, such as when woking with semantic time seies o multilingual embeddings. In addition to combining seveal models tained on the same dataset with the same configuation as demonstated in this pape, thee ae othe possible use cases fo the model ensembles which could be exploed in futue wok. Fo instance, cuently all ou input models had the same dimensionality and the same embedding size was also used in the combined model. In futue it would be inteesting to expeiment with combining models with diffeent dimensionality, in this way maginalising out the embedding size hypepaamete. Ou expeiments showed that the SOPP appoach pefoms well in both synonym and analogy tests when combining the models tained on the elatively small Estonian copus. In futue we plan to conduct simila expeiments on moe languages that, simila to Estonian, have limited esouces fo taining eliable wod embeddings. Anothe idea would be to combine embeddings tained with diffeent models. As all wod embedding systems lean slightly diffeent embeddings, combining fo instance Wod2Vec (Mikolov et al., 2013b), Glove (Pennington et al., 2014) and dependency based vectos (Levy and Goldbeg, 2014) could lead to a model that combines the stengths of all the input models. Yin and Schütze (2016) demonstated that the combination of diffeent wod embeddings can be useful. Howeve, thei esults showed that the model combination is less beneficial when one of the input models (Glove vectos in thei example) is tained on a huge text copus. Thus, we pedict that the ensemble of wod embeddings constucted based on diffeent embedding models also has the most effect in the setting of limited taining esouces. Finally, it would be inteesting to exploe the domain adaptation appoach by combining fo instance the embeddings leaned fom the lage geneal domain with the embeddings tained on a smalle domain specific copus. This could be of inteest because thee ae many petained wod embedding sets available fo English that can be feely downloaded fom the intenet, while the copoa they wee tained on (English Gigawod, fo instance) ae not feely available. The model combination appoach would enable to adapt those embeddings to the domain data by making use of the petained models. 7 Conclusions Although model ensembles have been often used to impove the esults of vaious natual language pocessing tasks, the ensembles of wod embedding models have been aely studied so fa. Ou main contibution in this pape was to combine seveal wod embedding models tained on the same dataset via linea tansfomation into an ensemble and demonstate the usefulness of this appoach expeimentally. We expeimented with two linea methods to combine the input embedding models the odinay least squaes solution to the linea egession poblem and the othogonal Pocustes which adds an additional othogonality constaint to the least squaes objective function. Expeiments on synonym and analogy tests on Estonian showed that the combination with othogonal Pocustes was consistently bette than the odinay least squaes, meaning that peseving the distances and angles between vectos with the othogonality constaint is cucial fo model combination. Also, the othogonal Pocustes combined model pefomed bette than the aveage of the individual initial models in all synonym tests and analogy tests suggesting that combining seveal embedding models is a simple and useful appoach fo impoving the quality of the wod embeddings. Acknowledgments We thank Alexande Tkachenko fo poviding the petained input models and the analogy test questions. We also thank the anonymous eviewes fo thei helpful suggestions and comments. Refeences Rami Al-Rfou, Byan Peozzi, and Steven Skiena Polyglot: Distibuted wod epesentations fo multilingual NLP. In Poceedings of the Seven- 103

9 teenth Confeence on Computational Natual Language Leaning, pages William L. Hamilton, Jue Leskovec, and Dan Juafsky Diachonic wod embeddings eveal statistical laws of semantic change. In Poceedings of the 54th Annual Meeting of the Association fo Computational Linguistics (Volume 1: Long Papes), pages th Annual Meeting of the Association fo Computational Linguistics, pages Wenpeng Yin and Hinich Schütze Leaning Wod Meta-Embeddings. In Poceedings of the 54th Annual Meeting of the Association fo Computational Linguistics, pages Vivek Kulkani, Rami Al-Rfou, Byan Peozzi, and Steven Skiena Statistically significant detection of linguistic change. In Poceedings of the 24th Intenational Confeence on Wold Wide Web, pages Ome Levy and Yoav Goldbeg Dependencybased wod embeddings. In Poceedings of the 52nd Annual Meeting of the Association fo Computational Linguistics (Volume 2: Shot Papes), pages Tomas Mikolov, Quoc V. Le, and Ilya Sutskeve. 2013a. Exploiting similaities among languages fo machine tanslation. CoRR, abs/ Tomas Mikolov, Ilya Sutskeve, Kai Chen, Geg S Coado, and Jeff Dean. 2013b. Distibuted epesentations of wods and phases and thei compositionality. In Advances in Neual Infomation Pocessing Systems 26, pages Tomas Mikolov, Scott Wen-tau Yih, and Geoffey Zweig. 2013c. Linguistic egulaities in continuous space wod epesentations. In Poceedings of the 2013 Confeence of the Noth Ameican Chapte of the Association fo Computational Linguistics: Human Language Technologies, pages Aditya Mogadala and Achim Rettinge Bilingual wod embeddings fom paallel and nonpaallel copoa fo coss-language text classification. In Poceedings of the 2016 Confeence of the Noth Ameican Chapte of the Association fo Computational Linguistics: Human Language Technologies, pages Jeffey Pennington, Richad Soche, and Chistophe D. Manning Glove: Global vectos fo wod epesentation. In Poceedings of the 2014 Confeence on Empiical Methods in Natual Language Pocessing, pages Tobias Schnabel, Igo Labutov, David Mimno, and Thosten Joachims Evaluation methods fo unsupevised wod embeddings. In Poceedings of the 2015 Confeence on Empiical Methods in Natual Language Pocessing, pages Pete H. Schönemann A genealized solution of the othogonal pocustes poblem. Psychometika, 31(1):1 10. Joseph Tuian, Lev Ratinov, and Yoshua Bengio Wod epesentations: A simple and geneal method fo semi-supevised leaning. In Poceedings of the 104