Feature-Based Matrix Factorization

Size: px

Start display at page:

Download "Feature-Based Matrix Factorization"

Silvester Wade
5 years ago
Views:

1 Feature-Based Matrx Factorzaton arxv: v3 [cs.ai] 29 Dec 2011 Tanq Chen, Zhao Zheng, Quxa Lu, Wenan Zhang, Yong Yu Apex Data & Knowledge Management Lab Shangha Jao Tong Unversty 800 Dongchuan Road, Shangha Chna Proect page: wk/svdfeature (verson 1.1) Abstract Recommender system has been more and more popular and wdely used n many applcatons recently. The ncreasng nformaton avalable, not only n quanttes but also n types, leads to a bg challenge for recommender system that how to leverage these rch nformaton to get a better performance. Most tradtonal approaches try to desgn a specfc model for each scenaro, whch demands great efforts n developng and modfyng models. In ths techncal report, we descrbe our mplementaton of feature-based matrx factorzaton. Ths model s an abstract of many varants of matrx factorzaton models, and new types of nformaton can be utlzed by smply defnng new features, wthout modfyng any lnes of code. Usng the toolkt, we bult the best sngle model reported on track 1 of KDDCup Introducton Recommender systems that recommends tems based on users nterest has become more and more popular among many web stes. Collaboratve Flterng(CF) technques that behnd the recommender system have been developed for many years and keep to be a hot area n both academc and ndustry aspects. Currently CF problems face two knds of maor challenges: how to handle large-scale dataset and how to leverage the rch nformaton of data collected. Tradtonal approaches to solve these problems s to desgn specfc models for each problem,.e wrtng code for each model, whch 1

2 demands great efforts n engneerng. Matrx factorzaton(mf) technque s one of the most popular method of CF model, and extensve study has been made n dfferent varants of matrx factorzaton model, such as [3][4] and [5]. However, we fnd that the maorty of matrx factorzaton models share common patterns, whch motvates us to put them together nto one. We call ths model feature-based matrx factorzaton. Moreover, we wrte a toolkt for solvng the general feature-based matrx factorzaton problem, savng the efforts of engneerng for detaled knds of model. Usng the toolkt, we get the best sngle model on track 1 of KDDCup 11[2]. Ths artcle serves as a techncal report for our toolkt of featurebased matrx factorzaton 1. We try to elaborate three problems n ths report,.e, what the model s, how can we use such knd of model, and addtonal dscusson of ssues n engneerng and effcent computaton. 2 What s feature based MF In ths secton, we wll descrbe the model of feature based matrx factorzaton, startng from the example of lnear regresson, and then gong to the full defnton of our model. 2.1 Start from lnear regresson Let s start from the basc collaboratve flterng models. The very baselne of collaboratve flterng model may be the baselne models ust consderng the mean effect of user and tem. See the followng two models. ˆr u = µ + b u (1) ˆr u = µ + b u + b (2) Here µ s a constant ndcatng the global mean value of ratng. Equaton 1 descrbe a model consderng users mean effect whle Equaton 2 denotes tems mean effect. A more complex model consderng the neghborhood nformaton[3] s as follows ˆr u = µ + b + b u + R(u) 1 2 s (r u b u ) (3) R(u) Here R(u) s the set of tems user u rate, b u s a user average ratng pre-calculated. s means the smlarty parameter from to. s s a parameter that we tran from data nstead of drect calculaton usng 1 wk/svdfeature 2

3 memory based methods. Note b u s dfferent from b u snce t s precalculated. Ths s a neghborhood model that takes the neghborhood effect of tems nto consderaton. Assumng we want to mplement all three models, t seems to be wastng to wrte code for each of the model. If we compare those models, t s obvous that all the three models are specal cases of lnear regresson problem descrbed by Equaton 4 y = w x (4) Suppose we have n users, m tems, and h total number of possble s n equaton 3. We can defne the feature vector x = [x 0, x 1,, x n+m+h ] for user tem par < u, > as follows Indcator(u == k) k < n Indcator( == k n) n k < n + m x k = 0 k m + n, / R(u), s means w k R(u) 1 2 (r u b u ) k m + n, R(u), s means w k (5) The correspondng layout for weght w shown n equaton 6. Note that choce of pars s can be flexble. We can choose only possble neghbors nstead of enumeratng all the pars. w = [b u (0), b u (1),, b u (n), b (1), b (m) s ] (6) In other words, equaton 3 can be reformed as the followng form ˆr u = µ + b 1 + b u 1 + s [ R(u) 1 2 (ru b ] u ) R(u) (7) where b, b u, s corresponds to weght of lnear regresson, and the coeffcents on the rght of the weght are the nput features. In summary, under ths framework, the only thng that we need to do s to layout the parameters nto a feature vector. In our case, we arrange frst n features to b u then b and s, then transform the nput data nto the format of lnear regresson nput. Fnally we use a lnear regresson solver to work the problem out. 2.2 Feature based matrx factorzaton The prevous secton shows that some baselne CF algorthms are lnear regresson problem. In ths secton, we wll dscuss feature-based generalzaton for matrx factorzaton. A basc matrx factorzaton model s stated n Equaton 8: ˆr u = µ + b u + b + p T u q (8) 3

4 Queston U User Factor Mergng Merged User Factor User Features I Item Features Item Factor Mergng Merged Item Factor Global Features Answer r U,I User Feature Bas Item Feature Bas Global Feature Bas Fgure 1: Feature-based matrx factorzaton The bas terms have the same meanng as prevous secton. We also get two factor term p u and q. p u models the latent peference of user u. q models the latent property of tem. Inspred by the dea of prevous secton, we can get a drect generalzaton for matrx factorzaton verson. ˆr u = µ + w x + b u + b + p T u q (9) Equaton 9 adds a lnear regresson term to the tradtonal matrx factorzaton model. Ths allows us to add more bas nformaton, such as neghborhood nformaton and tme bas nformaton, etc. However, we may also need a more flexble factor part. For example, we may want a tme dependent user factor p u (t) or herarchcal dependent tem factor q (h). As we can fnd from prevous secton, a drect way to nclude such flexblty s to use features n factor as well. So we adust our feature based matrx factorzaton as follows T y = µ+ b (g) γ + b (u) α + b () β + p α q β (10) The nput conssts of three knds of features < α, β, γ >, we call α user feature, β tem feature and γ global feature. The frst part of Equaton 10. The name of these features explans ther meanngs. α descrbes the user aspects, β descrbes the tem aspects, whle γ descrbes some global bas effect. Fgure 1 shows the dea of the procedure. We can fnd basc matrx factorzaton s a specal case of Equaton 10. For predctng user tem par < u, >, defne { { 1 k = u 1 k = γ =, α k = 0 k u, β k = (11) 0 k We are not lmted to the smple matrx factorzaton. It enables us to ncorporate the neghborhood nformaton to γ, and tme de- 4

5 pendent user factor by modfyng α. Secton 3 wll present a detaled descrpton of ths. 2.3 Actve functon and loss functon There, you need to choose an actve functon f( ) to the output of the feature based matrx factorzaton. Smlarly, you can also try varous of loss functons for loss estmaton. The fnal verson of the model s ˆr = f(y) (12) Loss = L(ˆr, r) + regularzaton (13) Common choce of actve functons and loss are lsted as follows: dentty functon, L2 loss, orgnal matrx factorzaton. ˆr = f(y) = y (14) Loss = (r ˆr) 2 + regularzaton (15) sgmod functon, log lkelhood, logstc regresson verson of matrx factorzaton. 1 ˆr = f(y) = 1 + e y (16) Loss = r ln ˆr + (1 r) ln(1 ˆr) + regularzaton (17) dentty functon, smoothed hnge loss[7], maxmum margn matrx factorzaton[8][7]. Bnary classfcaton problem, r {0, 1} Loss = h ((2r 1)y) + regularzaton (18) 1 2 z z 0 1 h(z) = 2 (1 z)2 0 < z < 1 0 z 1 (19) 5

6 2.4 Model Learnng To update the model, we use the followng update rule p = p + η êα q β λ 1 p (20) q = q + η êβ p α λ 2 q (21) b (g) b (u) b () = b (g) = b (u) = b () ( ) + η êγ λ 3 b (g) ( ) + η êα λ 4 b (u) ( ) + η êβ λ 5 b () (22) (23) (24) Here ê = r ˆr the dfference between true rate and predcted rate. Ths rule s vald for both logstc lkelhood loss and L2 loss. For other loss, we shall modfy ê to be correspondng gradent. η s the learnng rate and the λs are regularzaton parameters that defnes the strength of regularzaton. 3 What nformaton can be ncluded In ths secton, we wll present some examples to llustrate the usage of our feature-based matrx factorzaton model. 3.1 Basc matrx factorzaton Basc matrx factorzaton model s defned by followng equaton y = µ + b u + b + p T u q (25) And the correspondng feature representaton s γ =, α k = { 1 k = u 0 k u, β k = { 1 k = 0 k (26) 3.2 Parwse rank model For the rankng model, we are nterested n the order of two tems, gven a user u. A parwse rankng model s descrbed as follows P (r u > r u ) = sgmod ( µ + b b + p T u (q q ) ) (27) 6

7 The correspondng features representaton are lke ths { 1 k = 1 k = u γ =, α k = 0 k u, β k = 1 k = 0 k, k (28) by usng sgmod and log-lkelhood as loss functon. Note that the feature representaton gves one extra b u whch s not desrable. We can removed t by gve hgh regularzaton to b u that penalze t to Temporal Informaton A model that nclude temporal nformaton[4] can be descrbed as follows y = µ + b u (t) + b (t) + b u + b + (p u + p u (t)) T q (29) We can nclude b (t) usng global feature, and b u (t), p u (t) usng user feature. For example, we can defne a tme nterpolaton model as follows ( ) y = µ + b + b s e t u e s + t s be u e s + p s e t u e s + t s T pe u q (30) e s Here e and s mean start and end of the tme of all the ratngs. A ratng that s rated later wll be affected more by p e and b e and earler ratngs wll be more affected by p s and b s. For ths model, we can defne γ =, α k = e t e s t s e s k = u k = u + n 0 otherwse, β k = { 1 k = 0 k (31) Note we frst arrange the p s n the frst n features then p e n next n features. 3.4 Neghborhood nformaton A model that nclude neghborhood nformaton[3] can be descrbed as below: y = µ + s [ R(u) 1 2 (ru b ] u ) + b u + b + p T u q (32) R(u) We only need to mplement neghborhood nformaton to global features as descrbed by Secton

8 3.5 Herarchcal nformaton In Yahoo! Musc Dataset[2], some tracks belongs to same artst. We can nclude such herarchcal nformaton by addng t to tem feature. The model s descrbed as follows y = µ + b u + b t + b a + p T u (q t + q a ) (33) Here t means track and a denotes correspondng artst. Ths model can be formalzed as feature-based matrx factorzaton by redefnng tem feature. 4 Effcent tranng for SVD++ Feature-based matrx factorzaton can naturally ncorporate mplct and explct nformaton. We can smply add these nformaton to user feature α. The model confguraton s shown as follows: y = bas + ξ p + α d T β q (34) Here we omt the detal of bas term. The mplct and explct feedback nformaton s gven by α d, where α s the feature vector of feedback nformaton, α = for mplct feedback, and 1 R(u) α = r u, b u for explct feedback. d s the parameter of mplct R(u) and explct feedback factor. We explctly state out the mplct and explct nformaton n Equaton 34. Although Equaton 34 shows that we can easly ncorporate mplct and explct nformaton nto the model, t s actually very costly to run the stochastc gradent tranng, snce the update cost s lnear to the sze of nonzero entres of α, and α can be very large f a user has rated many tems. Ths wll greatly slow down the tranng speed. We need to use an optmzed method to do tranng. To show the dea of the optmzed method, let s frst defne a derved user mplct and explct factor p m as follows: p m = α d (35) The update of d after one step s gven by the followng equaton d = ηêα β q (36) 8

9 The resulted dfference n p m s gven by p m = ηê α 2 β q (37) Gven a group of samples wth the same user, we need to do gradent descent on each of the tranng sample. The smplest way s to do the followng steps for each sample: (1) calculate p m to get predcton (2) update all d assocates wth mplct and explct feedback. Every tme p m has to be recalculated usng updated d n ths way. However, we can fnd that to get new p m, we don t need to update each d. Instead, we only need to update p m usng Equaton 37. What s more, we can fnd there s a relaton between p m and d as follows: d = α p m (38) k α2 k We shall emphasze that Equaton 38 s true even for multple updates, gven the condton that the user s same n all the samples. We shall menton that the above analyss doesn t consder the regularzaton term. If L2 regularzaton of d s used durng the update as follows: d = η êα β q λd (39) The correspondng changes n p m also looks very smlar p m = η ê α 2 β q λp m (40) However, the relaton n Equaton 38 no longer holds strctly. But we can stll use the relaton snce t approxmately holds when regularzaton term s small. Usng the results we obtaned, we can develop a fast algorthm for feature-based matrx factorzaton wth mplct and explct feedback nformaton. The algorthm s shown n Algorthm 1. We fnd that the basc dea s to group the data of the same user together, for the same user shares the same mplct and explct feedback nformaton. Algorthm 1 allows us to calculate mplct feedback factor only once for a user, greatly savng the computaton tme. 5 How large-scale data s handled Recommender system confronts the problem of large-scale data n practce. Ths s a must when dealng wth real problems. For ex- 9

10 Algorthm 1 Effcent Tranng for Implct and Explct Feedback for all user u do p m α d {calculatng mplct feedback} p old p m for all tranng samples of user u do update other parameters, usng p m to replace α d update p m drectly, do not update d. end for for all, α 0 do d d + end for end for α k α2 k (p m p old ) {add all the changes back to d} ample Yahoo! Musc Dataset[2] conssts of more than 200M ratngs. A toolkt that s robust to nput data sze s desrable for real applcatons. 5.1 Input data bufferng The nput tranng data s extremely large n real applcaton, we don t try to load all the tranng data nto memory. Instead, we buffer all the tranng data through bnary format nto the hard-dsk. We use stochastc gradent descend to tran our model, that s we only need to lnearly terate over the data f we shuffle our data before bufferng. Therefore, our soluton requres the nput feature to be prevously shuffled, then a bufferng program wll create a bnary buffer from the nput feature. The tranng procedure reads the data from hard-dsk and uses stochastc gradent descend to tran the model. Ths bufferng approach makes the memory cost nvarant to the nput data sze, and allows us to tran models over large-scale of nput data so long as the parameters ft nto memory. 5.2 Executon ppelne Although nput data bufferng can solve the problem of large-scale data, t stll suffers from the cost of readng the data from hard-dsk. To mnmze the cost of I/O, we use a pre-fetchng strategy. We create a ndependent thread to fetch the buffer data nto a memory queue, then the tranng program reads the data from memory queue and do tranng. The procedure s shown n Fgure 2 Ths ppelne style of executon removes the burden of I/O from the tranng thread. So long as I/O speed s smlar or faster to tran- 10

11 Data n Dsk FETCH Buffer INPUT Thread 1 n Memory Thread 2 Matrx Factorzaton Stochastc Gradent Descent Fgure 2: Executon ppelne ng speed, the cost of I/O s neglgble, and our our experence on KDDCup 11 proves the success of ths strategy. Wth nput bufferng and ppelne executon, we can tran a model wth test RMSE=22.16 for track1 n KDDCup 11 2 usng less than 2G of memory, wthout sgnfcantly ncreasng of tranng tme. 6 Related work and dscusson The most related work of feature based matrx factorzaton s Factorzaton Machne [6]. The reader can refer to lbfm 3 for a toolkt for factorzaton machne. Strctly speakng, our toolkt mplement a restrcted case of factorzaton machne and s more useful n some aspects. We can support global feature that doesn t need to be take nto factorzaton part, whch s mportant for bas features such as user day bas, neghborhood based features, etc. The dvde of features also gves hnts for model desgn. For global features, we shall consder what aspect may nfluence the overall ratng. For user and tem features, we shall consder how to descrbe the user preference and tem property better. Our model s also related to [1] and [9], the dfference s that n feature-based matrx factorzaton, the user/tem feature can assocate wth temporal nformaton and other context nformaton to better descrbe the preference or property n current context. Our current model also has shortcomngs. The model doesn t support multple dstnct factorzatons at present. For example, sometmes we may want to ntroduce user vs tme tensor factorzaton together wth user vs tem factorzaton. We wll try our best to overcome these drawbacks n the future works. References [1] Deepak Agarwal and Bee-Chung Chen. Regresson-based latent factor models. In Proceedngs of the 15th ACM SIGKDD nterna- 2 kddcup.yahoo.com

12 tonal conference on Knowledge dscovery and data mnng, KDD 09, pages 19 28, New York, NY, USA, ACM. [2] Gdeon Dror, Noam Koengsten, Yehuda Koren, and Markus Wemer. The Yahoo! Musc dataset and KDD-Cup 11. In KDD- Cup Workshop, [3] Yehuda Koren. Factorzaton meets the neghborhood: a multfaceted collaboratve flterng model. In Proceedng of the 14th ACM SIGKDD nternatonal conference on Knowledge dscovery and data mnng, KDD 08, pages , New York, NY, USA, ACM. [4] Yehuda Koren. Collaboratve flterng wth temporal dynamcs. In Proceedngs of the 15th ACM SIGKDD nternatonal conference on Knowledge dscovery and data mnng, KDD 09, pages , New York, NY, USA, ACM. [5] A. Paterek. Improvng regularzed sngular value decomposton for collaboratve flterng. In Proceedngs of KDD Cup and Workshop, volume 2007, [6] Steffen Rendle. Factorzaton machnes. In Proceedngs of the 10th IEEE Internatonal Conference on Data Mnng. IEEE Computer Socety, [7] Jasson D. M. Renne and Nathan Srebro. Fast maxmum margn matrx factorzaton for collaboratve predcton. In Proceedngs of the 22nd nternatonal conference on Machne learnng, ICML 05, pages , New York, NY, USA, ACM. [8] Nathan Srebro, Jason D. M. Renne, and Tomm S. Jaakola. Maxmum-Margn Matrx Factorzaton. In Advances n Neural Informaton Processng Systems 17, volume 17, pages , [9] Davd H. Stern, Ralf Herbrch, and Thore Graepel. Matchbox: large scale onlne bayesan recommendatons. In Proceedngs of the 18th nternatonal conference on World wde web, WWW 09, pages , New York, NY, USA, ACM. 12

Collaborative Filtering Ensemble for Ranking

Collaborative Filtering Ensemble for Ranking Collaboratve Flterng Ensemble for Rankng ABSTRACT Mchael Jahrer commo research & consultng 8580 Köflach, Austra mchael.ahrer@commo.at Ths paper provdes the soluton of the team commo on the Track2 dataset