CS224W Final Report. 1 Introduction. 3 Data Collection and Visualization. 2 Prior Work. Cyprien de Lichy, Renke Pan, Zheng Wu.

Transcription

1 CS224W Final Report Cyprien de Lichy, Renke Pan, Zheng W 1 Introdction December 9, 2015 Recommender systems are information filtering systems that provide sers with personalized sggestions for prodcts or items. They have become an important research area since the appearance of the first papers on collaborative filtering. Mch work has been done on developing new approaches to recommender systems over the last years becase of the growing need of applications to help sers deal with an increasing amont of information and provide them with personalized recommendations. Nowadays, recommender systems are sally based on the sers past behaviors and the similarity between sers and/or between items. Bt sch traditional recommendation systems sffer from several problems, one of them being data sparsity. Some sers may have very few recorded (or even no) interactions with the items, and ths it is hard to predict what they might like, which is called cold start. However, there is a noticeable social trend that people that are socially related share interests and tastes, and ths, the knowledge abot a person s social neighborhood strctre cold be sed to infer its preferences to some degree. Bt conventional recommendation systems don t leverage this network strctre to make predictions, ths depriving themselves of a rich sorce of information. The idea of this project wold be to take into consideration the relational strctre of the network to improve the recommendation power, for both ser network and item network. 2 Prior Work Recommender systems are sally classified into collaborative filtering methods and content-based methods. There are also hybrid approaches that combine several recommendation method. For instance in [5] the athors sed a combination of collaborative filtering and content filtering. Except the two approaches mentioned above, people are exploring more and more other information sch as social ties between people to help improve recommendation system s performance. Zheng et al. [1] were the first to introdce the method of social network collaborative filtering. Hill et al. [2] introdce a way to se the information of social network in collective inference withot heavy comptations. However, the comptational complexity for condcting sch inference techniqe on traditional social network is high. Ths, Hill et al. [2] sggests to only limit or interest to a small sbpoplation (those with high commercial potential).on the other hand, featres extracted from recommended contents network cold also make the system more robst. According to the stdy done by Teng et al. [6] to bild recommendation system for recipes, two featres can be extracted from recipes network. Sharma et al.[4] proposed a network-centric approach to recommendations and introdced PopCore, a platform that implements this approach on Facebook. They implemented 6 different algorithms, some of which are network-centric, to stdy and compare the effect of the recommendations and fond that the network-centric algorithms indeed perform better. Marlin et al.[7] showed that to improve the recommendation accracy of a recommender system, the algorithm shold not ignore the missing data mechanism and the MAR (Missing at Random) assmption is not valid, and we shold model the MNAR (Missing Not at Random) phenomenon. In practice we observe that sers tend not to give a rating to an item they didn t like than giving a bad rating. As a reslt the rating distribtion is more skewed toward good ratings. 3 Data Collection and Visalization We se Yelp Data for recommendation system implementation. The dataset contains the following information: 1. Users: Yelping since, votes, review conts, name, serid, friends, fans, average stars, type, compliments, elite. 2. Bsinesses: bsiness ID, fll address, hors, open, categories, city, review cont, name, neighborhood, longitde, state, stars, latitde, attribtes, type. 3. Reviews: votes, ser ID, review ID, stars, date, text, type, bsiness ID. 4. Tips :ser ID, text, bsiness ID, likes, date, type. 5. Check-ins : check-in info, type, bsiness ID. It has 1.6 Million reviews in total, covering 10 cities across United States, Germany, Canada and UK. 366k Users in the dataset form a large social network with 2.9 Million connections. We choose only bsinesses in Pittsbrgh in this case to speed p the calclation for the milestone. And only sers who have reviewed one of the bsinesses inclded will be considered. Ths the data set we se contains 2724 bsinesses, sers, ser connections and ser ratings. 3.1 User Network The degree distribtion of ser network in the whole data set. 1

2 other words, we consider an edge exist if the weight is above t. We se Frchterman Reingold lay ot method, with threshold 5. The degree distribtion of ser network in the Pittsbrgh data set. For User network, if we only display nodes with above 33 degrees, below is the network. We cold observe that there is no obvios clsterization. 4 Evalating Metric 3.2 Bsiness Network We define or crrent bsiness network as follows: Each bsiness is a node in the network. If same ser have been to both bsiness A and bsiness B, then we add an edge between bsiness A and bsiness B with weight 1. If bsiness A and bsiness B share k common sers, then they have an edge between them with weight k. To simplify the network, we limit the size of edges by ptting certain threshold t for the formation of edges. In Social network based top-k recommendation recommends to a ser a small nmber of items(k items) at a time. Most of the existing social network based recommender systems se the root mean sqare error (RMSE) as the optimization goal. Top-k recommendations are more commonly sed in real world problem. For these recommendation systems, the top-k hit ratio or recall is a more natral and sefl metric compared with root mean sqare error. The top-k hit ratio or recall is defined as below[8]: For each ser, we sort the items i according to descending order of the predicted rating ˆR i,. When we have same predicted rating to different items, we rank them randomly. An item is relevant to a ser in the test set if s/he likes it (e.g., the assigned rating in the test data is above a certain threshold). For example, with Yelp data, the rating vales range from 1 to 5 stars and we consider a 5 star rating as relevant(the ser definitely liked these items), while other rating vales and missing rating vales are considered not relevant. The top-k hit ratio or recall is the fraction of relevant items in the test set that are in the top-k of the ranking list, denoted by N(k, ), from among all relevant items, N(). 2

3 For each ser, the top-k hit ratio is given by: H(k, ) = N(k, ) N() With ser top-k hit rate, we can compte the overall top-k hit ratio/recall by: recall = Σ N(k, ) Σ N() A higher top-k hit ratio or recall indicates a better performance of or algorithm. We also compted the RMSE when rnning or recommendation algorithms: RMSE = 1 T (,i) T ( ˆRtest,i R,i test Where T is a test set of observed ratings, R,i test is the grond trth vale, is the corresponding predicted rating. ˆR test,i 5 Nearest Neighbor Methods In a Nearest Neighbor method, top-k recommendations makes prediction only based on items adored by a sbset of sers who have some connection (nder certain distance metric) with the target ser. This kind of connection cold be based on collaborative filtering or edges in the ser-ser network. Then we can integrate social information into Nearest Neighbor top-k recommendation system. 5.1 Collaborative Filtering Approach The Collaborative Filtering Approach ses AllRank to obtain the ser latent featres[8]. The distance is compted throgh the Pearson correlation coefficient. Then we gather items from The U nearest neighbors of the target ser. The voting for the candidate items is defined as follows: V ote,i = Σ v N Σ i sim(, v)δ i Iv where δ is the Kronecker delta; I v denotes the set of relevant items of ser v; and N is the set of U nearest neighbors of ser (as determined by the Pearson correlation). For ser, item i, we have V ote,i ; We give weights to neighbors according to their similarity sim(, v) with the target ser, which is defined as the Pearson correlation coefficient. w t (, v) is set to be 1/d v, where d v is the depth of ser v in the BFS tree rooted at ser. 5.3 Trst-Collaborative Filtering Approach Trst-Collaborative Filtering Approach is designed to hybrid ser latent featre space based collaborative filtering approach and social network based approach. In this approach, we first find U closest neighbors from the CF neighborhood, then find U closest neighbors from the trst neighborhood which are not in the first U neighbors. Then we gather information from the above neighbors: V ote,i = Σ (c) v N Σ i w(, v)δ i Iv where N (c) is the combined neighborhood and w(,v) is defined as following: { sim(, v) if v N w(, v) = w t (, v) if v N (t) 5.4 Reslt Or reslts show that trst information significantly improves top-k hit ratio when incorporated properly in Nearest Neighbor models. The top-k hit ratio for Trst-Collaborative Filtering Approach is the highest, followed by that of PreTrst Approach, which is significantly higher than that of Collaborative Filtering Approach. We have three parameters: 1. U, the nmber of sers considered in the neighborhood, both in the collaborative filtering approach and the network based approaches. 2. K, the pper limit of the nmber of items recommended to a ser. 3. R, the rating threshold sed to determine whether a recommendation is relevant. By changing these parameters, we got different top-k hit ratios. The next three figres show how the top-k hit ratio change with respect to different vales of U, K and R: 5.2 PreTrst Approach PreTrst approach finds neighbors by implementing BFS in the ser-ser network starting from ser. The voting is as follow: V ote,i = Σ (t) v N Σ i w t (, v)δ i Iv where N (t) is the set of trsted sers of, and w t (, v) is the voting weight of ser v. The vale of 3

4 U, the larger the U, the longer it takes to finish the comptation. For K, we cannot recommend say 1000 restarants simltaneosly to a ser. A reasonable K wold be somewhere below 50. For R, 5 seems to be the best not only becase it is the largest possible vale, bt also becase it makes the most sense in practice. Giving a 5 star review sggests the ser absoltely loved the restarant. 5.5 Commnity Detection We hypothesized that adding in commnity detection to pre trst approach is going to improve the hit rate. The assmption is that sers within the same commnity share the same taste. Ths giving recommendation based on sers from the same commnity will likely increase the hit rate/recall. We first implemented a version of PreTrst that restricts neighbors to only those that are within the same commnity, detected by Claset-Newman-Moore commnity detection method. We can see that recall increases as U increases, fixing K and R. The recall for collaborative filtering approach reaches a platea after certain threshold. This is probably becase there is not enogh sers that rated the same restarants and ths we cannot get a neighborhood of size U after certain large enogh U. On the other hand, the recall for the two network-based approaches keep increasing as U increases. This is probably becase the largest connected component is pretty large (it trns ot 46% percent of all sers are in the largest connected component) so we are always able to find a large enogh neighborhood in the network. From the period before the collaborative filtering approach reaches the platea, we can conclde that introdcing network featres significantly improves the recommendation. We can see that recall increases as K increases, fixing U and R. This is expected becase we are increasing the nominator of recall while keeping denominator fixed. We can see that recall increases as R increases, fixing U and K. This is expected becase we are increasing the nominator of recall slightly bt the denominator increased drastically if we decrease R. So overall we see the recall increases with U, K, R. Larger U, K, R always lead to better recall. However, the choice of U, K, R cannot be arbitrarily large. For As can be seen from the above graph, it trned ot that the hit rate actally decreased, which is qite nexpected. Then we dig deep into this and brainstormed what cold be casing this. We think the reason is that not every one of a ser s nearest neighbor falls within the same commnity with the ser. Ths these sers, which actally has the best predictive power, are exclded when making the recommendation. As an attempt to jstify this hypothesis and also improve the recommendation, we introdced a heristic: all the sers who are in the same commnity as the ser of interest are closer to this ser in the network. So once we skipped throgh nmber of sers in the same commnity, we no longer enforce same commnity rle when determining if a ser shold be considered, i.e. always consider them. This heristic in fact increased the recall. Or previos hypothesis is ths jstified. However, the improvement in the recall is not enogh. It still falls mch lower than the original approach. Frther more, both crves with commnity featres reach a platea. This is becase they both skipped some close sers in the network bt are not in the same commnity and these sers have the most predictive power. Inclding more sers frther ot doesn t really help the prediction mch becase the similarity between sers is mch lower. 4

5 6 Matrix Factorization The Matrix Factorization approach was fond to be the most accrate approach to redce the dimensionality of the problem when a high level of data sparsity is observed. 6.1 Basic approach In this model, the rating matrix R of dimensions 0 i 0 ( 0 being the nmber of sers and i 0 the nmber of items) is represented as the prodct of 2 low-rank matrices P and Q. The matrices P and Q are latent-featres representation of items and sers respectively, and they are said to be low-rank: rank(p ) = rank(q) = d, with d 0 and d i 0. The MF model is a kind of dimension redction, the objective is to find the matrix ˆR = QP T that best fits the original matrix R (which is sparse becase we have only a few items rated by each ser). Q is of dimensions 0 d and P is of dimensions i 0 d. To take into accont the phenomenon of MNAR (Missing Not at Random) of the ratings, an offset rating vale r m is added in the model in the following form: ˆR = r m + QP T One way to proceed to maximize the recall on the top-k recommendations wold be to optimize directly on the top-k hit ratio bt this is too costly comptationally. Instead, we can optimize a modified sqared error criterion that will lead to a better top-k hit rate than the classical sqared error: min P,Q i ( W,i R,i ˆR (,i + λ P 2 F + Q 2 ) F (1) Where R,i = R,i if the rating is observed and R,i = r m otherwise, r m is the impted rating when there is a missing vale, it is a tning parameter of the model, and according to the stdy by Yahoo! (See [7]) we expect its optimal vale to be rather small. The matrix W is introdced to balance the dataset (which is very sparse in reality) by giving more weight to the observed ratings. In addition to that we also have the typical L2 reglarization term to avoid over-fitting. 6.2 Inclding the social network Now, we can inclde the networks (ser-ser and bsiness-bsiness networks) information by inclding more terms i the following way. We introdce two matrices U and B representing the relationship between two sers and two bsinesses respectively, U and B cold be weighted adjacency matrix, it is proposed in [8]: deg(v) U,v = A,v deg(v) + deg() where A is the ser-ser adjacency matrix. The objective fnction to minimize becomes the following: min P,Q,V ( W,i R,i ˆR,i i + γ v W U,v ( U,v Û,v + λ ( P 2 F + Q 2 F + V 2 ) F Where Û = QV T, V is of size 0 d. (2) 6.3 Inclding a bsiness-bsiness network In or approach we imagined doing the same for bsinesses, by constrcting a bsiness-bsiness network sing the data abot the bsinesses (like descriptions, reviews) and the sers. Basically, we wold bild a network by linking two bsinesses if they have mtal cstomers and similar descriptions. In that way we se more available data than jst the ratings history to drive the CF comptations for the recommender system. The objective fnction to minimize wold be the following: min P,Q,V,C ( W,i R,i ˆR,i i + γ ( W,v U U,v Û,v v + β Wi,j (B B i,j ˆB i,j i j + λ ( P 2 F + Q 2 F + V 2 F + C 2 ) F Where ˆB = P C T, and C is of size i 0 d. 6.4 Social inflence weighting (3) By varying the weights W,i of the observed ratings according to the eigenvector centrality measre of ser, we are giving more weights to the important/inflential sers, this forces the algorithm to better fit the matrix factorization to these sers and these better reslts may diffse in the network (via the social part of the objective fnction) and yield better overall reslts. This comes from the intition that better fitting the more inflential sers is more sefl for the overall performance. 6.5 Optimization algorithm for the MF models We implemented all of these MF models sing an alternating optimization algorithm: Alternating Least Sqares, which is an efficient algorithm for this task. At each step of the algorithm 3 of the nknown matrices (P, Q, V and C) are fixed and the other one can be pdated to maximize the objective fnction via an exact formla. 5

6 6.6 Reslts Model RMSE MAR (r m, w m = 0) 1.61 MNAR (r m, w m = 2, ) 1.11 MNAR Social Network 1.02 MNAR Social & bsiness Network 1.01 From this table we can see that by taking into accont the phenomenon of MNAR we can greatly improve the RMSE at the expense of finding the optimal vales of 2 tning parameters (r m and w m ). We can also see that sing information abot the strctre of the social network (the adjacency matrix and the eigenvector centrality measre) can also improve the RMSE. However, we didn t get a noticeable improvement by considering or bsiness-bsiness network maybe becase or basic constrction of the network does not bring new information, the convergence of or algorithm is also longer. 7 Temporal Effects When we first proposed the project, we were thinking abot incorporating the temporal effects of the ratings and the network. However, when we actally want to go ahead and do it, we fond ot there is no timestamp of the friendship formation. So the network is a static snapshot, not evolving over time. However, we do have the review timestamp. The sage of the review timestamp is limited thogh, besides splitting the data into train and test set, predicting ftre reviews from past reviews, there is not an intitive way of tilizing this information. Ths stdying the temporal effects is not well sited for this data set. 8 Reprodcibility/Robstness We focsed or analysis with Pittsbrgh becase Pittsbrgh s data is diverse enogh and complete yet small enogh for fast prototyping. One concern is that the overall network strctre is different than the one in Pittsbrgh. This is a totally valid concern. However, the data consists of restarants from the following cities: U.K.: Edinbrgh Germany: Karlsrhe Canada: Montreal and Waterloo U.S.: Pittsbrgh, Charlotte, Urbana-Champaign, Phoenix, Las Vegas, Madison First of all, this is not the complete graph of the overall yelp network anyway. It s only a small portion of the global graph strctre. Frther redcing the graph to one city isn t any fndamentally different. Secondly, the cities are in disjoint regions of the contry or even different contries. we wold expect the graphs of each city to be a relative independent connected component and there is mch less inter-city connections. Therefore, stdying each individal city by itself is a valid approach. In fact, to make the most relevant prediction, I sspect Yelp wold se local information heavily as well. Therefore, it doesn t really make sense to rn the analysis with the whole data set. If we did that, We wold spent way too mch comptation time on an improperly defined network. To make sre or analysis is robst and can be reprodced. We did the exact same analysis on Phoenix data, which is diverse and complete yet sbstantially larger than Pittsbrgh. All the trends, relationships, orderings are retained with Phoenix data. To save space, we did not inclde the plots here. We have ths gained confidence that or approach can be reprodced in an independent data set and is robst. 9 Difficlties 1. The existing recommendation packages do not satisfy or need (mostly becase the flexibility of the algorithms is not satisfying) and or data is not ready to se for recommendation, so we had to implement all the algorithms and data processing from scratch. 2. The size of the data poses problems for details of implementing the algorithms. For example, we rn ot of memory when doing matrix operations and had to se sparse matrices and resort to comptational tricks. 3. The dataset is hge and the rnning time for the algorithms is rather long. 10 Ftre Works 1. Frther optimize the model, minor tweak the algorithms to have the best performance. 2. Have another data set, different from Yelp, which has friendship formation timestamps, to stdy temporal effects. 11 Acknowledgment We wold like to thank Lada and the TAs for providing a wonderfl qarter of learning experience and insightfl feedback on the project. References [1] R. Zheng, F. Provost and A. Ghose. Social Network Collaborative Filtering: Preliminary Reslts. Proceedings of the Sixth Workshop on ebsiness (WEB2007), New York, USA, December 2007 [2] S. Hill, F. Provost and C. Volinsky. Learning and Inference in Massive Social Networks. 5th International Workshop on Mining and Learning with Graphs, Agst

7 [3] G. Adomavicis and A. Tzhilin, Toward the Next Generation of Recommender Systems: A Srvey of the State-of-the-Art and Possible Extensions, IEEE Trans. Knowl. Data Eng. 17(6): , 2005 [4] A. Sharma and D. Cosley, Network-Centric Recommendation: Personalization with and in Social Networks, IEEE International Conference on Privacy, Secrity, Risk, and Trst, 2011 [5] A. B. Barragns-Martnez, E. Costa-Montenegro, J.C. Brgillo, M. Rey- Lpez, F. A. Mikic-Fonte, and A. Peleteiro. A hybrid content-based and item-based collaborative filtering approach to recommend TV programs enhanced with singlar vale decomposition. Information Sciences, 180(22), [6] C.-Y. Teng, Y.-R. Lin, and L. A. Adamic. Recipe recommendation sing ingredient networks. Proceedings of the 3rd Annal ACM Web Science Conference, pages , [7] Marlin, B.; Zemel, R.; Roweis, S; Slaney, M., Collaborative Filtering and the Missing at Random Assmption [8] Yang, X.; Steck, H.; Go, Y; Li, Y., On Top-k Recommendation sing Social Networks 7