Bayesian Personalized Ranking for Las Vegas Restaurant Recommendation

Bayesian Personalized Ranking for Las Vegas Restaurant Recommendation Kiran Kannar A53089098 kkannar@eng.ucsd.edu Saicharan Duppati A53221873 sduppati@eng.ucsd.edu Akanksha Grover A53205632 a2grover@eng.ucsd.edu Abstract Item recommendation is a challenging task of predicting a personalized ranking for a set of items for each user. In this project, we build a collaborative filtering model BPR-MF based on a Bayesian analysis of the problem, to generate a total order of the personalized ranking for all restaurants, which can be used to recommend the next top restaurant for each user. The learning method is based on stochastic gradient descent with bootstrap sampling. We compare our primary model with other collaborative filtering methods like memory-based collaborative filtering and matrix factorization with SVD. We see that BPR-MF is indeed a state-of-art method for personalized recommendation, outperforming other models based on the AUC evaluation criterion. 1 Introduction An important aspect of item recommendation is making the recommendation on a set of items personalized i.e. specific to each user, based on his historical data. For example, Amazon may recommend specific movies based on what genre of movies one has watched before. This kind of personalized recommendation usually involves some kind of ranking of items specific to each user. Better, relevant recommendation increases engagement of users with the recommender system. Collaborative filtering is a commonly used recommendation technique; it involves making predictions per user or item based on collective preferences across all users or items. The idea is to exploit the similarity between users or items, or even user-item compatibility to make recommendations In this project, we aim at recommending the next best restaurant in Las Vegas to a user based on the restaurants he/she has already visited. A standard matrix factorization model would then involve building the latent factor representations for both users and restaurants based on review history of the restaurants in Las Vegas and that of the users. Our dataset is derived from the huge corpus provided by Yelp as a part of the Round 9 of their Dataset challenge [1]. We build a recommender system that uses a Bayesian analysis of the problem to derive its optimization criterion, and therefore termed Bayesian Personalized Ranking (BPR). This model is based on implicit feedback i.e. no specific feature (like rating) is required for ranking. The algorithm considers all of the observed implicit data as positive feedback and tries to differentiate it with large set of remaining items which is either negative feedback or missing values that will be obtained in future. In our case, all the restaurants the user visits fall under positive feedback and the restaurants he/she doesn t go to are a part of the other larger set. The algorithm produces a pairwise ordering for all user-restaurant pairs. We can use these pairwise ordering to find a total ordering of the ranking of restaurants for each user, which serves the purpose of personalizaed recommendation. In this report, we first discuss our findings from dataset exploratory analysis. We choose to provide personalized recommendations for restaurants in Las Vegas. The predictive task is described further along with the AUC evaluation criteria, which will be our method of evaluation of recommendations

across the models we implemented. We then compare this model with various models for personalized ranking, which involve the use of explicit feedback, in terms of ratings. We report the results of our experiments which indicate that the BPR with Matrix Factorization and regularization (BPR-MF-reg) is a superior algorithm for the task of personalized ranking as compared to all other models. 2 Related Work Matrix Factorization (MF) has become very popular in recommender systems, and it has been widely used in systems which accept implicit or explicit feedback. It finds the latent factor representations for users and restaurants which can be used in the service of the task.[3]. This project s main model is based on the generic learning algorithm proposed by Rendle et.al in [1], which uses a generic optimization criterion called BPR_Opt for personalized ranking. BPR measures the difference between the personalized rankings of the restaurants a user has visited and the rest of the restaurants. From our experimentation, we have seen similar results to what the paper authors observed with their datasets. The authors used two datasets, Rossman dataset having buying history of 10,000 users on 400 items, and the Netflix DVD rental dataset with 10,000 users and 5000 items. Our Las Vegas dataset has 11,264 users and 5431 restaurants. The subsamples specific to Las Vegas restaurants have been derived from the Yelp s dataset available as a part of Round 9 of their Dataset Challenge [1]. The dataset is bigger with more user ratings and more restaurants, in comparison to the dataset available in previous rounds. It was not possible to find any recent work on the round 9 dataset, as the challenge is expected to end in June 2017. However, in terms of related work, the Yelp datasets in general have been experimented with several learning model techniques from SVMs to collaborative filtering, especially for rating prediction. The combination of BPR and MF has also been used in recent works. In [4], He et.al. incorporate visual signals into BPR-MF for considering the visual appearance of the items for recommendation. In [5], Weike Pan and Li Chen extend the BPR-MF algorithm for incorporating group preferences. Additionally, there are other collaborative filtering models like the weighted Regularized Matrix factorization model (WR-MF) by Hu et.al. [3] and Pan et.al [7] which add weights to the error function to increase the impact of positive feedback. They also use regularization to control overfitting. However, we limit to using individual preferences and then comparing with other models, which include standard collaborative filtering techniques like matrix factorization and memory-based collaborative filtering using cosine similarity. 3 Dataset exploration The Yelp dataset from Yelp s website has data across 144,072 businesses of which around 48,485 are restaurants. We want to predict the next best restaurant each user would want to visit, by ranking the restaurants the user has not been to. The dataset available is distributed across different files each for user reviews, check-in s, user tip and files with data about user profiles and restaurant profiles. For the purpose of this project, we have only retained two files that pertain to user reviews and restaurant data. From all Yelp businesses, we first filtered out the restaurants and plotted the locations of them on Google Maps using gmplot in Figure 1. This showed us that the data we have is from approximately 10 cities across US (including Las Vegas, Phoenix, Pennsylvania etc.), Canada and Europe. For all the restaurants, we have a total of 2,577,298 reviews given by 7,21,779 users. Figure 1: Heat Map of the Restaurants in Yelp DataSet (in and around US) 2

As a next step, we built dictionary data structures to store all the restaurants that each user u reviewed (I u ) and all the users that reviewed a given restaurant (U i ). We plotted the lengths of items of these dictionaries, as a histogram. These histograms show the distribution of the number of the ratings across users 2 and the number of ratings received across restaurants 3. We included only the users and restaurants with at least 10 reviews. For better visibility, the plots have 99% of data. Figure 2: Number of Yelp Restaurants over number of ratings it received Figure 3: Number of Yelp users over number of ratings they have given Then, we calculated the average ratings user-wise and restaurant-wise., i.e the average rating a user tends to give across restaurants he/she reviewed (in figure 5) and the average rating a restaurant received across all users that reviewed it (in figure 4). Figure 4: Average Restaurant Ratings Figure 5: Average Users Ratings Seeing the immensity of the number of restaurants and reviews we had from the data exploration, we decided to focus and build a model for a specific city like Las Vegas. Apart from the scale of managing a model with extremely high number of user-item pairs leading us to make this decision, we found it fun and exciting to predict restaurants in Las Vegas! What happened in Las Vegas, we know it. What will happen next, we shall know it too! Thereafter, we proceeded to obtain a geographical view of all Vegas Restaurants in our data set. Figure 6is the rating distribution of the restaurants in Vegas. We also plotted the histograms for number of ratings given by users and obtained by restaurants specific to Vegas, and also the histograms for Average user-wise and restaurant-wise ratings. The plots have users/restaurants with at least 10 reviews and for visibility, we plotted 99% of data. Below are the plots in figures 7, 8, 9 and 10. 3

Figure 6: Map of Las Vegas Restaurants by Rankings Figure 7: Number of Yelp Vegas Restaurants over number of ratings it received Figure 8: Number of Yelp User over number of ratings given Figure 9: Average Vegas Restaurant Ratings Figure 10: Average Vegas Users Ratings 4

3.1 Key observations From the plots we observe the following points: Most restaurants have 10-20 review ratings and very few have up to 2000 reviews. Most users reviewed 10-15 restaurants and very few reviewed up to 200 restaurants Most restaurants received an average rating between 3.5 and 4 from all users who reviewed them and very few received 1 or 5 average rating. Most users gave an average rating between 3.5 and 4 across all restaurants they reviewed. Very few had average ratings of 1 and 5. After this, we used our Vegas data set to predict rankings of Vegas restaurants. Below are few statistics of our data: Table 1: Statistics of our Vegas Data Set Statistic Value Number of Restaurants 5431 Number of Users 11,264 Total Number of data points 61,174,784 Total Number of Ratings 7,61,678 For all the models described in this paper, we assume that if the user has reviewed a restaurant, it implies that the user has visited the restaurant. We also only considered those users which have reviewed/visited at least 10 restaurants. Since our BPR model uses implicit feedback, we didn t use any features but geographical distributions and setting thresholds for including users and restaurants really helped us to come to a set of data points which we had the computational power to work and predict on. For all other models that we built as baselines, described in this paper, we used ratings as the explicit feedback. 4 Predictive task and Model Evaluation criteria As defined previously, our prediction task is to determine the next best restaurant for each user or a ranking of restaurants for each user. Unlike regular prediction tasks, our pairwise algorithm will try to separate out the known data (which is already available) and the remaining unobserved data. In the case of explicit feedback, the task is relatively easier since we already have both the positive and negative data. In our case of implicit feedback, we will consider the observed data(restaurant s the user visited) as positive feedback, while the unobserved data(restaurant s the user didn t visit) could be either negative feedback or the data that will be available in future. We provide a recommendation by building a total order that ranks all the items for each user. This can be formed by considering pairwise ordering of items for each user. We borrow notations from [2] as we describe the total order (i.e ranking). Let U be the set of users and I be the set of all restaurants. Each user has been to a subset of these restaurants, and therefore the observed data S U I. The personalized total ranking > u I 2 of all items should satisfy the properties of anti-symmetry, totality and transitivity so that a total ranking can be formed from individual pair orders. We define the two sets I u = {i I : (u, i) S} and U i = {u U : (u, i) S} to be the sets of restaurants reviewed by each user, and the sets of users who reviewed each restaurant respectively. Since our model looks at both the observed data and unobserved data, we can create the training data D s = {(u, i, j) i I u j I \ I u } which is the set of triples for each user conjoined with a restaurant he has reviewed and a restaurant he has not. By training on such users, we can make the model learn that the user prefers restaurant i over j, thereby incorporating the anti-symmetry. The actual training data is a subset of Ds, as we use the leave one out evaluation scheme for testing. For every user, we remove one (u, i) and keep in S test. The remaining observed data becomes S train. Our evaluation criteria uses AUC (area under the curve) metric over the test set S test. 5

The AUC is a measure of ranking quality. It specifies the probability that the predicted pairwise ranking is correct when we draw out two items at random. It can also be defined as the expectation that a uniformly draw positive sample is ranked before a uniformly drawn negative sample. The average AUC statistic can be calculated as: AUC = 1 1 U E(u) where the evaluation pairs per user u are: u (i,j) E(u) δ(x ui > x uj ) E(u) = {(i, j) (u, i) S test (u, j) / (S test S train )} x ui and x uj are the values predicted by the standard collaborative filtering models like matrix factorization or memory-based collaborative filtering. δ function counts the pass of the evaluation criterion within it as 1, else it has the value 0. Our main model is Bayesian Personalized Ranking model(bpr-mf) and we used a number of baseline models like -Most popular, Memory-based collaborative filtering (using cosine similarity), Matrix Factorization, which are explained in detail in the next section. We chose the BPR-MF model because we believe that the type of task we are optimizing will give the best predictions if we create a personalized ranking for each user that is based on a pair of items. BPR-MF is the state-of-the-art model available for this. The models are discussed in detail below, and the ensuing section discusses the results 5 Models For every model we experimented with, we explain in detail any pre-processing we performed specific to each model. Else, we used the standard leave one-out evaluation method to construct S train and S test for training and AUC evaluation. The strengths and weaknesses of each model are discussed in the results section. 5.1 Baseline Model 1 - Most Popular (MP) This is the simplest baseline that is user-independent. For each restaurant, it assigns a common value across all users. We have chosen the most-popular value, which simply is the number of users who have reviewed/visited the restaurant. x most pop ui = U i x ui and x uj are calculated for all pairs of (i,j) for different users and the AUC is evaluated. This model is computationally simple, but also extremely naive in its assumptions. 5.2 Baseline Model 2 - Memory-based Collaborative Filtering(Using Cosine Similarity) This approach uses the entire training data, and not a subsample as we will see in BPR. In this approach, we want to extract the similarities between restaurants and predict the ratings for each user according to that. We first construct the matrix M of size R U where R and U are the set of the restaurants and number of users respectively. Next, we initialize each value of M with the explicit feedback ie., if user u reviewed the restaurant r then M[u, r] = rating(u, r) else we equate it to zero. Let M R be the column corresponding to restaurant R obtained by removing the mean across each dimension for normalization. We then construct the similarity matrix S R where S R i,j = (M R i )T (M R j ) M R i M R j To make a prediction, we need to calculate x uij which can be calculated in terms of r u,i and r u,j) For evaluation, we say the user u prefers restaurant i to restaurant j if r u,i > r u,j. The rating can be 6

calculated as: r u,k = M[u, z] M[u,z] 0 SR kz M[u,z] 0 SR kz 5.3 Baseline Model 3 - Matrix Factorization In the matrix factorization model, we predicted user rankings by learning the latent factors γ u, γ r for users and restaurants respectively from ratings, which is an explicit feedback. The loss function we minimize in this model is L = (rating(u, r) γ u.γ r ) 2 + λ u,r u γ u 2 + λ r γ r 2 Where the sum is taken over all (u,r) if the user u has reviewed restaurant r Differentiating above equation for loss L w.r.t to γ u and γ r and equating them to 0 gives the below closed form solutions γ u = γ r = r γ r rating(u, r) λ + r γ r 2 u γ u rating(u, r) λ + u γ u 2 By iterative updating the γ u and γ r by above equations for a few iterations ( 20-50) will give a stable loss value L In the evaluation of AUC and for further analysis, we say the user u prefers restaurant i to restaurant j if γ u.γ i > γ u.γ j We repeated the iterations for different values of K(rank of γ u and γ r ) and calculated AUC for all values. 5.4 Bayesian Personalized Ranking model The BPR model is a pairwise ranking framework which uses Stochastic Gradient Descent (SGD) for training. Following the notation from [2], it derives the BPR optimization criterion by using the maximum likelihood estimate for P (i > u j Θ). This criterion (BPR-OPT) is as follows: (u,i,j) D s ln σ(x uij ) λ Θ Θ 2 where x uij is value captured by our pairwise learning algorithm based on the parameter Θ. x uij = x ui x uj and the parameter Θ is the latent representation of each user and restaurant, and λ Θ is the regularization parameter. We have chosen the same regularization parameter for all users and restaurants. BPR-MF learns the model parameter Θ using Stochastic Gradient Descent (SGD). It is infeasible to use batch gradient descent since we have a extremely huge set of triples to consider. Therefore, in each iteration, we randomly sample a (u, i) S train and find a j from the restaurants the user has not visited, to construct the triple (u, i, j). The learning rule in BPR-MF is: ( Θ := Θ + η σ( x uij ). x uij Θ ) + λ ΘΘ The hyperparameters that we choose were λ Θ = 0.001 and η = 10 4. We tried out a number of different values for each of these parameters and choose the above best values. In the evaluation of AUC and for further analysis, we say the user u prefers restaurant i to restaurant j if x ui > x uj We repeated the iterations for different values of K and calculated AUC for all values. 7

6 Challenges and optimization techniques In this section we present the challenges we faced, as well as the optimizations we performed in our models. The dataset has a huge collection of business across may cities. To scale down the data, while still retaining the sufficient problem we have chosen only the restaurant data in Las Vegas and filtered users and reviews pertaining to these restaurants The filtered data still has lots of users who have rated very less number of restaurants and this data can be problematic as it can cause cold-start issues for new users. Therefore we have retained only those users who have reviewed more than or equal to 10 restaurants in Las Vegas. This kind of data preprocessing has been done before (Liu et.al) [6] In the memory-based collaborative filtering model, we have set the missing values in the matrix to be 0. This does not mean the user gave a rating of 0. Rather, Yelp s minimum rating is 1.0. Hence the value 0 can be used to indicate that the particular observation pair is missing. This is useful in fast processing of the similarity matrix with respect to the original rating matrix, to calculate the value of r u,k We performed the above experimentations for different values of K which is the latent factor representation size of each user and restaurant. We then computed the average AUC statistic and obtained the AUC curve for each model. To speed up the computations, we digress from bootstrap sampling with replacement method by precomputing five million samples, each of which is picked in order for the first five million iterations of SGD. After these iterations are completed, we randomly pick a sample from the generated five million samples. In fact, Rendle et.al in [8] state that uniform sampling can lead to slow convergence. Our sampling technique is uniform over the set of precomputation values, but not uniform over the entire set of iterations. The BPR model which is our primary model, requires a huge number of iterations of the order O(10 9 ) to actually result in predictions with high AUC. Therefore, instead of letting the stochastic gradient descent run all the way until convergence, we have restricted the number of iterations to 10 8 to generate results of all BPR instances. We still received considerably high results, though convergence would lead to even higher results. Also, due to this early stopping, we did not overfit our model. Use of regularization in all the matrix factorization models also prevented overfitting. 7 Results and discussion We compared the AUC metric of each baseline to our main BPR-MF model and we assessed the validity of our model by showing that its AUC is higher as compared to all the baselines. A trivial predictor would have an AUC value of 0.5 since its prediction is random. We do not show this in the below AUC curve, but we definitely perform better than a trivial predictor. The other models for which we plotted the AUC curve with varying values of K are: 1. Most-popular model (Most Popular) 2. Memory-based collaborative filtering (MCF) 3. Matrix factorization (MF) 4. BPR-MF with no regularization (BPR-MF-No-reg) 5. BPR-MF with regularization (BPR-MF-Reg) The parameter tuning is decribed for each model in the model section. To summarize, Cosine similarity based MCF model required no hyperparameter tuning since it is not a learning model, rather a memory-based model.the λ Θ value is 0.001 for BPR-MF-Reg, along with a small learning rate of 10 4 With a higher regularization parameter value (eg: 0.0025), we found that our BPR-MF model has an increase in loss. Therefore, the best value of regularization parameter is a very small value. 8

The results are shown in the AUC curve plotted in figure 11 Figure 11: AUC curve The AUC value would not change for the trivial predictor, the most-popular predictor and the MCF model, since they have no dependency on the latent factor size K. However, we see that in the simple matrix factorization, the AUC decreases with an increase in K. This is consistent with the results obtained in [2], as it is stated that matrix factorization with SVD is prone to overfitting. We see that the increase in K improves the values of the AUC for both the BPR-MF variants. But a more prominent observation is the sharp increase in the AUC and then a stagnation. This means that increasing K will not increase AUC indefinitely and also very high values of K will not help in improving model s recommendation either. This is consistent with the characteristics of each model. The Most-popular model is computationally simple and quick, but is a very naive model, and therefore has the least AUC. However, it is still a better model than a trivial predictor with AUC of 0.5. The cosine similarity based MCF model looks at correlations between restaurants. This model is very intuitive in terms of the similarity measurements. However, it is computationally expensive, especially when we are dealing with sparse data sets. It also requires the entire training data to exist in the memory. Also, memory-based algorithms do not generalize well, subject to high variation in user data. The Matrix factorization based approach is faster in terms of number of iterations to converge, but a SVD-MF model as we saw above is prone to overfitting. The best model is the BPR-MF model along with some small regularization. However, as we observed, it takes too long for convergence owing to the stochastic gradient descent algorithm for parameter updates. 7.1 Conclusion In this project, we have implemented the BPR-MF model using SGD to recommend restaurants to users in Las Vegas. This model uses a maximum posterior estimator derived from Bayesian analysis as its optimization criterion. We used AUC as our evaluation criteria. We demonstrated that this model is superior to other models, which include assigning the most-popular baseline, finding item-item similarity in a memory-based model, and SVD-based matrix factorization. Acknowledgements We would like to thank Professor Mcauley as he introduced us to the above model of BPR-MF when we explained our idea to him. He suggested us to read [2] which really helped us in formulating the model and coming up with great results. We thank him for his guidance and valuable suggestions. 9

References [1] Round 9 - Yelp Dataset Challenge, https://www.yelp.com/dataset_challenge [2] Steffen Rendle et.al., BPR: Bayesian Personalized Ranking from Implicit Feedback, CoRR, 2012, http://arxiv.org/abs/1205.2618 [3] Yifan Hu, Yehuda Koren, Chris Volinsky, "Collaborative Filtering for Implicit Feedback Datasets," 2008 Eighth IEEE International Conference on Data Mining, Pisa, 2008, pp. 263-272. doi: 10.1109/ICDM.2008.22 http://yifanhu.net/pub/cf.pdf [4] Ruining He, Julian McAuley. 2016. VBPR: visual Bayesian Personalized Ranking from implicit feedback. In Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence (AAAI 16). AAAI Press 144-150. [5] Weike Pan, Li Chen. 2013. GBPR: group preference based Bayesian personalized ranking for one-class collaborative filtering. In Proceedings of the Twenty-Third international joint conference on Artificial Intelligence (IJCAI 13), Francesca Rossi (Ed.). AAAI Press 2691-2697. [6] Yang Liu, Xiangji Huang, Aijun An, and Xiaohui Yu. 2008. Modeling and Predicting the Helpfulness of Online Reviews. In Proceedings of the 2008 Eighth IEEE International Conference on Data Mining (ICDM 08). IEEE Computer Society, Washington, DC, USA, 443-452. http: //ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=4781139 [7] Pan R., et.al., One-class collaborative filtering. In Data Mining, 2008. ICDM 08. Eighth IEEE International Conference on, 502 511. IEEE http://www.rongpan.net/publications/ pan-oneclasscf.pdf [8] Steffen Rendle, Christoph Freudenthaler. 2014. Improving pairwise learning for item recommendation from implicit feedback. In Proceedings of the 7th ACM international conference on Web search and data mining (WSDM 14). ACM, New York, NY, USA, 273-282 10