Yelp Star Rating System Reviewed: Are Star Ratings inline with textual reviews?

Transcription

1 Yelp Star Rating System Reviewed: Are Star Ratings inline with textual reviews? Eduardo Magalhaes Barbosa 17 de novembro de Introduction Star classification features are ubiquitous in apps world, but how good are people star rating business at Yelp, compared with user textual reviews? Are do they sending the same message? Using Yelp data and NLP Sentiment Analysis this paper compares the rating of stars given for each Review with textual scores from Sentiment Analysis. 2 Methods and Data Yelp data came packed into 5 JSON ziped files each one of them corresponding to Business, Checkin, Review, Tip and Users data. After initial data exploration and further decision on which question would be answered, only three datasets have been chose for further treatment: Review, Business and Users. The next step was to include an external dataset for PNL Sentiment Analysis. In this case datasets from Minqing Hu and Bing Liu (Hu and Liu, KDD 2004) were used. This dataset can be downloaded here ( lexicon English.rar). Basically this dataset provides Negative and Positive words to be tested against the Yelp review texts. The outcome is an variable associated with each review giving a score (negative scores denote negative sentiments while positive scores denotes positive sentiments). The range for Yelp reviews scores given were 55, 77. str(review) ## 'data.frame': obs. of 11 variables: ## $ review_id : chr "tbf1ki PGpXnE34hfIvPgQ" "2jfcTYa2v8b3foH9mxx57Q" "Q8dYqdnwHT xdh1cckyes5q" "45Ded2REO9ndfe88eRx6zA"... ## $ stars : Factor w/ 5 levels "1","2","3","4",..: ## $ date : chr " " " " " " " "... ## $ score : int ## $ longitude : num ## $ latitude : num ## $ cat : chr "Other" "Other" "Other" "Food"... ## $ review_count : int ## $ friends : int ## $ popularity : int ## $ years_yelping: num After including scores on Review, Business and Users data were mergerd into a single Review dataset and some data transformation done, some columns were left for future studies about demographics: Business Category was simplified by a two factor cat Food or Other ; Popularity is

2 the sum of all users votes (Cool + Useful + Funny); YearsYelping is the simple math from users YelpingSince and 2015 in years; Latitude and Longitude come from the business. The means of Sentiment Analysis Scores by Stars are below: ## ## However, the question remains: differences between score means and star ratings are statistically significant? Analysis Of Variance (AOV) was used to compare means of Sentiment Analysis scores and Stars, giving the null and alternative hypothesis: Null Hypothesis: all five stars means are equal > there is NO relationship between scores means and stars, which we can write as follows: H0: 1S = 2S = 3S = 4s = 5s Alternative Hypothesis: not all star score means are equal > there is a relationship between stars and score means: H1: not all S are equal The chart is clear that Score Means are different for each Star, as well as the number of reviews taken into account. That shows the means scores are lower for lower stars, and upper for greater stars. The underlying question is if is that enough to provide evidence against the null hypothesis? The boxplot chart below confirmed that means are different for each Star factor. However, it also showed that each Star presents a important amount of variation/spread in Review Scores, so that there were much overlap of values between stars. Thus, differences in means could have come about by chance (and the null hypothesis case shouldn t be rejected). An ANalysis Of VAriance (ANOVA) test was used to answer this question.

3 The question answered with ANOVA was: are the variations between the stars means due to true differences about the review score means or just due to sampling variability? To answer this question, ANOVA calculates a parameter called F statistics, which compares the variation among sample means (among different Stars in our case) to the variation within groups (within Stars). F statistics = Variation among sample means / Variation within groups F statistics were used to compare if the variation among sample means dominates over the variation within groups, or not. In the first case there are a strong evidence against the null hypothesis (means are all equals), while in the second case will reinforce the null hypothesis. aov_stars< aov(score ~ stars, Review) summary(aov_stars) ## Df Sum Sq Mean Sq F value Pr(>F) ## stars <2e 16 *** ## Residuals ## ## Signif. codes: 0 '***' '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1 ANOVA summary showed a consistent F values and p values are low. In plain words, the variation of score means among different stars (numerator) is much larger than the variation of score means within each stars, and p value is less than 0.05 (as suggested by normal scientific standard). Hence, given the confidence interval, the alternative hypothesis H1 was accepted and that there was a significant relationship between Stars and Score means.

4 Nevertheless ANOVA showed that not all means are equals!, however Stars is a category variable with more than 2 levels (indeed it has 5), and it might be that it s just one star that is not equal to the others. ANOVA does not tell which groups (stars) are different from the others. In this sense, there was a need to check each pair of stars to determine if there were significant differences. To determine which groups are different from the others a post hoc test was conducted, where the Tukey post hoc test was used, calling the function TukeyHSD in R as follows: tuk< TukeyHSD(aov_stars) tuk ## Tukey multiple comparisons of means ## 95% family wise confidence level ## ## Fit: aov(formula = score ~ stars, data = Review) ## ## $stars ## diff lwr upr p adj ## ## ## ## ## ## ## ## ## ## Results Taking the results of TukHSD function (stands for Tukey Honest Significant Differences), diff columns shows the difference in score means for each pairwise group of Stars. The p adj confirms their significance. Thus, this study concludes that: There was a significant difference in Sentiment Analysis Scores between Stars (p= 0.00) for all pair wise of stars studied. Finally, the next chart showed star pairs and analyses significant differences by plotting the tuk object in R. Significant differences are the ones which did not cross the zero value (none in this case).

5 4 Discussion The overall alignment between Sentiment Analysis Review Scores and the Stars Rating were demonstrated throughout this document, confirming that, in average, Stars given are in line with textual reviews users made. Users can quickly rate a business, or write further reviews with more details about the business and those two opinions will match. Better in the sense of favorable to the business text reviews yield more stars, as the opposite is equally true. The implications should not be disregarded by business owners, as the reviews, either Stars or textual, are replicated thru the world of mouth, in social networks and most important, on classification algorithms used by applications such Yelp. Better rated business are positioned atop the listing generating more business, more revenue. Notice, this is the mean of all reviews by Star factor, and future studies could aim to do the same test using demographic variables from business and users, i.e. are users that are more experienced or users that are more popular do better reviews than newbies?