Improving Popular Content Recommendation with Social Profiles

Transcription

1 Universidad Autónoma de Madrid Escuela Politécnica Superior Departamento de Ingeniería Informática Improving Popular Content Recommendation with Social Profiles Víctor Villasante Guerrero Supervisor: David Vallet Weadon Trabajo Fin de Máster Programa Oficial de Posgrado en Ingeniería Informática y de Telecomunicación. Universidad Autónoma de Madrid Marzo 2014

2

3 Abstract Microblogging services such as Twitter enable the detection of popular content that is mentioned (shared) by users. In addition, the social activity of users can be used to profile their interests, in a way that popular content can be recommended in a personalized manner. In this work, we investigate various alternatives of profiling content and users to recommend popular content. We make use of concept detection techniques as well as extraction of social tags from social bookmarking services (e.g. Delicious). The former representation has been proven to be effective for recommendation, but requires text to be analyzed, while the latter is content-agnostic, but has low coverage over recent content. We address these drawbacks by representing the content by exploiting the social profiles of the users that share it. An evaluation with real users shows that our proposed technique outperforms the other approaches. We also show how all the representations can be combined in an effective way, improving the recommendations significantly in both terms of accuracy and novelty. i

4

5 Contents 1. Introduction Related Work News Recommendation Social Sharing and Popularity Social Recommendations Social-Based Content Recommendation User and Item Profile Social Sharing and Popularity Recommendation Recommendation Service Personalizing News Recommendations News Recommendation Service Predicting News Popularity Social Sharing Concept Detection Social Annotation Scoring Scoring techniques Combination Experimental Setup An Evaluation Front-end Data Collection Results Overall Performance Basic scoring Extended scoring Combination Performance Profiling Technique Coverage and Size Conclusions Summary and Contributions Discussion and Future Work Bibliography iii

6

7 1. Introduction Nowadays, users have problems to find relevant content due to information overload. The non-stop growth of information available on the Web and the success of social networks, where people are constantly sharing content, complicate the access to information that truly interest users. In order to solve this problem, the analysis of social sharing networks such as Twitter 1 or Facebook 2 has proven to be effective in identifying popular content and popular trends (Bandari et al., 2012; Cui et al., 2012). However, the identification of popular content by itself is just one step, as these days using a one-size-fits-all solution for content recommendation is not feasible due to the diverse preferences of users. Therefore, the use of some sort of personalization has to be used to generate recommendations specifically tailored to the user s interests. Several works have focus on analyzing the content of items to perform personalized recommendations (Ricci et al., 2011). These techniques perform extraction of relevant keywords or concepts from the content of items that are then matched with the user profile (content-based recommendation). The main shortcoming of such techniques are that they are content dependent, and content such as images and videos cannot be analyzed easily. The use of social tagging activities for representing content has been another approach investigated previously in the context of Web search personalization and content recommendation (Cantador et al., 2010; Vallet et al., 2010). However, these techniques are dependent on having consolidated content that has been annotated by several users. Therefore, these techniques may not fit the domain of popular content recommendation, as many popular content is recent and has not yet been annotated (e.g. in the context of news recommendation). To deal with the possible lack of coverage of both content analysis or use of social tagging activities, we suggest to relate content to the users that are sharing it. We hypothesize that, to some degree, the content that users share in social networks or microblogging services is related to the users interests. These interests can be extracted from their social activity to ultimately personalize recommendations. We present an approach that exploits the posting activity of users in a social sharing service, such as Twitter, to both extract popular content, and personalize recommendations. For the former, we predict the number of times the content is going to be shared by users. For the latter, we exploit different content and user profile representations investigated in this work to personalize the detected popular content. We provide insight on the feasibility of such techniques by evaluating a news recommendation system, which follows our approach, with real users. This evaluation allows us to mimic a real setup in which a user is receiving real-time popular 1 Twitter, 2 Facebook, 1

8 2 1. Introduction recommendations based on their social profile and allows us to evaluate our techniques in terms of accuracy and novelty. The obtained results have proven that our innovative method for creating social based profiles greatly improves the coverage and size of the content profiles. We propose several approaches that make use of these profiles, and show how they can be used to improve the recommendation of popular content. Furthermore, our approach can be easily altered in order to take advantage of additional social network information and to perform other types of content analysis. The main contributions of our work are: 1. We perform a user-centered evaluation of popular content recommendation based on microblogging profiles. We evaluate a news recommender system, which uses Twitter profiles to provide personal recommendation over popular content identified in Twitter itself. We evaluate the different sources of content recommendation with respect to user perceived accuracy and novelty. 2. We explore multiples sources for popular content recommendation, such as concept extraction, and use of social tagging. In addition, we introduce the notion of representing content using the profiles of the users that shared such content. We show how the latter technique achieves significant improvements in terms of accuracy and coverage, and allows a content-agnostic representation of popular documents. 3. We investigate the combination of the different methods of content and user representation, showing that such combination produces better accuracy and novelty of recommendations. The rest of the document is structured as follows: Chapter 2 presents a comprehensive study of work related to the proposed approach. In chapter 3, we describe our content based recommendation approach using social profiles for improving popular content recommendation. Chapter 4 presents a news recommender system that implements our approach. Chapter 5 describes the front-end developed for the evaluation of our news recommender system and its collected evaluation data. In chapter 6 we discuss the obtained results from the evaluation. Finally, chapter 7 presents conclusions and future work.

9 2. Related Work In this section we present a comprehensive analysis of related works to our approach. We separate this analysis in three subtopics. First, we study works focused on general news recommendation. Second, we examine works studying the role of social networks in information spreading. Finally, we end this section describing and comparing studies focused on social recommendation that are closely connected to our proposed model. 2.1 News Recommendation There has been a lot of interest from academia and industry to enhance the experience of online news reading. Recommendation of news articles based on user preferences has become one of the key areas where researchers have focus recently. Various famous news aggregators, such as Google News 3 or Yahoo! News 4 provide news recommendations to millions of users on a daily basis. Collaborative filtering, content-based and hybrid approaches have been used as recommendation methods in these web services (Das et al., 2007; Lihong Li et al., 2010; Liu et al., 2010) and other types of applications (Adomavicius & Tuzhilin, 2005). Das et al. (2007) described a content agnostic approach of collaborative filtering to generate personalized recommendations in Google News. Agnostic approaches are content independent and therefore the same approach can be used to recommend different types of items, such as movies and news. They aggregate recommendation from three different collaborative filtering approaches using a linear model. The combined approaches were based on clustering, probabilistic latent semantic indexing and covisitation counts. More recently, Liu et al. (2010) described how they improve the recommendation technique from Das et al. (2007). They added a content-based Bayesian model that uses the click-behavior from the target user and from the reader community. Lihong Li et al. (2010) described a system based on a learning algorithm that also uses user-click information to select news to maximize total user clicks. This system was implemented in Yahoo! News successfully. As in other applications of recommender systems, news recommenders are affected by two problems: data sparsity and cold start (Adomavicius & Tuzhilin, 2005). However, in news recommendation these problems increase as new users, but especially new items, are constantly coming to the system. Lin et al. (2012) proposed a method that focuses on solving these problems. It integrates content based, collaborative filtering methods, and information diffusion (i.e. word of mouth ) using probabilistic matrix factorization techniques. They combine these techniques to be able to perform recommendations of 3 Google News, 4 Yahoo! News, 3

10 4 2. Related Work new items by focusing on the item content; provide recommendations to new users by trusting a community of experienced users (i.e. experts ); and recommend new items to new users by creating a default profile aggregating pseudo ratings from all experts. Lei Li et al. (2011) described a news recommendation system that shows articles using a two-level representation. The first level contains general topics summarized from multiple news articles that are selected from clusters similar to the user profile. The user can select these clusters to get the second level representation. This level contains representative news articles from the selected cluster. The system uses content from the news reading history of the user, access patterns, and named entities extracted from these news to perform the recommendations. The analyzed studies are several representative works regarding news recommendation. We decided to test our content recommendation approach by applying it to a news recommender system. However, the type of content is the only particularity that our work and these studies have in common. Our recommendation system focuses more on exploiting certain features of social networks than in the recommended item per se. Following, we present several studies focused on analyzing social networks and how the information spread among them. 2.2 Social Sharing and Popularity In this subsection we identify works that have studied how the activity in social networks, such as publicly sharing content, affects information spreading. These studies focus on content diffusion within social networks, and how to identify popular content or content likely to become popular. Java et al. (2007) performed a study of the topological and geographical properties of Twitter on its early days. They observed that people use the social network to seek and share information tending to share latest news or comment about current events. They also detected that like-minded people are likely to connect with each other. In this early study they identified that 13% of messages contained some URL. However, a recent study suggests that this ratio has increased over time up to 29.1% (Gao et al., 2012). A later work from Kwak et al. (2010) studied the topological characteristics of Twitter by analyzing trending topics based on their active time period, and the content of their messages. Their study revealed that the majority of topics (over 85%) were news that indicate the power of Twitter as an information sharing medium. In their work they also proposed three novel methods to rank the popularity of users. These were based on the numbers of followers, by using PageRank, and by the number of retweets of their messages. They found that the first two methods achieved similar results, being the number of the retweets the best method to identify popular users. Lehmann et al. (2013a) observed that users devote substantial amount of effort to share content through online social networking. Particularly, they identified that when a user shares a news article in Twitter, the following messages are correlated during a brief period of time. This observation allowed them to design a method to identify users who

11 2.2. Social Sharing and Popularity 5 may provide more information related to a news story. This work led the authors to develop a system (Lehmann et al., 2013b) aiming to help news editors by detecting follow-up stories of their news articles. The system aggregates users who tweet a particular news article and detects, using a random forest classifier, content shared by these users (i.e. URLs) that are related to the original article. Another study about news diffusion on social networks was performed by Petrovic et al., (2013). They analyzed articles from news agencies, and also a collection of tweets both from the same time period. The objective was to determine whether Twitter could replace news agencies as a source for discovering news events. Their results indicate that Twitter reports similar notorious events in the same way as news agencies, along with a large number of minor events ignored by mainstream media. However, neither of which performed better than the other when dealing with major news events. Therefore, using information from social networks, such as Twitter, helps to cover major news events, as in the system developed by Lehmann et al. (2013b), and also cover hyper-local news. Bakshy et al. (2012) performed a large-scale field experiment where they examined the role of social networks in online information diffusion. Specifically, their experiments evaluated how much exposure to a URL posted on Facebook increases an individual s propensity to share that URL. Their experiments indicated that social relationships characterized by infrequent contact have great effects on the diffusion of novel information. By means of individuals having access to information that the others do not. Another study from Lerman & Ghosh (2010) also shown that social networks play a crucial role in information spreading. They analyzed how news articles were shared within the social network of users from Digg and Twitter. They observed that in Twitter the articles were spread through the network slower during the initial stages than on Digg. This is due to a less interconnected social topology. However, they revealed that news on Twitter continue spreading as the stories age ultimately reaching more individuals than Digg stories. Bandari et al. (2012) described a method to predict accurately ranges of popularity of news articles on Twitter. To characterize an article they used four features: source of the article, the category, subjectivity in the language, and the named entities mentioned in the article. Their results indicated that these features were enough to predict ranges of popularity for articles; being the news article source the most important feature. Ahmed et al. (2013) described a content agnostic method to identify future popular content, which was tested against Digg 5, Vimeo 6 and YouTube 7 data sets. Their method uses two features: the attention that specific content gets from a user with respect to all other observed content; and the normalized rate of change in the attention that the content gets from the whole community, in both cases during the same time interval. Their results 5 Digg, 6 Vimeo, 7 YouTube,

12 6 2. Related Work revealed that, using these features, their approach performed better than baselines based on clustering, linear regression and comparable work in the area. In our work we use social sharing networks to detect popular content and to describe it. Some of the above studies have revealed that social networks play a crucial role in information diffusion. Particularly, some works note that much of the content shared in Twitter is related to news events. Therefore, using this network to detect and profile news articles is appropriate for our content recommendation approach. Following, we present several works which have used Twitter for recommendation purposes. 2.3 Social Recommendations In this subsection we analyze works that have used information from the social network Twitter to generate recommendations. There are two main trends among these works. Some works focus on recommendation of specific Twitter content, such as users to follow, tweets, etc. Other works use Twitter social information to support the recommendation of external content, such as news articles or URLs. Hannon et al. (2010) published an interesting work about the recommendation of relationships on Twitter. They defined seven user profiling strategies, which allowed them to perform recommendations by employing content based and collaborative filtering approaches. Four strategies used in the content based approach were based on weighted term vectors. These terms were extracted from the user own tweets, from its followees, and followers plus a combination of them. Three strategies based on the relationship between user followees, followers and its combination were used for the collaborative approach. Their study indicated that although all these strategies achieved good results, the collaborative approaches obtained slightly better results than the content based strategies. Michelson & Macskassy (2010) developed a novelty approach for discovering users topics of interest. Their defined user profile was based on Wikipedia 8 categories. To obtain these categories, they extracted entities from each tweet and retrieved from Wikipedia the associated sub-tree of categories. Diaz-Aviles et al. (2012) presented the first empirical study demonstrating the viability of online collaborative filtering as an online ranking problem on Twitter. They proposed a method that recommends userspecific set of tweets based on individual preferences inferred from the user s interactions. Other method for topic recommendation on Twitter was presented by Liang et al. (2012). They proposed the use of social network information, such as the relationship between users, content of the users timelines, and temporal information, to find topics that are not only semantically relevant to the user profile but pertinent on a specific time. Xu et al. (2012) performed an analysis of user s posting behavior on Twitter and concluded that it is mainly influenced by the user s intrinsic interests but also by breaking news and messages from the user social friends. They proposed to profile users combining the 8 Wikipedia,

13 2.3. Social Recommendations 7 above factors using a latent topic model. They then evaluated their approach by recommending retweets to users. Another interesting work from K. Chen et al. (2012) on tweet recommendation complemented the use of social relationship information with additional factors, such as the authority of the publisher and the quality of the tweet. The above works represent examples of systems that recommend specific content from Twitter, such as users, topics or tweets. Next, we present some works closely related to our approach. These works use Twitter as a source to discover and profile candidate items (i.e. URLs, news articles), or to model users preferences in a similar way to our approach. Phelan et al. (2011) proposed a system that recommends news using different strategies for profiling users and for obtaining candidate news. They created two sets of candidate news; from the user RSS feeds and also from the feeds set by random users of the system. News articles from these sets were profiled by extracting weighted terms from them. Similarly, they defined several strategies for profiling target users based on extracting terms from the user s tweets along with tweets from the user s followers and followees. The main difference with our approach is that their news profiling is only based on the analysis of the news content while we also use social sharing activity to represent content. In addition, they did not use Twitter for getting candidate news. Abel et al. (2011) analyzed various strategies for constructing users' profiles using their twitter activity and studied how these strategies benefit from semantic enrichment and temporal dynamics. These strategies were based on extracted hashtags, detected entities and topics within the user s tweets. To compare the performance of the different strategies, they applied them on a news recommendation system and concluded that their entity based profiling produced the most valuable results. The main differences with our approach are that their user profiling was performed by linking news articles with the user s tweets to extract entities from both, we only use the users tweets and explicit documents (URLs) shared by them to perform the extraction of entities. Furthermore, Twitter was not used to detect candidate articles based on their popularity nor these candidates were modelled using social activity as we propose. De Francisci Morales et al. (2012) presented a method to improve the recommendations from Yahoo News by exploiting twitter users' profiles. Their approach followed Abel et al. (2011) idea of linking news with user twitter messages but adding information from the users' social neighborhood and the popularity of the news topic in the entire Twitter activity (i.e. trending topics). They combined all these signals using a learning to rank approach with training data extracted from click-log data of Yahoo. The main difference with our approach is that they used Twitter to promote articles that were covering trending topics on the social network while we use Twitter to detect a candidate set of popular articles. Also their user profiling using twitter was based on the user s relationships instead of on the content of the user s messages. The most similar approach to our proposed model is the pages recommendation system from J. Chen et al. (2010). They proposed a method to recommend not only news but any type of URLs. They constructed a bag-of-words from the users' posts to model

14 8 2. Related Work the users interests. They followed a similar approach for profiling a candidate URL by using the Twitter posts where the URL was mentioned to construct its bag-of-words. The main difference with our method is that we collect the users who have shared the URL and use their profiles to model the candidate URL. Moreover, to collect the candidate URLs, they used an extinguished web service which provided trendy URLs from Twitter and also by URLs shared by people followed by the user followees. Instead, we generate the set of candidate items by picking short and mid-term term popular URLs from news media shared on Twitter based on predicting their future popularity. Table 1 presents a summary of the recommender systems presented in this subsection. It recapitulates the recommended items, approach of filtering used, how the user and items profiles were constructed, where the recommended content was coming from, and the type of evaluation performed. As shown, several different recommendation approaches use Twitter to obtain the recommended content. The recommended content can be specific content from the social network such as tweets, relationships, etc. up to external content as news articles or pages which benefit from the social information to model candidate items or target users. These works use several approaches for generating the recommendations, however, there is a predominant use of content based filtering or hybrid methods especially on approaches focused on news recommendations. This is probably due to the nature of news articles that complicates their recommendation by collaborative filtering approaches. News articles are by definition novel items and hence are unlikely to be discovered by the user s neighborhood. Furthermore, works using collaborative filtering, instead of using previous evaluations of common items to create the user neighborhood as classic collaborative approaches, they assume that the relationships between users indicate common interests. As a result, they consider that it is adequate to use these relationships to create the neighborhood used by collaborative filtering approaches to generate recommendations. Regarding the evaluation procedure, the use of offline experiments led over evaluations with real users on the analyzed works. Experiments over data sets collected from Twitter were used in all works that recommended specific content of the social network, such as tweets or users. Only Hannon et al. (2010) vaguely corroborated their offline results evaluating with real users the algorithm that achieved the best results over the data set. However, in the works addressing the recommendation of external content (i.e. news articles and pages) there is no predominant technique of evaluation. Analyzed works most similar to our approach are those focused on news or pages recommendations. These works recommend the same type of external content (i.e. news pages) as well as propose using content based approaches to recommend such content. However, the main difference is that we use users social activity not only to model target users but to profile candidate items. Furthermore, we also focus our work on obtaining a candidate list for recommendation: we use the social sharing characteristic of Twitter to detect long term shared popular content which is then personalized to provide both popular and relevant content to the user.

15 9 Hannon et al., 2010 Diaz-Aviles et al., 2012 Liang et al., 2012 Xu et al., 2012 K. Chen et al., 2012 Phelan et al., 2011 Abel et al., 2011 De Francisci Morales et al., 2012 J. Chen et al., 2010 Items Technique User profile Item profile Items from Evaluation Users Hybrid Tweets Collaborative filtering Topics Hybrid Retweets Content based Tweets Collaborative ranking News Content based News Content based News Hybrid URLs Content based Bag-of-word from tweets for content based. Twitter user neighborhood for collaborative filtering. Twitter user neighborhood. Bag-of-word from tweets for content based. Twitter user neighborhood for collaborative filtering. Latent topics within the user tweets. Twitter user neighborhood. TF-IDF vectors from the user s tweets. Topics, entities and hashtags identified in user s tweets and news linked to those tweets. Twitter user neighborhood. Bag of words from the users tweets. No item profile. Users were the recommendations. No item profile. Tweets were extracted from the user neighborhood. No item profile. Topics were extracted from similar users and from the user neighborhood. No item profile. Retweets were extracted from similar users. No item profile. Tweets extracted from the user neighborhood. TF-IDF vectors from news articles. Topics, entities and hashtags identified in news feeds. Popularity on twitter of the news article. Bag of words from tweets containing the URLs. Twitter Twitter Twitter Twitter Twitter External (articles from users RSS) Offline and online Offline Offline Offline Offline Online External (news feeds) Offline External (Yahoo News feed) Twitter (popular URLs on Twitter or from the user neighborhood) Offline Online Table 1: Summary of recommender systems on Twitter.

16

17 3. Social-Based Content Recommendation In this section we first describe our social based user and item profiling approach, followed by a description of two requirements that the social sharing networks have to meet in order to be used with our approach. We following present various recommendation algorithms and a description of a social based content recommender system that can be used with the described profiling techniques. 3.1 User and Item Profile Our profiling approach creates the user and item profiles in a similar manner. In order to create them, we use content from social networks profiles. This content is based on user activity on these social networks such as public messages (i.e. posts, tweets, etc.) and shared documents (e.g. URLs). Following we describe both profiles. User Profile: We create a social based user profile by extracting entities, such as concepts or social tags, from the activity of the user in social networks. Extracted entities from social network interactions have been shown to be a good indicator of user s preferences (Abel et al., 2011; Cantador et al., 2010). We define a user profile as a collection of entities =,,. These entities are extracted from social activity, which is represented by the user messages and shared documents in social networks. Therefore, the representation of the user is defined as a set of entities. Item Profile: We define an item as the content linked by a URL. To create its profile we extract the same type of entities =,, from the linked content. The result of this extraction, depending on the type of content, can be a limited number of different entities or even none. For example, if the items are non-text based such as images or videos, extraction of entities (e.g. concepts, categories, etc.) can be difficult. Furthermore, profiling items using social tags can be challenging if these items are fresh (i.e. undiscovered by the community) such as news articles. In order to solve this potential issue we decided to enrich the items profiles by adding social information to increase both the coverage and the description of the items features. We assume that users who have shared the same item have some common interests and therefore, aggregating their users profiles allow us to characterize this item. We define users =,, as the set of users who have mentioned the item in their user social activity (i.e. have shared publicly the URL). By aggregating the user social based profile of these users, we compose the item social based representations. In other words, a social based item profile is defined as = 11

18 12 3. Social-Based Content Recommendation. This profiling technique will permit us to circumvent the described problems as it will not profile the items by using their content. In summary, the item profile can be represented in two ways: the entity document representation ( ) extracted from the item itself and the social-based entity document representation ( ) created from the aggregation of users' profiles who shared the item in their social activity. Figure 1 shows the suggested representation of a user and a social-based item profile. The list of weighted entities ( ) representing the user or item profiles can come from different sources of information, such as concepts, annotations, or categories. User profile u e Social share 1 Social share n User u e 1,1 e 2,1 e m,1 e 1,n e 2,n e m,n u 1 u n Item d e 1,1 e 2,1 e m,1 e 1,n e 2,n e m,n e 1,1 e 2,1 e m,1 e 1,n e 2,n e m,n Item profile d se Figure 1: Diagram of user and social item entity vectors Detailed below are the entity weighting used for computing the entity representation of users and items. User-based entity frequency =, measures how popular an entity is to a user. Being, the number of appearances of the entity in the user profile. If the entities have weights,, will be the aggregation of these appearances. User-based entity inverse frequency = measures how common or popular an entity is across all users. Being the number of users, and the number of users with the entity in their profiles. Document-based entity frequency =, Measures how relevant an entity is to a document. Being, the number of appearances of the entity in the document profile. The document profile, as explained in this section, can be the created using the item itself or from the set of users profiles who shared the document in their social activity (social based profiles). Equally as the userbased entity frequency if the entities have weights,, will be the aggregation of these appearances. Document-based entity inverse frequency =log measures how common or popular an entity is across all documents. Being the

19 3.2. Social Sharing and Popularity 13 number of documents, and the number of documents with the entity in their profiles. User size calculated as the number of collected users profiles. These collected users can be target users of the recommendations or users collected to profile documents. Average user size calculated as the mean size of collected users' profiles. 3.2 Social Sharing and Popularity To be able to perform the described social based item profiling technique there are two essential requirements. First, the social sharing network must allow users to share items not only with their friendships but publicly with the whole community. Our item profiling technique needs access to the social activity of users who have shared the item, therefore this activity must be freely available. There are several social networks that allow performing this task, Twitter and Tumblr are two clear examples as well as other social networks such as Facebook, which changed recently its policy to allow sharing publicly the user social activity. Additionally, as we need several users to profile each item, these items must be popular within the social sharing network. Consequently, the system would need not only public profiles but popular items that are being shared by several users. This requirement can be easily met using the social network Twitter for detecting and profiling news articles. Previous studies have shown that Twitter users tend to share URLs (Gao et al., 2012; Java et al., 2007) and that majority of popular topics on Twitter are news (Kwak et al., 2010). 3.3 Recommendation In this subsection we present some scoring techniques that can be used with our user and item social profile representation. Our approach allows the usage of multiple scoring techniques based on statistical models such as Okapi BM25 and tf-idf, and vector models such as cosine similarity. Okapi BM25 We use a personalization approach which is an adaptation of the user based Okapi BM25 ranking model (Robertson & Walker, 1994) to a personalization ranking of similarity between a user and a document (Vallet et al., 2010). We define a weighting function for a single entity based either on user (1) or document (2) information: =

20 14 3. Social-Based Content Recommendation = Where and are set to the standard values of 0.75 and 2, respectively and is an entity. This weighting scheme allows to perform different types of similarity scoring. We present three alternatives to do this computation: user based 25 function (3) gives more relevance to the user profile by adding the weighted values of the user profile for entities appearing in the user and item profiles. 2 25, =, 25 3 Alternatively, document based 25 function (4) gives more relevance to the item profile by using the item scores when adding scores of common entities between the user and item profiles. 25, =, 25 4 Other alternative is to give the same importance to the user and item profiles when computing the similarity. User and document based 25 function (5) uses the product of scores of common entities between the user and item profiles. 25, =, These are three examples for computing the similarity scores using the Okapi BM25 weighting scheme. Furthermore, our social-based profiling approach allows the use of other classic similarity functions such as vector space model explained following. Cosine Similarity based personalization cos, = The numerator is the scalar product of the and weighted entity vectors associated with the user and document, respectively. The denominator is the user and document length normalization factors, calculated as the magnitude value of the weighted entity vectors. Length normalization usually works well when computing similarity scores, however, Vallet et al., (2010) indicated that normalization could not be applicable in case of annotations. Social annotation produces differences in items profiles sizes depending on the popularity of the items among the community. Considering the above observation we can define the following scoring technique.

21 3.4. Recommendation Service 15 Scalar tf-if based personalization By eliminating the user and document length normalization factors to the cosine similarity personalization we obtain the scalar personalization:, = 3.4 Recommendation Service Figure 2 shows a diagram of a content based recommendations service which uses social sharing to obtain and profile candidate documents and to capture the interests of target users. In the following, we describe the overall recommendation process: Popularity classifier Top N popular items d 1 d 2 d N Profiles that shared d Content Social Expansion e 1,d e 2,d e m,d Top P Recommendation d' 1 Social Sharing Entity extraction Scoring module d' 2 d P Social User Profile User Social Extraction e 1,u e 2,u e m,u Figure 2: High-level system diagram for a social sharing based document recommender The Popularity classifier identifies popular items (e.g. URLs) which are being shared (i.e. mentioned) on a social sharing network (e.g. Twitter, Tumblr) and forwards it content to the Content social expansion module. The Content social expansion module creates a content profile of each popular item, whether by analyzing the content itself, or by analyzing profiles of users who have shared the item. The output of this module is a vector of entities representing the content. Similarly, the User social extraction module extracts relevant shared content from the target user profile, such as posts and shared links. This content can be annotated by different annotation modules which extract the entities that form the social profile of the user. The final output of the module is also a weighted vector entities, which represent the user s interests. Each popular content's profile is matched by the Scoring module to the user social based profile to produce a final personalized recommendation.

22

23 4. Personalizing News Recommendations In this section we present the implementation details of the components that comprise the social based content recommendation described in section 3.4. This implementation is used to create a real time news recommender. We use the social network Twitter as a social sharing network to capture popular news articles. Twitter is also used for performing the profiling of target users and candidate articles following our profiling approach. In this implementation we use concepts and social tags as the entities that describe both users and items. We describe how we implement the components that extract concepts and social tags by using AlchemyAPI and the social bookmarking web service Delicious respectively. We finish this section presenting the scoring techniques used for performing the recommendation of news articles. 4.1 News Recommendation Service Figure 3 shows a diagram of a news recommendations service which uses social sharing to obtain and profile candidate news articles and to capture the interests of target users. The entities describing the content are concepts and social tags. In the following, we describe the overall recommendation process: Popularity classifier Top N popular news d 1 d 2 d N Profiles that shared d Content Social Expansion c 1,d c 2,d t 1,d t 2,d c m,d t m,d Top P Recommendation Social Sharing Concept detection Social tagging Scoring module d' 1 d' 2 d P Social User Profile User Social Extraction c 1,u c 2,u t 1,u t 2,u c m,u t m,u Figure 3: High-level system diagram for a social sharing based news recommender The Popularity classifier uses Twitter to detect popular news articles. This module is described in section 4.2. The Content social expansion uses two different annotation modules to represent content. The Concept detection module processes content to extract relevant related concepts. The Social tagging module analyzes users annotated content 17

24 18 4. Personalizing News Recommendations on the Delicious 9 social bookmarking service. The result of this process are a weighted concept vector and a tag vector that represent the item. In section 4.4 and 4.5 we describe implementations for these entity extraction modules which use AlchemyAPI and Delicious. The User social extraction module also makes use of the concept detection and social tagging modules to represent the user s interest. The way the user profile is constructed using such modules is described in section 4.3. An implementation of the Scoring module is presented in section Predicting News Popularity We use Twitter as a source to obtain candidate news articles for the recommendation. In order to get the set of candidate articles we use the Twitter Streaming API 10. This API allows filtering of tweets by setting a set of keywords that can be used to match content in the tweets as well as mentioned URLs in them. We use this functionality to filter tweets that mention a predefined set of news media domains. These domains were selected using Alexa 11. Alexa provides different information about websites including the most popular (i.e. most visited) sites. We selected the top 16 US news sources from Alexa ranking to ensure getting enough tweets containing their URLs on them. News media sources usually include web widgets in their pages to share their articles on the visitors social networks. When visitors use these widgets, the article URL usually gets transformed to a shortened version of the original URL. Therefore, we use Twitter s streaming API to filter messages in which the shortened URLs appear. Table 2, below, contains the list of news sources tracked on Twitter along their hostname and shortened version. 9 Delicious, 10 Twitter Streaming API, 11 Alexa Internet,

25 4.2. Predicting News Popularity 19 Source Hostname Short Hostname Topic Al Jazeera aljazeera.com aje.me General BBC bbc.com, bbc.co.uk bbc.in General Bloomberg bloomberg.com bloom.bg Financial CNET cnet.com cnet.co Technology CNN cnn.com, edition.cnn.com cnn.it General Engadget engadget.com engt.co Technology Fox News foxnews.com fxn.ws General Forbes Forbes.com onforb.es Financial Gizmodo gizmodo.com gizmo.do Technology NBC News nbcnews.com nbcnews.to General Reuters reuters.com reut.rs General The Guardian theguardian.com gu.com General The Huffington Post huffingtonpost.com huff.to General The New York Times nytimes.com nyti.ms General The Washington Post washingtonpost.com wapo.st General Time time.com ti.me General USA Today usatoday.com usat.ly General Table 2: US news sources tracked on Twitter. For each tweet containing an URL linking to one of the media sources on Table 2, we extract several features, the most important are shown in Table 3. Feature Description Original whether the tweet has been written by the user Retweet whether the tweet is a retweet Reply whether the tweet is a reply to another tweet English whether the language of the tweet is in English English user whether the user language is set to English User followers number of followers the user who has written the tweet has. User favorites number of times the user s tweets have been marked as favorite Table 3: Most important features extracted from tweets. Since each news article is associated with multiple tweets, the extracted features of individual tweets are aggregated. We use three techniques for aggregating tweets, counter, ratio and average. Counter aggregates the overall number of tweets having a certain feature (e.g. overall number of original tweets with a link to a news article); ratio indicates the ratio between the number of tweets having a feature amongst all observed tweets (e.g. ratio between the number of tweets in English and all tweets with a link to a news article); and average for the average value of a feature (e.g. average number of followers from users that posted a tweet with a link to a news article). As we have 16 features and 3 techniques to aggregate them there are a total of 48 features to describe each news article retrieved from Twitter. Using these features we train a Gradient Boost Regressor (Friedman, 2000) to predict the popularity of the news

26 20 4. Personalizing News Recommendations articles mentioned in Twitter. The prediction is based on a 12 hours sliding window that uses the first six hours to extract the training features and the following six hours to obtain which content from the training set was actually popular, in terms of number of original mentions. Once trained, the last 6 hours are used to predict what content will be popular in the following six hours. After 6 hours, the sliding window is shifted and the model is retrained by using the training features extracted from the data previously used to test for popular content, and tested over the newly acquired 6 hours of Twitter sharing data. We opted to retrain the model each time the sliding window is shifted to be able to adapt more easily to changes of the sharing behavior (e.g., spikes on sharing behavior due to specific event, such as the Oscar s academy awards). We use the number of original tweets an article will collect as an indicator of popularity as it has been proved to be a good signal of popularity (Vallet et al., 2014). By doing this we try to differentiate short term popularity from long lasting popularity. Finally, the news articles are ranked based on this prediction. 4.3 Social Sharing As well as a source to obtain popular news, the social sharing capabilities of Twitter can also be used to profile news articles and users. As we mentioned in section 3.2, this social network is ideal for our social based profiling approach since users are likely to share publicly their activity (i.e. tweets) allowing us to profile both target users and candidate news articles. In order to compute the profile of the target user we retrieve the latest 200 publicly posted messages from the user Twitter profile. This collection of posts is then filtered by removing conversations (messages which are interactions between users) as they may not represent their opinions and interests. Following, we remove common Twitter keywords such as 'RT' (used to indicate forwarded message) and user identifiers (user name prefixed with the ). However we decided to keep hashtags (word or unspaced phrase prefixed with the symbol # ). These decisions were based on manual inspection over the entities extracted with our concept detection component to minimize the extraction of wrong entities. Therefore, the filtering performed over the posts may vary depending on the implementation of the concept detection component. Finally, we create another collection of user shared URLs by extracting links from the user posts. At the end we have two collections: filtered texts and links on which we can perform entity extraction. This extraction can be of concepts over both collections and detection of social annotations over the collection of URLs which ultimately will constitute the user profile. Table 4 exemplifies the filtering performed over sample tweets. To profile a news article, we identify users that have mentioned the article by sharing it on a public message (i.e. link to a news article within one of their tweets). Specifically, we collect randomly up to 100 of these users discarding users with less than 200 posted messages. The goal of this filtering is to avoid users who have not enough messages to perform the extraction of entities. Finally, we generate the collection of filtered texts and shared documents from these 100 users following the same process as the described above

27 4.4. Concept Detection 21 for profiling target users. This collection, based on profiles of up to 100 users who have shared the news article, is used to create the social based entity representations of a news article. Note that we were limited by the Twitter API call limits, therefore the defined numbers of users and messages were selected to maximize the capture of information within these limits. These numbers can be adjusted to, for example, extract more information from target users or from candidate news articles. Original tweet Filtered tweet Extracted URL Policemen charged over claims they illegally arrested student protester Policemen charged over claims they illegally arrested student protester I advised House of Cards on its season two plot (spoiler chttp://gu.com/p/3mzq3/tw You know you're doing serious computer work when you have to fetch pen and paper #programming I advised House of Cards on its season two plot (spoiler alert) You know you're doing serious computer work when you have to fetch pen and paper #programming Table 4: Example of the filtering performed over retrieved tweets. Marked in bold elements removed from the tweets. 4.4 Concept Detection We use the AlchemyAPI 12 concept tagging service to extract the necessary entities needed for constructing a concept-based representation. This service returns up to 8 weighted concepts for a given input which can be either free text or a URL. Due to the AlchemyAPI service call limits we need to perform the detection slightly different depending on whether the user is a target user of the recommendation or is one modeling a candidate news article. If the user is a target user, we annotate each of its 200 messages and shared documents (i.e. URLs) independently to create its concept-based user representation ( ). However, if the user is one of the up to 100 users used to represent a news article we aggregate its messages and annotate them jointly to obtain the news article social-based document representation ( ). The objective sought by extracting entities from users slightly different is to limit the number of calls to the AlchemyAPI service. Extracting concepts in a similar manner will allow us to profile only a small number of news articles 12 AlchemyAPI,

28 22 4. Personalizing News Recommendations before reaching the AlchemyAPI limit. Finally, we extract relevant concepts from the news article themself to obtain their concept-based document representation ( ). To sum up, for each target user we perform one call to the AlchemyAPI service for each extracted message and URL from the user s Twitter profile. As we retrieve 200 messages a maximum of 400 calls are made. For each candidate news article we perform up to 100 calls as we aggregate the 200 messages of each user and we do not compute their shared URLs to avoid exceeding the AlchemyAPI limits. Note that AlchemyAPI gives a relevance value between 0 and 1 for each retrieved concept. To calculate the entity frequency from both user and item profile, we sum these values for concepts appearing multiple times in their profiles. 4.5 Social Annotation We use the social bookmarking Web service Delicious to obtain a tag-based representation. This service provides the tags that the Delicious community has assigned to a given URL. We use the service to annotate the shared documents (i.e. URLs) from target users and from users used to profile news articles. As in the concept detection, to limit the number of calls to the Delicious service we annotate the URLs slightly different depending on the type of user. We annotate all URL extracted from messages of the target user to obtain its tag-based user representation ( ) and up to 5 of the latest URLs shared by the users that represent the social profile of a news article to obtain its social-tag-based document representation ( ). As well as in the concept detection, we perform the annotation over the news article URL itself to get its tag-based document representation ( ). In summary, we call Delicious service up to 200 times for target users and a maximum of 500 times for each news articles (up to 5 times for each of the up to 100 users modeling a news article). Note that Delicious returns also the number of times a specific tag has been assigned to the URL by its community. To calculate the entity frequency of both item and user profile, we sum these values for tags appearing multiple times in their profiles. 4.6 Scoring We have two types of entity extraction techniques for items: item and social based, however, their outputs produce the same types of entities: concepts and tags. As we have used the same concept and tag representation for target users we can compare both profiles as following: urlconcept: uses the concept representation from the target user ( ) and the concept representation from the item created exclusively from the content linked by the item URL ( ).

29 4.6. Scoring 23 urlannotation: uses the social tags representation from the target user ( ) and the social tag representation from the item created exclusively from the item URL ( ). usersconcept: uses the concept representation of the target user ( ) and the concept representation from the item created from messages and shared documents of users who mentioned (shared) the item URL ( ). usersannotation: uses the social tags representation of the target user ( ) and the social tag representation from the item created from shared documents of users who mentioned (shared) the item URL ( ) Scoring techniques The scoring module implemented for the news recommender system uses the four combinations between candidate items and target users profiles defined above along some scoring techniques from section 3.3 to compute the recommended news articles. Following we present in detail the different approaches followed to compute the scoring. Above are the four different possible combinations between users and documents that our social based implementation allows: urlconcept, urlannotation, usersconcept and usersannotation. To compute the similarity between these combinations we used three scoring techniques from the presented in section 3.3. During an initial experiment, we used the Okapi BM25 user based 25 scoring function. However, in a later experiment we expanded the set of scoring functions by adding the user and document based 25 Okapi BM25 and the scalar tf-idf personalization. Although it would have been ideal to test the performance of our social profiling with other scoring techniques, we had to limit our experiments due to not having enough human resources to test them. Using the defined types of scoring techniques and possible combinations between user and document profiles we can calculate a total number of 12 different scores. Each of these recommendation techniques computes the score between the target user and a set of candidate items and retrieves the five highest scored items (i.e. top 5). Table 5 presents the name we gave to these different recommendation outputs along its scoring functions and type of combination between item and document profiles. Note that, even though we used scalar tf-idf, we did not use cosine similarity. Profile type tf-idf urlconcept urlconcept BM25U urlconcept BM25UD urlconcept TF-IDF urlannotation urlannotation BM25U urlannotation BM25UD urlannotation TF-IDF usersconcept usersconcept BM25U usersconcept BM25UD usersconcept TF-IDF usersannotation usersannotation BM25U usersannotation BM25UD usersannotation TF-IDF Table 5: Name of the different computed recommendations along its profiling and scoring technique

30 24 4. Personalizing News Recommendations Combination It is possible to combine different scoring methods into a single aggregated score. One approach for performing such combination is through machine learning techniques. By using relevance feedback from users for different scores techniques it is possible to train a system to predict the final relevance value. We train a Gradient Boost Regressor (Friedman, 2000) using relevance feedback obtained from users. Specifically, we train the system using 13 features. These features are the 12 scoring function outputs plus the popularity score. We use a user-based leaveone-out approach, i.e., for each user we use the feedback provided by other users to train the regressor. We create the set of candidate items from the top 5 recommended items retrieved by each of the 13 scoring approaches. The trained regressor is then applied to predict the final relevance value that the user will give to this set of candidate documents, ranging from 1 (not relevant) to 5 (extremely relevant). Finally we select the top 5 documents from the output regressor which is understood as an aggregated score of all different techniques.

31 5. Experimental Setup In order to test the performance of our social based profiling approach we developed an evaluation front-end for the news recommendation system described in the previous section. Next, we describe the implementation of the front-end followed by a description of how the data was collected and a characterization of this data. 5.1 An Evaluation Front-end We designed a web service to allow the interaction of users with our news recommendation system. The system collects every 3 hours profiles of the most recent 600 popular news identified by the Popularity classifier, described in section 4.2, during the preceding 24 hours. When a new user accesses the service, the system requires him to log in using his Twitter account. The User Social Extraction module uses the profile of this Twitter account to create the user social based profile. The social based profile is processed by the Concept detection and Social tagging modules described in section 4.4 and 4.5 respectively. On average, the entity extraction process takes 1.5 minutes and directly depends on the number of messages and URLs extracted from the user Twitter profile and the number of concurrent users accessing the system. Figure 4: Part of the evaluation front-end For each of the 600 candidate news we compute the scoring values using the 12 techniques described in section 4.6. This process returns 60 news articles which are stored for their evaluation along with the top 5 most popular news which is our baseline. We present these news randomly in sets of up to 10 pages (i.e. without taking into consideration any score nor algorithm). News articles are presented only once even if they are within the top 5 of different techniques (i.e. overlapped news articles). In order to identify users providing random answers, or not following the evaluation directives, we randomly introduce validation pages that require the user to set a specific 25

32 26 5. Experimental Setup value of interest. These pages are easily identifiable as they have a very descriptive title (e.g. Rate this page with 4 stars ) and they direct to a page with clear instructions (see Figure 5). Figure 5: Validation page introduced during the evaluations Figure 4 shows part of the interface that displays the recommended news articles and contains inputs to provide the user's judgments. The users have to provide feedback for three questions: 1. Readability: The aim is to identify if the link to the page is broken or if its content is in a known language by the user. If the user sets the page as not readable, the page is marked as not relevant. 2. Novelty: Whether the news article or reported event is known or unknown for the user. The aim is to check the novelty of the recommended news articles. 3. Interest: The level of interest on the news article. The aim is to evaluate the different recommendation techniques used. In order to seek uniform relevance feedback we provided a description (Figure 6) for each of the stars. We define 5 stars as really interested, 4 interested, 3 somewhat interested, 2 not particularly interested, and 1 for not interested at all. Figure 6: Instructions provided to users for performing the evaluations