Which movie we can suggest to Anne?

Transcription

1 ECOLE CENTRALE SUPELEC MASTER DSBI DECISION MODELING TUTORIAL COLLABORATIVE FILTERING AS A MODEL OF GROUP DECISION-MAKING You kow that the low-tech way to get recommedatios for products, movies, or etertaiig web sites is to ask your frieds. You also kow that some of your frieds have better taste tha others, somethig you ve leared over time by observig whether they usually like the same thigs as you. As more ad more optios become available, it becomes less practical to decide what you wat by askig a small group of people, sice they may ot be aware of all the optios. This is why a set of techiques called collaborative filterig was developed. A collaborative filterig algorithm usually works by searchig a large group of people ad fidig a smaller set with tastes similar to yours. It looks at other thigs they like ad combies them to create a raked list of suggestios. There are several differet ways of decidig which people are similar ad combiig their choices to make a list ; this tutorial cover a few of these. Which movie we ca suggest to Ae? The ratigs of seve movie critics, give betwee 0 ad 5 o six movies, are give i the table below. The titles of these movies are Lady i the waters (Lady), Sakes o the Plae (Sakes), Just My Luck (Luck), Superma Returs (Superma), You, Me ad Dupree (Dupree) ad The Night Listeer (Night). A empty cell meas that the critic does ot see the movie ad is ot able to evaluate it. Lady Sakes Luck Superma Dupree Night Lisa Rose Gee Seymour Michael Phillips Claudia Puig Mick Lasalle Jack Matthews Toby Ae, a studet of Cetrale Supelec, eeds your help to choose betwee three movies Sakes, Superma ad Night, the movie she could see based o your recommedatios. She gave the followig ratigs to the other movies she already saw. Lady Sakes Luck Superma Dupree Night Ae Our objective is to provide recommedatios to Ae, takig ito accout the ratigs of the seve movie critics. Pytho laguage is oly used here as a example i the implemetatio of this model, but you ca choose aother laguage (Java, C++,... ).. The term collaborative filterig was first used by David Goldberg at Xerox PARC i 992 i a paper called Usig collaborative filterig to weave a iformatio tapestry. He desiged a system called Tapestry that allowed people to aotate documets as either iterestig or uiterestig ad used this iformatio to filter documets for other people. There are ow more tha hudreds of web sites that employ some sort of collaborative filterig algorithm for movies, music, books, datig, shoppig, other web sites, podcasts, articles, ad eve jokes.

2 . Build a dictioary critiques cotaiig the seve movie critics ad their ratigs (the dictioary is oly a suggestio for the represetatio of the prefereces of movie critics. You ca choose aother data structure). 2. After collectig data about the thigs people like, you eed a way to determie how similar people are i their tastes. You do this by comparig each perso with every other perso ad calculatig a similarity score. I our case, we should determie which persos are similar to Ae. (a) To compute a simply similarity score, we use the Mahatta distace or a Euclidea distace. Hece, if represet the umber of movies both rated by the critics x ad y, the the similarity score betwee x ad y is give by : their Mahatta distace d(x, y) defied by d(x, y) = x i y i their Euclidea distace d(x, y) defied by d(x, y) = (x i y i ) 2 2. d(x, y) = ( where x = (x,..., x ) ad y = (y,..., y ) are the ratigs vectors of the movies evaluated by x ad y (the movies o evaluated by x or y are ot take ito accout i these formulas). We ca geeralize Mahatta Distace ad Euclidea Distace to what is called the Mikowski Distace Metric 2. i. Build fuctios sim_distacemahatta ad sim_distaceeuclidiee which retur a similarity score, based respectively o Mahatta distace ad Euclidea distace, for give two persos. Idicatio : For istace, we ca build a fuctio def sim_distaceeuclidiee(perso,perso2) where perso et perso2 are dictioaries cotaiig ratigs give by the users (critics) Example : the dictioary associated to the evaluatios of Lisa Rose is obtaied by : >>> critiques[ Lisa Rose ] { Lady : 2.5, Sake :3.5, Luck : 3.0, Superma : 3.5, Dupree : 2.5, Night : 3.0}. Hece we have : >>> sim_distaceeuclidiee(critiques[ Lisa Rose ], critiques[ Gee Seymour ]) ii. For each distace above, build the fuctio recommed(ouveaucritique, Critiques) returig a list of movies to recommed to the user ouveaucritique based o the tastes of the other critics. Idicatio : For istace, we ca use the followig fuctio, based o Mahatta distace, computenearestneighbor(ouveaucritique, critiques), returig a sorted list of critics close to ouveaucritique. x i y i r ) r. Whe r =, the formula is Mahatta Distace ad whe r = 2, the formula is Euclidea Distace 2

3 def computenearestneighbor(ouveaucritique, Critiques): distaces=[] for critique i Critiques: if critique!=ouveaucritique: distace=sim_mahatta(critiques[critique], Critiques[ouveauCritique]) distaces.apped((critique,distace)) distaces.sort() retur distaces By testig these fuctios, we should have : >>> computenearestneighbor( Lisa Rose, Critiques) [(.5, Michael Phillips ), (2.0, Claudia Puig ), (2.5, Ae ), (3.0, Mick LaSalle ), (3.0, Toby ), (3.5, Jack Matthews ), (4.5, Gee Seymour )] >>> recommed( Lisa Rose, Critiques) [] >>> recommed( Toby, Critiques) [( Lady,.5), ( Luck, 4.0)] iii. Fidig a good critic to read is great, but what Ae really wats is a movie recommedatio right ow. Ae could just look at the perso who has tastes most similar to her ad look for a movie he likes that Ae has ot see yet, but that would be too permissive. Such a approach could accidetally tur up reviewers who have t reviewed some of the movies that Ae might like. It could also retur a reviewer who stragely liked a movie that got bad reviews from all the other critics retured by topmatches. To solve these issues, you eed to score the items by producig a weighted score (global score) that raks the critics. The procedure which determie the recommedatios to suggest to Ae is describe i two steps below. For a give movie a, let us deote by C(a) the list of all the critics which gave a ratig to a, ad x(a) the ratig give to a by the critic x. A. Step : For each movie a ot see by Ae, compute the quatities total(a) = s(a) = x C(a) x C(a) + d(x, Ae) x(a) + d(x, Ae) s (a) = total(a) s(a) The quatity s (a) traslates the fact that a perso similar to Ae will more cotribute, to the global score, tha a perso which is differet to her. By usig a Mahatta distace, we have for the movie Night o see by Ae : C(a) = {Lisa Rose, Gee Seymour, Michael Phillips, Claudia Puig, Mick LaSalle, Jack Matthews} total(a) = = s(a) = s (a) = B. Step 2 : The movie to recommed to Ae will be the movie with the highest global score s (a). From these explaatios, build the fuctio Bestrecommed suggestig to Ae a recommedatio betwee the three movies Sakes, Superma ad Night. 3

4 iv. A slightly more sophisticated way to determie the similarity betwee people s iterests is to use a Pearso correlatio coefficiet. The correlatio coefficiet is a measure of how well two sets of data fit o a straight lie. The formula for this is more complicated tha the Euclidea distace score, but it teds to give better results i situatios where the data is ot well ormalized for example, if critics movie rakigs are routiely more harsh tha average. Hece, if represet the umber of movies both rated by the critics x ad y, the the similarity score betwee x ad y is give by the Pearso correlatio coefficiet p(x, y) as follows : ( x i )( y i ) ( x i y i ) p(x, y) = ( x i ) 2 ( y x 2 i i ) 2 yi 2 where x = (x,..., x ) ad y = (y,..., y ) are the ratigs vectors of the movies evaluated by x ad y (the movies o evaluated by x or y are ot take ito accout i this formula). This fuctio will retur a value betwee - ad iclusive. A value of meas that the two people have exactly the same ratigs for every item. - idicates perfect disagreemet. It is a icreasig fuctio, i.e. ulike with the distace metric, you do t eed to chage this value to get it to the right scale i the computatio of s (a). Build the fuctio PearsoRecommed suggestig to Ae, by usig the Pearso correlatio coefficiet, a recommedatio betwee the three movies Sakes, Superma ad Night. Idicatio : Oe may use the followig fuctio determiig the Pearso correlatio coefficiet betwee two users. def pearso(perso, perso2): sum_xy=0 sum_x=0 sum_y=0 sum_x2=0 sum_y2=0 =0 for key i perso: if key i perso2: += x=perso[key] y=perso2[key] sum_xy +=x*y sum_x += x sum_y += y sum_x2 += x**2 sum_y2 += y**2 deomiator = sqrt(sum_x2 - (sum_x**2) / ) * sqrt(sum_y2 - (sum_y**2) / ) if deomiator == 0: retur 0 else: retur (sum_xy - (sum_x * sum_y) / ) / deomiator v. By usig as a similarity score the Cosie similarity, suggestig to Ae, by usig the Pearso correlatio coefficiet, a recommedatio betwee the three movies Sakes, Superma ad Night. We recall the followig formula of 4

5 Cosie betwee two users x ad y : x i y i cos(x, y) = x 2 i y 2 i where x = (x,..., x ) ad y = (y,..., y ) are the ratigs vectors of the movies evaluated by x ad y (the movies o evaluated by x or y are ot take ito accout i this formula). The cosie similarity ratig rages from idicated perfect similarity to - idicate perfect egative similarity. Note that this fuctio, which is very popular i text miig, is a icreasig fuctio. Which similarity measure to use? If the data is subject to grade-iflatio (differet users may be usig differet scales) use Pearso. If your data is dese (almost all attributes have ozero values) ad the magitude of the attribute values is importat, use distace measures such as Euclidea or Mahatta. If the data is sparse cosider usig Cosie similarity. 5