Quantifying Search Bias: Investigating Sources of Bias for Political Searches in Social Media

Size: px

Start display at page:

Download "Quantifying Search Bias: Investigating Sources of Bias for Political Searches in Social Media"

Bruno Merritt
5 years ago
Views:

work with Motahhare Eslami, Johnnatan Messias, Muhammad

1 Quantifying Search Bias: Investigating Sources of Bias for Political Searches in Social Media Juhi Kulshrestha joint work with Motahhare Eslami, Johnnatan Messias, Muhammad Bilal Zafar, Saptarshi Ghosh, Krishna P. Gummadi and Karrie Karahalios

2 Social media as "search" platform Rich source of news & information Search is how users follow news about events & people Hashtags - recommended queries

3 Social media as "search" platform

4 Social media as "search" platform Ranked list (according to importance)

5 Potential bias in search results

6 Potential bias in search results

7 Search can shape user opinion Users place greater trust in higher ranked items [Pan et al., 2007] Biased search results can influence voting patterns [Epstein & Robertson, 2015]

8 Search bias in the headlines

9 Search bias in the headlines What does bias of a search system mean? How can we quantify it?

10 Identify sources of search bias Quantify bias of each source Study bias of political searches in Quantifying Search Bias CSCW 2017

11 Identify sources of search bias Quantify bias of each source Study bias of political searches in Quantifying Search Bias CSCW 2017

12 Social media search engine design Data corpus

13 Social media search engine design Data corpus q i1 i2 i3... (set) input (relevant items)

14 Social media search engine design Data corpus q i1 i2 i3... Ranking system (set) input (relevant items) ranking

15 Social media search engine design Data corpus q i1 i2 i3... (set) Ranking system i5 i1 i3... input (relevant items) ranking output (ranked list)

16 Social media search engine design Data corpus q i1 i2 i3... (set) Ranking system i5 i1 i3... input (relevant items) ranking output (ranked list)

17 Social media search engine design Data corpus q i1 i2 i3... (set) Ranking system i5 i1 i3... input (relevant items) ranking output (ranked list) Output bias may stem from Bias introduced by the ranking system

18 Social media search engine design Data corpus q i1 i2 i3... (set) Ranking system i5 i1 i3... input (relevant items) ranking output (ranked list) Output bias may stem from Bias introduced by the ranking system Bias in the input relevant item set

19 Identify sources of search bias Quantify bias of each source Study bias of political searches in Quantifying Search Bias CSCW 2017

20 Quantifying bias of each source Data corpus q i1 i2 i3... (set) Ranking system i5 i1 i3... input (relevant items) ranking output (ranked list) Step 1: Quantify bias of an individual item

21 Step 1: Quantifying bias of a single items

22 Step 1: Quantifying bias of a single items We use source bias as a proxy Infer bias of each individual item from the bias of the author High scalability

23 Step 1: Quantifying bias of a single items We use source bias as a proxy Infer bias of each individual item from the bias of the author High scalability Prior work on inferring content bias Could be plugged into our bias quantification framework

24 Step 1: Quantifying bias of a single items Evaluation: High agreement between source and content bias (75% or more)

25 Step 1: Quantifying bias of a single items Evaluation: High agreement between source and content bias (75% or more)

Step 1: Quantifying bias of a single items -1 0 1 Evaluation: High agreement

26 Step 1: Quantifying bias of a single items Evaluation: High agreement between source and content bias (75% or more) "R-NC 11th District, Republican party"

27 Step 1: Quantifying bias of a single items Evaluation: High agreement between source and content bias (75% or more)

28 Step 1: Quantifying bias of a single items Evaluation: High agreement between source and content bias (75% or more) "Republican party nominee for Presidential elections"

29 Quantifying bias of each source Data corpus q i1 i2 i3... (set) Ranking system i5 i1 i3... input (relevant items) ranking output (ranked list) Step 2: Quantify bias of a set of items

30 Step 2: Quantifying bias of set of items i1 i2 i3... (set) Compute bias score (s) for each item Take the average over the whole set input (relevant items)

31 Quantifying bias of each source Data corpus q i1 i2 i3... (set) Ranking system i5 i1 i3... input (relevant items) ranking output (ranked list) Step 3: Quantify bias of a ranked list of items

32 Step 3: Quantifying bias of ranked list of items MAP-style measure i5 i1 i3... output (ranked list) Bias till rank r Output bias

33 Quantifying bias of each source Data corpus q i1 i2 i3... (set) Ranking system i5 i1 i3... input (relevant items) ranking output (ranked list) Step 4: Quantify bias introduced by ranking system

34 Step 4: Quantifying bias introduced by ranking Ranking system Ranking bias = Output bias - Input bias ranking

35 Identify sources of search bias Quantify bias of each source Study bias of political searches in Quantifying Search Bias CSCW 2017

36 Studying bias for political searches in Twitter Search queries for 2016 Democratic and Republican presidential primary debates (#demdebate, rep debate,...) Presidential candidates (Hillary Clinton, Donald Trump,...)

37 Studying bias for political searches in Twitter Search queries for 2016 Democratic and Republican presidential primary debates (#demdebate, rep debate,...) Presidential candidates (Hillary Clinton, Donald Trump,...) Data collected Output: Twitter top search snapshots every 10 mins Input: All tweets containing the query, using streaming api

38 Studying bias for political searches in Twitter Search queries for 2016 Democratic and Republican presidential primary debates (#demdebate, rep debate,...) Presidential candidates (Hillary Clinton, Donald Trump,...) Data collected Output: Twitter top search snapshots every 10 mins Input: All tweets containing the query, using streaming api Non-personalized search data

39 Studying bias for political searches in Twitter Search queries for 2016 Democratic and Republican presidential primary debates (#demdebate, rep debate,...) Presidential candidates (Hillary Clinton, Donald Trump,...) Data collected Output: Twitter top search snapshots every 10 mins Input: All tweets containing the query, using streaming api Computed input, ranking, and output bias for each of the 25 queries

40 Bias in Twitter search: Input bias vs. Ranking Quantifying Search Bias CSCW 2017

41 Bias in Twitter search: Input bias vs. Ranking bias Does input bias Quantifying Search Bias CSCW 2017

42 Impact of input bias Query Output Bias Input Bias Bernie Sanders Martin O'Malley Rand Paul John Kasich dem debate #demdebate republican debate rep debate

43 Impact of input bias Query Output Bias Input Bias Bernie Sanders Martin O'Malley Rand Paul John Kasich dem debate #demdebate republican debate rep debate Input bias matters!

44 Impact of input bias Query Output Bias Input Bias Bernie Sanders Martin O'Malley Rand Paul John Kasich dem debate #demdebate republican debate rep debate Input bias varies across queries

45 Effect of query phrasing Query Input Bias dem debate 0.29 #demdebate 0.56 republican debate 0.27 rep debate 0.40

46 Effect of query phrasing Query Input Bias dem debate 0.29 #demdebate 0.56 republican debate 0.27 rep debate 0.40 Even for the same event, query phrasing can greatly effect the bias

47 Bias in Twitter search: Input bias vs. Ranking bias Does input bias matter? Input bias does matter Can vary significantly based on the query Even for the same event, different phrasings of queries have widely differing Quantifying Search Bias CSCW 2017

48 Bias in Twitter search: Input bias vs. Ranking bias Does input bias matter? Input bias does matter Can vary significantly based on the query Even for the same event, different phrasings of queries have widely differing biases Does the ranking bias Quantifying Search Bias CSCW 2017

49 Examining ranking bias Query Ranking Bias Hillary Clinton 0.18 Bernie Sanders 0.16 Martin O'Malley 0.07 Donald Trump 0.10 Ted Cruz Marco Rubio Ben Carson 0.26

50 Examining ranking bias Query Ranking Bias Hillary Clinton 0.18 Bernie Sanders 0.16 Martin O'Malley 0.07 Donald Trump 0.10 Ted Cruz Marco Rubio Ben Carson 0.26 Ranking bias does exist...

51 Examining ranking bias Query Ranking Bias Hillary Clinton 0.18 Bernie Sanders 0.16 Martin O'Malley 0.07 Donald Trump 0.10 Ted Cruz Marco Rubio Ben Carson but no evidence of systemic bias

52 Bias in Twitter search: Input bias vs. Ranking bias Does input bias matter? Input bias does matter Can vary significantly based on the query Even for the same event, different phrasings of queries have widely differing biases Does the ranking bias exist? Yes and varies across queries No evidence of systemic Quantifying Search Bias CSCW 2017

53 Open challenge: How to address search Quantifying Search Bias CSCW 2017

54 Open challenge: How to address search bias? Modify ranking system to account for bias Might lead to reduction in quality of Quantifying Search Bias CSCW 2017

55 Open challenge: How to address search bias? Modify ranking system to account for bias Might lead to reduction in quality of results Make the bias transparent Keep the current ranking Inform the users about the bias they are seeing Make biases related to query phrasing Quantifying Search Bias CSCW 2017

56 Quantifying Search Bias CSCW 2017

57 Quantifying Search Bias CSCW 2017

58 Quantifying Search Bias CSCW 2017

Quantifying Search Bias: Investigating Sources of Bias for Political Searches in Social Media

Quantifying Search Bias: Investigating Sources of Bias for Political Searches in Social Media Muhammad Bilal Zafar with Juhi Kulshrestha, Motahhare Eslami, Johnnatan Messias, Saptarshi Ghosh, Krishna Gummadi