Quantifying Search Bias: Investigating Sources of Bias for Political Searches in Social Media

Size: px

Start display at page:

Download "Quantifying Search Bias: Investigating Sources of Bias for Political Searches in Social Media"

Kristian Quinn
5 years ago
Views:

1 Quantifying Search Bias: Investigating Sources of Bias for Political Searches in Social Media Muhammad Bilal Zafar with Juhi Kulshrestha, Motahhare Eslami, Johnnatan Messias, Saptarshi Ghosh, Krishna Gummadi and Karrie Karahalios

2 Social media as search platform Two-thirds of Americans on social media use it to get news (informational queries)

3 Social media as search platform Two-thirds of Americans on social media use it to get news (informational queries)

4 Social media as search platform Two-thirds of Americans on social media use it to get news (informational queries) Ranked list (importance)

5 Potential bias in search results

6 Potential bias in search results

7 Search rankings can shape user opinion Users place greater trust in higher ranked items [Pan et al., 2007] Biased search results can influence voting patterns [Epstein & Robertson, 2015] This talk: Study sources of bias in social media search

8 This talk: Study sources of bias in social media search Identifying sources of search bias Quantifying bias in each source Studying bias in political searches in Twitter

9 This talk: Study sources of bias in social media search Identifying sources of search bias Quantifying bias in each source Studying bias in political searches in Twitter

10 Social media search engine design Data corpus

11 Social media search engine design Data corpus q i1 i2 i3... (set) input (relevant items)

12 Social media search engine design Data corpus q i1 i2 i3... Search engine algo. (set) input (relevant items) ranking

13 Social media search engine design Data corpus q i1 i2 i3... (set) Search engine algo. i5 i1 i3... input (relevant items) ranking output (ranked list)

14 Social media search engine design Data corpus q i1 i2 i3... (set) Search engine algo. i5 i1 i3... input (relevant items) ranking output (ranked list) Output bias may stem from - Bias in the (input) relevant item set

15 Social media search engine design Data corpus q i1 i2 i3... (set) Search engine algo. i5 i1 i3... input (relevant items) ranking output (ranked list) Output bias may stem from - Bias in the (input) relevant item set - Bias introduced by the ranking algorithm

16 This talk: Study sources of bias in social media search Identifying sources of search bias Quantifying bias in each source Studying bias in political searches in Twitter

17 Quantifying bias in each source Data corpus q i1 i2 i3... (set) Search engine algo. i5 i1 i3... input (relevant items) ranking output (ranked list) Quantify bias of each individual item

18 Quantifying bias of individual items

19 Quantifying bias of individual items Our method: Infer bias of each individual item from the bias of the author

20 Quantifying bias of individual items Our method: Infer bias of each individual item from the bias of the author

21 Quantifying bias of individual items Our method: Infer bias of each individual item from the bias of the author "...conservative political commentator..."

22 Quantifying bias of individual items Our method: Infer bias of each individual item from the bias of the author 75% accuracy Highly scalable, high coverage Other measures can be used [Zafar et al., 2016]

23 Quantifying bias in inputs (relevant item set) i1 i2 i3... (set) Compute bias for each item Take the average over the whole set input (relevant items)

24 Quantifying bias in outputs (ranked list) MAP-style measure i5 i1 i3... output (ranked list) Bias till rank r Output bias

25 Quantifying bias in ranking algorithm Search engine algo. Ranking bias = Output bias - Input bias ranking

26 This talk: Study sources of bias in social media search Identifying sources of search bias Quantifying bias in each source Studying bias in political searches in Twitter

27 Studying bias in political searches in Twitter Search queries for Democratic and Republican debates (dem debate, #demdebate,...) - Presidential candidates (Hillary Clinton, Donald Trump,...)

28 Key takeaways It is not just the algorithm, input data also matters

29 High ranking bias Query Input Bias Ranking bias Hillary Clinton Ted Cruz

30 High input bias Query Input Bias Ranking bias Bernie Sanders Donald Trump

31 Key takeaways It is not just the algorithm, input data also matters Analyzing the input bias - Search topic matters

32 Effect of search topic Query Input Bias Hillary Clinton (D) 0.03 Bernie Sanders (D) 0.55 Martin O'Malley (D) 0.64 Donald Trump (R) 0.19 Ted Cruz (R) Marco Rubio (R) -0.12

33 Effect of search topic Query Input Bias Hillary Clinton (D) 0.03 Bernie Sanders (D) 0.55 Martin O'Malley (D) 0.64 Donald Trump (R) 0.19 Ted Cruz (R) Marco Rubio (R) -0.12

34 Key takeaways It is not just the algorithm, input data also matters Analyzing the input bias - Search topic matters - Query phrasing can make a big difference

35 Query phrasing and search bias Query Input Bias democratic debate 0.38 #democraticdebate 0.19 #demdebate 0.56

36 Query phrasing and search bias Query Input Bias democratic debate 0.38 #democraticdebate 0.19 #demdebate 0.56

37 Query phrasing and search bias Query Input Bias republican debate 0.27 #gopdebate 0.10 rep debate 0.40 Even for the same topic, query phrasing can greatly affect the input bias

Query phrasing and search bias -1 0 1 Query Input Bias republican debate 0.27 #gopdebate 0.

38 Query phrasing and search bias Query Input Bias republican debate 0.27 #gopdebate 0.10 rep debate 0.40 Even for the same topic, query phrasing can greatly affect the input bias

39 Future work: Addressing search bias Alter the ranking algorithm - Might lead to reduction in quality of results or lead to unforeseen biases Make the bias transparent - Keep the current ranking - Inform the users about what they are seeing - Make biases related to query phrasing transparent

40 Questions Paper at:

Quantifying Search Bias: Investigating Sources of Bias for Political Searches in Social Media

Quantifying Search Bias: Investigating Sources of Bias for Political Searches in Social Media Juhi Kulshrestha joint work with Motahhare Eslami, Johnnatan Messias, Muhammad Bilal Zafar, Saptarshi Ghosh,