Modeling Information Seeking Behavior in Social Media Eugene Agichtein

Transcription

1 Modeling Information Seeking Behavior in Social Media Eugene Agichtein I lli I f i A L b (IRL b) Intelligent Information Access Lab (IRLab) Emory University

2 Intelligent Information Access Lab (IRLab) Modeling information seeking behavior Searching the Web and social media Text and data mining for medical informatics and public health Ablimit Aji Qi Guo Julia Kiseleva In collaboration with: - Beth Buffalo (Neurology) - Charlie Clarke (Waterloo) - Ernie Garcia (Radiology) - Phil Wolff (Psychology) - Hongyuan Zha (GaTech) Dmitry Lagun Qiaoling Liu Yu Wang 2

3 Our Approach to Intelligent Information Access Search logs: queries, clicks Data-Driven Di Model ldiscovery (machine learning/data mining) Intelligent search Information Health Cognitive sharing Informatics Diagnosticsi 3 3

4 Intelligent search Contextualized Intent Inference 4

5 Intelligent search Web-scale Text Mining Extract entities, relationships, events from text Estimate accuracy of web content DiseaseOutbreaks, The New York Times Some Applications: Incorporating extracted information into (web) search Finding implicit connections between events, entities Visualizing and exploring large text collections 18 November 2009 Eugene Agichtein, Emory 5 University, IR Lab [DL 00, ICDE 2003 best student paper, SIGMOD 2006 best paper, ]

6 Health Informatics Information Extraction for Decision Support with E. V. Garcia (Radiology) and A. Ram (Georgia Tech) Rule Discovery from Medical Literature (MERLIN project): Identify articles containing useful clinical knowledge Extract new expert system rules for the Emory Cardiac Toolbox IF LV_stress_perfusion_is_abnormal THEN Diseased_coronary_is(LAD) Personalized diagnosis i and care (PRETEX project): Extract clinical variables from text in patient records Personalize expert system rules for a given patient or population New: unexpected findings 18 November 2009 Eugene Agichtein, Emory University, IR Lab 6

7 Talk Outline IR Lab research overview Mining interactions in social media Content quality Information seeker satisfaction Question intent If time: inferring web search intent 7

8 Finding Information Online 8

9 From Searching to Finding 9

10 Social (Information) Sharing 10

11 11 11

12 (Text) Social Media Today Published: 4Gb/day Social Media: 10Gb/Day Yahoo Answers: 120M users, 40M questions, 1B answers Yes, we could read your blog. Or, you could tell us about your day 12

13 Finding Information Online (Revisited) Claim: next generation of search will provide support for real-time, mediated info exchange First step: web-scale collaborative question answering (CQA) sites Realistic information exchange 100M+ community Many immediate challenges 13

14

15 15

16 (Some) Related Work Adamic et al., WWW 2007, WWW 2008: Expertise sharing, network structure Elsas et al., SIGIR 2008: Blog search Glance et al.: Blog Pulse, popularity, information sharing Harper et al., CHI 2008, 2009: Answer quality across multiple CQA sites Krautetal: al.: community participation Kumar et al., WWW 2004, KDD 2008, : Information diffusion in blogspace, network evolution Third Workshop on Searching Social Media (SSM 2010) at WSDM: edu/ssm2010/ 16

17 Finding High Quality Content in SM E. Agichtein, C. Castillo, D. Donato, A. Gionis, and G. Mishne, Finding High Quality Content in Social Media, in WSDM 2008 Well-written Interesting Relevant (answer) Factually correct Popular? Provocative? Useful? As judged by professional editors 17

18 18 18

19 How do Question and Answer Quality relate? 19 19

20 20 20

21 21 21

22 22 22

23 23 23

24 Community 24

25 Link Analysis for Authority Estimation User 1 User 2 Question 1 Question 2 Answer 1 User 3 Answer 2 User 4 Answer 3 User 5 Answer 4 User 6 User 3 User 1 User 4 User 5 User 2 User 6 Question 3 Answer 5 Answer 6 A ( j ) = H ( i ) i= 0.. M H ( i) = A( j) j = 0.. K Hub (asker) Authority (answerer) 25

26 HITS effective Qualitative Observations HITS ineffective 26

27 Random forest classifier 27 27

28 Result 1: Identifying High Quality Questions 28

29 Top Features for Question Classification Asker popularity ( stars ) Punctuation density Topical category Page views KL Divergence from reference corpus LM 29

30 Identifying High Quality Answers 30

31 Top Features for Answer Classification Answer length Community ratings Answerer reputation Word overlap Kincaid readability score 31

32 User and Content Quality: Coupled Mutual Reinforcement 32

33 Can Improve Performance OR Reduce Training 33

34 Finding Information Online (Revisited) Next generation of search: human-machine-human CQA: a case study in complex IR Content quality Asker satisfaction Understanding the interactions 34

35 Dimensions of Quality Well-written Interesting Relevant (answer) Factually correct Popular? Timely? Provocative? Useful? As judged by the asker (or community) 35 35

36 Are Editor Labels Meaningful for CGC? Information seeking process: want to find useful information about topic with incomplete knowledge N. Belkin: Anomalous states of knowledge Want to model directly if user found satisfactory information Specific (amenable) case: CQA 36

37 Yahoo! Answers: The Good News Active community of millions of users in many countries and languages Effective for subjective information needs Great forum for socialization/chat Can be invaluable for hard-to-find information not available on the web 37

38 38

39 Yahoo! Answers: The Bad News May have to wait a long time to get a satisfactory answer 1. FIFA World Cup Optical Poetry Football (American) Soccer Medicine WinterSports 5 8. Special Education 0 9. General Health Care Outdoor Recreation Time to close a question (hours) May never obtain a satisfying i answer 39

40 Predicting Asker Satisfaction Y. Liu, J. Bian, and E. Agichtein, in SIGIR 2008 Yandong Liu Jiang Bian Given a question submitted by an asker in CQA, predict whether the user will be satisfied with the answers contributed by the community. Satisfied : The asker has closed the question AND Selected the best answer AND Rated best answer >= 3 stars (# not important) Else, Unsatisfied 40

41 ASP: Asker Satisfaction Prediction Question Answer Asker History Answerer History Category Text Classifier Wikipedia News asker is satisfied asker is not satisfied 41 41

42 Experimental Setup: Data Crawled from Yahoo! Answers in early 2008 Questions Answers Askers Categories % Satisfied 216,170 1,963, , % Anonymized dataset available at: 1/2009: Yahoo! Webscope : Comprehensive Answers dataset: ~5M questions & answers

43 Satisfaction by Topic Topic Questions Answers A per Q Satisfied Asker rating 2006 FIFA World Cup Mental Health Time to close by asker , % minutes % days Mathematics % minutes Diet & % days Fitness 43

44 Satisfaction Prediction: Human Judges Truth: asker s rating A random sample of 130 questions Researchers Agreement: 0.82 F1: P*R/(P+R) R) Amazon Mechanical Turk Five workers per question. Agreement: 0.9 F1: 0.61 Best when at least 4 out of 5 raters agree 44 44

45 Performance: ASP vs. Humans (F1, Satisfied) Classifier With Text Without Text Selected Features ASP_SVMSVM ASP_C ASP_RandomForest ASP_Boosting ASP_NB Best Human Perf 0.61 Baseline (random) 0.66 Human F1 is lower than the random baseline! ASP is significantly more effective than humans 45

46 Top Features by Information Gain Q: Askers previous rating Q: Average past rating by asker UH: Member since (interval) UH: Average # answers for by past Q UH: Previous Q resolved for the asker CA: Average asker rating for category UH: Total number of answers received 46

47 Offline vs. Online Prediction Offline prediction (AFTER answers arrive) All features( question, answer, asker & category) F1: 0.77 Online prediction (BEFORE question posted) NO answer features Only asker history and question features (stars, #comments, sum of votes ) F1:

48 Personalized Prediction of Satisfaction Y. Liu and E. Agichtein, You've Got Answers: Personalized Models for Predicting Success in Community Question Answering, ACL 2008 Same information!= same usefulness for different searchers! Personalization vs. Groupization? 48

49 Example Personalized Models 49

50 Outline Next generation of search: Algorithmically mediated information exchange CQA: a case study in complex IR Content quality Asker satisfaction Understanding the interactions 50

51 Social Media Language Analysis Social Media!= WSJ Text Subjectivity, Sentiment, Temporal Sensitivity 51

52 Subjectivity in CQA B. Li, Y. Liu, and E. Agichtein, CoCQA: Co-Training Over Questions and Answers with an Application to Predicting Question Subjectivity Orientation, EMNLP 2008 How can we exploit structure of CQA for categorization of social media content? Case Study: Text Subjectivity Subjective: Has anyone got one of those home blood pressure monitors? and if so what make is it and do you think they are worth getting? Objective: What is the difference between chemotherapy and radiation treatments? 52 52

53 Objective vs. Subjective Content in CQA Education 30% Arts Science 36% 70% 48% 52% 64% Health 34% Objective Subjective 21% Sports 36% 64% 66% 79% 53

54 54 54

55 Questions and Answers: Two Views Example: Q: Has anyone got one of those home blood pressure monitors? and if so what make is it and do you think they are worth getting? A: My mom has one as she is diabetic so its important for her to monitor it she finds it useful. Answer orientation usually matches question Idea: Co-Training (Blum & Mitchell, COLT 1998) 55 55

56 CoCQA: A Co-Training Framework over Questions and Answers EA1 Labeled Data Q A CQ C CA Class ify Q A Unlabeled Unlabeled Data Data???????????????????????????????????????? Stop Validation (Holdout training data) 56 56

57 Slide 56 EA1 Include one more box on lower right corner: after "stop" lights up, show box "apply final classifier on test data" Eugene Agichtein, 10/26/2008

58 Features Method Example result: CoCQA Outperforms State-ofthe-Art Partially Supervised ML Question (macro avg F1) Question+ Best Answer (macro avg F1) Supervised GE (-0.7%) (+3.2%) CoCQA (+1.9%) (+7.2%) Implications: Can reduce amount of required manual labels Can improve accuracy with more unlabeled data 57

59 Another Example: Question Urgency [Liu et al., SIGIR 2009 poster] Problem a growing volume of questions competing for visibility Urgent questions pushed out Delayed responses useless 58

60 Outline Next generation of search: Algorithmically mediated information exchange CQA: a case study in complex IR Content quality Asker satisfaction Understanding interactions 59

61 Current Work (in Progress) Partially supervised models of expertise (Bian et al., WWW 2009) Sentiment, temporal sensitivity i i analysis Influence of text on interactions Towards real-time hybrid social/web search 60

62 Intelligent search Goal: Hybrid Web/Social Search 61

63 Takeaways Robust machine learning over interaction data system improvements, insights into behavior Contextualized models for NLP and text mining system improvements, insights into interactions Mining social media:potentialfortransformative transformative impact for IR, sociology, psychology, medical informatics, public health, 62

64 More information, datasets, papers, slides: References Modeling search intent [SIGIR 06, 07, ECIR 09, WI 09] Estimating content quality [WSDM 2008] Estimating contributor authority [CIKM 2007] Searching CQA archives [WWW 2008, WWW 2009] Inferring asker intent [EMNLP 2008, SIGIR 09 poster] Predicting satisfaction [SIGIR 2008, ACL 2008, TKDE 09] Coping with spam in CQA [AIRWeb 2008] 63

65 Thank you! Diane Kelly and UNC for hosting my visit Supported by: 64