cocobean A Socially Ranked Question Answering System ethan deyoung jerry ye jimmy chen Srinivasan Ramaswamy advisor: marti hearst

Transcription

1 cocobean A Socially Ranked Answering System ethan deyoung jerry ye jimmy chen Srinivasan Ramaswamy advisor: marti hearst

2 introduction Traditional Keyword Search keywords treated independently no semantic meaning instant results

3 introduction Answering Systems highly specific domain hard problem human dependent latency

4 introduction Social Ranking Websites users pick what is good reorders results

5 introduction cocobean simple NLP processing instant results users votes improve ordering

6 a quick example Ask

7 a quick example Vote

8 a quick example Improve

9 a quick example Improve

10 how we did it User Interface Design System Architecture Ranking

11 ui design

12 usability study Initial Interviews Personas Jason Linda Lee Lurker Read but never contributes Casual User Participates and contributes occasionally Hard Core User Active Participant

13 usability study Initial Interviews Personas Jason Linda Lee Lurker Read but never contributes Casual User Participates and contributes occasionally Hard Core User Active Participant

14 usability study Initial Interviews Personas Jason Linda Lee Lurker Read but never contributes Casual User Participates and contributes occasionally Hard Core User Active Participant

15 usability study Initial Interviews Personas Jason Linda Lee Lurker Read but never contributes Casual User Participates and contributes occasionally Hard Core User Active Participant

16 usability study Initial Interviews Personas Jason Primary Persona Linda Lee Lurker Read but never contributes Casual User Participates and contributes occasionally Hard Core User Active Participant

17 usability study Low-fi Testing blocking two voting interfaces Results thumbs up/down up/down/remove down arrow/remove confusing thumbs up/down more natural

18 usability study before

19 usability study after

20 system architecture

21 system architecture components wikipedia corpus lucene search engine question transformation question classification ranking rendering

22 system architecture introduction ui design system architecture ranking

23 system architecture introduction ui design system architecture ranking

24 system architecture Who is the president of France? Transformation

25 system architecture Who is the president of France? Transformation Classification

26 system architecture Who is the president of France? Transformation Classification

27 system architecture Wikipedia The president of France is Who is the president of France? Transformation, the president of France is the president of France Lucene Classification introduction ui design system architecture ranking

28 system architecture Wikipedia The president of France is Who is the president of France? Transformation, the president of France is the president of France Lucene Classification introduction ui design system architecture ranking

29 system architecture Wikipedia The president of France is Who is the president of France? Transformation, the president of France is the president of France Lucene Classification Results: IR Score, title, snippet Re-ranking introduction ui design system architecture ranking

30 system architecture Wikipedia The president of France is Who is the president of France? Transformation, the president of France is the president of France Lucene Classification class (Human) Results: IR Score, title, snippet Re-ranking introduction ui design system architecture ranking

31 system architecture Wikipedia The president of France is Who is the president of France? Transformation, the president of France is the president of France Lucene Classification class (Human) Results: IR Score, title, snippet Re-ranking introduction ui design system architecture ranking

32 system architecture Wikipedia The president of France is Who is the president of France? Transformation, the president of France is the president of France Lucene Classification class (Human) Results: IR Score, title, snippet Re-ranking Ranking Database fetch data: votes, reputation introduction ui design system architecture ranking

33 system architecture Wikipedia The president of France is Who is the president of France? Transformation, the president of France is the president of France Lucene Classification class (Human) Results: IR Score, title, snippet Re-ranking Ranking Database fetch data: votes, reputation introduction ui design system architecture ranking

34 system architecture Wikipedia The president of France is Who is the president of France? Transformation, the president of France is the president of France Lucene Classification class (Human) Results: IR Score, title, snippet Rendering reordered results Re-ranking Ranking Database fetch data: votes, reputation introduction ui design system architecture ranking

35 system architecture Wikipedia The president of France is Who is the president of France? Transformation, the president of France is the president of France Lucene Classification class (Human) Results: IR Score, title, snippet Rendering reordered results Re-ranking Ranking Database fetch data: votes, reputation introduction ui design system architecture ranking

36 system architecture Wikipedia The president of France is Who is the president of France? Transformation, the president of France is the president of France Lucene Classification class (Human) Results: IR Score, title, snippet cleaned and highlighted results Rendering reordered results Re-ranking Ranking Database fetch data: votes, reputation introduction ui design system architecture ranking

37 system architecture Wikipedia The president of France is Who is the president of France? Transformation, the president of France is the president of France Lucene Classification class (Human) Results: IR Score, title, snippet cleaned and highlighted results Rendering reordered results Re-ranking Ranking Database fetch data: votes, reputation introduction ui design system architecture ranking

38 system architecture Wikipedia The president of France is Who is the president of France? Transformation, the president of France is the president of France Lucene Classification class (Human) Results: IR Score, title, snippet cleaned and highlighted results Rendering reordered results Re-ranking user votes Ranking Database fetch data: votes, reputation introduction ui design system architecture ranking

39 system architecture Wikipedia The president of France is Who is the president of France? Transformation, the president of France is the president of France Lucene Classification class (Human) Results: IR Score, title, snippet cleaned and highlighted results Rendering reordered results Re-ranking user votes updated votes displayed to user Ranking Database fetch data: votes, reputation introduction ui design system architecture ranking

40 ranking

41 features Images Votes Up Votes Down User Reputation Lucene IR Relevancy Type = Answer Type Rule Type

42 data set Gold Standard Relevancy score for each question/answer pair Mechanical Turk Pay people to vote

43 training Learn patterns from data Learn importance of each feature Support Vector Machine for Regression

44 evaluation NDCG = Z k i=1 score(i) log 2 log(i + 1) default random no votes with votes best NDCG Normalized Discounted Cumulative Gain Document ranking ordered by SVM score() provided by golden set DCG improved in tests

45 demo

46 results What Works What Doesn t

47 questions Answers