信息检索与搜索引擎 Introduction to Information Retrieval GESC1007

Transcription

1 信息检索与搜索引擎 Introduction to Information Retrieval GESC1007 Philippe Fournier-Viger Full professor School of Natural Sciences and Humanities Spring

2 Introduction Philippe Fournier-Viger Full professor ( 教授 ) at HIT since 2015 Previously: professor at University of Moncton (Canada / 加拿大 ) Ph.D. in Computer Science (Canada) My specialty is data mining ( 数据挖掘 / 大数据 ) I am fluent in English and French, and can speak some basic Chinese. 2

3 How to contact with me? We can discuss immediately after lectures ( 电子邮件 ): philfv8@yahoo.com I check my s once a day Office: G421 Office phone:

4 Teaching assistants There will be 2 teaching assistant(s) for this course: to grade homework, to check the attendance, help with other issues related to the course, QQ group:

5 About this course 16 hours (8 lectures) 1 credit, general course ( 本科专业课程, 通识课程 ) no requirements Evaluation: Attendance ( 上课 ): 10 % Course work (30%: two assignments 两个作业 ) Final exam (60% 考试 ) 5

6 Main objective ( 目的 ) Goal:understand how information retrieval systems work. For example: Web search engines (Baidu, Bing ) ( 搜索引擎 ), Text document retrieval systems ( 文本检索系统 ). 6

7 Specific goals What is an information retrieval system ( 信息检索系统 )? Why are they used? How they work? An introduction to information retrieval techniques: Boolean retrieval, dictionary-based retrieval, index construction and compression, scoring, the vector space model, computing scores in a search system, 7

8 Specific objectives computing scores in a search system, query processing, language models, text, vector space classification, clustering, web search including crawling, indexes and link analysis 8

9 Lecture 1 (today - 今天 ) Lecture 2 Course schedule ( 日程安排 ) Introduction Boolean retrieval Term vocabulary and posting lists Lecture 3 Lecture 4 Lecture 5 Lecture 6 Dictionaries and tolerant retrieval Index construction and compression Scoring, weighting, and the vector space model Computer scores, and a complete search system Lecture 7 Lecture 8 Evaluation in information retrieval Web search engines, advanced topics, and conclusion 9

10 Lecture slides (PPTs) I will provide detailed lecture slides (PPTs) I use this book to prepare the course: Manning, C. D., Raghavan, P., Schütze, H. Introduction to information retrieval. Cambridge: Cambridge University Press, 2008 Note: It is not necessary to buy the book! ( 没有必要买这本书 ) 10

11 Course website Slides, assignments, grades, and the updated calendar will be provided on the course website: 11

12 12

13 13

14 14

15 15

16 16

17 17

18 Rules 1. Students must attend all lectures. 2. Students must arrive on time. 3. Homework must be submitted before the deadline. No late homework will be accepted. 18

19 Rules 4. Plagiarism ( 抄袭 ) will not be tolerated. 5. Phones needs to be turned off during lectures 6. One should not talk or eat in the classroom, or do anything that may disturb other classmates. 19

20 Lectures Each lecture starts at 13:45 Each lecture ends at 15:30 We will take a 5 minutes break around 14:40. 20

21 CHAPTER 1 INTRODUCTION Picture from: 21

22 Introduction What is Information Retrieval (IR) 信息检索? Information 信息 Any kind of Information or data Retrieve (verb) 检索 To search, obtain, look up for.. 22

23 Introduction Reading your student number on your student card ( 学生证 ). Is it information retrieval? We will use a more restrictive definition of information retrieval 23

24 Introduction We will consider information retrieval as searching for documents. User ( 用户 ) Query ( 查询 ) Database ( 数据库 ) olympics Relevant documents Search for documents Documents 24

25 Information retrieval: finding documents of an unstructured nature (e.g. text) that satisfies some requirements. User ( 用户 ) Query ( 查询 ) Database ( 数据库 ) olympics Relevant documents Search for documents Documents 25

26 We assume that there is a large collection of documents (generally, stored on computers). User ( 用户 ) Database ( 数据库 ) Documents 26

27 In the past: information retrieval was mostly done by librarians ( 图书馆员 ) and other professional searchers. 27

28 Today: Billions of people performs information retrieval using computers and cellphones Searching for webpages ( 网页 ), Searching s 28

29 Today More and more data is collected and stored in computers (this is the big data era - 大数据时代 ). Why? Storing data is cheaper Networks are fast As a result, information retrieval is becoming the most important way of accessing information. 29

30 Types of documents Structured data ( 结构化数据 ): documents that have a clear and well-defined structure (e.g. data about bank customers) Name = P. Black Country = Canada Age = 20 Name = X. Zhang Country = China Age = 25 Searching structured data is easy ( 容易 ) 30

31 Types of documents Unstructured data ( 非结构化数据 ): documents that have no clear structure, or are not organized in a predefined manner. (e.g. text documents) Searching unstructured data is hard ( 难 ) Information retrieval is mostly about searching unstructured data. 31

32 Information retrieval is also To let users browse documents ( 浏览文件 ) To let users filter documents ( 过滤文件 ). To automatically group together books on similar topics (clustering - 聚类 ) To classify books into some categories (classification - 分类 ) To rank documents by relevance ( 相关排序 ) 32

33 Information retrieval at a large scale ( 大规模 ) Web search (very large scale) billions of documents on millions of computers Important to search quickly A challenge: some websites try to cheat ( 欺骗 ) to appear more often in search engine results 33

34 Information retrieval at a small scale ( 小规模 ) Personal search Search files using an operating systems ( 操作系统 ) such as Windows or Android Need to be fast Need to handle many different types of files Search s. service providers usually classify s into categories such as spam ( 垃圾邮件 ) and relevant e- mails. Need to be reliable 34

35 Information retrieval at a medium scale Enterprise ( 企业 ) search Search documents of an enterprise database of patents ( 专利 ) database of research papers Documents are typically stored in a centralized database. Computers are used to search the databases. 35

36 1.1 An example We have a set of books: We want to find all books containing: Beijing AND Shenzhen AND NOT Guangzhou How to search? 36

37 How to search? Naive approach ( 幼稚的方法 ): Read all books word by word to find the books containing Beijing and Shenzhen and not Guangzhou. For a human: this is very slow! For a computer: it is fast if there is not too many books Web: billions of documents. 37

38 Search quickly, We want to: Search using flexible matching operations ( 灵活的匹配 ) e.g. search for all documents with Shenzhen «whithin 5 words» of Beijing rank the results by relevance ( 相关性排序 ): we want to find the best answer(s) to queries made by users 38

39 How to search quickly? We can create an index ( 索引 ) of the documents. The index is created in advance. We search for documents using the index rather than reading the documents word by word. similar to how the table of content ( 目录 ) of a book helps us to find information in the book An example of index 39

40 An example of index ( 索引 ) A term-document incidence matrix ( 关联矩阵 ) can be used as index for searching documents. Terms Beijing Shenzhen Guangzhou ooks Book Book Book Book Book = the term appears in the document 0 = it does no appear 40

41 Example 1: Search books with Beijing AND Shenzhen Terms Beijing Shenzhen Guangzhou Book ooks Book Book Book Book The result of this query is: { Book 1, Book 5} 41

42 Example 2: Search books with Beijing AND NOT Shenzhen Terms Beijing Shenzhen Guangzhou Book ooks Book Book Book Book The result of this query is: { Book 2, Book 3} 42

43 Example 3: Search books with Beijing OR Shenzhen Terms Beijing Shenzhen Guangzhou Book ooks Book Book Book Book The result of this query is: { Book 1,2,3,4,5} 43

44 A term-document incidence matrix is said to be a boolean matrix ( 布尔矩阵 ) (a table containing 0 and 1). Terms Beijing Shenzhen Guangzhou Book ooks Book Book Book Book

45 SOME VOCABULARY 45

46 Boolean retrieval model ( 布尔检索模型 ) We can search for documents using Boolean expressions ( 布尔表达式 ). A Boolean expression is one or more terms combined with operators AND, OR and NOT. Example: Beijing AND Shenzhen AND NOT Guangzhou 46

47 Collection A collection ( 收集 ) is a set of documents that we may want to search. Example: N = 1 million documents Each document about 1000 words (2-3 pages) M = 500,000 distinct terms in these documents Size: 6 GB 47

48 Information need, query Information need ( 信息需求 ): It is the topic about which the user wants to know more (e.g. I want to know about Beijing Olympics) Query ( 查询 ): It is what the user enter in a computer to search for documents. (e.g. «Beijing AND Olympics AND 2008») 48

49 Relevance ( 关联 ) A document is relevant if it contains information that is valuable given the information need of a person. e.g. a user wants to find documents about outdoor activities in Shenzhen. Challenge: relevant documents may or may not contains these words. 49

50 Information retrieval system ( 信息检索系统 ) An IR system is a software or website that can be used to search for documents. e.g. 50

51 How to evaluate an IR system? A good IR system is an IR system that can provide relevant documents to users. But how can we measure the relevance of documents? Two evaluation measures 51

52 Precision ( 准确率 ) Precision: What fraction of the returned documents are relevant to the information need? Example: A person searches for webpages about Beijing The search engine returns: 5 relevant webpages 5 irrelevant webpages. Precision = 5 / 10 = 0.5 (50 %) 52

53 Recall ( 召回 ) Recall: What fraction of the relevant documents in a collection were returned by the system? Example: A database contains 1000 documents about HITSZ. The user search for documents about HITSZ. Only 100 documents about HITSZ are retrieved. Recall = 100 / 1000 = 0.1 (10 %) 53

54 Dictionary ( 词典 ) A dictionary is set of terms. Each term is associated to a list of documents where it appears. Each entry of a list is called a posting. Example: Dictionary City Shenzhen Located China Book1, Book2, Book 20, Book 7. Book1, Book3 Book1, Book 20, A posting Postings 54

55 Dictionary A dictionary is usually much smaller than a matrix because not all words appear in all documents. Matrix City Shenzhen Word 500,000 Book Book Book Book books 500, 000 words = a huge matrix of 500,000,000 entries!! 55

56 How to create an index? Step 1: collect the documents to be indexed Book1 Book2 Book3 Book100

57 How to create an index? Step 1: collect the documents to be indexed Book1 Book2 Book3 Book100 Step 2: tokenize the text ( 标记文本 ): separate it into words Book1 «The city of Shenzhen is located in China» token1 token2 token7 token8 57

58 How to create an index? Step 3: Linguistic preprocessing ( 语言的预处理 ) Keep only the terms that are useful for indexing documents. «The city of Shenzhen is located in China» token1 token2 token7 token8 During that step, words can be also transformed if necessary: friends friend wolves wolf eaten eat 58

59 How to create an index? Step 4: Create the dictionary City Shenzhen Located China Dictionary City Shenzhen Located China Book1, Book2, Book 20, Book 7. Book1, Book3, Book 5, Book 9. Book1, Book 20, Book 34

60 How to create an index? The index has been created! It can then be used to search documents. Dictionary City Shenzhen Located China Book1, Book2, Book 20, Book 7. Book1, Book3, Book 5, Book 9. Book1, Book 20, Book 34

61 Document ID ( 标识符 ) An IR system will generally assign a unique identifier (ID) to each document. Book 1, Book 2, Book 3. 61

62 Dictionary (cont d) A dictionary is typically sorted to allow faster search (e.g. in alphabetical order - 按字母顺序 ) Dictionary City Shenzhen Located China Book1, Book2, Book 20, Book 7. Book1, Book3, Book 5, Book 9. Book1, Book 20, Book 34 62

63 Dictionary (cont d) A dictionary is typically sorted to allow faster search (e.g. in alphabetical order - 按字母顺序 ) Dictionary China City Located Shenzhen Book1, Book 20, Book 34 Book1, Book2, Book 7, Book 20. Book1, Book3, Book 5, Book 9. 63

64 A dictionary may be used to store statistics about documents For example: Number of documents that contains each term (word) 64

65 How an IR system answers boolean queries? Dictionary China City Located Shenzhen Book1, Book 20, Book 34 Book1, Book2, Book 7, Book 20. Book1, Book3, Book 5, Book 9. 65

66 How an IR system answers boolean queries? Dictionary China City Located Shenzhen Book1, Book 20, Book 34 Book1, Book2, Book 7, Book 20. Book1, Book3, Book 5, Book 9. The query is: CITY AND CHINA 1) Locate CITY in the dictionary 66

67 How an IR system answers boolean queries? Dictionary China City Located Shenzhen Book1, Book 20, Book 34 Book1, Book2, Book 7, Book 20. Book1, Book3, Book 5, Book 9. The query is: CITY AND CHINA 1) Locate CITY in the dictionary 2) Retrieve its postings 67

68 How an IR system answers boolean queries? Dictionary China City Located Shenzhen Book1, Book 20, Book 34 Book1, Book2, Book 7, Book 20. Book1, Book3, Book 5, Book 9. The query is: CITY AND CHINA 1) Locate CITY in the dictionary 2) Retrieve its postings 3) Locate CHINA in the dictionary 68

69 How an IR system answers boolean queries? Dictionary China City Located Shenzhen Book1, Book 20, Book 34 Book1, Book2, Book 7, Book 20. Book1, Book3, Book 5, Book 9. The query is: CITY AND CHINA 1) Locate CITY in the dictionary 2) Retrieve its postings 3) Locate CHINA in the dictionary 4) Retrieve its postings 69

70 How an IR system answers boolean queries? Dictionary China City Located Shenzhen Book1, Book 20, Book 34 Book1, Book2, Book 7, Book 20. Book1, Book3, Book 5, Book 9. The query is: CITY AND CHINA RESULT: 1) Locate CITY in the dictionary Book 1, Book20 2) Retrieve its postings 3) Locate CHINA in the dictionary 4) Retrieve its postings 5) Do the intersection ( 交线 ) of the two lists 70

71 Answering more complex queries It is also possible to answer more complex queries using a similar approach: (Shenzhen OR CHINA) AND BEIJING) (Shenzhen AND CHINA) AND BEIJING) 71

72 Conclusion I presented the «Boolean retrieval model» where we search documents using terms and Boolean operators (e.g. AND, OR, NOT) This model is popular. e.g. often used to search for books in libraries. Why using this model? A document either match a query or does not. Thus, we can precisely search for documents. We will see extensions of this model with more features 72

73 Conclusion Before next week, please make sure that you can access the course website 73

74 EXERCISES 74

75 An exercise This is an exercise that you can do at home if you want to review what we have learnt in the Chapter 1 of the book. b. Draw the dictionary (also called inverted index representation) for this collection c. What are the returned result for these queries? - schizophrenia AND drug - for AND NOT (drug OR approach) 75

76 References Manning, C. D., Raghavan, P., Schütze, H. Introduction to information retrieval. Cambridge: Cambridge University Press,