Natural Language Processing as Key Component to Successful Information Products

Transcription

1 Natural Language Processing as Key Component to Successful Information Products Yves Schabes Teragram Corporation Tera monster in Greek 2 40 (=1,099,511,627,766) (one trillion) gram something written down, drawn, or recorded. Yves Schabes. Teragram. JEITA 10/15/2003 1

2 Natural Language Processing and Document Management Information Overflow Problem. Amount of Information growing at exponential rate. Internet Search Engines:» Year 1998: 10 million pages» Year 2003: 2 billion pages with more than 100 millions queries/day Personal Computing:» Year 1998: Hard-drives of 100 Megabytes-several Gigabytes» Year 2003: Hard-drives of 100s Gigabytes Enterprise Computing» Year 1998: storage 100 Gigabytes - 1 Terabyte» Year 2003: storage 1 Terabyte 100s Terabytes New information is being published continuously. Scalability of NLP Techniques is key. Yves Schabes. Teragram. JEITA 10/15/2003 2

3 Examples of use of NLP for Business Applications in the U.S. Search Internet Search Question-Answering Enterprise search Customer Relationship Management Call Center Knowledgebase Document Management Speech Recognition Automated Call Center (e.g. airline reservation) Dictation Translation Software Localization User Manual Localization Machine Translation Translation Memory Government applications. Interfaces Yves Schabes. Teragram. JEITA 10/15/2003 3

4 Examples of Market Size Internet Search ~$3 billions of 2003 revenues (projected) $5-$8 billions expected for Examples of revenues: Google: $700,000, revenue (projected) Overture: $1,000,000, revenue (projected) Ask Jeeves: $100,000, revenue (projected) Natural Language Technologies are key components of Internet Search Engines. Microsoft is building a web search engine. Internet Search is Generating significant revenues and attracts big players. Yves Schabes. Teragram. JEITA 10/15/2003 4

5 Information Overflow Information Retrieval: Look into existing documents Traditional Solution: Retrieval of Documents Objective: Retrieval of Information» Need for NLP techniques. Information Navigation: Go from one document to another Traditional Solution: Search for Documents Objective: Present Documents and information based on user profiles» Need for NLP techniques. Information Awareness: What is newly published? Traditional Solution: alerts based on keywords Objective: alerts based on user interests.» Need for NLP techniques. Yves Schabes. Teragram. JEITA 10/15/2003 5

6 Why Scalability? Example: Altavista Prisma Topics Related To Search Query To Refine Search Yves Schabes. Teragram. JEITA 10/15/2003 6

7 Example: Altavista Prisma In order to perform the computation needed for Prisma: Preprocessing: Process every week 1 billion web pages and perform the following NLP techniques: 1. Automatic Language Identification 2. Tokenization and Segmentation 3. Morphological Analysis 4. Disambiguation and Part-of-Speech Tagging 5. Concepts and phrase extraction Run-time speed: more than 100 million queries/s Scalability is key! Yves Schabes. Teragram. JEITA 10/15/2003 7

8 Spelling Correction. Example MSN.com Word Processor Speed: 1 user. Small Dictionary (~300,000 words) Internet Search Engine Requirements: Hundreds of millions of queries/day Use of context Use of multiple dictionaries (names, ) Dictionary Size: all words used on the Internet Yves Schabes. Teragram. JEITA 10/15/2003 8

9 Spelling Correction. User mistypes Christina Aguilera s first name as cristina in the search query box and then clicks the Search button. Before the search engine begins its search, Spelling Correction automatically corrects the first name to christina in order to return the correct Web pages. Yves Schabes. Teragram. JEITA 10/15/2003 9

10 Spelling Correction. User has mistyped the name for the query search. The search is made for pages with the mistyped name, but the Spelling Correction suggests a search with the correct spelling of the name. Yves Schabes. Teragram. JEITA 10/15/

11 Contextual Spelling Correction Yves Schabes. Teragram. JEITA 10/15/

12 Contextual Spelling Correction Yves Schabes. Teragram. JEITA 10/15/

13 New York Times Examples Functionalities: Retrieval and Browsing (find existing documents) Search by keywords. Browse documents by topics Search by keyword within Topic Related News/Documents Information Awareness (search in the future for newly available documents) Alert based on Topic and Keywords Yves Schabes. Teragram. JEITA 10/15/

14 Search Search Search Search within within Category Category Browse Browse by by Categories Categories Alerts Alerts Browse Browse by by Categories Categories Yves Schabes. Teragram. JEITA 10/15/

15 Search Query Search Query Categorized Yves Schabes. Teragram. JEITA 10/15/

16 Alerts Related Articles Related Categories Yves Schabes. Teragram. JEITA 10/15/

17 Search Query Categorization Yves Schabes. Teragram. JEITA 10/15/

18 Use of Categorization and Extraction for Alerts Category Concept Yves Schabes. Teragram. JEITA 10/15/

19 Yves Schabes. Teragram. JEITA 10/15/

20 New York Times. NLP Techniques. NLP Techniques: Document Categorization Concepts Extraction Phrase Extraction Part-of-speech tagging Stemming Tokenization Scalability in Precision. Yves Schabes. Teragram. JEITA 10/15/

21 Example: CNN Functionality: Alerts based on concepts and topics Scalability: millions of s per day Yves Schabes. Teragram. JEITA 10/15/

22 Yves Schabes. Teragram. JEITA 10/15/

23 Natural Language Question-Answering Scalability in terms of documents is required. Answers are computed across all documents. Techniques: Semantic Analysis Parsing Events Extraction Concepts Extraction Part-of-Speech Tagging Stemming Yves Schabes. Teragram. JEITA 10/15/

24 Question Documents Documents were were answer answer is is found found Answer 1 Excerpts Answer 2 Answer 3 Yves Schabes. Teragram. JEITA 10/15/

25 Yves Schabes. Teragram. JEITA 10/15/

26 Yves Schabes. Teragram. JEITA 10/15/

27 Conclusion Scalability is a requirement of NLP Techniques. Scalability makes NLP techniques useful for Information Retrieval Information Browsing Information Awareness Yves Schabes. Teragram. JEITA 10/15/