Entity Extraction Enables Discovery

Transcription

1 Entity Extraction Enables Discovery A discovery search is one in which you don t know or can t know all relevant search terms. Automated entity extraction lets you discover what you don t know. by Steve Cohen, EVP/COO

2 Pg. 1 ABOUT BASIS TECHNOLOGY Basis Technology provides software solutions for text analytics, information retrieval, digital forensics, and identity resolution in over forty languages. Our Rosette linguistics platform is a widely used suite of interoperable components that power search, business intelligence, e-discovery, social media monitoring, financial compliance, and other enterprise applications. Our linguistics team is at the forefront of applied natural language processing using a combination of statistical modeling, expert rules, and corpus-derived data. Our forensics team pioneers better, faster, and cheaper techniques to extract forensic evidence, keeping government and law enforcement ahead of exponential growth of data storage volumes. Software vendors, content providers, financial institutions, and government agencies worldwide rely on Basis Technology s solutions for Unicode compliance, language identification, multilingual search, entity extraction, name indexing, and name translation. Our products and services are used by over 250 major firms, including Cisco, EMC, Exalead/Dassault Systems, Hewlett-Packard, Microsoft, Oracle, and Symantec. Our text analysis products are widely used in the U.S. defense and intelligence industry by such firms as CACI, Lockheed Martin, Northrop Grumman, SAIC, and SRI. We are the top provider of multilingual technology to web and e-commerce search engines, including Amazon.com, Bing, Google, and Yahoo!. Company headquarters are in Cambridge, Massachusetts, with branch offices in San Francisco, Washington, London, and Tokyo. For more information, visit Basis Technology Corporation. Basis Technology Corporation, Rosette and Highlight are registered trademarks of Basis Technology Corporation. Big Text Analytics is a trademark of Basis Technology Corporation. All other trademarks, service marks, and logos used in this document are the property of their respective owners. ( WPREX)

3 Pg. 2 Consider three scenarios: 1. A real estate agent wants to search Craig s List for the prices of all two-bedroom/two-bathroom condominiums in Boston with over 1200 square feet and indoor parking. 2. An intelligence analyst wants to search global newspapers for the names, locations, and dates associated with a certain diplomat s visit to the U.S. 3. An industrial designer for a leading automaker wants to search through thousands of customer s received last year for all references to the color of the company s hot-selling new model. The advantage of such searches is clear. Yet each is impossible or impractical with today s web and enterprise search engines, because conventional search technology does not enable discovery search. What is discovery search? It is a search in which you are looking for something you cannot fully specify until you find it. You don t know all the search terms that will return all instances of the desired information. Even if you did know all of these terms, you probably would not have time to type them all into a search box. Take the real estate scenario. There are countless ways sellers describe the attributes of properties, yet a person reading these listings usually has no problem parsing the sentences and extracting the key entities of interest price, square footage, amenities, and so on. Whether bathroom is spelled out, abbreviated, shortened to bath, or preceded by a number or a word, the meaning is obvious to the buyer. But it s impossible for anyone to know, never mind manually type, all of the possible search-term combinations that would find every qualified apartment regardless of how the classified ads were written. The same difficulty applies to the other two scenarios. How would you specify a search for all the places a diplomat might have visited? Or how could you search for every possible occurrence of a color especially since new words for colors are invented all the time? You can conduct a discovery search by using a technology called entity extraction. Entity extraction automatically locates important search terms for you based on the same contextual cues people would use. The ability to locate these entities, or units of meaning, in any text is what enables discovery search to find all the information that fits a definition, even when you don t know all of the search terms that might fit the definition. This ability is something that the current generation of search technologies does not do.

4 Pg. 3 TWO-DIMENSIONAL SEARCH First generation search tools look for information in two ways: by word matching and by link analysis. Mid 1990s: Alta Vista & Keyword Lookup The earliest search tools, of which Alta Vista is the model example, looked for keywords in documents that match the search terms the user typed into the search box. In these systems, relevance is based on the document s internal content attributes, including the document title, the number of times the sought-after string appears in the document, how early in the document the terms appear, and the proximity of words to each other in a multi-word search term. Late 1990s: Google & Social Context Rather than just look at a document in isolation to determine search relevance, Google and most other web search engines today also look at how many other documents link to it. If more people link to a document, the document must be more important, and hence more relevant to most searches. Late 1990s-Early 2000s: Ask Jeeves & Topic Communities Ask.com (formerly Ask Jeeves) was the first to apply machine language processing to semantic content specifically when users pose questions. Where other search engines ignored words like Why or What because they appear too often to have relevance anywhere, Ask regards such words as question markers. Another technique Ask uses is dynamic topic clustering. This technique weighs both the raw number of inbound links, like Google, and the number of links from sites with a high degree of relevant subject content that also link to each other. If a site is highly referenced within a topic community, it is probably better suited to answer a question on that topic. DISCOVERY: THE NEXT GENERATION IN SEARCH None of these methods, however, let you search on meaning rather than on words. If you were looking for information on red cars, for example, even the technology that powers Ask.com would only return answers from sources explicitly including the word red. It would not return answers about burgundy or maroon unless those sources included the word red as well. That might pose a problem for, say, a car designer looking for the most popular color combinations mentioned in a year s worth of customer s.

5 Pg. 4 Adding the word color to the search string as a workaround might improve results, but again, not all red-related content would include the word color either. Another issue arises in the fact that the word red has meanings outside the context of design, especially as a metaphor for danger, as in red zone. A designer might want to exclude those. Bush is an even richer example of a word whose meaning is different depending on context. A search on that word might return articles about President Bush as well as articles about landscaping. A journalist looking for documents about President Bush would want to specify the person attribute in the search. What is missing in these scenarios is the ability to refine searches to target the desired meaning (semantics) of a term. The computer cannot automatically discover all the entities, i.e., concepts, expressed as words or phrases that have those attributes. With first generation search, you must already know all the right keywords to type into the search box. You must also have taken into account all the semantic misinterpretations (as in other kinds of bushes). And you would also need the skill to express terms so as to exclude those misinterpretations. This ability to automatically discover semantic matches and exclude semantic non-matches is what entity extraction is all about. HOW ENTITY EXTRACTION WORKS Words derive much of their meanings from the context in which they appear on a page. The same word will often have very different meanings depending on context. Similarly, the same meaning will often be expressed by very different words, which is also context-sensitive. These contextual features include: Proximity to other words Written forms of the word (e.g., abbreviations, capitalization) Parts of speech (e.g., is the word used as a subject, predicate, object, etc.) Punctuation And many more A person grasps a concept from a word or phrase choosing from one of several possible word definitions. That choice is heavily dependent on the context of the word or phrase. Entities are the words on a page that represent a concept in people s heads.

6 Pg. 5 Suppose you wanted to find all human names in a document all names, including those of people you had not previously considered. Suppose also that the document contains the quote: Mr. John Hillinhuetter spoke Even though you may never have seen the name Hillinhuetter before, you would infer that Hillinhuetter is a person s name. You would base that inference on that fact that the word is capitalized and otherwise matches a pattern: Mr. [capitalized noun] [capitalized noun] [verb] Entity extraction recreates the process of applying and recognizing context in a computer. Take the word date. When that word appears in a block of text, is it an entity with one set of contextual features, as in a point in time, or is it a completely different set of contextual features that add up to food? There are literally hundreds of different features that may indicate when the word date is about time or about food. Ko i Atta Annan is a Ghanaian diplomat who served as the seventh Secretary General of the United Nations from January 1, 1997, to January 1, 2007, serving two ive-year terms. Annan was the co-recipient of the Nobel Peace Prize in October Ko i Annan was born on April 8, 1938, to Victoria and Henry Reginald Annan in Kumasi, Ghana. He is a twin, an occurrence that is regarded as special in Ghanaian culture. Efua Atta, his twin sister, shares the same middle name, which means twin. As with most Akan names, his irst name indicates the day of the week he was born: Ko i denotes a boy born on a Friday. The name Annan can indicate that a child was the fourth in the family, but in his family it was simply a name which Annan inherited from his parents. Person In 1962, Annan started working as a Budget Of icer for the World Health Location Organization, an agency of the United Nations. From 1974 to 1976, he worked Organization as the Director of Tourism in Ghana. Annan then returned to work for the United Date Nations as an Assistant Secretary General in three consecutive positions. Nationality Title Ko i Annan (né le 8 Avril 1938), est l ancien secrétaire général des Nations-unies. Ko i Annan fut le septième Secrétaire général de l Organisation des Nations Unies et le premier à sortir des rangs du personnel. Il a entamé son premier mandat le 1 er janvier Le 29 juin 2001, sur recommandation du Conseil de sécurité, l Assemblée générale l a réélu par acclamation pour un second mandat, commençant le 1 er janvier 2002 et s'achevant au 31 décembre Ko i Annan est né à Kumasi (Ghana) le 8 avril Il a étudié à l'université scienti ique et technologique, à Kumasi, et a achevé sa licence d'économie au Macalester College, à Saint Paul (Minnesota) aux États-Unis, en En Personne 1961 et 1962, il a fait des études de troisième cycle en économie à l Institut universitaire Lieu des hautes études internationales, à Genève. En 1971 et 1972, en qualité de Organisation Sloan Fellow au Massachusetts Institute of Technology, M. Annan a obtenu son diplôme Date de maîtrise en sciences de gestion. Nationalité Titre For decades, linguists and computational linguists have examined word context to create natural language processing software. Natural language refers to normally written or spoken language. The barrier to automating entity extraction has been the question of how to write software to correctly recognize entities. Manually programming the rules to do this is impractical; there are just too many different possible contexts that determine a word s meaning. Some features are obvious, but many are not. It would be virtually impossible to specify all of them explicitly as rules for a computer for every possible context a computer might encounter in real-world applications. Rather than programming computers to recognize contexts, an alternative approach has been to essentially install the language itself into computers. In other words, the goal is to teach computers the grammatical and morphological rules of English, French, Chinese, or whatever human language is to be parsed. Once the computer understands a language, it could presumably recognize entities within a text based on the context in which the entity appears.

7 Pg. 6 A natural language parser, however, has two main drawbacks: Natural languages are enormously difficult to express as explicit rule sets, so a parser would take years to construct Every language has a different set of rules, so the parser built for one language would not work for another A third approach, different from either programming computers to match contexts explicitly or constructing natural language parsers, is statistical machine learning. Whereas other approaches might take years just to write the code to begin to extract even one entity (color, for example), a statistical approach uses an efficient three-step process that can be completed in a few weeks. 1. A computational linguist specifies an entity s contextual features 2. Native speakers tag a statistically sufficient number of examples of the entity (typically thousands) in the target text genre (e.g., news, medical, financial, etc.) 3. The features and tagged examples are fed into a computer, which then generates a model that recognizes entity matches in the text genre. Statistical machine learning makes entity extraction practical and offers several key benefits: Language Independence The software algorithms are the same regardless of language or script; the only difference is the entity-specific context features and language-specific, tagged training samples used. Entity Extensibility Adding new features and examples to an existing model is straightforward, so adding new entities to accommodate linguistic changes is relatively easy. Automated Discovery The model will return new context-appropriate instances of an entity even if they didn t exist when the entity was first modeled. Users don t have to search for terms they don t know (or don t know yet) to define an inclusive search.

8 Pg. 7 THE IMPACT Statistically powered entity extraction distinguishes the newest generation of search from earlier generations. People will no longer have to rely on personal knowledge (or simple brute force) to find all occurrences of the information they wish to find in unstructured content, but people are not the only ones capable of using this technology. Adding entity extraction to XML tagging tools, for example, provides an efficient way to bring the mountains of unstructured data both enterprise and web into a database where it can be manipulated and processed just like structured data can. In fact, in the very near future the idea of unstructured data as a separate category may itself seem quaint. We are looking at a new kind of convergence, one just as important as the convergence between data and communications. When that happens, it will be because computers can identify what words mean, not just what they look like. BASIS TECHNOLOGY S APPROACH TO ENTITY EXTRACTION The Rosette Entity Extractor is a hybrid mechanism that integrates the results from three techniques: list-based, pattern matching, and statistical. The target text is fed to all three modules and then a fourth module called the redactor balances the results and acts as judge when answers conflict. Rosette uses a weighted set of criteria to merge results and identify people, places, and other entities. Rosette Entity Extractor Features Foreign language capabilities Rosette extracts entities from text in many languages, including English, Arabic, Pashto, Persian, Urdu, Chinese, Japanese Korean, and major European languages. Context-sensitive extraction Rosette s statistical models consider context when extracting key entities such as person, place, and organization (including company names). Seamless integration Rosette is a software development kit (SDK) accessible via via single C, C++,.NET, or Java application programming interface (API). It has been designed for simple integration with Apache Lucene, Apache Solr, dtsearch,and other search engines. Easy customization Users can add custom entities via regular expressions or lists, or enhance the statistical model with training data with additional data relevant to the user s problem domain. High accuracy and throughput Rosette s accuracy and speed is industry-tested and used by customers such as Microsoft Bing, which handle a high volume of transactions and require high quality for every system component.

9 Pg. 8 ABOUT THE AUTHOR Steve Cohen is Executive Vice President and Chief Operating Officer of Basis Technology, where he is responsible for worldwide sales and the planning and operations of the company s linguistic product research and development. Before starting Basis Technology with Carl Hoffman, Steve was engineering manager for Cognex Corporation s Tokyo office and development manager for SMT device inspection. He has also consulted on software internationalization engineering and developed software for embedded systems and electronic test equipment. Steve earned a bachelor s degree in electrical engineering from MIT and studied at Waseda University in Tokyo.