The Challenges of Search for Enterprise Search and Search-based Applications. Gregory Grefenstette

Transcription

1 The Challenges of Search for Enterprise Search and Search-based Applications Gregory Grefenstette

2 1

3 2 Differences between Enterprise & Web Search Motivators for Enterpise Search 3 types of Search Facets and Semantics Semantics and Databases and Text And Merging And Search Based Applications

4 Difference 1: The Enterprise is not just "a small Internet Mark Bennett, New Idea Engineering, Inc. - Volume 5 Number 4 - Summer

5 4 Difference 2: Not web pages in Web pages out

6 Difference 3: Document,Title vs. Hybrid Structured Data 5

7 Difference 4: Getting a Job Done 6

8 Difference 5: Single Shot Relevancy vs Refining Navigation 7 7

9 Difference 6: No page Rank behind firewall

10 Difference 7: identified users, ACL and specific interfaces Company management Access to company accounts/books Access to a search by License and by Service Commercial Access to a search by License and by Service Assistant Manager Access to a search of unpaid invoices. 9

11 Difference 8: Expectations: sliders, bar graphs, pie charts 10 10

12 11 Motivators for Enterpise Search

13 Information everywhere 210 Billion s per day 4.5 trillion pages copied/printed per year 300TB average size of a datawarehouse Source IDC 80% of enterprise information unstructured 988 Exabytes worldwide data volume in

14 Digital data growth IDC Digital Universe Study, sponsored by EMC, May 2010, [Gantz and Reinsel, 2010].) 13

15 Enterprise Search Motivators Compliance, ediscovery Sarbanes-Oxley, 2002 post-enron HIPAA, Health Insurance Portability and Accountability Act, 1996 FRCP, Federal Rules of Civil Procedure equal to paper-based documents as valid and admissible in a court of law Obligation to store all business communication 40,000 per year per employee

16 Enterprise Data all over the place Analyst SRO Report C R M BI Business Analyst.Doc Excel BI Delivery Systems Oracle ERP Client Customer ODS FIDM isb Budget CRM ABC RCK CIM ODS Off-line analysis Data Warehouse CRM DW Accelerator Mapping MDM Product Information JDE Access SAP Siebel Scala Oracle PS Source Systems Source: Leveraging Search to Improve Contact Center Performance Richard Snow, VP & Research Director, Customer & Contact Center Management March types of systems 6% 6% with 20+ with 50+ types of systems 50+ ERP ERP systems systems alone alone

17 Old World View 16

18 17 Three Types of Search

19 Three Types of Search 1. Unique Search Box

20 Three Types of Search 2. Faceted Search

21 Three Types of Search 3. Form Based Search Traditional search on database is complicated: - Several fields to fill in - Need to know field values

22 Three Types of Search In the past ten years, people have learned to use the unique search box People have learned to use facets on shopping sites Forms are still boring

23 Faceted Search and Semantics Facets are semantic dimensions They are visualisations of semantics that users can understand Semantics can come from databases Semantics can come from text Common semantic dimensions link together structured and unstructured data Search Based Applications are possible because of semantics

24 23 Semantics in Search Engine

25 Search Engine now handle rich semantics Text fields «the certification test is documented in the report» Numerical fields Date 16/11/1957 GPS coordinates, real world coordinates , Categories Top/Animals/Pets/Dog Value separated fields Color: outside red ; interior: white ; trimming: silver Metadata (attribute:value) Source : dailymotion

26 Database semantics is imported into search engine facets 1 2 3

27 Semantics is extracted from unstructured text by NLP autosuggestion language identification related terms related queries spell checker faceted search local search phonetic lemmatisation stopwords sentiment analysis named entity lemmatisation synonyms phonetic cross language Ontology Matcher INDEX 26 QueryMatcher FastRules HTMLRelevantContextExtractor Categorizer Clusterer

28 Facets can make search results into Business intelligence aggregating numerous unstructured data sources Feature rich navigation without Navigation having is refined access to dynamically the structured information

29 Provide interactive reporting Infinite number of dimensions Manage simultaneously both dimensions and single records

30 Provide Interactive and real time operational tools for operational arbitrage Enable Real Time reporting on ANY question that could NOT BE ANTICIPATED

31 Build decisional tools on unstructured information Reconnect the causes (unstructured) with the effect (structured)

32 31 Search Based Applications

33 Search Based Applications

34 Advantages of Search Based Applications Usability 360, Google-like, interactif, conversational Agility Flexible, Agile, Days vs Months Performance Real time, millions of end-users, Terabytes of information

35 Search Based Applications advantages Standard traditional Archi SBA Users Interface Limited number of users Usage complexity, production costs Large number of users Ease of use, traffic scalability Functional Heavy one shot development Agile applications Simple data access, use of standard web technologies Querying Data Source Dedicated resources Datamarts, additional hardware Structured data Generic data layer Real time data, high performance querying All data Connectors, structuration of data

36 How Search Based Applications Work Complete DIY Perfect SBA

37 SBA Step 1: Connect to the data sources Instantiate a connector with a connection string, and an extraction query

38 SBA Step 2 : Process the data

39 SBA Step 3: Create Virtual Feeds Create your feeds!

40 SBA Step 4 : Design your pages

41 When to SBA and not to SBA Beware that a SBA is not A replacement for transactional applications A SBA won t manage your workflows, lifecycles, and won t modify the existing systems A good excuse to drop your business intelligence software A SBA goal is not produce the pixel perfect highly complex final report you have to submit to the SEC Replace all your complex, historical business systems A SBA goal is not to reproduce all the business logic of existing applications. It s to simplify it for information access A SBA addresses critical business issues by enabling easy Search & Discovery into key data by key users

42 Examples of Search Based Applications 41

43 Hybrid Search: Unify structured and unstructured data sources A unique search bar Structured results from a Database Unstructured data from the Web Merged Facets

44 Unify structured and unstructured data sources Structured results from a Database Unstructured data from the Web Semantics from Text

45 Directories: Urbanizer Semantic Analytics for Mood-Based Local Search YPG Database + Web & UGC content + Social Networking

46 Government: Sourcier Map-Based Access and Reporting for Water Quality Data

47 Web B2B Description Agile MDM project Part catalogue normalization & enrichment Aggregation of content from ERP & Directories ERP (SAP) & Directories (online directories) Aggregation of : Part number Technical description (size, length) Description Semantic analysis : Merging information from SAP and vertical directories on parts Similar product feature bases on technical description

48 Current and Future Needs: Domain Specific Semantics 47

49 Remaining challenges: Intelligent query interpretation Intelligent analysis of query to extract relevant terms: -2 stars: mandatory field -Swimming pool: Optional with a boost on matching results

50 Travel Description B2C Touristic Portal Automatic aggregated content Exalead Business Console modules Aggregation of : Online news content Hotel directory Entertainment directory Railway station location Semantic analysis : Type Corrida or Beaujolais and the system auto-discover entertainment or railways stations co-cited in the news.

51 Conclusions Search engines can handle the semantics of databases (but not the transactions) Facets are semantic dimensions Semantics allows for Business intelligence type reporting Search Based Applications use the power of search engines (intuitive, scaling, agility) to extract and merge information from databases and text More semantic types are needed for text 50

52 Publisher: Morgan & Claypool Copyright: 2011 ISBN: Paperback Ebook Pages: 141 Authors: Gregory Grefenstette & Laura Wilber, Exalead, S.A. 51