Getting Lost in Semantics Selecting the Right Search Engine

Transcription

1 Getting Lost in Semantics Selecting the Right Search Engine Steve Mann VP Sales Concept Searching Robert Piddocke VP Channel and Business Development Concept Searching

2 Steve Mann Vice President of Sales at Concept Searching has expertise in consulting, solutions delivery, sales, and project management. He is experienced in Microsoft technologies, and in providing guidance on best practices, technical advice, and solutions to address a wide range of business challenges, leveraging a combination of Microsoft and third-party technology. Robert Piddocke Vice President of Business and Channel Development at Concept Searching has over fifteen years of business development experience. With extensive Microsoft product knowledge and information management understanding, he has implemented hundreds of site and enterprise search solutions, and has authored two books on SharePoint Search.

3 Agenda Who we are and what we do Search engines Search components Demo How can we help?

4 Who Is Concept Searching? Company founded in 2002 Product launched in 2003 Focus on management of structured and unstructured information Profitable, debt free Technology Platform Delivered as a web service Automatic concept identification, content tagging, auto-classification, taxonomy management Only statistical vendor that can extract conceptual metadata 9 years KMWorld 100 Companies that Matter in Knowledge Management 9 years KMWorld Trend Setting Product Authority to Operate enterprise wide US Air Force, NETCON US Army, and Canadian SLSA Client base: Fortune 500/1000 organizations in Healthcare, Financial Services, Manufacturing, Energy, Professional Services, Pharmaceutical, Public sector and DoD Microsoft Gold Certification in Application Development Member of SharePoint PAC and TAP programs Suitable for all versions of SharePoint on-premises and SharePoint Online, including the vnext dedicated platform and the government cloud

5 What Is the Perfect Enterprise Search Engine? Google-esque Relevant Matches user intent Finds similar content, even when words in search string are not used Balances recall and precision Image Voice Video Safe Fast

6 Precision and Recall This is extremely important in data discovery and classification Enterprise search Knowledge management Research Collaboration Tunable In data discovery and classification, the ability to alter precision or recall easily is a significant advantage Manual is cumbersome and prone to error

7 Cognitive Models of Information Retrieval Cognitive Definition of cognitive 1: of, relating to, being, or involving conscious intellectual activity (such as thinking, reasoning, or remembering) cognitive impairment 2: based on or capable of being reduced to empirical factual knowledge cognitively adverb [kog-ni-tiv] Relating to cognition; concerned with the act or process of knowing, perceiving, etc. :cognitive development; cognitive functioning. Relating to the mental processes of perception, memory, judgment, and reasoning, contrasted with emotional and volitional processes.

8 Findability NOT Usability 2.73 words Users will attempt a search only twice in a given session If the right information isn t located in those two attempts, users assume it does not exist or that it is inaccessible Gartner Will spend 5-25 minutes Inability to find, share, reuse information Inability to find all relevant information presentations, documents, graphics, html Only 50% of content is correctly indexed, meta tagged or searchable 60% of documents are obsolete elaw Less than 1% of content is analyzed

9 Transactional search queries Users want to execute a transaction buy books or purchase an iphone 6 Navigational search queries Users want to visit a specific site or find a certain vendor Frankfurt city library Informational search queries Users are looking for certain information how to make coffee Search engines often support a fourth type of query that is used far less frequently Connectivity queries Questions that report on the connectivity of the indexed web graph Which links point to this URL?, How many pages are indexed from this domain name? How Do Your Users Search?

10 Four Modes When Searching for Information Known knowledge Users have an idea of what they want, can express it, aknow where to start looking. Exploring Users have an idea of what they want to know but may have difficulty expressing it, or cannot use the correct terms. Users often know when they have found the right content, but do not know if the amount of information is sufficient. Do not know what they need Users often do not know exactly what they need to know. They may believe that they need to know one thing, when in reality it is something else. Sometimes they visit an information source without any specific purpose. Retrieve In this mode, users are looking for information of which they have prior knowledge. They can remember where they saw it, the source, where to find it.

11 Google s Take On It

12 Why Is Web Search Different from Enterprise Search? Web Enterprise Content Web pages; largely homogenous Variety of data formats, data sources, heterogeneous Relevance Tolerates large number of results, as well as duplicated and overlapping information Demands small number of unique results with high degree of specifity Personalization Little personalization expected; expect list of returned results Analysis Generic Domain Specific Expectation of customized results (data access) aligned with user profiles (role, group, project, etc.)

13 38% of companies never tune the search engine 8% have tools that were never deployed Keywords Proximity User clicking 5 year replacement cycle How Do We Find Anything?

14 Intelligent Search with Machine Learning (Concept Searching) Document interpretation/topic and concept identification Categorization/ classification of data Query analysis/clustering Ranking results Sentiment analysis Entity identification Personalization of content/ recommendations Auto-complete/directed navigation

15 Natural Language Processing (NLP) Subset of machine learning (ML) Unstructured data acquisition Raw language processing Text mining, text expansion, query understanding for improved search Entity extraction Content categorization Content clustering Statistical natural language processing In many NLP projects, statistical techniques can provide a general understanding of documents as a whole Clustering Categorization Similarity Fact extraction Relationship extraction Topic analysis Word clouds Summarization

16 Branch of artificial intelligence that deals with generating, understanding, and analyzing the languages that humans naturally use in order to communicate with computers in both spoken and written ways, using natural human languages instead of computer languages Referred to as computational linguistics as some tasks involved, such as named entity extraction, deep analytics, sentence segmentation, co-reference resolution, part-of-speech tagging, and parsing Sits at the intersection of artificial intelligence, computational linguistics, and computer science Need to analyze maximum amount of data Not to be underestimated Natural Language Processing (NLP)

17 Machine Learning (ML) Subset of artificial intelligence (AI) Large data sets training set is critical Data set emphasis on data quality, expert input is vital Continuously self-learning Machine-generated predictions Using a subset of machine learning called natural language processing (NLP), users are able to ask a question in a conversational manner, and enterprise search applications can predict which data sources are most relevant Can get complicated to manage and maintain

18 Artificial Intelligence (AI) Can identify the past but not the future Use when and where it delivers specific and measurable business value to the organization Lack of support and investment from big vendors Splintered into areas of expertise Coveo/customer service Attivio/data source discovery Splunk/log file analysis It s neither here yet nor will it overtake organizations like a magical wave of utopian pixie dust. This lack of inevitability is because there are two significant problems when it comes to leveraging machine learning in the enterprise: data and bias. Charles Araujo, CIO Magazine

19 Artificial Intelligence (AI) The term developing systems endowed with the intellectual processes characteristic of humans, such as the ability to reason, discover meaning, generalize, or learn from past experience Moving forward on a project? Expert data scientists Accurate data Lots of data Watson swallowed 800 million pages of Wikipedia alone Extensive testing, patience Deep pockets Address future changes Best suited for questions and answers We need not decide if a machine can think ; we need only decide if a machine can act as intelligently as a human being.

20 Concept Searching

21 Must-Haves Centralized metadata management that supports users' language connects experts terminology with novices The metadata management should also provide support to other information systems, or directly to content creators, to provide feedback on how findable the information will be once published Master data management (MDM), Product information management (PIM) and reference data to identify businesses physical and digital items from information available Directories to traverse the structure of the organization and commonly used directories

22 Solution A pragmatic, easy-to-deploy taxonomy approach More effective than the typical academic approach, where complex ontologies are used Creates metadata repositories and taxonomies that are optimized for each organization Delivers the right information, distilled from a variety of distinct content, making available useable knowledge Concept Searching

23 conceptclassifier Platform Automatically generates multi-term metadata, auto-classifies and manipulates it, eliminating manual, time-consuming, and error-prone tasks Cleans up, optimizes, and organizes file share content Improves search, mitigates risk and information exposures, manages records, and prepares content for intelligent migration Integrates and operates in Microsoft environments Unique compound term processing Advanced taxonomy tools Concept Searching

24 Benefits Reduces the cost of content ownership Identifies and remediates threats of noncompliance Applies and enforces governance policies, in real time Delivers a platform for defensible deletion Improves search and ediscovery Reduces risk with searchable, findable, and organized content Increases accuracy of metadata by eliminating end user tagging, improving productivity, and enabling concept-based searching Unique concept identification enables the creation of organizationallydefined taxonomies Reduces taxonomy development time by 80 percent compared with competitive products Tagging and auto-classification of content aligned to business goals Concept Searching

25 Next Webinar in Metadata-Driven World Series Why You Need Metadata-Driven Records Management Wednesday, October 17, 2018 Register Microsoft is continually adding new features to Office 365, and it is easy to get lost in information. This is particularly true when you need to deploy new functionality in your own organization. Join us as we explore records management in Office 365 and SharePoint. What is useful, what could be improved, and what are potential drawbacks? Understand the importance of metadata in driving records, the synergy with classification labels in the Office 365 Security and Compliance Center, and how it is part of effective records management. Read more and register in the Upcoming Webinars area of our website. Concept Searching

26 Thank You Steve Mann VP Sales Concept Searching Robert Piddocke VP Channel and Business Development Concept Searching Concept Searching