Unstructured Text in Big Data The Elephant in the Room

Transcription

1 Unstructured Text in Big Data The Elephant in the Room David Milward ICIC, October 2013

2 Click Unstructured to to edit edit Master Master Big title Data style title style Big Data Volume, Variety, Velocity Estimated 80% of all data is unstructured Need to be able to make decisions based on this data Ever increasing amount of data makes it harder and harder to filter by hand Need more automation 80% 20% Linguamatics Ltd.

3 Click From to to Unstructured edit edit Master Master title style Data title style to Structured Identify concepts and relationships Structured, relational data Normalized concept identifiers Unstructured or Semi-Structured Content Sources Traditional text mining works in batch mode Requires each extraction to be programmed in, or learnt from a large amount of annotated material Linguamatics Confidential

4 Click I2E Agile to to edit edit Text Master Master Mining title style title style Querying Results Information Consumers Confidential

5 Click Increasing to to edit edit Master Range Master title Of style title Applications style Vocabulary Development Target Identification & Prioritization Safety/Tox Sentiment Analysis in Social Media Mining FDA Drug Conference In the Labels beginning... Abstract SAR Competitive Protein- Extraction Intelligence Protein Interactions Plant Metabolic Pathways Mining Key Opinion Leader Identification Biomarker Discovery Mining Electronic Medical Records Patent FTO Chemical Search In-licensing Opportunities Patent Analysis Extracting Numerical and Experimental Data Systems Biology Drug Repositioning Confidential

6 Click The to Database to edit edit Master Master Approach title style title style Can we predetermine what is required? One approach is to try to extract everything but in practice have to make assumptions, whether by hand or automatically Do we include negative statements? Do we include speculation? What context do we include? Need a good idea of how the data will be used to know what needs to be extracted There are cases where this works well e.g. have useful structured data and want to add to it from free text have new records structured, but want to bring in information from legacy free text Linguamatics Ltd.

7 Click Extracting to to edit edit Master for Master a title Relational style title style Database Extracting all relevant information from pathology reports Linguamatics Ltd.

8 Click Extracting to to edit edit Master for Master a title Semantic style title style Store Output as a set of triples Because all the parts have URIs, the parts are webaddressable Peptide Treat Express Associate Psoriasis Infectious Disease Web Linguamatics Ltd.

9 Click The to Ad to edit Hoc edit Master Approach Master title style title style No need to build a database Regard text mining queries as ways to create a database on the fly Keep all the unstructured free text, and query for what you need when you need it Even index structured data with text mining to link information together Used very successfully by knowledge professionals: Just like a search engine, there are thousands of questions people want to ask of the data You can t predict them all beforehand Linguamatics Ltd.

10 Click Ad Hoc to to edit Querying edit Master Master title style title style Find relationships that you want for any particular information request Treat the text mining as an agile database to answer thousands of different questions e.g. metabolites from fruit Linguamatics Ltd.

11 Click Do We to to edit Need edit Master Master More? title style title style I2E Agile Text Mining has opened up text mining to the end user, but it is still mainly used by knowledge professionals Given the importance of unstructured big data, can we bring the benefits of text mining to an even wider audience? Linguamatics Ltd.

12 Click Workflow to to edit edit Master Automation Master title style title style Successful queries converted into regular workflows using Pipeline Pilot or KNIME Real-time workflows for up-to-date dashboards, alerts Further analysis, visualization, integration with other data Pipeline Pilot 12

13 Click Clinical to to edit Trials edit Master Master Analysis title style title from style I2E Text Mining Results Colours indicate Phase Dates Using Pipeline Pilot visualization Linguamatics Ltd.

14 Click Embedding to to edit edit Master I2E Master title within style title Web style Apps 10/21/ _8.pptx

15 Click Form-Based to to edit edit Master Master Querying title style title style Create web interfaces connecting to I2E server Information professionals develop sophisticated queries Queries are parameterized to allow end users to customize Javascript allows portability to mobile devices such as ipads

16 Click Enhancing to to edit edit Master Enterprise Master title style title Search style Use text mining to annotate concepts to feed into a search engine to: provide concept search provide concepts for facets provide intelligent thumbnails for documents, pulling out key information Linguamatics Ltd.

17 Value Click Semantic to to edit edit Master annotation Master title style title of documents style Relationships Concepts Keywords NLP-based Text Mining Search Engine

18 Click Provide to to edit edit Concepts Master Master title style title Facets style for Enterprise Search Document Refine the search information and based on teaser terminologies Document preview and additional information Linguamatics Ltd.

19 Click Dictionary to to edit edit Master Matching Master title style title - Companies style

20 Click Pattern to to edit edit Matching Master Master title - style Institutions title style Not a fixed list: can find previously unknown institutions

21 Click Pattern to to edit edit Matching Master Master title - style Mutations title style

22 Click Pattern to to edit edit Matching Master Master title - style Chemicals title style Assign structure to novel chemicals Distinguish exemplified compounds Distinguish compounds given properties

23 Click Whatever to to edit edit Master the Master Content... title style title style... I2E can mine and extract with precision Scientific literature Twitter

24 Click. and to to edit Wherever edit Master Master title style title style Internal Content MEDLINE Patents Drug Labels Clinical Trials Publisher Content Provider I2E Enterprise I2E OnDemand I2E Platform External Content I2E Platform I2E 4.1 Linked Servers Users can query local information or information on the cloud Proprietary content can reside within Enterprise Standard sources e.g. patents can be hosted on the cloud, and shared by all users Data from content providers can reside on their sites 24

25 Click Bringing to to edit edit the Master Master Elephant title style title Down style to Size Data warehouses for data that you know you need Embedding text mining in other applications e.g. Enterprise Search to provide benefits of semantic search Building specialized new interfaces to provide self-service applications for end-users Continuing role for ad hoc text mining text mining can ask almost any question of the data you can never predict every question Linguamatics Ltd.