Aufbau agiler BI- & Discovery-Applikationen mit Oracle Endeca. DOAG 2012 Nürnberg, 20. November Harald Erb Solution Architect BI & DWH

Transcription

1 Aufbau agiler BI- & Discovery-Applikationen mit Oracle Endeca DOAG 2012 Nürnberg, 20. November 2012 Harald Erb Solution Architect BI & DWH 1

2 Agenda Oracle Big Data & Analytics Overview Endeca Information Discovery Live-Demo Developing Endeca Solutions Q & A 2

3 The Mystery of your Business 3

4 Dialogues Contain Critical, Untapped Insights 4

5 New Insights Drive New Opportunities 5

6 The Challenges of Unstructured Data 6

7 Organizations Need Help to Bring together diverse and unstructured data sources Easily explore those sources to draw new insight and meaningful conclusions Quickly analyze to make more informed decisions In an intuitive, easy to use manner without burdening IT. 7

8 8

9 9

10 Comprehensive Big Data & Analytics Solution 10

11 Oracle Big Data Appliance Acquiring and Organizing Big Data Get up and running quickly Prepackaged solution Improve performance of Hadoop InfiniBand Interconnect between nodes Easily integrate into your data center Optimized for Exadata Lower TCO for big data Minimize integration and support costs 11

12 Oracle Big Data Appliance Integrated Hardware and Software Hardware 18 Sun X4270 M2 Servers 48 GB memory per node = 864 GB memory 12 Intel cores per node = 216 cores 36 TB storage per node = 648 TB storage Software Cloudera CDH3 Cloudera Manager Open Source R Distribution Oracle NoSQL Database (CE or EE*) 12

13 Oracle Exadata Hardware for extreme Data Warehouse & OLTP Performance Database server pool Intel-based database servers Oracle Linux or Solaris Gb/sec Ethernet (to data center) Storage server pool Up to 504 TB disk 5 TB flash storage Oracle Exadata Storage Software 40 Gb/sec InfiniBand network 13

14 Oracle Exadata Software Database server pool Oracle Database 11g Oracle Real Application Clusters Oracle Automatic Storage Management Storage server pool Smart Scan Hybrid Columnar Compression Smart Flash Cache 14

15 Oracle Exalytics First In-Memory Engineered System for Analytics Spans Relational, Multi- Dimensional, and Unstructured Analysis, Combined with Financial & Operational Planning In-Memory Optimized Hardware In-Memory Oracle BI, TimesTen, Essbase, and Endeca Several In-Memory Software Innovations Tightly integrated with Exadata and Big Data Appliance 15

16 Oracle Exalytics Memory Optimized Hardware 16

17 Oracle s Complete Big Data Solution Oracle Endeca Information Discovery Oracle Big Data Appliance Cloudera Hadoop Oracle NoSQL Open-Source R Big Data Connectors InfiniBand Oracle Data Integrator Oracle Exadata Oracle Advanced Analytics Oracle Spatial and Graph Oracle Database InfiniBand Oracle Endeca Information Discovery Oracle Business Intelligence Oracle Real-Time Decisions Oracle Exalytics Acquire Organize Analyze Decide 17

18 Agenda Oracle Big Data & Analytics Overview Endeca Information Discovery Live-Demo Developing Endeca Solutions Q & A 18

19 Have you ever heard of Endeca? Hitcount always and fast updated Graphical selection of filter criteria i.e. with histogram view Guided Navigation shows only remaining filter attributes and no. of occurrences for each selection step Powerful text search and geospatial search 19

20 Oracle Endeca Information Discovery Rapid, intuitive exploration and analysis of data from any combination of structured and unstructured sources Unique Features Contextual Search, Navigation, Analytics Dynamic Data and Metadata Content Acquisition and Text Enrichment In-Memory Performance Benefits Unprecedented Information Visibility Leverage Existing BI Investments Self-Service Data Discovery Reduced IT Costs, Better Business Decisions 20

21 Global Automotive Manufacturer: Gets Ahead of Quality Issues Investigate Root Cause Reduce warranty costs and detection to correction time Common view of 6 years of product and quality data from multiple systems Quality engineers explore root cause (why) of product issues from unstructured data. Operates along side Oracle BI EE, Essbase, EPM and PeopleSoft Result: 80% reduced time-to-market 21

22 Consumer Goods: Discover New Insights from Big Data Consumer Sentiment Impact on Sales Understand consumer dialog to aid sales performance Integrated customer service center conversations, social media dialog, loyalty card transactions and ad spend Discovered how viral events influence consumer brand sentiment and impact to sales performance 22

23 Oracle Endeca Information Discovery Extend Business Analytics with Unstructured Data 23

24 Oracle Exalytics with Information Discovery Fastest, most powerful hardware platform for Unstructured Analytics In-Memory BI, EPM, and Unstructured Analytics Designed for Performance Endeca Software Breakthroughs Multi-Core Parallel Query Evaluation In-Memory Columnar Storage and Dynamic Cache Adaptive In-Memory Data Mapping Embedded Search Indices Free Form Search, Exploration, and Analysis Optimized Hardware Processor: 4 Intel Xeon E7-4870, 40 cores total Memory: 1 Terabyte DRAM Hard Disk: 3.6 Terabytes 24

25 Business Intelligence and Information Discovery Complete Business Analytics Solution Analysis Problems Measure, Analyze, Report Discovery Problems Investigate, Explore, Understand Structured Data Modeled and conforming Oracle Business Intelligence Proven Answers to Known Questions Optimized for Exalytics In-Memory Machine Unstructured Data Diverse, textual, uncertain quality Oracle Endeca Information Discovery Fast Answers to New Questions 25

26 Agenda Oracle Big Data & Analytics Overview Endeca Information Discovery Live-Demo Developing Endeca Solutions Q & A 26

27 Endeca Information Discovery in Action 27

28 IT System Integrator: Project Staffing 28

29 Major Retailer: Identifying root causes of drop in sales 29

30 Agenda Oracle Big Data & Analytics Overview Endeca Information Discovery Live-Demo Developing Endeca Solutions Q & A 30

31 Discovery Application Lifecycle Structured Perform Content Acquisition and Enrichment Ingest information on the Endeca Server no predefined model required Build the Information Discovery application with a Drag-and-drop approach Semi-Structured Unstructured Oracle Corporation Confidential Oracle Internal

32 Discovery Application Lifecycle Structured Perform Content Acquisition and Enrichment Ingest information on the Endeca Server no predefined model required Build the Information Discovery application with a Drag-and-drop approach Semi-Structured Oracle Endeca Integrator Oracle Endeca Server Oracle Endeca Studio Unstructured Oracle Corporation Confidential Oracle Internal

33 Oracle Endeca Information Discovery Enterprise Platform for Business Self-Service and Discovery Studio Interactive Business Discovery Create and Share Apps Endeca Server Hybrid Search/Analytical Database Flexible Data Model Integration Suite Data Integration and Enrichment Structured and Unstructured Helps organizations quickly explore all relevant data. Combines diverse and changing information from disparate systems. Automatically organizes information for exploration and analysis. Enables rapidly assembly of easy-touse discovery applications. 33

34 Oracle Endeca Information Discovery Endeca Server Studio Interactive Business Discovery Create and Share Apps Endeca Server Hybrid Search/Analytical Database Flexible Data Model Integration Suite Data Integration and Enrichment Structured and Unstructured Oracle Endeca Server Hybrid search/analytical database Search, navigation, and analytics on diverse, changing information Designed for Discovery Flexible data model Columnar storage In-memory analytics 34

35 Endeca Server In-Memory, But Not Memory-Bound value, id Storage columns are mapped into virtual memory Data resides on disk Fetched into RAM when referenced Embedded index trees mean only needed data is scanned Data that is referenced will reside in RAM Data that is not frequently needed remains on disk Especially common with search indexes and wide records Provides key benefits of in-memory analytics, without being bound by the size of RAM 35

36 Endeca Server Full Text Search Search index storage and analysis build on same column storage core as structured data store / indexes Structured Data Column value, id Acme Corp, 375 Search Index Column word, id, position Corp, 375 (89,72) Supports a breadth of structured and unstructured search capabilities: Guided Navigation Keyword search Boolean search Parametric search Wildcard search Dimension search Dimension filters Dimension precedence rules Numeric range filters Geospatial filters Date/Time filters Security filters Spell correction/suggestion, DYM Find similar 1- and 2-way synonyms Stemming and lemmatization Keyword-in-context snippeting Results clustering Relevance ranking Sorting and paging Language support 36

37 Endeca Data Model Table-free Schema-free Semi-structured Denormalized Multi-assign 37

38 Endeca Data Model Structured Data PRODUCT_ID SUPPLIER_ID WEIGHT QTY START_SELL UNIT_COST AA /1/2009 $2.25 AA A 8 oz /1/ AA lb 500 1/1/ BB Dec. 1, :34: BB A /1/ Structured Information RDBMS, ERP, DWH /12/2008 $1.25 each <ENDECA RECORD> <PROP NAME= PRODUCT_ID"><PVAL>AA12345</PVAL></PROP> <PROP NAME= SUPPLIER_ID"><PVAL>000123</PVAL></PROP> <PROP NAME= WEIGHT"><PVAL>8</PVAL></PROP> <PROP NAME= QTY"><PVAL>1500</PVAL></PROP> <PROP NAME= START_SELL"><PVAL>01/01/2009</PVAL></PROP> <PROP NAME= UNIT_COST"> <PVAL>2.50</PVAL></PROP> </ENDECA RECORD> Record Manipulator Pipelines support arbitrary series of data manipulation steps Supports aggregate and multi-assigned fields Pre-packaged manipulators for Strings, Regex, Date/Time, Geocodes, and Hierarchy Extensible via Java-based framework Configured through administrative GUI Final Endeca Record 38

39 Endeca Data Model Structured Data Table-free Schema-free Semi-structured Denormalized Multi-assign System A unique attributes Present ONLY in System A e.g. Location, Sales Staff, Cust Group Global attributes Common across all or many systems e.g. Time, City, Brand, Key Account System B unique attributes Present ONLY in System B e.g. Volume, Trade Area, Product Details, etc Supports jagged records For diverse structures such as customer attributes 39

40 Endeca Data Model Unstructured Data Table-free Schema-free Semi-structured Denormalized Multi-assign System A unique attributes Present ONLY in System A e.g. Location, Sales Staff, Cust Group Global attributes Common across all or many systems e.g. Time, City, Brand, Key Account System B unique attributes Present ONLY in System B e.g. Volume, Trade Area, Product Details, etc Supports jagged records For diverse structures such as customer attributes 40

41 Endeca Data Model Unstructured Data Crawled Properties Document Conversion Language, Encoding Detection Tokenization Property Mapping Index Crawled Properties 8 universal attributes 15 attributes for file systems 20 attributes for CMS Sample properties include: Endeca.SourceType Endeca.SourceId Endeca.FileSystem.Path Endeca.FileSystem.ACL.AllowRead Endeca.FileSystem.Extension Endeca.File.Size Endeca.CMS.RepositoryType Endeca.CMS.RepositoryVersion Endeca.CMS.ItemID Endeca.CMS.Path Endeca.CMS.CreationDate Document Conversion Properties Automatic identification of formats Extraction of file metadata, e.g. Author, Title Full text of file extracted as Text property Sample properties include: Endeca.Document.Type Endeca.Document.Metadata.Title Endeca.Document.Metadata.Author Endeca.Document.Text Language, encoding Properties Over 40 encoding types automatically detected, including e.g. UTF-8, ASCII Option to specify undetectable values, e.g. Language= Pig Latin Sample properties include: Endeca.Document.Language Endeca.Document.Encoding 41

42 Endeca Data Model Unstructured Data Table-free Schema-free Semi-structured Denormalized Multi-assign System A unique attributes Present ONLY in System A e.g. Location, Sales Staff, Cust Group Global attributes Common across all or many systems e.g. Time, City, Brand, Key Account System B unique attributes Present ONLY in System B e.g. Volume, Trade Area, Product Details, etc Supports jagged records For diverse structures such as customer attributes 42

43 Text Enrichment using Rules-based Tagging SearchTerm Google Rick Whiting MSFT Los Angeles... TagValue Corporate Person Corporate City... Text tagging Framework for applying metadata properties when matching criteria met Tag list specified based on any authoritative term lists Sample Text Tagger outputs: Endeca.Document.KeyTerm 43

44 Text Enrichment with Entity Extraction Entity Extraction Identify categorized entities such as Persons, Places, Organizations, Dates Generates metadata dimensions to drive Guided Navigation, Analytics, other features Sample Entity Extraction outputs: Endeca.Document.Person= Rick Whiting Endeca.Document.Organization= Google Endeca.Document.Organization= Novell Endeca.Document.Ticker= MSFT Endeca.Document.Ticker= CSC Endeca.Document.Location= Los Angeles 44

45 Text Enrichment with Sentiment Analysis Sentiment Analysis Categorize documents based on overall positive, neutral, or negative tone Provides insight into balance of opinion across the corpus Sample Sentiment Analysis output: Endeca.Document.Sentiment=

46 Oracle Endeca Information Discovery Integration Suite Studio Interactive Business Discovery Create and Share Apps Endeca Server Hybrid Search/Analytical Database Flexible Data Model Integration Suite Data Integration and Enrichment Structured and Unstructured Integration Suite Lightweight ETL with native Endeca Server connectivity Java SDK for extensibility easily connect to additional data feeds and APIs Modules for unstructured data: Text Enrichment Sentiment Analysis Content Management System Connectors 46

47 Endeca Server Integration Suite Data Integration APIs Integration Suite Integrator Text Enrichment Text Enrichment with Sentiment EDW Content Acquisition System (CAS) CMS Connectors Quickly load and enrich any data. Structured or unstructured. Internal or external. Endeca Server Provides load-and-go web services APIs for incremental, bulk ingest Enables integration with other data tools Integrator Included agile enterprise ETL tool Adapters for JDBC and common file types (e.g. XML, delimited, fixed-width), library of structured and unstructured data manipulators Production management and monitoring interfaces Add-on modules for Text Enrichment, Sentiment Analysis Content Acquisition System (CAS) Crawls and extracts text from documents in CMS, web, file systems Add-on modules for common CMS 47

48 Integration Suite Dimensional Modeling 48

49 Integration Suite Text Enrichment 49

50 Oracle Endeca Information Discovery Studio Studio Interactive Business Discovery Create and Share Apps Endeca Server Hybrid Search/Analytical Database Flexible Data Model Integration Suite Data Integration and Enrichment Structured and Unstructured Studio Interactive, visual discovery environment Drag-and-drop configuration Full-featured component library Based on best practice UI design patterns Built on Java portal technology standards 50

51 Endeca s Unique User Experience Interactive Data Exploration and Analysis Advanced Search Search across all data Dynamic typeahead Automatic spell correction Unlocks unstructured data + + Contextual Navigation Data-Driven. Freely browse data without predefined paths or writing queries Interactive. Shows only valid next steps Easy to Use. Familiar online experience Visual Analysis Charts, crosstabs, key metrics Geospatial visualization Tag clouds 51

52 Studio Component-Based Application Composition Each component includes a user control and an editor 52

53 Studio Component Library Configurable components: Alerts Bookmarks Breadcrumbs Chart Compare Crosstab Data Explorer Guided Navigation Map Metrics Bar Range Filters Record Details Results Table Results List Search Box Tag Cloud 53

54 Oracle Exalytics with Information Discovery Fastest, most powerful hardware platform for Unstructured Analytics In-Memory BI, EPM, and Unstructured Analytics Designed for Performance Endeca Software Breakthroughs Multi-Core Parallel Query Evaluation In-Memory Columnar Storage and Dynamic Cache Adaptive In-Memory Data Mapping Embedded Search Indices Free Form Search, Exploration, and Analysis Optimized Hardware Processor: 4 Intel Xeon E7-4870, 40 cores total Memory: 1 Terabyte DRAM Hard Disk: 3.6 Terabytes 54

55 OEID Project Methodology (typical) 1-2 Weeks Duration: 8-10 Weeks Discovery Design Workshop (Iteration 0) Iteration Driven Business Release Approach User Adoption Solution Architecture Definition Business Release Scope & Approach Multi-Release Approach Confirm Requirements Solution Design User Interface Sketching Iteration Plan Development Kickoff Communication Plan Iterations every 2-4 weeks Working application released to users for each iteration Iterations build upon each other with additional data, users, functionality and scale Feedback from constant user acceptance testing to refine functionality Rollout Tools Feedback for next Business Release Oracle Corporation Confidential Oracle Internal

56 Agenda Oracle Big Data & Analytics Overview Endeca Information Discovery Live-Demo Developing Endeca Solutions Q & A 56

57 WHY? engineered for better outcomes Business Analytics Business Intelligence Analytic Applications Data Warehousing of analytics Systems engineered system for in-memory analytics 57

58 58

59 59