TIBCO Analytics Meetup. Michael O Connell and the TIBCO Data Science Team April 25th, 2017

Transcription

1 TIBCO Analytics Meetup Michael O Connell and the TIBCO Data Science Team April 25th, 2017

2 CONFIDENTIALITY The following information is confidential information of TIBCO Software Inc. Use, duplication, transmission, or republication for any purpose without the prior written consent of TIBCO is expressly prohibited.

3 DISCLAIMER This document (including, without limitation, any product roadmap or statement of direction data) illustrates the planned testing, release and availability dates for TIBCO products and services. This document is provided for informational purposes only and its contents are subject to change without notice. TIBCO makes no warranties, express or implied, in or relating to this document or any information in it, including, without limitation, that this document, or any information in it, is error-free or meets any conditions of merchantability or fitness for a particular purpose. This document may not be reproduced or transmitted in any form or by any means without our prior written permission. The material provided is for informational purposes only, and should not be relied on in making a purchasing decision. The information is not a commitment, promise or legal obligation to deliver any material, code, or functionality. The development, release, and timing of any features or functionality described for our products remains at our sole discretion. During the course of this presentation TIBCO or its representatives may make forward-looking statements regarding future events, TIBCO s future results or our future financial performance. These statements are based on management s current expectations. Although we believe that the expectations reflected in the forward-looking statements contained in this presentation are reasonable, these expectations or any of the forward-looking statements could prove to be incorrect and actual results or financial performance could differ materially from those stated herein. TIBCO does not undertake to update any forward-looking statement that may be made from time to time or on its behalf.

4 TIBCO Analytics Meetup opportunities to learn and network Group keeps growing and now has 345+ members! Join this group and receive automatic invite to future TIBCO Analytics Meetups

5 Agenda Welcome and AI and Machine Learning introduction by Michael O'Connell Anomaly Detection by David Katz Geo-Analytics Demo and Community content by Peter Shaw TIBCO Community update by Heleen Snelting Live Q&A Please submit your questions at any time via Q&A option We will answer at the end or will get back to you via

6 AI and Machine Learning introduction Michael O Connell, Chief Analytics Officer TIBCO Software

7 TIBCO Spotfire - Smart Analytic Apps for Customers Out of box & Right-click ML 2. Custom Expression functions 3. Embed Data Functions

8 Data Science at Your Fingertips with Spotfire and R Mass Communicate the Power of Data Science Spotfire has an embedded R Engine Client and Server Spotfire Analyst Spotfire Consumer Spotfire ios Spotfire Automation Spotfire makes it easy for Data Scientists to share their work Spotfire Data Functions support many Stats Engines Spotfire Library manages components for re-use & sharing TIBCO Community shares data science components LDAP Spotfire Library Data

9 TIBCO Enterprise Runtime for R (TERR) and Spotfire What does TERR do in Spotfire? Adds stats on the Visual environment Can be used directly in Expressions (Data Wrangling) Runs R Data Functions in Spotfire analyses Powers the Predictive Modeling Tools Can extend Spotfire graphics palette

10 Spotfire TERR Data Function Spotfire Map Spotfire TERR Data Function contourlines(x,y)

11 Spotfire TERR Data Function Spotfire Map Data Functions Wafer diameter Die width, height Spotfire/TERR(parameters) Spotfire Map Chart fields package for smoothing Spotfire TERR Data Functions wafer(die width, height) fields: tps()

12 Wafer Maps in Spotfire

13 Wafer Analysis in Spotfire

14 Spotfire TERR Data Function Anomaly Detection Spotfire TERR Data Function h2o.deeplearning(x,..) h2o.deeplearning( )

15 Spotfire TERR Machine Learning Data Functions Spotfire Modeling Data Functions Machine Learning Gradient Boosting Random Forests Deep Learning Anomaly Detection Segmentation Propensity Affinity Non-Linear Regression; Decline Curves Modeling & Simulation Optimization Genetic Algorithms

16 The TIBCO Insight Platform Data Discovery ACCESS PREDICT Insight Deployment WRANGLE DECIDE Insight Action ANALYZE ACT MODEL MONITOR

17 The TIBCO Insight Platform TIBCO Spotfire ACCESS PREDICT TIBCO Streambase WRANGLE DECIDE Insight Action ANALYZE ACT MODEL MONITOR

18 The TIBCO Insight Platform The TIBCO Insight Platform Closed-Loop Continuous Learning to Transform Your Business The TIBCO Insight Platform fuses our visual, predictive, and streaming analytics technologies providing a platform for optimizing business operations. The platform creates a digital nervous system on your business operations, generating actionable insights in real time. The platform helps you source and analyze the salient data, to surface insights and drive actions in a closed-loop solution with continuous learning.

19 The power of the Insight Platform applies to numerous industries Financial Fraud Detection Accelerator Accelerator for Apache Spark Freely available on TIBCO Community wiki/accelerators Connected Vehicles Accelerator Intelligent Equipment Accelerator FX Dealing Accelerator Real-Time Inventory Accelerator

20 Insight Platform: Anomaly Detection 1. Visualization Dashboard Features Insights 2. Fit autoencoder and score history rae.mod <- deeplearning.h2o(x) econ <-h2o.anomaly(x, ae.mod) Data Function - Model Reconstruction error 3. Fit gbm on error gbm.mod <-gbm(score, x) Root Cause Analysis 4. Score new live events Raise notification, alert gbm.mod Intervention Action Insights Model

21 Insight Platform - Documents

22

23 Anomaly Detection Autoencoders in Spotfire David Katz, Principal Consultant TIBCO Software

24 Autoencoders in Spotfire For Outlier Detection David Katz Principal Consultant

25 Challenge: Media Server Fails Video, Telephony, Internet Typical case: The first we know of it is that customers call to complain. Lots of them Can we detect the problem sooner and take corrective action?

26 Challenge: Media Server Fails Video, Telephony, Internet Timely Alerts Root Cause Analysis Note how this is an example of a very general problem Monitor and understand the Black Box Justify the recommendation of a smart assistant

27 Current Environment: System produces log files Complex, Verbose, Voluminous Impenetrable by users. then press the 're-sync alarms' button on the NE object (if available) or deactivate and reactivate the NE Connection object. This will remove this alarm and replace it with separate alarms if the alarm limits are no longer breached. If this does not correct the problem, and you do not see a large number of individual alarms on the same type of object, this collective alarm could be due to an error in the alarm counts reported by the NE. In this case, a restart of the MetaView Server, or a scheduled Software Protection Switch on the affected NE, during a maintenance window may be necessary to correct the problem. You should discuss this with your Support Contact. " enterprise=" " OS_EventId="SNMPTRAP-metaswitch-METASWITCH-MIB-alarmTrap" alarmtraplogid=" " alarmtrapelementname="boonetworkelementconnse.80/boonetworkelementse.default/ BOOCallFeatureServerSE.1/BOOControlledGatewaysContainerSE/BOOSubscriberGatewayContainerSE" alarmtrapalarmtype="clear ( 1 )" alarmtrapcountresettimestamp="0" alarmtrapequipmentidentifier="87:ef:99:80:5d:28:38:03:72:23:1e:ea:08:c0:c5:99" alarmtrapseverity="80" alarmtrapalarmfaulttype="alarm ( 1 )" alarmtrapelementdisplaytype="" generic-trap="6" alarmtrapobjectadditionalinformation="" alarmtrapalarmedobjectprimaryoid=" " ="( ) 2 days, 9:00:56.22" alarmtrapequipmentsoftwareversion=" " ="ec9ecd0aec0e" ="ems01olvemo-sec" alarmtrapequipmentipaddress=" " alarmtrapequipmenttype="unknown ( 112 )" alarmtrapalarmoid=" " =" " alarmtrapgroup="42" ="112" ="87:ef: 99:80:5d:28:38:03:72:23:1e:ea:08:c0:c5:99" ="agc14olvemo" =" " ReceivedTime=" " alarmtrapcause="a

28

29 Step 1 Reformat log as structured table using R/TERR (TIBCO R) then press the 're-sync alarms' button on the NE object (if available) or deactivate and reactivate the NE Connection object. This will remove this alarm and replace it with separate alarms if the alarm limits are no longer breached. If this does not correct the problem, and you do not see a large number of individual alarms on the same type of object, this collective alarm could be due to an error in the alarm counts reported by the NE. In this case, a restart of the MetaView Server, or a scheduled Software Protection Switch on the affected NE, during a maintenance window may be necessary to correct the problem. You should discuss this with your Support Contact. " enterprise=" " OS_EventId="SNMPTRAPmetaswitch-METASWITCH-MIB-alarmTrap" alarmtraplogid=" " alarmtrapelementname="boonetworkelementconnse.80/ BOONetworkElementSE.default/BOOCallFeatureServerSE.1/ BOOControlledGatewaysContainerSE/ BOOSubscriberGatewayContainerSE" alarmtrapalarmtype="clear ( 1 )" alarmtrapcountresettimestamp="0" alarmtrapequipmentidentifier="87:ef:99:80:5d:28:38:03:72:23:1e:ea: 08:c0:c5:99" alarmtrapseverity="80" alarmtrapalarmfaulttype="alarm ( 1 )" alarmtrapelementdisplaytype="" generic-trap="6" alarmtrapobjectadditionalinformation="" alarmtrapalarmedobjectprimaryoid=" " ="( ) 2 days, 9:00:56.22" alarmtrapequipmentsoftwareversion=" " ="EC9ECD0AEC0E" ="ems01olvemo-sec" alarmtrapequipmentipaddress=" " alarmtrapequipmenttype="unknown ( 112 )" alarmtrapalarmoid=" " =" " alarmtrapgroup="42" ="112" ="87:ef:99:80:5d:28:38:03:72:23:1e:ea: 08:c0:c5:99" ="agc14olvemo" =" " ReceivedTime=" " alarmtrapcause="a

30 Step 2 Use Spotfire s Autoencoder Template

31 Unsupervised = no outcome variable in the data. Examples: Principal Components Clustering Autoencoder / Anomaly Detection Generalizes Principal Components

32 Autoencoders In use for decades Recent developments have brought them into increased use

33 Applications of Autoencoders Pre-Training Deep Learning Networks Compression Classification Anomaly detection

34 Applications of Autoencoders Anomaly Detection Fraud Detection Manufacturing Processes Monitor and Control

35 Autoencoder with One Hidden Layer

36 Autoencoder In Spotfire Using R Package h2o

37 Autoencoder In Spotfire Using R Package h2o Spotfire Desktop Cloud WebPlayer TIBCO Enterprise Runtime for R Desktop Server Hadoop Cluster

38 Autoencoder Simple Interface

39 Autoencoder Multiple Hidden Layers

40 Autoencoder Output Histogram

41 Autoencoder Output Histogram Typical Unusual Anomaly

42 Autoencoder Drill-Down on Most Severe Anomalies

43 Autoencoder Time Series View

44 Autoencoder Drill-Down to Log Messages by alarmtrapdescription

45 Using Supervised Learning to replicate the Reconstruction MSE

46 Using Supervised Learning to replicate the Reconstruction MSE

47 Using Supervised Learning to replicate the Reconstruction MSE

48 Summary Spotfire provides access to advanced analytics including Deep Learning R/TERR packages and scripts SAS Python via PythonInR package MATLAB Seamless integration with visual analytics Visual Drill-down Multiple procedures linked in a data discovery workflow

49 Geo-Analytics Demo and Community content Peter Shaw - Senior Data Scientist TIBCO Software This section was mainly delivered through a live demo please refer to the video recordings posted on Youtube and TIBCO Community

50 New GeoAnalytics Main and Gallery Wiki pages Major update with comprehensive overview of latest capabili7es & how to use them New Page

51 Spatial Heatmap Data Function

52 TIBCO Community update Heleen Snelting Data Science Manager EMEA TIBCO Software

53 TIBCO Community the platform for our users TIBCO Community has the answer to your questions! community.tibco.com

54 TIBCO Analytics Meetup - all recordings on Community Spotfire online user group keeps growing and now has 345+ members! Join this group and receive automatic invite to future TIBCO Analytics Meetups Target is our more advanced Spotfire users, Both Analysts and Data Scientists

55 Doctor Spotfire more help with your Spotfire work Another valuable option for our Spotfire users to ask questions Sign up for the Live Webex Tweet question with the hashtag #DrSpotfire Post your question to the TIBCO Community "Answers" section with the hashtag #DrSpotfire Contact Dr. Spotfire directly at If you are new to Spotfire, download a trial and sign up for the regular introduction demo webinar sessions via the regular Spotfire Website Copyright TIBCO Software Inc.

56 Sign up for the TIBCO Spotfire quarterly newsletter Newsletter with reminders of new content on the TIBCO Community, events, customer stories anything interesting for the Spotfire user

57 New and Updated TIBCO Community Assets Geoanalytics Updated Geoanalytics main page and new Gallery page Heatmap and density map data functions Exchange Machine Learning Components Random Forest data function and template Autoencoder Anomaly Detection template CPG & Retail Industry Page Insight Platform page Big Data page Extending Spotfire Scripts What s new in Spotfire page v7.8 features Tips and Tricks Wiki pages

58 Anomaly Detection Template Detect anomalies in datasets Highlight previously unknown problems Machine Learning Autoencoder, neural networks algorithm

59 Random Forest Data Function and Template Machine Learning Model Supervised Learning Variable Importance & Predic7ons Classifica7on & Regression

60 New CPG & Retail Industry page Use Cases Data Sources Customer Stories

61 Insight Platform Wiki Page What is an Insight PlaKorm? Industry Solu7ons Customer Stories

62 Big Data Wiki Page Architecture Technology Use Cases

63 Extending Spotfire Scripts and Extensions Extending & Automa7ng SpoKire Scripts & Extensions 80+ Wiki pages Collabora7ve effort Data Science, Product & Support groups Iron Python, JavaScript, C#, JSViz, etc.

64 What s New in Spotfire 7.8 Overview and links to detailed descrip7ons of new features

65 New Tips and Tricks Content Creating Control Charts Unleash the power of text areas Set the minimum page width Use the data panel Set the number of records displayed in table visualizations Create you own functions using TERR & R Use map zoom visibility options Use TMS layers Use TIBCO GeoAnalytics Using Automation Services to share insight

66 TIBCO and Spotfire specific local Meetups Easily find your local meetups from the Meetup star7ng page Linked from main page of the TIBCO Community

67

68 Q&A and Contact info Michael O Connell, PhD Chief Analytics Officer TIBCO Next TIBCO Analytics Meetup tbc ~ June 2017 Heleen Snelting Data Science Manager EMEA TIBCO Copyright TIBCO Software Inc.

69