Integrate Analytics into Enterprise Applications Aurélie Urbain MathWorks Consulting Services 2015 The MathWorks, Inc. 1
Data Analytics Workflow Data Acquisition Data Analytics Analytics Integration Business Systems Smart Connected Systems : Single Platform 2
Challenges How to share algorithms with multiple internal and external consumers Challenging and time consuming to re-code algorithms for integration into IT frameworks Difficult to migrate solutions to enterprise scale web or cloud frameworks How to deliver fast results with large volumes of data 3
Key Takeaways 1. Distribute applications to non- users royalty-free. 2. Integrate functions into existing workflows and development platforms. 3. Deploy Analytics for Big Data on Spark-enabled Hadoop clusters. 4. Build a scalable and reliable platform for sharing applications. 4
Sharing and Deploying Applications Write Your Programs Once Then Share to Different Targets Compiler Compiler SDK Coder Embedded Coder Apps Files Standalone Application Excel Add-in Hadoop C/C++ shared library Java.NET Python Production Server.lib/.dll.exe MEX Custom Toolbox Runtime With Users With People Who Do Not Have With Users 5
Share with People Who Do Not Have Compiler Compiler SDK Standalone Application Excel Add-in Hadoop C/C++ shared library Java.NET Python Production Server Share Applications with No Additional Programming Integrate -based Components With Your Own Software Runtime Royalty-free Sharing IP Protection via Encryption 6
Integrate -based Components With Your Own Software Application Author Toolboxes 1 Software Developer Compiler SDK 2 C/C ++.NET Production Server 3 4 Runtime Python Java 7
Using Compiler SDK to create Java Classes 8
and Production Server is the easiest and most productive environment to take your enterprise analytics or IoT solution from idea to production Idea Production 9
Why Production Server Matters to You Domain Expert Solution Architect No need to learn another programming language Web and cloud friendly architecture Reliably service large numbers of concurrent requests with low latency Client connections can be protected with SSL encryption Platform independency (win, mac, linux) Automatically deploy updates without server restart 10
Production Server Enterprise Class Framework For Running Packaged Programs Server software Manages packaged programs and worker pool Runtime libraries Single server can use runtimes from different releases RESTful JSON interface and lightweight client library (C/C++,.NET, Python, and Java) Enterprise Application MPS Client Library Enterprise Application RESTful JSON Production Server Request Broker & Program Manager Runtime 11
Calling Functions Enterprise Application Production Server HttpClient object HTTP(S) Request Broker & Program Manager Calculation Process Calculation Process Worker Pool 12
Example - Integrating with IT systems Compiler SDK Web Applications Web Server Production Server Portfolio Optimization Pricing Desktop Applications Application Server Risk Analytics Database Server 13
Customer examples: Financial customer advisory service Production Server Global financial institution with European HQ Request Broker o Saved 2 million annually for an external system Algorithm Developers Compiler SDK Request Broker o Quicker implementation of adjustments in source code by the quantitative analysts Request Broker o Knowledge + = Build your own systems 15
Industrial IoT Analytics on AWS Global industrial equipment manufacturer Industrial Equipment Networked communication Embedded sensors Data reduction Production Server Request Broker Business Systems Users Compiler SDK Algorithm Developers 16
Building Automation IoT Analytics on Azure Global heavy duty electrical equipment manufacturer Building/HVAC automation control system Variety of sensors and controls Networked communication Data reduction Azure EventHub Azure Blob Production Server Request Broker Azure SQL Compiler SDK Business Systems Users Algorithm Developers 17
Front-end scalability Production Server Application server for Manage large numbers of requests to run short-running deployed programs Back-end scalability Distributed Computing Server Cluster framework for /Simulink Speed up computationally intensive programs on computer clusters, clouds, and grids.net Deployed Application Deployed Application Deployed Application Deployed Application 18
Distinct Offerings Scale Application Access and Computation Compiler SDK Deployed Application Desktop (client) Parallel Computing Toolbox code with batch, parfor, or other parallel constructs Request broker Deployed Application Deployed Application Deployed Application Deployed Application GPU Multi-core CPU Production Server Distributed Computing Server 19
Distinct Offerings Scale Application Access and Computation Compiler SDK Deployed Application Desktop (client) Parallel Computing Toolbox code with batch, parfor, or other parallel constructs Compiler SDK Request broker Deployed Application Deployed Application Deployed Application Deployed Application GPU Multi-core CPU Production Server Parallel workers on remote hardware Distributed Computing Server 20
and Distributed Computing Server allow you to speedup your computations on multiple CPUs and GPUs overcome memory limitations and offload computations to clusters and clouds. Desktop Clusters and Clouds 21
Distributed Computing Server Benefits Domain Expert Solution Architect Transparently get access to more workers on a cluster. Get access to advanced parallel workflows directly through their desktop. Don t need to build a framework for submission and retrieval. Can easily scale up with more workers Platform independency (win, mac, linux) Client connections can be protected with SSL encryption 22
Data Intensive: Big Data support in Datastores Apache Spark on Hadoop 11 26 41 12 27 42 13 28 43 15 30 45 16 31 46 17 32 47 20 35 50 21 36 51 22 37 52 Distributed Arrays Tall Arrays 23
Summary for tall arrays Local disk, Shared folders, Databases Run on Compute Clusters or Spark + Hadoop (HDFS), for large scale analysis Process out-of-memory data on your Desktop to explore, analyze, gain insights and to develop analytics Use Parallel Computing Toolbox for increased performance Distributed Computing Server, Spark+Hadoop 24
Technology Stack Data Analytics Business System Databases Distributed Computing Server Visualization Cloud Storage Azure Blob Production Server Web Request Broker IoT Custom App Public Cloud Platform Private Cloud 25
Online Resources Documentation Create and Share Toolboxes Website Desktop and Web Deployment Free White Paper Building a Website with Analytics Website Using With Other Programming Languages 26
MathWorks Consulting Services can help you! We can support you during the 3 different phases of the application development process Data Analysis Presentation Data Management Data Modeling & Computation Enterprise Deployment 27