Get Data, Build Apps and Analyze Data Using IBM Bluemix Data and Analytics (Session 6748)

Transcription

1 Get Data, Build Apps and Analyze Data Using IBM Bluemix Data and Analytics (Session 6748) Eric Cattoir Yves Debeer Bert Waltniel 2015 IBM Corporation

2 IBM Bluemix The Digital Innovation Platform

3 Innovation is the new currency Two guys in a Starbucks can have access to the same computing power as a Fortune 500 company. Jim Deters Founder, Galvanize

4 4 Anatomy of a Disruptive Idea

5 To really disrupt, a business should focus on building differentiation and rent the rest Devs can quickly compose apps with new APIs and digital services to add features and increase engagement in areas like: Analytics, cognition Mobile, location Internet of Things Social engagement Identity Reviews Travel Messaging His/her company s private APIs and services

6 Bluemix works for disruptors.

7 Bluemix started as a public PaaS Bluemix started with a major focus on developer productivity in the public cloud. Customer Managed Service Provider Managed Infrastructure as a Service Platform a Service as Code Code Data Data Runtime Runtime Middleware Middleware OS OS Virtualization Virtualization Servers Servers Storage Storage Networking Networking IBM SoftLayer 7

8 We listened. Now we re evolving to become even more flexible. Capabilities in Bluemix now span PaaS and IaaS and can be delivered as a public, dedicated, or on-premises* implementation. Customer Managed Service Provider Managed Infrastructure as a Service Platform a Service as Code Data Runtime Middleware OS Virtualization Servers Storage Networking Code Data Runtime Middleware OS Virtualization Servers Storage Networking Built on technologies: IBM SoftLayer 8

9 How does Bluemix work? + Always focused on what s next Bluemix is underlined by three key open compute technologies: Cloud Foundry, Docker, and OpenStack. It extends each of these with a growing number of services, robust DevOps tooling, integration capabilities, and a seamless developer experience. DevOps Tooling Your Own Hosted Apps / Services Integration and API Mgmt Catalog of Services that Extend Apps Functionality Web Data Mobile Cognitive Analytics IoT Security Yours + Flexible Compute Options to Run Apps / Services Instant Runtimes Containers Virtual Machines Platform Deployment Options that Meet Your Workload Requirements + Bluemix Bluemix Bluemix Public + 9 Powered by IBM SoftLayer In Your Data Center *Bluemix Local coming Summer 2015

10 Bluemix is built on IBM SoftLayer The highest performing cloud infrastructure available. A different kind of data center Every location designed, built, and operated to the same standardized, pod based spec 24/7 on-site security and rigorous controls Expanding to 40 data centers worldwide Global network of networks Public, private, and management networks all separate More than 2,000Gbps between data centers and network points of presence (PoPs) Unmetered inbound public bandwidth and fully unmetered bandwidth between data centers Entirely automated London (now) SoftLayer API controls everything - more than 3000 documented methods and 180 distinct services Bare metal and virtualized servers in the same platform Dallas Bluemix Public Location SoftLayer Data Center 10

11 11 Sign up in minutes. Pay for what you use. Cloud based pricing models to serve developer needs. Friction free adoption 30 day trial (no credit card required) - Designed to allow testing of an entire application on the platform Free tier for every service - encourages experimentation of new services for applications already running on Bluemix Multiple Commitment Models Pay-as-you-go - optimized for flexibility, no term commitment Subscription - term based optimized for cost, discounted from pay as you go rates Self Service Zero to coding in less than 5 minutes Credit card over the web in many countries - or through your IBM rep

12 Let s see it! *click* 12

13 Business Agility through Data Traditional Requirements Based Top-Down Design Integration and Reuse Competence Centers Better Decisions Enterprise Focus Agile Data Analytics Opportunity-Oriented Experimentation Throwaway Hackathons Business Innovation Functional Focus 13

14 But How?

15 Agility and Elasticity through Cloud Elastic Provisioning Pay-as-You-Go Manage High Volume External Data Sources Self-Service Through a Browser SQL / NOSQL Unstructured Data Access Data Anywhere, Anytime Leverage Current Cloud Apps 15

16 IBM Data and Analytics Services Open for Data A comprehensive portfolio of open source data services

17 Work with Cloud Data Services in Bluemix Iterate Sensors Internet Social Media Customer Conversations GET DATA PUT DATA TO WORK ANALYZE DATA INTERACT GAIN INSIGHT VISUALIZE Internal & External Data Sources DataWorks Cloudant MongoDB Notebooks Predictive Analytics Back Office Applications Apache Kafka Streams DB2 Postgres ReThinkDB Redis Object Storage Graph DB DashDB Your Own Data & Analytics Applications 17

18 Example: Health Management Platform Clinical & Wearable Device Sensors Fitbit, Jawbone Device Data Lab Results Patient Conversations GET DATA PUT DATA TO WORK ANALYZE DATA INTERACT GAIN INSIGHT VISUALIZE Internal & External Data Sources Health Results from RDBMS DataWorks Streams Cloudant DashDB Notebooks 18

19 Get Data from Own or Public Data Import Data into data services e.g. dashdb, Cloudant, Mongo, through respective load tools Create Connections to diverse data sources on-premise or cloud for use in analytics e.g. Notebooks Load Data from diverse sources into cloud data services in-context, powered by DataWorks X

20 Get Data from Bluemix s Analytics Exchange Explore available data sets Find interesting data Access data from Bluemix apps Analyze Data in Apache Spark & Notebooks Dash DB Watson Analytics X

21 Build Applications using Bluemix Data Services Connect your applications to use Data in Bluemix Select the Database Service Instance you want to use and pick your plan (upper right) Get service credentials to use in your code (lower right) Use the APIs, passing the credentials you obtained from Node.js, Liberty, or other apps on Bluemix or from apps running on other platform or devices Manage from the context of Bluemix under Bluemix login X

22 Analyze Data with IBM Analytics for Spark Go to Work with Data -> Analytics and create a new service instance Interactive Notebooks Use Python, Scala with Spark Associate an Object Storage for accessing&uploading of Data Connect to Data Sources e.g. Files, Cloudant, DashDB, on-premise DBs, Spark Submit Download Apache Spark Submit Develop your own Spark Jobs Run and monitor your Jobs + X

23 Example: New York Accidents Analytics NY City Public Data Accident Data from NYPD Road Condition Data Weather History Data We created a set of Notebooks to Cleanse Data to get it in proper shape for visualization and analytics Visualize Data to better understand its content and structure Analyze Data to identify patterns and correlations in the data Predict future Incident Likelyhoods from data Visualize Insights from descriptive and predictive analytics X

24 X

25 X

26 X

27 X

28 X

29 X

30 X

31 X

32 X

33 X

34 Spark Technical Discussion X

35 Resilient Distributed Datasets (RDDs) A collection of elements that Spark works on in parallel. May be kept in memory or on disk. Applications can also explicitly tell Spark to cache an RDD, which is great for iterative algorithms. An RDD contains the raw data, plus the function to compute it. Fault-tolerance: if any partition of an RDD is lost, it will automatically be recomputed using the transformations that originally created it. RDD built from a Java collection RDD built from an external dataset (local FS, HDFS, Hbase, )

36 Working with RDDs: Transformations and Actions Transformations are lazy: they do not compute their results right away. They are added to the operations of the RDD optimize the required calculations recover from lost data partitions Examples: map(func), filter(func), union(), join(), groupbykey Actions are executed immediately, and trigger execution of all prior transformations on an RDD Examples: reduce(func), collect(), saveassequencefile() func are Java/Scala/Python functions that you write Call persist() on an RDD if you plan to reuse it later

37 Execution Model spark-submit Executor Driver Program Executor notebook Executor X

38 Spark in Action Word Count in Scala val conf = new SparkConf().setAppName( WordCount ) val sc = new SparkContext(conf) 1 RDD = 1 line of the document val file = sc.textfile( swift://filecontainer.spark/input.txt ) val words = file.flatmap(line => tokenize(line)) Transformations val wordmap = words.map(x => (x, 1)) Action val wordcounts = wordmap.reducebykey(_ + _) wordcounts.saveastextfile( swift://filecontainer.spark/output.txt ) Tokenize is def tokenize(text : String) : Array[String] = { text.tolowercase.replaceall("[^a-za-z0-9\\s]", "").split("\\s+") } // Adapted from Word Count example on

39 X

40 DataFrames X

41 Combine Services: Analytics of Twitter Data Node.js Reader receives Twitter data stream and writes to Topic Reader Node.js Message Hub provides elastic high velocity message queue Topic Kafka Algorithms in Scala detect Tweets of interest Notebook Spark Streaming Watson Tone Analyzer Watson enriches Tweets with tone & sentiment info Insight App lets users explore and interact with results Insights App Node.js REST API Node.js Results Cloudant DB Cloudant stores insight data with HADR at scale X

42 Streaming Analytics using multiple Data Sources Insight App Node.js Notebook REST API Node.js Reader Node.js Topic Kafka Results CloudantDB Reader Node.js Topic Kafka Spark Streaming Stock Quotes Reader Node.js Topic Kafka Topic Kafka Alert Gen Node.js Push Service Watson Tone Analyzer Predictive Analytics X

43 Conclusion You can achieve greater Business Agility and faster Insights through Cloud based Innovation without upfront investment IBM Cloud Data Services provide open, cloud based data and analytics services that enable fast cloud based innovation Bluemix - Data & Analytics at features and integrates cloud data services, enabling you to Get Data from your own or public data sources Build Applications using cloud data & analytics services Analyze Data with Spark&Notebooks at Hadoop, dash DB, Combine and Integrate cloud data & analytics services with each other, as well as with other Bluemix services, e.g. through the new Message Hub based on Apache Kafka X

44 Combine Services: Analytics of Twitter Data 19

45 clouddataservices/sentiment-analysis-oftwitter-hashtags/ 20

46 Thank You