Hortonworks and The Internet of Things

Transcription

1 Hortonworks and The Internet of Things Dr. Bernhard Walter Solutions Engineer

2 About Hortonworks Customer Momentum ~700 customers (as of November 4, 2015) 152 customers added in Q Publicly traded on NASDAQ: HDP Founded in 2011 Original 24 architects, developers, operators of Hadoop from Yahoo! 800+ E M P L O Y E E S E C O S Y S T E M P A R T N E R S Hortonworks Data Platform Completely open multi-tenant platform for any app and any data Consistent enterprise services for security, operations, and governance Partner for Customer Success Leader in open-source community, focused on innovation to meet enterprise needs Unrivaled Hadoop support subscriptions

3 O P E N C O M M U N I T Y T H E I N N O VAT I O N A D VA N TA G E P R O P R I E T A R Y H A D O O P T I M E I N N O V A T I O N Hortonworks Is Genuinely Open Eliminates Risk of vendor lock-in by delivering 100% Apache open source technology Maximizes Community Innovation with hundreds of developers across hundreds of companies Integrates Seamlessly through committed co-engineering partnerships with other leading technologies M A X I M U M C O M M U N I T Y I N N O VAT I O N Page 3

4 Challenges of the IoT

5 The Internet of Things is Driving Growth of Data Much of the new data Internet of Anything Sensors and machines Geolocation Server logs The Opportunity Unlock transformational business value from new data sources data from the Internet of Any Thing NEW exists in-flight between systems comes with a high frequency has little structure Clickstream delivers value only in high volumes Web & social Files & s Traditional Data Sources ERP, CRM, SCM TRADITIONAL Page 5

6 Unique Challenges of the Edge IoT often operates on data flow edges that require a small footprint and operate on low power Bandwidth is limited and high latency can be the norm Access to data often exceeds the bandwidth to transmit it PRIORITIZE GATHER DELIVER Agents need recoverability capacity Must be secured across both the data plane and control plane Track from the edge Through the datacenter Page 6

7 Challenges of the Internet of Things IoT creates new dataflow challenges of communications, distribution, power and footprint Businesses need trusted, realtime access to data and analysis capabilities Page 7

8 How Hortonworks Supports IoT

9 HDP and HDF: A Connected Data Platform for IoT INTERNET OF ANYTHING HORTONWORKS DATAFLOW (powered by Apache NiFi) HORTONWORKS DATA PLATFORM (powered by Apache Hadoop) Page 9

10 Hortonworks IoT Dataflow: From Data to Analysis to Action SOURCES REGIONAL INFRASTRUCTURE CORE INFRASTRUCTURE Constrained High-Latency Localized Context Hybrid Cloud/On-Premise Low-Latency Global Context Page 10

11 Operationally Effective for IoT, Proven at Scale Visual User Interface Drag and drop for efficient, agile operations Immediate Feedback Start, stop, tune, replay dataflows in realtime Adaptive to Volume and Bandwidth Any data, big or small Provenance Metadata Governance, compliance & data evaluation Powered by Apache NiFi Secure Data Acquisition & Transport Fine grained encryption for controlled data sharing and selective data democratization Page 11

12 Drag a processor to the canvas Page 12

13 Filter all processors Page 13

14 Configure the selected processor Page 14

15 Configure the selected processor Page 15

16 Add a Kafka target processor Page 16

17 Drag a connection from one processor to the other Page 17

18 Configure connection Page 18

19 The simple flow Page 19

20 The more realistic flow Page 20

21 And in action Page 21

22 HDF: Designed for IoAT (Internet of Any Thing) Securely move data of any size - between source and destination Reliably deliver data to real-time applications and storage platforms in point-to-point and bi-directional data flows Seamlessly adapt to fluctuating conditions of bandwidth, latency, throughput with built in recoverability Flexible, scalable and extensible to accommodate variability and change Page 22

23 Hadoop is the Perfect Data Platform for IoT Any Data, Any Volume Existing and new datasets Business systems Any Application Batch, interactive, and real-time Existing applications New analytics Partner applications Batch Interactive Real-time Anywhere Complete range of deployment options Clickstream Web/ social Geo location YARN: data operating system Commodity Appliance Cloud Internet of things Server logs Files, s Page 23

24 Map Reduce Apache Hive Apache Pig Apache HBase Apache Accumulo Apache Solr Apache Spark Apache Storm ISV Engines Hortonworks Data Platform 2.3 GOVERNANCE DATA ACCESS SECURITY OPERATIONS Apache Atlas Apache Ranger Apache Ambari Apache Falcon Apache Knox Cloudbreak Apache Kafka Apache Flume YARN: Data Operating System (Cluster Resource Management) Apache Atlas HDFS Encryption Apache ZooKeeper Apache Oozie Apache Sqoop HDFS 1 (Hadoop Distributed File System) Deployment Page 24

25 Example: IoT Data Analysis Batch and Real-Time INTERNET OF ANYTHING Guaranteed, dynamic, secure data collection HDF Fast, scalable durable queuing Kafka Exploration Machine learning Model creation BATCH Hive / Spark Access of SoE, Low latency, Multi concurrency HDFS / Hbase / Solr SERVING Clients Storm / Spark Streaming Other Databases SPEED Fast, realtime transformation & analytics Page 25

26 Hortonworks in IoT Page Hortonworks 26 Inc Hortonworks All Inc. Rights 2011 Reserved All Rights Reserved

27 Snapshot telemetry data for usage-based insurance products Page 27

28 Page 28

29 HDP and HDF: A Connected Data Platform for IoT ACTIONABLE INTELLIGENCE PERISHABLE INSIGHTS HISTORICAL INSIGHTS INTERNET OF ANYTHING DATA IN MOTION DATA AT REST

30 Early Bird All Access Pass

31 Early Bird All Access Pass

32 Thank You Page 32