Hortonworks DataFlow Sam Lachterman Solutions Engineer

Size: px

Start display at page:

Download "Hortonworks DataFlow Sam Lachterman Solutions Engineer"

Cassandra Rice
5 years ago
Views:

2 Disclaimer This document may contain product features and technology directions that are under development, may be under development in the future or may ultimately not be developed. Project capabilities are based on information that is publicly available within the Apache Software Foundation project websites ("Apache"). Progress of the project capabilities can be tracked from inception to release through Apache, however, technical feasibility, market demand, user feedback and the overarching Apache Software Foundation community development process can all effect timing and final delivery. This document s description of these features and technology directions does not represent a contractual commitment, promise or obligation from Hortonworks to deliver these features in any generally available product. Product features and technology directions are subject to change, and must not be included in contracts, purchase orders, or sales agreements of any kind. Since this document contains an outline of general product development plans, customers should not rely upon it when making purchasing decisions. 2 Hortonworks Inc All Rights Reserved

3 ONLY Hortonworks Company Profile Apache 100 open source TM % Hadoop data platform Founded in ST provider to go public HADOOP IPO 4Q14 (NASDAQ: HDP) 1,000+ subscription customers technology partners ~1100 employees across 2, countries 3 Hortonworks Inc All Rights Reserved

4 Accelerating Momentum Multi-Platform Expansion: HDP & HDF International Adoption Company Founded 2011 IPO Dec 2014 Fastest company to $100mm revenue 800+ customers and growing 4 Hortonworks Inc All Rights Reserved

Connecting Data Between Ecosystems Without Coding: 180+ Processors FTP SFTP HL7 UDP XML Hash Merge Extract Duplicate Encrypt Tail Evaluate Execute GeoEnrich Scan Replace Translate Split Fetch Convert

8 Connecting Data Between Ecosystems Without Coding: 180+ Processors FTP SFTP HL7 UDP XML Hash Merge Extract Duplicate Encrypt Tail Evaluate Execute GeoEnrich Scan Replace Translate Split Fetch Convert HTTP WebSocket HTML Image Syslog AMQP All Apache project logos are trademarks of the ASF and the respective projects. 8 Hortonworks Inc All Rights Reserved Route Text Route Content Route Context Control Rate Distribute Load Generate Table Fetch Jolt Transform JSON Prioritized Delivery

NiFi Integration of Many Things Like when the iphone was new, people compare NiFI to something they know, to create a frame of reference based on their experience iphone: a little of everything,

9 NiFi Integration of Many Things Like when the iphone was new, people compare NiFI to something they know, to create a frame of reference based on their experience iphone: a little of everything, integrated together People compare NiFi to a messaging bus (Kafka), ETL, scheduler (Oozie), CEP - Complex Event Processing (Spark/Storm), Visual UI flow designer, Forwarder (Splunk), while NiFI is an integration of many things and complements many different systems. Event Processing Messaging Bus ETL It is a telephone? Is it an client? Is it video camera? Is it digital camera? Is it a music player? Is it a gameboy? Scheduler GUI Forwarder 9 Hortonworks Inc All Rights Reserved

11 HDF: Visual Data Flow Management between Datacenter and Cloud H O R T O N W O R K S D ATA F L O W Dynamically adjust the pipeline Add and adjust data sources Visually trace the data path Add and Adjust Data Sources to maximize the opportunity that you capture from perishable insights Visually Trace the Data Path to manage the what, who, where and how around data in motion Dynamically Adjust the Pipeline to match the dataflow with your bandwidth Deliver Continuous Insights from real-time stream processing 11 Hortonworks Inc All Rights Reserved

12 Architectural Transformation Edge Analytics Machine Learning Edge Data Data in Motion (Cloud) Data at Rest (Cloud) Cloud Data in Motion Edge Data Closed Loop Analytics On-premises Data at Rest (on-premises) Data in Motion (on-premises) Data at Rest (on-premises) Deep Historical Analysis 12 Hortonworks Inc All Rights Reserved

Transaction logs CDC MSG MSG MSG n 2 1 Message broker Bulk

13 Attunity Replicate-HDF Architecture In-memory optimized metadata management and data transport Data Streaming Transaction logs CDC MSG MSG MSG n 2 1 Message broker Bulk Load Data Streaming MSG MSG MSG n 2 1 Message broker 2016 Attunity

Optimize Log Analytics Platforms with Content Based Routing Edge analytics for costeffective and efficient movement of machine data Intelligent, content based routing,

14 Optimize Log Analytics Platforms with Content Based Routing Edge analytics for costeffective and efficient movement of machine data Intelligent, content based routing, transformation and enrichment Send data to alternative systems based on value, content, priority Transformation Routing Enrichment 14 Hortonworks Inc All Rights Reserved

Apache NiFi What is data logistics? Guaranteed delivery Data buffering Backpressure Pressure release Prioritized queuing Flow-specific QoS Latency vs.

16 Apache NiFi What is data logistics? Guaranteed delivery Data buffering Backpressure Pressure release Prioritized queuing Flow-specific QoS Latency vs. throughput Loss tolerance Data provenance Recovery / recording a rolling log of fine-grained history Designed for extension Visual command and control Flow templates Policy-based security Clustering 16 Hortonworks Inc All Rights Reserved

17 Key Components of Streaming Analytics in Enterprise Environments Easy, Secure, Reliable Way to Get the Data You Need Because there s no data science without the data Flow Management Stream Processing Immediate and Continuous Insights Because acting on perishable insights in real time maximizes value Enterprise Services Provisioning, Management, Monitoring, Security, Audit, Compliance, Governance Because it all has to work together in an enterprise environment 17 Hortonworks Inc All Rights Reserved

18 Data in Motion Needs DataFlow Management and Stream Processing 1. Acquire truck telemetry events from different Geo Cloud Instances and from the edge (telemetry devices on certain trucks) 2. Move and Filter Data from Customer X Cloud Instances to on-premise instance 3. Intelligent Routing of data. The routing and filtering rules will be often changed at run-time. 4. Deliver the data data to various downstream systems. Flow Management 5. Parse the device data to standardized format that downstream sysem can understand 6. Enrich the data with contextual information HR, Payroll, Truck Maintenance records, weather 7. Join multiple sensor streams via attributes in real-time 1. E.g: Join separate streams like geo-location stream of a truck with the driver speed stream. 8. Detect complex patterns in real-time: 1. E.g: Over a 3 minute period, if a driver has multiple infractions across 2 different infraction types, then alert (Breaking, Speeding) 2. E.g: Over a 5 minute period, if the average speed of a driver is > 80 in known dangerous routes, then alert. Stream Processing 18 Hortonworks Inc All Rights Reserved

HDF Components HDF provides the flow management, stream processing, and enterprise

transformation and data routing Simple event processing End to end provenance Edge

Complex transformation Complex Streaming analytics Continuous Insights SAM UI

19 HDF Components HDF provides the flow management, stream processing, and enterprise services needed to collect, curate, analyze and act on data-in-motion across the data center and cloud. Flow Management Stream Processing Java Agent Data acquisition and delivery Simple transformation and data routing Simple event processing End to end provenance Edge intelligence & bi-directional comms Scalable data broker for streaming apps Scale out Complex transformation Complex Streaming analytics Continuous Insights SAM UI Enterprise Services Provisioning, Management, Monitoring, Security, Audit, Compliance, Governance, Multi-tenancy) SmartSense 19 Hortonworks Inc All Rights Reserved

Logical Architecture (Zero-Master Clustering)

quality validation based on schema Stream Processing Schema Registry usage in Stream Processing Schema evolution and

21 Use cases for Schema Registry Flow Management Schema Registry usage in Flow Management Generic/flexible format conversion Generic routing, SQL-based routing for any structured data Reusable schema, avoid schema overhead Data quality validation based on schema Stream Processing Schema Registry usage in Stream Processing Schema evolution and version compatibility Consumers and producers can evolve at different rates 21 Hortonworks Inc All Rights Reserved

22 What is Schema Registry? What Value Does it Provide? What is Schema Registry? A shared repository of schemas that allows applications to flexibly interact with each other - in order to save or retrieve schemas for the data they need to access What Value does Schema Registry Provide? 1. Data Governance Provide reusable schema (centralized registry) Define relationship between schemas (version management) Enable generic format conversion, and generic routing (schema validation) 2. Operational Efficiency To avoid attaching schema to every piece of data (centralized registry) Consumers and producers can evolve at different rates (version management) Data quality (schema validation) Example Use Register Schemas for Kafka Topics to be used by consumers of Kafka Topic (e.g: Streaming Analytics Manager) 22 Hortonworks Inc All Rights Reserved

23 Truck CSV data NiFi TailFile Attribute: Content: CSV bytes schema fingerprint SplitText Attribute: Content: CSV bytes schema fingerprint ExtractText ConvertFormat Attribute: Schema Version Schema name Schema Content: CSV bytes FetchSchema FromRegistry Attribute: Schema Version Schema name Content: CSV bytes Input CS Output CS CSV xml Avro JSON CSV xml Avro JSON 23 Hortonworks Inc All Rights Reserved

24 Truck CSV data NiFi TailFile Attribute: Without schema Attribute: No generic format conversion: only Content: SplitText JSON to Avro, CSV to Avro, etc Content: 1 to 1 mapping CSV bytes in conversion: 100-column CSV to 100-field CSV bytes JSON schema fingerprint schema fingerprint ExtractText ConvertFormat CovertFormat Input CS Output CS Attribute: Schema Version Schema name Schema Content: CSV bytes FetchSchema FromRegistry Attribute: Schema Version Schema name Content: CSV bytes CSV xml Avro JSON CSV xml Avro JSON With schema Generic format conversion: single processor + plugin controller services Flexible conversion: 100-column CSV to 5-field JSON 24 Hortonworks Inc All Rights Reserved

25 Streaming Analytics Manager What is it? A tool used to design, develop, deploy and manage streaming analytics app using a drag drop visualize paradigm Streaming Analytics Manager will allow you to build streaming analytics applications that do event correlation, context enrichment, complex pattern matching, analytical aggregations and create alerts/notifications when insights are discovered. The tool is agnostic to the underlying streaming engine and will support multiple streaming substrates (e.g: Storm, Spark Streaming, Flink) Extensibility is a first class citizen (add custom sinks, processors, spouts, etc..) Guiding Principle Write Complex Streaming Applications With No Code 25 Hortonworks Inc All Rights Reserved

26 Streaming Analytics Manager Key Capabilities Building Stream Apps using the following primitives Connecting to Streams Joining Streams Forking Streams Aggregations over Windows Stream Analytics Descriptive, Predictive, Prescriptive Rules Engine Transformations Filtering and Routing Notifications / Alerts Deploying Stream Apps Deploying the Stream App on a a supported streaming engine Monitoring the Stream App with app specific metrics. 26 Hortonworks Inc All Rights Reserved

29 DATAWORKS SUMMIT/HADOOP SUMMIT JUNE 13 15, 2017 San Jose McHenry Convention Center REGISTER NOW AND SAVE $1,000 REGISTER NOW! > dataworkssummit.com

Hortonworks and The Internet of Things

Hortonworks and The Internet of Things Dr. Bernhard Walter Solutions Engineer About Hortonworks Customer Momentum ~700 customers (as of November 4, 2015) 152 customers added in Q3 2015 Publicly traded