Apache Ignite and Apache Spark Where Fast Data Meets the IoT

Transcription

1 Apache Ignite and Apache Spark Where Fast Data Meets the IoT Denis Magda GridGain Product Manager Apache Ignite PMC #apacheignite #denismagda

2 Agenda IoT Demands to Software IoT Software Stack Device OS/RTOS Data Collection and Enrichment HTAP Platform Application APIs Demo

3 IoT Demands to Software

4 IoT Demands to Software Real-time ingestion Real-time processing Time-series support Flexible Querying APIs SQL Full-text search Geo-spatial Analytics BI Machine Learning High-availability Simple scalability

5 IoT Software Stack

6 IoT Software Stack Application APIs HTAP Platform Data Collection and Enrichment Device OS / Real-Time OS

7 Apache IoT Software Stack Application APIs HTAP Platform Data Collection and Enrichment Device OS / Real-Time OS

8 Device OS/RTOS

9 Open Source RTOS Cortex M0-M4 MIPS & RISC-V Networking Bluetooth Low Energy Wi-Fi TCP/IP & UPD Apache MyNewt Build & Package Management Secure bootloader and signed images Remote Firmware Upgrade

10 Data Collection and Enrichment

11 Data Collection and Enrichment Spark Streaming Fault-Tolerant Streams Processing Data Collection Sockets Kafka Flume Data Enrichment Spark API Data Storage Apache Ignite

12 HTAP Platform

13 Use Case: Smart Metering and Utilities delivers a comprehensive IOT platform SilverSpring Requirements Migrate to in-memory processing Add scalability & elasticity Use open source technologies SilverSpring Solution SaaS Architecture Multi-Tenancy Advanced Security Strong compute capabilities Co-located in-memory processing Demonstrated best On-demand elasticity & scalability ANSI-99 SQL Support Transactional consistency 50+ Million Meters SilverSpring IoT Platform GridGain Security

14 Apache Ignite In-Memory Data Fabric Supports Applications of various types and languages Open Source Apache 2.0 Simple Java APIs 1 JAR Dependency High Performance & Scale Automatic Fault Tolerance Management/Monitoring Runs on Commodity Hardware Supports existing & new data sources No need to rip & replace

15 Distributed Key-Value Data Store Data Reliability High-Availability Active replicas, automatic failover Data Consistency ACID distributed transactions In-Memory Data Grid

16 In-Memory Data Grid: Partitioning Partitioned Cache Replicated Cache

17 Ignite Data Streamer Fastest Data Injection Automatic data partitioning Stream Receivers Custom logic execution Additional data transformation Stream Adapter Socket Kafka Flink RocketMQ etc. Streaming to Ignite

18 Application APIs

19 ANSI-99 SQL Geo-spatial Queries Full-text Search Always Consistent Fault Tolerant Cross-Platform Apache Ignite SQL Grid JDBC and ODBC drivers DML (INSERT, UPDATE, etc.) DDL (CREATE, DROP, etc.)

20 Apache Ignite SQL Grid: Queries Distributed Joins Automatic Group By, Aggregations, Sorting Cross-Cache Joins, Unions, etc. Ad-Hoc SQL Support

21 Apache Ignite Compute Grid Direct API for MapReduce Direct API for ForkJoin Zero Deployment State Checkpoints Load Balancing Automatic Failover

22 IgniteRDD Share RDD across jobs on the host Share RDD across jobs in the application Share RDD globally Faster SQL In-Memory Indexes SQL on top of Shared RDD Spark Shared RDDs

23 Machine Learning Grid

24 Demo

25 Documentation: Resources Apache Ignite: Apache Spark Streaming: Ignite and Spark Integration: Apache MyNewt: Demo Source Code:

26 ANY QUESTIONS? Thank you for joining us. Follow the conversation. #apacheignite #denismagda