In-Memory Performance Durability of Disk GridGain Systems, Inc.

Transcription

1 In-Memory Performance Durability of Disk

2 Apache Ignite In-Memory Hammer for Your Data Science Toolkit Denis Magda Ignite PMC Chair GridGain Director of Product Management

3 Agenda Apache Ignite Overview Use Cases Data Science Toolkit Box Data Grid Durable Memory Distributed SQL Compute Grid Machine Learning Grid (Beta) Q&A

4 Apache Ignite In-Memory Computing Platform Financial Services Telco Travel & Logistics E-Commerce Pharma & Healthcare IoT SQL Key/Value Transactions Compute Services Streaming ML Memory-Centric Storage Ignite Native Persistence (Flash, SSD, Intel 3D XPoint) Third-Party Persistence (RDBMS, HDFS, NoSQL)

5 Apache Ignite Users Financial Services Software Logistics & Travel E-commerce Telco FinTech IoT Pharma & Healthcare Adtech

6 - Drug Discovery and Network Biology e-therapeutics provides a computer-based drug discovery platform and a specialized approach to network biology. e-therapeutics Platform Problem Analysis of a network of proteins influencing a disease and drugs discovery could be measured in weeks Could not parallelize existing algorithms Cache & ComputeAPI Apache Ignite Solution 80x speed increase over the non-parallelized environment Analysis projects completion in hours and minutes Computational resources for abandoned research projects s Clients Nodes 100x Cluster Nodes 5x Physical Nodes

7 Data Grid JCache & SQL JCache Transactions Compute SQL ACID Transaction Distributed partitioned hash map Distributed Key-Value Store DURABLE MEMORY DURABLE MEMORY DURABLE MEMORY RDBMS Dynamic Scaling HDFS NoSQL 3rd party storage caching

8 Durable Memory Automatic Defragmentation Predictable memory consumption Ignite Cluster DURABLE MEMORY DURABLE MEMORY DURABLE MEMORY Off-heap Removes noticeable GC pauses Fully Transactional (Write-Ahead Log) Stores Superset of Data Instantaneous Restarts

9 Ignite Native Persistence 1. Update 2. Persist Write-Ahead Log 3. Ack RAM 4. Checkpointing Partition File 1 Partition File N

10 Distributed SQL Cross-platform Compatibility Java.NET C++ BI Tools DDL, DML Support JDBC ODBC SQL API SELECT, UPDATE, INSERT, MERGE, DELETE, CREATE and ALTER Apache Ignite Cluster Indexes in RAM or Disk DURABLE MEMORY DURABLE MEMORY DURABLE MEMORY Dynamic Scaling

11 Compute Grid Zero Deployment Load Balancing C1 DURABLE MEMORY C = Compute C = C1 + C2 R1 R = Result in T/2 time R = R1 + R2 C2 R2 DURABLE MEMORY Automatic Failover Ignite Cluster

12 Client-Server Processing Co-located Processing 2 Data Client Node ON-DISK 1 Client Node ON-DISK 2 Data ON-DISK 3 ON-DISK 1. Initial Request 2. Fetch data from remote nodes 3. Process entire data-set 1. Initial Request 2. Co-located processing with data 3. Reduce multiple results in one

13 Genetic Algorithm Grid Biological Evolution Simulation Chromosome and Genes Cluster F1, C1, M1 DURABLE MEMORY F = F1 + F2 C = C1 + C2 M = M1 + M2 DURABLE MEMORY F = Fitness Calculation C = Crossover M = Mutation Collocated Computation F2, C2, M2 Ignite Cluster

14 Machine Learning Grid R C++ Python Java Scala REST Multi-Language Support Distributed Algorithms K-Means Regressions Decision Trees Random Forest Distributed Core Algebra DURABLE MEMORY DURABLE MEMORY DURABLE MEMORY Dense and Sparse Algebra Large Scale Parallelization No ETL

15 Any Questions? Thank you for joining us. Follow the conversation. #apacheignite #denismagda