MariaDB MaxScale 2.0 and ColumnStore 1.0 for the Boston MySQL Meetup Group Jon Day, Solution Architect - MariaDB

Transcription

1 MariaDB MaxScale 2.0 and ColumnStore 1.0 for the Boston MySQL Meetup Group Jon Day, Solution Architect - MariaDB 2016 MariaDB Corporation Ab 1

2 Tonight s Topics: MariaDB MaxScale 2.0 Currently in Beta MariaDB ColumnStore Currently in Alpha

3 MariaDB Corporation Company Overview

4 MariaDB Highlights Founded by Original MySQL team Michael Monty Widenius and David Axmark Venture Capital - Intel Ventures Driving Innovation & Committed to Open Source Red Hat & Major Linux Distributions Standardized on MariaDB Increasing OEM/MariaDB Embedded Software Solutions

5 The MySQL/MariaDB Timeline Sun Buys MySQL AB Oracle Buys Sun 2014 MariaDB 10.0 fork of MySQL 5.5 SkySQL and MariaDB Merger 2014 MariaDB default in Red Hat & other Linux Distributions 2015 MaxScale 1.0 GA Release 2016 MariaDB ColumnStore 6

6 MariaDB MaxScale 2.0 (Beta) Product Overview

7 MariaDB MaxScale Application-to-Database Insulates client applications from the complexities of backend database cluster Database-to-Database Simplifies interoperability across databases Secure Your Data Scale for Growth Ensure Availability Manageability 7

8 MariaDB MaxScale concept An Intelligent Data Gateway (IDG) Decouple applications from database deployment environment Improve availability without adding application complexity Improve data security Handle scale-out issues Add flexibility without burdening every application Enable data replication from OLTP databases to external data stores Improve database scalability Remote data disaster recovery Real-time data streaming to OLAP/DW and Big Data stores Copy data to other applications, QA databases

9 What is MariaDB MaxScale? A flexible data gateway for scalability, high availability, security, interoperability and migration beyond MySQL and MariaDB Highly configurable gateway platform Database Aware Pluggable Architecture

10 MariaDB MaxScale Database aware Understands the database environment Is aware of the state of the database components Understands the data that flows through it Routes requests based on a combination of Defined algorithms Component state Request contents Session state

11 MariaDB MaxScale - Core Provides core services for Configuration Networking Scheduling Query classification Logging Buffer management Plugin loading Request flow Designed to make plugins easy to write

12 MariaDB MaxScale Pluggable architecture Generic Core Flexible, easy to write plugins for Protocol support Database monitoring Query Transformation and Logging Load balancing and Routing Authentication

13 MariaDB MaxScale Flow of requests Client Application Protocol Monitor Filter Filter Router Protocol Router Protocol

14 MariaDB MaxScale Use Cases

15 MariaDB MaxScale Classic Load Balancing Connection based routing Low overhead Balances a set of connections over a set of servers Uses monitoring feedback to identify master and slaves Connection weighting if configured Load balances queries in round robin across configured servers

16 MariaDB MaxScale Read Load distribution Can be used for master-slave replication or master-master replication environments Two approaches possible Either using connection routing with separate read connections Or statement routing, classify the statements to read, write or session modification Monitor identifies the master and slave nodes Simplifies applications Cluster configuration, Node failures transparent to applications

17 MariaDB MaxScale Schema Sharding Multi-tenant database hosting MaxScale Each tenant with its own schema Sharding Router Multiple schema per shard Schema Sharding Router Shard 1 Shard 2 Shard 3 Shard 4 Shard 5 All applications connect to single MaxScale MaxScale routes the shard server on based on query from client No impact on existing client application New client or shard server is added Scale the database environment as user base and data volume grows Without impacting existing user base

18 MariaDB MaxScale Database firewall Block queries matching rules for specified users Multiple ordered rules Match on and block queries with certain patterns Date or time WHERE clause Wildcard or regular expression Query failed: 1141 Error: Required WHERE/HAVING clause is missing SELECT * FROM CUSTOMER WHERE id = 5; SELECT * FROM CUSTOMERS; MaxSc ale Data ba se Fire w l Filt al er Block queries that match a set of rules rule safe_select deny no_where_cla use on_queries select rule safe_customer _select deny regex '.*from.*custo mers.*'

19 MaxScale 2.0 New Capabilities 19

20 Data Streaming Provide real time transactional data to data lake environment for machine learning or real-time analytics. Capture change data in the binary log events and replicate the events from MariaDB to Kafka producer in real-time from Master to slave to offload the replication load from master Master Binary log events Avro or JSON events MaxScale Binlog, Avro, JSON Binlog, Avro, JSON Slaves Slaves Data Warehouse 20

21 MaxScale 2.0 New Capabilities Better Security Transport layer security with end-to-end SSL through MaxScale MaxAdmin security improvements to enable configurable prevention of remote access Connection rate limitation feature to protect against DDoS attacks 21

22 MaxScale 2.0 New Capabilities High Availability Minimize downtime with read mode for MariaDB/MySQL master-slave clusters 22

23 High Availability Ensure High Availability with no single point of failure 4 1 CHANGE MASTER to new_master; START SLAVE Master Ensure database uptime Automatic failover No impact on read transaction when master fails Minimize database downtime Database upgrade without impacting user experience 3 STOP SLAVE Failover Script 2 master_down event binlog cache Slaves 4 Promote as master 23

24 Business Source License

25 MariaDB ColumnStore (Alpha) Product Overview

26 MariaDB ColumnStore Massively parallel, distributed data engine for powerful analytics on big data Price to Performance at Scale Unified Simplicity Data Analytics using SQL or SPARK SQL Scaling to petabytes of data Read performance scales linearly with data growth Built-in high availability at access and data layers Exceptional performance Transactional and Analytics processing under the same roof Encryption for data in motion, role based access and audit features Simplified installation, management maintenance, and scaling Same interface as MariaDB Attaches to wide range of BI tools Real-time response to analytics queries and high speed data loading Open-source GPL2 26

27 Brief History Created by Calpont originally as InfiniDB no longer in business GPL License several years ago Calpont was a MariaDB partner, we brought developers, managers and support staff onboard Older MySQL 5.1 front end 27

28 MariaDB ColumnStore Architecture User Module : Processes SQL Requests Performance Module : Multi Threaded Distributed Processing Engine Clients User Connections User Modules Performance Module 1 Performance Module 2 MariaDB SQL Front End Performance Module 3... Columnar Distributed Data Storage Performance Module N Distributed Query Engine Local Disks, SAN, EBS, GlusterFS, HDFS 28

29 MariaDB ColumnStore 1.0 Performance Columnar Storage, multi-threaded and Massively Parallel distributed execution engine High Availability Built in redundancy and high availability Scale Linear scalability Analytics In database analytics with Complex and Cross Engine JOINs Windowing functions and UDFs Out of box BI Tools connectivity, Analytics integration with R Ease of Use ANSI SQL compatible ACID compliant No indexes, No materialized views No manual partitioning Data Ingestion High speed parallel data load and extract Create Table as Select, Like -- locally, cross database joins, or over ODBC Security SSL support, Audit Plugin, Authentication Plugin, Role Based Access Deployment Options On premise, AWS Supports local disks, SAN, HDFS, GlusterFS 2016 MariaDB Corp 29

30 Client Access ODBC/JDBC MariaDB/MySQL Connectors BI tools

31 Row-Oriented vs Column-Oriented Row-oriented: rows stored sequentially in a file Key Fname Bugs Yosemite Daffy Elmer Witch Lname Bunny Sam Duck Fudd Hazel State NJ CT IA CT CT Zip Phone (123) (234) (345) (456) (567) Age Sales Column-oriented: each column is stored in a separate file Each column for a given row is at the same offset. Key Fname Bugs Yosemite Daffy Elmer Witch Lname Bunny Sam Duck Fudd Hazel State NJ CT IA CT CT Zip Phone (123) (234) (345) (456) (567) Age Sales

32 Data Storage Vertical Partitioning by Column Each column in its own column file Only do I/O for columns requested Horizontal Partitioning by range of rows Logical grouping of 8 million rows of each column file In-memory mapping of extent to physical layer Physical Layer Logical Layer Table Column1 Extent 1 (8MB 64MB 8 million rows) Server DB Root ColumnN Extent N (8MB 64MB 8 million rows) Segment File1 (Extent) Segment FileN (Extent) Blocks (8KB)

33 Data Storage - Extents and PMs Extent 7 Extent 8 Extent 5 Extent 6 Extent 5 Extent 6 Extent 7 Extent 8 Extent 3 Extent 4 Extent 1 Extent 2 Extent 3 Extent 4 Extent 1 Extent 2 PM 1 PM 2 PM 3 PM 4 PM 1 PM 2 Extent Map In memory meta-data of an extent s min, max value for a column, extent s physical block offset and PM on which the extent resides

34 Data Storage - Local Disks Each PM nodes stores data on local disk No PM node can access the data on another PM node Shared Nothing No data redundancy

35 Data Storage - SAN Each PM node is attached to a set of volumes on SAN called DBRoots Upon failure of PM node, another PM attaches to the failed PM s DBRoots Shared nothing during running state No data redundancy

36 Data Storage - GlusterFS Distributed file system Software based storage system GlusterFS runs on every PM node Creates distributed file system with each PM node s local disks and network interface across PM nodes Data redundancy across multiple nodes Automatic data failover Data availability during failover and failback

37 Data Storage - EBS Dynamic scaling to handle variable workloads Data layer high availability with Elastic Block Store (EBS)

38 Data Ingestion Bulk data load cpimport : CSV and Binary LOAD DATA INFILE: CSV Apache Sqoop Integration: Integration with cpimport and sql interface Future Release Data Streaming from MariaDB/MySQL database to MariaDB ColumnStore cluster via Kafka Avro data record

39 Data Ingestion - cpimport Fastest way to load data Load data from CSV file Load data from Standard Input Load data from Binary Source file Multiple tables in can be loaded in parallel by launching multiple jobs Read queries continue without being blocked Successful cpimport is auto-committed In case of errors, entire load is rolled back

40 Data Ingestion - LOAD DATA INFILE Traditional way of importing data into any MariaDB storage engine table Up to 2 times slower than cpimport for large size imports Either success or error operation can be rolled back

41 MariaDB ColumnStore on Hadoop Native scoop integration Runs on existing Apache Hadoop hardware SQL access to Apache Hadoop data libhdfs integration Batch Processing High Performance analytics Pig/Hive HBase Map Reduce Hadoop Distributed File System MariaDB ColumnStore

42 MariaDB ColumnStore on AWS Automated cluster installation on AWS Dynamic scaling to handle variable workloads Data layer high availability with Elastic Block Store (EBS)

43 Use Case: Scaling Big Data Analytics MariaDB ColumnStore OLAP Solution Business Challenge An organization is generating large amount of operational data Multiple terabytes of historical data With growth in business and in operational data Analytics query performance degrades Impractical to do analytics , GB Put past data into MariaDB ColumnStore As data grows Add new node(s) MariaDB ColumnStore 1.0 Perform analytics without performance degradation Linear Scalability with data growth 1,000, ,000,000 10,000,000, ,000,000, GB 1-10TB TB...PB Rows/DataSize Scope MariaDB Enterprise OLTP MariaDB ColumnStore OLAP Harvest new value from large historical datasets by deriving new insights Support growth in your business, whileinternal continue to deliver high service levels for data analytics Only 43

44 Sizing Minimum Spec: UM: 2 GHz, 4 core, 32 G RAM PM: 2 GHz, 4 core, 16 G RAM Typical Server spec UM: 8 core, 64G to 256G RAM PM: 8 core 64G RAM Data storage: External data volumes: Maximum 2 data volume per IO channel per PM node server up to 2TB on the disk per data volume Max 4 TB per PM node Local disk Up to 2TB on the disk per PM node server 44

45 Sizing - Example Initial DB 60TB uncompressed data = 6TB compressed data at 10x compression 2UM - 8 core, 128G to 256G (based on workload) PM: 8 core 64G RAM 6 TB compressed = 3 data volume (at 2TB per volume) with 1 data volume per PM node - 3PMs Data growth - 2TB per month, Data retention - 2 years Plan for 2TB X24 = 48 TB additional 48 TB = 4.8TB compressed 3 data volume(at 2TB per volume) with 1 data volume per PM node - 3 additional PMs Total 6 PMs, 2 UMs 45

46 MariaDB Solution for Big Data Analytics High performance data management solution for big data analytics Descriptive Analytics What is Happening? Predictive Analytics What is likely to happen? Diagnostic Analytics Why did it Happen? Prescriptive Analytics What should I do about it? Analytics Insight Connectors, SPARK Integration etc Data Processing UM Data Collection UM ETL Tools Mobile PM Node 1 Biometrics MariaDB MaxScale Sensors PM Node 2 PM Node 3... MariaDB ColumnStore MariaDB ColumnStore. Social Media Transactional, Operational PM Node N 46

47 Thank you! Jon Day Solution Architect MariaDB Corporation 47