The growing global data platform market

Transcription

1

2 OSS DB on Azure

3 The growing global data platform market Global Data Platform Market is growing at 11.2% CAGR Growth is expected to exceed $100B in FY22 Primary growth is driven by relational workloads (on-premise & cloud) Sustained double-digit growth is expected through adoption of newer architectures and data models (Hadoop, NoSQL) along with more investment in predictive analytics, ML, and AIlike capabilities FY17 FY18 FY19 FY20 FY21 FY22 Traditional, Relational Data Advanced, Predictive Analytics Distributed Architectures / Data Models Other Source: IDC + Internal Assessments

4 Savings opportunity for modernizing your data estate is significant Platform (as a Service) On-premise costs tends to be driven by hardware and data center management costs On-premises Applications Infrastructure (as a Service) Applications Intelligent performance/security Applications Infrastructure-as-a-Service reduces cost categories related to data center and compute Data Data Data Platform-as-a-Service off-loads customers most administrative tasks to Azure, further improving efficiency with machine-learning capabilities for performance and security High availability /DR/Backups Database Provision/ Patch/Scaling O/S provision /patching High availability /DR/Backups Database Provision/ Patch/Scaling O/S High Availability/ DR/Backups Database Provision/ Patch/Scaling O/S The ML capabilities are key technology differentiation of Azure vs AWS RDS Virtualization Virtualization Virtualization Azure provides 99.99% SLA with built-in HA, significantly reduce financial burden on customers compared to RDS Hardware Datacenter Management Open Source on-prem deployment Hardware Datacenter Management Open Source VMs Hardware Datacenter Management Azure Database for MySQL/PostgreSQL/MariaDB Managed by customer Managed by Microsoft Machine-learning capability

5 Open source database engines popularity 80% of companies invest in public cloud technologies 1 By 2020, 74% of cloud workloads will be software-as-a-service (SaaS) workloads, up from 65 percent in By 2018, more than 70% of new in-house apps will be developed on OSS databases 2 PostgreSQL and MariaDB are the fastest growing relational database management systems 3 PostgreSQL named 2017 Database of the year by DB-Engines.com 4 1 Cisco Global Cloud Index: Forecast and Methodology, editions & The State of Open Source RDBMSs, 2015, Gartner (Donald Feinberg and Merv Adrian) 3 DB-Engines. DB-Engines Ranking. solid IT Gmbh, DB-Engines. Kerschberg, Ben. How Postgres and Open Source are Disrupting the Market for Database Management Systems. Forbes. 8 Mar 2016

6 Highlights of open source databases Free Community focused Runs on Linux and everywhere PostgreSQL runs on Linux, macos, Solaris, Windows, and more MySQL and MariaDB run on Linux Straight forward deployment into production compared to commercial DBs that restrict free developer version for production Less DBA needs, ease of use favored by developers Wide community support with developers adding and asking things of community Large number of updates and releases from PostgreSQL

7 Azure Database for MySQL, PostgreSQL, & MariaDB Cloud scale Elastic Get the scale full benefits in of seconds the cloud and unlock developer productivity Performance Security HA/DR #1 Query Benchmark-setting Store and performance intelligent insights vs. great performance for its purpose Auditing, Most secure data vs. not-yettops in the world classification, and of threat detection security attacks Built-in Best-in-class High Availability HA and DR and vs. just-enough Disaster Recovery Solves for engine gaps with platform benefits

8 DBaaS: common service fabric enabling best customer experience and value Azure SQL Data Warehouse Azure SQL Database Azure Database for PostgreSQL Azure Database for MySQL Azure Database for MariaDB Database Services Platform Intelligent Advisors, tuning, monitoring Flexible On-demand scaling, resource governance Trusted HA/DR, backup/restore, security, audit, isolation Azure compute Azure storage

9 Global Reach, Local Proximity

10 REGIONAL INDUSTRY US GOV GLOBAL More certifications than any other cloud provider ISO ISO ISO ISO SOC 1 Type 2 SOC 2 Type 2 SOC 3 CSA STAR Self-Assessment CSA STAR Certification CSA STAR Attestation Moderate JAB P-ATO High JAB P-ATO DoD DISA SRG Level 2 DoD DISA SRG Level 4 DoD DISA SRG Level 5 SP FIPS Section 508 VPAT ITAR CJIS IRS 1075 PCI DSS Level 1 CDSA MPAA FACT UK Shared Assessments FISC Japan HIPAA / HITE CH Act HITRUST GxP 21 CFR Part 11 MARS-E IG Toolkit UK FERPA GLBA FFIEC Argentina PDPA EU Model Clauses UK G-Cloud China DJCP China GB China TRUCS Singapore MTCS Australia IRAP/CCSL New Zealand GCIO Japan My Number Act ENISA IAF Japan CS M ark Gold Spain ENS Spain DPA India MeitY Canada Privacy Laws Privacy Shield Germany IT Grundschutz workbook

11 My Azure Database for MySQL, PostgreSQL & MariaDB Enterprise-ready performance, high availability and security on top of cloud scaling and integration Cloud HA & Scaling PaaS ++ Azure Integration Truly Global Reach Save time and money with built-in High Availability (HA) and elastic scaling Enterprise-grade performance and security from Intelligent Insights and Advanced Threat Protection (ATP) Productive development seamless use of App services and one-click connections to Azure ecosystem 2x global reach vs. AWS RDS for OSS

12 My High Availability built-in: save effort and money Azure Database for MySQL/PostgreSQL AWS RDS for MySQL/PostgreSQL Cloud HA & Scaling PaaS ++ Built-in HA: offers HA on a single node No additional configuration required No extra cost from a 2nd node SLA at 99.99% for all open source DB services HA from Multi-AZ Multi-AZ not configured out-of-the-box for HA 2x costs due to required 2nd VM No SLA for single-az deployment; 99.95% for Multi- AZ deployment Azure Integration Truly Global Reach Secure container-enabled elastic scaling: independently scale compute and storage; scale storage instantly and compute in seconds. All based on Microsoft IP. VM based scaling takes minutes, and compute/storage not fully decoupled

13 My Intelligent insights and ATP to enhance performance and security Azure Database for MySQL/PostgreSQL AWS RDS for MySQL/PostgreSQL Cloud HA & Scaling PaaS ++ Azure Integration Truly Global Reach ML-based, intelligent insights & recommendation (Preview) that learns your unique database patterns and provides customized recommendations, enabling customers to maximize the performance of PostgreSQL database. Advanced Threat Protection (ATP) (Preview): the same ATP that protects SQL DB with data discovery & classification, vulnerability assessment, and advanced threat detection are coming to Azure DB for MySQL and PostgreSQL. Performance insights for MySQL/PostgreSQL* Based on historical, static query snapshots only No time-series enabled drill-down on dynamics of each query processing. No recommendation based on root-cause analysis for easy query plan fix No ATP such as Threat Detection, Vulnerability Assessment, Information Classification ATP on Storage (Preview) and full VNET service endpoints for Azure OSS DBs (GA) AWS Performance Insights

14 My Ease of integration with the rest of Azure services Azure Database for MySQL/PostgreSQL AWS RDS for MySQL/PostgreSQL Cloud HA & Scaling PaaS ++ Azure Integration Seamless use of App services for best developer productivity. e.g. integrated with Web App such that customers can deploy the database as part of deployment of the app. One-click connection to Power BI, Azure Monitor, Azure function, Cognitive APIs etc. Customers must deploy the database and app separately and stitch them together Requires additional connector for PBI Integration; lack the wealth/breadth in integration for better productivity similar to that in Azure Truly Global Reach

15 My 2x global reach, better local proximity Azure Database for MySQL/PostgreSQL AWS RDS for MySQL/PostgreSQL Cloud HA & Scaling PaaS ++ Currently in 35 datacenters Reference this for MySQL regions Reference this for PostgreSQL regions Reference this for MariaDB regions. Expanding soon to all 54 Azure Datacenters Currently in 17 datacenters SLA at 99.95%, for Multi-AZ only Azure Integration SLA at 99.99% out of the box Truly Global Reach

16 Azure Cosmos DB A globally distributed, massively scalable, multi-model database service Elastic scale out of storage & throughput Guaranteed low latency at the 99 th percentile Five well-defined consistency models Turnkey global distribution Comprehensive SLAs

17 Azure Cosmos DB A globally distributed, massively scalable, multi-model database service Column-family Document Key-value Graph Elastic scale out of storage & throughput Guaranteed low latency at the 99 th percentile Five well-defined consistency models Turnkey global distribution Comprehensive SLAs

18 Azure Cosmos DB A globally distributed, massively scalable, multi-model database service Table API MongoDB Column-family Document Key-value Graph Elastic scale out of storage & throughput Guaranteed low latency at the 99 th percentile Five well-defined consistency models Turnkey global distribution Comprehensive SLAs

19 Turnkey Global Distribution Worldwide presence as a Foundational Azure service Automatic multi-region replication Multi-homing APIs Manual and automatic failovers Designed for High Availability

20 System topology (behind the scenes)

21 Guaranteed low latency at P99 (99 th percentile) Reads (1KB) Indexed writes (1KB) Requests are served from local region Single-digit millisecond latency worldwide P50 <2ms <6ms Write optimized, latch-free database engine designed for SSD P99 <10ms <15ms Synchronous automatic indexing at sustained ingestion rates

22 Provisioned request / sec Black Friday Elastically scalable storage and throughput Single machine is never a bottle neck Transparent server-side partition management Elastically scale storage (GB to PB) and throughput (100 to 100M req/sec) across many machines and multiple regions Nov 2016 Dec 2016 Time Hourly throughput (request/sec) Automatic expiration via policy based TTL Pay by the hour, change throughput at any time for only what you need

23 % CPU Request Units % Memory % IOPS Request Units (RU) is a rate-based currency Abstracts physical resources for performing requests Key to providing isolation in a multi-tenant environment Enables SLAs for predictable performance Foreground and background activities

24 GET Request Units Normalized across various access methods POST PUT 1 RU = 1 read of 1 KB record Each request consumes fixed RUs Applies to reads, writes, queries, and stored procedure execution Query

25 Incoming Requests Request Units Rate limit Provisioned in terms of RU/sec Rate limiting based on amount of throughput provisioned Can be increased or decreased instantaneously Max RU/sec No throttling Billing is metered hourly Min RU/sec Background processes like TTL expiration, index transformations scheduled when quiescent Replica Quiescent

26 Schema-agnostic, automatic indexing At global scale, schema/index management is painful Automatic and synchronous indexing Hash, range, and geospatial Schema Works across every data model Highly write-optimized database engine Physical index

27 Multi-model, multi-api Database engine operates on Atom-Record-Sequence type system All data models can be efficiently translated to ARS Multi-model: Key-value, Document, Column, and Graph Multi-API: SQL (DocumentDB), MongoDB, Table, Cassandra and Gremlin More data-models and APIs to be added

28 Multiple, well-defined consistency choices Global distribution forces us to navigate the CAP theorem Writing correct distributed applications is hard Five well-defined consistency levels Intuitive and practical with clear PACELC tradeoffs Programmatically change at anytime Can be overridden on a per-request basis

29 (West US) (East US) (North Europe)

30 Value = 5 Value = 5 Value = 5

31 Value = 5 6 Update 5 => 6 Value = 5 Value = 5

32 Value = 5 6 Update 5 => 6 Value = 5 6 Value = 5 What happens when a network partition is introduced?

33 Value = 5 6 Update 5 => 6 Value = 5 6 Value = 5 What happens when a network partition is introduced? Reader: What is the value? Should it see 5? (prioritize availability) Or does the system go offline until network is restored? (prioritize consistency)

34 Brewer s CAP Theorem: impossible for distributed data store to simultaneously provide more than 2 out of the following 3 guarantees: Consistency, Availability, Partition Tolerance

35 Latency: packet of information can travel as fast as speed of light. Replication between distant geographic regions can take 100 s of milliseconds PACELC Theorem: In the case of network partitioning (P) in a distributed computer system, one has to choose between availability (A) and consistency (C) (as per the CAP theorem), but else (E), even when the system is running normally in the absence of partitions, one has to choose between latency (L) and consistency (C).

36 Programmable Data Consistency Strong consistency High latency Eventual consistency, Low latency

37 Consistency Level Strong Bounded Staleness Session Consistent Prefix Eventual Guarantees Linearizability (once operation is complete, it will be visible to all) Consistent Prefix. Reads lag behind writes by at most k prefixes or t interval Similar properties to strong consistency (except within staleness window), while preserving 99.99% availability and low latency. Consistent Prefix. Within a session: monotonic reads, monotonic writes, read-your-writes, write-followsreads Predictable consistency for a session, high read throughput + low latency Reads will never see out of order writes (no gaps). Potential for out of order reads. Lowest cost for reads of all consistency levels.

38 Security & Compliance Always encrypted at rest and in transit Rest AES256 Transit SSL / TLS Fine grained row level authorization User/Permissions with Resource Tokens Network security with IP firewall rules and VNET Comprehensive Azure compliance certification: ISO 27001, ISO 27018, EUMC, HIPAA PCI, SOC1 and SOC2 FEDRAMP, HITRUST

39

40

41 Globally Distributed Apps Business Needs: High scalability to support massive user base Low latency access across multiple geographies to build highly responsive applications Automatic indexing over flexible schemas to support rich queries over user-defined content (e.g. custom form builder) High availability across multiple data centers

42 Content Management Systems Azure region A Azure region B Azure Cosmos DB (app + session state) Azure Traffi c Manager Azure region C Globally distributed across regions

43 Internet of Things Telemetry & Sensor Data Business Needs: High scalability to ingest large # of events coming from many devices Low latency queries and changes feeds for responding quickly to anomalies Schema-agnostic storage and automatic indexing to support dynamic data coming from many different generations of devices High availability across multiple data centers

44 Internet of Things Telemetry & Sensor Data Azure IoT Hub Azure Databricks Spark (Structured Streaming) Azure Cosmos DB (Hot) (TTL = 90 days) Azure API App (user facing app) Azure Function (triggered via Cosmos DB change feed) Azure Storage (Cold)

45 Retail Product Catalog & Order Processing Business Needs: Elastic scale to handle seasonal traffic (e.g. Black Friday) Low-latency access across multiple geographies to support a global user-base and latency sensitive workloads (e.g. real-time personalization) Schema-agnostic storage and automatic indexing to handle diverse product catalogs, orders, and events High availability across multiple data centers

46 Retail Product Catalogs Azure Web App (e-commerce app) Azure Cosmos DB (product catalog) Azure Search (f ull-text index) Azure Storage (logs, static catalo g content) Azure Cosmos DB (session state)

47 Retail Order Processing Pipelines Azure Functions (E-Commerce Checkout API) Azure Cosmos DB (Order Event Store) Azure Functions (Microservice 1: Tax) Azure Functions (Microservice 2: Payment)... Azure Functions (Microservice N: Fufillment)

48 Real-time Recommendations Online Recommendations Service Azure Container Service (Recommendations API) Azure Cosmos DB (Product + User Vectors) Shoppers E-commerce Store Apache Spark on Azure Databricks Azure Container Service (Order Transaction API) Azure Cosmos DB (Customer Orders) Order Transactions

49 Multiplayer Gaming Business Needs: Elastic scale to handle bursty traffic on day Low-latency queries to support responsive gameplay for a global user-base Schema-agnostic storage and indexing allows teams to iterate quickly to fit a demanding ship schedule Change-feeds to support leaderboards and social gameplay

50 Multiplayer Gaming Azure CDN Azure Storage (game files) Azure Traffic M anager Azure API Apps (game backend) Azure Cosmos DB (game database) Azure Databricks (game analytics) Azure Functions Azure Notification Hubs (push notifications)

51 Customer Operational Analytics Business Needs: High scalability to handle large user base Rich queries and automatic indexing over flexible schemas to consolidate data from a variety of sources Built-in aggregates and efficient spark connector to analyze user behavior, drive insights, and build a single view of the customer.

52 Customer Social Analytics 1 st -Party and 3 rd -Party Data Sources Azure Cosmos DB (Master Data) Spark on Azure HDInsight