Architecture of a Real-Time Operational DBMS

Size: px
Start display at page:

Download "Architecture of a Real-Time Operational DBMS"

Transcription

1 Architecture of a Real-Time Operational DBMS Srini V. Srinivasan Founder, Chief Development Officer Aerospike CMG India Keynote Thane December 3, 2016 [ CMGI Keynote, Thane, India Aerospike Inc. All rights reserved. 1 ]

2 Real-Time Workloads [ CMGI Keynote, Thane, India Aerospike Inc. All rights reserved. 2 ]

3 Reliability at Massive Scale Developments Internet growth High rate of transactions Millions/second Storage advances expansion of DRAM, rise of SSDs New distributed consensus algorithms e.g., CAP Theorem, Paxos Application developers prefer not using SQL Python, Go, PHP, Traditional DB Guarantee strong consistency to replicated data Limited in scalability and availability Cannot handle network partitions NoSQL DB Focus on massive scalability and high availability Use concepts from Operating Systems and Distributed Systems [ CMGI Keynote, Thane, India Aerospike Inc. All rights reserved. 3 ]

4 SQL è NoSQL SQL databases are architected for Disk oriented storage and indexing structures Multithreading to hide latency Locking-based concurrency control mechanisms Log-based recovery NoSQL or NewSQL databases are architected for In-Memory Incremental upgrades (no fork-lift upgrade) High Availability Self-managing (self-healing, self-maintaining, self-tuning) [ CMGI Keynote, Thane, India Aerospike Inc. All rights reserved. 4 ]

5 Next Generation of Database Systems Speed at Scale Many Choices Scaling up affordably TCO Restricted Functionality Real-time Problem Set Observed TCO Speed TPS Rich functionality nonreal-time use cases TCO ($) Desired TCO Scale TB Scale TB TCO - Total cost of ownership Goal: Deliver Predictable Performance, Highest Availability, and Lowest TCO [ CMGI Keynote, Thane, India Aerospike Inc. All rights reserved. 5 ]

6 Use Cases [ CMGI Keynote, Thane, India Aerospike Inc. All rights reserved. 6 ]

7 Billion Dollar Advertising Market uses RTB 1 to 6 billion cookies tracked Auctions at about 3.0M / sec in North America 100ms ad rendering, 50ms real-time bidding Low Latency, High Throughput, High Uptime [ CMGI Keynote, Thane, India Aerospike Inc. All rights reserved. 7 ]

8 RTB Tech Stack [ CMGI Keynote, Thane, India Aerospike Inc. All rights reserved. 8 ]

9 Beyond Ad Tech FinServ Marketing Tech Telco AdTech Gaming [ CMGI Keynote, Thane, India Aerospike Inc. All rights reserved. 9 ]

10 Financial Services Tech Stack [ CMGI Keynote, Thane, India Aerospike Inc. All rights reserved. 10 ]

11 Fraud Detection Tech Stack [ CMGI Keynote, Thane, India Aerospike Inc. All rights reserved. 11 ]

12 Telco Tech Stack [ CMGI Keynote, Thane, India Aerospike Inc. All rights reserved. 12 ]

13 Operational Scale in Enterprises Decisioning Engine BUSINESS TRANSACTIONS LEGACY RDBMS HDFS BASED Web views ( Payments ) ( Mobile Queries ) ( Recommendation ) ( And More ) High Performance NoSQL XDR REAL-TIME BIG DATA DECISIONING 500 Business Trans per sec LEGACY DATABASE (Mainframe) DATA WAREHOUSE/ DATA LAKE X 5000 = 2.5 M Calculations per sec Database Transactions per sec [ CMGI Keynote, Thane, India Aerospike Inc. All rights reserved. 13 ]

14 Technology [ CMGI Keynote, Thane, India Aerospike Inc. All rights reserved. 14 ]

15 Architecture Overview 1) No Hotspots Distributed Hashing simplifies data partitioning 2) Smart Client 1 hop to data, load balancing 3) Shared Nothing Architecture every node is identical 4) Smart Clustering auto-sharding, auto-failover, auto-rebalancing, rack aware, rolling upgrades 5) Transactions and long-running tasks prioritized in realtime 6) XDR sync replication across data centers ensures near Zero Downtime [ CMGI Keynote, Thane, India Aerospike Inc. All rights reserved. 15 ]

16 Cluster Formation Say N1 is seed node and N3 is the paxos principal N2 and N3 send themselves in list to N1; N1 discovers them N1 sends adjacency list [N1, N2, N3] to newly discovered node N3 (and also N2) N3 discovers N2 and starts sending the cluster node list [N3, N2, N1] to N1 and N2 [ CMGI Keynote, Thane, India Aerospike Inc. All rights reserved. 16 ]

17 Distributed Hash Based Partitioning Distributed Hashing with No Hotspots Every key hashed with RIPEMD160 into an ultra efficient 20 byte (fixed length) string Hash + additional (fixed 64 bytes) data forms index entry in RAM Some bits from hash value are used to calculate the Partition ID (4096 partitions) Partition ID maps to Node ID in the cluster [ CMGI Keynote, Thane, India Aerospike Inc. All rights reserved. 17 ]

18 Data Distribution Index and data are colocated 1. Distribute workload uniformly 2. Provide predictable read/write performance 3. Scale up and down by simply adding cluster nodes 4. Rebalance data non-disruptively and efficiently Partition assignment objectives 1. Deterministic, so each node can operate by itself 2. Uniform distribution of partitions across nodes 3. Minimize partition moves during cluster changes [ CMGI Keynote, Thane, India Aerospike Inc. All rights reserved. 18 ]

19 Partition Assignment Algorithm function REPLICATION_LIST_ASSIGN(partitionid) node_hash = empty map for nodeid in succession_list: node_hash[nodeid] = NODE_HASH_COMPUTE(nodeid, partitionid) replication_list = sort_ascending(node_hash using hash) return replication_list function NODE_HASH_COMPUTE(nodeid, partitionid): nodeid_hash = fnv_1a_hash(nodeid) partition_hash = fnv_1a_hash(partitionid) return jenkins_one_at_a_time_hash(<nodeid_hash, partition_hash>) [ CMGI Keynote, Thane, India Aerospike Inc. All rights reserved. 19 ]

20 Real-Time Prioritization master replica Writing with Immediate Consistency Adding a Node transactions continue 1. Write sent to row master 2. Latch against simultaneous writes 3. Apply write to master and replica synchronously 4. Queue operations to disk 5. Signal completed transaction 6. Master merges duplicate copies (if any) 1. Cluster discovers new node via gossip protocol 2. Paxos vote determines new data organization 3. Partition migrations scheduled (only deltas copied) 4. When a partition migration starts, write journal starts on destination 5. Partition moves atomically 6. Journal is applied and source data deleted [ CMGI Keynote, Thane, India Aerospike Inc. All rights reserved. 20 ]

21 Intelligent Client The Aerospike Client is implemented as a library, JAR or DLL, and consists of 2 parts: Operation APIs These are the operations that you can execute on the cluster CRUD+ etc. First class observer of the Cluster Monitoring the state of each node and aware of new nodes or node failures. 1 Hop to data Smart Client simply calculates Partition ID to determine Node ID Client performs load balancing [ CMGI Keynote, Thane, India Aerospike Inc. All rights reserved. 21 ]

22 Designed for Wire-Line Speed Multi-core architecture Optimized C based DB kernel 1. Multi-threaded data structures 2. Nested locking model for synchronization 3. Lockless data structures 4. Partitioned single threaded data structures 5. Index entries are aligned to cache line (64 bytes) 6. Custom memory management (arenas) Memory Arena Assignment [ CMGI Keynote, Thane, India Aerospike Inc. All rights reserved. 22 ]

23 In-Memory and Hybrid-Memory Storage Architecture 1. Direct device access 2. Large Block Writes 3. Indexes in DRAM 4. Highly Parallelized Highlights 5. Log-structured FS copy-on-write 6. Fast restart with shared memory Storage Layout [ CMGI Keynote, Thane, India Aerospike Inc. All rights reserved. 23 ]

24 Benchmarks [ CMGI Keynote, Thane, India Aerospike Inc. All rights reserved. 24 ]

25 Hybrid-Memory Performance HIGH THROUGHPUT LOW LATENCY 350, ,000 Balanced Workload Read Latency Throughput, ops/second 250, , , ,000 50,000 0 Balanced Read-Heavy Average Latency, ms , , , ,000 Throughput, ops/sec Aerospike Cassandra Aerospike Balanced 50/50 read-write ratio Read-Heavy 95/5 read-write ratio [ CMGI Keynote, Thane, India Aerospike Inc. All rights reserved. 25 ]

26 In-Memory performance YCSB Benchmark 50 million records YCSB Workload A (50/50 R/W) YCSB Workload B (95/5 R/W) Zipfian key distribution 8 Core Dual Socket Intel Xeon CPU E5-2665@2.4GHz 32GB DRAM with 16 queues [ CMGI Keynote, Thane, India Aerospike Inc. All rights reserved. 26 ]

27 DRAM Vs SSD on GCE GCE instance n1-standard-8 10 node cluster 150 byte record with 3 columns 100 million records [ CMGI Keynote, Thane, India Aerospike Inc. All rights reserved. 27 ]

28 Predictable Performance During Failures Phases 1) 100KTPS 4 nodes 2) Clients at Max 3) 400KTPS 4 nodes 4) 400KTPS 3 nodes 5) 400KTPS 4 nodes Aerospike Node Specs: CentOS 6.3 Intel i5-2400@ 3.1 GHz (Quad core) 16 GB RAM@1333 MHz [ CMGI Keynote, Thane, India Aerospike Inc. All rights reserved. 28 ]

29 TCO: In-Memory Vs Hybrid-Memory Actual deployment analysis. Deployment requires 500K TPS, 10 TB of storage, with 2x replication factor. In-Memory System 186 SERVERS REQUIRED Hybrid-Memory System ONLY 14 SERVERS REQUIRED 14 SERVERS REQUIRED Storage per server 180 GB (196 GB Server) 2.4 TB (4 x 700 GB) TPS per cluster 500, ,000 Cost per server $8,000 $11,000 Server costs $1,488,000 $154,000 Power/server 0.9 kw 1.1 kw Power (2 years) $0.12 per kwh ave. US $352,000 $32,400 Maintenance (2 years) $3,600 per server $670,000 $50,400 Total $2,510,000 $236,800 [ CMGI Keynote, Thane, India Aerospike Inc. All rights reserved. 29 ]

30 Future Work Software Application Requirements New Hardware Linearize with CP Mode Eventual consistency with conflict detection and resolution Pipelined execution of client transactions for increased performance Security enhancements Customers demand Real-time decisions based on recent data High Consistency Security 3D XPoint High core CPUs NVMe Multi-queue network cards Virtualized IO 30 [ CMGI Keynote, Thane, India Aerospike Inc. All rights reserved. 30 ]

31 Thank You Questions? [ CMGI Keynote, Thane, India Aerospike Inc. All rights reserved. 31 ]

Flash-Optimized, High-Performance NoSQL Database for All

Flash-Optimized, High-Performance NoSQL Database for All Flash-Optimized, High-Performance NoSQL Database for All Khosrow Afroozeh Engineer Aerospike Aerospike Built for the Age of Scale The Gold Standard 6 of top 16 powered by Aerospike (after Google, FB, from

More information

VOLTDB + HP VERTICA. page

VOLTDB + HP VERTICA. page VOLTDB + HP VERTICA ARCHITECTURE FOR FAST AND BIG DATA ARCHITECTURE FOR FAST + BIG DATA FAST DATA Fast Serve Analytics BIG DATA BI Reporting Fast Operational Database Streaming Analytics Columnar Analytics

More information

Have your cake, and eat it too. Strong Consistency and High Performance

Have your cake, and eat it too. Strong Consistency and High Performance Have your cake, and eat it too Strong Consistency and High Performance Brian Bulkowski, CTO & Founder March 7, 2018 Qcon London Aerospike in a nutshell Hybrid Memory Enables Digital Transformation Fast

More information

New Oracle NoSQL Database APIs that Speed Insertion and Retrieval

New Oracle NoSQL Database APIs that Speed Insertion and Retrieval New Oracle NoSQL Database APIs that Speed Insertion and Retrieval O R A C L E W H I T E P A P E R F E B R U A R Y 2 0 1 6 1 NEW ORACLE NoSQL DATABASE APIs that SPEED INSERTION AND RETRIEVAL Introduction

More information

<Insert Picture Here> Oracle NoSQL Database A Distributed Key-Value Store

<Insert Picture Here> Oracle NoSQL Database A Distributed Key-Value Store Oracle NoSQL Database A Distributed Key-Value Store Charles Lamb The following is intended to outline our general product direction. It is intended for information purposes only,

More information

Aerospike Scales with Google Cloud Platform

Aerospike Scales with Google Cloud Platform Aerospike Scales with Google Cloud Platform PERFORMANCE TEST SHOW AEROSPIKE SCALES ON GOOGLE CLOUD Aerospike is an In-Memory NoSQL database and a fast Key Value Store commonly used for caching and by real-time

More information

CIS 601 Graduate Seminar. Dr. Sunnie S. Chung Dhruv Patel ( ) Kalpesh Sharma ( )

CIS 601 Graduate Seminar. Dr. Sunnie S. Chung Dhruv Patel ( ) Kalpesh Sharma ( ) Guide: CIS 601 Graduate Seminar Presented By: Dr. Sunnie S. Chung Dhruv Patel (2652790) Kalpesh Sharma (2660576) Introduction Background Parallel Data Warehouse (PDW) Hive MongoDB Client-side Shared SQL

More information

Oracle Exadata X7. Uwe Kirchhoff Oracle ACS - Delivery Senior Principal Service Delivery Engineer

Oracle Exadata X7. Uwe Kirchhoff Oracle ACS - Delivery Senior Principal Service Delivery Engineer Oracle Exadata X7 Uwe Kirchhoff Oracle ACS - Delivery Senior Principal Service Delivery Engineer 05.12.2017 Oracle Engineered Systems ZFS Backup Appliance Zero Data Loss Recovery Appliance Exadata Database

More information

Dell PowerEdge R730xd Servers with Samsung SM1715 NVMe Drives Powers the Aerospike Fraud Prevention Benchmark

Dell PowerEdge R730xd Servers with Samsung SM1715 NVMe Drives Powers the Aerospike Fraud Prevention Benchmark Dell PowerEdge R730xd Servers with Samsung SM1715 NVMe Drives Powers the Aerospike Fraud Prevention Benchmark Testing validation report prepared under contract with Dell Introduction As innovation drives

More information

Jargons, Concepts, Scope and Systems. Key Value Stores, Document Stores, Extensible Record Stores. Overview of different scalable relational systems

Jargons, Concepts, Scope and Systems. Key Value Stores, Document Stores, Extensible Record Stores. Overview of different scalable relational systems Jargons, Concepts, Scope and Systems Key Value Stores, Document Stores, Extensible Record Stores Overview of different scalable relational systems Examples of different Data stores Predictions, Comparisons

More information

VoltDB for Financial Services Technical Overview

VoltDB for Financial Services Technical Overview VoltDB for Financial Services Technical Overview Financial services organizations have multiple masters: regulators, investors, customers, and internal business users. All create, monitor, and require

More information

VoltDB vs. Redis Benchmark

VoltDB vs. Redis Benchmark Volt vs. Redis Benchmark Motivation and Goals of this Evaluation Compare the performance of several distributed databases that can be used for state storage in some of our applications Low latency is expected

More information

MySQL Cluster Web Scalability, % Availability. Andrew

MySQL Cluster Web Scalability, % Availability. Andrew MySQL Cluster Web Scalability, 99.999% Availability Andrew Morgan @andrewmorgan www.clusterdb.com Safe Harbour Statement The following is intended to outline our general product direction. It is intended

More information

When, Where & Why to Use NoSQL?

When, Where & Why to Use NoSQL? When, Where & Why to Use NoSQL? 1 Big data is becoming a big challenge for enterprises. Many organizations have built environments for transactional data with Relational Database Management Systems (RDBMS),

More information

Conceptual Modeling on Tencent s Distributed Database Systems. Pan Anqun, Wang Xiaoyu, Li Haixiang Tencent Inc.

Conceptual Modeling on Tencent s Distributed Database Systems. Pan Anqun, Wang Xiaoyu, Li Haixiang Tencent Inc. Conceptual Modeling on Tencent s Distributed Database Systems Pan Anqun, Wang Xiaoyu, Li Haixiang Tencent Inc. Outline Introduction System overview of TDSQL Conceptual Modeling on TDSQL Applications Conclusion

More information

Fusion iomemory PCIe Solutions from SanDisk and Sqrll make Accumulo Hypersonic

Fusion iomemory PCIe Solutions from SanDisk and Sqrll make Accumulo Hypersonic WHITE PAPER Fusion iomemory PCIe Solutions from SanDisk and Sqrll make Accumulo Hypersonic Western Digital Technologies, Inc. 951 SanDisk Drive, Milpitas, CA 95035 www.sandisk.com Table of Contents Executive

More information

CISC 7610 Lecture 2b The beginnings of NoSQL

CISC 7610 Lecture 2b The beginnings of NoSQL CISC 7610 Lecture 2b The beginnings of NoSQL Topics: Big Data Google s infrastructure Hadoop: open google infrastructure Scaling through sharding CAP theorem Amazon s Dynamo 5 V s of big data Everyone

More information

Introduction to Database Services

Introduction to Database Services Introduction to Database Services Shaun Pearce AWS Solutions Architect 2015, Amazon Web Services, Inc. or its affiliates. All rights reserved Today s agenda Why managed database services? A non-relational

More information

10. Replication. Motivation

10. Replication. Motivation 10. Replication Page 1 10. Replication Motivation Reliable and high-performance computation on a single instance of a data object is prone to failure. Replicate data to overcome single points of failure

More information

Next-Generation Cloud Platform

Next-Generation Cloud Platform Next-Generation Cloud Platform Jangwoo Kim Jun 24, 2013 E-mail: jangwoo@postech.ac.kr High Performance Computing Lab Department of Computer Science & Engineering Pohang University of Science and Technology

More information

Oracle TimesTen Scaleout: Revolutionizing In-Memory Transaction Processing

Oracle TimesTen Scaleout: Revolutionizing In-Memory Transaction Processing Oracle Scaleout: Revolutionizing In-Memory Transaction Processing Scaleout is a brand new, shared nothing scale-out in-memory database designed for next generation extreme OLTP workloads. Featuring elastic

More information

Accelerating Big Data: Using SanDisk SSDs for Apache HBase Workloads

Accelerating Big Data: Using SanDisk SSDs for Apache HBase Workloads WHITE PAPER Accelerating Big Data: Using SanDisk SSDs for Apache HBase Workloads December 2014 Western Digital Technologies, Inc. 951 SanDisk Drive, Milpitas, CA 95035 www.sandisk.com Table of Contents

More information

Oracle NoSQL Database Enterprise Edition, Version 18.1

Oracle NoSQL Database Enterprise Edition, Version 18.1 Oracle NoSQL Database Enterprise Edition, Version 18.1 Oracle NoSQL Database is a scalable, distributed NoSQL database, designed to provide highly reliable, flexible and available data management across

More information

4 Myths about in-memory databases busted

4 Myths about in-memory databases busted 4 Myths about in-memory databases busted Yiftach Shoolman Co-Founder & CTO @ Redis Labs @yiftachsh, @redislabsinc Background - Redis Created by Salvatore Sanfilippo (@antirez) OSS, in-memory NoSQL k/v

More information

Memory-Based Cloud Architectures

Memory-Based Cloud Architectures Memory-Based Cloud Architectures ( Or: Technical Challenges for OnDemand Business Software) Jan Schaffner Enterprise Platform and Integration Concepts Group Example: Enterprise Benchmarking -) *%'+,#$)

More information

BENCHMARK: PRELIMINARY RESULTS! JUNE 25, 2014!

BENCHMARK: PRELIMINARY RESULTS! JUNE 25, 2014! BENCHMARK: PRELIMINARY RESULTS JUNE 25, 2014 Our latest benchmark test results are in. The detailed report will be published early next month, but after 6 weeks of designing and running these tests we

More information

Improving Altibase Performance with Solarflare 10GbE Server Adapters and OpenOnload

Improving Altibase Performance with Solarflare 10GbE Server Adapters and OpenOnload Improving Altibase Performance with Solarflare 10GbE Server Adapters and OpenOnload Summary As today s corporations process more and more data, the business ramifications of faster and more resilient database

More information

MySQL Cluster for Real Time, HA Services

MySQL Cluster for Real Time, HA Services MySQL Cluster for Real Time, HA Services Bill Papp (bill.papp@oracle.com) Principal MySQL Sales Consultant Oracle Agenda Overview of MySQL Cluster Design Goals, Evolution, Workloads,

More information

DATABASE SCALE WITHOUT LIMITS ON AWS

DATABASE SCALE WITHOUT LIMITS ON AWS The move to cloud computing is changing the face of the computer industry, and at the heart of this change is elastic computing. Modern applications now have diverse and demanding requirements that leverage

More information

GridGain and Apache Ignite In-Memory Performance with Durability of Disk

GridGain and Apache Ignite In-Memory Performance with Durability of Disk GridGain and Apache Ignite In-Memory Performance with Durability of Disk Dmitriy Setrakyan Apache Ignite PMC GridGain Founder & CPO http://ignite.apache.org #apacheignite Agenda What is GridGain and Ignite

More information

Accelerate Database Performance and Reduce Response Times in MongoDB Humongous Environments with the LSI Nytro MegaRAID Flash Accelerator Card

Accelerate Database Performance and Reduce Response Times in MongoDB Humongous Environments with the LSI Nytro MegaRAID Flash Accelerator Card Accelerate Database Performance and Reduce Response Times in MongoDB Humongous Environments with the LSI Nytro MegaRAID Flash Accelerator Card The Rise of MongoDB Summary One of today s growing database

More information

Oracle Exadata: Strategy and Roadmap

Oracle Exadata: Strategy and Roadmap Oracle Exadata: Strategy and Roadmap - New Technologies, Cloud, and On-Premises Juan Loaiza Senior Vice President, Database Systems Technologies, Oracle Safe Harbor Statement The following is intended

More information

MapR Enterprise Hadoop

MapR Enterprise Hadoop 2014 MapR Technologies 2014 MapR Technologies 1 MapR Enterprise Hadoop Top Ranked Cloud Leaders 500+ Customers 2014 MapR Technologies 2 Key MapR Advantage Partners Business Services APPLICATIONS & OS ANALYTICS

More information

NVMFS: A New File System Designed Specifically to Take Advantage of Nonvolatile Memory

NVMFS: A New File System Designed Specifically to Take Advantage of Nonvolatile Memory NVMFS: A New File System Designed Specifically to Take Advantage of Nonvolatile Memory Dhananjoy Das, Sr. Systems Architect SanDisk Corp. 1 Agenda: Applications are KING! Storage landscape (Flash / NVM)

More information

CS 655 Advanced Topics in Distributed Systems

CS 655 Advanced Topics in Distributed Systems Presented by : Walid Budgaga CS 655 Advanced Topics in Distributed Systems Computer Science Department Colorado State University 1 Outline Problem Solution Approaches Comparison Conclusion 2 Problem 3

More information

CISC 7610 Lecture 5 Distributed multimedia databases. Topics: Scaling up vs out Replication Partitioning CAP Theorem NoSQL NewSQL

CISC 7610 Lecture 5 Distributed multimedia databases. Topics: Scaling up vs out Replication Partitioning CAP Theorem NoSQL NewSQL CISC 7610 Lecture 5 Distributed multimedia databases Topics: Scaling up vs out Replication Partitioning CAP Theorem NoSQL NewSQL Motivation YouTube receives 400 hours of video per minute That is 200M hours

More information

HyperDex. A Distributed, Searchable Key-Value Store. Robert Escriva. Department of Computer Science Cornell University

HyperDex. A Distributed, Searchable Key-Value Store. Robert Escriva. Department of Computer Science Cornell University HyperDex A Distributed, Searchable Key-Value Store Robert Escriva Bernard Wong Emin Gün Sirer Department of Computer Science Cornell University School of Computer Science University of Waterloo ACM SIGCOMM

More information

Distributed File Systems II

Distributed File Systems II Distributed File Systems II To do q Very-large scale: Google FS, Hadoop FS, BigTable q Next time: Naming things GFS A radically new environment NFS, etc. Independence Small Scale Variety of workloads Cooperation

More information

HP NonStop Database Solution

HP NonStop Database Solution CHOICE - CONFIDENCE - CONSISTENCY HP NonStop Database Solution Marco Sansoni, HP NonStop Business Critical Systems 9 ottobre 2012 Agenda Introduction to HP NonStop platform HP NonStop SQL database solution

More information

Tools for Social Networking Infrastructures

Tools for Social Networking Infrastructures Tools for Social Networking Infrastructures 1 Cassandra - a decentralised structured storage system Problem : Facebook Inbox Search hundreds of millions of users distributed infrastructure inbox changes

More information

Crescando: Predictable Performance for Unpredictable Workloads

Crescando: Predictable Performance for Unpredictable Workloads Crescando: Predictable Performance for Unpredictable Workloads G. Alonso, D. Fauser, G. Giannikis, D. Kossmann, J. Meyer, P. Unterbrunner Amadeus S.A. ETH Zurich, Systems Group (Funded by Enterprise Computing

More information

Highly Scalable, Non-RDMA NVMe Fabric. Bob Hansen,, VP System Architecture

Highly Scalable, Non-RDMA NVMe Fabric. Bob Hansen,, VP System Architecture A Cost Effective,, High g Performance,, Highly Scalable, Non-RDMA NVMe Fabric Bob Hansen,, VP System Architecture bob@apeirondata.com Storage Developers Conference, September 2015 Agenda 3 rd Platform

More information

Migrating Oracle Databases To Cassandra

Migrating Oracle Databases To Cassandra BY UMAIR MANSOOB Why Cassandra Lower Cost of ownership makes it #1 choice for Big Data OLTP Applications. Unlike Oracle, Cassandra can store structured, semi-structured, and unstructured data. Cassandra

More information

Introduction to Oracle NoSQL Database

Introduction to Oracle NoSQL Database Introduction to Oracle NoSQL Database Anand Chandak Ashutosh Naik Agenda NoSQL Background Oracle NoSQL Database Overview Technical Features & Performance Use Cases 2 Why NoSQL? 1. The four V s of Big Data

More information

TrafficDB: HERE s High Performance Shared-Memory Data Store Ricardo Fernandes, Piotr Zaczkowski, Bernd Göttler, Conor Ettinoffe, and Anis Moussa

TrafficDB: HERE s High Performance Shared-Memory Data Store Ricardo Fernandes, Piotr Zaczkowski, Bernd Göttler, Conor Ettinoffe, and Anis Moussa TrafficDB: HERE s High Performance Shared-Memory Data Store Ricardo Fernandes, Piotr Zaczkowski, Bernd Göttler, Conor Ettinoffe, and Anis Moussa EPL646: Advanced Topics in Databases Christos Hadjistyllis

More information

Oracle NoSQL Database Enterprise Edition, Version 18.1

Oracle NoSQL Database Enterprise Edition, Version 18.1 Oracle NoSQL Database Enterprise Edition, Version 18.1 Oracle NoSQL Database is a scalable, distributed NoSQL database, designed to provide highly reliable, flexible and available data management across

More information

TITLE. the IT Landscape

TITLE. the IT Landscape The Impact of Hyperconverged Infrastructure on the IT Landscape 1 TITLE Drivers for adoption Lower TCO Speed and Agility Scale Easily Operational Simplicity Hyper-converged Integrated storage & compute

More information

How do we build TiDB. a Distributed, Consistent, Scalable, SQL Database

How do we build TiDB. a Distributed, Consistent, Scalable, SQL Database How do we build TiDB a Distributed, Consistent, Scalable, SQL Database About me LiuQi ( 刘奇 ) JD / WandouLabs / PingCAP Co-founder / CEO of PingCAP Open-source hacker / Infrastructure software engineer

More information

Maximizing Fraud Prevention Through Disruptive Architectures Delivering speed at scale.

Maximizing Fraud Prevention Through Disruptive Architectures Delivering speed at scale. Maximizing Fraud Prevention Through Disruptive Architectures Delivering speed at scale. January 2016 Credit Card Fraud prevention is among the most time-sensitive and high-value of IT tasks. The databases

More information

CMU SCS CMU SCS Who: What: When: Where: Why: CMU SCS

CMU SCS CMU SCS Who: What: When: Where: Why: CMU SCS Carnegie Mellon Univ. Dept. of Computer Science 15-415/615 - DB s C. Faloutsos A. Pavlo Lecture#23: Distributed Database Systems (R&G ch. 22) Administrivia Final Exam Who: You What: R&G Chapters 15-22

More information

Table of contents. OpenVMS scalability with Oracle Rdb. Scalability achieved through performance tuning.

Table of contents. OpenVMS scalability with Oracle Rdb. Scalability achieved through performance tuning. OpenVMS scalability with Oracle Rdb Scalability achieved through performance tuning. Table of contents Abstract..........................................................2 From technical achievement to

More information

Megastore: Providing Scalable, Highly Available Storage for Interactive Services & Spanner: Google s Globally- Distributed Database.

Megastore: Providing Scalable, Highly Available Storage for Interactive Services & Spanner: Google s Globally- Distributed Database. Megastore: Providing Scalable, Highly Available Storage for Interactive Services & Spanner: Google s Globally- Distributed Database. Presented by Kewei Li The Problem db nosql complex legacy tuning expensive

More information

The Google File System. Alexandru Costan

The Google File System. Alexandru Costan 1 The Google File System Alexandru Costan Actions on Big Data 2 Storage Analysis Acquisition Handling the data stream Data structured unstructured semi-structured Results Transactions Outline File systems

More information

NoSQL BENCHMARKING AND TUNING. Nachiket Kate Santosh Kangane Ankit Lakhotia Persistent Systems Ltd. Pune, India

NoSQL BENCHMARKING AND TUNING. Nachiket Kate Santosh Kangane Ankit Lakhotia Persistent Systems Ltd. Pune, India NoSQL BENCHMARKING AND TUNING Nachiket Kate Santosh Kangane Ankit Lakhotia Persistent Systems Ltd. Pune, India Today large variety of available NoSQL options has made it difficult for developers to choose

More information

A Non-Relational Storage Analysis

A Non-Relational Storage Analysis A Non-Relational Storage Analysis Cassandra & Couchbase Alexandre Fonseca, Anh Thu Vu, Peter Grman Cloud Computing - 2nd semester 2012/2013 Universitat Politècnica de Catalunya Microblogging - big data?

More information

Agenda. AWS Database Services Traditional vs AWS Data services model Amazon RDS Redshift DynamoDB ElastiCache

Agenda. AWS Database Services Traditional vs AWS Data services model Amazon RDS Redshift DynamoDB ElastiCache Databases on AWS 2017 Amazon Web Services, Inc. and its affiliates. All rights served. May not be copied, modified, or distributed in whole or in part without the express consent of Amazon Web Services,

More information

NVMe SSDs Future-proof Apache Cassandra

NVMe SSDs Future-proof Apache Cassandra NVMe SSDs Future-proof Apache Cassandra Get More Insight from Datasets Too Large to Fit into Memory Overview When we scale a database either locally or in the cloud performance 1 is imperative. Without

More information

Changing Requirements for Distributed File Systems in Cloud Storage

Changing Requirements for Distributed File Systems in Cloud Storage Changing Requirements for Distributed File Systems in Cloud Storage Wesley Leggette Cleversafe Presentation Agenda r About Cleversafe r Scalability, our core driver r Object storage as basis for filesystem

More information

VEXATA FOR ORACLE. Digital Business Demands Performance and Scale. Solution Brief

VEXATA FOR ORACLE. Digital Business Demands Performance and Scale. Solution Brief Digital Business Demands Performance and Scale As enterprises shift to online and softwaredriven business models, Oracle infrastructure is being pushed to run at exponentially higher scale and performance.

More information

PRESENTATION TITLE GOES HERE. Understanding Architectural Trade-offs in Object Storage Technologies

PRESENTATION TITLE GOES HERE. Understanding Architectural Trade-offs in Object Storage Technologies Object Storage 201 PRESENTATION TITLE GOES HERE Understanding Architectural Trade-offs in Object Storage Technologies SNIA Legal Notice The material contained in this tutorial is copyrighted by the SNIA

More information

State of the Dolphin Developing new Apps in MySQL 8

State of the Dolphin Developing new Apps in MySQL 8 State of the Dolphin Developing new Apps in MySQL 8 Highlights of MySQL 8.0 technology updates Mark Swarbrick MySQL Principle Presales Consultant Jill Anolik MySQL Global Business Unit Israel Copyright

More information

IBM System Storage DCS3700

IBM System Storage DCS3700 IBM System Storage DCS3700 Maximize performance, scalability and storage density at an affordable price Highlights Gain fast, highly dense storage capabilities at an affordable price Deliver simplified

More information

Exadata Implementation Strategy

Exadata Implementation Strategy Exadata Implementation Strategy BY UMAIR MANSOOB 1 Who Am I Work as Senior Principle Engineer for an Oracle Partner Oracle Certified Administrator from Oracle 7 12c Exadata Certified Implementation Specialist

More information

Pivot3 Acuity with Microsoft SQL Server Reference Architecture

Pivot3 Acuity with Microsoft SQL Server Reference Architecture Pivot3 Acuity with Microsoft SQL Server 2014 Reference Architecture How to Contact Pivot3 Pivot3, Inc. General Information: info@pivot3.com 221 West 6 th St., Suite 750 Sales: sales@pivot3.com Austin,

More information

NoSQL Databases MongoDB vs Cassandra. Kenny Huynh, Andre Chik, Kevin Vu

NoSQL Databases MongoDB vs Cassandra. Kenny Huynh, Andre Chik, Kevin Vu NoSQL Databases MongoDB vs Cassandra Kenny Huynh, Andre Chik, Kevin Vu Introduction - Relational database model - Concept developed in 1970 - Inefficient - NoSQL - Concept introduced in 1980 - Related

More information

MySQL High Availability. Michael Messina Senior Managing Consultant, Rolta-AdvizeX /

MySQL High Availability. Michael Messina Senior Managing Consultant, Rolta-AdvizeX / MySQL High Availability Michael Messina Senior Managing Consultant, Rolta-AdvizeX mmessina@advizex.com / mike.messina@rolta.com Introduction Michael Messina Senior Managing Consultant Rolta-AdvizeX, Working

More information

A Brief Introduction of TiDB. Dongxu (Edward) Huang CTO, PingCAP

A Brief Introduction of TiDB. Dongxu (Edward) Huang CTO, PingCAP A Brief Introduction of TiDB Dongxu (Edward) Huang CTO, PingCAP About me Dongxu (Edward) Huang, Cofounder & CTO of PingCAP PingCAP, based in Beijing, China. Infrastructure software engineer, open source

More information

MySQL & NoSQL: The Best of Both Worlds

MySQL & NoSQL: The Best of Both Worlds MySQL & NoSQL: The Best of Both Worlds Mario Beck Principal Sales Consultant MySQL mario.beck@oracle.com 1 Copyright 2012, Oracle and/or its affiliates. All rights Safe Harbour Statement The following

More information

Course Content MongoDB

Course Content MongoDB Course Content MongoDB 1. Course introduction and mongodb Essentials (basics) 2. Introduction to NoSQL databases What is NoSQL? Why NoSQL? Difference Between RDBMS and NoSQL Databases Benefits of NoSQL

More information

<Insert Picture Here> MySQL Web Reference Architectures Building Massively Scalable Web Infrastructure

<Insert Picture Here> MySQL Web Reference Architectures Building Massively Scalable Web Infrastructure MySQL Web Reference Architectures Building Massively Scalable Web Infrastructure Mario Beck (mario.beck@oracle.com) Principal Sales Consultant MySQL Session Agenda Requirements for

More information

Achieving the Potential of a Fully Distributed Storage System

Achieving the Potential of a Fully Distributed Storage System Achieving the Potential of a Fully Distributed Storage System HPCN Workshop 2013, DLR Braunschweig, 7-8 May 2013 Slide 1 Scality Quick Facts Founded 2009 Experienced management team HQ in the San Francisco,

More information

A Gentle Introduction to Ceph

A Gentle Introduction to Ceph A Gentle Introduction to Ceph Narrated by Tim Serong tserong@suse.com Adapted from a longer work by Lars Marowsky-Brée lmb@suse.com Once upon a time there was a Free and Open Source distributed storage

More information

DataON and Intel Select Hyper-Converged Infrastructure (HCI) Maximizes IOPS Performance for Windows Server Software-Defined Storage

DataON and Intel Select Hyper-Converged Infrastructure (HCI) Maximizes IOPS Performance for Windows Server Software-Defined Storage Solution Brief DataON and Intel Select Hyper-Converged Infrastructure (HCI) Maximizes IOPS Performance for Windows Server Software-Defined Storage DataON Next-Generation All NVMe SSD Flash-Based Hyper-Converged

More information

Cloud Computing with FPGA-based NVMe SSDs

Cloud Computing with FPGA-based NVMe SSDs Cloud Computing with FPGA-based NVMe SSDs Bharadwaj Pudipeddi, CTO NVXL Santa Clara, CA 1 Choice of NVMe Controllers ASIC NVMe: Fully off-loaded, consistent performance, M.2 or U.2 form factor ASIC OpenChannel:

More information

NoSQL Performance Test

NoSQL Performance Test bankmark UG (haftungsbeschränkt) Bahnhofstraße 1 9432 Passau Germany www.bankmark.de info@bankmark.de T +49 851 25 49 49 F +49 851 25 49 499 NoSQL Performance Test In-Memory Performance Comparison of SequoiaDB,

More information

Copyright 2013, Oracle and/or its affiliates. All rights reserved.

Copyright 2013, Oracle and/or its affiliates. All rights reserved. 1 Oracle NoSQL Database: Release 3.0 What s new and why you care Dave Segleau NoSQL Product Manager The following is intended to outline our general product direction. It is intended for information purposes

More information

São Paulo. August,

São Paulo. August, São Paulo August, 28 2018 A Modernização das Soluções de Armazeamento e Proteção de Dados DellEMC Mateus Pereira Systems Engineer, DellEMC mateus.pereira@dell.com Need for Transformation 81% of customers

More information

DEMYSTIFYING BIG DATA WITH RIAK USE CASES. Martin Schneider Basho Technologies!

DEMYSTIFYING BIG DATA WITH RIAK USE CASES. Martin Schneider Basho Technologies! DEMYSTIFYING BIG DATA WITH RIAK USE CASES Martin Schneider Basho Technologies! Agenda Defining Big Data in Regards to Riak A Series of Trade-Offs Use Cases Q & A About Basho & Riak Basho Technologies is

More information

April 21, 2017 Revision GridDB Reliability and Robustness

April 21, 2017 Revision GridDB Reliability and Robustness April 21, 2017 Revision 1.0.6 GridDB Reliability and Robustness Table of Contents Executive Summary... 2 Introduction... 2 Reliability Features... 2 Hybrid Cluster Management Architecture... 3 Partition

More information

NewSQL Databases. The reference Big Data stack

NewSQL Databases. The reference Big Data stack Università degli Studi di Roma Tor Vergata Dipartimento di Ingegneria Civile e Ingegneria Informatica NewSQL Databases Corso di Sistemi e Architetture per Big Data A.A. 2017/18 Valeria Cardellini The reference

More information

Spanner: Google's Globally-Distributed Database. Presented by Maciej Swiech

Spanner: Google's Globally-Distributed Database. Presented by Maciej Swiech Spanner: Google's Globally-Distributed Database Presented by Maciej Swiech What is Spanner? "...Google's scalable, multi-version, globallydistributed, and synchronously replicated database." What is Spanner?

More information

Database Architecture 2 & Storage. Instructor: Matei Zaharia cs245.stanford.edu

Database Architecture 2 & Storage. Instructor: Matei Zaharia cs245.stanford.edu Database Architecture 2 & Storage Instructor: Matei Zaharia cs245.stanford.edu Summary from Last Time System R mostly matched the architecture of a modern RDBMS» SQL» Many storage & access methods» Cost-based

More information

5 Fundamental Strategies for Building a Data-centered Data Center

5 Fundamental Strategies for Building a Data-centered Data Center 5 Fundamental Strategies for Building a Data-centered Data Center June 3, 2014 Ken Krupa, Chief Field Architect Gary Vidal, Solutions Specialist Last generation Reference Data Unstructured OLTP Warehouse

More information

NEC Express5800 A2040b 22TB Data Warehouse Fast Track. Reference Architecture with SW mirrored HGST FlashMAX III

NEC Express5800 A2040b 22TB Data Warehouse Fast Track. Reference Architecture with SW mirrored HGST FlashMAX III NEC Express5800 A2040b 22TB Data Warehouse Fast Track Reference Architecture with SW mirrored HGST FlashMAX III Based on Microsoft SQL Server 2014 Data Warehouse Fast Track (DWFT) Reference Architecture

More information

An Intelligent & Optimized Way to Access Flash Storage Increase Performance & Scalability of Your Applications

An Intelligent & Optimized Way to Access Flash Storage Increase Performance & Scalability of Your Applications An Intelligent & Optimized Way to Access Flash Storage Increase Performance & Scalability of Your Applications Saeed Raja Director Product Management Enterprise Storage Solutions Corporation August 6,

More information

Performance Benefits of Running RocksDB on Samsung NVMe SSDs

Performance Benefits of Running RocksDB on Samsung NVMe SSDs Performance Benefits of Running RocksDB on Samsung NVMe SSDs A Detailed Analysis 25 Samsung Semiconductor Inc. Executive Summary The industry has been experiencing an exponential data explosion over the

More information

Cascade Mapping: Optimizing Memory Efficiency for Flash-based Key-value Caching

Cascade Mapping: Optimizing Memory Efficiency for Flash-based Key-value Caching Cascade Mapping: Optimizing Memory Efficiency for Flash-based Key-value Caching Kefei Wang and Feng Chen Louisiana State University SoCC '18 Carlsbad, CA Key-value Systems in Internet Services Key-value

More information

Big and Fast. Anti-Caching in OLTP Systems. Justin DeBrabant

Big and Fast. Anti-Caching in OLTP Systems. Justin DeBrabant Big and Fast Anti-Caching in OLTP Systems Justin DeBrabant Online Transaction Processing transaction-oriented small footprint write-intensive 2 A bit of history 3 OLTP Through the Years relational model

More information

RIGHTNOW A C E

RIGHTNOW A C E RIGHTNOW A C E 2 0 1 4 2014 Aras 1 A C E 2 0 1 4 Scalability Test Projects Understanding the results 2014 Aras Overview Original Use Case Scalability vs Performance Scale to? Scaling the Database Server

More information

Drilling Through The Stack. Burkhard Neidecker-Lutz Technical Director, SAP Research SAP AG

Drilling Through The Stack. Burkhard Neidecker-Lutz Technical Director, SAP Research SAP AG Drilling Through The Stack Burkhard Neidecker-Lutz Technical Director, SAP Research SAP AG Talk Outline Where I come from Layered stacks Violating layers for fun and profit The mother of all stacks: Future

More information

CA485 Ray Walshe Google File System

CA485 Ray Walshe Google File System Google File System Overview Google File System is scalable, distributed file system on inexpensive commodity hardware that provides: Fault Tolerance File system runs on hundreds or thousands of storage

More information

Windows Servers In Microsoft Azure

Windows Servers In Microsoft Azure $6/Month Windows Servers In Microsoft Azure What I m Going Over 1. How inexpensive servers in Microsoft Azure are 2. How I get Windows servers for $6/month 3. Why Azure hosted servers are way better 4.

More information

CIT 668: System Architecture. Distributed Databases

CIT 668: System Architecture. Distributed Databases CIT 668: System Architecture Distributed Databases Topics 1. MySQL 2. Concurrency 3. Transactions and ACID 4. Database scaling 5. Replication 6. Partitioning 7. Brewer s CAP Theorem 8. ACID vs. BASE 9.

More information

Introduction to Distributed Data Systems

Introduction to Distributed Data Systems Introduction to Distributed Data Systems Serge Abiteboul Ioana Manolescu Philippe Rigaux Marie-Christine Rousset Pierre Senellart Web Data Management and Distribution http://webdam.inria.fr/textbook January

More information

SCYLLA: NoSQL at Ludicrous Speed. 主讲人 :ScyllaDB 软件工程师贺俊

SCYLLA: NoSQL at Ludicrous Speed. 主讲人 :ScyllaDB 软件工程师贺俊 SCYLLA: NoSQL at Ludicrous Speed 主讲人 :ScyllaDB 软件工程师贺俊 Today we will cover: + Intro: Who we are, what we do, who uses it + Why we started ScyllaDB + Why should you care + How we made design decisions to

More information

IBM Spectrum NAS. Easy-to-manage software-defined file storage for the enterprise. Overview. Highlights

IBM Spectrum NAS. Easy-to-manage software-defined file storage for the enterprise. Overview. Highlights IBM Spectrum NAS Easy-to-manage software-defined file storage for the enterprise Highlights Reduce capital expenditures with storage software on commodity servers Improve efficiency by consolidating all

More information

YCSB++ benchmarking tool Performance debugging advanced features of scalable table stores

YCSB++ benchmarking tool Performance debugging advanced features of scalable table stores YCSB++ benchmarking tool Performance debugging advanced features of scalable table stores Swapnil Patil M. Polte, W. Tantisiriroj, K. Ren, L.Xiao, J. Lopez, G.Gibson, A. Fuchs *, B. Rinaldi * Carnegie

More information

EMC Virtual Infrastructure for Microsoft Applications Data Center Solution

EMC Virtual Infrastructure for Microsoft Applications Data Center Solution EMC Virtual Infrastructure for Microsoft Applications Data Center Solution Enabled by EMC Symmetrix V-Max and Reference Architecture EMC Global Solutions Copyright and Trademark Information Copyright 2009

More information

FAWN. A Fast Array of Wimpy Nodes. David Andersen, Jason Franklin, Michael Kaminsky*, Amar Phanishayee, Lawrence Tan, Vijay Vasudevan

FAWN. A Fast Array of Wimpy Nodes. David Andersen, Jason Franklin, Michael Kaminsky*, Amar Phanishayee, Lawrence Tan, Vijay Vasudevan FAWN A Fast Array of Wimpy Nodes David Andersen, Jason Franklin, Michael Kaminsky*, Amar Phanishayee, Lawrence Tan, Vijay Vasudevan Carnegie Mellon University *Intel Labs Pittsburgh Energy in computing

More information

Performance comparisons and trade-offs for various MySQL replication schemes

Performance comparisons and trade-offs for various MySQL replication schemes Performance comparisons and trade-offs for various MySQL replication schemes Darpan Dinker VP Engineering Brian O Krafka, Chief Architect Schooner Information Technology, Inc. http://www.schoonerinfotech.com/

More information