VoltDB for Financial Services Technical Overview

Transcription

1 VoltDB for Financial Services Technical Overview Financial services organizations have multiple masters: regulators, investors, customers, and internal business users. All create, monitor, and require access to vast amounts of data, generated and viewed on myriad devices and platforms. This data must be immediately accessible, always-correct, and stored for varying periods of time, depending on local, country and global regulations. For finserv companies, data is currency. Yet aging, largely proprietary infrastructures lack the flexibility and scale required to respond to today s highly-networked, regulated, data-intensive financial applications. Fraud, a global, Internet-scale business, is an evolving threat. Financial services institutions must manage vast streams of data in real time while storing troves of transaction and profile data, with audit trails, to maintain relevance in a highly-regulated, competitive marketplace. Traditionally, these institutions relied on legacy relational database management systems; in the past decade, the rise of NoSQL has changed the options for enterprise architects and developers in financial services. Let s look at a range of data management options, and describe the technical benefits of VoltDB, an in-memory, NewSQL on-line transaction processing (OLTP) database. The operational complexity of many databases, from legacy RDBMSs to open source options, can be daunting. Full-time DBA support isn t an option for many small-medium companies, and can represent a significant seven-figure sum for larger ones. Architectural complexity, scale out vs. scale up issues, HA and cross-datacenter replication, data consistency, cloud-readiness, capacity for virtualization, even old-school locking and latching present issues more familiar to a distributed systems expert than to an app developer or DBA. More importantly, operational complexity inevitably bubbles up to affect end users. Many NoSQL offerings, which offer a more flexible approach to scale out, flexible schema and data types, fail on support for scalable transaction support when working with shared, finite resources: credit balances or trade verification, risk management, fraud detection and management, and customer interaction and personalization, to name a few use cases. Financial services organizations build value on transactional applications: Fraud and risk management Preventing credit card fraud requires banks to protect their customers and contain losses by monitoring each card swipe to detect unusual or fraudulent activity, and make an immediate decision to allow a purchase to go through or to block it as fraudulent. Trade reconciliation Proprietary workflows for processing trades, where managing high volumes of financial transactions require the ability to monitor, record, log and index transactions to comply with regulations and maintain an accurate view of transaction history. Two common problems are requesting history replay and state recovery, both of which are necessary to maintain accurate records and avoid regulatory fines.

2 Bid & offer management Brokers must route trade orders to the market with the best price, and by law must guarantee customers the best available price, to comply with the National Best Bid and Offer (NBBO) regulation. NBBO is defined as the lowest available ask price and highest available bid price across participating markets for a given security. Regulatory compliance Regulations such as Dodd-Frank, Sarbanes-Oxley, Basel III and the pending MiFID II require institutions to prove all databases and replicas are the same, with audited consistency across different data sources. In addition, institutions must comply with the SEC s National Best Bid & Offer regulation. Financial services applications directly affect an institution s revenue stream. Institutions require tight, predictable latencies for physical transactions, such as approval of credit card swipes in the range of sub 20ms so performance and scalability are non-negotiable requirements. VoltDB is the best solution available for ingesting, analyzing and acting on the massive volumes of real-time data streaming from trading, fraud detection and bid & offer management systems. It combines accuracy, scalability and manageable TCO, even for cutting edge scenarios such as managing trading operations, detecting credit card fraud in real-time, and managing quality of service for many millions of users based in multiple data centers simultaneously. VoltDB Basics VoltDB is an in-memory, SQL, cloud-ready operational database for modern applications that require the ability to manage data at unprecedented scale and volume, with 100% accuracy. VoltDB rapidly imports, operates on, and then exports vast amounts of data at lightning speed. Its robust architecture combines the best of traditional transactional databases with the speed and scalability of newer entrants. Unlike OLTP, Big Data, and NoSQL offerings that force users to compromise, only VoltDB supports all three modern financial services application data requirements: 1. Millions VoltDB processes relentless volumes of data from users, devices and sources. 2. Milliseconds VoltDB ingests, analyzes, and acts on data in milliseconds, with predictable low latency % Data managed by VoltDB is always accurate, all the time, for all decisions. Financial Services organizations use VoltDB to modernize revenue and business-critical applications, including: Fraud and risk management Trade reconciliation SLA management Regulatory compliance VoltDB was founded by a team of world-class database experts, including Dr. Michael Stonebraker. 2 VOLTDB FOR FINANCIAL SERVICES TECHNICAL OVERVIEW

3 Why VoltDB? VoltDB is adopted in Financial Services because it s well suited to both the current needs of vendors and the challenges they anticipate in future. VoltDB has been written from scratch to work in a 21st century RAM-centric environment and to meet the demanding requirements of Financial Services institutions. VoltDB makes instantaneous decision-making possible by combining the best elements of modern and traditional database technology: The speed and scalability of the best distributed data architectures, combined with the ACID transactionality of traditional RDBMSs without the licensing hassles. The consistency and reliability financial institutions need, deployed with a more streamlined, cloud-ready, highly-available, simple architecture. Active-active, multi-version cross-datacenter replication. The tools and languages developers already know. Technical Details VoltDB was designed by Dr. Michael Stonebraker to address the shortcomings of traditional online transaction processing ( OLTP) systems. With VoltDB, Stonebraker and his team were able to eliminate performance issues such as latching and locking, buffer management, and transaction management. For a more detailed look at the decisions behind VoltDB s architecture, read the Technical Overview here. Useful Work 12% Index Management 11% Logging 20% Buffer Management 29% Locking 18% Latching 10% 3 VOLTDB FOR FINANCIAL SERVICES TECHNICAL OVERVIEW

4 VoltDB s Architecture VoltDB was built to bring speed, scalability and performance to operational applications. Below are some of the features that enable VoltDB to meet the needs of modern applications. Partitioned by core, with a single thread per core VoltDB partitions workload by CPU core, forcing all work for a given partition towards a single thread running on a single core. Transactions cannot span invocations Applications cannot lock records in VoltDB. All transactions begin, do their work and end without ever leaving the core. This means that applications that work with shared finite resources can t trip each other up as they all try to decrement the same counter at the same time. It also cuts out CPU overhead that otherwise would be spent on latching and locking. Stored Procedures Because transactions cannot span invocations, stored procedures are required to provide ACID for transactions. Stored Procedures drastically reduce the number of network trips for complicated transactions. VoltDB s Stored Procedures are written in Java, which avoids developers having to learn a new language. JDBC access also is supported. An open and asynchronous API VoltDB s underlying client API is asynchronous. It s both published and relatively simple. This works well in a financial services context, as it means clients don t have to create more and more threads as workloads go up. Workload replicated by partition for High Availability Because VoltDB partitions work by CPU core, HA can be implemented by having two or more different cores to do the same queue of work in the same order. Unlike a legacy RDBMS, where an outage leads to an IO storm as the surviving database nodes try and figure out the deceased node was doing when it died, VoltDB will wait 1-3 seconds to make sure the deceased node is in fact gone and then continue doing the same work on the surviving partitions. This work is hidden behind the scenes from the client. In-memory only The only time VoltDB directly reads from disk is when starting. You can configure it to flush to disk after each transaction, but most customers are happy to use a HA cluster and rely on the fact that transactions will be micro-batched to disk on two separate servers within a few 10s of milliseconds. Shared Nothing architecture, no hardware dependencies VoltDB does not require a SAN, SSD or shared disk storage to work. It s agnostic in its choice of Linux and supports virtualization. It runs well on private, hybrid and public cloud implementations. Rule-based optimizer Query plans in VoltDB never change. This is a significant advantage as the behavior of a SQL statement will not change once deployed. Feeds to downstream systems A classic weakness of OLTP databases is the need to issue SELECT * queries to unload data for downstream systems to use. VoltDB has an at least once queue mechanism that looks to developers like a SQL table when you insert into it you create an entry in the queue, which can then be sent to HTTP, Kafka, HDFS, wherever. Active-Active-Active Cross-Datacenter replication Not only does VoltDB support Active-Active clusters at different locations, as of v7.0 it supports Active-Active-Active clusters. This means that data can be in multiple locations and the data center nearest the user can handle the request. Obviously, there is a need to manage conflicts in this scenario. 4 VOLTDB FOR FINANCIAL SERVICES TECHNICAL OVERVIEW

5 SQL and other methods of communication VoltDB supports JDBC-type access. VoltDB s design uses stored procedure calls where you send in a set of parameters and get back a list of results as a transaction. Procedure calls are both asynchronous and sent to all the nodes handling the data in question. Each node then executes the procedure in a deterministic manner. Node failures do not require reading from disk to recover, as the surviving nodes are already caught up. The asynchronous nature of the procedure call has big advantages in the financial services space at no point does code have to wait for a response while milliseconds pass. An example would be ending a trading session, which can be a fire and forget operation in VoltDB the application creates an asynchronous call to end the session and continues on its way without having to wait for a confirmation. The format VoltDB uses for messages is published, so writing new clients in new languages is easy. VoltDB supports C#, C++, Erlang, Go, Java, JDBC, JSON + HTTP (REST), Node.js, PHP, Python, and Ruby ACID compliant VoltDB supports repeatable reads - none of the data an application is working with can be changed by anyone else during a transaction. Procedure calls are effectively transactions - everything in a procedure either happens or it doesn t. VoltDB s HA architecture involves executing the same code with the same inputs on multiple servers at the same time. Even if a node dies in the middle of a transaction, the surviving nodes will complete the same transaction. VoltDB Disk Interaction/Export Tables Export tables, which look like tables from a SQL level, are at least once queues to CSV, HDFS, Kafka etc. Use VoltDB adaptors or write your own. Download VoltDB here. 5 VOLTDB FOR FINANCIAL SERVICES TECHNICAL OVERVIEW

6 VoltDB Scale In-memory deployments of up to 32 Nodes with in excess of 5TB total RAM. Support for individual nodes with 384GB of RAM. VoltDB Speed VoltDB holds the YCSB benchmark record 2.4 million TPS. See softlayer_benchmark_0.pdf By using a C++ core VoltDB takes advantage of CPU cache pre-fetching. No server stalling for Java garbage collection much of the operational work is done off the heap About 12X faster than Oracle Using a single thread per partition avoids overhead of latching, locking, and read consistency VoltDB s architecture is great for counting things or working with shared, finite resources. VoltDB Licensing Development and production licenses available. VoltDB one-line Comparisons MongoDB VoltDB has better support for transactions Cassandra VoltDB is immediately consistent for 100% accuracy Oracle 12x faster with much lower license costs SMACK Stack better performance with much less complexity and no glue code About VoltDB VoltDB is the only in-memory transactional database for modern applications that require an unprecedented combination of data scale, volume, and accuracy. Unlike other databases, including OLTP, Big Data, and NoSQL, that force users to compromise, only VoltDB supports all three modern application data requirements: 1. Millions VoltDB processes a relentless volume of data points from users and data sources. 2. Milliseconds VoltDB ingests, analyzes, and acts on data in less than the blink of an eye % Data managed by VoltDB is always accurate, all the time, for all decisions. Telcos, Financial services, Ad Tech, Gaming, and other companies use VoltDB to modernize their applications. VoltDB is preparing energy, industrial, telco and other companies to meet the challenges of the IoT. VoltDB was founded by a team of world-class database experts, including Dr. Michael Stonebraker, winner of the coveted ACM Turing award. VoltDB, Inc. 209 Burlington Road, Suite 203, Bedford, MA voltdb.com