What s New in DataStax Enterprise 3.1? A Guide for Developers, Architects and IT Managers. White Paper BY DATASTAX CORPORATION November 2013

Transcription

1 What s New in DataStax Enterprise 3.1? A Guide for Developers, Architects and IT Managers White Paper BY DATASTAX CORPORATION November

2 Table of Contents Abstract 3 Introduction 3 What s New in DataStax Enterprise Edition 3.1? 3 Performance Enhancements 4 Improved Native Memory Management 4 Faster Node Bootup/Startup 4 Murmur3Partitoner 4 Query Profiling/Tracing 4 New Compression Option 4 Miscellaneous Cassandra Performance Enhancements 4 Management Enhancements 5 Virtual Nodes 5 Parallel Leveled Compaction 5 Improved JBOD Functionality 5 Developer Enhancements 5 Collections 5 Atomic Batches 5 Native/Binary CQL Transport 6 Concurrent Schema Changes 6 Composite Column Support for Hive and Solr 6 Other CQL Enhancements 6 Enterprise Search Improvements 7 Updates to Security 7 Improvements to Visual Enterprise Manageability 7 Why DataStax Enterprise? 8 Use Cases Handled by DataStax Enterprise 9 Conclusion 10 About DataStax 10 2

3 Abstract DataStax Enterprise Edition is an enterprise NoSQL platform, built with the C* Seal of Approval: a production-certified version of Apache Cassandra. Architected to manage real-time, batch analytics, and enterprise search data all in the same database cluster, DataStax Enterprise 3.1 features numerous components and upgrades that make management and development easier than ever before, while delivering unmatched performance. DataStax Enterprise 3.1 also continues to provide the most comprehensive security feature set of any NoSQL solution provider. This paper details the enhancements you ll find in DataStax Enterprise Edition 3.1. Introduction DataStax Enterprise Edition is an enterprise NoSQL platform, built with the C* Seal of Approval: a production-certified version of Apache Cassandra. It is architected to manage real-time, batch analytics, and enterprise search data all in the same database cluster. DataStax Enterprise has three components: 1. DataStax Enterprise Server, which is an enterprise-class NoSQL data management platform that uses Apache Cassandra for real-time data management, and provides easy methods for running analytics and enterprise search operations on Cassandra data. 2. OpsCenter Enterprise, which is a visual, browser-based management tool that allows administrators to manage all database clusters, whether they are on premise, across multiple data centers, or in the cloud, from a single interface. 3. Expert support, which is delivered by the NoSQL professionals at DataStax and includes around-the-clock coverage. This paper discusses the enhancements included in version 3.1 of DataStax Enterprise Edition. What s New in DataStax Enterprise Edition 3.1? DataStax Enterprise Edition 3.1 includes a number of improvements in the following areas: Performance new additions provide for faster query and DML response times, more efficient data compression, and better insight into performance bottlenecks. Management altering an existing database cluster (adding, removing nodes, etc.) is now much easier in DataStax Enterprise 3.1, with the end result being even less time needed for cluster management and maintenance. Developer Enablement many new Cassandra and Solr features are included in DataStax Enterprise 3.1, giving developers more flexibility and capabilities for developing online applications plus reducing the amount of manual code that needs to be written. The following sections explain these feature additions in more detail. 3

4 Performance Enhancements DataStax Enterprise 3.1 contains a number of performance enhancements that greatly contribute to better performance and scale. For many use cases, up to 10x the amount of data per Cassandra node can now be managed with the same levels of performance as prior versions. Such capabilities can reduce hardware costs and personnel overhead for large scaleout deployments. The following features contribute to 3.1 s enhanced performance and scale. Improved Native Memory Management DataStax Enterprise 3.1 uses native memory vs. the Java heap for Cassandra bloom filters, compression metadata, and the partition summary. This allows Cassandra to handle much more data per node than in the past, and reduces garbage collection operations. Faster Node Bootup/Startup Faster startup/bootup times for each node in a cluster is realized in 3.1, with internal tests performed at DataStax showing up to 80% less time needed to start a DataStax Enterprise node. The startup reductions were realized through more efficient sampling and loading of SSTable indexes into memory caches. Murmur3Partitoner Version 3.1 introduces a new Cassandra partitioner: the Murmur3Partitioner, based on the Murmur3 hash. The Murmur3 hash is 3x-5x faster than the prior MD5 has used in earlier versions of Cassandra; this translates into performance gains of over 10% for index-heavy workloads. Query Profiling/Tracing A new Cassandra performance diagnostic utility is available in 3.1, which is aimed at helping to understand, diagnose, and troubleshoot CQL statements sent to a Cassandra cluster. Users can interrogate individual CQL statements in an ad-hoc manner, or perform a system-wide collection of all queries/commands that are sent to a cluster. New Compression Option LZ4 compression is now included as a compression option. LZ4 provides about a 50% performance improvement over the current default Snappy compression. Miscellaneous Cassandra Performance Enhancements Other performance enhancements include: A new approach to index maintenance, improving the speed at which indexes are updated. More efficient and faster streaming of data during bootstrap or repair operations. Faster replica recovery via a new concurrent hint delivery mechanism. The compaction_throughput_mb_per_sec setting now works better with large partitions. In prior releases, Cassandra only checked the compaction throughput between partitions, so large partitions could still cause spikes of i/o demand. The removal of cell-name bloom filters results in faster queries against large partitions, since they are no longer part of the partition header. The enablement of a thread-local allocation setting results in a 15% gain in read performance. 4

5 Management Enhancements Database clusters running DataStax Enterprise 3.1 are now easier to manage due to the following management enhancements. Virtual Nodes Virtual nodes or vnodes change the previous Cassandra paradigm of using one token or range per node, to many per node, which makes a cluster much easier to manage and grow. Vnodes provide the following core benefits: Rather than just one or a couple of nodes participating in bootstrapping new nodes, all nodes participate in the operation, thus parallelizing the task. End result: much faster performance for node addition operations. Vnodes automatically maintain the data distribution / balances of a cluster so there is no need to perform any rebalance operation after a cluster has been modified. Note that virtual nodes may be combined with traditional token-assigned nodes in a DataStax Enterprise cluster. For example, Cassandra nodes may use virtual nodes, whereas Solr nodes may use traditional token assignments in the same database cluster. Parallel Leveled Compaction Cassandra parallel leveled compaction provides more efficient and faster compaction operations for deployments that especially target SSD hardware. Whereas the general idea for compaction processes is to mitigate impact on the overall operation of nodes (which typically results in longer compaction times but less resource intensive operations), SSD implementations lend themselves to speeding up compaction tasks, which is what paralleled level compaction does. Improved JBOD Functionality In prior versions, a single disk going down in a JBOD (just a bunch of disks) configuration had the potential to make an entire node unavailable for I/O operations. A new disk_failure_policy configuration setting allows architects to choose from a number of new policies that deal with disk failure so this doesn t have to be the case. Developer Enhancements Version 3.1 provides a number of new features that help developers create applications with DataStax Enterprise. Collections DataStax Enterprise 3.1 includes a new mechanism for storing Cassandra data called collections. The general idea behind collections is to provide easier methods for inserting and manipulating data that consists of multiple items that a user wants to store in a single column; for example, multiple addresses for a single employee. There are three different types of collections: (1) sets; (2) lists; (3) maps. Atomic Batches Prior versions supported Cassandra batch operations, which allowed users to group related updates into a single statement. If some of the replicas for the batch failed mid-operation, the coordinator would hint those rows automatically. However, if the coordinator itself failed in mid operation, users could end up with partially applied batches. In DataStax Enterprise 3.1, batch operations are guaranteed by default to be atomic. The default functionality for batches is for any batch to be atomic (i.e. all or nothing). It should be noted that there is a performance penalty for using atomic batches, so for use cases that necessitate batch operations, but either have client side workarounds or other methods for ensuring batch 5

6 atomicity, a BEGIN UNLOGGED BATCH command is supplied for cases when performance is more important than atomicity guarantees. Native/Binary CQL Transport Prior to version 3.1, the Cassandra Query Language (CQL) API had been using Thrift as a network transport, but now a new binary protocol is available for CQL that does not require Thrift. There are a number of benefits that the new native CQL transport provides: Thrift is a synchronous transport meaning only one request can be active at a time for a connection. By contrast, the new native CQL transport allows each connection to handle more than one active request at the same time. This translates into client libraries only needing to maintain a relatively low number of open connections to a Cassandra node in order to maximize performance, and helps scale large clusters. Thrift is an RPC mechanism, which means a user cannot have a Cassandra server push information to a client. However the new native CQL protocol allows clients to register for certain types of event notifications from a server. Currently supported events include [1] cluster topology changes (e.g. a node joins the cluster, is removed, is moved, etc.); [2] status changes (e.g. a node is detected up/down); [3] schema changes (e.g. a table has been modified, etc.). These new capabilities allow clients to stay up to date with the state of the Cassandra cluster without having to poll the cluster regularly. The new native protocol allows for messages to be compressed if desired. Thrift is still the default transport in 1.2. To use the new binary protocol, the start_native_transport option is changed to true in the cassandra.yaml file. Also needed are client drivers that support the new binary protocol such as the new DataStax Java and.net drivers. Concurrent Schema Changes While earlier versions introduced the ability to modify objects in a concurrent fashion across a cluster, they did not include support for programmatically creating and dropping column families / tables (either permanent or temporary) in a concurrent manner. This is now supported in 3.1, which means multiple users may add/drop tables at the same time in the same cluster. Composite Column Support for Hive and Solr Composite columns are now no longer constrained to only work with Cassandra. Users can now create objects with composite columns in both Hive/Hadoop and Solr. Other CQL Enhancements Numerous CQL enhancements are in version 3.1. Changes include a new ALTER KEYSPACE statement, syntax additions to understand how long a TTL column has remaining, support for conditional operators, and much more. For a full list of all CQL additions, please see the DataStax online documentation. 6

7 Enterprise Search Improvements DataStax has certified Solr 4.3 for production use in DataStax Enterprise. With Solr 4.3 come nearly 60 new features, mostly targeted at providing developers with more functionality for creating search applications. Solr 4.3 also brings a number of improvements that provide greater quality and stability to the platform. In addition to Solr 4.3, version 3.1 of DataStax Enterprise includes the following new search features, which help make Solr near real-time in its data management handling: Per segment filters. Multi-value facets. Updates to Security DataStax Enterprise 3.1 includes a number of maintenance patches to several of the security features that were introduced in version 3.0. DataStax Enterprise 3.1 continues to provide the most comprehensive security feature set of any NoSQL solution provider, with data protection capabilities including the following: Internal authentication supports standard login ID/password external security for Cassandra. External authentication provides integration with Kerberos and LDAP (with Kerberos) to secure Cassandra, Hadoop and Solr using popular 3 rd party security software. Object permission management controls who can create and edit objects as well as view and manipulate data via the familiar RDBMS GRANT and REVOKE paradigm. Client to node encryption ensures that data in flight from client machines to/from a database cluster cannot be stolen. Node to node encryption guarantees that data being moved between nodes cannot be intercepted and accessed. Transparent data encryption protects data at rest so that sensitive data is not compromised. Data auditing allows for administrators to understand who is accessing the database and what actions they have performed. Improvements to Visual Enterprise Manageability OpsCenter Enterprise 3.2 is included with the DataStax Enterprise NoSQL platform and provides visual management and monitoring for Cassandra, Hadoop, and Solr. Enhancements were provided in OpsCenter 3.0 for visually creating and editing database clusters, as well as restoring a cluster from backup. Version 3.2 supplies a number of improvements to these features including: Support for provisioning database clusters with virtual nodes. Recognizing objects created with CQL3. Improved monitoring with a new caching component that results in performance metrics being displayed about 10x faster than in previous versions. Numerous stability and visual user interface improvements. 7

8 Why DataStax Enterprise? Version 3.1 of DataStax Enterprise continues to help modern businesses transform the way they do business. Today, line-of-business (LOB) applications are evolving to meet the need of providing greater capabilities and data insights more than ever before, and this necessitates a new kind of technology aimed at handling today s data in a technically efficient and cost effective way. Using Apache Cassandra, DataStax Enterprise provides the type of technology and architecture that targets modern LOB applications. Moreover, DataStax Enterprise is unique in the data management marketplace in that it smartly handles all key data dimensions real time, batch analytic, and enterprise search all in one easily managed database cluster. The benefits delivered by DataStax Enterprise can be summarized as follows: Feature Benefit Production-Certified Cassandra Continuously Available Analytics on Cassandra data Fault-Tolerant Search on Cassandra data Workload Separation/Isolation No Need for ETL Easy Data Migration Simple Log Integration Enterprise Management Control Expert Support Cost Efficient Apache Cassandra is a massively scalable NoSQL database that is an acknowledged industry leader at handling today s LOB systems. DataStax certifies a version of Cassandra for its big data platform via lengthy testing, benchmarking, validation with 3 rd party software, and defect resolution to ensure a chosen version is ready for production environments. Running analytics on Cassandra data is made easy via the integration of a number of Hadoop components. Support for MapReduce, Hive, Pig, Mahout, and Sqoop are included in DataStax Enterprise. Enterprise search operations are run on Cassandra data with built-in Solr integration. The mixed workload problem is solved in the platform as no workload (real time, analytics, search) competes with any other for compute resources; all are isolated and yet integrated together. The need to extract-transform-load data from real time to analytic to search systems goes away as builtin replication keeps all data domains in sync. Migrating data from existing RDBMS systems is easy via built-in migration utilities. Third party migration tools also support the platform. Application logging data is easily consumed via logging interfaces, and then can be analyzed and searched.. Administrators are instantly productive and save time with OpsCenter Enterprise, which allows management of all database clusters within a single interface. Professional, around-the-clock support ensures questions get quickly answered and help is available to ensure applications stay online. Cost reductions over typical RDBMS vendors routinely run 80%. 8

9 Use Cases Handled by DataStax Enterprise Because DataStax Enterprise is a comprehensive and integrated big data platform, it is capable of handling use cases that have real-time, batch analytics, and enterprise search requirements. Example use cases should such as the following can be managed by DataStax Enterprise: Real-Time: Time series data Device/Sensor/Data exhaust systems Distributed applications Media streaming Online Web retail (transactional, shopping carts, etc.) Real-time data analytics Social media capture and analysis Web click-stream analysis Write-intensive transactional systems Batch Analytics: Buyer behavior analytics Compliance/regulatory analysis Customer recommendation output Fraud detection Risk analysis Sales program campaign analysis Supply chain analytics Batch Web clickstream analysis Enterprise Search: General Web search Web retail faceted (categorization) search Search/hit prioritization and highlighting Application log search and analysis Document (PDF, MS Word, etc.) search and analysis Geospatial search Real estate location and property search Social media match ups 9

10 Conclusion DataStax Enterprise Edition 3.1 provides increased performance, scale, and developer flexibility for creating modern line-of-business applications that have real-time, analytic, and enterprise search requirements. For more information on DataStax Enterprise Edition 3.1, visit for downloads, online documentation, and more. Note that DataStax Enterprise Edition 3.1 may be downloaded and used free of charge in development environments with no restrictions (e.g. data size, RAM, CPU, etc.), however production deployments do require a subscription be purchased. For information on subscription pricing, please send an to DataStax at info@datastax.com. About DataStax DataStax powers the online applications that transform business for more than 300 customers, including startups and 20 of the Fortune 100. DataStax delivers a massively scalable, flexible and continuously available big data platform built on Apache Cassandra. DataStax integrates enterprise-ready Cassandra, Apache Hadoop for analytics and Apache Solr for search across multi-data centers and in the cloud. Companies such as Adobe, Healthcare Anytime, ebay and Netflix rely on DataStax to transform their businesses. Based in San Mateo, Calif., DataStax is backed by industry-leading investors: Lightspeed Venture Partners, Crosslink Capital and Meritech Capital Partners. For more information, visit DataStax or follow 10