Pro MySQL NDB Cluster

Transcription

1 Pro MySQL NDB Cluster Jesper Wisborg Krogh Mikiya Okuno

2 Pro MySQL NDB Cluster Jesper Wisborg Krogh Sydney, New South Wales, Australia Mikiya Okuno Tochigi, Japan ISBN-13 (pbk): ISBN-13 (electronic): Library of Congress Control Number: Copyright 2017 by Jesper Wisborg Krogh and Mikiya Okuno This work is subject to copyright. All rights are reserved by the Publisher, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation, broadcasting, reproduction on microfilms or in any other physical way, and transmission or information storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar methodology now known or hereafter developed. Trademarked names, logos, and images may appear in this book. Rather than use a trademark symbol with every occurrence of a trademarked name, logo, or image we use the names, logos, and images only in an editorial fashion and to the benefit of the trademark owner, with no intention of infringement of the trademark. The use in this publication of trade names, trademarks, service marks, and similar terms, even if they are not identified as such, is not to be taken as an expression of opinion as to whether or not they are subject to proprietary rights. While the advice and information in this book are believed to be true and accurate at the date of publication, neither the authors nor the editors nor the publisher can accept any legal responsibility for any errors or omissions that may be made. The publisher makes no warranty, express or implied, with respect to the material contained herein. Managing Director: Welmoed Spahr Editorial Director: Todd Green Acquisitions Editor: Jonathan Gennick Development Editor: Laura Berendson Technical Reviewer: Charles Bell Coordinating Editor: Jill Balzano Copy Editor: Kezia Endsley Compositor: SPi Global Indexer: SPi Global Artist: SPi Global Distributed to the book trade worldwide by Springer Science+Business Media New York, 233 Spring Street, 6th Floor, New York, NY Phone SPRINGER, fax (201) , orders-ny@springer-sbm.com, or visit Apress Media, LLC is a California LLC and the sole member (owner) is Springer Science + Business Media Finance Inc (SSBM Finance Inc). SSBM Finance Inc is a Delaware corporation. For information on translations, please rights@apress.com, or visit rights-permissions. Apress titles may be purchased in bulk for academic, corporate, or promotional use. ebook versions and licenses are also available for most titles. For more information, reference our Print and ebook Bulk Sales web page at Any source code or other supplementary material referenced by the author in this book is available to readers on GitHub via the book s product page, located at For more detailed information, please visit Printed on acid-free paper

3 Contents at a Glance About the Authors...xxi About the Technical Reviewer...xxiii Acknowledgments...xxv Introduction...xxvii Part I: The Basics... 1 Chapter 1: Architecture and Core Concepts... 3 Chapter 2: The Data Nodes Part II: Installation and Configuration Chapter 3: System Planning Chapter 4: Configuration Chapter 5: Installation Chapter 6: Replication Part III: Daily Tasks and Maintenance Chapter 7: The NDB Management Client and Other NDB Utilities Chapter 8: Backups and Restores Chapter 9: Table Maintenance Chapter 10: Restarts Chapter 11: Upgrades and Downgrades Chapter 12: Security Considerations Chapter 13: MySQL Cluster Manager iii

4 at a Glance Part IV: Monitoring and Troubleshooting Chapter 14: Monitoring Solutions and the Operating System Chapter 15: Sources for Monitoring Data Chapter 16: Monitoring MySQL NDB Cluster Chapter 17: Typical Troubles and Solutions Part V: Development and Performance Tuning Chapter 18: Developing Applications Using SQL with MySQL NDB Cluster Chapter 19: MySQL NDB Cluster as a NoSQL Database Chapter 20: MySQL NDB Cluster and Application Performance Tuning Index iv

5 Contents About the Authors...xxi About the Technical Reviewer...xxiii Acknowledgments...xxv Introduction...xxvii Part I: The Basics... 1 Chapter 1: Architecture and Core Concepts... 3 Terminology... 3 Characteristics and Features... 6 Architecture... 6 Features... 6 Limitations... 8 Use Cases Node Types Management Nodes Data Nodes API and SQL Nodes Built-In High Availability Shared Nothing Architecture Heartbeats Data Node Failure Handling and Arbitration Summary v

6 Chapter 2: The Data Nodes Single-Threaded Versus Multi-Threaded Data Nodes Thread Types Performance Considerations Replicas Primary and Backup Replicas MySQL NDB Cluster 7.5: Read from Backup Replica Node Groups Partitions Automatic Partitioning User-Defined Partitioning MySQL NDB Cluster 7.5: Partition Balancing MySQL NDB Cluster 7.5: Fully Replicated Tables Case Study: Partition Distribution D for Durability Duplication of Data Local Checkpoints (LCPs) The Redo Logs Global Checkpoints (GCPs) Restarts and Processes Data Node Internals Memory Usage Kernel Blocks Signals Job Buffer Send and Receive Buffers Triggers Epochs Master Node vi

7 Data and Indexes Data Memory and Index Memory On-Disk Data BLOB, TEXT, and JSON Columns Cases Studies: Investigating the Schema Objects The ndb_show_tables Utility The ndb_desc Utility The NDB File System Summary Part II: Installation and Configuration Chapter 3: System Planning Determine Your Priorities High Availability Requirements Data Node SQL Node Management Node Prepare for Network Partitioning Scalability Disaster Recovery Typical Topologies Number of Replicas Maximum Number of Data Nodes Maximum Number of Total Nodes Arbitration Rank Placing the SQL Node and the Data Node on the Same Machine Typical Topology Examples Platform Considerations Processor Type and Operating System CPU Performance and Characteristics vii

8 Memory Consumption Disk Performance Virtual Machines Network Design Network Devices Network Redundancy Direct Connection Security Considerations Extending Capacity Using Disk Objects Performance Considerations Storage Requirements Memory Consumption Summary Chapter 4: Configuration Configuration Overview Formatting config.ini...85 Restart Types Management Node Options Major Options for Management Node Vital Point for Management Node Configuration Data Node Options Basic Options Memory Data Storage Options Schema Object Options Transaction Options Estimate for Total Memory Consumption Checkpoint Options Estimating Redo Log Size Multi-Threading Options Consideration for CPU Properties viii

9 Backup Options Transporter Options Disk Object Options Heartbeat and Watchdog Options Logging Options Recommended Configuration Strategy SQL Node Options Major Options for SQL Node Vital Point for SQL Node Configuration TCP Transporter Options Major Options for Transporter Transporter via Separate Network Path Program Startup Options (my.cnf) Common Options Major Options for ndb_mgmd Options for ndbd/ndbmtd Options for mysqld Location of Option Files Summary Chapter 5: Installation Package Installation Obtaining Packages Installation on Linux Installation on Windows Installation on macos Installing MySQL NDB Cluster Instances Using Auto Installer Verifying Installation Configuration Files Initial Startup Checking the Status ix

10 Uninstalling Packages Tar.gz and Zip Archive Package RPM Package Windows Installer Package macos Native Package Summary Chapter 6: Replication NDB Cluster Replication Overview Replication Architecture Overview Replication Channel Failover NDB Cluster Replication Tables Use Cases and Advantages of NDB Cluster Replication Setting Up NDB Cluster Replication Setting Up NDB Cluster Replication with an Empty Database Setting Up NDB Cluster Replication with an Existing Database (Offline) Setting Up NDB Cluster Replication with Existing Database (Online) Failing Over NDB Cluster Replication Channel NDB Cluster Replication Daily Maintenance Monitoring NDB Cluster Replication Restarting Master Cluster NDB Cluster Replication Performance Tuning Conflict Detection and Resolution Multi-Master Replication Conflicts Caused by Multi-Master Replication Conflict Detection Methods Conflict Detection for Read Operations Setting Up Conflict Detection and Resolution Monitoring Conflict Detection Conflict Detection Case Study Application Modifications Required for Conflict Detection x

11 Cautions and Limitations of Conflict Detection Replication to InnoDB Requirements and Limitations Setting Up Replication to InnoDB Tips When Using InnoDB as a Slave MySQL Server Options Related to Cluster Replication Notes and Limitations of NDB Cluster Replication Summary Part III: Daily Tasks and Maintenance Chapter 7: The NDB Management Client and Other NDB Utilities The NDB Management Client Invoking the NDB Management Client Getting Help from Inside the Client Setting the Prompt Display the Cluster Status Single User Mode Create Reports Purge Stale Sessions Other NDB Utilities Summary Chapter 8: Backups and Restores Backups and Backup Procedures Native NDB Cluster Online Backups Overview Implementation Details Starting and Aborting Backups Choosing the Backup ID Backup Monitoring Backup Configuration xi

12 Logical Backups and Binary Logs Consistency Considerations Creating Logical Backups Logical Backups from MySQL Workbench Backing Up Privileges Binary Logs Restores The ndb_restore Program Restore Schema Full Data Restores Restore to a Different Number of Data Nodes Partial Data Restores Restores Using MySQL Workbench Point-In-Time Recovery (PITR) Summary Chapter 9: Table Maintenance Schema Changes Distributing Schema Changes and the Global Schema Lock Online Versus Offline Schema Changes Offline Schema Changes Online Schema Changes ALTER TABLE Algorithm ALTER TABLE Examples Default Behavior Adding a Column with the Explicit Column Format Specifying Algorithm and Lock Type Attempting Unsupported In-Place Changes Schema Changes in Version 7.2 and Earlier Reorganize Partitions xii

13 Defragmentation Index Statistics Index Statistics Internals Maintaining Index Statistics Options and Status Variables Summary Chapter 10: Restarts Restart Types Node Restart Initial Node Restart System Restart Initial System Restart Rolling Restart Stopping and Starting Nodes Management Nodes Data Nodes API/SQL Nodes Restart Related Configuration Startup Process Monitoring Restarts The Management Client The ndb_waiter Utility The ndbinfo.restart_info Table A Restart Seen in the Logs Example Restart Scenarios Configuration Change Adding a Management Node Adding Data Nodes Adding Data Nodes with Node Group Pre-Allocated Adding API/SQL Node xiii

14 xiv Recovering from a Corrupt NDB File System Initial System Restart Summary Chapter 11: Upgrades and Downgrades Upgrades Upgrade Types Upgrade Considerations Downgrades Performing Upgrades and Downgrades Online Upgrades and Downgrades Offline Upgrades and Downgrades Case Studies Online Upgrade Using Generic Binaries Upgrade from 7.4 to 7.5 Using RPM Online Downgrade Offline Upgrade Summary Chapter 12: Security Considerations Network Security Updates Accounts and Privileges in the SQL Nodes Accounts and Password Management SSL/TLS Certificates The Access Control and Privilege System Distributed Privileges Enabling Distributed Privileges Disabling Distributed Privileges Special Considerations The Operating System and the Rest of the Infrastructure Summary

15 Chapter 13: MySQL Cluster Manager Background Terminology Architecture Commands Limitations Download, Installation, and Configuration Downloading Installation on Linux Installation on Microsoft Windows Upgrading Configuration Starting and Stopping MySQL Cluster Manager The MySQL Cluster Manager Client Managing a Cluster Installing the Cluster Binaries Preparing the Cluster Through the mcm Client Cluster Configuration: Auto Tuning Cluster Configuration: The set Command Cluster Configuration: The get Command Cluster Configuration: The reset Command Starting and Stopping Processes Configuration of an Online Cluster Backups Restoring a Backup Upgrades Troubleshooting MySQL Cluster Manager Error Messages and the Log Self-Healing Agents Summary xv

16 Part IV: Monitoring and Troubleshooting Chapter 14: Monitoring Solutions and the Operating System Why Monitor? Establish a Baseline Perform a Root Cause Analysis Perform Preventive Maintenance Monitoring Solutions MySQL Monitoring Solutions MySQL Enterprise Monitor (MEM) Components Installation and Upgrades Features The Operating System CPU Usage Network Usage Disk Usage Memory Usage Logs Summary Chapter 15: Sources for Monitoring Data The Information Schema Information Schema Tables Using the Information Schema The Information Schema and NDB Cluster The Performance Schema Performance Schema Threads Performance Schema Tables Overview Setup Tables and Configuration Event Tables xvi

17 Summary Tables Connection and Thread Tables Variable and Status Tables Replication Tables Instance Tables Lock Tables The sys Schema Installation sys Schema Objects sys Schema Configuration Command-Line Usage and Examples MySQL Workbench Performance Reports SHOW Statements MySQL Logs SQL Node Error Logs Audit Logs Summary Chapter 16: Monitoring MySQL NDB Cluster The NDB Cluster Information Database (ndbinfo) Configuration The ndbinfo Views NDB Cluster Reports Memory Usage Report Disk Page Buffer Report Transporters Report Disk Write Speed Report Locks Report Log Buffers and Spaces Report Configuration Report xvii

18 NDB Cluster Logs Cluster Logs Data Node Logs Summary Chapter 17: Typical Troubles and Solutions Typical Problems on Data Nodes General Information about Node Failures Cluster Log Node Log Error Log Trace Files Core Files NDB Error Reporter Utility Watchdog Timeout LCP Watchdog Timeout Swap Insanity GCP Stop Network Partitioning (Split Brain) Unplanned Shutdown of Entire System Typical Problems on SQL Nodes Errors While Executing Queries Resource Temporary Errors Non-Temporary Errors Connection to Data Node Is Lost Errors Related to Transaction Handling Crashes Typical Problems on Management Nodes Summary xviii

19 Part V: Development and Performance Tuning Chapter 18: Developing Applications Using SQL with MySQL NDB Cluster Designing Tables Creating NDB Cluster Tables Supported Data Types Three Types of Indexes Defining Indexes The T-Tree Index Estimating Table Size Estimating Required Objects per Table Defining Foreign Key Constraints Reviewing Table Definition Disk Data Tables Consideration for Normalization Major Limits Regarding Table Design Accessing Data via SQL Connecting to SQL Node Transaction Handling for NDBCluster Tables Error-Handling Techniques Summary Chapter 19: MySQL NDB Cluster as a NoSQL Database Why NoSQL? Accessing Data via memcached Why Use NDB-memcached Setting Up NDB-memcached Defining Mapping to NDB Cluster Tables Accessing Data via the NDB API Why Use the NDB API? Installing Header Files and Libraries for the NDB API Building an Application with the NDB API xix

20 References and Examples Typical Program Flow Simple Read Example Accessing Data Using NdbRecord Scan Example Error-Handling Considerations Accessing Data via ClusterJ Installing ClusterJ Writing a ClusterJ Application ClusterJ Example Summary Chapter 20: MySQL NDB Cluster and Application Performance Tuning MySQL NDB Cluster Tuning Disabling Powersave and CPU Frequency Scaling CPU Binding Strategy Disk Type and File System Block Size SQL Tuning Commit Sizing Non-Transactional Batch Processing Engine Condition Pushdown Optimization Optimizing Joins Optimizing Partitioning Optimizing Access from SQL Node to Data Node Adding Nodes Using NoSQL API in Conjunction with SQL Summary Index xx

21 About the Authors Jesper Wisborg Krogh is a member of the Oracle MySQL Support team and has spoken on several occasions at Oracle OpenWorld. He has a background with a Ph.D. in computational chemistry before changing to work with MySQL and other software development in Jesper lives in Sydney, Australia and enjoys spending time outdoors walking, traveling, and reading. His areas of expertise include MySQL Cluster, MySQL Enterprise Backup, and the Performance and sys schemas. He is an active author in the Oracle Knowledge Base and regularly blogs on MySQL topics. Mikiya Okuno is a member of the Oracle MySQL Support team based in Japan and has written four database-related books in Japanese. He joined the MySQL support team in 2007 when MySQL AB was an independent company. Before joining MySQL, he worked at Sun Microsystems (which acquired MySQL AB, then was acquired by Oracle) as a technical support engineer. He has over 16 years of technical support experience in the computer industry. Mikiya is a self-confessed computer geek and loves Gentoo Linux OS and ErgoDox keyboard for his desktop. Emacs is his preference. xxi

22 About the Technical Reviewer Dr. Charles Bell conducts research in emerging technologies. He is a member of the Oracle MySQL Development team as a senior developer working on a variety of database administration and high availability projects. He lives in a small town in rural Virginia with his loving wife. He received his Ph.D. in engineering from Virginia Commonwealth University in His research interests include database systems, software engineering, sensor networks, and 3D printing. He spends his limited free time as a practicing Maker focusing on microcontroller and 3D printers and printing projects. xxiii

23 Acknowledgments We would like to thank all of the people who made this book possible. First of all, Jonathan Gennick, Jill Balzano, and Laura Berendson ensured its progress. Jonathan and Jill, your guidance has been invaluable. However, the rest of the team at Apress should not be forgotten either: thank you. A special thanks to our technical reviewer, Charles Bell. It has been great to get your feedback and learn from your experience as an author. Our colleagues at Oracle also deserve credit for this book becoming a reality. The discussions with the MySQL NDB Cluster developers and our direct colleagues in the MySQL Support team were key to becoming an expert on MySQL NDB Cluster. It is hard to start mentioning names while not forgetting any, but a definite thanks to our manager Adam Dixon for the support. Thanks to the developers Frazer Clement and Mikael Ronström for, time and time again, explaining technical matters of MySQL NDB Cluster, as well as Pekka Nousiainen for his feedback on the index statistics implementation. Of course, Jon Stephens must not be forgotten for his work on the MySQL NDB Cluster chapter in the MySQL Reference Manual. Last but not least, a warm-hearted thanks to our wives, Ann-Margrete and Junko, as well as Mikiya s children, who put up with us while we were busy writing this book. xxv

24 Introduction MySQL NDB Cluster is part of the MySQL family of products. There are few databases matching as many buzzwords as MySQL NDB Cluster: high availability, SQL, NoSQL, in-memory database, automatic failure handling, sharding, etc. This book discusses how these properties are integrated into MySQL NDB Cluster and how they are not just buzzwords, but real features. Whom This Book Is For The book has been written for database administrators who are looking into deploying MySQL NDB Cluster or already have a cluster in production and want to increase their knowledge to be able to handle routine administrative tasks and troubleshooting. While we have attempted to write the book assuming as little existing knowledge as possible, it is an advantage to have at least some SQL database background. How This Book Is Structured The chapters have been divided into five parts, with each part covering related topics. The journey starts with some background knowledge of how MySQL NDB Cluster works, then moves on to installation and configuration, daily tasks and maintenance, monitoring and troubleshooting, and completes with development and performance tuning. Part I The first part consists of two chapters and discusses how MySQL NDB Cluster works from a technical point of view. There is not a lot of focus on practical tasks, but some of the theory is exemplified through case studies. The two chapters are: Chapter 1: Architecture and Core Concepts. The first chapter goes into how the cluster works at a high level as well as more specific details for two of the nodes types: management and API nodes. Chapter 2: The Data Nodes. The second chapter exclusively focuses on the data nodes, which are where the data is stored and queries are executed. xxvii

25 Introduction Part II The second part explains how to deploy MySQL NDB Cluster, starting with system considerations, then discusses configuration, installation, and replication. The four chapters are: Part III Chapter 3: System Planning. This chapter discusses what to consider when choosing hardware and how to plan for network partitioning and scalability. There are also several examples of typical topologies. Chapter 4: Configuration. This chapter shows how MySQL NDB Cluster is configured and goes through the most important configuration options. Chapter 5: Installation. It is finally time to install MySQL NDB Cluster. The chapter has installation instructions for Linux, Microsoft Windows, and MacOS. Chapter 6: Replication. Replication allows the data from one cluster to be replicated to another cluster, or an InnoDB database. MySQL NDB Cluster s conflict resolution features for active-active replication between two clusters are also included. The cluster is up and running, so this third part moves on to a series of practical focused chapters describing how day-to-day tasks and maintenance are performed. There are seven chapters in this part: Chapter 7: The NDB Management Client and Other NDB Utilities. This chapter discusses the utilities available in the MySQL NDB Cluster installation. Most of the focus is on the NDB management client, which can be used to perform a range of tasks such as starting, stopping, and restarting nodes as well as creating backups, getting the cluster status, etc. Chapter 8: Backups and Restores. The data is worth no more than its ability to be recovered in the event of a disaster. This chapter discusses the ins and outs of backups and restores. Chapter 9: Table Maintenance. This chapter goes through the online and offline schema change features, defragmentation, and index statistics. Chapter 10: Restarts. In a product that aims at making itself highly available, online restarts are important. This chapter goes through the concepts of rolling restarts and system restarts. There are also several detailed examples of tasks requiring a restart. Chapter 11: Upgrades and Downgrades. This chapter discusses and gives practical examples of how it is possible to perform upgrades and downgrades, both online and offline. Chapter 12: Security Considerations. Security must be a primary part of the design decisions, starting from the initial planning and continuing through the lifetime of the cluster. This chapter goes through the most important security aspects for MySQL NDB Cluster. Chapter 13: MySQL Cluster Manager. This chapter provides a tutorial for the MySQL Enterprise offering MySQL Cluster Manager (MCM), which provides an easier way to manage a cluster. The chapter includes the main steps, from the initial installation through an upgrade. xxviii

26 Part IV Introduction A special part of a database s daily routines is monitoring and troubleshooting. That is the topic of Part IV. There are four chapters in this part: Part V Chapter 14: Monitoring Solutions and the Operating System. This chapter goes through how monitoring solutions, particularly MySQL Enterprise Monitor, can be used to prevent and solve issues, and ends with considerations of what to monitor at the operating system level. Chapter 15: Sources for Monitoring Data. This chapter goes through the traditional sources in MySQL for collecting and monitoring data, including the Information Schema, the Performance Schema, the sys schema, the SHOW statements, and the MySQL error log. Chapter 16: Monitoring MySQL NDB Cluster. MySQL NDB Cluster provides some additional monitoring sources: The ndbinfo schema and the logs on the management and data nodes. Chapter 17: Typical Troubles and Solutions. No matter how careful a database is managed, there will be times when something goes wrong. This chapter goes through some general troubleshooting techniques for MySQL NDB Cluster and some typical issues. The last part moves on to the development part of MySQL NDB Cluster. Developing using SQL and NoSQL and performance tuning are all discussed. The three chapters in Part V are: Chapter 18: Developing Application Using SQL with MySQL NDB Cluster. This chapter goes through table and index creation and other considerations, such as error handling when using SQL statements in the development. Chapter 19: MySQL NDB Cluster as NoSQL Database. MySQL NDB Cluster supports several NoSQL APIs. This chapter discusses NDB-memcached, the C++ NDB API, and ClusterJ. The chapter also includes several code examples. Chapter 20: MySQL NDB Cluster and Application Performance Tuning. The final chapter of the book covers performance tuning at the system and SQL levels. The system level, for example, includes binding data node threads to CPUs and disk types. The SQL level includes optimizing joins and partitioning, mixing NoSQL and SQL, and more. Downloading the Code The code for the examples shown in this book is available on the Apress web site, A link can be found on the book s information page. xxix