腾讯云 DCDB for TDSQL. Best Practices 产品文档

Transcription

1 腾讯云 DCDB for TDSQL Best Practices 产品文档

2 版权声明 腾讯云版权所有 本文档著作权归腾讯云单独所有, 未经腾讯云事先书面许可, 任何主体不得以任何形式复制 修改 抄袭 传 播全部或部分本文档内容 商标声明 及其它腾讯云服务相关的商标均为腾讯云计算 ( 北京 ) 有限责任公司及其关联公司所有 本文档涉及的第三方 主体的商标, 依法由权利人所有 服务声明 本文档意在向客户介绍腾讯云全部或部分产品 服务的当时的整体概况, 部分产品 服务的内容可能有所调整 您所购买的腾讯云产品 服务的种类 服务标准等应由您与腾讯云之间的商业合同约定, 除非双方另有约定, 否则, 腾讯云对本文档内容不做任何明示或模式的承诺或保证 第 2 页共 25 页

3 文档目录 文档声明... 2 Programming and Usage Specification... 4 Import from Standalone Instance to Distributed Instances Choose DCDB Instance and Sharding Configuration Import Between Distributed Instances 第 3 页共 25 页

4 Programming and Usage Specification Programming and Usage Specification of Distributed Version DCDB refers to DCDB for Percona and MariaDB, unless otherwise specified in this document. 1. Overview 1.1 Introduction to Distributed Solution DCDB for Percona and MariaDB is a distributed database deployed on Tencent Cloud's public cloud. It is compatible with MySQL protocol and syntax and supports automatic horizontal split. With distributed database, a logic database table is acquired as a whole and then at backend, is evenly split into multiple shards and distributed to multiple physical nodes. Currently, DCDB for Percona and MariaDB deploys master/slave architecture by default and provides complete solutions for disaster recovery, backup, recovery, monitoring, migration, and so on. It is suitable for massive database scenarios at TB or PB level. DCDB for Percona and MariaDB supports horizontal split, which is a distributed solution to split the data of a table to multiple databases (hosts) according to the logic of the data. This solution is also called horizontal data sharding. 1.2 Glossary DCDB for Percona and MariaDB: It is a distributed version of DCDB and is named DCDB in short in the follow-up document where does not involve other engines. Vertical split: Tables are classified according to business logic and distributed to different databases to share the pressure of single database. Horizontal split: Instead of being classified, tables are split to different databases in certain rules. Each database is a shard containing only a part of data. 第 4 页共 25 页

5 Sharding rules: Or data split rules. As relational database is a two-dimensional model, two split methods are available: Manually split by a regular field, such as date ( for example, 2015 and 2016 can be considered as a shard respectively), user's ID card number... Hash a specific field and distribute data to different databases according to the specified field, which is called ShardKey Instance: Multiple logically unified shards compose a logic instance. Shard: A physical-logic instance consists of database engines including "a master node (Master), some slave nodes (Slave_n), and some remote slave nodes (Watcher_m)". Node: It is a physical device carrying shards. Node could be a physical machine, a virtual machine or a small cluster. Logic table: Horizontal split is a physical split. While logically, the split database or database table is still a complete database table. proxy: With Tproxy, DCDB achieves automatic databases and tables splitting, manages the underlying physical database instances and provides a unique service port that is compatible with mysql database. shard: It is a physical database instance. A logical instance that is visible to users is composed of multiple physical instances. 2. Usage Specification 2.1 Database Connection DCDB provides a unique IP and its ports can be accessed and used by users, for example: mysql -h P3306 -utest12 -ptestpassword 第 5 页共 25 页

6 2.2 Overview of Incompatibility You should specify the shardkey (a sharding field) when create a table. A sample code is as follows: create table test(a int, b int, ) shardkey=b Cross-node "join" and "transaction" A batch operation like batch insert will fail when data is split into two different shards. It only supports the batch operation when the value of shardkey is always same. View, storage procedure, trigger Self-built partition It does not support the version below MySQL 3.4.0, SSL, and compression protocol. Compatibility: Compatibility Description Creating a Table When creating a table, you must specify the value of shardkey at the end, which is a field name in the table used for SQL routing selection: mysql> create table test1 ( a int, b int, c char(20) ); ERROR 1005 (07000): Proxy Warning - sql is not legal,tokenizer_gram went wrong mysql> create table test1 ( a int, b int, c char(20) ) shardkey=a; Query OK, 0 rows affected (1.56 sec) The field of "shardkey" has the following restrictions: 1. If there is a primary key or a unique index, "shardkey" must be a part of the primary key and all unique indexes; 2. If it contains multiple unique indexes (including primary key), these indexes must have an intersection field that is the "shardkey"; 第 6 页共 25 页

7 3. The type of "shardkey" must be int, bigint, smallint/char/varchar; 4. The value of "shardkey" should not contain Chinese characters. Otherwise, different character sets could be routed to different partitions because the gateway cannot convert character set; 5. Do not update the value of "shardkey"; 6. Put "shardkey=a" at the end of sql Ordinary sql select: It is recommended to contain the "shardkey" field. If the field is not contained, you need to scan the whole table, and then the gateway aggregates the result set. But this will cause a great impact on performance: mysql> select * from test1 where a=2; a b c record2 2 4 record rows in set (0.00 sec) insert/replace: Field "must" contain "shardkey", or the sql will not be executed. Because the proxy cannot confirm which shard should be inserted into: mysql> insert into test1 (b,c) values(4,"record3"); ERROR 1105 (07000): Proxy Warning - sql have no shardkey mysql> insert into test1 (a,c) values(4,"record3"); Query OK, 1 row affected (0.01 sec) delete/update: When we execute this kind of sql, for safety, the sql "must" contain "where" conditions, or it will not be executed: 第 7 页共 25 页

8 mysql> delete from test1; ERROR 1005 (07000): Proxy Warning - sql is not legal,tokenizer_gram went wrong mysql> delete from test1 where a=1; Query OK, 1 row affected (0.01 sec) join join: Currently, DCDB only supports the join operation in a single shard. This means all sqls in a transaction must be based on the same shard, so the shardkey must be specified. mysql> create table test1 ( a int, b int, c char(20) ) shardkey=a; Query OK, 0 rows affected (1.56 sec) mysql> create table test2 ( a int, d int, e char(20) ) shardkey=a; Query OK, 0 rows affected (1.46 sec) mysql> insert into test1 (a,b,c) values(1,2,"record1"),(2,3,"record2"); Query OK, 2 rows affected (0.02 sec) mysql> insert into test2 (a,d,e) values(1,3,"test2_record1"),(2,3,"test2_record2"); Query OK, 2 rows affected (0.02 sec) mysql> select * from test1 join test2 on test1.a=test2.a; ERROR 1105 (07000): Proxy Warning - join shardkey error mysql> select * from test1 join test2 on test1.a=test2.a where test1.a=1; a b c a d e record1 1 3 test2_record row in set (0.00 sec) 第 8 页共 25 页

9 Note: The statement mysql> select * from test1 join test2 on test1.a=test2.a; will be supported in subsequent versions, but with the following preconditions: 1. Must be inner join 2.The "where" condition of inner join must be that the "shardkey" of two tables are equal Transactions DCDB supports transactions in a single shard, which means all sqls in a transaction must be based on the same shard mysql> select * from test1; a b c record2 1 2 record rows in set (0.00 sec) mysql> begin; Query OK, 0 rows affected (0.00 sec) mysql> insert into test1 (a,b,c) values(2,4,"record3"); Query OK, 1 row affected (0.00 sec) mysql> insert into test1 (a,b,c) values(1,4,"record3"); ERROR 1105 (07000): Proxy ERROR - In transaction,this sql use a diffent backend 第 9 页共 25 页

10 mysql> select * from test1; ERROR 1105 (07000): Proxy ERROR - In transaction,this sql use more than one backend mysql> rollback; Query OK, 0 rows affected (0.01 sec) Note: Subsequent versions will support part of simple DML statements Auto-increment Field DCDB supports auto-increment field in a certain sense to ensure that a field is globally unique, but monotonic increment is not ensured. Specific usage is as shown below: Create: create table auto_inc ( a int,b int,c int auto_increment,d int) shardkey=d; Insert: mysql> insert into shard.auto_inc ( a,b,d,c) values(1,2,3,0),(1,2,3,0); Query OK, 2 rows affected (0.01 sec) Records: 2 Duplicates: 0 Warnings: 0 mysql> select * from shard.auto_inc; a b c d rows in set (0.01 sec) 第 10 页共 25 页

11 Note that when the proxy is scheduling the processes of switching, restarting and others, there could be void in auto-increment field, for example: mysql> insert into shard.auto_inc ( a,b,d,c) values(11,12,13,0),(21,22,23,0); Query OK, 2 rows affected (0.03 sec) mysql> select * from shard.auto_inc; a b c d rows in set (0.01 sec) Modify the current value alter table auto auto_increment=100 The current version does not support the auto-increment field as shardkey, but subsequent versions will support SQL Command Restrictions Currently, DCDB supports the sql commands as follows: 1. delele, update, insert, replace, select 第 11 页共 25 页

12 2. alter, create, drop, truncate 3. show, describe (desc, explain), help 4. start, begin, commit, rollback, savepoint 5. set Note: For general sql used to query the information of database status, proxy will sent a default shard. In this case, if you query the statistics information, the result is the information of a single shard Restrictions on User Permission You cannot use sql command to configure user permission through the proxy, please go to Tencent Cloud Console -> Cloud Database -> DCDB -> Management to operate Other Unsupported MySQL features 1. Cross-node join and transaction (Note that join and transaction can be supported when they are not cross-code and use same shardkey in data operation) 2. View, storage procedure, trigger 3. Self-built partition 4. Batch data import: It does not support the sql like "load data local infile", which needs to be converted to "insert" statement, such as "insert into values (xxx,xxx), (xxx,xxx)"; 5. Sub-query, "having" clause Restrictions on Aggregate Functions 1. If you aggregate after "distinct", "where" condition must contain "shardkey": For example: select count(distinct a), sum(distinct a), avg(distinct a) from table where sk=\*\* 第 12 页共 25 页

13 2. If distinct, order by, group by are followed by a function, the function must appear in the "select" field and must be defined by an alias which is used after corresponding "distinct", "order by" and "group by": For example: select concat(...) as substr from table where... order by substr 3. The "group by" field must be contained in the "select" field: For example: select count(a),b from test group by b, where "select" field must contain the "b" field 第 13 页共 25 页

14 Import from Standalone Instance to Distributed Instances 1. Overview of Data Import Distributed architecture of distributed database is transparent to users, so generally, you only need to create table structure first. Then you can migrate data using MySQL client, such as mysqldump, Navicat and SQLyog. Procedures of migration are as follows: Step 1: Prepare import and export environment Step 2: Export the table structure and data of the source table Step 3: Modify the statement for creating table and create table structure in the destination table Step 4: Import data 2. Prepare import and export environment Before migrating data, you need to get the following prepared: CVM 2-core CPU, MEM 8 GB, Disk 500 GB+ (depending on data volume) (recommended) Linux MySQL client installed If the data volume is small (<10 GB), you can also import it directly via public network (Internet) with nothing to be prepared. DCDB: Select expected volume and make initialization according to the source library character set, table name case status and the value of innodb_page_size. Create an account with all global permissions enabled (recommended) Use public IP if necessary 3. Export the table structure and data of the source table 3.1 Demonstration environment Operating database: caccts 第 14 页共 25 页

15 Operating table: t_acct_water_0 Source database: single instance mysql Destination database: DCDB for Percona, MariaDB 3.2 Export table structure in the source database Execute mysqldump -u username -p password -d dbname tablename > tablename.sql to export table structure //Command example mysqldump -utest -ptest1234 -d -S /data/4003/prod/mysql.scok caccts t_acct_water_0 > table.sql 3.2 Export table data in the source database Execute mysqldump -c -u username -p password dbname tablename > tablename.sql to export table structure //Command example mysqldump -c -t -utest -ptest1234 -S /data/4003/prod/mysql.scok caccts t_acct_water_0 > data.sql Note: Data must be exported through mysqldump with parameter -c in order to get data with column name field. Sql without column name field will not be accepted by DCDB for Percona and MariaDB. Parameter -t specifies that only table data will be exported and table structure will not. 第 15 页共 25 页

16 3.3 Upload files to a directory on CVM Before uploading, you need to enable CVM public IP and read the CVM file upload guideline: File Upload from Linux Machine Using SCP. You should upload at least the following exported contents: Table structure sql: table.sql Data sql: data.sql 4. Modify the statement for creating table and create table structure in the destination table Open the exported table structure file table.sql, add primary key and shardkey through statements like the followings, and save as tablenew.sql. Pay attention to shardkey selection techniques and notes: CREATE TAbLE ( column name 1 data type, column name 2 data type, column name 3 data type,..., PRIMARY KEY('column name n')) ENGINE=INNODB DEFAULT CHARSET=xxxx shardkey=keyname Note: Primary key and shardkey must be set. Pay attention to the table name case status. It is recommended to delete redundant notes, or the table may not be created. 第 16 页共 25 页

17 5. Import data 5.1 Connect to DCDB for Percona and MariaDB instance 第 17 页共 25 页

18 On CVM, use mysql -u username -p password -h IP -P port to log in to MySQL server, and use use dbname to enter the database. Note: You may need to create a database first. 5.2 Import table structure Use the uploaded file and import data with source command. 1. Import table structure: source /file path/tablenew.sql 2. Import data: source /file path/data.sql 3. Verify the import result: select count(*) from tablename 第 18 页共 25 页

19 Note: You need to import table creation statement before importing data. You can also import sql directly using the source command in mysql. 6. Others Generally, you can import data smoothly as long as you have created the corresponding table structure with shardkey specified in the destination table before importing data. 第 19 页共 25 页

20 Choose DCDB Instance and Sharding Configuration Overview of DCDB selection DCDB is made up of shards. The specification and number of shards determine the processing abilities of DCDB instances. Theoretically, DCDB instance read and write concurrency performance = (performance of a shard with certain specification * number of the shards) DCDB instance transaction performance = (transaction performance of a shard with certain specification 70% number of the shards) Therefore, high specification and large quantity of shards means strong processing ability to the instance. The sharding performance of a DCBD instance is mainly related to CPU/memory and measured by QPS. Specific performance indicators can be found in the sharding performance part. How to decide on DCDB shard specification Specification of DCDB shard is determined by three factors: performance, capacity and other requirements. Performance: By predicting the performance capacity and possible increments for at least six months, you can define the total CPU/memory size required for your distributed instance. Capacity: By predicting the disk capacity and possible increments for at least one year, you can define the total disk size required for your distributed instance. Other requirements: That one shard stores at least 50 million lines of data is recommended. broadcast table and single table, join and other businesses within the node shall also be considered. Note: It is recommended to ensure high specification for single shard with relatively small quantity of shards. 第 20 页共 25 页

21 Judging from the above, we recommend to you the following policies based on the predicted business models: For functional testing and no special performance requirements: 2 shards, and the specification for each one: memory: 2 GB, disk: 25 GB. In initial stage of business, total size of data is small but grows fast: 2 shards, and the specification for each one: memory: 16 GB, disk: 200 GB. In stable development stage, sharding is based on actual business conditions: 4 shards, and the specification for each one: current business peak * growth rate / 4. Test of DCDB Sharding Performance Database benchmark performance test is for modifying descriptions of sysbench 0.5: otlp script in sysbench is modified; read/write ratio is set to 1:1 and controlled by executing test command parameters "oltp_point_selects" and "oltp_index_updates". All test cases in this document do 4 Select operations and 1 Update operation, and keep read/write ratio to 4:1. vcpu (Core) Memory Storage Data set Number of Number of QPS TPS (GB) Space (GB) (GB) clients concurrence in single client 1 2 GB 100 GB 46 GB GB 200 GB 76 GB GB 200 GB 142 GB GB 400 GB 238 GB GB 700 GB 238 GB GB 1 T 378 GB GB 1.5 T 378 GB GB 2 T 378 GB GB 3 T 567 GB GB 6 T 567 GB TPS is a single TPS, but not a distributed transaction TPS. 第 21 页共 25 页

22 According to the requirements on operation policies, current DCDB instances (or part of them) adopt the policy of excess usage of idle resources, so CPU utilization may exceed 100% on your monitoring screen. 第 22 页共 25 页

23 Import Between Distributed Instances Since the solution for data importing from one distributed database to another is special, we take mysqldump as an example to summarize the importing steps as follows: Step 1: Install mysqldump (MariaDB version) Purchase linux CVM and use yum install mariadb-server to install mysqldump. Step 2: Export the table structure of the source database mysqldump --no-tablespaces --default-character-set=utf8 --hex-blob --skip-add-drop-table -c --skip-lock-tables --skip-opt --skip-triggers --skip-comments --skip-tz-utc --skip-extended-insert --skip-disable-keys --skip-quote-names --no-data -uusername -ppassword -hxxx.xxx.xxx.xxx -Pxxxxx dbname grep -v '^\/\*\!40' > schema.sql; Note: --default-character-set=utf8 will be set based on your source table. -uusername is an account with permissions (-u: keyword). -ppassword is a password (-p: keyword). -hxxx.xxx.xxx.xxx -Pxxxxx is the IP and port of the database instance. dbname is the database name. Filter /!40.../ statements out with grep, because they are not supported in DCDB. Step 2: Export the table data of the source database 第 23 页共 25 页

24 mysqldump --no-tablespaces --default-character-set=utf8 --hex-blob --skip-add-drop-table -c --skip-lock-tables --skip-opt --skip-triggers --skip-comments --skip-tz-utc --skip-extended-insert --skip-disable-keys --skip-quote-names --no-create-info -uandelwu -pandelwu -hxxx.xxx.xxx.xxx -Pxxxxx dbname grep -v '^\/\*\!40' > data.sql; Note: --default-character-set=utf8 will be set based on your source table. -uusername is an account with permissions (-u: keyword). -ppassword is a password (-p: keyword). -hxxx.xxx.xxx.xxx -Pxxxxx is the IP and port of the database instance. dbname is the database name. Filter /!40.../ statements out with grep, because they are not supported in DCDB. Step 3: Create a database in the destination database mysql --default-character-set=utf8 -uusername -ppassword -hxxx.xxx.xxx.xxx -Pxxxxx -e "create database dbname;"; Note: --default-character-set=utf8 will be set based on your destination table. -uusername is an account with permissions (-u: keyword). -ppassword is a password (-p: keyword). -hxxx.xxx.xxx.xxx -Pxxxxx is the IP and port of the database instance. dbname is the database name. Step 4: Import table structure in the destination database+ 第 24 页共 25 页

25 Powered by TCPDF ( Best Practices 产品文档 mysql --default-character-set=utf8 -uusername -ppassword -hxxx.xxx.xxx.xxx -Pxxxxx dbname < schema.sql Note: --default-character-set=utf8 will be set based on your destination table. -uusername is an account with permissions (-u: keyword). -ppassword is a password (-p: keyword). -hxxx.xxx.xxx.xxx -Pxxxxx is the IP and port of the database instance. dbname is the database name. Step 5: Import table data in the destination database mysql --default-character-set=utf8 -uusername -ppassword -hxxx.xxx.xxx.xxx -Pxxxxx dbname < data.sql Note: If auto-increment field is used in the source table and error "Column 'xx' specified twice" occurred while importing data, you need to delete the backquotes in the autoincrement field (cat schema.sql tr "`" " " > schema_tr.sql) of schema.sql, drop database, and then repeat steps 3 to 5 using the modified schema_tr.sql. 第 25 页共 25 页