POLARDB for MyRocks Extending shared storage to MyRocks. Zhang, Yuan Alibaba Cloud Apr, 2018

Transcription

1 POLARDB for MyRocks Extending shared storage to MyRocks Zhang, Yuan Alibaba Cloud Apr, 2018

2 About me Yuan Zhang database engineer Work at Ailbaba for 5 years Focus on MySQL & MyRocks zhangyuan.zy@alibaba-inc.com

3 Agenda Background Basic Architecture Implementation details Performance Improment Future plan

4 Background Why POLARDB for MyRocks Benifits from MyRocks MyRocks + Polarstore Greate space efficiency, better compression Greate write efficiency, lower write amplification Fast data loading Compatiable with MySQL Benifits from share-storage(polarstore) Promising data consistency Ability to scale read node immediately without full copy of data

5 Basic Architecture Primary Accept Read/Write workload Replica Only Accept Read workload Share sst/wal with primary

6 Let s Begin prepare for rocksdb wal replication Base on AIiSQL5.7 Port MyRocks from Facebook Only support RocksDB and MyISAM engine Convert system tables to RocksDB

7 Convert system tables to RocksDB Prepare for RocksDB WAL replication Convert system tables to RocksDB Except mysql.slow_log, mysql.general_log, they store in local disk, primary and replica have their owen mysql.slow_log, mysql.general_log tables.

8 Rocksdb WAL/Manifest replication Architecture

9 Rocksdb WAL/Manifest replication Asynchronous replication WAL Replication Replay PUT/DELETE/MERGE Manifest Replicaion Replay flush & compaction WAL and Manifest Coordination Only apply VEdit while Applied lsn > VEdit lsn

10 Rocksdb WAL/Manifest replication Control Primary WAL and SST files deletion WAL deletion - original wal deletion will lead Replica lost wal Lm: min_log_number on Primary Ln: min_log_number on all Replicas new_min_log_number= min(lm,ln) When WAL s number < new_min_log_number, then this WAL can be deleted SST deletion- original SST deleteion will lead Replica cannot find SST and crash min_version_number: the minimal version number replica is using SST can be deleted only when It will t be used by Primary and all Replicas

11 DDL&Cache replication Architecture

12 DDL Replication Remove frm,par files Frm,par files Table metadata information If Master and replica share frm,par files, DDL replication must be synchronous Remove frm,par files Store these contents in RocksDB Replica can read multi version of table schema DDL replication is asynchronous

13 DDL Replication Remove frm,par files DDL replication is asynchronous Multiple Table schema version in rocksdb Row data also have different verisions

14 DDL Replication We have MDL lock to protect DDL operation in Primary. This lock also need in Replica s DDL. Primary Log MDL lock start and end. Replica Replay MDL lock start A. lock MDL Replay MDL lock end A. update table cache in myrocks B. unlock MDL

15 Cache Replication ACL, Procedure, Query cache Replicaition Primary Log cache change in RocksDB WAL ACL, Procedure Replica Replay this change from WAL and invaild this cache

16 Index Statistics Replication Persistent Part index statistics information persist in each SST Total index statistics store in INDEX_STATISTICS Memory Rdb_dey_def::m_stats Update Analyze table Flush memtable Compact Replica listen PUT operation in INDEX_STATISTICS and reload statistic info to memory.

17 New Log Format log change for replication Log Types DDL(START, END) Cache change, ACL/Proc Log format PUT/DELETE Log store location system column family

18 New Log Format New type in data dictionary // Data dictionary types enum DATA_DICT_TYPE { DDL_ENTRY_INDEX_START_NUMBER = 1, INDEX_INFO = 2, CF_DEFINITION = 3, BINLOG_INFO_INDEX_NUMBER = 4, DDL_DROP_INDEX_ONGOING = 5, INDEX_STATISTICS = 6, MAX_INDEX_ID = 7, DDL_CREATE_INDEX_ONGOING = 8, POLAR_LOG = 100, // for polar replication END_DICT_INDEX_ID = 255 }; enum POLAR_LOG_TYPE { TABLE_DDL = 1, CACHE_CHANGE = 2, END_POLAR_ROCK_TYPE = 255 };

19 New Log Format New type in data dictionary DDL_START type: PUT key: POLAR_LOG+TABLE_DDL+dbname.tablename value: NULL DDL_END type: DELETE key: POLAR_LOG+TABLE_DDL+dbname.tablename value: NULL CACHE_CHANGE type: PUT key: POLAR_LOG+CACHE_CHANGE+ACL/Proc value: NULL

20 New Log Format Problems DDL_START and DDL_END must be a pair. Problem 1: Primary Crash Primary crash after DDL_START, Primary will resent DDL_START when restart, and the previous DDL_END will lost. Replica replay DDL_START and hold MDL lock, It will not unlock with DDL_END DDL_START type: PUT key: POLAR_LOG+TABLE_DDL+dbname.tablename value: NULL DDL_END type: DELETE key: POLAR_LOG+TABLE_DDL+dbname.tablename value: NULL

21 New Log Format Problems DDL_START and DDL_END must be a pair. Problem 1: Primary Crash Primary crash after DDL_START, Primary will resent DDL_START when restart, and the previous DDL_END will lost. Replica replay DDL_START and hold MDL lock, It will not unlock with DDL_END Solution Primary Scan RocksDB to find record TABLE_DDL when restart, if found, Primary should resent DDL_END, and Replica will unlock the old lock

22 New Log Format Problems DDL_START and DDL_END must be a pair. Problem 2: Replica Crash Replica carsh after DDL_START, Replica will continue to replay DDL_END when restart But the lock with DDL_START will not exist after restart, Replica replay DDL_END to unlock a MDL lock which is not exist

23 New Log Format Problems DDL_START and DDL_END must be a pair. Problem 2: Replica Crash Replica carsh after DDL_START, Replica will continue to replay DDL_END when restart But the lock with DDL_START will not exist after restart, Replica replay DDL_END to unlock a MDL lock which is not exist Solution Replica Scan RocksDB to find record TABLE_DDL when restart, if found, Replica should replay DDL_START to lock

24 MVCC MVCC based on RocksDB snapshot Keep a consistent snapshot in Replica Replica can t get the record after Primary compact Control compact in Primary Compact in Primary should consider about Replica s snapshot Only delete record when sequnce >=Sn, Sn is the laste seqence in Replica Primary s snapshot list merge with replica s snapshot list.

25 MVCC MVCC based on RocksDB snapshot Keep a consistent snapshot in Replica

26 Performance Improment Optimize write performance Async-commit Optimize auto_increment

27 Performance Improment Async-commit Original pipeline write

28 Performance Improment Async-commit Async-commit

29 Performance Improment Optimize write performance Optimize auto_increment write need check unique Do Get first then write Get is expensive Actually, most auto_increment check uniqueness is not necessary. Espacially, when all the auto_incment column is automatically generated.

30 Performance Improment Optimize write performance Optimize auto_increment max_specify_pk: user sepcified max auto_increment value if pk > max_specify_pk, skip unique check if pk <= max_specify_pk nead unique check max_specify_pk update when user use sepcified auto_increment value

31 Future Feature Online DDL Multiple-Master Performance Compaction optimize

32 Q&A

33