A Brief Introduction of TiDB. Dongxu (Edward) Huang CTO, PingCAP

Transcription

1 A Brief Introduction of TiDB Dongxu (Edward) Huang CTO, PingCAP

2 About me Dongxu (Edward) Huang, Cofounder & CTO of PingCAP PingCAP, based in Beijing, China. Infrastructure software engineer, open source hacker Codis / TiDB / TiKV Golang / Python / Rust

3 What would you do when RDBMS is becoming the performance bottleneck of your backend service The amount of data stored in RDBMS is overwhelming You want to do some complex queries on a sharding cluster e.g. simple JOIN or GROUP BY Your application needs ACID transaction on a sharding cluster

4 TiDB Project - Goal SQL is necessary Transparent sharding and data movement 100% OLTP + 80% OLAP Transaction + Complex query Compatible with MySQL, at most cases 24/7 availability, even in case of datacenter outages Thanks to Raft consensus algorithm Open source, of course.

5 Agenda Technical overview of TiDB / TiKV Storage Distributed SQL Tools Real-world cases and benchmarks Demo

6 Architecture Stateless SQL Layer TiDB... TiDB... TiDB Metadata / Timestamp request grpc grpc Placement Driver (PD) TiKV... TiKV... TiKV... TiKV Raft Raft Raft Distributed Storage Layer grpc Control flow: Balance / Failover

7 Storage stack 1/2 TiKV is the underlying storage layer Physically, data is stored in RocksDB We build a Raft layer on top of RocksDB What is Raft? Written in Rust! TiKV API (grpc) Transaction MVCC Raft (grpc) RocksDB Transactional KV API ( /tidb/blob/master/cmd/ben chkv/main.go) Raw KV API ( ap/tidb/blob/master/cmd /benchraw/main.go)

8 Storage stack 2/2 Data is organized by Regions Region: a set of continuous key-value pairs RPC (grpc) Transaction MVCC Raft RocksDB Region 1:[a-e] Region 1:[a-e] Region 2:[f-j] Region 1:[a-e] Region 3:[k-o] Region 2:[f-j] Region 3:[k-o] Region 2:[f-j] Raft group Region 4:[p-t] Region 5:[u-z] Region 3:[k-o] Region 4:[p-t] Region 5:[u-z] Region 4:[p-t] Region 5:[u-z] RocksDB... Instance RocksDB... Instance RocksDB... RocksDB... Instance Instance

9 Dynamic Multi-Raft What s Dynamic Multi-Raft? Dynamic split / merge Safe split / merge split Region 1.1:[a-c] Region 1:[a-e] split Region 1.2:[d-e]

10 Safe Split: 1/4 Raft group Region 1:[a-e] Region 1:[a-e] Region 1:[a-e] TiKV1 raft TiKV2 raft TiKV3 Leader Follower Follower

11 Safe Split: 2/4 Region 1.1:[a-c] Region 1.2:[d-e] raft Region 1:[a-e] TiKV2 raft Region 1:[a-e] TiKV3 TiKV1 Follower Follower Leader

12 Safe Split: 3/4 Split log (replicated by Raft) Region 1.1:[a-c] Region 1.2:[d-e] TiKV1 Split log Region 1:[a-e] TiKV2 Region 1:[a-e] TiKV3 Leader Follower Follower

13 Safe Split: 4/4 Region 1.1:[a-c] raft Region 1.1:[a-c] raft Region 1.1:[a-c] Region 1.2:[d-e] TiKV1 raft Region 1.2:[d-e] TiKV2 raft Region 1.2:[d-e] TiKV3 Leader Follower Follower

14 Scale-out (initial state) Node B Region 1* Region 1 Region 2 Region 1 Region 3 Node D Region 2 Region 2 Region 3 Region 3 Node A Node C

15 Scale-out (add new node) Node B Region 1* Region 1^ Region 2 Region 1 Region 3 Node D Region 2 Region 2 Node A Region 3 Region 3 Node C Node E 1) Transfer leadership of region 1 from Node A to Node B

16 Scale-out (balancing) Node B Region 1 Region 1* Region 2 Region 1 Region 3 Node D Region 2 Region 2 Region 3 Region 3 Node C Node A Node E Region 1 2) Add Replica on Node E

17 Scale-out (balancing) Node B Region 1* Region 2 Region 2 Region 2 Region 1 Region 3 Node D Node A Region 3 Node E Region 1 Region 3 Node C 3) Remove Replica from Node A

18 ACID Transaction Based on Google Percolator Almost decentralized 2-phase commit Timestamp Allocator Optimistic transaction model Default isolation level: Repeatable Read External consistency: Snapshot Isolation + Lock SELECT FOR UPDATE

19 Distributed SQL Full-featured SQL layer Predicate pushdown Distributed join Distributed cost-based optimizer (Distributed CBO)

20 TiDB SQL Layer overview

21 What happens behind a query CREATE TABLE t (c1 INT, c2 TEXT, KEY idx_c1(c1)); SELECT COUNT(c1) FROM t WHERE c1 > 10 AND c2 = percona ;

22 Query Plan Physical Plan on TiDB Final Aggregate SUM(COUNT(c1)) DistSQL Scan COUNT(c1) Physical Plan on TiKV (index scan) Partial Aggregate COUNT(c1) Filter c2 = percona Row Row COUNT(c1) COUNT(c1) COUNT(c1) Read Row Data by RowID RowID TiKV TiKV TiKV Read Index idx1: (10, + )

23 What happens behind a query CREATE TABLE left (id INT, TEXT,KEY idx_id(id)); CREATE TABLE right (id INT, TEXT, KEY idx_id(id)); SELECT * FROM left join right WHERE left.id = right.id;

24 Distributed Join (HashJoin)

25 Supported Distributed Join Type Hash Join Sort merge Join Index-lookup Join

26 No silver bullet (anti-patterns for TiDB SQL) Join between large tables without index or any hints Get distinct values from large tables without index Sort without index Result set is too large (forget LIMIT N?)

27 Best practices Random, massive, read / write workload No hot small table Use transaction, but not much conflicts

28 Tools matter Syncer TiDB-Binlog Mydumper/MyLoader(loader) Open sourced, too.

29 Syncer Synchronize data from MySQL in real-time Hook up as a MySQL replica MySQL (master) Syncer MySQL binlog Syncer Syncer Syncer Rule Filter or TiDB Cluster Fake slave Save Point (disk) TiDB Cluster TiDB Cluster

30 TiDB-Binlog Subscribe the incremental data from TiDB Output Protobuf formatted data or MySQL Binlog format(wip) TiDB Server Pumper Cistern 3rd party applications TiDB Server Pumper Sorter Protobuf MySQL Binlog Another TiDB-Cluster MySQL TiDB Server Pumper

31 MyDumper / Loader Backup/restore in parallel Works for TiDB too Actually, we don t have our own data migration tool for now

32 Use case 1: OLTP + OLAP One of the most popular bike sharing companies in China 7-nodes TiDB cluster for order storage (OLTP). Hook up as MySQL Replica, synchronize data to a 10-nodes TiDB cluster for Ad-hoc OLAP.... Master Master Master Master Master syncer syncer syncer syncer syncer Slave cluster

33 Use case 1: Ad-hoc OLAP TiDB Elapse (3 nodes) MySQL Elapse s 19.93s s 43.23s s 43.33s s >20 mins s 36.81s s 1 min 0.27 sec s 44.05s s 43.18s

34 Use case 2: Distributed OLTP One of the biggest MMORPG game in China. 2.2 T, 18 nodes. Drop-in replacement for MySQL Distributed OLTP

35

36 Sysbench OS linux (ubuntu 14.04) CPU RAM DISK 28 ECUs, 8 vcpus, 2.8 GHz, Intel Xeon E5-2680v2 16 G 80 G (SSD) Notice: 3 replicas

37 Sysbench (Read) table count table size sysbench threads qps latency(avg/. 95) 3 nodes 16 1M rows ms / 19.87ms 6 nodes 16 1M rows ms / 10.96ms 9 nodes 16 1M rows ms / 7.36ms

38 Sysbench (Read)

39 Sysbench (Insert) table count table size sysbench threads TPS latency(avg/. 95) 3 nodes 16 1M rows ms / 78.21ms 6 nodes 16 1M rows ms / 44.61ms 9 nodes 16 1M rows ms / 86.85ms

40 Sysbench (Insert)

41 Roadmap TiSpark: Integrate TiKV with SparkSQL Better optimizer (Statistic && CBO) Json type and document store for TiDB MySQL X-Plugin Integrate with Kubernetes Operator by CoreOS

42 Thanks Contact me: