Database Sharding with Oracle RDBMS

Size: px

Start display at page:

Download "Database Sharding with Oracle RDBMS"

Reginald Parrish
6 years ago
Views:

1 Database Sharding with Oracle RDBMS First Impressions Robert Bialek Principal Consultant BASEL BERN BRUGG DÜSSELDORF FRANKFURT A.M. FREIBURG I.BR. GENEVA HAMBURG COPENHAGEN LAUSANNE MUNICH STUTTGART VIENNA ZURICH

Backup/Recovery Troubleshooting/Performance Tuning Linux Operating System Trainer for

2 Who Am I Principal Consultant and Trainer at Trivadis GmbH in Munich MSc in Computer Engineering Focus: Oracle Database High Availability Database Architecture/Internals Backup/Recovery Troubleshooting/Performance Tuning Linux Operating System Trainer for the following Trivadis courses: Oracle Grid Infrastructure, RAC, Data Guard

3 Our company. Trivadis is a market leader in IT consulting, system integration, solution engineering and the provision of IT services focusing on and technologies in Switzerland, Germany, Austria and Denmark. We offer our services in the following strategic business fields: O P E R A T I O N Trivadis Services takes over the interacting operation of your IT systems. 3

4 With over 600 specialists and IT experts in your region. COPENHAGEN HAMBURG 14 Trivadis branches and more than 600 employees 200 Service Level Agreements Over 4,000 training participants DÜSSELDORF Research and development budget: CHF 5.0 million FRANKFURT Financially self-supporting and sustainably profitable BASEL FREIBURG STUTTGART BRUGG ZURICH MUNICH VIENNA Experience from more than 1,900 projects per year at over 800 customers GENEVA BERN LAUSANNE 4

5 Technology on its own won't help you. You need to know how to use it properly. 5

6 Database Scalability Limits? For some business requirements: extreme scalability/elasticity, global data distribution, know database techniques might not be sufficient. The solution: Database Sharding (horizontal scaling) Not really a new concept: Cassandra, MongoDB, MySQL, Split one into many One Database Only 1/N components 6

7 Agenda 1. Sharded Database Architecture 2. Data Sharding Methods 3. Data High Availability/Disaster Recovery 4. Session/Statement Routing 5. Conclusions 7

8 Sharded Database Architecture 8

Duplicated tables: data exist in all shards Sharded Database (SDB) Up to 1000 non-cdb shards Distributed

9 Unsharded DB (Tables) SDB Architecture Distributed Partitioning Horizontally distributed data partitioning using shared-nothing database architecture Sharded tables: data exist in one shard Duplicated tables: data exist in all shards Sharded Database (SDB) Up to 1000 non-cdb shards Distributed Partitioning Shard 1 Shard 2 Shard 3 Server B Server C Server D Server A Don t share physical resources No cluster software 9

10 SDB Architecture Big Picture Shard Director 1 Shard Director 2 Shard Director 3 Shard Director 4 Shard Catalog Shards Global Services: OLTP_RW, OLTP_RO, Shard Catalog Shards Data Center (Region): Germany Data Center (Region): Switzerland 10

Table Family SDB Architecture Sharded Tables All tables in one table family must be equi-partitioned (sharding key) Two methods to create a table family: reference partitioning or the PARENT clause

11 Table Family SDB Architecture Sharded Tables All tables in one table family must be equi-partitioned (sharding key) Two methods to create a table family: reference partitioning or the PARENT clause CUSTOMERS CUSTID FIRSTNAME ORDERS ORDERID CUSTID ORDERDATE CREATE SHARDED TABLE customers (... CREATE SHARDED TABLE orders (... ORDERITEMS LINEITEMID CUSTID ORDERID

12 SDB Architecture Duplicated Tables Used to avoid cross-shard operations Use materialized view replication The master table is created in the Shard Catalog database R/O materialized views in all shards All supporting objects created automatically in SDB SHRD_DUPL_TABLE_REFRESH_RATE initialization parameter controls refresh frequency Cannot coexist in a tablespace with sharded tables CREATE DUPLICATED TABLE products( productid INTEGER PRIMARY KEY,... ) TABLESPACE PRODUCTS_TBS; Shard 1 Shard 2 Shard 3 Shard Catalog 12

13 SDB Architecture DDL Execution The application schema name as well as object names are identical in all shards DDLs are executed in the Shard Catalog database or using GDSCTL Automatically propagated to all shards SQL> CONNECT SQL> ALTER SESSION ENABLE SHARD DDL; SQL> CREATE USER <app_name>... SQL> GRANT CREATE TABLE TO <app_name>... SQL> CREATE DUPLICATED TABLE <name>... SQL> CREATE SHARDED TABLE <name>... GDSCTL> sql "CREATE USER..." GDSCTL> sql "CREATE TABLESPACE SET..."... Shard Catalog 13

SDB Architecture Sharded Tables (Chunks) Physical data distribution based on chunks Each table partition is stored in different tablespace Shard 1 Chunks: 1-6 Shard 2 Chunks: 7-12 0 357913941

14 SDB Architecture Sharded Tables (Chunks) Physical data distribution based on chunks Each table partition is stored in different tablespace Shard 1 Chunks: 1-6 Shard 2 Chunks: Chunk #1 Chunk #2 Chunk #7 Chunk #8 CUSTOMERS_P1 ORDERS_P1 CUSTOMERS_P2 ORDERS_P2 CUSTOMERS_P7 ORDERS_P7 CUSTOMERS_P8 ORDERS_P8 TBS_P1 TBS_P2 TBS_P7 TBS_P8 14

SDB Architecture Resharding (Hotspot Elimination) Adding/Removing shards or hotspot elimination requires chunk movement Automatically or manually GDSCTL> MOVE CHUNK -CHUNK 12 -SOURCE sh01 -TARGET

15 SDB Architecture Resharding (Hotspot Elimination) Adding/Removing shards or hotspot elimination requires chunk movement Automatically or manually GDSCTL> MOVE CHUNK -CHUNK 12 -SOURCE sh01 -TARGET sh12 Chunk level 0 RMAN backup Copy files and RMAN restore FAN R/O notification to clients Chunk R/O on source Drop the chunk on the source FAN chunk up/down events File move/ restore Chunk level 1 RMAN backup A chunk can also be spit manually within a shard GDSCTL> SPLIT CHUNK -CHUNK 12 Hotspot Database Sharding with Oracle RDBMS 12c Release 2 - First Impressions

16 Data Sharding Methods 16

17 Data Sharding Methods System-Managed Sharding Data is automatically distributed across shards using partitioning by consistent hash Expression: MOD(ORA_HASH('<SHARDING_KEY'), ) Useful for data distribution Automatic resharding in case of cardinality change The number of chunks is defined during CREATE SHARDCATALOG and cannot be changed later A tablespace set needs to be created in the Shard Catalog database SQL> CREATE TABLESPACE SET TBS_SET USING TEMPLATE (DATAFILE SIZE 1G AUTOEXTEND ON NEXT 512M MAXSIZE UNLIMITED EXTEND MANAGEMENT LOCAL SEGMENT SPACE MANAGEMENT AUTO); 17

18 Tablespace Set Data Sharding Methods System-Managed Sharding Shard Catalog #Chunks=12 #Shards=3 CREATE SHARDED TABLE customers ( custid varchar2(20) CONSTRAINT pk_customers PRIMARY KEY, firstname varchar2(50), lastname varchar2(50),... ) TABLESPACE SET tbs_set PARTITION BY CONSISTENT HASH (custid) PARTITIONS AUTO; Shards C001TS_SET C002TS_SET C003TS_SET C004TS_SET C005TS_SET C006TS_SET C007TS_SET C008TS_SET C009TS_SET C00ATS_SET C00BTS_SET C00CTS_SET customers_p1 customers_p2 customers_p3 customers_p4 customers_p5 customers_p6 customers_p7 customers_p8 customers_p9 customers_p10 customers_p11 customers_p12 18

19 Data Sharding Methods Composite Sharding Data is first partitioned by range or list method across multiple shardspaces (set of shards), then by consistent hash in all shards within a shardspace GDSCTL> ADD SHARDSPACE germany; GDSCTL> ADD SHARDSPACE switzerland; GDSCTL> ADD SHARD CONNECT shdb01 SHARDSPACE germany; GDSCTL> ADD SHARD CONNECT shdb02 SHARDSPACE germany; GDSCTL> ADD SHARD CONNECT shdb02 SHARDSPACE switzerland;... SQL> CREATE TABLESPACE SET de_tbs IN SHARDSPACE germany; SQL> CREATE TABLESPACE SET ch_tbs IN SHARDSPACE switzerland; 19

20 ch_tbs de_tbs Data Sharding Methods Composite Sharding Shards CREATE SHARDED TABLE customers ( custid varchar2(20), country varchar2(2),... ) PARTITIONSET BY LIST (country) PARTITION BY CONSISTENT HASH (custid) PARTITIONS AUTO ( PARTITIONSET germany VALUES ('DE') TABLESPACE SET de_tbs PARTITIONSET switzerland VALUES ('CH') TABLESPACE SET ch_tbs... Shardspace GERMANY Shardspace SWITZERLAND C001DE_TBS C002DE_TBS C003DE_TBS C004DE_TBS C005CH_TBS C006CH_TBS C007CH_TBS C008CH_TBS customers_p1 customers_p2 customers_p3 customers_p4 customers_p5 customers_p6 customers_p7 customers_p8 20

21 Data High Availability Disaster Recovery 21

22 Data High Availability / Disaster Recovery Overview Data replication with Data Guard is a crucial component in SDB environment High availability, disaster recovery, read offloading Replication deployment performed fully automatically The logical unit of data replication is a shardgroup Configuration Data Guard Broker with FSFO Local replica (HA), remote replica (DR) With or without ADG Shard Director 1 Shard Director 2 Observers Primary Standby 22

23 Data Replication For System-Managed Sharding Shard 1 Shard 2 Shard 3 shgrp1 Server B DC1 Shard 4 Shard 5 Shard 6 shgrp2 HA Shard 7 Shard 8 Shard 9 shgrp3 Server B DR DC2 23

24 Data Replication For System-Managed Sharding GDSCTL> create shardcatalog -database shd01:1521:repo -chunks 12 -user mygdsadmin/<pwd> -sdb shdb -region germany,switzerland repl DG sharding system -protectmode maxavailability... GDSCTL> add shardgroup -shardgroup shgrp1 -deploy_as primary -region germany GDSCTL> add shardgroup -shardgroup shgrp2 -deploy_as active_standby -region germany GDSCTL> add shardgroup -shardgroup shgrp3 -deploy_as active_standby -region switzerland HA DR 24

25 Data Replication For System-Managed Sharding GDSCTL> create shard -shardgroup shgrp1 -destination sh01 -credential oracle_cred GDSCTL> create shard -shardgroup shgrp2 -destination sh02 -credential oracle_cred... GDSCTL> create shard -shardgroup shgrp3 -destination sh03 -credential oracle_cred... GDSCTL> deploy 25

26 Data Replication For Composite Sharding Shard 1 Shard 2 shgrp1 Shard 9 Shard 10 Server B shgrp5 DC1 Shard 3 Shard 4 shgrp2 Shard 7 Shard 8 shgrp4 Server B DC2 Shard 5 Shard 6 shgrp3 Server B DC3 Shardspace GERMANY Shardspace SWITZERLAND 26

27 Data Replication For Composite Sharding GDSCTL> create shardcatalog -database shd01:1521:repo -user mygdsadmin/<pwd> -sdb shdb chunks 12 -region germany,switzerland repl DG sharding COMPOSITE... GDSCTL> add shardspace shardspace germany -PROTECTMODE maxavailability GDSCTL> add shardspace shardspace switzerland -PROTECTMODE maxavailability 27

28 Data Replication For Composite Sharding GDSCTL> add shardgroup -shardgroup shgrp1 -shardspace germany -deploy_as primary -region germany GDSCTL> add shardgroup -shardgroup shgrp2 -shardspace switzerland -deploy_as active_standby -region switzerland... GDSCTL> add shardgroup -shardgroup shgrp4 -shardspace switzerland -deploy_as primary -region switzerland GDSCTL> create shard -shardgroup shgrp1 -destination sh01 -credential oracle_cred... GDSCTL> deploy 28

29 Session/Statement Routing 29

30 Session Routing Session routing is done based on a sharding key (super sharding key) SQL operations are related to the specified sharding key (shard) To work on another sharding key value, a new session needs to be created 30 OLTP_RW.TRIVADIS.COM = (DESCRIPTION= (CONNECT_TIMEOUT=5)(RETRY_COUNT=5)(TRANSPORT_CONNECT_TIMEOUT=3) (ADDRESS_LIST= (LOAD_BALANCE=ON) (ADDRESS=(PROTOCOL = TCP)(HOST=gsm01.trivadis.com)(PORT=1530)) (ADDRESS=(PROTOCOL = TCP)(HOST=gsm02.trivadis.com)(PORT=1530))... (CONNECT_DATA= (SERVICE_NAME=oltp_rw.shdb.trivadis) Shard Directors (SHARDING_KEY=robert.bialek@trivadis.com)(SUPER_SHARDING_KEY=germany) (REGION=germany))) Required Only Composite Sharding

31 2 Session Routing (GDS) 3 Shard Director 1 Region: GERMANY 1 Shard Director 2 4 SHARDING_KEY=robert.bialek@trivadis.com 1 2 Connection Pool 3 Shard Director 3 Shard Director 4 Region: SWITZERLAND Shard Catalog Shard Catalog Data Replication 31

32 Statement Routing/Cross-Shard Query Client connection to the Coordinator (Catalog) Database is required No sharding key necessary in the connect descriptor SQL excuted via DB-Link on Shards Partition and Shard Shard Director 1 Region: GERMANY Shard Director 2 Shard Catalog Shard Director 3 Shard Director 4 Region: SWITZERLAND Data Replication 32

If the global service cardinality drops to 0, client is routed to other region GDSCTL> add service -service oltp_ro role

33 Global Services (GDS) Affinity to a Local Region Connection requests are routed to the least-loaded database only in a local region Affinity to a Local Region with Interregion Failover Connection requests are routed only to the least-loaded database in a local region If the global service cardinality drops to 0, client is routed to other region GDSCTL> add service -service oltp_ro role physical_standby -locality LOCAL_ONLY -region_failover lag 20 Region: GERMANY Global Services: OLTP_RW, OLTP_RO, Region: SWITZERLAND 33 Data Replication

34 Conclusions 34

35 Conclusions Use case Custom OLTP applications which require extreme scalability (global data distribution) Existing database applications need to be adopted for database sharding Primarily single shard access With cross-shard operations performance may suffer Existing database frameworks/features have been extended to allow for massive data scalability and fault isolation Special business requirements can now also be addressed with Oracle RDBMS But sharding is complex and requires a lot of knowledge! 35

Trivadis @ DOAG 2016 Booth: 3rd Floor next to the escalator Know how, T-Shirts, Contest

36 DOAG 2016 Booth: 3rd Floor next to the escalator Know how, T-Shirts, Contest and more We look forward to your visit Because with Trivadis you always win!

Backup Methods from Practice

Backup Methods from Practice Optimized and Intelligent Roland Stirnimann @rstirnimann_ch BASEL BERN BRUGG DÜSSELDORF FRANKFURT A.M. FREIBURG I.BR. GENEVA HAMBURG COPENHAGEN LAUSANNE MUNICH STUTTGART VIENNA