Summer Student Project Report

Transcription

1 Summer Student Project Report Steffen Pade September 6, 2013 Abstract During my summer project here at CERN I was able to work with a - to me - completely new database technology: Oracle s 12c. My task was to investigate on a new feature called Far Sync. I was asked to run performance tests using Oracle s Real Application Testing framework. The main goal was to find out whether Far Sync is working as promised by Oracle with real workloads from production database systems. After having finished this, I was given the task to extend an existing python tool by a new module. The python tool called DBCheck is supposed to check the setup of database systems for their proper configuration. My module will not only check one system but a group of systems, further called Data Guard system. Including at least two database systems interacting. The tools checks for proper system setup and configuration so the whole group of databases can interact properly as supposed to. 1

2 1 Performance Tests with Oracle s 12c and Far Sync The usual Data Guard setup consists of a primary and a standby database system (each single instance or RAC system). While in production use, the primary system keeps sending redo logs to the standby system constantly on every data modification transaction which is made. The standby system receives the logs and applies them. According to its configuration it can write them to disk and apply them so it s in the exact same state as primary with a delay of a few seconds (or even milliseconds). The purpose of this is mainly to have a backup system that will replace the primary system on failure. The backup / standby system will be in the same state as the primary system and therefore be able to completely replace it. Different Modes of Protection There s three different possible configurations for the so-called protection mode with a data guard setup: Maximum Performance, Maximum Availability and Maximum Protection. The main difference is to be found in how the primary system transports redo logs to the standby database. It s either a synchronous or asynchronous mechanism. Where synchronous means, that the primary database will wait for standby to confirm the reception of redo logs before continuing production work. Accordingly, asynchronous configuration means that primary will immediately continue working after having sent the logs. While it seems quite clear that Maximum Protection always uses synchronous transport and Maximum Performance goes well with asynchronous connections, the Maximum Availability mode is a mix of both. It basically means that the usual connection is synchronous to ensure data safety. Unless it takes too long due to network delays or other problems, then it can, without notification switch to asynchronous transport of redo logs. This way it ensures the maximum availability of the primary system. The Problem Currently at CERN, data guard systems are run using Maximum Performance. Unfortunately, when having a system fail, this could cause data loss since transactions that have not been received by standby will get lost. Inside a local setup in the data center on the CERN site that loss of data is even on a system fail quite likely but will be very small thanks to very fast network connections between the database systems. However, a newly opened data center in Hungary will provide hardware for future migrations of standby systems over there. The expected network 2

3 delay is approximately 25 milliseconds. That sounds quite low but compared to only a fraction of a millisecond in the local setup, it is quite high and makes it more likely to experience a bigger data loss during the process of failover. Named by Oracle this is what happens when a primary system fails and the standby system replaces it in the scope of a data guard configuration. The solution to avoid the loss of data is - obviously - using Maximum Protection configuration. With the delay of 25 milliseconds in between primary and standby systems in the future, this mode would cause a slow down of our primary systems that cannot be accepted. Releasing the newest version 12c Oracle provides us with its Far Sync feature that directly addresses our needs in this very case: A high network delay between primary and standby. Supposed to be the solution, it was my task to investigate its functionality with a production workload. Concept of Far Sync For a better understanding I will make a short draw of the Far Sync concept. As explained earlier the usual setup consists of a primary and a standby system. With Far Sync there will be a third actor in between these two: A Far Sync instance. This system is basically a lightweight database instance only configured for redo forwarding. The primary database will send its redo logs to the Far Sync system in synchronous mode. Far Sync writes them to disc and confirms the reception. Primary can then continue working. After confirmation, Far Sync sends the received logs to the standby system asynchronously. The trick is that the primary and the far sync systems are in a close location to each other with fast network connections. This way it is possible to enable synchronous log transport without having a large performance impact on primary systems. As soon as the logs have reached the far sync system, they are safe and their transport to the standby database can be asynchronous without endangering the data. In case of a failover, primary will be replaced by standby. If there are still logs not applied on standby due to asynchronous transport from far sync, standby and far sync can close that gap and a new primary database can start working fast and without data loss. Figure 1: Usual Data Guard Setup 3

4 Figure 2: Data Guard Setup with Far Sync Testing and Results Using Oracle s Real Application Testing framework one is able to record a sql workload on a production system and replay the exact happening of the production system on a test system. The framework captures statistics on runtime and different other parameters during the replay. These statistics were the basis for my analyses. The process of my tests was basically divided into the following main steps: 1. Import data of production system into test system 2. Replay the captured workload from production 3. Create replay report and collect figures from it 4. Remove imported data and start over The test system was configured as follows: 12c 2-node RAC database as primary system (itrac1238 and -39) 12c 2-node RAC database as standby system (itrac1240 and -41) One Far Sync instance (itrac1216) Both standby machines were setup with an artificial network delay on OS level of 25 milliseconds to simulate the expected delay between CERN in Meyrin and the Hungary data center. I ran the tests with and without the far sync instance in between to compare the runtimes of both setups. Each setup and workload test was run at least three times to gain an average of the needed figures for my analyses. For my tests I was given two workload captures: One from LCGR and one from CMSR production database. Another test run with a simulated workload done by my supervisor Emil Pilecki is part of the analyses as well. As we can see in the figures, the results of all tests show that Far Sync keeps up to its promises. It provides a performance gain of about 60% 4

5 compared to the usual setup with synchronous transport of the logs. The results of the LCGR test runs seem to draw another picture. After taking a deeper look, it turned out that the captured workload was mainly read-only. These loads are not generating any data changes to be sent to standby, so there was basically no difference between Far Sync and no Far Sync due to the lack of redo log generation. Figure 3: Runtimes of Production Workloads Figure 4: Runtime of Simulated Workload Nevertheless, there were different problems with the new feature of Far Sync. It still suffers from many bugs that needed to be worked around. Identifying these bugs and working around them was time-intensive and led to a long time spending on the whole process of testing. 5

6 Thankfully, I am able to say, that the results show a promising feature that needs to be fixed on some parts but works quite well already. 2 Extension of DBCheck for Data Guard DBCheck is a command line tool developed in Python to check existing databases inside the CERN environment for their proper configuration. Where the term proper configuration refers to the correct application of existing conventions. DBCheck can be seen as a framework for modules performing different kinds of checks. While there are modules for general checks on OS level, database level and in LDAP, I have developed a module to check whole setups of data guard configurations. Idea and Concept Since the check of a data guard setup does not simply go down to comparing values of parameters from a system with the given values conform to conventions for the parameters, my module needed to consider a slightly different approach. Inside the programme it is necessary to represent a data guard setup consisting of primary, standby and possibly far sync systems. So the idea evolved to build data structures that apply to these requirements. Using these data structures the module will then be able to run defined test operations on them, returning a text report to the user. The basic idea of the framework is that each module encapsulates data structures and tests and a class Module that has a method text report returning a string containing the whole report of this module. Fulfilling these requirements each module can internally be totally different from the others though. Data Structures The data guard module is based on data structures that represent primary, standby and far sync databases with their corresponding parameters and values needed for the checks. All the classes inherit from a base class encapsulating data and functionality that is common to all systems. Another class, the DGSystem, contains pointers to each database that is part of it representing a data guard setup with primary, standby and (optionally) far sync systems. Check Process The process that is run during the checks can mainly be described as follows: 1. Setting up LDAP connection 6

7 Figure 5: Conceputal: Data Structures of Data Guard Module 2. Gather database entities from LDAP that are standby systems 3. Build data structures (giving them connections to databases and LDAP (resulting in a list of DGSystem objects) 4. Let the objects collect the needed data for given parameters 5. Gain a list of all scheduled checks 6. Iterate over all tests and DGSystem objects, performing each test with each system A consolidated report of all checks with all systems will then be passed to the main module of the framework which in turn creates a file containing this text report. The Checks The performed checks in the data guard module are divided into three groups: LDAP, database and OS related checks. While the OS checks only consist of checking the tnsnames.ora file for completeness, LDAP and database checks monitor the existence and correctness of parameters and their values. Cross checks between the three groups are part of the idea. For example there is check like: Is the DB UNIQUE NAME in the database the same as the LDAP entity name of the corresponding LDAP entity. Further checks include the check of correct pointers between primary, far sync and stanby inside a data guard setup. Status As of now, I could not finish the implementation of all checks. While the first part of my summer student project took longer than expected due to 7

8 so many bugs and problems with the Far Sync setup and testing, the second part of my project is not fully finished. Although it is not complete considering the checks, the data guard module is set up with its data structures and their needed attributes and all functionality as well as one existing check class. This status enables my successor to continue writing check classes to complete all checks needed for the data guard module. 8

9 Acknowledgments The Summer Student Programme is a great idea. I was able to spend an amazing time here at CERN improving and developing professionally as much as personally during these last 13 weeks. First and foremost, I would like to thank Emil Pilecki for being a great supervisor and having lots of patience with answering all my questions and helping me with each and every problem I had during my time here. The same thanks go to Eric Grancher for greatly supervising me during Emil s absence. Further, I want to thank my colleagues from the office, Bartosz Niemczura, Georgios Kaklamanos and my former colleague who has already left CERN Mariusz Piorkowski for being great office mates making the time I spent at work a nice time with nice people! More thanks go to the whole DB group, that provided me with the opportunity of being a Summer Student and a very friendly and open-minded work environment. The last thanks go to the Summer Student Team that did a great job supporting all of us and taking care of our problems. Thank you, Sharon, Laura and Eva! 9