How to Minimize Brownouts in Oracle RAC Reconfiguration Session ID# UGF9795. Amit K Das PayPal

Size: px

Start display at page:

Download "How to Minimize Brownouts in Oracle RAC Reconfiguration Session ID# UGF9795. Amit K Das PayPal"

Jane Patterson
5 years ago
Views:

1 How to Minimize Brownouts in Oracle RAC Reconfiguration Session ID# UGF9795 Amit K Das PayPal

2 Introduction: about our team Sehmuz Bayhan Our visionary director. Executed great changes in lightning speed. Saibabu Devabhaktuni Our fearless leader at PayPal for at least 10 years. Kyle Towle Our fearless database architect at Paypal for at least 8 years. Dong Wang Goldengate expert, speaker at multiple conferences, PayPal DBA for going on 8 years. John Kanagaraj Author, Oracle ACE, frequent speaker at Oracle conferences Sarah Brydon One of the very few Oracle Certified Masters. Samrat Roy One of our Expert DBA.

3 Who Am I? 11 years in Oracle RAC Development team. Technical lead for world s first Exadata production golive (Apple), while at Oracle. Currently Engineering lead/architect for World largest Exadata OLTP system (PayPal). Frequent presenter in Oracle Open World, IOUG, NoCOUG. Love fishing.

4 Largest OLTP database Fast paced VLDB OLTP environment on Oracle 500+ database instances OLTP databases commonly TB 5,000-14,000 concurrent processes; 120,000 executions/second, 25MB Redo/Sec Continuously growing Database size double every 18 months Workload double every 3 years Stringent uptime requirements We are a financial company that operates at Internet scale Need high performance, scalability, and consistency We rely on MAA for our high availability architecture 200+ Active Data Guard deployed 50+ RAC databases; 50+ GoldenGate instances

5 Agenda 1. Understand Oracle RAC and its advantages 2. Understand Oracle RAC reconfiguration 3. Measure the reconfiguration time 4. Oracle RAC reconfiguration vs. business SLA 5. Understand the workload and behavior 6. Minimizing the impact of Oracle RAC reconfiguration

6 Understand Oracle RAC and its Advantages Why Oracle RAC? 1. Without it, we cannot scale our fast growing business. 2. Without it, we will not have high availability. 3. The largest single machine cannot be a replacement for Oracle RAC. 4. Without it, we will not able to do rolling patching. 5. Without it, we cannot build a real cloud. 6. And More

7 Understand RAC Reconfiguration Every Oracle Block in RAC is protected by a lock element, when it is accessed by an instance. Total number of master locks will be equally distributed between all instances. Other non mastered instance will own a copy/shadow lock if he does not own the master lock but interest with that block. Open Block count can measure from v$bh Open lock count can measure from X$ view

8 Understand RAC Reconfiguration Why do we need Reconfiguration? To protect cache consistency when a node joins or leaves the cluster. When it can trigger? If an instance joins or leaves the cluster Start or Stop recovery of ADG Partial reconfiguration triggers: o o o In certain situations when DRM is ON Flushing of Buffer Cache on any node in the cluster. When redirecting locks

9 Understand RAC Reconfiguration What is the effect? While running the reconfiguration, applications will have transient delay to access some database objects, depending on access pattern. Duration of reconfiguration depends on Size of buffer cache Heat map of objects in cache Number of active sessions

10 Understand RAC Reconfiguration Note: We have huge OLTP load and very large buffer cache, we observe things depending our workload and business needs.

11 Measure the reconfiguration time When node leave or join, dump the ASH data to a offline table Create table ash_ as select * from v$active_session_history; Generate report from ash_ with a SQL like below col MAXTIME format a15 col SAMPLE_TIME format a26 col BLOCKING_SESSION format col event format a25 col AVGTIME format a15 select a.sample_time, a.blocking_session,b.event,a.blocking_session_status,a.blocking_hangchain_info, a.event, count(*), round(max(a.time_waited)/1000,2) ' ms' maxtime,round(avg(a.time_waited)/1000,2) ' ms' avgtime from ash_ a, ash_ b where a.sample_time=b.sample_time and b.session_id=a.blocking_session group by a.sample_time,a.blocking_session,a.blocking_session_status,b.event, a.blocking_hangchain_info,a.event having count(*)>5 order by a.sample_time ;

12 Measure the reconfiguration time Analyze the report of last SQL SAMPLE_TIME BLK_SESSION Blocker_EVENT BLK_Stat Waiter_EVENT COUNT(*) MAXTIME AVGTIME JUL PM 3521 ges enter server mode VALID log file sync ms ms 09-JUL PM 7419 latch: enqueue hash chains VALID latch free ms ms 09-JUL PM 9162 gcs resource directory to be unfrozen VALID gc buffer busy acquire ms 0 ms 09-JUL PM 4442 gcs resource directory to be unfrozen VALID gc buffer busy acquire ms 0 ms 09-JUL PM gcs resource directory to be unfrozen VALID gc buffer busy acquire ms 0 ms JUL PM gcs resource directory to be unfrozen VALID gc buffer busy acquire ms ms 09-JUL PM 3433 latch: gcs resource hash VALID latch: gc element ms ms 09-JUL PM 3521 VALID log file sync ms ms 09-JUL PM 3697 VALID gc domain validation ms ms 09-JUL PM db file sequential read VALID read by other session ms ms 09-JUL PM 3521 log file parallel write VALID log file sync ms ms 09-JUL PM 3521 log file parallel write VALID log file sync 6.57 ms.45 ms Total Time of RAC reconfiguration time is :(09-JUL PM - 09-JUL PM )=15sec+

13 Oracle RAC reconfiguration vs. business SLA If your application is a DSS? most of the time reconfiguration will be invisible for the business. If your application is a large OLTP? define your SLA and check whether reconfiguration can meet your expectation

14 Understand the workload and behavior Discover the data access pattern. Is the application partitioned by node? Is the workload random read/write accessing from all nodes? Is the workload read mostly accessing from all nodes and write only from one node? Is the application configured as active/passive?

15 Understand the workload and behavior Measure the object distribution across all nodes. Maintain high uptime: avoid stopping or starting any instance frequently (i.e. avoid reconfiguration)

16 How to minimize the impact of the reconfiguration in Oracle RAC. 1. Proper sizing of buffer_cache. 2. Optimal number of LMS processes 3. Move application traffic and wait as long as possible to cool down the buffer cache from the node which is to be brought down. 4. Increase the bandwidth and low latency for private interconnect.

17 Proper sizing of Buffer Cache Larger Buffer cache => more objects => more time required for RAC reconfiguration If your application meets its I/O SLA with smaller Buffer Cache, keep it as small as possible Prior to increasing buffer cache size, consider effect on RAC reconfiguration Note: RAC will require a larger buffer cache than a single node to serve same amount of traffic for failover situation

18 Increasing LMS processes optimally Do not simply increase the number of LMS processes Too many LMS processes cause overhead for synchronization Need to limit the # of LMS processes when you have multiple DBs on same cluster Check: The usage of interconnect CPU usage increase the number of LMS processes only if you have enough bandwidth on the Interconnect and idle CPU cycles Improve LMS performance further by applying Fix : LONG LATCH WAIT ON GCS RESOURCE VALIDATE LIST ONLARGE NUMBER OF LMSES

19 Planning for reconfiguration Move your traffic to other nodes for planned shutdown Allow more time for auto remastering Fix for bug will help faster reconfiguration (40-50% improvement) (needs latest JAN PSU with 11203) Avoid unwanted double reconfiguration with fix SQL>Alter system set db_cache_size=256m scope=memory sid=<sid_name_which_is_going_down>;

20 From our experience Need good amount of patient to execute special tool perfectly. Make sure one node can handle the entire traffic. Transfer all connections to one node. Aggressive lock conversion can cause more session spike and make things worse. Monitor the lock conversion.

21 2 Sharing the result from active node Active Node DLM Locks Before redirecting locks 18,143,660 3 Hours after 1,173,860 5 Hours after 638,420 * Note: This is from a test system Your mileage may vary!

22 2 Sharing the result from passive node Passive Node DLM Locks Before redirecting locks 12,340,238 3 Hours after 2,001,760 5 Hours after 435,628

23 Determining the best time to shutdown passive instance Shutdown the passive instance when you see Small number of DLM locks on passive node Shrink the buffer_cache of passive node Shutdown passive instance with immediate option will cause close to ZERO impact.

24 Lessons learned Working with Oracle for further improvement WIP Do not flush buffer cache on passive node to close the open DLM locks But shrink buffer cache before shutdown Optimization of LMS process count will improve reconfiguration. The fix of bug will help for faster reconfiguration (40-50% improvement). Avoid unwanted double reconfiguration with fix Instance startup from mount to open will be impacted (2-10 sec) with CF Enqueue contention, received the fix.- Bug : DELAY IN ARC0 STARTING UP Disabled DRM using the parameter _gc_policy_time=0

25 My Next Presentation Session ID: CON2468 Session Title: Tech System Refresh: Zero Downtime with Oracle RAC/Oracle Automatic Storage Management Venue / Room: Moscone South Date and Time: 9/26/13, 15:30-16:30

26 COLLABORATE14.IOUG.ORG April 7 11, 2014 The Venetian Resort, Las Vegas, NV Network with 5,000+ database and application professionals 5 days of in-depth education built by users, for users Complimentary Pre-Conference Workshops for IOUG registrants Attend for free! IOUG s Call for Speakers is now open collaborate14.ioug.org/call-for-speakers

November 8: Atlanta, GA IOUG Master Class Series Returns Coming soon to a city near you!

27 Training Close to Home Maximum Availability Architecture with Oracle s Larry Carpenter and Joe Communities Meeks October 15: Milwaukee, WI October 16: Rochester, NY Performance and Internals with Craig Shallahamer November 6: Chicago, IL November 8: Atlanta, GA IOUG Master Class Series Returns Coming soon to a city near you! One Day Interactive Training Events to meet your educational needs Get more information at or visit the IOUG kiosk in Moscone West, 2 nd floor

28 Come See IOUG at the User Group Pavilion Win a Free Registration to COLLABORATE 14! Stop by the IOUG kiosk in the User Group Pavilion in Moscone West, 2nd floor, to pick up a lucky poker chip and see if you are a winner of a free registration to COLLABORATE 14 in Las Vegas!

30 THANK YOU Amit K Das (amidas@paypal.com)

Internals of Active Dataguard. Saibabu Devabhaktuni

Internals of Active Dataguard. Saibabu Devabhaktuni Internals of Active Dataguard Saibabu Devabhaktuni PayPal DB Engineering team Sehmuz Bayhan Our visionary director Saibabu Devabhaktuni Sr manager of DB engineering team http://sai-oracle.blogspot.com