Taming Banner 7 on Oracle 10g

Transcription

1 SUNGARD SUMMIT 2007 sungardsummit.com 1 Taming Banner 7 on Oracle 10g Presented by: Scott Harden University of Illinois March 20, 2007 Course ID: 58 A Community of Learning

2 Session Rules of Etiquette Please turn off your cell phone/beeper If you must leave the session early, please do so as discretely as possible Please avoid side conversation during the presentation Thank you for your cooperation! 2

3 Introduction Purpose: To share performance lessons learned after upgrading to Banner 7 and Oracle 10g leading to our best performance ever! Benefits: Learn Top 5 things that helped our performance Avoid the same problems we hit Tools to assist you in your tuning efforts Tuning tips particularly for large school volume 3

4 Topics of Discussion How large are we? Results of our tuning efforts Top 5 solutions Problems we hit Tools to help you tame the beast 4

5 SUNGARD SUMMIT 2007 sungardsummit.com 5 How Large Are We? A Community of Learning

6 How Large Are We? 70,000 students 40,000 employees 3 campuses Nearly 10,000 potential forms users Max. connections: 2635, Median: 1500 Max. self service connections: 1147 SUN F15K - 44 CPUs, 88 GB memory Database size: 330 GB Growth: ~250 MB/day FGBTRND over 140 million rows NHRDIST over 72 million rows SORLCUR over 4 million rows Live for roughly 3.5 years now 6

7 SUNGARD SUMMIT 2007 sungardsummit.com 7 Results of Our Tuning Efforts A Community of Learning

8 Results of Our Tuning Efforts 45% reduction in CPU load 62% reduction in logical reads/sec 50% increase in max transactions/sec 8

9 Results of Our Tuning Efforts (Details) Before Avg. Before Max. After Avg. After Max. Host CPU Util.* 27% 52% 15% 40% Logical read/sec 236,853 1,080,098 89, ,734 Logical reads/txn , ,350 User txns/sec* *numbers since adding 2 additional system boards (8 CPUs, 16 GB memory) 9

10 SUNGARD SUMMIT 2007 sungardsummit.com 10 Top 5 Solutions A Community of Learning

11 Top 5 Solutions #1 Registration Class Search SunGard Corrected Defect 1-BQAXW Cursor was searching across multiple terms resulting in excessive load (60% in perf. testing) Added cursor to search only one term when specified Fixed in Banner

12 Top 5 Solutions #2 New Banner 7.3 APIs SunGard Corrected Defect 1-G7KLZ Banner 7.2 APIs were using literals rather than bind variables causing shared pool fragmentation and high latch waits Implemented Student 7.3 APIs that were backwards compatible with 7.2 Partially fixed in Banner

13 Top 5 Solutions #3 Upgraded from Oracle to Oracle fixed many Cost-Based Optimizer Bugs Many poor execution plans were observed under causing us to use SQL profiles and RULE hints to overcome poor plans Plans improved considerably under , SQL Profiles removed 13

14 Top 5 Solutions #4 Turned off Oracle 10g Automatic Memory Mgt During heavy load shared pool was resizing itself several thousand times a minute creating massive library latch waits Oracle Bugs / (unpublished) Set SGA_TARGET= 0, SHARED_POOL_SIZE = , DB_CACHE_SIZE =

15 Top 5 Solutions #5 Turned off Oracle 10g Automatic Statistics Default stats gathering caused plans to change and random performance degradation from one day to the next Default stats run nightly, use SIZE=AUTO and generate histograms indiscriminately Implemented script to calc stats without histograms, SIZE=1, added histograms as needed 15

16 SUNGARD SUMMIT 2007 sungardsummit.com 16 Problems We Hit A Community of Learning

17 Problems We Hit Week 1 Problem: Query from Document Imaging against ALL_SYNONYMS responsible for 20% of database load Detected by: ADDM and Top Activity from Oracle 10g Enterprise Manager Grid Control Short term solution: Implement SQL Profile using Oracle 10g SQL Tuning Advisor Long term solution: Replaced 10g ALL_SYNONYMS view with 9i view. See Metalink Note:

18 Problems We Hit Week 1 Problem: SAAADMS form slow, issuing 19 million calls per hour to get effective term code from SGBSTDN and 14 million calls to SORLCUR, responsible for 28% of load Detected by: Reports of slow response, ADDM, AWR and Top Activity in Oracle EM Short Term Solution: Paper patch for defect CMS- DFCT Long Term Solution: Defect is fixed in Student

19 Other Problems We Hit Problem: Performance issues on NBQPOSN, TGRAGES, TSRCBIL, FAIVNDH, FAIINVL, FGIBDSR, PERJOBS, RORRULE, GLBDATA, RPEDISB due to bad plans/stats Short term solutions: Restore stats, RULE hint, SQL Profiles Long term solution: Implemented our own statistics gathering with histograms only where required 19

20 Other Problems We Hit Problem: Poor performance in FZRACTG, SFRFASC, and Registration Solutions: Added index on FGBTRND on FGBTRND_DEFER_GRNT_IND; Added index on SFRRGFE based upon our usage of columns; dropped index SSBSECT_GSCH_INDEX 20

21 Other Problems We Hit Problem: Poor performance in FGITRND, FZIGITD, FZAGRNT, FGIGLAC due to parallel queries against partitioned FGBTRND; lots of PX waits skewing monitors Solution: Turned off parallel query by setting PARALLEL_MAX_SERVERS = 0 21

22 Other Problems We Hit Problem: Poor performance in Web for Admissions on two queries against SARHEAD due to conditional NULL logic in SQL statements Solution: Rewrote portions of 10 packages locally, SunGard issued Defect 1-17GNAT SELECT SARHEAD_APPL_SEQNO, STVWAPP_CODE, STVWAPP_DESC, STVTERM_DESC, SARHEAD_ADD_DATE, SARHEAD_APPL_STATUS_IND, SARHEAD_APPL_COMP_IND, SARHEAD_WSCT_CODE_BOOKMARK, SARHEAD_APPL_PREFERENCE FROM SARHEAD, STVTERM, STVWAPP WHERE ( ( :B2 IS NOT NULL AND EXISTS (SELECT 1 FROM SABIDEN WHERE SABIDEN_PIDM = :B2 AND SARHEAD_AIDM = SABIDEN_AIDM) ) OR ( :B2 IS NULL AND SARHEAD_AIDM = :B1 ) ) Allowed us to process over 13,000 admission decision queries in 30 minutes on Dec. 15, 2006 with almost no impact on the system!! 22

23 Other Problems We Hit Problem: Receiving ORA-6550 and PLS-00306: wrong number or types of arguments affecting Web For users due to Oracle Bug Short Term Solution: Used workaround of recompiling packages Long Term Solution: Implemented the Oracle patch for the bug 23

24 SUNGARD SUMMIT 2007 sungardsummit.com 24 Tools to Help You Tame the Beast A Community of Learning

25 Tools to Help You Tame the Beast #1 Must Have! Oracle 10g Enterprise Manager Grid Control w/diagnostic & Tuning Packs Make sure you have this up and running as soon as you put up Oracle 10g! 25

26 Tools to Help You Tame the Beast Sample Top Activity graph part of Oracle 10g Diagnostic Pack 26

27 Tools to Help You Tame the Beast Advanced Database Diagnostic Monitor (ADDM) Report part of Oracle 10g Diagnostic Pack FINDING 1: 46% impact (30464 seconds) SQL statements consuming significant database time were found. RECOMMENDATION 1: SQL Tuning, 21% benefit (14283 seconds) ACTION: Run SQL Tuning Advisor on the SQL statement with SQL_ID "9zyggvkjs7pdt". RELEVANT OBJECT: SQL statement with SQL_ID 9zyggvkjs7pdt and PLAN_HASH select table_owner, table_name from all_synonyms where owner = 'OTGMGR' and synonym_name = 'AE_USRPRO' order by decode(owner, 'PUBLIC', 2, 1) RATIONALE: SQL statement with SQL_ID "9zyggvkjs7pdt" was executed 8268 times and had an average elapsed time of 1.7 seconds. 27

28 Tools to Help You Tame the Beast Automatic Workload Repository (AWR) part of Oracle 10g Diagnostic Pack 28

29 Tools to Help You Tame the Beast Active Session History (ASH) Report Run reports by: Date/time range only SID SQL ID Wait Class Service Module Action Client 29

30 Tools to Help You Tame the Beast RULE Hint Majority of Banner still runs well with the RULE hint Use only as a last resort! Investigate statistics, particularly histograms, before implementing RULE hint 30

31 Tools to Help You Tame the Beast SunGard Oracle 10g FAQ 1-S35GU The FAQ of all other Oracle 10g related FAQs Start here when configuring your first 10g environment 31

32 SUNGARD SUMMIT 2007 sungardsummit.com 32 Summary A Community of Learning

33 Summary NO FEAR! - Banner 7 and Oracle 10g can perform great together! Scrutinize Oracle s automatic memory management and statistics gathering Use Oracle s 10g tools to monitor performance and drill into problems Follow SunGard s FAQs for Oracle 10g 33

34 Questions & Answers 34

35 Thank You! Scott Harden Please complete the online class evaluation form SunGard, the SunGard logo, Banner, Campus Pipeline, Luminis, PowerCAMPUS, Matrix, and Plus are trademarks or registered trademarks of SunGard Data Systems Inc. or its subsidiaries in the U.S. and other countries. Third-party names and marks referenced herein are trademarks or registered trademarks of their respective owners SunGard. All rights reserved. 35