Running E-Business Suite on Exadata: Technical Details April 25, 2017
Gary Gordhamer Principal Technologist ERP Infrastructure Power & Water Technology GE Power & Water 25 years of IT experience 24 years with Oracle (6.x up to 12c) Worked in many different industries including healthcare, manufacturing, utilities, banking, and printing @ggordham linkedin.com/in/ggordhamer/ oraontap.blogspot.com Note: The views in this presentation are my own and do represent the views of the company I work for. Oracle and NLS, #C17LV Session 333 April 25, 2017 2
Topics 1. What is Exadata 2. Networking 3. OS Configuration 4. Cluster Settings 5. Storage settings 6. DB Settings 7. EBS application items 8. Maintain the Exadata Presentation Title April 25, 2017 3
Notes! Oracle has lots of technical notes, you need to follow them! But which ones? My general order of importance: 1. Start with EBS on Exadata notes 2. Then Exadata notes (ignoring most DB specific items) 3. Then EBS notes The EBS on Exadata notes supersede any other notes! List at end of presentation to start with Presentation Title April 25, 2017 4
1. What is Exadata? Hyper converged Infrastructure Compute (DB NODES) Storage (Storage NODES) Network (10G Ethernet + IB) OS + Cluster Management tools (ExaCheck, ExaPatch, etc ) Presentation Title April 25, 2017 5
Exadata Architecture, ½ Rack (X6-2) Complete Optimized Standardized Hardened Database Platform Fully Redundant Standard Database Servers 4x 2-socket servers è 176 cores, 3TB DRAM Unified Ultra-Fast Network 40 Gb InfiniBand internal connectivity è all ports active 10 Gb or 1 Gb Ethernet data center connectivity Scale-out Intelligent Storage Servers 7x 2-socket servers è 140 cores in storage 40 SAS disk drives è 672 TB total capacity HC drives 20 Flash PCI cards è ~90 TB Flash
Redundancy notes All networking is redundant in all configurations (IB and 10GB Ethernet) The administrative network is never redundant (1GB ILOM / ADMIN port on Exadata / eth0) If you are doing on-line storage cell patching, I would highly recommend a minimum of 4 storage nodes E.G. ability to lose 2 storage nodes Presentation Title April 25, 2017 7
2. Networking The network in Exadata can be complex Minimum of 3 networks 1. Admin network (eth0) 1GB Ethernet Minimal usage 2. IB network (ib0 / ib1) 40GB InfiniBand Storage, Cluster, Cache Fusion, possibly NFS, SQL*NET 3. Public network (bondeth0 / eth4 / eth5) 10GB Ethernet Primary interface for applications Note: You can have custom configurations beyond this Presentation Title April 25, 2017 8
Public network / LISTENER Primary connectivity for DB Recommend hostname of DB nodes matches PUBLIC network IP (note, default from Oracle is admin network IP) LISTENER VIP LISTENER, SCAN LISTENER will run on this network Allows for external interfaces (SOA, data loads, DW, etc ) Make sure LOCAL_LISTENER and REMOTE_LISTENER are set right for your EBS database Presentation Title April 25, 2017 9
Admin Network By default, local LISTENER on the server will listen on this IP Make sure application connections are not using this If the ADMIN IP is your hostname: Make sure LOCAL + REMOTE_LISTENER are set right LISTENERS can re-direct connections to wrong network Autoconfig will pick up hostname by default Presentation Title April 25, 2017 10
IB Network If you have application servers on IB, add IB_LISTENER (MOS Note 1580584.1) BUT EBS 12.2 does not support both PUBLIC and IB LISTENER EBS 12.1 does support this You can probably hack the DB configuration, but will break your EBS 12.2 clone process IB Network MTU should be 65520 (Exadata version 12.1.2.1.0 +) Presentation Title April 25, 2017 11
NFS You will probably be using NFS Shared storage between APP and DB tier Backups (ZFS?) DB files for non-production Oracle Note: 2087231.1 covers NFS settings for DB file and backup usage For shared storage between APP and DB tier (CSF) If supported NFSv4 recommended (ZFS supports this) NFSv4 requires shared directory system (NIS or better yet LDAP) Your OS users and groups should exist in your shared directory Presentation Title April 25, 2017 12
NFS Continued NFS Mount Options MOS Note 359515.1 covers NFS v3 options NFSv4 options for IB are covered in EBS MAA Whitepaper nfs4 rw,bg,hard,nointr,rsize=131072,wsize=131072 If not using NFS over IB (ZFS), recommend HA NFS over public network for light usage (shared App storage) DB backups on NFS, recommend physically segmented network Jumbo frames if supported 10GB Ethernet if possible IB if using ZFS appliance for backup Presentation Title April 25, 2017 13
Additional IB / ZFS / Network best practices 1. Enable scatter / gather on Exadata (ver 12.1.2.1.1 +) cat /sys/module/ib_ipoib/parameters/cm_ibcrc_as_csum 1 = enabled 2. Confirm ZFS supports scatter / gather (version 2013.06.05.3.0,1-1.7 + ) 3. Exadata IPoIB receive queue size of 2048 cat /sys/module/ib_ipoib/parameters/recv_queue_size 4. Exadata TCP Segmentation Offload (TSO) is disabled /sbin/ethtool --show-offload <interface> 5. Exadata, confirm IB MTU of 65520 Note: check Exalogic ZFS IB interface firmware for fencing bug 17951397 Presentation Title April 25, 2017 14
3. OS Configuration - HugePages HugePages you need to configure this /etc/sysctl.conf - vm.nr_hugepages=<number of pages> Pages are 2MB on Linux x86-64 by default You can not use all the memory on the server! Save Non-hugepage memory for: Operating System (OS) PGA DB utilities (RMAN, SQL*Loader, SQL*Plus, etc ) Other non database utilities Recommend setting this higher than minimum memory needed to account for DB changes over time. Start with MOS note: 401749.1 for sizing advice Presentation Title April 25, 2017 15
Memory Management Exadata uses RAC / GI clustering, memory management is key! Memory race conditions / out of memory issues = node evictions You must use HugePages for DB memory (OS + DB settings MOS ID 1392497.1) /etc/sysctl.conf - vm.min_free_kbytes = 524288 Reserves low memory for the OS to prevent race conditions (4194304 for X?-8) Also required for NFS usage When consolidating databases: Keep track of all DB memory usage Do no over-use memory on individual nodes Keep enough memory free for sessions + non-database processes Note: many EBS systems have 1,000+ sessions taking up 1MB each Presentation Title April 25, 2017 16
OS Users OS Users need to be consistent across all nodes of the cluster, and on the application tiers! Recommend central directory (NIS or better LDAP) UID and GID number matter! If you have existing, make sure OEDA has those numbers during Exadata install OS Limits - /etc/security/limits.conf + /etc/security/limits.d/* Settings in limits.d directory override the limits.conf file! Be consistent on all nodes (DB + App tiers) Recommend making a custom configuration file for limits.d with your needed setup and copy to all nodes (E.G. 98-mylimits.conf) Presentation Title April 25, 2017 17
Local Storage You need unique storage for EBS database homes EBS DB homes on RAC can not be shared across nodes Two main options Use /u01 and increase the size for what you need Add a new mountpoint (E.G. /p01) Carved out of local disk / LVM volumes on each node Exadata Database Machine Maintenance Guide ' => 'Resizing LVM Partition Note: be sure to leave room for LVM snapshots for backup! Presentation Title April 25, 2017 18
4. Cluster Settings RAC node timeout CSS miscount - seconds of failure for cluster before restart Previously was default to 60 seconds For missions critical apps recommend 30 seconds New default is 30 seconds Reduces cluster stall or hang during incidents (termed as brownout) As GRID owner user: crsctl get css misscount Presentation Title April 25, 2017 19
LISTENER Dead Connection Detection For critical systems, recommended to enable DCD Server side verifies the client is alive SQLNET.ORA setting on server side (LISTENER) $GRID_HOME/network/admin/sqlnet.ora $ORACLE_HOME/network/admin/sqlnet.ora SQLNET.EXPIRE_TIME = <minutes> MOS Note: 151972.1 Presentation Title April 25, 2017 20
5. Storage Settings - IORM For EBS on Exadata Oracle recommends IORM Plan objective=balanced CellCLI command: alter iormplan objective=balanced This setting: Allows large IO s for 90% of disk capacity If there is any queuing of IO, putts small requests at the front of the queue Presentation Title April 25, 2017 21
Flash Cache settings Write Through writes go to disk, Conservative setting Write Back writes go to cache, written to disk out of band You can see more performance, if your EBS has a lot of write operations, with Write Back turned on X5 and bellow defaults to Write Through X6 defaults to Write Back More details and how to change: 1500257.1 Note: Monthly brownout of flash cache for backup battery cycle test Presentation Title April 25, 2017 22
6. DB Settings Log Writer Exadata can accommodate large redo log buffers Minimum of 128MB, can go larger Unset LOG_CHECKPOINT_INTERVAL Use FAST_START_MTTR_TARGET instead MTTR advisor can provide advice (if licensed) Should be > 300 (Exacheck will report this) Helps with batch (CM) program performance Exadata Flash Logging enabled after Exadata 11.2.2..4 + Note: flash logging uses only 512MB of the 300+GB of flash, so a small investment for huge improvement! Presentation Title April 25, 2017 23
Fixed Object Statistics Should be done on the new platform (Exadata) Needs to be done after the database is warmed up Should be done during off-peak time exec dbms_stats.gather_fixed_objects_stats( ALL ); Note 1: 12c has automatic collection for this (unless automatic collection is turned off, which is normal for EBS) Note 2: This is normally a one time item event (may need to be repeated after major patches) Presentation Title April 25, 2017 24
INIT.ORA settings Exacheck will give you good guidance in this space, But, it is just guidance, you will need to sort through what is needed by your system and all the best recommendations. Presentation Title April 25, 2017 25
Database patches Must follow specific note: MOS ID 1392527.1 Database Patches Required by Oracle EBusiness Suite on Oracle Engineered Systems: Exadata Database Machines and SuperClusters This note supersedes any other note when running EBS on Exadata Presentation Title April 25, 2017 26
7. EBS configuration items Concurrent Managers Recommend hosting on application server or dedicated server Recommend not running on Exadata tier Different patching cycles / levels from DB to app tier Exadata = quarterly QFSDP or BP App Tier =??? Easier to manage with application technology stack May still share NFS mounts with DB tier for UTL_FILE access Presentation Title April 25, 2017 27
8. Maintaining Exadata Exacheck Exacheck built in and free tool with Exadata Upgrades at least quarterly, MOS ID 1070954.1 Has specific checks for EBS Recent version = 1,902 checks (cluster, hardware, software, OS, network, database) Collection Manager Free application express (APEX) application Provides central repository for Exacheck reports Alerts, trends, graphs, etc.. You should review the reports and take actions! Presentation Title April 25, 2017 28
April 25, 2017 Presentation Title 29
Patching Exadata Recommend using the QFSDP Quarterly Full Stack Download Patch Patches the OS, Network, Storage, Grid Infrastructure Database can be patched at same time, or latter time Exadata / GI must be at newer or same level as DB patch Patches come out quarterly, contain critical fixes OS CVE Security bugs (shell shock, HeartBlead, DirtyCOW, etc ) Critical fixes NFS buffers, memory leaks, GI stability Patching process will depend a lot on your environment Recommend a N-1 patching level Presentation Title April 25, 2017 30
Suggested Exadata Patching Process Exadata support on-line patching EBS does not support on-line infrastructure patching Recommended method: Shutdown EBS (outage starts) Shutdown DB Patch DB server OS / Firmware Patch GI Bring DB + EBS back up (outage ends) Patch Storage cells on-line (if high redundant, and 4 or more storage cells) Patch IB switch on-line Total outage depends on number of DB nodes, but should be less than 5 or 6 hours. There are other methods and models, but they are much more complex. Presentation Title April 25, 2017 31
Exadata Backups Recommend OS backups on Exadata Use in-house OS backup tools (Tivoli, NetBackup, etc ) Use LVM snapshots + simple OS tools like TAR Oracle Exadata Database Machine Maintenance Guide => chapter 2.10 Creating a Snapshot-Based Backup of Oracle Linux Database Server Example script: /opt/oracle.supporttools/dbserver_backup.sh Note: this script only backups up the OS, not /u01 or any other added mount points. Presentation Title April 25, 2017 32
Wrap up Lots of documents out there Pretty well documented but not all in one place Exadata is a great start, but Still need to piece together a full EBS system Presentation Title April 25, 2017 33
Questions April 25, 2017 34
References Oracle E-Business Suite and Oracle MaximumAvailability Architecture Best Practices April 2014 MOS ID 1392527.1 Database Patches Required by Oracle EBusiness Suite on Oracle Engineered Systems: Exadata Database Machines and SuperClusters MOS ID 2087231.1 Guidelines When Using ZFS Storage in an Exadata Environment MOS ID 1354980.1 Oracle ZFS Storage: FAQ: Exadata RMAN Backup with The Oracle ZFS Storage Appliance MOS ID 1580584.1 Setup Listener on Infiniband Network using both SDP and TCP Protocol MOS ID 401749.1 Oracle Linux: Shell Script to Calculate Values Recommended Linux HugePages / HugeTLB Configuration MOS ID 1392497.1 USE_LARGE_PAGES To Enable HugePages Presentation Title April 25, 2017 35
References MOS ID 1500257.1 Exadata WriteBack Flash Cache FAQ MOS ID 1070954.1 Oracle Exadata Database Machine exachk or HealthCheck Presentation Title April 25, 2017 36