Oracle E-Business Suite 12.2 Administration Prepared by: Ivo Dujmovic Ivo@patchivo.com Entrepreneur Patchivo, Interana REMINDER Check in on the COLLABORATE mobile app
About the Presenter Ivo Dujmovic Patchivo : 12.2 PatchStatus free iphone app Specialized tools delivering insight into IT ivo@patchivo.com Interana : Hadoop accelerator for customer behavior analytics Stealth VC-backed startup ivo@interana.com Formerly Oracle Snr Director of Development Managed development organization for EBS Administrative functionality: Install, Cloning, Configuration, Patching, EM plugins, OAM, isetup, TechStack Wrapped up 12.2 to pursue entrepreneurship Holds Online Patching patents
Main Changes from 12.0/1 to 12.2 Re-platforming to FMW 11, DB 11.2 same WLS, webtier, DB as used by Fusion Applications Fusion applications use a different FND core version Native consumption of technology Install: bootstraps then defers to native tech installs Configuration: bootstraps then defers to native tech config Cloning: wraps native tech cloning 10.1.2 Forms & Reports considered EBS internal Online Patching Uses EBR from 11.2 of DB New concepts to EBS: dual FS, FS cloning Cheat sheet for Online Patching
Fusion Middleware (FMW) 11g WebLogic Server (WLS) 11g BEA acquisition, first partially integrated version Key concepts Admin Server manages Managed Servers (farm) Managed Servers have local copy of config, apps if Admin Server is unavailable Industry leadership due to great administrative functionality Deploy new version of staged App from AS to select MS s Run two versions side-by-side or preview new version with secret url Roll-back to previous version if needed
Other Technology WebTier delivers Oracle HTTP Server, aka Apache 2.2 Needed for mod_security Database 11.2 was the latest at development time, as well as the first delivering EBR, the key feature for Online Patching Grid Install concept: Lay down grid infrastructure (storage management, fast network interconnects) prior to installing the database binaries on top Forms & Reports 10.1.2 Same as in other R12 releases Forms servlet deployed to WLS instead of OC4J
12.2 Install Preparation Virtualization makes sense virtualbox with 64b Linux 9GB mem for the install; otherwise hard to troubleshoot bugs Platform install doc has gotcha s listed Hostname/network sensitivity even more pronounced OS patching and settings: install 32b libs before 64b ones Staging script When downloading zip files You do not need all You should not unzip them your self Start with startcd and follow readme See blog on patchivo.com for more tips & details
Dual File System File System: 1 R2 File System: 2 Oracle HTTP Server WebLogic Server Developer 10.1.2 APPL_TOP COMMON_TOP INST_TOP Edition-Based Redefinition Synchronization managed by patching tools Oracle HTTP Server WebLogic Server Developer 10.1.2 APPL_TOP COMMON_TOP INST_TOP File System: Non Editioned APPL_TOP_NE PATCH_TOP LOGS
12.2 File System for Install $ ls /oracle ebs122 orainventory stage $ ls /oracle/stage/ zipmedia client Disk1 EBSInstallMedia misc startcd current Disk2 examples patches TechInstallMedia database Disk3 gateways readme.htm TechPatches deinstall Documents grid README.txt wls1036_generic.jar
12.2 File System: FS1, FS2, FS_NE $ cd /oracle/ebs122/ $ ls EBSapps.env fs1 fs2 fs_ne VIS $. EBSapps.env E-Business Suite Environment Information ---------------------------------------- RUN File System : /oracle/ebs122/fs1/ebsapps/appl PATCH File System : /oracle/ebs122/fs2/ebsapps/appl Non-Editioned File System : /oracle/ebs122/fs_ne DB Host: ip-10-28-88-207.ec2.internal Service/SID: VIS
12.2 Key Environment Variables FILE_EDITION=run RUN_BASE=/oracle/ebs122/fs1 PATCH_BASE=/oracle/ebs122/fs2 CONTEXT_FILE=/oracle/ebs122/fs1/inst/apps/VIS_i p-10-28-88-207/appl/admin/vis_ip-10-28-88-207.xml ADMIN_SCRIPTS_HOME=/oracle/ebs122/fs1/inst/apps /VIS_ip-10-28-88-207/admin/scripts
FS1 - File System One $ ls fs1 EBSapps FMW_Home inst $ ls fs1/ebsapps/ 10.1.2 appl comn $ ls fs1/inst/ apps $ ls fs1/inst/apps/vis_ip-10-28-88-207/ admin appltmp logs out portal soa appl conf_vis.txt ora pids rgf temp
Instance Top of FS1 $ ls fs1/inst/apps/vis_ip-10-28-88-207/appl admin fnd $ ls fs1/inst/apps/vis_ip-10-28-88-207/appl/admin/ adovars_vis_ip-10-28-88-207.env oamextensions.xml cutover/ forms-c4ws_wls.properties forms_wls.properties fsclone_config.txt oacore_wls.properties ojspcompile.conf ojspcompile.properties VIS_ip-10-28-88-207_patch.txt VIS_ip-10-28-88-207_run.txt VIS_ip-10-28-88-207.xml oafm_wls.properties
FS_NE Non-Editioned File System $ ls fs_ne EBSapps inst $ ls fs_ne/ebsapps/ appl log patch $ ls fs_ne/ebsapps/appl/ ad $ ls fs_ne/ebsapps/log adop adopreports $ ls fs_ne/inst/ VIS_ip-10-28-88-207 $ ls fs_ne/inst/vis_ip-10-28-88-207/ certs logs oracle_common soa
12.2 Database File System $ ls EBSapps.env fs1 fs2 fs_ne VIS $ ls VIS/ 11.2.0 checkpoints data diag $ ls VIS/checkpoints/ $ ls VIS/diag/ asm clients crs diagtool lsnrctl netcman ofm rdbms tnslsnr VIS/11.2.0 is the database ORACLE_HOME
12.2 Admin Scripts $ ls $ADMIN_SCRIPTS_HOME 01041419.log adexecsql.pl adstpall.sh jtffmctl.sh adadminsrvctl.sh adformsrvctl.sh adstrtal.sh msc adalnctl.sh admanagedsrvctl.sh cz64bitengine.pl mwactl.sh adapcctl.sh adnodemgrctl.sh gsmstart.sh mwactlwrpr.sh adautocfg.sh adopmnctl.sh ieo sqlnet.log adcmctl.sh adpreclone.pl java.sh $ ls /oracle/ebs122/vis/11.2.0/appsutil/scripts/vis_ip-10-28-88-207/ adautocfg.sh addbctl.sh adexecsql.pl adpreclone.pl adstrtdb.sql adchknls.pl addlnctl.sh adlsnodes.sh adstopdb.sql
New Cloning Type: FS Clone Clone an Instance: Prod to Copy-of-Prod, Test1 to Test2 Clone a Node: scale-out, rolling upgrades New: Clone a File System Replicate one File System on a node, from the other File System on that node Instance and Node Cloning still copy bits for only one FS At target bits are recycled for creating both FS s
ADOP AD Online Patching $ which adop /oracle/ebs122/fs_ne/ebsapps/appl/ad/bin/adop $ adop -help Applications DBA Online Patching Tool (adop) Usage: adop [phase=<phase,phase,...>] [patches=<patch#,patch#,...>] [<parameter>=<value>...] [input_file=<filename>] Enter adop -examples for a detailed list of parameters and their usage. See Oracle E-Business Suite Maintenance Guide for a full description of adop features, operation, and usage.
Online Patching Affects DBA s Life Traditional Patching involved: scheduling downtime, kicking off users, stopping services, applying patch, restarting services Online Patching allows for patch application to happen during online use: Schedule quiet time to apply patch, bounce services Downtime slims to 5-20mins range, easier to schedule If applying patchsets, load can be non-negligible Target 1 RAC node (similar to reporting node ) Apply patches during quiet periods Adworker count logic is improved, but can still be manually set No workshifts/ramp-up
ADOP Phases Standard phases: prepare apply finalize cutover cleanup - Prepare the instance for patch application. - Apply patches (to the patch edition). - Ready the instance for cutover. - Make the patch edition the new run edition. - Drop obsolete objects and data from old editions. Special phases: abort - Abort the current patching cycle. actualize_all - Create new copies of all code objects in the patch edition. fs_clone - Copy the run file system to the patch file system.
ADOP Phase Parameters Apply parameters: patches=<patch#>[,<patch#>...] A single patch or comma-separated list of patches to apply. This parameter is required when executing the apply phase. For language patches, you must also specify the driver file: patches=10124646_ar:10124646.drv,10124646_ko:10124645.drv restart=(yes no) [default: no] Resume a failed apply action where processing left off. abandon=(yes no) [default: no] Re-apply a failed patch from the beginning. Finalize parameters: finalize_mode=(full quick) [default: quick] Quick mode will provide the shortest execution time, by
More ADOP Phase Parameters Finalize parameters: finalize_mode=(full quick) [default: quick] Quick mode will provide the shortest execution time, by skipping non-essential additional actions. Full mode performs additional actions such as gathering statistics, which may improve performance after cutover. Cutover parameters: mtrestart=(yes no) [default: yes]. Specifies whether to restart application tier servers after cutover. Leave at default unless performing manual steps during downtime. Cleanup parameters:
More ADOP Phase Parameters Cleanup parameters: cleanup_mode=(full quick) [default: quick] Quick mode provides the shortest execution time, by skipping non-essential additional actions. Full mode performs additional processing to remove all unused code, data, and old editions. May take a long time. General parameters apply to all phases: workers=<number> [default: computed] Number of parallel workers used to execute tasks. Default value is computed principally according to number of available CPU cores.
All ADOP Phases: Shared Parameters input_file=<file_name> As well as being entered directly on the command line, adop parameters can be specified in a text file, with one <parameter>=<value> on each line of the file. Examples: phase=prepare,apply,finalize,cutover,cleanup patches=123456 workers=4 Command line parameters override input file parameters. loglevel=(statement procedure event warning error unexpected) [default: event] Controls the level of diagnostic log detail displayed.
More Shared Parameters allnodes=(yes no) [default: yes] Specifies whether actions should be executed on all application tier nodes of a multi-node system. action=(db nodb) [default: db] Specifies whether to execute database actions (as well as file system actions). -status [<session_id>] Display status of the latest adop session, or a specified session. -help -examples
Adop Examples Complete patching cycle, running each phase separately: adop phase=prepare adop phase=apply patches=12345,67890 workers=4 adop phase=finalize workers=4 adop phase=cutover workers=4 adop phase=cleanup Complete patching cycle, running all phases in a single adop command: adop phase=prepare,apply,finalize,cutover,cleanup patches=12345 Complete patching cycle, specifying all parameters in an input_file: adop input_file=adop2013_05_13.txt
Online Patching Cycle - Cutover Cutover Requires a Brief Downtime Middle-tier processes stopped End Users are disconnected File system roles swapped Patched (FS-2) promoted to Run FS-1 available for next patching cycle Database Patch Edition promoted to Run Edition Middle-tier processes restarted Users reconnect
Great Presentation on Online Patching Slides with red corner from Kevin Hudson s: https://oracleus.activeevents.com/2013/connect/sessiondetail.w w?session_id=8432
ADOP Special Phases abort - Abort the current patching cycle. actualize_all - Create new copies of all code objects in the patch edition. fs_clone - Copy the run file system to the patch file system.
ADOP Behind The Curtain: Prepare Prepare Checks if cleanup was run after last cutover If not, runs it now Checks what was done to the other FS (currently run) Patches applied Configuration changed (both EBS and WLS,Webtier) Redoes those on this FS: patches applied and configuration cloned Checks if there was an aborted session on this FS (current patch) Losses confidence in the patchfs, so calls fs_clone to recreate the patchfs from the runfs
ADOP Behind the Curtain: Apply Apply Applying many patches within a single list merges them To apply patches without merging, use separate adop calls Applying many patches unassisted in test systems Use abandon = yes Cutover Does a finalize first if it was not run as last action before cutover Run finalize before cutover Runs certain (deferred) db actions from patch apply Duration of cutover can be impacted by patches applied in cycle
Hotpatching Apply a patch to runfs with your historical hotpatching processes adop phase=apply hotpatch=yes patches= Must be done outside normal patching cycle, so if you already did a prepare, you need to do a cutover (empty cycle) Empty cycle does not require an fs_clone unlike the other option: abort current cycle Not recommended unless you are really sure in what you are doing: review the patch, read all files delivered and actions listed in driver (some actions are not really documented) Know which users will be connected Bounce processes
Patching Many Nodes Adop automatically synchronizes patching across nodes FS_NE is shared across nodes, with patches and logs Sharing the file system amongst many nodes is possible to the extent that the underlying technologies allow it DB has Oracle Home sharing Webtier / Apache 2.2 is easy to share WLS has some competing infrastructure Rolling patching is possible, but has no benefit
Applying Technology Stack Changes [worse] Outside a patching cycle it should be done to the Run file system Followed by an FS_Clone to separate propagation of change from patching [better] Within the patching cycle After prepare finishes, perform on target Patch file system Applies to techstack config as well, although those changes should be automatically detected and propagated
Custom Code Patching During ADOP cycle Apply to Patch FS Register files in FS_NE/EBSapps/appl/ad/custom/adop_sync.drv Will propagate from Run to Patch FS
Special Cutover Processing See doc or code for pre-cutover and post-cutover hooks You can hang your special cutover processing functionality
Q & A
Oracle E-Business Suite 12.2 Administration Session ID#: This is a subtitle for the presentation that can be extended to three lines Prepared by: Joseph Josephine Head of Oracle Product Usage Acme Manufacturing REMINDER Check in on the COLLABORATE mobile app