Lessons Learned: A DoD Exadata Migration. Chris Bradham

Size: px

Start display at page:

Download "Lessons Learned: A DoD Exadata Migration. Chris Bradham"

Norah Hicks
6 years ago
Views:

1 Lessons Learned: A DoD Exadata Migration Chris Bradham

Vizuri an Operating Division of AEM Applied Engineering Management (AEM) Corporation Founded in 1986 as a 100% woman-owned business More than 25 years of profitable growth Headquartered in Chantilly,

2 Vizuri an Operating Division of AEM Applied Engineering Management (AEM) Corporation Founded in 1986 as a 100% woman-owned business More than 25 years of profitable growth Headquartered in Chantilly, VA with offices located in major metropolitan areas including; Los Angeles, San Antonio, and Jacksonville Diversified client base including Fortune 500 and major government agencies Industry recognized awards and certifications for performance, capability and delivery

3 Chris Bradham Oracle DBA experience 1997 to present (Oracle 7 to 11.2) Replication, Materialized Views, Data Guard, GoldenGate, Exadata, RAC Part-time Instructor George Mason University (OCA/OCP) Oracle Certified Exadata Implementation Specialist, Oracle Certified Professional (11g), Oracle Performance Tuning Certified(11g), ITIL Foundation, Security+ Oracle Application Server, SAN Administration, Linux Administration, VMWare, IBM / Sun Blade Administration cbradham@vizuri.com

4 What s being covered? Technology Refresh Legacy Environment / Options Operational / Migration Considerations Results of Migration Lessons Learned References Q & A

5 Background Information Global multi-service DoD Web-based application Over 300 schemas 750 Gb of data 4,300 Active Users 4.2 million annual log ins 4,500 Reports Generated Per Day AEM Corporation responsible for Hosting / Operations & Maintenance / Technology Refresh

6 PowerVault 720N Legacy Configuration Network Topology Diagram Production Production Environment Hosting Facility Physical Application Servers JBoss JVM IDM Server - BOXI R3 JBoss/CF Server FM Server Tomcat/PPV Physical Application Servers Oracle Database Servers Real Application Cluster RAC Gigabit Switch 4 Gbps Fibre Channel Disk Storage/ Tape Vault RAC-1 RAC-2 RAC-3 Fiber Channel Switches Clarion Disk Array (10k / 15k RPM) 5 Node RAC Servers RAC-4 RAC-5 Oracle 10gR2 RAC install Disk Array for: Transaction Database Archive data Binary Media Files Fiber Cable Ethernet Cable

7 Pre-Tech Refresh Issues Legacy hardware over six years old Patches (5 nodes, slower machines) Deployments, data updates time consuming Large or complex reports often hang Node evictions Oracle Support ended 6/31/11 Data Warehouse delay due to performance requirements (Oracle Streams attempt)

8 PowerVault 720N Alternative 1 : Based on Legacy Solution Proposed Network Topology Diagram Production Production Environment Hosting Facility Virtual Application Servers JBoss JVM IDM Server - BOXI R3 JBoss/CF Server FM Server Tomcat/PPV Virtualized application servers Disk Storage/ Tape Vault Oracle Database Servers Real Application Cluster RAC RAC-1 RAC-2 RAC-3 Gigabit Switch Fiber Channel Switches Network bonding 8 Gbps Fibre Channel EMC Disk Array RAC-4 RAC-5 5 Node RAC Servers Oracle 11gR2 RAC install Extended Disk Array Fiber Cable Ethernet Cable PHASE 1 PHASE 2 Disk Array w/ partitions for: Transaction Database Data Mart (Star Schema) Archive data Binary Media Files (Share Point storage) *VM Templates

9 Alternative 2 : Based on Exadata Solution Virtualized application servers Vmware App Servers Network bonding DNS 40 Gbps backbone MGT Server CommVault ORACLE EXADATA READY/ ACTIVITY CLEAN MEDIA ERROR DRIVE ATTENTION PowerVault 122T CANCEL PREVIOUS NEXT ENTER LTO Autoloader Tape Library Oracle Storage Servers 2 node Quarter Rack Oracle 11gR2 RAC preconfigured EMC CX4-120 EMC Fibre switch EMC Fibre switch V~,1.0A,50/60Hz LAN T 2A, 250 VAC CAUTION: For continued protection against risk of fire, replace only with same type and rating of fuse. KVM PDU1 PDU2 Surprise, we chose Exadata!

10 Exadata Selection Points Licensing fees made Exadata the low cost solution Total database hardware solution o 2 nodes vs. 5 nodes o Number of Vendors Patch Strategy Sizable and expandable Oracle vested to help DoD succeed Storage Indexes / Smart Scan / Smart Flash Cache

Throughput Gb/Second 80.0 70.0 75 60.0 50.0 2 Gbps Fibre Channel x2 40.0 30.0 20.

11 Throughput Gb/Second Gbps Fibre Channel x Gbps Fibre Channel x2 8 Gbps Fibre Channel x2 Exadata 1/4 - Disk Exadata 1/2 - Disk Exadata Full - Disk Exadata 1/4 - Disk & Flash Exadata 1/2 - Disk & Flash Exadata Full - Disk & Flash

12 Exadata Planning/Considerations Pre-configuration customer worksheet (DBM Configuration spreadsheet) Initial Setup Activities Network Configuration / Software Installation Procurement considerations (2 Quarter Racks) Space Choices High Capacity (3 Tb / 7.2 K RPM) High Performance ( 600 Gb / 15k RPM) Space Allocation (ASM Disk Group layout) 78% DATA, 20% RECO, 2% DBFS_DG o 25 Tb <- Production (Triple Mirror) o 8 Tb <- Test (Mirror)

Tech Refresh Challenges 100% hardware replacement and Data Center move Narrow window for outage to avoid negative impact on end users Performance of system, database

13 Tech Refresh Challenges 100% hardware replacement and Data Center move Narrow window for outage to avoid negative impact on end users Performance of system, database growth, and network bandwidth Exadata unproven in DoD space at the time (Security) Upgrading Database versions (data/code/reports) Lots of change, what if issues surface???

simultaneous write to flash / disk (faster writes) with minimal space (512 Mb) Write-Back cache 11.2.0.3.

14 Smart Flash Cache Considerations Helps with Write-Through cache voids caching data that will not be reused Holds hot data, much faster than disk (small, random I/O) Data not duplicated from cache in other Storage Servers Reduce latency of log write by simultaneous write to flash / disk (faster writes) with minimal space (512 Mb) Write-Back cache Don t touch except for Alter table <table_name> flash_cache keep; Create Flash Disks out of the Flash Cache Reassign portion for TEMP tablespace on index builds

Database Node Considerations Database Consolidation SGA Settings AMM Bad! ASMM Good! (set minimum values) Huge Pages (use_large_pages=only) Where s the shared storage space?

15 Database Node Considerations Database Consolidation SGA Settings AMM Bad! ASMM Good! (set minimum values) Huge Pages (use_large_pages=only) Where s the shared storage space? DBFS is the answer (fix_control=8,ac_timeout=60 and SGA=2Gb) Is everything setup correctly? Exachk is the answer Indexes / Hints / Compression Large Segments <- 8 Mb Initial / Next Size with Autoallocate TEMP <- BIGFILE, Autoextend 1 Gb, Uniform 1 Mb

Exadata Patch Management Multiple Patches Infiniband (once per year) DB Nodes / Storage Server (quarterly) Bundle Patch (BP) DB Software (quarterly)

16 Exadata Patch Management Multiple Patches Infiniband (once per year) DB Nodes / Storage Server (quarterly) Bundle Patch (BP) DB Software (quarterly) Additional components (Ethernet switch, KVM, PDU) Bug Fixes included so important to apply Proceed with caution: one-off patches Rolling option time a consideration

Security DoD 8570 Requirements Security Technical Implementation Guide (STIG) o Oracle installation not customizable o DBFS and idle_time don t play well together o Automatic Service Request (ASR) /

17 Security DoD 8570 Requirements Security Technical Implementation Guide (STIG) o Oracle installation not customizable o DBFS and idle_time don t play well together o Automatic Service Request (ASR) / Configuration Manager Limitation o Grid Control / Third Party Certificates (September 2012 release) o Banners / SQLNET.ORA settings impact on tools Don t assume security settings will not have impact. Must TEST!!!

18 Migration Strategies to x Options Considered DBFS with external tables (5 to 7 GB/sec file system I/O throughput) GoldenGate with datapump (near-zero downtime) Datapump Factors Maintenance window Risk of data loss Familiarity with technology Whatever the choice, perform multiple trial runs for optimal settings.

19 2011 Technical Refresh (Data Center move) On 9/9/11 at 7pm application servers at legacy site were Turned Off: Transferred all data to Data Center Network outage occurred during data transfer (2 hours) On 9/10/11at 7am New System Testing was Initiated Users were on the system by 3pm Vmware App Servers DNS MGT Server CommVault ORACLE EXADATA READY/ ACTIVITY CLEAN MEDIA ERROR DRIVE ATTENTION PowerVault 122T CANCEL PREVIOUS NEXT ENTER LTO Autoloader Tape Library EMC CX4-120 EMC Fibre switch In Future-> X3 or Storage Expansion EMC Fibre switch V~,1.0A,50/60Hz LAN T 2A, 250 VAC CAUTION: For continued protection against risk of fire, replace only with same type and rating of fuse. KVM PDU1 PDU

20 Migration Timeline 1/4 Rack Delivered Migrate (Test) Grid Control Setup Setup DBFS Text Test Oracle CAB DB Setup / Load STIG Prod/ Apply BP Cutover to Exadata 1/11 2/11 3/11 4/11 5/11 6/11 7/11 8/11 9/11 Chris First Day Ini@al DB Setup / Load STIG Test Apply BP Test Migra@on Selec@on 1/4 Rack Delivered (Prod) Migra@on Test Grid Control Setup

21 Post Tech Refresh Performance (in hours) Event Legacy Exadata Process 1 4 hours 1 hour Process 2 4 minutes 23 seconds Datamart 3 hours 35 minutes MV refresh 37 minutes 13 minutes

Exadata Lessons Learned Ensure hosting center can accommodate

necessary) Testing required, ideally 2 Exadata Database

fast full index scans, parallel with parallel_degree_policy

performance (DML) Data Warehouse (EHCC) IORM not heavily used

22 Exadata Lessons Learned Ensure hosting center can accommodate Exadata s dimensions Staff requirements (more communication necessary) Testing required, ideally 2 Exadata Database machines Smart Scan <- direct path reads, table access full, fast full index scans, parallel with parallel_degree_policy not auto Chained rows void smart scans EHCC 10x space and performance (DML) Data Warehouse (EHCC) IORM not heavily used by customers X In-Memory Massive Memory All I/Os to Memory X3 2012

23 Exadata Lessons Learned (cont.) Grid Control for monitoring / managing components Expect CPU utilization to decrease Expect Disk failures Can t mix drive types and pricy to switch Standard tuning principles apply (OLTP) DB link opportunities for tuning Platinum Support, major assistance Exachk and opatch before / after patching Time, Experience keys to stability

24 References performance pdf exadata-storage-technical-overview pdf

25 References (cont.) Database Machine and Exadata Storage Server ( ) Oracle Exadata Database Machine exachk ( ) Oracle Exadata Best Practices ( ) Best Practices for OLTP on the Sun Oracle Database Machine ( ) Best Practices for Data Warehousing on Database Machine ( ) Oracle Sun Database Machine Application Best Practices for Data Warehousing ( ) Oracle Sun Database Machine Diagnosability and Troubleshooting Best Practices ( ) Expert Oracle Exadata (Osborne, Johnson, Poder)

26 Questions??? Thank you!!!

27 Backup Slides

28 X2-2 Quarter Rack Specifications 2 Xeon-based Dual-processor Database Servers (Sun Fire X4170 M2) 24 cores (12 per server) 192 GB memory expandable to 288 GB (96 GB per server expandable to 144 GB) 10 GigE connectivity to Data Center 4 x 10GbE ports (2 per server) 1.1 TB High Speed Flash 3 Exadata Storage Servers X2-2 All with High Performance 600GB disks OR All with High Capacity 3 TB disks 2 Sun Datacenter InfiniBand Switch port Managed QDR (40Gb/s) switch 1 Admin Cisco Ethernet switch Keyboard, Video, Mouse (KVM) hardware Redundant Power Distributions Units (PDUs) Can Upgrade to a Half Rack or just add storage

Quarter Rack 2 DB nodes 2 Infiniband switches Half Rack 4 DB nodes 3 Infiniband

29 Exadata Rack Options 2 six Core Processors / 96 Gb RAM per DB node 2 six Core Processors / 24 Gb RAM per Storage Server Dual ported 40 Gb/sec InfiniBand Quarter Rack 2 DB nodes 2 Infiniband switches Half Rack 4 DB nodes 3 Infiniband switches Full Rack 8 DB nodes 3 Infiniband switches Infiniband 10x faster than Fiber Channel

International Journal of Computer Engineering and Applications,

International Journal of Computer Engineering and Applications, EXADATA DATABASE MACHINE: CLOUD IMPLEMENTATION- IN A BOX: Lecturer, Dept of Information technology Ranchi Women s College, Ranchi-834001, Jharkhand. dollyviv@gmail.com ABSTRACT: After remaining happy with