Lessons Learned Migrating a Major Application to Exadata v2

Transcription

1 Leon Learned Migrating a Major Application to Exadata v2 Oracle Open World 2010 Aaron Werman Bank of America aaron.werman@gmail.com

2 Diclaimer All opinion are thoe of the author No endorement are intended: thi i a technical preentation intending to help other following in a imilar path

3 Application Ue Cae Overview Complex integration of capital market trading data Hundred of ETL, Thouand of table 10K+ ETL execution per day, many highly complex Near real time SLA ODS with data haring for entire line of buine Several web application, each with multiple hundred of uer, doing reporting and analytic querie Buine, not traditional BI, SLA for availability and recovery

4 Evaluation Conideration Very uncomfortable with technology rik! Zero rik POC Run entire production load Migrate full application Tet with full production volume for a long period (1 month) Scale to identified 3 year growth goal Run a week of load in a day Determine how long it take to run a day of load in a compreed period (ee how overlap affect performance) Reporting caling goal: 25K report/hour Recovery time: RTO goal 2 hour

5 Alternative Conidered and Rejected Key Iue: mixed workload. Obviou candidate are Teradata, DB2, Netezza, Oracle for ODS and a column databae for DW General concern: migration from Oracle entail time/cot and rik. Application i highly tuned to Oracle phyical deign Teradata concern: too mall a ue cae for their weet pot DB2 concern: migration, organizational iue, and new upport iue Netezza: recovery model, fact/fact join, mixed workload, tuning, LOB upport New column databae for reporting: organizational rik and current ODS I/O bottleneck No other POC were done! We choe Exadata baed on migration rik avoidance

6 Exadata Jutification for the Application Migrating from Oracle to another platform migration entail ignificant chedule rik Scaling Oracle ourelve i not jutified by cot / rik / technology tack (but may be le painful in term of corporate architecture) Potential lo of buine capability and likely mi of critical SLA if we do not cale adequately Current gap in corporate SAN engineering to upport VLDB (and 100TB application)

7 Prep Work: Oracle 10g to 11g Converion/Validation We created a mall copy of the app in Oracle 11g and teted for functional gap No iue were raied depite ome of our tack (Informatica v8.1) not being certified for 11g)

8 Data Migration Key iue are time to migrate and dik pace requirement and complexity We rejected RMAN Oracle 10g ingle intance Exadata Require migration to 11g, ASM, RAC too many tep We choe to ue export: data pump (network) for almot everything export claic for large LOB table Be careful about conidering ASM torage in FRA a a copy target there are limitation; review with Oracle!

9 Exadata Target State Architecture Primar y IIS,.NET Java MQ, WMB Cogno Informatica EMC SRDF Synchronou Replication DR IIS,.NET Java MQ, WMB Cogno Informatica X Oracle Phyical Data guard (TCP/IP) X

10 Bug Some major bug encountered: High CPU utilization of orarootagent.bin proce with CRS ASM AND DATABASE HANG - CONNECT: OSSNET: CONNECTION FAILED TO SERVER, RESULT= ORA-00600: internal error code Thee are now corrected a part of the current Exadata Oracle releae

11 Performance Tuning Mot due to SQL optimization difference between Oracle 10g and 11g ~40,000 SQL tatement in app 68 tatement identified a ubtantially lower 37 conidered non-sla relevant and ignored 31 important SQL tatement with ignificant SLA impact 26 reolved uing pro 3 reolved uing hint 2 reolved by query rewrite optimizer_ue_ql_plan_baeline? Note that mot tatement improved in performance, and improved in proportion to how much work/time they took

12 RAC Tuning Our DBA, baed on prior RAC trategie, initially partitioned the app to egregate load and prevent potential lock/block overhead 8K page ize only teted After tuning, we determined there wa no gain, and all load wa allowed acro all node Your mileage may vary

13 Support Model Really complex to implement in our enterprie Diruptive technology require change in trategy for many takeholder, epecially infratructure upport group Include time in your plan to allow for the tranition Include ALL takeholder in your planning Backup: NAS/NFS... RMAN uing a certified agent Who manage the SA role? Monitoring (corporate tandard v. Oracle practice ) SNMP

14 Application Deign Future Baed on Exadata Application change, uch a reducing our real time ETL SLA by 2/3 Index removal We will experiment and remove many for purpoe indexe Incremental trategy with ufficient teting required ILM uing Hybrid Columnar Compreion Reducing duplication of data between operational and reporting requirement Likely BI (read-only reporting) againt diater recovery ite uing Active Data Guard