Independent Electricity System Operator Rapid Migration of Big Data - Oracle DB using IBM Enterprise Storage Tools

Transcription

1 Independent Electricity System Operator Rapid Migration of Big Data - Oracle DB using IBM Enterprise Storage Tools

2 Presented by: Sajid Rizvi, Oracle Database Consultant / IT Architect IBM GBS Customer Representative: Noel Sabandal, Architecture and Quality Assurance Lead IESO Contributions Mark Bunn, Lead IT Architect- IESO account - IBM GBS 2 2

3 IESO Rapid Migration of Big Data Business Context 3

4 Application (MDM/R) processes at a Glance Smart Meter Entity (SME) - IESO On March 28, 2007, the Independent Electricity System Operator (IESO) was designated as the SME by Ontario Regulation 393/07 under the Ontario Electricity Act. Responsible for the operation of a central meter data repository the Meter Data Management and Repository (MDM/R) Smart meters installed across Ontario : 4.77M Active SDP/meters enrolled in MDM/R : 4.48M Processing intervals and register reads daily over : 107M Projected upon enrolment of all SDP/meters : 120M Delivering Billing responses daily : 133K Shared Service for 71 Local Distribution Companies (LDCs) and 9 LDC Authorized Agents 4

5 Operational Challenges CPU utilization reaches 100% SAN storage utilization reaching maximum capacity RMAN Backup execution time taking over 8 hours for full Backup No more CPU can be added due to hardware limitation No more SAN can be added to the existing infrastructure RMAN run time increases with data growth hitting application performance. Additional test environment required Test Environment refresh from Production takes 18 hours using RMAN Restore causing delays in testing activities. Disaster Recovery RPO and RTO not meeting business requirement Unable to add SAN and Server to the current infrastructure Fast Refresh option unavailable on current infrastructure DR needs to be un-squished and CPU memory borrowed. 5

6 Infrastructure Upgrade Objectives Rapid Migration of Production and all other testing environments to new Infrastructure Provision production size test environment without production down time Improve Performance with upgraded Power system and Storage Instant Recovery in case of failure Improve Backup process reducing load on Production Meet Disaster Recovery RPO and RTO. 6

7 IESO Rapid Migration of Big Data Technical Context 7

8 Migration Challenges 45 TB of Production and DR Oracle Database environments to be migrated from Old to New infrastructure during one weekend outage. Long testing cycles for Functional, Performance and User acceptance tests requiring multiple cut over from Production. DR environment to be tested as Independent primary Production first followed by conversion to DR environment during Parallel run. A parallel run of one month to be performed on old and new production Two Production like test environments to be cut over before Production. Multiple cut over required to run latest input files to compare old and new production environments. Multiple iterations of a test cycle to be done using Oracle Flash Copy 8

9 Migrating 45 TB Storage (Oracle Databases) within a 12 hour maintenance window. Design and build 9 new environments on the new infrastructure Initial creation/restore of large Oracle database from tape RMAN backup took 14 hours (not including the logistics of tape movement) How is it possible for us to synchronize this enormous OLTP Database fast enough to allow a seamless transition to the new infrastructure? Need to synchronize the database at minimum twice for parallel testing and for production cutover Need to be able to repeat the synchronization at short notice to allow for rapid recovery of testing an cutover testing or cutover 9

10 Existing Environment Challenges SAN Storage reaching its maximum capacity and Database size continues to grow Continuous performance degradation with increased volume. RMAN backup conflicting with application processing CPU and Memory capacity maxed out on frame and no new CPU power can be added Additional test environments required for Performance and volume testing RMAN Backup execution time taking over 8 hours affecting application performance Test Environment refresh from Production takes 18 hours using RMAN Restore. Application performance slowness. Unable to process expected volumes within allotted window. Enhanced Disaster Recover capability and other services to address LDCs evolving requirements Upgrade and Migration Bendfits Mix of Solid State & IBM DS8870 System Storage with Easy Tier feature to automatically optimize performance Gold Tier Storage added with Flash drive using Easy Tier and IBM Flash Memory. Increased CPU and Memory with new P7 Frames and DS8870 SVC Storage New Production size environment for Performance test added to the frame. New Backup server added to off load RMAN backup from Production using IBM Flash copy & Oracle RMAN Removed impact to Production completely IBM FlashCopy with Oracle RMAN Restore to reduce database Restore window to 4 hours With upgraded state of the art hardware and Oracle changes, the application performance increased New Disaster Recovery Environment with RTO of 6 hours compare to 24 hours in old environment 10

11 11 11

12 12 12

13 13

14 14

15 15

16 16

17 IESO Rapid Migration of Big Data Flash Copy 17

18 IBM Flash Copy for Oracle Database FlashCopy runs at block level and across all LUNS Consistency Groups required for Flash copy Operation (Data+Index & FRA) Works well with file system, ASM over multiple LUNs or a mixture of both ASM and file system DATA consistency group contains data and index files FRA consistency group contains Redo logs, Archivelogs and Flashback logs FlashCopy must be performed in two stages, Data Consistency followed by FRA. 18

19 19

20 IESO Big Data Migration Storage Tier 20

21 Storage Tiers TIER Description Proposed Solution TIER 0 TIER 1 Ultra High Performance. Meet QoS for High End Mission Critical applications High Performance and/or Availability. Drive up utilization of high-ended storage subsystems and still maintain performance Q0S objectives For low capacity requirements smaller less powerful devices may meet tier definition < 2ms Avg Response Time > 1000 Avg IOPS/LUN DS8870 with SSD < 12ms Avg Response Time 120 Avg IOPS/LUN DS8870 with 300GB 15,000 rpm Disk drive set in RAID5/RAID10 arrays. TIER 2 TIER 3 Medium Performance and/or Availability. Revenue generating applications. Meet QoS for non-mission critical Applications For low capacity requirements smaller less powerful devices may meet tier definition Low Performance and/or Availability. Non-Mission critical applications. Archiving. < 12ms Avg Response Time 80 Avg IOPS/LUN Not Used. < 10sec Avg Response Time 50 Avg IOPS/LUN DS8870 with 3TB 7.2K Disk drive set in RAID5. TIER4 Long term retention, backup. Client data recovery from Tape TS35 21

22 Oracle Backup comparison before and after Migration PROCESS ENVIRONMENT FREQUENCY EXECUTION TIME Old Production Weekly 10 hours RMAN Full Backup New Production Moved to BKP Moved to BKP New Backup server Daily 5 hours Flash Copy Backup New Backup Daily 20 Minutes 22 22

23 Infrastructure Upgrade Objectives Rapid Migration of Production and all other testing environments to new Infrastructure Provision production size test environment without production down time Improve Performance with upgraded Power system and Storage Instant Recovery in case of failure Improve Backup process reducing load on Production Meet Disaster Recovery RPO and RTO. 23

24 Call to Action IBM Announces New DS8880 To get more Info on DS8880, click here. To learn more about this migration or to see the brand new DS8880 just announced this week, please stop by and visit IBM at Booth # 1301 at Oracle Open World You can meet our presenters and see the new hardware system. 24

25 Thanks to Vishal Verma Khurram Shiraz Jack Gao 25

26 Contacts Sajid Rizvi Mark Bunn - markbunn@ca.ibm.com Noel Sabandal - noel.sabandal@smi-ieso.ca 26

27 Thank You 27