MUFG Capacity Management Transformation

Transcription

1 MUFG Capacity Management Transformation APRIL 15, 2015 A member of MUFG, a global financial group

2 Meeting with you today Ellen Friedman (Director) MUFG UB Leader of Capacity, Performance & Availability Management Joined the bank in March 2009 and developed the Capacity Management and Performance Engineering Practice In the past 6 years, bank has grown 3-fold, huge investments in new technology, new application platforms, Program Hudson, New Core Banking Program, New Market Systems initiatives Performance Engineering/Testing for 25 Major New Applications in the past 18 months Roger Lee (Assistant Vice President) MUFG Lead Capacity & Performance Management Lead Engineer Joined the bank in August 2013 and serves as Lead Engineer responsible for Teamquest strategy, architecture, capacity management reporting framework leads capacity management initiatives particularly for batch optimization (Data warehouse, Market Systems, Core Banking) Prior financial experience (Credit Suisse, Merrill Lynch) and software engineering (Lockheed Martin, PepsiCo), degree in Computer Science, special interest in statistical modeling and operating systems 2

3 MUFG Overview MUFG operates in over 40 countries with more than 1,100 offices. In the Americas we service our corporate and investment banking customers under the MUFG brand, and we go to market with our retail, wealth, and commercial banking customers under the Union Bank brand Americas #of locations: under UNBC EMEA #of locations: 34 Japan #of locations: 1104 Company Name CEO Mitsubishi UFJ Financial Group, Inc. Nobuyuki Hirano Head Office Tokyo, Japan Asia and Oceania #of locations: 58 Countries of Operation 40+ Our vision is to be the world s most trusted financial group 3

4 Agenda 1 BTMU Capacity and Performance Management Services 2 History & Environment Overview 3 Realized & Future Benefits 4

5 MUFG UB Capacity and Performance Management services Planning Phase Business Driven Requirements & Volume Analysis Select vendor, platform, architecture Cost vs. performance tradeoffs: maintainability, reliability, availability and scalability of architecture (VMware vs. Stand-alone Windows Server) Design Phase Performance Estimation & Modeling Evaluate Design Trade-offs Design for Performance & Scalability Risk Prioritization: POCs may be required to validate Testing Phase Performance Testing Application instrumentation to facilitate monitoring End-to-End Performance Monitoring and Tuning Necessary Changes to System Infrastructure/Capacity/Priorities to meet SLA Necessary Changes to Application (e.g., code optimization, heap size, application tuning) Production Support Early-Life Implementation Support Monthly Management Reports Performance Analysis, Patterns & Trending Forecasting, Capacity Planning & Capacity Management Performance Problem Resolution 5

6 Our history with TeamQuest After joining the bank, we leveraged TQ for major new VMware virtualization effort. (June 2009) Purchased licenses for VMware only and floating licenses to evaluate if application met performance/capacity criteria and ROI criteria for virtualization <2 vcpus, < 8 GB memory <200 IOPS (vsphere 2.5- Now vsphere 5.1) Increased the VMware footprint and continued to purchase (project basis) Within past 18 months we have standardized on TQ as our Go-To Solution for capacity/performance monitoring Installation increasing on stand-alone Market Systems Installation on all Citrix, VDI infrastructure, Exchange, File Server (physical and virtual) Installation on Linux on VMware New AIX deployment initiative- rolling out to 400 AIX LPARs 6

7 Environment Overview TeamQuest Deployment Targeted Systems Anticipated Strategic Benefits 54% (1,500) 7% (200) Platform Landscape 14% (400) AIX LPARS 25% (700) Standard tool set with standard data collection to automate/build capacity/performance reports for management, infrastructure and application teams Integration with Intellimagic for Showback of storage usage by application/server/storage array Ability to identify near real-time bottlenecks (past minute) for storage, server and application performance issues Process Level data Citrix Session level data SQL Server data TeamQuest Standardization Progress Full Windows Coverage We are here Full Environment Coverage 7

8 Benefits realized to date: Capacity Planning, Performance Testing, Software License Savings, Production Support - with minimal overhead Report Automation: VMware Capacity Dashboard Report Automation: Storage Showback TeamQuest Deployment and overhead costs Software Licensing Requirements Leveraging TeamQuest during Performance Testing Leveraging TeamQuest for production support/ Root Cause Analysis Realized $ Savings through the automation of monthly management and exception reports Identified top resource consumers to provide management/business users planning data Demonstrated that TeamQuest footprint is lightweight Leveraged TeamQuest to identify actual CPU usage to reduce software license costs Used pre-defined templates for monitoring and reporting while testing Able to identify bottlenecks more quickly for problem determination 8

9 Benefit: Automated Monthly Report Generation: VMware Monthly Capacity Report: Automatically Generated with Surveyor Narrative: 1. What changed from last month 2. How much more can we add 3. When do we need to add more hardware? 4. Show Growth over time 9

10 Benefit: Automated Monthly Report Generation: Showback: Storage Usage by Application Application: Record Keeping 10

11 Benefit: Light-weight footprint for deploying on the servers COMPARISON: TEAMQUEST VS. OTHER TOOLS (PROGNOSIS & TIVOLI) Approach: Deploy in Development, SIT, UAT prior to installing in Prod Show the overhead after 2 weeks of installation in Test environments A week of performance measurement data was reviewed. Presented a representative subset to the teams Results: Broker LPARs-Sensitive to CPU overhead and Memory Overhead In all cases TeamQuest used less processor than Prognosis, and averaged half as much as Tivoli In all cases TeamQuest used less memory than Prognosis, and averaged less than Tivoli 11

12 Benefit: Demonstrate Software Licensing Requirement Savings in Dev: Implement Virtual Pools for IBM Software Licensing (Broker) License Rules: For Capped Partitions: The highest actual level of Entitled Capacity (EC) EC was formerly referred to as PrU (Processing units) For uncapped Partitions: The Highest actual number of Online VP ( Virtual Processor Cores) Shared Pool Capping Rule: the lower of the sum of each partition for a product or the processor core capacity of the shared pool Aggregate fractional processor cores, apply shared pool capping rules, and round up at the server level to the next whole processor core Purchased 8 Required 5 Savings 3 The greater of what the partition starts with or the result of a DLPAR operation App Env Entitled vcpus AMH Dev/UAT AMH Dev/UAT EPP-GPP Dev/UAT EPP-GPP Dev/UAT EPP Dev/UAT EPP-GPP Dev/UAT Total Broker Licenses 8 12

13 Benefit: Leveraging TeamQuest for Performance Testing/ Engineering: Analyzing Broker Performance/Resource Footprint Broker CPU and Memory Usage is balanced across both LPARs- 2 VCPUs and 32 GB memory each Showing Memory Decomposition Total (Computational, Non-computational, unused) Activity ramped up at 17:19 13

14 Benefit: Leveraging TeamQuest for Performance Testing/ Engineering: Drilldown: Who is using the resources? Top 5 Processes/Workloads for CPU and Memory Top CPU Consumers: 1. EPP Common 2. EPP_RT1 3. EPP_RT1_EPF_01 4. EPP_RT2 5. EPP_RT2_EPF_01 Top Memory Consumers: 1. EPP_RT1 2. EPP_RT2 3. EPP_RT2_EPF_01 4. EPP_RT1_EPF_01 5. EPP_COMMON 14

15 Benefit: Leveraging TeamQuest for Production Support: Root Cause Analysis: Memory Leak Chart shows memory utilization increases over month Memory demands exceed Total system memory System was restarted Spike in memory usage on 3/18 Memory Demand exceeds capacity (24GB) System became unresponsive and was restarted The CULPRIT process suddenly increased its memory demand causing total memory to be exhausted. CULPRIT Process has a Memory LEAK Culprit 15

16 Our future-state vision Extract data from TeamQuest to generate Service-Based capacity reports through a common capacity reporting database to drive Leverage Surveyor to automate manual reports (exception, storage frame, etc.) Tie storage capacity to application consumption Future Report Definitions Capacity Management Maturity 16

17 TeamQuest Infrastructure 17

18 Thank you! 18