Robust QAD Infrastructure Leveraging VMware and Netapp. Sekhar Athmakuri IT Manager

Size: px

Start display at page:

Download "Robust QAD Infrastructure Leveraging VMware and Netapp. Sekhar Athmakuri IT Manager"

Meagan Blankenship
6 years ago
Views:

1 Robust QAD Infrastructure Leveraging VMware and Netapp Sekhar Athmakuri IT Manager

2 Agenda Tower International Overview QAD deployments at Tower QAD infrastructure upgrade and virtualization DR Architecture Additional application and infrastructure upgrades 2

Tower International at a glance Tower International is a leading

components and assemblies primarily serving automotive original

Headquarters: Livonia, Michigan, USA Locations: Our products are

our customers in North America, South America, Europe and Asia.

3 Tower International at a glance Tower International is a leading integrated global manufacturer of engineered structural metal components and assemblies primarily serving automotive original equipment manufacturers Revenue: 2.1 Billion Employees: 9000 Corp. Headquarters: Livonia, Michigan, USA Locations: Our products are manufactured at 29 production facilities strategically located near our customers in North America, South America, Europe and Asia. We support our manufacturing operations through eight engineering and sales locations throughout the world. 3

4 QAD at Tower QAD deployed globally at all Tower locations QAD infrastructure centralized at our USA based Global Data Center QAD versions deployed at Tower include: MfgPro 9.0 / Progress 9.1E EB2 (various SP levels) / Progress 9.1E QAD 2008 SE / Progress 10.1.C QAD 2010 SE with.net / Progress 10.2.B Total QAD users around

5 Pre virtualization QAD at Tower MfgPro 9.0 and EB2 run centrally from USA based Data Center Infrastructure: HP PARISC servers running HPUX 11.x Obsolete hardware. Slow performance. No hardware fault tolerance. Risk of single server failure bringing down QAD for an entire region. EMC SAN storage Costly and complex outsourced storage solution DR (Disaster Recovery) solution with 2 day recovery time from tape backups Recovery time too long Solution does not scale well Costly outsourced solution 5

6 Infrastructure upgrade project Project started in 2010 with the following objectives: 1. Upgrade QAD server infrastructure (replace obsolete hardware) 2. Improve QAD performance (5X) 3. Replace outsourced SAN storage solution (reduce cost and complexity) 4. Implement scalable (across multiple applications including QAD) DR solution with a recovery time objective of 4 hours or less 6

7 New Solution Key infrastructure changes Migrate from HPUX to Redhat Linux Linux made hardware upgrade possible without requiring a QAD application upgrade Utilize latest Intel X86 CPUs for improved performance Virtualize servers on VMware Provide hardware fault tolerance without the complexity Simplify future QAD upgrades. New servers to test upgrades can be setup easily without having to purchase new hardware. Server portability for future Data Center move Migrate from EMC SAN storage to Netapp NAS storage Reduce cost and complexity of storage solution Already proven technology at Tower New DR solution based on VMware virtualization and Netapp storage replication technologies QAD servers and databases to be replicated to DR site 7

8 New VMware/Netapp Infrastructure 8

9 Migration to Linux Key Success Factors Redhat Enterprise Linux 4 with increased CPU and Memory capacity Increased DB cache for improved performance Single tier QAD/Progress architecture. Client and DB on a single server. This later allowed for self client mode to boost performance. QAD databases, binaries, users on NFS version 3 TCP mounts. Simplified storage management. Leveraged Bravepoint s Pro dump/load utility to minimize outage windows during migrations 9

10 Migration to Linux Contd. Key Benefits LDAP (AD) authentication for users Improved performance (5X) Several large operations including MRP runs saw over 5X speed improvement. (MRP runs went from around 50min to around 10min) Larger DB Cache, Large read cache on Netapp and faster CPUs Lessons Learned Code compilation issues. Source code for some custom programs could not be properly identified. Better source code management was needed. Slow telnet performance on Linux especially for regions outside USA. Nagel algorithm had to be disabled (set NODELAY flag in /etc/xinetd.d/telnet) to fix telnet performance. Direct printing to Korea printers did not work in Linux. NetTerm (Terminal emulation software) print to local functionality was used as a workaround. 10

11 Virtualization of QAD / Progress Key success factors: VMware Vsphere 4 cluster with Intel Nehalem CPUs optimized for virtualization No over subscription of cpu/memory resources to ensure consistent performance Simplified VMware configuration with NFS volumes for data stores Separate gigabit network dedicated for storage access VMware best practices including Jumbo frames (MTU 9000) Key Benefits: VMware snapshots: Facilitates simple back out from system changes and recovery from OS corruption. Automatic load balancing of virtual servers across hosts Ease of virtual server resource (cpu/mem) changes and new server deployments Cloning: Test servers could be easily cloned from production ensuring same OS configuration and patches on test. 11

12 Virtualization of QAD / Progress contd. Lessons learned: Vmotion of systems with large amounts of memory was disruptive to systems being vmotioned. Setup rules to pin systems with large memory to specific hosts in order to prevent automatic relocation. Did not impact hardware fault tolerance. IP hash based load balancing across 4 gigabit ports on VMhost storage network was not optimal. 95% of the traffic stayed on a single gigabit port. Storage NICs on the VMhosts were shared for both storage traffic as well as vmotion traffic. This caused contention at times. 12

13 Netapp Storage Key success factors Dual FAS 3160 Storage controllers for high availability Netapp sizing based on peak IOPS usage on EMC storage 250GB of flash based read cache per controller High performance 15K RPM fiber channel disks with large stripe size Ether channeled storage network interface with 8 gigabit ports Netapp best practices including jumbo frames (9000 MTU) on storage interfaces 13

14 Netapp Storage contd Key benefits Eliminated frequent dump and loads for QAD databases Old infrastructure required frequent (multiple times a year) dump/loads due to poor performance Performance has been good and consistent since the upgrade minimizing the need for dump/loads Snapshots: easily create logical snapshots of volumes for backup and recovery Dedupe: Eliminates duplicate data blocks on volumes. Saw a 35% space savings on QAD DB volumes NFS and CIFS access to same volume. Eliminated the need for users to FTP to QAD servers to access print files. Lessons learned IO inefficiencies due to misaligned Windows 2003 and Redhat Linux 4/5 virtual servers. Administrative correction needed to fix the issue 14

15 DR Architecture 15

16 QAD Disaster Recovery Key network design considerations: Different system IP addresses in DR Employ DNS to manage IP change Fenced DR environment to protect production environment during DR testing Access to DR environment controlled via access lists on Cisco layer 3 switch Optimize replication traffic between Data Center and DR site Over 80% optimization achieved with Riverbed WAN accelerators Adequate network bandwidth to replicate daily changes between Data Center and DR site 45Mbps line used at DR site 16

17 QAD Server/Data Replication Process Replication Process: Daily replication of all QAD servers and ancillary systems to DR site using Netapp SMVI utility Swap partitions are on separate volumes and are not replicated on a daily basis Daily QAD DB Replication using Netapp SnapMirror technology: Proquiet QAD DBs and take a snapshot (takes a few seconds) Release QAD DB to normal operation Replicate snapshot copy of DB to DR site Only data blocks that changed since previous day s snapshot are replicated 17

18 QAD DR Startup and Testing DR Startup process: VMware SRM (Site Recovery Manager) used to automate the following: Change IP addresses of all servers Change DNS entries to reflect new IPs Change DB mounts on all QAD servers to reflect DR Netapp Create a logical copy (flex clone) of DR volumes Start all systems (Entire process takes less than 4 hours) DR Testing: All QAD environments are validated annually Logical copies of volumes are discarded after testing is completed. No impact to production systems or replicated data in DR 18

19 Key accomplishments of this project 5X QAD performance improvement Reduced storage cost and complexity with Netapp storage Less than 2 year return on investment based on outsourced storage expense savings Eliminated frequent database dump/loads saved 0.5 FTE (Fulltime Equivalent) Automatic hardware failure protection with VMware Standby scalable (across multiple apps) DR solution with a less than 4 hour system recovery time. 19

20 QAD 2010SE multi-domain virtualization Server environment: Redhat 5 64bit, 8 vcpus and 64GB of RAM. 10 NA plants were migrated from a MfgPro 9.0 environment to a QAD 2010SE multi-domain environment System performance was good until a large plant came on board Linux system load frequently exceeded 80 resulting in severe performance degradation High cpu utilization Resolution steps: Implemented a larger server with 24 vcpus and 128GB of RAM System load issues were resolved after enabling -q parameter (only reads code once) in client connections Performance was significantly improved after implementing self client mode Corrected several custom programs that were consuming large amounts of cpu due to indexing issues 20

21 Minutes MRP (23.2 Full Regen) Run time improvements 36:00 Post Upgrade 28:48 21:36 14:24 auburn bardstown chicago clinton elkton madison meridian ohio plymouth smyrna 07:12 00:00 4/9/2012 4/10/2012 4/11/2012 4/12/2012 4/13/2012 4/14/2012 4/15/2012 4/16/2012 4/17/2012 4/18/2012 4/19/2012 4/20/

22 Netapp and VMware upgrades 10Gbit storage network Simplified storage network due to reduced number of ports Much faster Vmotion speeds. No impact on systems with large amounts of memory during Vmotion. VMware upgrade to Vsphere 5 along with distributed virtual switch implementation Eliminated VMhost load balancing inefficiencies across multiple NICs Upgraded Netapp to FAS 3250 cluster SSDs used for read/write caching Aggregate / Volume level caching control with SSD cache 22

23 Questions? 23

EMC VSPEX SERVER VIRTUALIZATION SOLUTION

Reference Architecture EMC VSPEX SERVER VIRTUALIZATION SOLUTION VMware vsphere 5 for 100 Virtual Machines Enabled by VMware vsphere 5, EMC VNXe3300, and EMC Next-Generation Backup EMC VSPEX April 2012