Session: 9172 Large Scale Linux Consolidation: IBM Moving it's Domino Infrastructure to Linux under z/vm

Transcription

1 Session: 9172 Large Scale Linux Consolidation: IBM Moving it's Domino Infrastructure to Linux under z/vm Mike Wojton Domino on System z Specialist - ATS

2 Legal Disclaimer IBM Corporation All Rights Reserv ed. The inf ormation contained in this publication is prov ided f or inf ormational purposes only. While ef f orts were made to v erif y the completeness and accuracy of the inf ormation contained in this publication, it is prov ided AS IS without warranty of any kind, express or implied. In addition, this inf ormation is based on IBM s current product plans and strategy, which are subject to change by IBM without notice. IBM shall not be responsible f or any damages arising out of the use of, or otherwise related to, this publication or any other materials. Nothing contained in this publication is intended to, nor shall hav e the ef f ect of, creating any warranties or representations f rom IBM or its suppliers or licensors, or altering the terms and conditions of the applicable license agreement gov erning the use of IBM sof tware. Ref erences in this presentation to IBM products, programs, or serv ices do not imply that they will be av ailable in all countries in which IBM operates. Product release dates and/or capabilities ref erenced in this presentation may change at any time at IBM s sole discretion based on market opportunities or other f actors, and are not intended to be a commitment to f uture product or f eature av ailability in any way. Nothing contained in these materials is intended to, nor shall hav e the ef f ect of, stating or imply ing that any activ ities undertaken by y ou will result in any specif ic sales, rev enue growth or other results. Perf ormance is based on measurements and projections using standard IBM benchmarks in a controlled env ironment. The actual throughput or perf ormance that any user will experience will v ary depending upon many f actors, including considerations such as the amount of multiprogramming in the user's job stream, the I/O conf iguration, the storage conf iguration, and the workload processed. Theref ore, no assurance can be giv en that an indiv idual user will achiev e results similar to those stated here. All customer examples described are presented as illustrations of how those customers hav e used IBM products and the results they may hav e achiev ed. Actual env ironmental costs and perf ormance characteristics may v ary by customer. IBM, the IBM logo, Lotus, Lotus Notes, Notes, Domino, Quickr, Sametime, WebSphere are trademarks of International Business Machines Corporation in the United States, other countries, or both. Uny te is a trademark of WebDialogs, Inc., in the United States, other countries, or both. UNIX is a registered trademark of The Open Group in the United States and other countries. Linux is a registered trademark of Linus Torv alds in the United States, other countries, or both. Other company, product, or serv ice names may be trademarks or serv ice marks of others. 2

3 Agenda A Brief History How did we get here Project Green ECM Domino Linux and Virtualization Overview of IBM's Domino Deployment Historical and Current Development/Deployment Strategy Migrations Current Status Applications Mail/Messaging Production Beta Linux guest Quick Snapshot Lotus Connections

4 Project 'Big Green' Major proof point for Project Big Green Double compute capacity with no increase in consumption or impact by 2010 IBM to reallocate $1 billion each year To accelerate green technologies and services To offer a roadmap for clients to address the IT energy crisis while leveraging IBM hardware, software, services, research, and financing teams To create a global green team of almost 1,000 energy efficiency specialists from across IBM Re-affirming a long standing IBM commitment Energy conservation efforts from have resulted in a 40% reduction in CO2 emissions and a quarter billion dollars of energy savings Annually invest $100M in infrastructure to support remanufacturing and recycling best practices IBM will consolidate and virtualize thousands of servers onto approximately 30 IBM System z mainframes Substantial savings expected in multiple dimensions: energy, software and system support costs The consolidated environment will use 80% less energy and 85% less floor space This transformation is enabled by the System z sophisticated virtualization capability 4

5 IT Organizations are Challenged by Operational Issues Challenges Costs & Service Delivery Business Resiliency & Security Rising costs of systems and networking operations Explosion in volume of data and information Difficulty in deploying new applications and services Security of your assets & your clients information Landslide of compliance requirements Systems and applications need to be available Energy Requirements Rising energy costs & rising energy demand Power & thermal issues inhibit operations Environmental compliance & governance mandates 5

6 IBM s Globally Integrated Enterprise Data Centers 6

7 IBM System z Linux Virtualization Progress Established phased approach for quick wins Migrated initial servers from early wave teams Thousands of servers inventoried Multiple successful migrations delivering benefits as expected Decommission pipeline of hundreds of servers for reuse or removal Comprehensive project plan and management system in place Integrated business priorities with transformational objectives Work in progress approach to maximize server migrations Pipeline, process, technical, finance and communications support Phase 1 Quick wins Phase 2 Highest Savings First Phase 3-n Finish Virtualization Developed internal business case (RACE*) Created detailed cash flow and labor analysis, migration expense, iterated Technical solution, education plan and operational plan developed Built upon IBM prior consolidation/simplification efforts, utilizing IBM offerings and capabilities Highest level of support from IBM senior executive team *Formerly zrace 7

8 IBM System z Linux Virtualization Progress IBM implementing New Enterprise Data Center through achievements in Server and storage virtualization Energy efficiency and resiliency improvements Benefits are on track with expectations Migration management key Business case is compelling Using System z10 technology, the number of machines could be cut by about half, with greater savings in energy, floor space, software and support costs Cumulative 5-Year Cost Comparison Distributed Cum z9 Cumulative Lessons Learned, including: Enterprise strategy and sponsorship needed to drive business case and execution Compelling business imperative accelerates execution and drives support Enterprise view of migration managed by waves drives experience; savings for investment 1 st Year 2 nd Year 3 rd Year 4 th Year 5 th Year IBM experience is driving Time to Value initiatives, integrated into IBM capabilities Dramatic reduction in labor through new processes supporting workload migrations Fall in/out analysis, working with business units, to close gaps in workload pipeline Piloting new testing strategy, processes & tools to automate 8

9 Virtualization Benefits are Significant; Migration Management is Key Expected Benefits of Virtualization Substantial savings in multiple dimensions: energy, software and system support costs 80% less energy, 85% less floor space for consolidated environment Improved inventory hygiene, including application to server mapping Dramatically faster provisioning Improved security and resiliency Higher quality through reduced complexity, increased stability and availability Large Scale Migration Challenges Exist Decision-making: Integrating Enterprise and Business Unit view Mindset/Culture related to distributed and mainframe worlds Workload selection - multidimensional nature of selection process Dated inventory records that are not centrally maintained Detailed data required for internal business case Project and program complexity integrating multiple priorities Clients are able to leverage IBM experience and capabilities to accelerate value 9

10 TCO: A Range of IT Cost Factors Often Not Considered Availability High availability Hours of operation Backup / Restore / Site Recovery Backup Disaster Scenario Restore Effort for Complete Site Recovery SAN effort Infrastructure Cost Space Power Network Infrastructure Storage Infrastructure Initial Hardware Costs Software Costs Maintenance Costs Additional development/implementation Investment for one platform reproduction for others Controlling and Accounting Analyzing the systems Cost Operations Effort Monitoring, Operating Problem Determination Server Management Tools Integrated Server Management Enterprise Wide Security Authentication / Authorization User Administration Data Security Server and OS Security RACF vs. other solutions Deployment and Support System Programming Keeping consistent OS and SW Level Database Effort Middleware SW Maintenance SW Distribution (across firewall) Application Technology Upgrade System Release change without interrupts Operating Concept Development of an operating procedure Feasibility of the developed procedure Automation Resource Utilization and Performance Mixed Workload / Batch Resource Sharing Integration Integrated Functionality vs. Functionality to be implemented (possibly with 3rd party tools) Balanced System Integration of / into Standards Further Availability Aspects Planned outages Unplanned outages Automated Take Over Uninterrupted Take Over (especially for DB) Workload Management across physical borders Business continuity Availability effects for other applications / projects End User Service End User Productivity Virtualization Skills and Resources Personnel Education Availability of Resources shared nothing vs. shared everything Parallel Sysplex vs. Other Concepts Response Time Performance Management Peak handling / scalability Routinely Assessed Cost Factors 10

11 Overview of IBM's Current Deployment Doing Business in 170 Countries 2,041 Locations 140,000 + Remote Employees 5 Mega Centers Mega Data Centers IBM Locations Mobile Employees 11

12 Overview of IBM's Current Deployment Over 385,000 Employees in 75 countries Europe 93,000 Americas 157,000 Asia Pacific 135,000 12

13 Overview of IBM's Current Deployment Over 800 Domino servers ~500 Mail Servers Majority are at 8.0.x Some beta/pilots at 8.5.x Approximately 300 Application Servers Predominantly at 8.0.x 70+ Domino domains Predominantly mail domains, typically one domain per country Applications in two domains Staging Production Servers located in 5 Mega Data Centers Worldwide 13

14 Overview of IBM's Current Deployment Mail Users by Domino Server Platform Platform System p, AIX System i, OS/400 Percentage of Users 78% 2% System z, zos 1% System z, Linux 19%

15 Overview of IBM's Current Deployment Entire messaging computing infrastructure moving to Linux on System z Part of Project Big Green Application servers went first- Completed in 4Q 2009 Mail is ongoing with different Geos at different states Not just Domino servers, but entire messaging infrastructure Designed to save Space Energy Money

16 Overview of IBM's Current Deployment Client Deployment Largely Microsoft Windows XP Over 30,000 Linux workstations Over 2,000 Apple Mac Over 200,000 on Notes 8.0.x Rest of IBM on Notes 7.+HF

17 Development Strategy Successful Product Deployment Lotus Everybody 1 enthusiastic beta group 1 heaping helping of cooperation 1 large set of early adopters 2 healthy doses of try again Combine all ingredients thoroughly, and simmer on medium heat for several months, stirring regularly. For best results, fold in a generous portion of humor.

18 Development Strategy Three basic stages of Deployment Beta deployment Beta code (pre-gold) Early adoption Gold code Full production deployment Beta code and Gold code Production Early Beta

19 Development Strategy Beta deployment Over 25,000 Notes Beta clients Diverse Beta population 19 Domino Beta servers Full production servers, not test servers Beta servers clustered with servers running Gold code Focus on quality and user experience Improved deployment experience for the Marketplace

20 Development Strategy Early adoption Gold code, Notes and 8.5 Target Beta users Focus Deployment Packaging for distribution User experience

21 Development Strategy Full production deployment Client began with Server began with 8.0 Versions chosen based on Early Adopter work, business needs

22 Development Strategy Deployment done on a rolling basis Always a Beta or early deployment project ongoing Current Production Deployment: Notes/Domino 7 and Current Early Adoption project: Notes 8.5 Current Beta project: Notes/Domino

23 Development Strategy IBM's broad internal deployment part of the development process Improves quality of shipping code Augments test lab efforts with real-world production deployment in a large, complex enterprise Widely adopted code translates to higher quality code for a broader set of environments

24 IBM's North America Domino Environment before Linux on System z Domino mail servers on AIX and z/os Approx. 160k mail users 19 primary mail servers on z/os (Poughkeepsie) 33 primary mail servers on AIX (Boulder) Multiple DPARs within one OS Image HUB servers on AIX Domino application servers on AIX and z/os Approx. 45k databases on over 100 Domino DPARs 32 AIX LPARs (Boulder and Poughkeepsie) 2 zos LPARs (Poughkeepsie) Multiple DPARs within one OS Image

25 A Brief Domino Structural Overview Domino Provides all of the services that it will utilizes for each platform Security Services Like RACF Transaction Manager Like CICS Database Manager Like DB/2 Programming Environment Like REXX Interface to 3GL/4GL languages C, C++, Java, Like LE Web Server Like Apache etc... Domino databases (nsf files) can be binary ftp'ed between Domino servers on any platforms and run unmodified. The same Domino administration client and tools are used to manage and monitor Domino servers on all platforms Common administrator Our same Notes Administrators managed both System p and System z DPARs But least common denominator, limited awareness to System z's +40 year history of management/diagnostic tools

26 Migrating to Linux on System z Linux guest configuration Application servers: 5CP and 10GB of memory per Linux guest 2GB of memory for each Domino DPAR 2GB of memory for Linux OS Note: Memory is for 32 bit Domino 8.0.x and not for 64 bit 8.5.x 4 Domino DPARs per guest DASD mixture of FICON and SAN Note: This is a starting point as application milage will vary! Domino Application Guest Domino None Appls. Domino DPAR DPAR Guest DPAR... IP Address 1 Ports for HTTP, SHTTP, NRPC, SMTP, DPAR zvm z/vm... IP Address n

27 Migrating to Linux on System z Linux guest configuration Mail servers: 6CP and 12GB of memory per Linux guest 6 to 12 IFLs per z/vm LPAR 2 mail servers per guest (to start) Milage will vary due to: Client type used User profile/active rates Third party products Features enabled Strong management archival policy Peak Loads/Time of year Guests can be mixed within one z/vm Domino Mail DPAR DPAR Domino Appls. DPAR DPAR zvm / None Domino Guest

28 Migrating to Linux for System z - Migration Strategy Applications Started with a 1-month pilot of 4 different application servers to measure performance and stability Domino Applications server DPARs migrated as-is Retained server hostname No change for end-user Data migrated via FTP during a maintenance window Potential Migration Issues Case sensitivity Platform dependent call (ex. dir c:) Standard calls versus non standard calls (LotuScript, VisualBasic, etc...) Platform dependency (BlackBerry servers Windows only)

29 Migrating to Linux for System z Migration Strategy Mail A Domino server consolidation was already done before migration to Linux for System z was started 32 bit Domino code Virtual Storage limitations limited registered users per DPAR DPARs targeted with about +3K registered users each Scaling Domino vertically instead of horizontally provides CPU savings IBM Lotus Domino, Linux, virtualization, scalability: No longer mutually exclusive 0.05 terms lotus/library/domino-linux/ Platform must be able to scale to handle Domino requirements, especially IO 0.01 Fewer DPARs means less to: 0 Manage Upgrade However, without good policies/standard/controls a server outage will affect more people CPU Seconds Per Active 15 Minute User 6 DPARs 3 DPARs

30 Migrating to Linux for System z Migration Strategy Mail Domino Mail server DPARs migrated like mass mail migration Temp server started on Linux for System z guest Temp server placed into Domino cluster with production server Mail file stubs placed on target and replicated over Cluster maintains/updates new server During maintenance window host name swap is done Mail server retains original name After validation period, original server removed from cluster/deleted With the Notes client, this is transparent to the end user

31 Current IBM NA Domino Environment on Linux for System z As of 1Q 2010 Domino under z/vm 45K databases/applications 54,7694 mail files 46,192 Primary Users Cluster Users 13 z/vm LPARs 8 Applications, 4 Mail Primary, 1 Mail Cluster 48 guests 8 primary mail, 3 cluster mail, 36 application, 1 test 140 Domino application DPARs, 16 primary mail servers, 6 cluster mail servers All mail server guests are running SLES10 Domino 8.0.2FP2 Applications servers guests are running SLES9 and SLES10

32 TSM Backups and Restores for Domino TSM servers have been migrated from AIX to Linux for System z for the Domino backups Required move to meet encryption requirements with adequate backup performance TSM runs in guest within a z/vm LPAR HiperSockets exploited for communications Backup are executed on the passive side of the Domino cluster Active D P A R 1 Domino cluster Passive D P A R 2 TSM Server

33 TSM Backups and Restores for Domino Currently 6 TSM servers 5 for mail and 1 for applications With encryption support 49,930 users Without encryption support 8,388 users On a typical Wednesday night 451 Domino backup jobs Multiple jobs run against each Domino Data is either staged to disk or written directly to tape for largest servers Data staged to disk can be written during the day to tape Tape compression/space reclamation performed during the day 20.1TB of data Domino data is not the only backups running in this environment (shared library)

34 Supporting Domino on Linux for System z Prior to migrating to Linux for System z we had servers on multiple platforms making support more challenging Consolidating to one platform brings the following benefits to the support team: Common tool set 1 OS team to interact with Highly scalable environment with Linux for System z Ability to handle the large IO requirements of Domino Since zlinux environment is very similar to AIX, our support team was able to easily grasp supporting Domino on Linux for System z Challenge is understanding all the layers of virtualization Other than fighting for Resources Hardware, People, and IBM Change Freeze(s) Implementation has been quite smooth

35 Quick view of our Beta Linux guest and two DPARs

36 Daily Messages Routed Almost ½ Million Messages per day

37 Cost per message Delivered

38 Domino Transaction Counts Between 35 to 45 Million Transactions per day

39 CPU Used by DPAR

40 Weekend Backups and Maintenance tasks have heavier IO than users Significantly higher write activity

41

42 Lotus Connections System z Environment Shared with 15 existing WAS applications on W3 All Servers running on SLES bit 4 WAS Nodes: WAS R bit Separate WAS install dedicated to LC apps Each LC app running in it's own 4-node WAS cluster All 11 LC apps installed (LC R ) 2 DB2 Servers DB2 R9.1 64bit (moving to R9.5 by March 2010) HA config in active-passive mode 4 HTTP Nodes Co-resident on same nodes as WAS app servers Leveraging existing w3 infrastructure for caching proxies and load balancing services Added resources to each existing server to handle Connections + 4 CPU's and 16GB memory for each WAS node + 12GB memory for each DB2 node

43 W3 Lotus Connections Next Steps Next phases of BluePages to Profiles Team with product development and focus on enhancing Profile edit capabilities Direct realtime sync from ED to Profiles Address new and unsatisfied IBM internal requirements Migrate data from other internal custom apps to LC BlogCentral --> LC Blogs Cattail --> LC Files WikiCentral --> LC Wikis IBM Forums --> LC Forums Tuning and ifixes through the year Planning for deployment of LC R3.0 beta and GA Assessment of performance and deployment to refine architecture & more

44 Thank You