WebSphere Application Server

Transcription

1 Test Infrastructure: Repeated system management and flexible management operations on a large scale environment under stress Author: Brian Hanczaryk Date: 2/25/2010

2 Disclaimer THE INFORMATION CONTAINED IN THIS REPORT IS PROVIDED FOR INFORMATIONAL PURPOSES ONLY. ALTHOUGH EFFORTS WERE MADE TO VERIFY THE COMPLETENESS AND ACCURACY OF THE INFORMATION CONTAINED IN THIS PRESENTATION, IT IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED. IN ADDITION, THIS INFORMATION IS BASED ON IBM S CURRENT PRODUCT PLANS AND STRATEGY, WHICH ARE SUBJECT TO CHANGE BYIBM WITHOUT NOTICE. IBM SHALL NOT BE RESPONSIBLE FOR ANY DAMAGES ARISING OUT OF THE USE OF, OR OTHERWISE RELATED TO, THIS REPORT OR ANY OTHER DOCUMENTATION. NOTHING CONTAINED IN THIS REPORT IS INTENDED TO, OR SHALL HAVE THE EFFECT OF CREATING ANY WARRANTY OR REPRESENTATION FROM IBM (OR ITS AFFILIATES OR ITS OR THEIR SUPPLIERS AND/OR LICENSORS); OR ALTERING THE TERMS AND CONDITIONS OF THE APPLICABLE LICENSE AGREEMENT GOVERNING THE USE OF IBM SOFTWARE. 2

3 Contents Trademarks and Service Marks Executive Summary Audience Scenario Description Topology Applications Automation Installation and Configuration Procedures Results 3

4 Trademarks and Service Marks The following terms are trademarks of the IBM Corporation in the United States or other countries or both: IBM WebSphere WebSphere Application Server Java and all Java-based trademarks and logos are trademarks or registered trademarks of Sun Microsystems, Inc. in the United States and other countries. Other company, product and service names may be trademarks or service marks of others. 4

5 Executive Summary The goal of this document is to outline the steps by which the WebSphere System Verification Test (SVT) team installed, configured, and tested the repeated system management and flexible management operations on a 300 app server environment under stress during the alpha timeframe. After reading this test report, you should gain a better understanding of how WebSphere SVT tested the following : Creating large WebSphere topologies with hundreds of application servers and applications. Performing repeated administrative operations such as app deployment in a large topology. Managing multiple deployment managers asynchronously through job manager. 5

6 Audience This document is intended for administrators, testers and others who desire an understanding of the type of testing performed prior to the release of the WebSphere Application Server v alpha. 6

7 Persona Description The WebSphere Application Server test team has defined several customer personas that describe fictitious types of customers with unique, but consistent patterns of WebSphere usage. The test described here simulated the activities for one large topology environment under our Norman BigIron (NBI) company persona. Norman BigIron is a fictitious insurance company and this particular test has many simple applications deployed in a large environment. NBI typically deploys WebSphere Application Server on distributed platforms such as Linux, Solaris, Windows, and AIX. Workload management is obtained through the use of large clusters of application servers fronted by web servers. One or more applications may be deployed on each cluster. Internally, NBI is a risk averse organization. They value stability and maintenance of the status quo over chasing new technological solutions. NBI is also focused on improving their operating efficiency. They've used scripting and automation to standardize as many of their system management and operational functions as possible. 7

8 Scenario Description This scenario is designed to test repeated management of a large scale enterprise topology. Management operations centered around continuous deployment of applications, continuous job submissions, and concurrent administrative console access. The WebSphere environment encompasses 31 Network Deployment (ND) cells, all of which are registered to a single job manager. One of the ND cells was large and consists of 7 nodes with 50 clusters and 300 application servers. The other 30 ND cells consist of a deployment manager (dmgr) and a single node with up to two application servers. 8

9 Topology Large ND Cell The Large ND Cell environment included: IBM WebSphere Application Server Deployment Manager v alpha Seven IBM WebSphere Application Server v alpha nodes IBM DB2 Enterprise Server Edition v9.5 fixpack 1 Sun Java System Directory Server v6.0 9

10 Topology Details Large ND Cell Machine OS # of CPUs CPU Speed CPU Type RAM (GB) Function Alpha V WebSphere Deployment Manager Red Hat Enterprise Linux AS release 5.4 (x86_64) Ghz X86_64 4 Deployment manager Alpha V WebSphere Application Servers Microsoft Windows Server 2003 R2 Enterprise x64 Edition GHz X86_64 32 Node agent + servers Alpha V WebSphere Application Servers Red Hat Enterprise Linux AS release 5.4 (x86_64) GHz X86_64 32 Node agent + servers Alpha V WebSphere Application Servers Red Hat Enterprise Linux AS release 5.4 (x86_64) GHz X86_64 32 Node agent + servers Alpha V WebSphere Application Servers Red Hat Enterprise Linux AS release 5.4 (x86_64) GHz X86_64 32 Node agent + servers Alpha V WebSphere Application Servers Red Hat Enterprise Linux AS release 5.4 (x86_64) GHz X86_64 32 Node agent + servers Alpha V WebSphere Application Servers Red Hat Enterprise Linux AS release 5.4 (x86_64) GHz X86_64 32 Node agent + servers Alpha V WebSphere Application Servers Red Hat Enterprise Linux AS release 5.4 (x86_64) Ghz X86_64 4 Node agent + servers Red Hat Enterprise Linux IBM DB2 v9.5 FP1 Server release GHz PPC64 16 Sun Java System Directory Server v6.0 Trade Database GS60 Database Solaris GHz Sparcv9 4 LDAP 10

11 Topology Deployment manager server farm environment The W ebsphere deployment manager server farm environment included: IBM WebSphere Application Server job manager v alpha IBM WebSphere Application Server deployment manager v alpha with one node and up to two application servers (30 of these smaller ND cells) Sun Java System Directory Server v6.0 11

12 Topology Details Deployment manager server farm environment Machine OS # of CPUs CPU Speed CPU Type RAM (GB) Function Alpha V WebSphere Job Manager Alpha V WebSphere Deployment Manager + Application Servers Alpha V WebSphere Deployment Manager + Application Servers Sun Java System Directory Server v6.0 Windows 2003 Ent. X64 Ed SP2 Red Hat Enterprise Linux AS release 5.4 (x86_64) Red Hat Enterprise Linux AS release 5.4 (x86_64) GHz X86_64 8 Job manager Ghz X86_ Ghz X86_ Deployment managers + application servers 15 Deployment managers + application servers Solaris GHz Sparcv9 4 LDAP 12

13 Applications DayTrader is benchmark application built around the paradigm of an online stock trading system. Originally developed by IBM as the Trade Performance Benchmark Sample, DayTrader was donated to the Apache Geronimo community in The application allows users to login, view their portfolio, lookup stock quotes, and buy or sell stock shares. With the aid of a Web-based load driver such as Mercury LoadRunner, Rational Performance Tester, or Apache JMeter, the realworld workload provided by DayTrader can be used to measure and compare the performance of Java Platform, Enterprise Edition (Java EE) application servers offered by a variety of vendors. In addition to the full workload, the application also contains a set of primitives used for functional and performance testing of various Java EE components and common design patterns. DayTrader is built on a core set of Java EE technologies that includes Java Servlets and JavaServer Pages (JSPs) for the presentation layer and Java database connectivity (JDBC), Java Message Service (JMS), Enterprise JavaBeans (EJBs) and Message-Driven Beans (MDBs) for the back-end business logic and persistence layer. 13

14 Automation Python automation scripts which invoke silent install were used to install WebSphere binaries, create profiles and federate nodes. Jython scripts were used for further configuration. Enable LDAP security Mapping additional users to roles Creating resources Creating clusters Installing applications Tuning JVM parameters Creating coregroup configurations 14

15 Detailed Installation and Configuration These steps assume the databases, the LDAP, and the web server you are using are already installed and configured on separate machines. 2 Install WebSphere Application Server Network Deployment (ND) binaries on each machine to be used for WebSphere deployment manager and WebSphere application server nodes. 3 Create required profiles on a each machine using the WebSphere Application Server ND for your platform. Create job manager profiles on the designated machine. Create deployment manager profile on the machine designated for the large ND cell WebSphere deployment manager. Create twelve node profiles on the machines designated for the large ND cell WebSphere application servers. Create 30 deployment manager & node profile pairs on the machine(s) designated for WebSphere deployment manager and application servers in the deployment manager server farm. 4 Enable LDAP security for job managers and deployment managers. Start the required job manager and deployment manager processes. Enable LDAP using the Sun Java System Directory Server v6.0 for the job manager and the deployment manager used for the large ND cell. Enable LDAP using the Sun Java System Directory Server v6.0 for the 30 deployment managers used for the deployment manager server farm. 5 Register nodes with their appropriate deployment manager. Register the twelve nodes intended for the large ND cell to the intended deployment manager. In the deployment manager server farm, register the 30 nodes with their corresponding deployment manager. 6 Register deployment managers with job managers using the AdminTask.registerWithJobManager() command. After registering all deployment managers with job managers, there were 31 deployment managers registered. After registering the deployment managers, created a group consisting of all the deployment managers for use with job submission. Changed the polling interval from default of 30 seconds to 90 seconds. 7 Set JVM parameters for job managers, deployment managers, node agents, and future servers through scripting. For all JVMs, set the JVM log size to 100Mb with one rollover. Set IBM_HEAPDUMP=true and IBM_HEAPDUMP_OUTOFMEMORY=true. Enabled verbosegc. For job managers and deployment managers, set the maximumheapsize to For node agents, set the maximumheapsize to 768. For future app servers, set the maximumheapsize to 512. Set the maximumsize for WebContainer threadpool to 1000 for future TradeCluster and GS60Cluster servers. 15

16 Detailed Installation and Configuration Create clusters and servers for dmgr server farm through scripting. For the each of the 30 small ND cells in the dmgr server farm, created a single cluster with one application server. 9 Create cluster, servers and install DayTrader application for the large ND cell using scripting. The first cluster created on the large ND cell was TradeCluster consisting of one cluster member from each of the first six nodes. Using the scripting provided with the DayTrader application, created the cluster and application resources which include JAAS authentication alias, JDBC Provider, Datasource, JMS resources, and SIBus. Also installed the DayTrader application on the cluster. 10Create cluster, servers and install GarageSale 6.0 application for the large ND cell using scripting. The second cluster created on the large ND cell was GS60Cluster consisting of one cluster member from each of the first six nodes. Using the scripting, created the cluster and application resources which include JAAS authentication alias, JDBC Provider, Datasource. Following resource creation, installed the GarageSale 6.0 application on the cluster. 11Create cluster, servers and install Sample applications for the large ND cell using scripting. The third cluster created on the large ND cell was SamplesCluster consisting of one cluster member from each of the first six nodes. Using the scripting, created the cluster and application resources which include JDBC Provider, Datasource. On the Samples cluster, installed the following applications: SamplesGallery, PlantsByWebSphere, JaxWSServicesSamples. 12Create proxy. In the large ND cell, create proxy server on the seventh node. 13Create 47 clusters and 282 servers for the large ND cell using scripting. Each cluster consists of members from each of the first six nodes. 14Organize clusters into coregroups for large ND cell. Create 11 coregroups. The TradeCluster, GS60Cluster, and SamplesCluster were moved to coregroup 1. The other 47 clusters and their members created in the previous step were moved into coregroups After moving all clusters into the specified coregroup, perform a Full Resynchronization through the administrative console. 15Create CoreGroupAccessPoint (CGAP_1) and bridge interfaces for large ND cell. Under the seventh node, create application servers for DefaultCoreGroup and CGAP_1 for use as bridge interfaces and move to designated coregroup. 16

17 Detailed Installation and Configuration Map additional users to various Admin roles for the large ND cell. Using scripting, AdminTask.mapUsersToAdminRole(), create two additional users with Administrator role. Also, create two additional users with Configurator/ Operator role. 17Add any needed virtual host alias for default_host in the large ND cell. To account for the ports used for the TradeCluster, GS60Cluster, and SamplesCluster servers, added a HostAlias of * for the ports used by these appservers. By default, this was 9080 for TradeCluster servers, 9081 for GS60Cluster servers, and 9082 for SamplesCluster servers. 18Restart deployment manager and node agents in the large ND cell. 19Start application clusters in the large ND cell. Start TradeCluster, GS60Cluster, and SamplesCluster. Ensure that DayTrader, GarageSale 6.0, and PlantsByWebSphere applications are functioning properly before proceeding. 17

18 Alpha scalability improvement Application server creation port conflict resolution improvements were made to the v8 alpha which allows for faster creation of applications servers. This improvement is more noticeable as the number of application servers in the cell increase. This v8 alpha testing experienced a 22% reduction in the time to create and save 300 JVMs to the configuration versus the v7005 on the same hardware. 18

19 Procedures To test the configuration, we: Performed continuous deployment of applications Performed continuous job submission Drove application load Tested concurrent administrative console access Monitored system activity Gathered and analyzed logs Detailed steps to test the configuration are shown in the charts that follow. 19

20 Detailed Procedures Begin continuous deployment of applications scripting for the large ND cell. Continuous deployment of applications used the Query, School, IVT, and DefaultApplication applications and performed the following operations for a period of 24 hours : Randomly pick one of the 47 clusters If application is running on the cluster, then it stops the applications. If application is installed on the cluster, then it uninstalls the applications. Install the application. Perform node synchronization with each node. If cluster members are stopped, then start the cluster members. Start the application 2 Begin continuous job submission for the job managers. Utilizing scripting, executed a general job flow against the deployment manager server farm endpoints. The sequence below was continually run sequentially against the deployment manager server farm for a period of over 24 hours. In addition to the continuous job submissions, various other jobs were executed against the group of all 31 deployment managers. Continuous job submission sequence number one was the following : Distribute files (jython and ear) from job manager to dmgr RunWsadminScript to create application resources create cluster and cluster members. Start cluster Install DayTrader 2.0 application Start application Stop application Stop cluster Uninstall DayTrader 2.0 application Delete cluster RunWsadminScript to cleanup resources Remove files (jython and ear) that were previously distributed to dmgr. 20

21 Detailed Procedures Drive application load to applications in large ND cell. While the continuous deployment of applications and continuous job submissions are executing, used internal load generation tool to drive application load to DayTrader, GarageSale6.0, and PlantsByWebSphere applications for a period of 24 hours. DayTrader and GarageSale 6.0 were configured to use DB2 backend databases and PlantsByWebSphere utilizes Derby. Used built-in OS tools such as vmstat and top to monitor memory/cpu utilization. 4 Test concurrent administrative console access with different users/roles in the large ND cell. During the application load, continuous deployment of applications, and continuous job submissions, test concurrent administrative console access with the two administrative users and two configurator/operator users detailed in earlier steps. With the intent to have all four users access the administrative console at the same time and perform a variety of operations, such as : Start/stop clusters and servers not experiencing load Node synchronization Application modifications JDBCProvider/Datasource modifications Edit tracing for deployment manager, node agents, and application servers 21

22 Detailed Procedures Periodically monitor WebSphere machines. In particular, focus on the following : Check CPU on application server machines Check SystemOut.log and SystemErr.log for exceptions/failures Check verbosegc output for memory leaks Took interim thread/heapdumps on large ND cell deployment manager and used IBM HeapAnalyzer to review for heap growth. 6 Gather and analyze logs after 24 hour run of application load, continuous deployment of applications, and continuous job submission. Review SystemOut.log and SystemErr.log for exceptions/failures. Used verbosegc monitoring tool IBM PMAT to analyze the verbosegc logs for any memory leaks over the 24 hour run. Used HeapAnalyzer tool to review the interim heapdumps which were generated. 22

23 Tooling WebSphere Application Server Performance Tools ISA 4.1 IBM PMAT downloaded from alphaworks HeapAnalyzer from alphaworks Internal load generator tool 23

24 Results Conclusion and Limits After reading this test report, you should have a better understanding of how WebSphere SVT tested the following during the v alpha timeframe : Creating large WebSphere topologies with hundreds of application servers and applications. Performing repeated administrative operations such as app deployment in a large topology. Managing multiple deployment managers asynchronously through job manager. High level limits Core group recommendation is up to 50 members, although hardware specifications may allow for variation. Larger core groups result in additional communication overhead. Cluster sizes are bound by the core group limit as all cluster members must be in the same core group. Job manager polling interval, in seconds, should be at least 1.5 times the number of registered endpoints. Smaller polling intervals will result in higher CPU for job manager. 24

25 Results Tuning Maximum Heap Size 1024 for job manager and deployment manager 768 for nodeagents 512 for application servers JVM log file rollover size 100 Mb for SystemOut.log and SystemErr.log 200 Mb for trace.log JVM custom properties IBM_HEAP_DUMP=true IBM_HEAPDUMP_OUTOFMEMORY=true Verbose garbage collection enabled WebContainer Thread Pool MaximumSize = 1000 MinimumSize = 50 InactivityTimeout = Soap timeout value In properties/soap.client.props, com.ibm.soap.requesttimeout=

26 Results sample dmgr SystemOut.log contents [2/25/10 10:55:35:607 EST] 00003ea8 AppBinaryProc I ADMA7021I: Distribution of application Query_12 completed successfully. [2/25/10 10:55:35:618 EST] 00003e86 FileRepositor A ADMR0015I: User k04sun06.rtp.raleigh.ibm.com:689/user1 created document cells/testcell/applications/query_12.ear/deployments/query_12/querybean.jar/meta-inf/ibm-ejb-jar-bnd.xmi. [2/25/10 10:55:35:619 EST] 00003e86 FileRepositor A ADMR0015I: User k04sun06.rtp.raleigh.ibm.com:689/user1 created document cells/testcell/applications/query_12.ear/deployments/query_12/meta-inf/ibm-application-bnd.xmi. [2/25/10 10:55:35:619 EST] 00003e86 FileRepositor A ADMR0015I: User k04sun06.rtp.raleigh.ibm.com:689/user1 created document cells/testcell/applications/query_12.ear/deployments/query_12/meta-inf/manifest.mf. [2/25/10 10:55:35:620 EST] 00003e86 FileRepositor A ADMR0015I: User k04sun06.rtp.raleigh.ibm.com:689/user1 created document cells/testcell/applications/query_12.ear/deployments/query_12/querybean.jar/meta-inf/ibm-application-bnd.xmi. [2/25/10 10:55:35:620 EST] 00003e86 FileRepositor A ADMR0015I: User k04sun06.rtp.raleigh.ibm.com:689/user1 created document cells/testcell/applications/query_12.ear/deltas/query_12/delta [2/25/10 10:55:35:620 EST] 00003e86 FileRepositor A ADMR0015I: User k04sun06.rtp.raleigh.ibm.com:689/user1 created document cells/testcell/cus/query_12/cver/base/cu.xml. [2/25/10 10:55:35:620 EST] 00003e86 FileRepositor A ADMR0015I: User k04sun06.rtp.raleigh.ibm.com:689/user1 created document cells/testcell/applications/query_12.ear/deployments/query_12/meta-inf/ibm-application-ext.xmi. [2/25/10 10:55:35:621 EST] 00003e86 FileRepositor A ADMR0015I: User k04sun06.rtp.raleigh.ibm.com:689/user1 created document cells/testcell/cus/query_12/cver/base/cu-ref.xml. [2/25/10 10:55:35:621 EST] 00003e86 FileRepositor A ADMR0015I: User k04sun06.rtp.raleigh.ibm.com:689/user1 created document cells/testcell/applications/query_12.ear/deployments/query_12/querybean.jar/meta-inf/application.xml. [2/25/10 10:55:35:621 EST] 00003e86 FileRepositor A ADMR0015I: User k04sun06.rtp.raleigh.ibm.com:689/user1 created document cells/testcell/applications/query_12.ear/deployments/query_12/querybean.jar/meta-inf/ejb-jar.xml. [2/25/10 10:55:35:622 EST] 00003e86 FileRepositor A ADMR0015I: User k04sun06.rtp.raleigh.ibm.com:689/user1 created document cells/testcell/applications/query_12.ear/deployments/query_12/querybean.jar/meta-inf/ibm_ejbext.properties. [2/25/10 10:55:35:622 EST] 00003e86 FileRepositor A ADMR0015I: User k04sun06.rtp.raleigh.ibm.com:689/user1 created document cells/testcell/applications/query_12.ear/deployments/query_12/deployment.xml. [2/25/10 10:55:35:622 EST] 00003e86 FileRepositor A ADMR0015I: User k04sun06.rtp.raleigh.ibm.com:689/user1 created document cells/testcell/applications/query_12.ear/deployments/query_12/meta-inf/application.xml. [2/25/10 10:55:35:623 EST] 00003e86 FileRepositor A ADMR0015I: User k04sun06.rtp.raleigh.ibm.com:689/user1 created document cells/testcell/applications/query_12.ear/query_12.ear. 26

27 References WebSphere Application Server 7.0 InfoCenter Main URL: For details of Jython scripting, see the article Jython (cxml_jython) found at: topic=/com.ibm.websphere.nd.doc/info/ae/ae/cxml_jython.html Best Practices for Large WebSphere Topologies Tactics and Tradeoffs in a large shared topology Techniques for Managing Large WebSphere Installations 27

28 End WebSphere Application Server 28