How WLM routing and HA Manager work together in WebSphere Application Server ND

Transcription

1 IBM Software Group How WLM routing and HA Manager work together in WebSphere Application Server ND Krishna Jaladhi Kumaran WebSphere Application Server Level 2 support May WebSphere Support Technical Exchange Click to add text

2 Agenda WLM Overview Clusters HA Basic WLM routing Logic How nodeagent involved in WLM How ORB involved in WLM Why bridging Coregroups Request Flow Important Custom Property Common issues WebSphere Support Technical Exchange 2

3 WLM Overview WebSphere Support Technical Exchange 3

4 WLM WorkLoad Management (WLM) is a WebSphere Application Server (WAS) facility that provides load balancing, fail over and affinity between application servers in a WebSphere clustered environment. Types of Workload Management (WLM) in WebSphere Application Server HTTP Server Plug-In WLM Enterprise Java Bean (EJB ) WLM EJB WLM balances WLM enabled RMI/IIOP requests between clients and clusters JNDI Lookups, EJB creates, EJB business methods and EJB removes WebSphere Support Technical Exchange 4

5 WLM - HTTP Server Plug-In Detects Failure Marks Container as Unavailable Tries Next Server in ServerCluster Round Robin or Random for Web Server Web Container Servlet Requests HTTP Server Plug-in HTTP(S) Protocol Traffic App Server Web Container App Server WebSphere Support Technical Exchange 5

6 WLM EJB Client ORB Plug-in Detects Failure Marks Container as Unavailable Tries Next Server in ServerCluster Weighted Round Robin Request Distribution WebSphere Support Technical Exchange 6

7 Clusters WebSphere Support Technical Exchange 7

8 Clusters Clusters are sets of servers that are managed together and participate in workload management. Clusters also enable enterprise applications to be highly available because requests are automatically routed to the running servers in the event of a failure. The servers that are members of a cluster can be on different host machines. A cell can include no clusters, one cluster, or multiple clusters. Servers that belong to a cluster are members of that cluster set and must all have identical application components deployed on them. WebSphere Support Technical Exchange 8

9 Types of clusters Vertical Cluster A vertical cluster has cluster members on the same node, or physical machine. Horizontal Cluster A horizontal cluster has cluster members on multiple nodes across many machines in a cell. You can configure either type of cluster, or have a combination of vertical and horizontal clusters WebSphere Support Technical Exchange 9

10 WLM Routing Policy WebSphere Support Technical Exchange 10

11 WLM Routing Policy Routing is based on weights associated with cluster members. Round robin algorithm used when weights are equal Weights can be modified to send more requests to a particular cluster member or members More information about WLM routing from the V8.5 Knowledge Center: If the EJB client is on the same physical box as the cluster member, the Prefer Local setting will ensure that all requests from the client go to the local cluster member. WebSphere Support Technical Exchange 11

12 WLM Routing Policy Cont.. Change the configuration and runtime weights by using the Update button Weights are only meaningful when they are compared to the other member weights. Make Idle button sets the selected member s configuration and runtime weight to 0 WebSphere Support Technical Exchange 12

13 HA Basics WebSphere Support Technical Exchange 13

14 HA Basics IBM Software Group The HAManager Framework (HAM) An integral part starting WAS* 6.0 (and WAS XD) designed to provide an infrastructure for making selected WAS services highly available. Present in all JVM s including Deployment Manager and Node Agents Can be used by other internal WebSphere components to provide automatic failover support. Significant only in ND** environments HAM provides no extra value for Base or any configuration consisting of a single server. Provides various asynchronous callbacks for interested internal WAS components. The Configuration of High Availability Systems is Simplified Works out of the box in most cases - No additional administration required for most commonly used topology *WAS WebSphere Application Server ** ND Network Deployment WebSphere Support Technical Exchange 14

15 HA Basics IBM Software Group Four basic HAManager services Bulletin Board (BB) A server state data exchange mechanism Typically used for carrying routing information (WLM and ODC) High Availability Groups (HAGroups) Provides the HAManager policy based fail-over mechanism Users : ME, Transaction Manager, JCA Adapter and more The most visible service Agent Framework Provides hot backup service from primary to N backup members High throughput service used by DRS for data replication Partitioned Managed Group (PMG) A distributed state manager (conceptually a distributed cache). Used by the core group bridge service to forward bulletin board communication between core groups WebSphere Support Technical Exchange 15

16 CoreGroup A core group is a collection of processes (JVM s) A core group is statically defined in the cell-scoped coregroup.xml configuration file A cluster can not span across core groups All of the cluster members must belong to the same core group A core group can have more than one clusters A core group can not span across cells All the members of a core group must belong to same cell A cell can contain more than one core group The recommended number of active members in a core group is 50 but can be extend to 100 with IBM_CS_WIRE_FORMAT_VERSION custom property enabled. WebSphere Support Technical Exchange 16

17 Multiple Core Groups - Bridging HA Manager services are normally restricted to a single core group (HA domain). There is one exception. The HAM state data exchange (bulletin board) service can be configured to communicate across core groups. The Core Group Bridge Service (CGBS) is the component that allows this cross core group communication. CGBS allows HA Manager bulletin board communication to span core groups. Core Groups within a cell can be bridged (intra-cell) Core Groups across cells can be bridged (inter-cell) WebSphere Support Technical Exchange 17

18 Role of Nodeagent and ORB in WLM WebSphere Support Technical Exchange 18

19 How Nodeagent involved in WLM Node Agent process provides service called Location Service Daemon(LSD) Application Server will register with Node agent during server startup LSD provides known direct (Interoperable Object Reference) IORs to caller WLM uses LSD cluster info for routing the EJB requests Don'ts Node agent should not be used for bootstrap in distributed OS environments. Only z/os environment is allowed for bootstrap.. WebSphere Support Technical Exchange 19

20 How Object Request Broker (ORB) is involved WebSphere Application Server uses the ORB to facilitate client/server communication EJB container depends on ORB for interactions between client/server ORB manages inbound and outbound requests for remote Java objects ORB provides a framework for clients to locate EJBs on the server Inter ORB communication is accomplished via the IIOP/GIOP using Interoperable Object Reference (IOR) which ORBs can understand and act. WebSphere Support Technical Exchange 20

21 Why bridging Coregroups WebSphere Support Technical Exchange 21

22 Why bridging Coregroups are important? WebSphere Application Server large topologies can be configured with multiple HA core groups to reduce usage of system resources such as memory and CPU. HA core group provides bulletin board service for each core group managed independently and bulletin board information can be shared within a core group WLM depends on the HAManager component s BulletinBoard service to aggregate and propagate the run time cluster description information. Due to practical limitations it may not be possible to have a nodeagent in each cluster that belongs different coregroups. HA manager core group bridge service allows communication between multiple core groups. Bridging between coregroups can be done between core groups within a cell or across multiple cells. WebSphere Support Technical Exchange 22

23 Request Flow WebSphere Support Technical Exchange 23

24 Request Flow servername="nodeagent" BOOTSTRAP_ADDRESS = "2810" ORB_LISTENER_ADDRESS= "9101" servername="clientserver1" BOOTSTRAP_ADDRESS = "2827" ORB_LISTENER_ADDRESS = 0 servername="ejbserver1" BOOTSTRAP_ADDRESS = "2829" ORB_LISTENER_ADDRESS = 9401 ic = new InitialContext(); Object ejbobject = ic.lookup("java:comp/env/beentherebean"); BeenThereHome beentherehome = (BeenThereHome)javax.rmi.PortableRemoteObject.narrow(ejbObject, BeenThereHome.class); beenthere = beentherehome.create(); WebSphere Support Technical Exchange 24

25 Request Flow Contd.. After obtaining InitialContext code does ic.lookup("java:comp/env/beentherebean"); Lookup Request sent from ClientServer1: ================================== OUT GOING: Locate Request Message Date: April 30, :23:58 PM EDT Thread Info: WebContainer : 0 Local Port: (0xEF40) Local IP: Remote Port: 2829 (0xB0D) Remote IP: Request ID: 68 (0x44) Object Key: length = 21 (0x15) Lookup Response from EJBServer1: ====================================== OUT GOING: Locate Reply Message Date: April 30, :23:58 PM EDT Thread Info: ORB.thread.pool : 0 Local Port: 2829 (0xB0D) Local IP: Remote Port: (0xEF40) Remote IP: Request ID: 68 (0x44) Reply Status: OBJECT_FORWARD IOR forwarded by EJBServer1 is indirect IOR pointing to LSD [4/28/15 15:06:31:209 EDT] WLMIOR > getwlmior Entry [4/28/15 15:06:31:209 EDT] WLMIOR 3 getwlmior - typeid= IDL:com.ibm/WsnOptimizedNaming/NamingContext:1.0 [4/28/15 15:06:31:209 EDT] WLMIOR 3 getwlmior - host= ADMINIB-NDLUO2T [4/28/15 15:06:31:209 EDT] WLMIOR 3 getwlmior - port= 9101 [4/28/15 15:06:31:210 EDT] WLMIOR 3 getwlmior - objectkey= 0x4a4d c WebSphere Support Technical Exchange 25

26 Request Flows Contd.. Request sent from ClientServer1:lookup contd.. ================================= [4/30/15 13:23:58:491 EDT] ORBRas 3 com.ibm.rmi.ras.trace dump:84 WebContainer : 0 OUT GOING: Request Message Date: April 30, :23:58 PM EDT Thread Info: WebContainer : 0 Local Port: (0xEECA) Local IP: Remote Port: 9101 (0x238D) Remote IP: Message header fragmented Request ID: 70 Response sent from Node agent ========================== [4/30/15 13:23:58:542 EDT] c6 ORBRas 3 com.ibm.rmi.ras.trace dump:84 ORB.thread.pool : 0 OUT GOING: Reply Message Date: April 30, :23:58 PM EDT Thread Info: ORB.thread.pool : 0 Local Port: 9101 (0x238D) Local IP: Remote Port: (0xEECA) Remote IP: Fragment to follow: Yes Message size: 1012 (0x3F4) -- Message header fragmented Request ID: 70 IOR forwarded by Node agent is indirect IOR pointing to EJBServer1 [4/30/15 13:23:58:545 EDT] WLMIOR > getwlmior Entry [4/30/15 13:23:58:545 EDT] WLMIOR 3 getwlmior - typeid= [4/30/15 13:23:58:545 EDT] WLMIOR 3 getwlmior - host= ADMINIB-NDLUO2T [4/30/15 13:23:58:545 EDT] WLMIOR 3 getwlmior - port= 9104 [4/30/15 13:23:58:545 EDT] WLMIOR 3 getwlmior - objectkey= R - typeid= WebSphere Support Technical Exchange 26

27 Request Flows Contd.. Request sent from ClientServer1 lookup contd..: ======================================== [4/28/15 15:06:31:524 EDT] ORBRas 3 com.ibm.rmi.ras.trace dump:84 WebContainer : 0 OUT GOING: Request Message Date: April 28, :06:31 PM EDT Thread Info: WebContainer : 0 Local Port: (0xEF42) Local IP: Remote Port: 9104 (0x2390) Remote IP: Message header fragmented Request ID: 76 Response sent from EJBServer1:Lookup contd.. ======================================== [4/30/15 13:23:58:959 EDT] f ORBRas 3 com.ibm.rmi.ras.trace dump:84 ORB.thread.pool : 0 OUT GOING: Reply Message Date: April 30, :23:58 PM EDT Thread Info: ORB.thread.pool : 0 Local Port: 9104 (0x2390) Local IP: Remote Port: (0xEF42) Remote IP: Request ID: 76 Reply Status: NO_EXCEPTION 0050: 636F6D2E 69626D2E com.ibm.webspher 0060: 652E7361 6D706C65 732E E7468 e.samples.beenth 0070: E E F6D ere.beentherehom 0080: 653A e: : BD EC WebSphere Support Technical Exchange 27

28 Request Flow Contd.. Request sent from ClientServer1 create: ================================= [4/30/15 13:23:59:002 EDT] ORBRas 3 com.ibm.rmi.ras.trace dump:84 WebContainer : 0 OUT GOING: Request Message Date: April 30, :23:59 PM EDT Thread Info: WebContainer : 0 Local Port: (0xEF42) Local IP: Remote Port: 9104 (0x2390) Remote IP: Message header fragmented Request ID: : AC F5F 686F6D65...)... home 0070: 4F66486F 6D F5F686F 6D654F66 OfHomes# homeof 0080: 486F6D F5F 686F6D65 4F66486F Homes# homeofho 0090: 6D E mes.beenthere#be 00A0: 656E E 6A E enthere.jar#been 00B0: EBDBDBD ThereBean... 00C0: BD D25 create...ibm% Response received from EJBServer1 create: ==================================== [4/30/15 13:23:59:030 EDT] f ORBRas 3 com.ibm.rmi.ras.trace dump:84 ORB.thread.pool : 0 OUT GOING: Reply Message Date: April 30, :23:59 PM EDT Thread Info: ORB.thread.pool : 0 Local Port: 9104 (0x2390) Local IP: Remote Port: (0xEF42) Remote IP: Request ID: 78 Service Context: length = 2 (0x2) Context ID: (0x49424D12) Reply Status: NO_EXCEPTION 0050: 636F6D2E 69626D2E com.ibm.webspher 0060: 652E7361 6D706C65 732E E7468 e.samples.beenth 0070: E E A3030 ere.beenthere: : BD WebSphere Support Technical Exchange 28

29 Important Custom Properties WebSphere Support Technical Exchange 29

30 IBM_CLUSTER_FEEDBACK_MECHANISM By default WLM plug-in uses a combination of the cluster member weights and the number of outstanding requests for each cluster member. Feedback mechanism can be changed by setting the cell custom property IBM_CLUSTER_FEEDBACK_MECHANISM on the target cell. Allowed values are: 0: Use only the configured weights to determine routing. 1: Use blending of weights and outstanding requests (default behavior). 2: Use only the outstanding requests to determine routing. 3: No extra feedback mechanism, does not take configured weights or outstanding requests into account. This is functionally equivalent to routing based on all servers having equal weights; any changes to the configured weights would be ignored When trace enabled following trace point can be used to check if default value has been changed. [4/28/15 15:00:36:394 EDT] WLMCustomProp 3 forcefeedbackstring is currently set to false WebSphere Support Technical Exchange 30

31 com.ibm.corba.connecttimeout The com.ibm.corba.connecttimeout property specifies the maximum time, in seconds, that the client ORB waits prior to timing out when attempting to establish an IIOP connection with a remote server ORB. Typically, client applications use this property. You can specify the property for each individual application server through the administrative console. Note : The default for the com.ibm.corba.connecttimeout property for Version 8 and later is 10. Earlier than Version 8, the default is 0 WebSphere Support Technical Exchange 31

32 com.ibm.corba.requesttimeout Specifies the number of seconds to wait before timing out on a request message Default Value is 180 seconds Click Servers > Server Types > WebSphere application servers > server_name > Container services > ORB service If you use standalone client or thin client, set com.ibm.corba.requesttimeout as system property WebSphere Support Technical Exchange 32

33 Common Issues WebSphere Support Technical Exchange 33

34 Issue 1: Forward limit reached Caused by: org.omg.corba.no_implement: Forward limit reached vmcid: 0x minor code: 40 completed: No at com.ibm.ws.cluster.router.selection.selectionmanager.targetforwarded(selectionmanager.java:366) at com.ibm.ws.wlm.client.wlmclientrequestinterceptor.receive_other(wlmclientrequestinterceptor.java:363) at com.ibm.rmi.pi.interceptormanager.invokeinterceptor(interceptormanager.java:599)... at com.ibm.corba.iiop.clientdelegate.invoke(clientdelegate.java:1320) NO_IMPLEMENT exception means that a requested object could not be located. For example, a NO_IMPLEMENT error is raised when a server does not exist or is not running when a clients initiates a request. Creating multiple instances of ORB can cause this issue. This can be easily identified using javacore or thread dump This is what an ORB reader thread looks like in javacore RT=383:P=570960:O=123:WSSSLTransportConnection[addr=XXX.XXX.XXX,port=47765,local=46289]" (TID:0x57FD8938, This is what an ORB listener thread id looks like. "LT=496:P=570960:O=123:port=46288" (TID:0x57FD8990, sys_thread_t:0x9ab1ef10, state:r, native ID:0x22E7FB) prio=5 The O= value is the ORB instance id. WebSphere Support Technical Exchange 34

35 Issue 2: No Cluster Data Available org.omg.corba.no_implement: No Cluster Data Available vmcid: 0x minor code: 42 completed: No at com.ibm.ws.cluster.router.selection.wlmlsdrouter.select(wlmlsdrouter.java:295) at com.ibm.ws.cluster.propagation.serverclustercontextlistenerimpl.forwardrequest(serverclustercontextlistenerimpl.java:625) at com.ibm.ws.cluster.propagation.serverclustercontextlistenerimpl.validaterequest(serverclustercontextlistenerimpl.java:669) at com.ibm.ws.wlm.server.wlmserverrequestinterceptor.notifyvalidationlisteners(wlmserverrequestinterceptor.java:317) at com.ibm.ws.wlm.server.wlmserverrequestinterceptor.receive_request_service_contexts(wlmserverrequestinterceptor.java:206) at com.ibm.rmi.pi.interceptormanager.invokeinterceptor(interceptormanager.java:621) This message is seen in Nodeagent, When a client makes the first request to a cluster, WLM plugin has no information yet about the target cluster members in order to do routing in an attempt to route the request, it sends it to node agents in the target cell. The node agents are expected to have data about the clusters which they can use to forward the request to a cluster member. If there are multiple core groups, make sure they are bridged. Verify the core groups are bridged in the coregroupbridge.xml Inside every <coregroupaccesspoint> there must be a <bridgeinterface> defined. If there is not a bridgeinterface for every coregroupaccesspoint, the core groups are not bridged. WebSphere Support Technical Exchange 35

36 Issue 3 : NoAvailableTargetExceptionImpl com.ibm.ws.cluster.selection.noavailabletargetexceptionimpl: Removal () Applicable Targets [] Removal () com.ibm.ws.cluster.selection.selectioncriteriaimpl@7666a71[{cellname=kumarancell01, CLUSTERNAME=cluster1}:{rules.restriction=[Lcom.ibm.wsspi.cluster.selection.SelectionRule;@28334}]] at com.ibm.ws.cluster.selection.selectioncriteriaimpl.select(selectioncriteriaimpl.java:261) Above message inform that WLM doesn't know about endpoints it can select to route the request to a target member Check for if HA Manager is enabled in all core group members Bridge Coregroups on the server side (if the nodeagent is not part of the cluster member coregroup) WebSphere Support Technical Exchange 36

37 Issue 4 : HA Manager View instability In order for WLM to work properly, the HA Manager view should be stable HA Manager view may not be stable, if there are any (OutOfMemory) OOM Error, Firewall blocking DCS Ports, Port conflict issue or Network Issues Refer and for more information You can confirm the HA View stability using DCSV8050I in the SystemOut.log file DCSV8050I: DCS Stack DefaultCoreGroup at Member dmgr\dmgr\dmgr: New view installed, identifier (291:0.dmgr\dmgr\dmgr), view size is 10 (AV=10, CD=10, CN=10, DF=40) Split Brain" (Split View) can occur due to connection/network problem. During this condition multiple views can be formed in a single coregroup. This scenario should be avoided and the problem should be fixed for WLM to spray the request properly Refer WebSphere Support Technical Exchange 37

38 Reference IBM Software Group Object Request Broker (ORB) Problem Determination and Best Practices Understanding how EJB calls operate in WebSphere Application Server V6.1 Ensuring enterprise availability when deploying Enterprise JavaBeans in WebSphere Application Server n.html Best Practices for Large WebSphere Topologies _largetopologies.html WebSphere Support Technical Exchange 38

39 Summary What is WLM and Clusters Why and How HA is important for WLM to work properly Importance of nodeagent and ORB Common WLM issues WebSphere Support Technical Exchange 39

40 Connect with us! 1. Get notified on upcoming webcasts Send an to with subject line wste subscribe to get a list of mailing lists and to subscribe 2. Tell us what you want to learn Send us suggestions for future topics or improvements about our webcasts to wsehelp@us.ibm.com WebSphere Support Technical Exchange 40

41 Questions and Answers WebSphere Support Technical Exchange 41

42 Additional WebSphere Product Resources Learn about upcoming WebSphere Support Technical Exchange webcasts, and access previously recorded presentations at: Discover the latest trends in WebSphere Technology and implementation, participate in technically-focused briefings, webcasts and podcasts at: Join the Global WebSphere Community: Access key product show-me demos and tutorials by visiting IBM Education Assistant: View a webcast replay with step-by-step instructions for using the Service Request (SR) tool for submitting problems electronically: Sign up to receive weekly technical My Notifications s: WebSphere Support Technical Exchange 42