CA Workload Automation (DE) Internals and Troubleshooting. Lee Stecklov

Size: px

Start display at page:

Download "CA Workload Automation (DE) Internals and Troubleshooting. Lee Stecklov"

Hugh Scott
6 years ago
Views:

1 CA Workload Automation (DE) Internals and Troubleshooting Lee Stecklov

2 Terms of This Presentation This presentation was based on current information and resource allocations as of October 2009 and is subject to change or withdrawal by CA at any time without notice. Notwithstanding anything in this presentation to the contrary, this presentation shall not serve to (i) affect the rights and/or obligations of CA or its licensees under any existing or future written license agreement or services agreement relating to any CA software product; or (ii) amend any product documentation or specifications for any CA software product. The development, release and timing of any features or functionality described in this presentation remain at CA s sole discretion. Notwithstanding anything in this presentation to the contrary, upon the general availability of any future CA product release referenced in this presentation, CA will make such release available (i) for sale to new licensees of such product; and (ii) to existing licensees of such product on a when and if-available basis as part of CA maintenance and support, and in the form of a regularly scheduled major product release. Such releases may be made available to current licensees of such product who are current subscribers to CA maintenance and support on a when and if-available basis. In the event of a conflict between the terms of this paragraph and any other information contained in this presentation, the terms of this paragraph shall govern. 2 October CA Workload Automation (DE) Internals and Troubleshooting Copyright 2009 CA

3 For Informational Purposes Only Certain information in this presentation may outline CA s general product direction. All information in this presentation is for your informational purposes only and may not be incorporated into any contract. CA assumes no responsibility for the accuracy or completeness of the information. To the extent permitted by applicable law, CA provides this document as is without warranty of any kind, including without limitation, any implied warranties or merchantability, fitness for a particular purpose, or non-infringement. In no event will CA be liable for any loss or damage, direct or indirect, from the use of this document, including, without limitation, lost profits, lost investment, business interruption, goodwill, or lost data, even if CA is expressly advised of the possibility of such damages. 3 October CA Workload Automation (DE) Internals and Troubleshooting Copyright 2009 CA

4 Abstract > CA dseries Workload Automation is a java based clientserver cross-platform scheduler consisting of 3 components: CA dseries Workload Automation server, Agents and Desktop Client. Along with the 3 way communication the CA dseries Workload Automation server communicates with a relational database system through JDBC. The presentation discusses the Server architectural components, the life cycle of an application, environmental factors and troubleshooting. 4 October CA Workload Automation (DE) Internals and Troubleshooting Copyright 2009 CA

5 Architecture: background > The Server Pure Java (1.6 except AIX 1.5) Installed and started by any user Launched by Shell on Unix/Linux or Service/command on Windows 5 October CA Workload Automation (DE) Internals and Troubleshooting Copyright 2009 CA

7 Architecture: Server > Design Ground-breaking Object-Oriented design written in Java. Object oriented-design allows flexibility and adaptability. Multi-threaded -Java threading JDBC connectors for database enhanced caching Java RMI interface for Graphical Client and Command Line SOAP Web Services interface Single instance or High Availability 7 October CA Workload Automation (DE) Internals and Troubleshooting Copyright 2009 CA

8 Architecture: Server The Result: Highly scalable, high performance scheduling server. Benchmarking 4 million jobs per day Components Number of generations triggered CA WA Server R11.1 Sp1 (Build 58) with default system agent Number of jobs run in a day 3,809,800 Database(Oracle 11g) 30 Unix Agents(physical) 30 Unix Agent(physical) 20 Unix Agents(physical) 20 Unix Agent(physical) 8 October CA Workload Automation (DE) Internals and Troubleshooting Copyright 2009 CA

9 Memory(KB) Architecture: Server > Performance considerations: Memory Java requires the heap to be pre-allocationed Xmx defines the maximum heap which is 1 GB by default. However process will only acquire the necessary memory Typical Memory profile for continuous scheduling: Memory Usage of server process Time(Minutes) Memory 9 October CA Workload Automation (DE) Internals and Troubleshooting Copyright 2009 CA

10 % cpu usage Architecture: Server > Performance considerations: CPU Java Multi-threading means robust exploitation of available cpu s 40.00% cpu usage of server process 35.00% 30.00% 25.00% 20.00% 15.00% 10.00% 5.00% 0.00% Time(Minutes) cpu usage 10 October CA Workload Automation (DE) Internals and Troubleshooting Copyright 2009 CA

11 Architecture: background > Database considerations Configuration, Schedule and run-time data stored in relational database (Oracle / MSSQL / DB2) JDBC used for database connection High volume, simple SQL insert, update, delete To ensure persistence, all run-time data is saved in database Database requirements are straightforward: one tablespace (Oracle), or one database (MSSQL). 11 October CA Workload Automation (DE) Internals and Troubleshooting Copyright 2009 CA

12 Architecture: background > Relational Database Schema Logically divided into Runtime, History and Definition tables Runtime tables are used as queue and persistence tables they are highly transactional. History tables record job history, less volatile but grow over time Definition tables contain XML definitions saved in CLOB data types. 12 October CA Workload Automation (DE) Internals and Troubleshooting Copyright 2009 CA

13 Architecture: communications > The Server Communications: TCP/IP Server requires 4 LISTENING ports Manager Client RMI export RMI Registry 13 October CA Workload Automation (DE) Internals and Troubleshooting Copyright 2009 CA

14 Architectural diagram the network DB Port i.e DB Port i.e Server: Manager Port:7507 Agent: Port 7520 RMI Ports 7598/7599 SNMP Receiver Client Port: 7500 RMI Ports: 7598/7599 RMI ports RMI ports CLI Server 14 October CA Workload Automation (DE) Internals and Troubleshooting Copyright 2009 CA

15 Architecture: communications > Server Communications: TCP/IP All ports bind to default installation address EXAMPLE: tcp 0 0 simon.ca.com:58500 *:* LISTEN tcp 0 0 simon.ca.com:58598 *:* LISTEN tcp 0 0 simon.ca.com:58599 *:* LISTEN tcp 0 0 simon.ca.com:58507 *:* LISTEN 15 October CA Workload Automation (DE) Internals and Troubleshooting Copyright 2009 CA

17 Architecture: UI > Desktop Client Based on the Eclipse graphical platform, pure Java Installs on user s workstation Started by a windows executable stub Personal workspace is C:\Documents and Settings\%user%\workspace Can automatically failover to the primary server 17 October CA Workload Automation (DE) Internals and Troubleshooting Copyright 2009 CA

Architecture: UI The Desktop Client 18 October 13-16 2009 CA

19 Architecture: UI > Desktop Client Creates java RMI (Remote Method Invocation) interface to Server Primary Functions: Define Workload Monitor Server Administration 19 October CA Workload Automation (DE) Internals and Troubleshooting Copyright 2009 CA

22 Architecture: Web Services > Web Services Interface uses Apace Tomcat and AXIS 1. Simple Object Access Protocol (SOAP) over HTTP or HTTPS use Web Services Description Language (WSDL) You can use web services to invoke the following CA WA functions: Trigger an Event in addition to its usual schedule Bypass a scheduled Event Hold (postpone) an Event Release a held Event Suspend an Event from triggering Resume a suspended Event Replace an Event's next scheduled execution with a new time 22 October CA Workload Automation (DE) Internals and Troubleshooting Copyright 2009 CA

23 Architecture: Application Life Cycle > Scheduler thread The scheduler initiates the Application Life Cycle. Loads the TDR (Time Driven Request), which is an XML representation of the event. RUN.TEST this TDR is scheduled on the :00. Processes JavaScript, and sends the Application definition to the Manager. Builds the new TDR, and modifies the Event 23 October CA Workload Automation (DE) Internals and Troubleshooting Copyright 2009 CA

24 Architecture: Application Life Cycle > Manager The Manager builds, executes and tracks. Builds the job definitions into memory, executes JavaScript if necessary Sends commands to Agent Receives and processes state information from Agent Persists states in RDBMS Persists relevant job information in RDBMS for history reporting 24 October CA Workload Automation (DE) Internals and Troubleshooting Copyright 2009 CA

25 Architecture: Application Life Cycle Manager completes application life cycle. If the application is complete, terminates the thread, updates the database By querying the database, we can see how many applications are active. This is what the desktop client does when subscribing. The same information can be seen in esp_wss_appl: SQL> select count(*) from esp_wss_appl where STATUS=1; COUNT(*) There are 45 active applications. 25 October CA Workload Automation (DE) Internals and Troubleshooting Copyright 2009 CA

26 Architecture: Job Life Cycle > Job Life Cycle Data Flow is best demonstrated by the job life cycle. Jobs are build when APPLICATION is created When a job has satisfied all it s requirements: resources, time constraints, predecessors, etc.. server considers it runnable and can send the run AFM (message) to the Agent AGENT LSESP5 WINNT1/VERIFYD.444/MAIN RUN. Data(Command=C:\Batches\test.bat) MFUser(LEE) Agent acknowledges reception At this point, job is in READY state on server 26 October CA Workload Automation (DE) Internals and Troubleshooting Copyright 2009 CA

28 Troubleshooting > Tracelog and AFM log Tracelog Logs errors termed exceptions in Java Various filter ids correspond to server subsystems Filter ids can be turned on as needed (on the fly or permanently) Afmlog Logs AFM traffic (remote and internal) Can be used to watch agent traffic Audit log Who did what and when 28 October CA Workload Automation (DE) Internals and Troubleshooting Copyright 2009 CA

29 Troubleshooting > Logging Exceptions Basis of troubleshooting server problems. Exception logging include call stacks Searching Log Files Quick test (Unix or Windows using GNU grep): grep -i exception tracelog.txt egrep -v -i 'send connect ping ack' Case insensitive search for exception, not including exceptions with the strings commonly found in agent connection exceptions :06: [ID: 0] SS: org.mozilla.javascript.evaluatorexception: Property 0 not found. 29 October CA Workload Automation (DE) Internals and Troubleshooting Copyright 2009 CA

Troubleshooting > Logging Examining the exception stack 30 October 13-16

31 Troubleshooting > Logging Example: JavaScript Exception stack :05: [ID: 0] SS: org.mozilla.javascript.evaluatorexception: Property 0 not found. at org.mozilla.javascript.defaulterrorreporter.runtimeerror(unknown Source) at org.mozilla.javascript.context.reportruntimeerror(unknown Source) at org.mozilla.javascript.context.reportruntimeerror(unknown Source) at org.mozilla.javascript.context.reportruntimeerror1(unknown Source) at org.mozilla.javascript.scriptableobject.setattributes(unknown Source) at cybermation.library.script.javascript.cybjsflatobjectfactory.restore(cybjsflatobjectfact ory.java:593) In this example, the SS thread is throwing an exception while evaluating a JavaScript. A good starting point. 31 October CA Workload Automation (DE) Internals and Troubleshooting Copyright 2009 CA

32 Troubleshooting > Logging Example: SQL error Exception stack :57: [ID: 0] HAC: PRIMARY SQLException on update/insert - java.sql.sqlexception: Io exception: Connection reset by peer: socket write error at oracle.jdbc.driver.databaseerror.throwsqlexception(databaseerror.java:125) at oracle.jdbc.driver.databaseerror.throwsqlexception(databaseerror.java:162) at oracle.jdbc.driver.databaseerror.throwsqlexception(databaseerror.java:274) ) :57: [ID: 0] HAC: Could not rollback the connection; the exception is Closed Connection - java.sql.sqlexception: Closed Connection at oracle.jdbc.driver.databaseerror.throwsqlexception(databaseerror.java:125) at oracle.jdbc.driver.databaseerror.throwsqlexception(databaseerror.java:162) at oracle.jdbc.driver.databaseerror.throwsqlexception(databaseerror.java:227) at oracle.jdbc.driver.physicalconnection.rollback(physicalconnection.java:994) at com.cybermation.espresso.rdbms.dbpooledconnectionwrapper.rollback(dbpooledconnec tionwrapper.java:141) at com.cybermation.espresso.rdbms.relationaldatabasemanager.unconditionalrollback(rela tionaldatabasemanager.java:382) High Availability thread reporting an I/O error on RDBMS 32 October CA Workload Automation (DE) Internals and Troubleshooting Copyright 2009 CA

33 Troubleshooting > Logging SQL I/O error example 911 Thread monitors critical errors :57: [ID: 0] 911-Service: ************************************************************ * The thread HAC reported a fatal server error. * Server 911 service is initiating the shutdown procedure. ************************************************************ Server is programmed to shutdown on certain SQL errors. In this case, could have been a simple Network glitch, or database down. The actual error SQLCode: SQLState was returned by the JDBC driver. This tells us that the problem occurred at the jdbc network interface 33 October CA Workload Automation (DE) Internals and Troubleshooting Copyright 2009 CA

34 Troubleshooting > Logging.. Server monitoring 911 Thread responsible for server health Server state thread tracks memory and database connection pools :37: [ID: 0] Thread-5: Esp Server State: Memory: Using 18 MB of Currently Allocated 38 MB. Total Free Memory is 998 MB of the Max Heap Size 1016 MB Threads: [Thread group=system;active groups=6;active threads=131] [Thread group=main;active groups=4;active threads=119] [Thread group=rmi Runtime;active groups=0;active threads=0] Database Connections:.. In Use Connections: Connection Wrapper[usageCount: 241, lastaccesstime: :37:03.204, Used by Thread: RDBOutputMessageQueue_13] 34 October CA Workload Automation (DE) Internals and Troubleshooting Copyright 2009 CA

35 Troubleshooting > Quick Log Searches Using GNU grep of Windows findstr command Track server memory usage Unix: grep Memory tracelog.txt Windows: findstr Memory tracelog.txt Track the progress of the SS thread: grep SS: tracelog.txt Track on particular Application generation: grep INITDIR.889 tracelog.txt Track High Availability Component: grep HAC tracelog.txt 35 October CA Workload Automation (DE) Internals and Troubleshooting Copyright 2009 CA

36 Troubleshooting > Thread Dumps Very helpful in case of server hangs A Java feature available on Unix/Linux only kill -3 pid_of_server creates a thread dump in ESP_HOME/MonitorAndStatus/server.log(Linux/Solaris/HP), <ESP_HOME>javacorennnn.nnnn.txt (AIX). Thread dump contains call stacks for all active threads Capture 3 thread dumps 1-2 minutes apart Valuable tool for support / development to analyze problems not found in log files. 36 October CA Workload Automation (DE) Internals and Troubleshooting Copyright 2009 CA

CA Automation Capabilities A Technical Look at Process and Runbook Automation. Tom Kouhsari and AJ Dennis

CA Automation Capabilities A Technical Look at Process and Runbook Automation Tom Kouhsari and AJ Dennis Terms of This Presentation This presentation was based on current information and resource allocations