Best Practices: Agent Updates Performing Large Scale Updates & Troubleshooting

Transcription

1 Christopher Hambridge Software Engineer 24 August 2012 Best Practices: Agent Updates Performing Large Scale Updates & Troubleshooting 2009 IBM Corporation

2 IBM Presentation Template Full Version Agenda Methods for Performing Large Scale Updates Using Groups to Perform Bulk Updates Group Deployment Commands Identifying Agents to Update Large Scale Deployments Suggestions & Tuning Using Agent Prerequisite Checking Executing a Bulk Deployment Using File Based Input to Perform Bulk Updates File Based Processing Identifying Agents to Update Executing a File Based Deployment Using an Alternate Tivoli Deployment Technology Tivoli Provisioning Manager Tivoli Endpoint Manager Updating the Framework Troubleshooting Agent Deployment Issues IBM Corporation

3 Methods for Performing Large Scale Updates ITM Installation & Configuration Guide: Chapter on Bulk agent deployment 2Fcom.ibm.itm.doc_6.2.3%2Fitm623_install429.htm Using Groups to Perform Bulk Updates itm623_install440.htm?path=3_1_2_0_6_4_6_3#wq851 Creating Deployment Groups (list of machines) & Bundle Groups (list of agent packages) for deployment Using File Based Input to Perform Bulk Updates 2Fcom.ibm.itm.doc_6.2.3%2Fitm623_cmdref12.htm Creating a Deployment File as input to the deployment commands Using an Alternate Tivoli Deployment Technology 2Fcom.ibm.itm.doc_6.2.3%2Fitm623_cmdref67.htm Exporting Bundles for use with an alternate deployment technology IBM Corporation

4 Using Groups to Perform Bulk Updates Group Deployment Commands Creating Deployment Groups & Bundle Groups: tacmd creategroup Adding Members to Groups: tacmd addgroupmember Identifying Agents to Update Using the Tivoli Enterprise Portal (TEP) to collect agents to update Format TEP data for input to Group Deployment Commands Large Scale Deployments Suggestions & Tuning Tips for creating Deployment Groups Tuning your bulk deployment Using Agent Prerequisite Checking Examine new function for checking endpoint prerequisites for an agent Stand-alone & remote prerequisite checking (preparatory scan or in-line deployment) Executing a Bulk Deployment Trigger Deployment using Groups Track Deployment Status IBM Corporation

5 Group Deployment Commands Create Deployment Groups & Bundle Groups Command Reference: 3D3_1_5_0_3_19 tacmd creategroup -g <group name> -t <group type> (DEPLOY or BUNDLE) A group with type 'DEPLOY' is a group of machines. A group with type 'BUNDLE' is a group of agent bundles. Examples: tacmd creategroup -g LZUpdate_623F2_DG -t DEPLOY tacmd creategroup -g LZUpdate_623F2_BG -t BUNDLE Add Members to Groups Command Reference: 3D3_1_5_0_3_5 Examples: tacmd addgroupmember -g LZUpdate_623F2_DG -t DEPLOY -m md44718 This is a hostname tacmd addgroupmember -g LZUpdate_623F2_BG -t BUNDLE -m LinuxAgentBundle -y LZ IBM Corporation

6 Identifying Agents to Update Using the Tivoli Enterprise Portal (TEP) to collect agents to update 1) Open the TEP and right click on the Enterprise and Select the 'Managed System Status' Workspace 2) Select the 'Properties' for the table to filter the list of agents that need to be updated IBM Corporation

7 Identifying Agents to Update Continued Filter Tab Filter on 'LZ' Use the 'Filter' tab to select only the agents you want to update. You can filter on items like product and version. You need to turn on the 'Host Address' column as this contains the hostname that will be used in the 'tacmd addgroupmember' command, as seen earlier. When completed this table should contain only a list of agents you wish to update. Deselect every column but the 'Host Address' column, so it is the only thing in the table IBM Corporation

8 Identifying Agents to Update Continued 1) After this filtering in the TEP you should be left with a set of host addresses associated with the agents that need to be updated, like the graphic on the right. 2) Next step will be exporting this data so it can be formatted for input into the 'tacmd addgroupmember' command 1 3) As you can see on the right, you can export this data to a CSV file. This option is available through the right click menu and selecting 'Export' IBM Corporation

9 Identifying Agents to Update Continued Now that you have the exported 'Host Address' data in CSV format you can open it in any spreadsheet program After importing it into your spreadsheet program you can construct a simple formula to extract the hostname information as shown below. Form ula: MID(A1;(SEARCH( <NM> ;A1)+4;SEARCH( </NM> ;A1)-((SEARCH( <NM> ;A1)+4)) IBM Corporation

10 Identifying Agents to Update Continued Once you have extended that formula to all of your rows, you can then copy the raw text into a new spreadsheet and save that as a new CSV file. Now after about 5 minutes of work you have a CSV file with all of the hostnames you need as input to the 'tacmd addgroupmember' command IBM Corporation

11 Large Scale Deployment Suggestions & Tuning Suggestions: We do suggest you create your groups per product and per platform. We also suggest you limit your group sizes to member range. The above suggestions should be followed at least early in your large scale deployment, as this will allow you to triage and troubleshoot any common problems (per product or platform) without wasting TEMS CPU time on deployments. Tuning: Build your deployment groups for parallelization Fan out your updates by spreading them across multiple RTEMS The RTEMS that will perform the update is determined by the OS Agent connection Controlling the Deployment Threads Remote deploy has 10 processing threads by default Alter the deployment thread count by changing the 'DEPLOYTHREADPOOLSIZE' environment variable in the TEMS environment file IBM Corporation

12 Using Agent Prerequisite Checking Publication Link Introduced in ITM some agents now package an Agent Prerequisite Checker Look in the Agent Guide for more information on this capability and the level of support. This bundle can be found under the agent image in the 'prereqchecker' directory. This bundle will be added to the depot automatically when when the agent is added. Use the 'tacmd viewdepot' command with bundle name <product code>-checker Prerequisite Checker test the target endpoint for required libraries, disk space, memory, etc. Prerequisite Checking can be run stand-alone on the target endpoint prereq_checker(.sh.bat) <three letter product code> prereq_checker(.sh.bat) KLZ Produces a results.txt file Prerequisite Checking can be run remotely on the target endpoint tacmd checkprereq -h <hostname> -t <product code> --collectall Collects results to deploying TEMS 'CANDLHOME\logs\checkprereq' directory Prerequisite Checking can be run in-line with a deployment (createnode,addsystem, updateagent) This can be done by including the command line '-o' flag properties 'EXECPREREQCHECK=Y' 'COLLECTALL=Y' IBM Corporation

13 Executing Bulk Deployment Executing a bulk deployment just takes a few more steps Taking the hostname file and creating a script for the groups and members Now you just need to make some pro-active checks: Use the 'tacmd viewdepot' command to check all TEMS impacted by the update for the required agent bundle. > tacmd viewdepot -t <product code> [-p <platform>] Use the 'tacmd addbundles' command to add any missing packages to the deploying TEMS. Now you can execute the 'tacmd updateagent' command: tacmd updateagent -g LZUpdate_623FP2_DG -b LZUpdate_623FP2_BG Filtering is new in ITMv623 You can track the deployment using 'tacmd getdeploystatus' command: tacmd getdeploystatus -g <transactionid> -o Overview flag new in ITMv623 Shows a count for status types IBM Corporation

14 Using File Based Input to Perform Bulk Updates File Based Processing What is File Based Processing? Identifying Agents to Update Using the Tivoli Enterprise Portal (TEP) to collect agents to update Format TEP data for input to File Based Processing Executing a File Based Deployment Trigger Deployment using File Based Processing Track Deployment Status IBM Corporation

15 File Based Processing Command Reference: 2Fcom.ibm.itm.doc_6.2.3%2Fitm623_cmdref12.htm The ITM command line 'tacmd' allows users to provide file based input to a command using the following format: tacmd <subcommand> <input file> This mechanism can be used by the 'tacmd updateagent' command Instead of supplying -- tacmd updateagent --type <product code> --node <OS agent> -- on the command line you can create a file with the following contents: update_lz_md44718.txt --type LZ --node md44718:lz This file can then be passed to the command tacmd updateagent update_lz_md44718.txt IBM Corporation

16 Identifying Agents to Update Using the Tivoli Enterprise Portal (TEP) to collect agents to update Just as before you can use the TEP to collect the input information you need for the File Based Input. Instead of collecting the 'Host Address' column, the 'Name' column must be collected. However, in this case the OS agents need to be collected. This is done by identifying all the UX, LZ and NT agents that match the hostnames of the agents to be update. This process is a bit more complex and time intensive than the previous method IBM Corporation

17 Executing a File Based Deployment Using the same export process you should be able to generate a CSV file of OS Agents needed to perform the Large Scale Update. The next step would be to create a small script that reads each line of the CSV and generates the input file in the form discussed earlier. You may want to use a naming scheme with a numeric (i.e. 3Q12_LZ_Update_0.txt thru 3Q12_LZ_Update_400.txt) Next you execute each update using a script that loops through the input files generated for the 'tacmd updateagent' command. You can track the deployment using 'tacmd getdeploystatus' command and overview flag: tacmd getdeploystatus -o Overview flag new in ITMv623 Shows a count for status types IBM Corporation

18 Using an Alternate Tivoli Deployment Technology Publication Link 2Fcom.ibm.itm.doc_6.2.3%2Fitm623_cmdref67.htm ITM also provides the capability for an alternative deployment technology to be used through the 'tacmd exportbundles' command. tacmd exportbundles -i <image> -e <exportdir> -o {LOCAL SPD SPB} -t <product> Tivoli Provisioning Manager can use the SPD output to deploy agents on a large scale Tivoli Endpoint Manager can be made to use the LOCAL (disk) output on a large scale An effective feature added in ITM v6.2.3 is exportbundles -os <Operating System> which allows the user to get all bundles for an Operating System instead of a specific platform type Operating Systems: LINUX, LINUX_Z, HP, WINDOWS, SOLARIS, AIX Trade off - Size vs. Matching Platform Type (i.e. li6264, lx8266,...) IBM Corporation

19 Updating the Framework Publication Link Introduced in ITM FP7 is the new command line 'tacmd updateframework' The new command line allows a user to update the agent framework or TEMA on a remote system. tacmd updateframework -n <OS Agent> -v <Version> tacmd updateframework -n asas1wd:lz -v This command also supports bulk deployment. tacmd updateframework -g <Deploy Group> -b <Bundle Group> -v <Version> tacmd updateframework -g UnixGroup -b UXBundle -v This command depends on the new package 'TF' found on the base agent image. You will need to add this bundle to your depot to perform this action. tacmd addbundles -i <image> -t tf IBM Corporation

20 Troubleshooting Agent Deployment Issues Troubleshooting any agent update failure starts with the 'tacmd getdeploystatus' command First check the associated message If it mentions a time out issue you may want to check the node status to see if the update succeeded, just not within configured time out. If this happens a lot you may want to consult the Installation Guide for configuring the default time out for a deployment. The current default time out is 600 seconds (10 minutes) and can be controlled by the 'TIMEOUT' environment variable in the TEMS environment file When altering the default time out, do so in incremental steps by 60 seconds until 98% of your deployments complete within the time out specified. Determine if the associated message provides the operator with an action to take Had the agent you are trying to update registered with the TEMS yet? Is the agent package located in the TEMS Depot (tacmd viewdepot)? Can the OS Agent be restarted without a problem (KDY1024E)? Determine where the error originated Messages with format KDY0 # # # E Originated from the TEMS Messages with format KDY1 # # # E Originated from the endpoint Knowing where the error originated can help you to zero in on what log to investigate Capture the transaction ID as this identifier can will help you to zero in on the deployment steps and issues related to this deployment within the log files IBM Corporation

21 Troubleshooting Agent Deployment Issues Continued Deployment Logs & Location From the 'tacmd getdeploystatus' you can obtain the deploying TEMS name. The serverside logs will reside on this machine in the following location: %CANDLEHOME%\logs\<TEMS hostname>_ms_*.log $CANDLEHOME/logs/<TEMS hostname>_ms_*.log The '<TEMS hostname>_ms.inv' file lists all the log files with the most recently written to at the top. The endpoint-side logs depend on the OS Agent (deploy runs as a child process) For Windows these files reside in the following location: %CANDLEHOME%\TMAITM6\logs\<agent hostname>_nt_kntcma_*.log %CANDLEHOME%\TMAITM6_x64\logs\<agent hostname>_nt_kntcma_*.log For Linux these files reside in the following location: $CANDLEHOME\logs\<agent hostname>_lz_klzagent_*.log For UNIX these files reside in the following location: $CANDLEHOME\logs\<agent hostname>_ux_kuxagent_*.log You can enable remote deploy tracing using (UNIT:KDY ALL) IBM Corporation

22 Troubleshooting Agent Deployment Issues Continued Installation Logs & Location If you don't see an issue in the deployment logs you can look at the installation logs Installation Logs on Windows: %CANDLEHOME%\InstallITM\<Product Name>(.log.trc) Installation Logs on UNIX or Linux: $CANDLEHOME/logs/candle_installation.log $CANDLEHOME/logs/itm_install_output_*.log $CANDLEHOME/logs/itm_install(.log.trc) $CANDLEHOME/logs/itm_config(.log.trc) After verifying that you are examining the correct installation log, my tip is to always start a the end of the file and work backwards. On UNIX or Linux start with the itm_install(.log.trc) as this contains data about a single installation where as the candle_installation.log contains all installation calls IBM Corporation

23 QUESTIONS? IBM Corporation