IBM Lab Services Tools

Transcription

1 IBM Lab Services Tools

2 Lab Services Tools Capacity Planning Tool (CPT) VSCSI-to-NPIV Migration Tool LPM Automation Tool w/ Simplified Remote Restart IBM Advanced PowerVM Provisioning Toolkit v5 Provisioning with buildframe capture/create/build Orchestrating LPAR Migrations Decommissioning LPARs PowerDraw Proactive Monitoring w/ Promon

3 Where does he get those wonderful toys Options for obtaining the IBM Lab Services Tools The tools are obtained through a Lab Services engagement. There are several easy funding approaches For further questions on funding, or to schedule obtaining one of the tools, please contact: Linda Hoben hoben@us.ibm.com OR- Stephen Brandenburg sbranden@us.ibm.com

4 Size it correctly with the Capacity Planning Tool

5 Moving to a new version of Power?... Use CPT Make sure it is sized properly with Capacity Planning Tool (CPT) CPT uses customer s utilization data and suggests right-sizing partitions CPU Sizing for migration from one power platform to another Allows for customer to group (tier) their partitions into different sizing categories Customer can do what-if sizings based on level of risk for partitions NEW Web Browser Interface to view your data

6 Capacity Planning Toolkit This toolkit was released in 4Q2013 and has been delivered to over 65 customers. IBM Power Customers often do not have proper focus on capacity planning. Enterprise Capacity Planning teams are typically crossplatform, and sometimes Power skills are lacking Partitions are sized once, and oft-forgotten If it s not broken, don t fix it This can lead to under-subscribing a server/frame Likewise, IBM Power Customers often do not properly monitor their Power environment. Infrequent sampling Short retention of sample data We often see folks leaving default retention periods with monitoring tools. No way to get good data more than a few weeks back.

7 Capacity Planning Tool Sizing Strategy Frequent sampling is better Minutes or seconds, not hours or days. Calculate consumption by percentile Look at the frequency physical consumption exceeds certain values The Capacity Planning Tool has been designed to take historical CPU consumption data for logical partitions (LPARs) on IBM Power systems and fit them to a more appropriate sizing, based on a combination of IBM best practices, and customer business requirements.

8 Averaging in graphic detail LPAR with EC=1.0

9 Capacity Planning Tool - INPUTS The Capacity Planning Tool (CPT) is was designed to be flexible, and handle a number of different input formats: NMON ITM BMC LPAR2RRD HMC LPARUTIL HMC PCM AIX VMSTAT CUSTOM

10 Perl preprocessor CPT Analytic Engine Capacity Planning Tool High Level NMON HMC lparutil ITM CPT CSV common format CSV file For review with application teams MS Excel reports BMC Workload Estimator CSV WLE output for server consolidation study

11 Flexibility in sizing algorithms and defining HA clusters The tool allows you to categorize LPARs into 4 different tiers. You can specify different percentiles to be covered with the utilization data and growth rates and EC/VP ratios. For example, Tier 1 may be for your production and VIOS LPARs and you want the tool to size them with a higher percentile for the EC and to allow for some percentage of growth. Tier 2 could be lower-priority LPARs with a different set of sizing goals. The customer can also define which LPARs are in a HA cluster and the type of cluster (Active/Passive, Active/Active.) and the tool can size based on those settings.

12 Excel file Summary Worksheet

13 Excel file LPAR worksheet There is a worksheet tab for each partition and a Summary tab

14 New Browser Based Interface

15 New Browser Based Interface

16 New Browser Based Interface

17 Stuck on VSCSI? Use the VSCSI-to-NPIV Migration Tool

18 VSCSI-to-NPIV Migration Tool This tool is a new addition to Lab Based Services in Spring2015. It has been used by several dozen customers with very good success. There are many customers that are still using VSCSI technology but want to move to NPIV technology This tool will do the step-by-step processes needed for a customer to convert while leaving the partition up and running. The VSCSI to NPIV Conversion tool does the following tasks: Convert a VSCSI partition to NPIV with no downtime NPIV virtual adapter creation/mapping LUN mirroring and mirror breaking VSCSI virtual adapter unmapping/deletion Support for EMC, 3PAR, NAS, Hitachi and IBM storage PowerHA Support

19 Why move to NPIV? Simplicity Cleaner configuration Ease of use Ease of configuring LPM Won t have no_reserve disks on various VIO s What Do You Need? 8 Gig or 16 Gig FC Adapters Power 6 or later NPIV capable Switch

20 New to Version 1.3 The tool now allows the following: Support for sudo for the clients. PowerHA Support is now available (cspoc for shared disks, replace the repository disk.) PowerHA 7.1. Setting the disk attributes on NPIV, such as queue_depth, algorithm, and reserve_policy Pick your VG s and disks for Mirroring. Batch job for steps 1-5 in the Menu. Force WWPN s to vfc adapters

21 Conversion process Create the VFC adapters and vfcmaps. Create and zone duplicate storage to the new VFC s WWPN s. Mirror with the LVM and sync the Mirror. Starting LPAR Add Zone New Disks MirrorDisks Client LPAR VFCS VFCS WWPN WWPN VSCSI VSCSI VIO1 VIO2 WWP N WWP N Mirror disks

22 Conversion process (continued) Break the mirror, unmap and remove the VSCSI and return VSCSI storage to the SAN Starting LPAR Add Zone New Disks MirrorDisks Client LPAR VSCSI VSCSI VFCS VFC WWPN WWPN VIO 1 WWP N VIO 2 WWP N

23 Choosing the HMC account and Managed System Which user would you like to use to login to the HMC? hscroot Which managed system would you like to use? 0: kurtkp8 1: bobfp8 2: Server-9117-MMC-SN105C627

24 Choosing the LPAR to convert Which lpar would you like to use? 0: tkvios1 1: tkvios2 2: bb2vios1 3: tk_client1 4: tk_client2 5: tk_client3 6: tk_client4 7: tk_client61 8: tkdemo2 4

25 VSCSI-to-NPIV Menu VSCSI to NPIV Menu 1: Get LUNid's and LUN Sizes 2: Create VFC Adapters 3: Collect WWPN's 4: Create VFC Mappings 5: Login/Logout your WWPNs 6: Set Disk Attributes 7: MirrorVG 8: Break the Mirror 9: Clean up VSCSI Mappings 10: PowerHA Options 11: Exit You can exit the tool and resume at the next entry

26 Get LUN ids and LUN sizes Collecting LUN sizes and ID's Please check /bonnie/tk_client2.lunlist.out for the List of Luns and Sizes Would you like to Create your VFC adapters? Y

27 Example list of LUN ids and sizes

28 Creating VFC adapters How many VFCs per VIO How many VFC adapters would you like to add to each VIOS? Ex. Best practice would be at least 2 per VIOS one per fabric 2

29 Creating VFC adapters Creating Please enter the client slot(s) for tkvios1. Ex. 3,5 10,12 Please enter the client slot(s) for tkvios2. Ex. 2,4 11,13 Creating VFC adapters, please wait... Saving current configs Running cfgdev on tkvios1 Running cfgdev on tkvios2 VFC's created successfully Would you like to Map the VFC's on the VIOS? y

30 Mapping VFC Adapters Menu On tkvios1, which FC adapter would you like to map to vfchost8 at VIO slot adapter 2? name physloc fabric tports aports swwpns awwpns fcs0 U78C0.001.DBJF101-P2-C1-T fcs1 U78C0.001.DBJF101-P2-C1-T fcs2 U78C0.001.DBJF101-P2-C5-T fcs3 U78C0.001.DBJF101-P2-C5-T : fcs0 at U78C0.001.DBJF101-P2-C1-T1 2: fcs1 at U78C0.001.DBJF101-P2-C1-T2 3: fcs2 at U78C0.001.DBJF101-P2-C5-T1 4: fcs3 at U78C0.001.DBJF101-P2-C5-T2 1

31 MirrorVG Introducing the new NPIV disks

32 Break the Mirror - Menu Finding VG's with SCSI Disks Verifying VG's are Synced The following VG's are Mirrored. Which VG would you like to Break the Mirror? 1: datavg 2: rootvg 2

33 Break the Mirror Unmirror rootvg Locating SCSI disks to unmirror Unmirroring rootvg rmlvcopy: If hd5 is the boot logical volume, please run 'chpv -c <diskname>' as root user to clear the boot record and avoid a potential boot off an old boot image that may reside on the disk from which this logical volume is moved/removed chvg: The quorum change takes effect immediately unmirrorvg: rootvg successfully unmirrored, user should perform bosboot of system to reinitialize boot records. Then, user must modify bootlist to just include: hdisk2. Making LV Copy of lg_dumplv Syncing lg_dumplv Removing LV copy of lg_dumplv Note: The script checks to see if the dump device has been mirrored (it shouldn t have but sometimes ) if not it will do a mklvcopy > sync> so it will move to the NPIV disk.

34 Break the Mirror Unmirror rootvg (continued)

35 Break the Mirror Remove VSCSI Client Devices Removing disk from tk_client2 Removing hdisk0 hdisk0 deleted Verifying all SCSI hdisks are removed Gathering VSCSI adapters Gathering VSCSI adapter slots Removing VSCSI Adapter vscsi0 vscsi0 deleted Removing VSCSI adapter slots Saving current tk_client2 configs

36 Break the Mirror Remove VSCSI Client Devices Gathering VSCSI adapters Gathering VSCSI adapter slots Removing VSCSI Adapter vscsi1 vscsi1 deleted Removing VSCSI adapter slots Saving current tk_client2 configs

37 Break the Mirror Unmapping VSCSI at VIO1 Unmapping the following disks hdisk1 on tkvios1 rvg_client2 deleted hdisk1 on tkvios2 rvg_client2 deleted Removing vhost2 from tkvios1 vhost2 deleted vhost2 was removed successfully Slot 20 has been removed Saving current configs

38 Break the Mirror Unmapping VSCSI at VIO2 Removing vhost1 from tkvios2 vhost1 deleted vhost1 was removed successfully Slot 20 has been removed Saving current configs All Mirrored VGs are complete. Please check /bonnie/tk_client2.lunlist.post for the LUNS that were removed

39 Ready to Migrate LPM Automation Tool Frame Outage Simplified Remote Restart

40 LPM Automation Tool and Simplified Remote Restart This tool was released in 4Q2014. Over 160 customers worldwide have already adopted it with very high customer satisfaction. This tool is the fastest growing ever in lab-services and is used on every continent and sub-continent (other than Antarctica). With the new SRR feature of Power8 servers, this tool is becoming a must for any customer wanting to use SRR. Customers current use of LPM Customers are using LPM more and more as part of their daily tasks (workload balancing) and planned maintenance procedures The HMC GUI only allows a single partition to be selected at at time Imagine having to move 100 partitions from a server (esp. using the GUI - (that s a lot of clicks!) Some customers are writing scripts to move many partitions at a time

41 LPM Automation Tool and Simplified Remote Restart The PowerVM LPM and Remote Restart Automation tool provides: Easy to use GUI to plan and execute LPM between multiple frames Can build plans in advance of a nightly/weekend migration window Can specify different VIOS mappings and HBA mappings New support added for Remote/Restart of partitions (no need for HMC CLI) which you will want to use on your new P8 servers!!!

42 LPM Automation Tool and Simplified Remote Restart Automated use of LPM This tool was developed to allow a customer to quickly move one or many partitions from a server using LPM to other servers (as few as 4 clicks) It can also return the partitions back to the original server with the original mappings (HBA/virtual slot id) (again, as few as 4 clicks) You can build a plan(s) in advance then execute the plans during maintenance windows. You can schedule LPM operations. Automated use of Simplified Remote Restart (SRR) Use of SRR on the HMC is only thru command line (CLI). There is no GUI for the SRR function and you can only move 1 partition at a time via the HMC CLI. The tool works similarly for both LPM and SRR, you can use a GUI to quickly SRR many or all of the partitions to one or more destination servers. You can build a plan(s) in advance then execute the plans during an outage. A huge benefit of the tool is that once the server is repaired, the tool can move all the partitions back to the original server with just a few clicks using LPM.

43 New features in Version 8.5 The tool version numbering has changed to match the features of the highest HMC version it can exploit. For example, since HMC V8.5 can specify vfc mappings and shared proc pools when doing SRR, version 8.5 of the tool supports those features as well. The tool still supports older HMC levels. New features: Specifying new VSWITCH mapping during LPM Specifying SRIOV VNIC mappings during LPM Increasing number of vfc mapping support from 63 to unlimited Subdividing partitions into different groups of concurrent LPMs Specifying concurrency performance levels on a per partition basis Returning SRR partitions back to original server with original settings Remote HMC support for SRR Use of ssh keys instead of HMC passwords Enabling SRR during LPM operations

44 New features in Version 8.6 The tool still supports older HMC levels. New features: Specifying redundant MSPs when doing LPM via GUI Specifying Shared Proc Pool and Vswitch during Inactive LPM via GUI Specifying LUN Validation during LPM validation via GUI Fixed Shared Processor Pool Checkbox on Perform LPM (defect was introduced in V8.5) Changed label to NO CONNECTION on the SRR Panel (it was incorrectly labeled FORCE SRR) More control over LPM and SRR options in config.properties

45 Home Screen Version 8.6 Nota bene: These graphics will change in v 8.7

46 LPM Away Choose servers This icon gets you to this screen

47 LPM Away Placement Panel

48 LPM Away Concurrency Level in v8.5 In HMC V8R8.5, the number of threads per LPM can be set via the concurrency performance level parameter. 4 is default (1 thread per partition). If you need to move a large partition over a >30 Gbit network, you can set this to 1 and then 4 threads in a VIOS will be used to transfer data for that 1 partition

49 LPM Return Return LPM or SRR to original server This screen will show servers with partitions that haven t been returned to their source server. If the partitions were moved to multiple servers, this operation will bring them back from all the servers. If multiple plans were used to move partitions off of this server, you have 2 options. Let the tool bring all the partitions back at once or you can import the plans one by one to restore the server. When moving the partitions back to the original managed system, the tool will restore the virtual adapter numbers and shared proc pools and HBA mappings that were originally being used before the managed system was evacuated

50 LPM Automation Tool also supports SRR One click will select all the partitions or select/deselect individual partitions Can choose one or many destination servers This icon gets you to this screen

51 Remote Restart - Placement Change dest server with drop down As dest server is changed, the Remaining values change

52 Videos on youtube.com Short Videos (these URLs are case-sensitive) ibm.biz/lpm_overview demo of LPM/SRR tool (10 minutes May 2016) ibm.biz/lpm_scheduler demo of LPM/SRR tool scheduling a group of LPMs (4 minutes May 2016) ibm.biz/lpm_pep demo of LPM/SRR tool automating Power Enterprise Pool resources moves as part of LPM operations (5 minutes May 2016) ibm.biz/srr_benefits video presentation on what SRR is and why you need to use it (12 minutes May 2016) ibm.biz/srr_tool demo of LPM/SRR tool performing SRR operations and cleanup of a failed server (8 minutes May 2016) ibm.biz/srr_enterprise_tool video presentation on why you need the LPM/SRR tool to do enterprise-level SRR operations (12 minutes long Aug 2016) ibm.biz/srr_bikeride enjoyable video/demo on how quick SRR can recover a failed server using the LPM/SRR Automation tool (5 minutes long May 2016) Longer, more detailed videos - detailed video presentation on LPM/SRR tool including new features in V8.5 of the tool (1:23 long Aug 2016) - detailed presentation on what SRR is and why customers need this and some information on the LPM/SRR tool (1:31 long Aug 2016)

53 Consistent, fast deployment of VIOS and LPAR s Create DR environment from Production Need Remote Restart on Power7?... Decommission arbitrary LPARs Migrate servers that can t be LPM d to another frame The IBM Advanced PowerVM Provisioning Toolkit v5

54 IBM PowerVM Provisioning Toolkit Use Cases The IBM Advanced PowerVM (Provisioning) Toolkit helps with the following tasks: Install/Deploy P8 VIOS with new VIOS infrastructures (i.e. VLANs, NPIV, SEA, device attributes, etc) Deploy new virtualized client LPARs anywhere in your environment. Similarly, it can help re-deploy your P7 client partitions onto new VIOS infrastructure on P8. Use DR features of tool (captureframe/createfame/buildframe) to easily deploy into DR site. NEW FEATURE Wrapper script to migrate LPARs to another server if they can t LPM. Simplifies capture/create/build sequences.

55 Manual provisioning flow The way MANY customers still work TODAY HMC you NIM Managed System, VIO Gather system build information: HMC, managed system VIO and client names, IP addresses, VLANs, storage, CPU, memory, virtual switches, shared processor pools, SEA/network designs, Repeat a series of manual interactions using that data: HMC GUI or command line execution NIM commands VIO commands Consistency can be a struggle

56 IBM Advanced PowerVM Provisioning Toolkit v5 Since 2009, more than 200 customers served worldwide Makes PowerVM deployment and configuration: Easier Faster Consistent Fun! With a process that is: Flexible (simple or complex designs) Scalable (large or small enterprises) Easily automated Can be integrated as loosely or tightly into your current process flow as you wish. This tool is also used internally by the IBM BCRS (Business Continuity and Resiliency Services) to help rebuild customer s configurations quickly and accurately for DR situations.

57 IBM Advanced PowerVM Toolkit v5 - Provisioning buildframe.ksh The buildframe script is the original Provisioning Toolkit component dating back to v1 (circa. 2009) Provisions VIO servers and clients from easy-to-create ASCII text files ( build_profiles and config_files ). KEYWORD with key=value arguments. Speeds and simplifies PowerVM provisioning and gives repeatable, consistent results every time. Can be used stand-alone with user-generated, monolithic build profiles. With wrapper scripting and config_files (or include_files), can be integrated and automated further into any environment for build-on-demand possibilities. Toolkit GUI /Editor Added to Toolkit in v2 timeframe (since 2012). Editor/syntax assistance for build_profile/config_file inputs to buildframe.ksh. KEYWORD samples buildframe.ksh execution assistance work on files and provision right from your laptop! Toolkit wiki TiddlyWiki v5 now used for Tookit documentation. Embedded graphics Searchable/browsable single file - syntax and function reference for all Toolkit files and components.

58 IBM Advanced PowerVM Toolkit v5 unique instance information design pattern [-c config_file] TARGET FRAME -p build_profile buildframe.ksh ssh keys FSP, DLPAR, viosvrcmd User inputs ksh script Generated outputs ksh93,nim Java GUI: assist w/ syntax, remote execution HMC provision, install, configure: lpar_env=vioserver lpar_env=aixlinux HMC cmds (e.g. mksyscfg, ) NIM cmds (nim o bosinst, ) VIO cmds (mkvdev, vfcmap )

59 Simplified build_profile example SERVERINFO VETHMAP FABRIC FABRIC VIO VFCGEN VFCMAP VIO VFCGEN VFCMAP servername=server-9117-mmc-sn105c627,buildhmc=mghmc.rchland.ibm.com clientslot=2,vlan=50 name=fabrica,clientadapters=4 name=fabricb,clientadapters=7 hostname=tkvios1,profilename=normal mapping=4:auto port=dbjf101-p2-c1-t1,clientinfo=4:tk_client1 hostname=tkvios2,profilename=normal mapping=7:auto port=dbjf103-p2-c1-t4,clientinfo=7:tk_client1 BOSINST bosinstid=aix71,source=mksysb,mksysb=mksysb_aix710tl02sp2, spot=aix710tl2sp2spot,bosinst_data=bosinst_data_aix,accept_licenses=yes,boot_client=no PARTITION name=tk_client1,profile_name=normal,min_mem=6144,desired_mem=16384, max_mem=24576,min_proc_units=0.1,desired_proc_units=0.3,max_proc_units=2.0,min_procs=1, desired_procs=2,max_procs=3,lpar_env=aixlinux,proc_mode=shared,uncap_weight=128,mem_mode=ded, allow_perf_collection=1,sharing_mode=uncap,max_virtual_slots=11,all_resources=0,conn_monitoring=1, redundant_err_path_reporting=0,bosinstid=aix71,lpar_netboot= ,tk_client1, , , ,NA,auto,auto,NA

60 buildframe.ksh 101 NPIV client LPAR creation: Assuming the build profile on the previous page is named tk_client1.def, we can build that defined client this way 1. buildframe.ksh p tk_client1.def -f carve # create partition profile and devices 2. buildframe.ksh p tk_client1.def -f vfcmap # map vfcs to VIOS-owned fibre ports 3. buildframe.ksh p tk_client1.def -f wwpn # harvest WWPNs for SAN team buildframe.ksh p tk_client1.def -f carve,vfcmap,wwpn # can do 1-3 all in one operation 4. SAN LUN allocation and zoning 5. buildframe.ksh p tk_client1.def -f nim # lpar_netboot from NIM

61 Using buildframe with config files VIO servers e.g. Initial frame provisioning Config file (instance1) Frame1 Config file (instance2) Build profile (r-o design) Server-9119-MME- SNABCDEFG Config file (instancen) Frame N e.g. buildframe.ksh -p E880ViosPattern.def f build -c Server-9117-MMC-SNABCDEFG.cfg

62 Using buildframe with config files VIO clients e.g. clients on demand Config file (instance1) Config file (instance2) Build profile (r-o design) LPAR: tk_client77 Config file (instancen) LPAR: 1 e.g. buildframe.ksh -p E880ClientPattern f carve,vfcmap,wwpn -c tk_client77.cfg

63 IBM Advanced PowerVM Provisioning Toolkit v5 Extended Value Proposition Business Challenge Clients struggle with consistency in the build process when deploying VIO servers/clients. Differences in configuration depend on who built the systems and when. Attempts to develop in-house scripts and automation struggle with adding new features and support. Issues migrating partitions from one managed system to another. Solution Introduced the Advanced PowerVM Toolkit and process to accelerate the clients transition from Power6 Power7 Power8 across their infrastructure. Provide assessment, planning and on-site execution to implement the Toolkit and provisioning process based on client requirements. Work with customers to integrate Toolkit into their existing build process and automation. Work with customers to exploit latest Toolkit features: migration and/or DR solution creation. Key Benefits Reduced manual errors associated with current build process. Create a repeatable methodology for deploying new VIO servers and client LPARs. Integrated into your environment. Assist migrations to new hardware and technologies. Facilitate customized DR event response.

64 capture/create/build application - conceptual overview SAN Source/target see same SAN, Or SAN is replicated SAN LPAR1 LPAR2, LPAR3 LPAR4, LPAR5 LPAR6, LPARn LPAR1, LPAR2, LPAR3 LPAR4, LPAR5, LPAR6, LPARn Capture source LPAR profile & mappings Create build definition for target frame Build LPARs onto target frame A group of source LPARs, spanning some number of systems A migration or DR target frame for the group

65 IBM Advanced PowerVM Provisioning Toolkit v5 At a glance In 2013, Toolkit v3 added two new scripts: captureframe and createframe. captureframe.ksh Captures data for LPAR re-provisioning. Partition profiles, resources, mapping information collected from parent VIOs. Rich command line interface allows a variety of capture scenarios (e.g. by HMC, frame by frame, LPAR by LPAR, etc.) Or use a capture_profile to surgically specify individual LPARs with unique profiles to collect. NOTE: mksysb is not captured. createframe.ksh createframe is guided by a managedsystem_file which defines a target frame for a reprovisioning operation. managedsystem_file defines source-target mapping schema, can add or remove adapters, attributes, etc. Static file, prepared once, usually one per target frame/environment. createframe uses the captured data set to generate buildframe-compatible build profiles to map all or a subset of the captured LPARs into a *new* build_profile/config_file set. In-flight changes at a partition-level possible via partition_files. buildframe can then provision that group of LPARs onto the new target frame. Migration Custom DR solutions

66 IBM Advanced PowerVM Provisioning Toolkit v5 captureframe/createframe/buildframe data replication or same SAN [-c capture_profile] -m managed_system_file captureframe [-p partition_override file] target system definition ksh93 syssnap _files createframe summary_file re-provisioning data set ksh93 [-c config_file] User inputs ksh scripts Generated outputs Generated outputs -or- user input -p build_profile Java GUI buildframe ksh93,nim

67 Orchestrating LPAR Migrations with migrate_lpars.ksh <GROUP_NAME> <TARGET_FRAME> LPAR1 LPAR2, LPAR3 LPAR4, LPAR5 LPAR6, LPARn Group migration to Power8 capture,create,build, validate, shutdown,activate, test, decomm LPAR1, LPAR2, LPAR3 LPAR4, LPAR5, LPAR6, LPARn migrate_lpars.ksh migrate_lpars.ksh will migrate a group of LPARs, from one or more frames in a common Power frame ensemble (assumed managed by the same HMC) together to a mutual target frame via a series of atomic Provisioning Toolkit and other HMC-based operations. Usage: Execute a series of migrate_lpars.ksh operations until all such migratory LPAR groups have been moved to their respective target frames. Prereqs: SAN zoning and networks ready, firmware updates, P8-compatible AIX on migrating LPARs.

68 <GROUP_NAME> LPAR subset build_profile Specify source LPAR set: hmc, frames, lpars captureframe./syssnap_files/<group_name>/ createframe <GROUP_NAME>_client.def -g <GROUP_FILE> repro 1. capture syssnap _files 2. create [-l <PARTITION_LIST>] Restrict ops to subset of <GROUP_FILE> LPARs LPAR-specific overrides: entitlements,lpar_id, [-p <PARTITION_FILE>] Specify target environment: VIOs, mapping, globals e.g. simplified_remote_restart [-T <MANAGEDSYSTEM_FILE>] ms_filemaker.ksh [-c <CONFIG_FILE>] Specify ensemble-level or global changes e.g. VLANs, vswitches, LPAR names 3. build 4. validate buildframe chsysstate, chsyscfg migrate_lpars.ksh User input data Generated files delete_lpar.ks h toggle 5. shutdown 6. activate 7. test 8. decomm buildframe <GROUP_NAME> LPAR subset 2 on power8 <TARGET_FRAME> nominal flow

69 IBM Advanced PowerVM Provisioning Toolkit v5 migrate_lpars.ksh, ms_filemaker.ksh Toolkit v5 added three new scripts: migrate_lpars.ksh, ms_filemaker.ksh, delete_lpars.ksh (the latter is Use Case 4). migrate_lpars.ksh Wraps and (hopefully!) simplifies certain capture/create/build process scenarios. Used in beta at ~6 customers prior to release in Toolkit v5. Major financial customer using for about a year now, staging and completing migrations to P8. Same functional environment and directories as for Provisioning Toolkit core scripts. Uses same user(s) and SSH key(s). Assumes short, symbolic names to existing Toolkit scripts. For example: captureframe.ksh -> captureframe.ksh.v5.0 createframe.ksh -> createframe.ksh.v5.0 buildframe.ksh -> buildframe.ksh.v5.3 ms_filemaker.ksh Generates starter managedsystem_files for use with capture/create/build and migrate_lpars.ksh. Same functional environment and directories as for Provisioning Toolkit core scripts. Uses same user(s) and SSH key(s). Gather VIO and profile names. Query VIOs for lsnports information and attempt to determine NPIV mapping possibilities.

70 IBM Advanced PowerVM Provisioning Toolkit v5 delete_lpars.ksh Simplifying wrapper script around the powerful `buildframe f delete` function. May be used whether or not the Toolkit was used to build the LPAR in the first place. Dynamically generates build_profile for input to buildframe. Same functional environment and directories as for Provisioning Toolkit core scripts. Uses same user(s) and SSH key(s). Assumes short, symbolic names to existing Toolkit scripts. buildframe.ksh -> buildframe.ksh.v5.3 Simple calling convention: single HMC, single frame, prompt or no-prompt, single LPAR name Optional HMC user (default hscroot). Checks/requires that LPAR is Not Activated Unmaps and deletes any vfchost, vhost devices on parent VIO servers then saves VIO profiles if needed. Deletes LPAR and its profile../delete_lpar.ksh -u toolkit -h mghmc -m Server-9117-MMC-SN105C627 -n gw_client77

71 PowerDraw Value Proposition Business Challenge Customers wanted to visualize what has been configured on a given managed system. Customers wanted the ability to document key components of their PowerVM design. Customer wanted to graphically explore key elements of their PowerVM managed system design to query attributes or debug current problems or issues. Solution Introduced PowerDraw - providing the ability to gather and display key configuration information within the managed system. Layered approach allows for viewing details at all or different levels within the PowerVM design. Key Benefits Ability to view any level of configuration for documentation purposes. Ability to perform problem determination using a graphical approach with current configuration settings.

72 PowerDraw at a glance Implemented as drawframe.ksh: Capture and draw from existing design Capture all/any managed systems and/or partitions from a managing HMC Draw all/any managed systems and/or partitions from an HMC Merge with user defined data so PowerDraw becomes launch point for any systemrelated data Generated views: Network Storage SR-IOV / vnic All Implementation: SVG (Scalable Vector Graphics) with JavaScript Browser-based (e.g. FireFox) Consumer of draw files produced by PowerDraw Displays detailed data utilizing onclick, onmouseover, onmouseout

73 PowerDraw at a glance drawframe performs both capture and draw functions: drawframe f capture h HMC o outname drawframe f draw i inputcapture -o outgraphic

74 PowerDraw Network View

75 PowerDraw Storage View

76 PowerDraw VNIC View

77 PowerDraw All View

78 PowerDraw mouse over Managed System

79 PowerDraw mouse over LPAR

80 Your system is deployed perfectly, keep it that way with proactive monitoring ProMon

81 ProMon Why is Proactive Monitoring needed at all? All systems tend toward chaos Assuming that properly defined and documented build and change procedures are in place, people make mistakes Redundancy built into an environment and properly verified during initial installation only protects against the first failure. Most monitoring systems report catastrophic failures in the environment but under report failure of the standby systems.

82 ProMon Do you need Strategic Monitoring? Are your system administrators responsible for less than 15 LPAR each? Does your admin team routinely audit the current configuration of each LPAR? Does the storage and network configuration of your LPARs remain static over its lifetime? Do the system administrators, network administrators and storage administrators keep each other informed of the details of upcoming changes? Is adding a new LPAR to your environment a rare event? If you can honestly answer yes to all of the above questions, congratulations, you have good command and control of your environment. If not ProMon can help.

83 Murphy s Law Another way of looking at things The 2 nd law of thermodynamics states that the entropy of any isolated system always increases. Murphy s Law: If anything can go wrong, it will. O Toole s commentary on Murphy s Law: Murphy was an optimist.

84 Use Case 1 for ProMon Time to upgrade the VIOS servers on all the frames. Are you sure none of the LPARs will lose access to one or more of their disks? Maybe we should wait until we have had a chance to verify the system better.

85 Use Case 2 for ProMon Regular use can alert the system administrator team that changes in the SAN or network subsystems have had an unanticipated effect on the LPAR redundancy and availability.

86 Use Case 3 for ProMon ProMon provides a regular method of checking the systems to insure that normal system maintenance has not adversely impacted the overall system performance.

87 ProMon Customer case study There are over 35 customers using it now. A financial customer started with 27,000+ errors spread across 600+ LPAR. At that time they were unable to LPM any of their LPARs. After correcting issues for several months, they were below 15,000 errors and able to perform LPM on about 40% of their environment.

88 System x3550 M3 System x3550 M3 Power 780 Power 780 Power 780 Power 780 Power 780 Power 780 Power 780 Power 780 Power 780 Typical ProMon environment HMC Power 795 Power 795 Centralized Monitoring Server

89 ProMon Audits AIX LPAR Active access to storage via multiple paths OS inconsistency check Unused/wasted storage check Storage devices without multiple access paths Required settings on all storage, including reserve_policy and hcheck_mode NFS file system verification Verification of backups started VIOS Redundant access to any storage presented to LPARs thru VSCSI PowerPath configurations, if present Audits SEA configurations for: Transmit errors Failed devices in aggregation Amount of data sent/received

90 Required File System and Backup Software SHARED_FS="" ### Set Backup software in use ### ### BACKUP_SOFTWARE="Avamar" ### BACKUP_SOFTWARE="DataDomain" ### BACKUP_SOFTWARE="CommVault" ### BACKUP_SOFTWARE="TSM" BACKUP_SOFTWARE=""

91 Monitor Software Up and Running ### Configure to check for monitoring software active ### ### Valid options are Nagios OpenView Ganglia and OpenView ### ### MONITOR="Nagios" ### MONITOR="OpenView" ### MONITOR="BMCPatrol" ### MONITOR="Ganglia" ### MONITOR="ITM6 MONITOR=""

92 Required Disk Attributes ### Configure required Disk Attributes #### HC_MODE="enabled" HC_INTERVAL="60" RESERVE_POLICY="no_reserve" QUEUEDEPTH="8" VSCSI_ALGORITHM="fail_over" FSCSI_ALGORITHM="round_robin" NBR_VSCSI_PATH="2" NBR_FSCSI_PATH="4"

93 Critical System Processes Are the RSCT services running? If I can t DLPAR more CPU to the e-commerce server we will never get through the day Is the Time Synchronization Daemon running? Please don t make me follow a transaction across multiple LPARs if they can t agree on the time.

94 Sample LPAR Report

95 Sample VIOS Report Starting VIOS audits at 0201 on florida-vio2 There is/are 4 configured Shared Ethernet Adapters on florida-vio2 functioning normally florida-vio2 For ent12 there were 0 MB sent and 1454 MB received with 0 xmit errors and 0 receive errors florida-vio2 Maximum Transmit Queue was 0 packets with no overflow errors florida-vio2 For ent20 there were 0 MB sent and 26 MB received with 0 xmit errors and 0 receive errors florida-vio2 Maximum Transmit Queue was 0 packets with no overflow errors florida-vio2, For ent32 there were 9096 MB sent and MB received with xmit errors and 568 receive errors florida-vio2 Maximum Transmit Queue was 4 packets with no overflow errors florida-vio2, Dynamic tracking not set on fscsi0 florida-vio2, Fast_Fail attribute not set on fscsi0 florida-vio2, Dynamic tracking not set on fscsi7 florida-vio2, Fast_Fail attribute not set on fscsi7 florida-vio2, Check for failed adapter in SEA ent16 florida-vio2, Unmirrored LV found rootvg audit_lv jfs2 /audit florida-vio1 There is/are 4 configured Shared Ethernet Adapters on florida-vio1 functioning normally florida-vio1, For ent32 there were 0 MB sent and 1218 MB received with 0 xmit errors and 568 receive errors VIOS audits complete at 0222 on

96 ProMon Commands used: lsdev lsattr lspv lsvg fcstat lspath powermt ioscli entstat ps grep ssh lppchk

97 Once a day, against the entire environment, or You can always run an audit against a single LPAR at any time. Provided documentation details all of the options available.

98 ProMon Identifying non-standard issues

99 Notice and disclaimers Copyright 2017 by International Business Machines Corporation (IBM). No part of this document may be reproduced or transmitted in any form without written permission from IBM. U.S. Government Users Restricted Rights use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM. Information in these presentations (including information relating to products that have not yet been announced by IBM) has been reviewed for accuracy as of the date of initial publication and could include unintentional technical or typographical errors. IBM shall have no responsibility to update this information. This document is distributed as is without any warranty, either express or implied. In no event shall IBM be liable for any damage arising from the use of this information, including but not limited to, loss of data, business interruption, loss of profit or loss of opportunity. IBM products and services are warranted according to the terms and conditions of the agreements under which they are provided. IBM products are manufactured from new parts or new and used parts. In some cases, a product may not be new and may have been previously installed. Regardless, our warranty terms apply. Any statements regarding IBM's future direction, intent or product plans are subject to change or withdrawal without notice. Performance data contained herein was generally obtained in a controlled, isolated environments. Customer examples are presented as illustrations of how those customers have used IBM products and the results they may have achieved. Actual performance, cost, savings or other results in other operating environments may vary. References in this document to IBM products, programs, or services does not imply that IBM intends to make such products, programs or services available in all countries in which IBM operates or does business. Workshops, sessions and associated materials may have been prepared by independent session speakers, and do not necessarily reflect the views of IBM. All materials and discussions are provided for informational purposes only, and are neither intended to, nor shall constitute legal or other guidance or advice to any individual participant or their specific situation. It is the customer s responsibility to insure its own compliance with legal requirements and to obtain advice of competent legal counsel as to the identification and interpretation of any relevant laws and regulatory requirements that may affect the customer s business and any actions the customer may need to take to comply with such laws. IBM does not provide legal advice or represent or warrant that its services or products will ensure that the customer is in compliance with any law.

100 Notice and disclaimers continued Information concerning non-ibm products was obtained from the suppliers of those products, their published announcements or other publicly available sources. IBM has not tested those products in connection with this publication and cannot confirm the accuracy of performance, compatibility or any other claims related to non-ibm products. Questions on the capabilities of non-ibm products should be addressed to the suppliers of those products. IBM does not warrant the quality of any third-party products, or the ability of any such third-party products to interoperate with IBM s products. IBM expressly disclaims all warranties, expressed or implied, including but not limited to, the implied warranties of merchantability and fitness for a particular, purpose. The provision of the information contained herein is not intended to, and does not, grant any right or license under any IBM patents, copyrights, trademarks or other intellectual property right. IBM, the IBM logo, ibm.com, AIX, BigInsights, Bluemix, CICS, Easy Tier, FlashCopy, FlashSystem, GDPS, GPFS, Guardium, HyperSwap, IBM Cloud Managed Services, IBM Elastic Storage, IBM FlashCore, IBM FlashSystem, IBM MobileFirst, IBM Power Systems, IBM PureSystems, IBM Spectrum, IBM Spectrum Accelerate, IBM Spectrum Archive, IBM Spectrum Control, IBM Spectrum Protect, IBM Spectrum Scale, IBM Spectrum Storage, IBM Spectrum Virtualize, IBM Watson, IBM z Systems, IBM z13, IMS, InfoSphere, Linear Tape File System, OMEGAMON, OpenPower, Parallel Sysplex, Power, POWER, POWER4, POWER7, POWER8, Power Series, Power Systems, Power Systems Software, PowerHA, PowerLinux, PowerVM, PureApplica- tion, RACF, Real-time Compression, Redbooks, RMF, SPSS, Storwize, Symphony, SystemMirror, System Storage, Tivoli, WebSphere, XIV, z Systems, z/os, z/vm, z/vse, zenterprise and zsecure are trademarks of International Business Machines Corporation, registered in many jurisdictions worldwide. Other product and service names might be trademarks of IBM or other companies. A current list of IBM trademarks is available on the Web at "Copyright and trademark information" at: Linux is a registered trademark of Linus Torvalds in the United States, other countries, or both. Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle and/or its affiliates.