ASR9xxx IOS XR Release Upgrade/Downgrade Procedure

Transcription

1 ASR9xxx IOS XR Release Upgrade/Downgrade Procedure Table of Contents 1. Summary Perform Pre-Upgrade Checks Mandatory SMUs (Upgrade/Downgrade): Selection of packages for Upgrade Upgrade to IOS XR Release ISSU SMU installation Perform Post-Upgrade/Downgrade Checks FPD Upgrade Mandatory SMUs for Downgrade Newly Released Hardware Matrix Downgrade from IOS XR Release Caveats Appendix Summary The following is a quick summary of steps. Please see the following sections for more details a. Download the right image tarball from the CCO on to a tftp server. The image tarball on CCO cannot be used AS-IS. CCO image tarball for will need to be downloaded and only the relevant SMU s taken from it and tarred into another tarball for use on the router. b. Install add/activate all the mandatory SMU s required to be active on the current release before being upgraded to The list of SMUs to be used is mentioned in the mandatory SMU s section of this document. The mandatory SMUs are available in the tarfile downloaded from CCO. c. Check if firmware upgrade is needed prior to the upgrade to and upgrade as necessary. If firmware was upgraded, then DO NOT reload the box in this step. It will be reloaded as part of the package activation step. d. Select appropriate packages from CCO image tarfile and create a separate tar file specific to your environment. Install add the tar file. Refer to the link below and add

2 any C-SMU s or recommended SMU s for once tar file is added. 0_423_Release_Recommendation.pdf e. Once install add has completed successfully, Install activate packages along with the SMU that was added. Once the router has reloaded and come back up with packages, perform install commit. f. Check if firmware upgrade is needed after upgrade to and upgrade as necessary. If firmware was upgraded, then reload the router for firmware changes to take effect. Please refer the firmware upgrade section for further details. 2. Perform Pre-Upgrade Checks Sl. No Commands Verification to be performed 1 Admin show platform verify that all Nodes are in "IOS XR RUN" state, SPAs in "OK" state, Fan Tray and Power Modules are in READY state 2 Admin show redundancy verify that Active & Standby RSP are available and in ACTIVE role & is ready states 3 Show interface summary Verify that all the concerned interfaces are Up 4 Admin show install active / inactive / committed summary Verify that proper set of packages are active / inactive / committed. 5 Show filesystem Check the actual disk0: size to ensure that sufficient disk space is available for the upgrade. A minimum of 700 MB free disk space is needed for upgrade to succeed. 6 dir bootflash: location 0/RSP0/CPU0 Bootflash device should not be used to save user specific files or configurations. Extraneous files such as configs files should be removed from bootflash. Copy the running-configuration to a temporary storage location eg: tftp, compactflash:, harddisk: Copy the admin running-configuration to a temporary storage location, eg: tftp, compactflash:, harddisk: 7 copy running-config tftp:// running_config.txt 8 Admin copy running-config tftp:// admin-running_config.txt 9 ping <tftp_server_address> If unable to reach the Mgmt LAN or TFTP_SERVER, please check the Mgmt Port states and route configuration for Mgmt LAN 10 no mirror location 0/rsp0/cpu0 Disable disk mirroring. if enabled on the router, before going to disk0:disk1: next step. Disk mirroring can increase the upgrade time. 11 mirror location 0/rsp0/cpu0 Enable disk mirroring after the upgrade has completed. disk0:disk1: 12 Admin install remove inactive Remove old versions of the operating system and old SMU files from router, use the admin-command. The install remove inactive test sync commands can be used first to show which packages will be removed from the disk 3. Mandatory SMUs (Upgrade/Downgrade): Release Install Mandatory SMUs(p) Install Mandatory SMUs(px) Upgrade SMU s Downgrade SMU s Upgrade SMU s Downgrade SMU s CSCtk65746 CSCtk65746 N/A N/A CSCtj90504, CSCtj90504, N/A N/A Cisco Systems, Inc. Page 2

3 CSCto05713 CSCto No No N/A N/A No No N/A N/A No No N/A N/A CSCty67156 CSCty67156 N/A N/A CSCub41271* CSCub41271* CSCtx89601, CSCtx28089, CSCtl84822, CSCua20673, CSCua50217 CSCtx89601, CSCtx28089, CSCtl84822, CSCua50217, CSCtz47970, CSCtz35514, CSCtz65963, CSCty98459 CSCtx89601, CSCtx28089, CSCtl84822, CSCua50217, CSCtz47970, CSCtz35514, CSCtz65963, CSCty98459 CSCtx89601, CSCtx28089, CSCtl84822, CSCua20673, CSCua No No No No No No No No *Install CSCub41271 and follow the instructions mentioned in the RSP2 disk repartition guide. This guide is added as part of the README file provided along with the SMU. Once the repartiton is completed, remove the repartition SMU CSCub After disk repartitioning, install the rest of the SMU s and perform the upgrade to Note: This repartition SMU might be applicable for other releases based on how many SMU s the customer has installed and are active on current release. The repartition SMU (CSCub41271) should be used if disk space required for upgrade is inadequate for upgrade to succeed. Here is some guideline on when to use the repartitioning SMU. Step1: check for disk usage on customer box. Step2: Disk space needed for base software + optional pies is: ~740 MB (excluding FPD package and any recommended SMU s) Step3: Space required for any mandatory SMU s on target release (4.2.3): Lookup the CSMU pack size. Step4: Check total disk space on the disk0: from customer router. (this should be approx. 1.6GB) If Total [(Step1 + Step2 + Step3) < (Step 4)] then repartition SMU is not needed. If Total {[(Step1 + Step2 + Step3) < (Step 4)] >=0MB and <=300MB,} then repartition SMU is warranted for upgrade to succeed. Note: If the disk space is still not adequate after repartitioning. Step3 can be skipped in the event upgrade is possible with removal of step3. Step3 can be added once is active on the box. In that case, box might have to go through an additional reload. 4. Selection of packages for Upgrade 1) Download the tar file from CCO and untar it onto a tftp location. Depending on optional packages (eg: mpls, mcast, optic, doc, fpd, nv, video, services, mgbl) already active on the disk, the user will need to select those specific packages for upgrade to ) The tar file from CCO can be untarred via following command tar xvf ASR9K-iosxr-p-k tar. The difference in the ASR9K-iosxr-p-k tar and ASR9K-iosxr-p tar should be addition of the crypto pie in the former and rest all the packages should be same. This holds true for the px packages aswell. Cisco Systems, Inc. Page 3

4 3) Now select the base package and individually select optional packages that are needed for your environment. For Eg: If the current active packages contain optional pies such as mpls, doc, mgbl, k9sec, optic pies, then the same packages will need to be added while upgrading to Include the Recommended SMU s provided in the link below along with other packages being selected for n.pdf There are 2 ways in which these packages can be added to the router. a. One way is to add the packages individually Install add source <source path> asr9k-mpls-p.pie asr9k-doc-p.pie asr9kmgbl-p.pie asr9k-k9sec-p.pie asr9k-optics-p.pie synchronous b. Another way is to create a tar file and add it using tar option. Create a separate directory on tftp and copy all the above 5 packages and base package. Creating a tar file can be done via following command from that directory tar cvf ASR9k-p tar *. Once the file has been tarred, the following will be contents of the tar file. The tar file can then be added to router via following command Install add tar <tar file source location/file name> synchronous Adding tar file to router Install add tar <tar file source/asr9k-p tar> synchronous 5. Upgrade to IOS XR Release Note1: All install operations should be done in admin mode. Check for system Stability (section 3) 1) Add the required packages to disk using one of the options (source, tar option, individual packages (specifying source-path for each package)): I. RP/0/RSP0/CPU0:router(admin)#install add <source>/<source-path>/ <pie-1> <pie-2> <pie-3> <pie-n> sync II. III. RP/0/RSP0/CPU0:router (admin)# install add tar <source-path>/<tar-file> sync RP/0/RSP0/CPU0:router(admin)# install add <source-path>/asr9k-mini-p.pie <sourcepath>/asr9k-mcast-p.pie <source-path>/asr9k-mgbl-p.pie <source-path>/asr9kmpls-p.pie <source-path>/asr9k-k9sec-p.pie <source-path>/asr9k-video-p sync Note 2: In case there are any other optional packages installed with prior release, new package list should match or supersede the prior release package list and need to be added using the above-mentioned command. Otherwise all the conflicting optional packages have to be deactivated before the upgrade/downgrade [followed by install commit operation]. Side effect of this is loss of the configuration supported by the pie. Cisco Systems, Inc. Page 4

5 2) Active/Inactive/Committed packages summary. Inactive Packages Active Packages Committed Packages Inactive Packages: disk0:asr9k-mini-px disk0:asr9k-mcast-px disk0:asr9k-mgbl-px disk0:asr9k-mpls-px disk0:asr9k-video-px disk0:asr9k-k9sec-px disk0:asr9k-optic-px disk0:asr9k-doc-px Active Packages: disk0:asr9k-mini-px disk0:asr9k-mcast-px disk0:asr9k-mgbl-px disk0:asr9k-mpls-px disk0:asr9k-video-px disk0:asr9k-k9sec-px disk0:asr9k-optic-px disk0:asr9k-doc-px Committed Packages: disk0:asr9k-mini-px disk0:asr9k-mcast-px disk0:asr9k-mgbl-px disk0:asr9k-mpls-px disk0:asr9k-video-px disk0:asr9k-k9sec-px disk0:asr9k-optic-px disk0:asr9k-doc-px ) Perform Test Activation of the inactive packages, using the following activate test CLI: RP/0/RSP0/CPU0:router(admin)# install activate <source> :<pie-1><source> :<pie-2> <source> :<pie-n> test sync Note 3: Testing the activation will give you a preview of the activation. No actual changes will be made when test option is used. Any config that is incompatible with the new version being activated will be identified via show configuration removed command and the same can be reapplied via load config removed <config>.cfg command. Activate inactive packages added in step 1 using one of the following commands: RP/0/RSP0/CPU0:router(admin)# install activate <source> :<pie-1><source> :<pie-2> <source> :<pie-n> sync RP/0/RSP0/CPU0:router (admin)# install activate disk0 :*4.2.3* sync Note 4: All install operations to be performed from Admin mode. The Router will reload at the end of activation to start using the new packages. This reload operation will impact traffic for small duration. Under idle conditions, this operation may take at least 20 minutes to complete. *Warning message: The following packages are specified multiple times could be observed with wild card option. 6. ISSU SMU installation Prior to the introduction of ISSU (In Service Software Upgrade), SMUs could be built as either "process restart" SMU or "reload" SMU. When installing these SMUs, the install operation will require process restart (least amount of impact), or system reload (larger impact). ISSU helps improve the experience and system availability as it allows a subset of these reload SMUs to be built as ISSU SMUs. The impact of installing ISSU SMUs are less than installing reload SMUs as Cisco Systems, Inc. Page 5

6 traffic outage should be in seconds vs amount of time it takes to reload a system. So in summary we have 3 types of SMUs in 4.2.x. Restart SMU(s) - Install manager will prompt user to continue operation as 'Parallel Process Restart' Reload SMU(s) - Install manager with prompt user to continue operation as 'Parallel Reload' ISSU SMU(s) - 'In-service Upgrade' Prior to performing an ISSU install or upgrade, take note of the following: - Verify NSF/NSR is enabled for all routing protocols where possible - Verify all protocol (e.g. BGP) hold time timer are adquitly configured 1) Verify SMU was built for ISSU. SMUs cannot be activated with ISSU method if it was not built for ISSU. ISSU SMUs on CCO are clearly marked ISSU capable in the SMU download folder. In addition one can use "show install package <pie> detail" from the router and verify the 'component restart type' is "ISSU (quick) warm reload". It is normal for the 'root package' restart type to be "parallel impacted processes restart". 2) Add ISSU SMU to the router, this is straight forward and is just a standard install add; Install add tar <source/asr9k-p pie> synchronous 3) Begin ISSU SMU upgrade operation by specifying the "issu" keyword with the "install activate" command. After pre-check, install director will inform how each node will be impacted and ask if the install operation should be continued in 'prompted mode', 'unprompted mode', or if the operation should be aborted. In 'unprompted' mode, install director will perform the 'load', 'run', and 'complete' phases sequentially without further user input. In 'prompted' mode, install director will pause between each phase and user input is required before each phase is started. It is recommended that Customers perform issu operation in unprompted mode. install activate <disk0:smu> issu sync or install activate <disk0:smu> issu <prompt-level> none (Another variation of running issu in unprompted mode) 4) Choose between 'prompted' or 'unprompted' mode. If we continue in 'prompted' mode. The first phase of ISSU installation is 'load' phase. Time needed for 'load' phase depends on system card scale. 5) During any point of the ISSU operation, you can do "show install request" and/or "show issu" to check the progress of the ISSU operation. At the completion of 'load' phase, see output of "show install request" for instruction on how to start the 'run' phase. 6) Start 'run' phase with the install instruction from the previous step. (admin)#install operation 3 run Cisco Systems, Inc. Page 6

7 7) During 'run' phase RP switchover will occur. After switchover occurs, progress can be monitored from the new active RP. After the completion of the 'run' phase, do "show install request" for the install instruction to start the 'complete' phase. 8) Finish the ISSU operation by starting the 'complete' phase (admin)#install operation 3 complete Note: Configuration will be locked once ISSU load phase starts and will be unlocked once ISSU has completed. If ISSU SMU install is aborted during load phase, router will be reverted back to its original state and it will be hitless. If ISSU is aborted during RUN Phase, router will be reloaded as part of the recovery process in bringing it back to original state. For more information on ISSU SMU s, please refer to the tech doc published on CCO Perform Post-Upgrade/Downgrade Checks Sl. No Commands Verification to be performed 1 Admin show platform verify that all Nodes are in "IOS XR RUN" state, SPAs in "OK" state, Fan Tray and Power Modules are in READY state 2 Admin show redundancy verify that Active & Standby RSP are available and in ACTIVE role & is ready states 3 Admin show install active/inactive/committed summary After Upgrade has been successful, Perform install comit from admin mode. Verify that proper set of packages are in active, inactive and commited state. See section Show configuration failed startup Check to see if there were any failed startup config. 5 Clear config inconsistency In an event there is a config loss, perform Clear configuration inconsistencies. 6 Show install verify packages / install verify package / install verify package repair Upon Sucessful upgrade check for package and image integrity. Any anamolies found will need to be repaired using the command. install verify package repair. 7 Cfs check Verify/ fix configuration file system. 8 Show interface summary Verify that all the concerned interfaces are Up 9 Show hw-module fpd location all Verify if firmware needs to be updated right after the successful Upgrade. If Firmware upgrade was needed, upgrade it and reload the LC/router for firmware changes to take effect. 10 Disk Cleanup (optional) Once software upgrade or downgrade has been completed, disk space can be recovered by removing any inactive packages that are no longer needed (if the packages are required at a later time, they can be re-added): Note 1: Verify system stability through commands described under Check System Stability [See Section 3 for details]. If system issues are detected or if the upgrade needs to be backed out for Cisco Systems, Inc. Page 7

8 any reason, please follow the steps described in Downgrade [See Section 11 for details] to rollback the software to the starting point FPD Upgrade After running the install commit command on upgrade to software, run the show hw-module fpd location all command to check which firmware files need to be upgraded, by inspecting the Upg/Dng column. If there is any Yes marked, upgrade is required. Note: For all upgrades from prior releases to release 4.2.3, It is recommended that user disables the Auto FPD upgrade command in admin mode. Once the router is upgraded to 4.2.3, please check for fpd firmware and upgrade to the latest firmware manually via the command upgrade hw-module fpd all location all. In the event that fpd autoupgrade is already configured in admin mode, please perform a no version of the command to unconfigure it, prior to the upgrade to Steps to Upgrade FPD firmware [R4.2.3]: 1) Run the show hw-module fpd location all command to check which firmware files need to be upgraded. RP/0/RSP0/CPU0:router(admin)# show hw-module fpd location all ========================================================= Existing Field Programmable Devices ========================================= HW Current SW Upg Location Card Type Version Type Subtype Inst Version Dng? ========================================================= 0/RSP0/CPU0 A9K-RSP-4G 4.8 lc fpga Yes lc fpga No lc fpga No lc cbc No lc fpga No lc hsbi No lc rommon Yes /RSP0/CPU0 ASR-9010-FAN 1.0 lc cbc No /RSP0/CPU0 ASR-9010-FAN 1.0 lc cbc No Snipped Note1: In the output above, the column Upg Dng? points to the down-rev firmware software versions, which need to be upgraded. 2) Issue the following command to upgrade fpd: RP/0/RSP0/CPU0:router(admin)#upgrade hw-module fpd all location all Note 2: Except CBC update, router reload is required after running the upgrade hw-module fpd all location all command, to make the changes in effect. Cisco Systems, Inc. Page 8

9 Note 3:No reload is required after running the upgrade hw-module fpd cbc location all command. The new CBC firmware will be active. The software automatically resets the local CAN Bus. 9. Mandatory SMUs for Downgrade Please refer to Section 3 for SMU s needed during downgrade from release to previous releases (4.0.x, 4.1.x, 4.2.x) 10. Newly Released Hardware Matrix Sl. No Release New Hardware Introduced A9K-RSP440-SE, A9K-RSP440-TR A9K-24x10GE-SE, A9K-24x10GE-TR A9K-MOD80-TR, A9K-MOD80-SE A9K-MPA-4x10GE, A9K-MPA-20x1GE A9K-2x100GE-SE, A9K-2x100GE-TR ASR-9001 Chassis A9K-MPA-2x10GE, A9K-MPA-2x40GE A9K-MOD160-TR, A9K-MOD160-SE ASR-9922 Chassis A9K-36x10GE-SE, A9K-36x10GE-TR A9K-1x100GE-SE, A9K-1x100GE-TR A9K-MPA-1x40GE 11. Downgrade from IOS XR Release Note 1: Downgrade requires atleast equal number of matching packages for downgrade to be successful. Any additional packages that are not needed need to be deactivated first (example: asr9k-adv-video.pie and/or asr9k-doc.pies). Note 2: Before downgrade to 4.0.x or 4.1.x, the Doc pie package (asr9k-doc-p-4.2.0) needs to be deactivated and removed. Before downgrade to 4.0.x or 4.1.x, any installed newly supported line card/spas need to be removed. Note 3: Before downgrade to 4.0.x or 4.1.x, we recommend to backup any installed Software licenses. Refer to Section 9 for list of the new HW Linecards, SPAs in that particular release. Note 4: Downgrade can be performed via 2 methods. install activate or rollback via install rollback <id> option. Both the options requires an least equal number of matching packages to be present as part of inactive packages. If packages aren t present, they will need to be added to the router before proceeding with the downgrade. One of the Install activate Method Install Rollback Method install activate disk0:<mini package>, List of rollback points & Identify the Cisco Systems, Inc. Page 9

10 disk0:<pie-1>.. disk0:<pie-n> sync install activate disk0:*4.0.x* sync (*Warning message: The following packages are specified multiple times: may be observed.) rollback point with these commands. # show install rollback? # show install rollback <rollback point> if the packages listed in the rollback point are not present as part of the inactive package list, then you will need to add the pies/packages to the router via install add <mini pie + optional packages listed in the rollback id> before downgrade via rollback method is attempted. Test Rollback procedure with: # install rollback to <rollback point> sync test Perform rollback with: # install rollback to <rollback point> sync Note 5: The router will reload at the end of activation/rollback to start using the new packages. Under idle conditions, this operation may take at least 30 minutes to complete. Note 6: Execute Section 6 and ensure that there are no isues with the downgrade. Note 7: Force downgrade fpd once downgrade is successful. (detailed in Section 7): RP/0/RSP0/CPU0:router(admin)#upgrade hw-module fpd all force location all Note: Except CBC, router reload is requested after running the upgrade hw-module fpd all force location all command. Upgrade firmware (mandatory) ROMMON, CBC, HSBI, CPLD, and FPGA firmware needs to be upgraded after the image installation on the system. For detailed upgrade procedure please refer IOS XR Firmware Upgrade Guide document which can be accessed at: Caveats 1) In Cisco ASR 9000 Series Router Software Release 4.0.0, the minimum configurable logging buffered size has been increased to Any configuration with a value less than fails to upgrade to Release Run the show configuration failed startup command on startup to display the failed configuration. Workaround: Prior to upgrading to Release 4.2.3, set the logging buffer size to a value of or greater (logging buffered ). 13. Appendix 1) Show fpd package and show version outputs can be found in the document link referenced below: s/reln_a9k_423.pdf 2) Choose asr9k-mini-p.pie for RSP routers and asr9k-mini-px.pie for RSP440 based routers. P = RSP, px =RSP440, RPi Cisco Systems, Inc. Page 10