Upgrading and Downgrading Firmware

Transcription

1 Firmware Management, page 1 Upgrading and Downgrading Considerations, page 3 Upgrading the Fabric, page 4 Firmware Management ACME Inc., in partnership with Cisco, has evaluated the requirements for their deployment based on the software features required, the support for the hardware platforms they have selected, and the maturity of the software releases. They have selected a target version of software for their deployment. Additionally, they have put a proactive plan in place to revisit this decision periodically to determine if future upgrades are required. Firmware Versions The software versions for Cisco Application Centric Infrastructure (ACI) are listed in the following format: major.minor(maintenance) major Represents major changes in the product architecture, platform, or features content. minor Represents a minor release with new software features. maintenance Represents bug fixes to a feature release of Application Policy Infrastructure Controller (APIC). This changes when there are fixes for product defects in the software, but no additional new features. The following example shows some APIC versions: 1.0(1e) 1.1(1j) 1.2(1i) Both the software for the APIC and the fabric nodes are denoted by the same version scheme. For example, the APIC 1.2(1i) release corresponds to the switch software 11.2(1i) release. The release notes for the APIC versions reference the corresponding switch versions, and vice versa. 1

2 Firmware Components All components of the ACI infrastructure including the APIC, leaf switches, and spine switches, should be on the same version. While at the time of upgrading, disparate versions may exist between APIC and the switches, do not operate the fabric for extended periods of time in this state. When considering the impact and risk of upgrading, you can assume that a maintenance version upgrade, such as upgrading from 1.1(1j) to 1.1(1o), will have less impact than a major/minor version upgrade, as there will be only bug fixes and no new features added. Firmware Components There are three main components that can be upgraded: Switches (leaf and spine) Application Policy Infrastructure Controller (APIC) Catalog firmware Firmware Policies Firmware Groups Firmware group policies on the Application Policy Infrastructure Controller (APIC) define the group of nodes on which firmware will be upgraded. For most deployments, a single firmware group is adequate. Maintenance Groups Maintenance group policies define a group of switches that will be jointly upgraded to the associated firmware set. Maintenance groups can be upgraded on demand or according to a schedule, making it possible to defer an upgrade task to a business maintenance window. Typically, there are two maintenance groups, each containing a set of leafs and spines. Each maintenance group is upgraded separately. Controller Firmware The APIC firmware policy applies to all controllers in the cluster, but the upgrade is always done sequentially. The APIC GUI provides real-time status information about firmware upgrades. Controller firmware policies can be upgraded on demand or according to a schedule. Catalog Firmware Each firmware image includes a compatibility catalog that identifies supported switch models. The APIC maintains a catalog of the firmware images, switch types, and models that are allowed to use that firmware 2

3 Upgrading and Downgrading Considerations image. The APIC, which performs image management, has an image repository for compatibility catalogs, APIC firmware images, and switch images. Figure 1: Firmware Upgrade Policy Relationships Upgrading and Downgrading Considerations Before starting the upgrade or downgrade process, verify the following things: Application Policy Infrastructure Controller (APIC) cluster health Before starting the upgrade process, your controllers should be in good health. Verify that the health state of all of the controllers in the cluster are Fully Fit before you proceed. To resolve issues for controllers that are not fully fit see the Troubleshooting Cisco Application Centric Infrastructure document. Configuration backup Before starting any upgrade, always export your configuration to an external source. For information about exporting configurations, see the "Import and Export Policies." Permissions A user must have the fabric administrator role to perform firmware upgrade tasks. Verify free space Confirm that the /firmware partition is not filled beyond 75%. If the partition is filled beyond 75%, you might be required to remove some unused firmware files from the repository to accommodate the compressed image as well as provide adequate space to extract the image. The APIC automatically extracts the image. Upgrade order Typically, the controllers should be upgraded first, followed by the switch nodes. Always refer to the relevant release notes of the destination firmware version for any changes to this order. Maintenance windows Although it is possible to upgrade the fabric without impacting the dataplane, you should perform an upgrade during a scheduled maintenance window according to your change control policy. This window should account for any unforeseen issues that might arise during the upgrade, and allocate enough time to troubleshoot or perform a rollback. Maintenance groups To help minimize the impact to hosts during an upgrade, you should set up at least two separate maintenance groups. A common separation is by odd and even node IDs. Assuming that your hosts are dual-connected to at least one odd and one even leaf node, there should not be any impact to your hosts. Maintenance group creation is covered in detail later in the chapter. Another consideration is that your leaf vpc pairs should contain one odd and one even node. Upgrading a fabric with the Application Virtual Switch (AVS) deployed The AVS software is not specifically tied to the APIC or switch software version. 3

4 Upgrading the Fabric Device packages Device packages are not always tied to the APIC software. You can confirm the device compatibility for Layer 4 to Layer 7 devices using the online Cisco Application Centric Infrastructure (ACI) Compatibility tool. Upgrading the Fabric Downloading the Firmware Images Using the GUI You must download both the controller software package and switch software package for the Application Policy Infrastructure Controller (APIC) from Cisco.com. Procedure Step 1 Step 2 Step 3 Step 4 Step 5 On the menu bar, choose Admin > Firmware. In the Navigation Pane, choose Fabric Node Firmware. In the Work pane, the list of all switches in the fabric and the status of when the firmware was last upgraded are displayed. In the Navigation Pane, choose Download Tasks. In the Work pane, choose Actions > Create Firmware Download Task. In the Create Firmware Download Task dialog box, perform the following actions: a) In the Source Name field, enter a name for the switch image, such as "apic_1.2.1i". b) For the Protocol radio buttons, click the Secure copy or HTTP radio button. c) In the URL field, enter the URL from where the image must be downloaded. HTTP Example: SCP Example: :/tmp/aci-firmware/aci-apic-dk i.iso For SCP, enter your username and password. d) Click Submit. Step 6 Step 7 Step 8 Step 9 Step 10 Step 11 (Optional) You can instead upload the image from your local machine by performing the following actions: a) In the Navigation pane, choose Download Tasks. b) Right click and choose Upload Firmware to APIC. c) Browse to the image that is saved on your local machine. d) Click Submit. In the Navigation Pane, choose Download Tasks. In the Work pane, choose the Operational tab to view the download status of the images. Repeat this procedure for the switch image. After the download reaches 100%, in the Navigation pane, choose Firmware Repository. In the Work pane, choose the Images tab to view the downloaded version numbers and image sizes. 4

5 Downloading the Firmware Images Using the Object Model CLI Downloading the Firmware Images Using the Object Model CLI You must download both the controller software package and switch software package for the Application Policy Infrastructure Controller (APIC) from Cisco.com. Procedure Step 1 Step 2 Step 3 Step 4 SSH to an APIC in the fabric. # ssh admin@node_name Switch to the object model CLI: apic1# bash admin@apic1:~> Place the image into the image repository: admin@apic1:~> firmware add ver_no.iso Verify that the software has been added to the repository: admin@apic1:~> firmware list Name : aci-apic-dk i.bin Type : controller Version : 1.2(1i) Upgrading an APIC Using the GUI The catalog firmware image is upgraded when an Application Policy Infrastructure Controller (APIC) image is upgraded. You do not need to upgrade the catalog firmware image separately. To upgrade an APIC: 1 On the menu bar, choose Admin > Firmware. 2 In the Navigation pane, click Controller Firmware. 3 In the Work pane, choose Actions > Upgrade Controller Firmware Policy. 4 In the Upgrade Controller Firmware Policy dialog box, perform the following actions: a b c In the Target Firmware Version field, from the drop-down list, choose the image version to which you want to upgrade. In the Apply Policy field, click the Apply now radio button. Alternately, you can apply a schedule policy if you wish to defer the task to a specific date/time. Click Submit to complete the task. The Status dialog box displays the "Changes Saved Successfully" message, and the upgrade process begins. The APICs are upgraded serially so that the APIC cluster is available during the upgrade. 5 Verify the status of the upgrade in the Work pane. Each APIC takes about 10 minutes to upgrade. Once an APIC image is upgraded, it drops from the cluster and reboots with the newer version while the other APICs in the cluster are still operational. Once the APIC reboots, it joins the cluster again. Then, the cluster converges and the next APIC image starts to 5

6 Upgrading an APIC Using the NX-OS-Style CLI upgrade. If the cluster does not immediately converge, and is not fully fit, the upgrade will wait until the cluster converges and is Fully Fit. During this period, a "Waiting for Cluster Convergence" message is displayed in the Status column for each APIC as it upgrades. When the APIC that the browser is connected to is upgraded and it reboots, the browser displays an error message. Note During the upgrade process, while the APIC reboots with the newer image, you will not be able to use the GUI of that specific APIC. If you are logged into the APIC GUI during the upgrade process, you may receive a browser error message and may be logged off. Once the status of that specific APIC if Fully Fit, you can log in to that APIC again. Upgrading an APIC Using the NX-OS-Style CLI You can upgrade an Application Policy Infrastructure Controller (APIC) using the NX-OS-style CLI. Before you upgrade the switches, the APICs must have completed upgrading and have a health state of Fully Fit. In the NX-OS-style CLI, you must first set the catalog firmware. The following procedure sets the catalog firmware and starts the upgrade. Procedure Step 1 Step 2 Step 3 Step 4 Step 5 Step 6 SSH to an APIC in the fabric. # ssh admin@node_name Enter the configure mode: apic1# configure apic1(config)# Enter the firmware mode: apic1(config)# firmware apic1(config-firmware)# The firmware mode allows you to set the catalog version. Set the catalog version: apic1(config-firmware)# catalog-version aci-catalog-dk bin Now you are ready to update the controller firmware. Enter the controller-group mode and verify the current version: apic1(config-firmware)# controller-group apic1(config-firmware-controller)# show version Role Id Name Version controller 1 apic1 1.2(0.139g) controller 2 apic2 1.2(0.139g) controller 3 apic3 1.2(0.139g) Set the controller firmware to the version that you want: apic1(config-firmware-controller)# firmware-version aci-apic-dk bin 6

7 Upgrading an APIC Using the Object Model CLI Step 7 Start the upgrade. You can specify a time for the upgrade to start, or you can start the upgrade immediately. To specify the time for the upgrade to start, enter: apic1(config-firmware-controller)# time start 23:30 You must always specify a time; specifying the date is optional. To start the upgrade immediately, enter: apic1(config-firmware-controller)# exit apic1(config-firmware)# exit apic1(config)# exit apic1# firmware upgrade controller-group Upgrading an APIC Using the Object Model CLI The catalog firmware image is upgraded when an Application Policy Infrastructure Controller (APIC) image is upgraded. You do not need to upgrade the catalog firmware image separately. Cisco recommends that you perform the firmware upgrade from the GUI. When you use the GUI, the APIC performs additional verification and integrity checks on the software image. To upgrade an APIC using the object model CLI: 1 List the current software in the repository that was previously downloaded. Example: admin@apic1:~> firmware list Name : aci-apic-dk j.bin Type : controller Version : 1.1(1j) 2 Upgrade the firmware on the APICs. Example: admin@apic1:~> firmware upgrade controllers ver_no.bin The APICs are upgraded serially so that the APIC cluster is available during the upgrade. The upgrade occurs in the background. 3 Check the status of the upgrade. Example: admin@apic1:~> firmware upgrade status Node-Id Role Current- Target- Upgrade- Progress-Percent Firmware Firmware Status (if inprogress) controller 1.1(1.200j) apic-1.2(1.202i) complete 0 2 controller 1.1(1.200j) apic-1.2(1.202i) inprogress 0 3 controller 1.1(1.200j) apic-1.2(1.202i) inqueue 0 The Upgrade-Status field will show "inqueue", "inprogress", or "completeok". If you see "unknown" in this field, the APIC has upgraded and is rebooting. During this time, you may lose connectivity to the APIC CLI and have to relog in to the CLI. 7

8 Upgrading a Switch Using the GUI Upgrading a Switch Using the GUI Before you upgrade the switches, the Application Policy Infrastructure Controllers (APICs) must have completed upgrading and have a health state of Fully Fit. To upgrade a switch using the GUI: 1 On the menu bar, choose Admin > Firmware. 2 In the Navigation pane, choose Fabric Node Firmware. In the Work pane, the switches that are operating in the fabric are displayed. 3 If you have not created a firmware group, perform the following substeps: a b c d In the Navigation pane, choose Fabric Node Firmware > Firmware Groups. In the Work pane, choose the Policy tab. Choose Actions > Create Firmware Group. In the Create Firmware Group dialog box, perform the following actions: a b c d In the Group Name field, enter the name of the firmware group. In the Target Firmware Version drop-down list, choose the firmware version to which you will upgrade. In the Group Node IDs field, enter a comma-separated list or a range of node IDs to include in the group. For example, "101, , 108". Click Submit. e To verify that the firmware group was created, in the Navigation pane, choose Fabric Node Firmware > Firmware Groups > new_firmware_group. The Work pane displays details about the firmware policy that was created earlier. 4 If you have not created maintenance groups, perform the following substeps: a b c d In the Navigation pane, choose Fabric Node Firmware > Maintenance Groups. Cisco recommends that you create two maintenance groups for all of the switches. For example, create one group with the even-numbered nodes and the other group with the odd-numbered nodes. Ensure at least one spine and one leaf are in a different maintenance group than others so as not to lose total connectivity. In the Work pane, choose Action > Create POD Maintenance Group. In the Create POD Maintenance Group dialog box, perform the following actions: a b c d e f In the Group Name field, enter the name of the maintenance group. For example, "Even-Nodes". For the Run Mode drop-down list, choose Pause Upon Upgrade Failure. This is the default mode. In the Group Node IDs field, enter a comma-separated list or a range of node IDs to include in the group. For example, "102, 104, 106, 108, 110". In the Scheduler drop-down list, you can choose to create a schedule for upgrading or leave the drop-down list blank so that you can upgrade on demand. Click Submit. Repeat this step for the second maintenance group. For example, a group named "Odd-Nodes". Verify that the maintenance group was created. 8

9 Upgrading a Switch Using the NX-OS-Style CLI a In the Navigation pane, choose Fabric Node Firmware > Maintenance Groups > new_maintenance_group b c Choose the name of the maintenance group that you created. In the Work pane, verify that the nodes are attached to that maintenance group. 5 Right-click one of the maintenance groups that you created and choose Upgrade Now. 6 In the Upgrade Now dialog box, for Do you want to upgrade the maintenance group policy now?, click Yes. Note: In the Work pane, the Status displays that all the switches in the group are being upgraded simultaneously. The default concurrency in a group is set at 20. Therefore, up to 20 switches at a time will get upgraded, and then the next set of 20 switches are upgraded. In case of any failures, the scheduler pauses and manual intervention is required by the APIC administrator. The switch upgrade takes up to 12 minutes for each group. The switches will reboot when they upgrade, connectivity drops, and the controllers in the cluster will not communicate for some time with the switches in the group. Once the switches rejoin the cluster after rebooting, you will see all the switches listed under the controller node. If there are any VPC configurations in the cluster, the upgrade process will upgrade only one switch at a time out of the two switches in a vpc domain. 7 In the Navigation pane, click Fabric Node Firmware. Note: In the Work pane, view all of the switches that are listed. In the Current Firmware column, view the upgrade image details listed against each switch. Verify that the switches in the fabric are upgraded to the new image. Upgrading a Switch Using the NX-OS-Style CLI You can upgrade a switch using the NX-OS-style CLI. Before you upgrade the switches, the APICs must have completed upgrading and have a health state of Fully Fit. The following procedure upgrades a switch. Procedure Step 1 Step 2 Step 3 Step 4 Step 5 SSH to an APIC in the fabric. # ssh admin@node_name Add images to the firmware repository: apic1# firmware repository add aci-n9000-dk bin Enter the configure mode: apic1# configure apic1(config)# Enter the firmware mode: apic1(config)# firmware apic1(config-firmware)# Check the firmware version: apic1(config-firmware)# show version Role Id Name Version

10 Upgrading a Switch Using the Object Model CLI leaf Leaf-1 n (0.65l) leaf Leaf-2 n (0.65l) spine Spine-1 n (0.65l) spine Spine-2 n (0.65l) Step 6 Step 7 Step 8 Step 9 Step 10 Step 11 Enter the firmware-switch mode by creating a switch-group: apic1(config-firmware)# switch-group EvenNodes apic1(config-firmware-switch)# Add switches to the switch-group: apic1(config-firmware-switch)# switch 102, 202 (Optional) Verify that the switches were added: apic1(config-firmware-switch)# show run # Command: show running-config firmware switch-group all-nodes # Time: Fri Nov 6 15:18: firmware switch-group EvenNodes switch 102 switch 202 Set the switch firmware to the version that you want: apic1(config-firmware-controller)# firmware-version aci-apic-dk bin Set the switch run-mode to pause-on-failure so that the upgrade will pause in the event of any failures: apic1(config-firmware-switch)# run-mode pause-on-failure Start the upgrade. You can use a scheduler specify a time for the upgrade to start, or you can start the upgrade immediately. To use a scheduler, enter: apic3(config-firmware-switch)# schedule upgradetimerevennodes To start the upgrade immediately, go back to execsh mode and enter: apic1# firmware upgrade switch-group Upgrading a Switch Using the Object Model CLI Before you upgrade the switches, the Application Policy Infrastructure Controllers (APICs) must have completed upgrading and have a health state of Fully Fit. To upgrade a switch using the object model CLI: 1 Check that the output of the following command appears like the output shown below, with the correct version number: Example: admin@apic1:~> firmware list Name : aci-n9000-dk i.bin Type : switch Version : 11.2(1i) The name changes from ".iso" to ".bin". 10

11 Verifying Cluster Convergence Using the GUI 2 Upgrade the switches. Example: admin@apic1:~> firmware upgrade switch node 101 ver_no.bin Firmware Installation on Switch Scheduled You must upgrade each switch separately. 3 Check the upgrade status for the switch. The output that appears from the following command will appear like the following sample: Example: admin@apic1:~> firmware upgrade status node node_id Node-Id Role Current- Target- Upgrade- Progress-Percent Firmware Firmware Status (if inprogress) leaf n (1.869s1) n (1i) completeok 100 You can check the status of all nodes at once, by entering the firmware upgrade status command. 4 Repeat Steps 2 and 3 for each additional switch. Verifying Cluster Convergence Using the GUI You can monitor the progress of the cluster convergence after a scheduled maintenance. You can view the progress on the Controller Firmware screen of the GUI, which presents you with a series of messages during the process of converging. These messages are displayed in the Status field. As the controller and switches move through the upgrade, you will see messages about the number of nodes queued and the number in the process of upgrading, as well as how many have upgraded successfully. The following are the possible upgrade states for a node: NotScheduled: No upgrade is currently scheduled for this node. Scheduled: Upgrade is scheduled for this node. Queued: There is a currently active window (schedule) and the node is requesting permission to upgrade. Inprogress: Upgrade is currently in progress on this node. CompleteOK: Upgrade completed successfully. CompleteNOK: Upgrade failed on this node. Inretryqueue: Node is queued again for upgrade retry (5 attempts are made before declaring failure). This may take a while. When all the clusters have converged successfully, you will see "No" in the Waiting for Cluster Convergence field of the Controller Firmware screen. Troubleshooting Failures During the Upgrade Process There is one scheduler per maintenance policy. By default, when an upgrade failure is detected, the scheduler pauses, and no more nodes in that group begin to upgrade. The scheduler expects manual intervention to debug any upgrade failures. Once manual intervention is complete, you must resume the paused scheduler. If you notice that switches are in the "queued" state, then check the following: 11

12 Troubleshooting Failures During the Upgrade Process Is the controller cluster healthy? The controller cluster must be healthy. If you see "waitingforclusterhealth = yes" in the API or "Waiting for Cluster Convergence" showing "Yes" in the GUI, that means the controller cluster is not healthy. Until the controller cluster is healthy, switches which have not already started their upgrade will be in "queued" state. Is the switch maintenance group paused? The group will be paused if any switch fails its upgrade. If the system takes longer than about 60 minutes for a switch to display "waitingforclusterhealth = no" in the API or "Waiting for Cluster Convergence" showing "No" in the GUI, you should work through the steps for verifying a pause in the scheduler. For additional troubleshooting procedures, see Troubleshooting Cisco Application Centric Infrastructure. 12