#shut, no shutHave you tried turning it off and on again?

Cisco Smart Install – IOS-XE Upgrade Caveat

Preamble

This was created for and used with FreeZTP, but can likely be used with any Jinja2 templating system.

IOS-XE 3.7.4 cannot upgrade to 16.3.6 via smart-install because new force isn’t appended. This workaround utilizes EEM applets in a Jinja2 switch template to download install the updated image.

Switch log output from failure

Would you like to enter the initial configuration dialog? [yes/no]:
Loading ztp_ios_upgrade from 172.17.251.251 (via Vlan1): !
[OK - 38 bytes]
Preparing install operation ...
[1]: Downloading file tftp://172.17.251.251/cat3k_caauniversalk9.16.03.06.
SPA.bin to active switch 1
[1]: Finished downloading file tftp://172.17.251.251/cat3k_caauniversalk9.16.03.06.
SPA.bin to active switch 1
[1]: Copying software from active switch 1 to switch 2
[1]: Finished copying software to switch 2
[1 2]: Starting install operation
[1 2]: Expanding bundle cat3k_caa-universalk9.16.03.06.SPA.bin
[1 2]: Copying package files
[1 2]: Package files copied
[1 2]: Finished expanding bundle cat3k_caa-universalk9.16.03.06.SPA.bin
[1 2]: Verifying and copying expanded package files to flash:
[1 2]: Verified and copied expanded package files to flash:
[1 2]: Starting compatibility checks
[1]: % Candidate package compatibility checks failed because the following
package dependencies were not satisfied. Operation aborted.


[2]: % Candidate package compatibility checks failed because the following
package dependencies were not satisfied. Operation aborted.


[1]: % An internal error was encountered. Operation aborted.
[2]: % An internal error was encountered. Operation aborted.

ERROR: Software Installation Failed: 35 2

Loading network-confg from 172.17.251.251 (via Vlan1): !
[OK - 69 bytes]
Loading ZTP-23CFBA478F-confg from 172.17.251.251 (via Vlan1): !
[OK - 77012 bytes] 

TAC confirmation

The behavior is expected due to the command syntax difference as you suspected. We documented the behavior in a bug below: https://bst.cloudapps.cisco.com/bugsearch/bug/CSCvd49193. The running code 3.7.4E is affected and this issue is fixed on 3.6.8E.

Is it possible for you to try the following?

1- Upgrade from 3.7.4E to 3.6.8E via smart install

2- Upgrade from 3.6.8E to 16.3.6 via smart install

BugID_Screenshot_from_TAC

Considerations/Notes

  • Validated on Cisco 3850-12X48U-S switches with IOS-XE 3.7.4E installed out of the box; upgrading to IOS-XE 16.3.6.
  • Default TFTP blocksize is 512 on IOS-XE 3.7.4E, and 8192 on IOS-XE 16.3.6; adding this to the template significantly reduces the image transfer time.
  • The J2 template should not contain any configuration or syntax that is incompatible in IOS-XE version (3.7.4E). Any commands that are compatible with later IOS-XE versions only should be added to the post_ztp_2 applet.

Applet: post_ztp_1

  • Triggered by the switch receiving a DHCP address on Vlan1; notes regarding the maxrun and wait;
    • action 01.00 ... maxrun 900 – 15 minutes to accommodate the 2 minute wait, the 4-5 minute TFTP download, and the 5-6 minute install.
    • action 01.01 wait 120 – Vlan1 obtains a DHCP address approximately 1.5 minutes before the switch will allow configurations if stacked (see IOS-XE 3.7.4E Log below). This wait can be omitted for stand-alone switches.

IOS-XE 3.7.4E Log

*21:32:15.992: %DHCP-6-ADDRESS_ASSIGN: Interface Vlan1 assigned DHCP address 172.17.250.6, mask 255.255.254.0
...
*21:33:42.831: %HA_CONFIG_SYNC-6-BULK_CFGSYNC_SUCCEED: Bulk Sync succeeded
*21:33:43.824: %RF-5-RF_TERMINAL_STATE: 1 ha_mgr:  Terminal state reached for (SSO)

Applet: post_ztp_2

  • Triggered by IOS-XE 16.x specific redundancy syslog message; notes regarding the maxrun and wait;
    • action 01.00 ... maxrun 600 – 10 minutes to accommodate the 2 minute wait, any optional configuration changes and the software package clean process.
    • action 01.01 wait 120Redundancy syslog message is logged approximately 1.75 minutes before the switch will allow configurations if stacked (see IOS-XE 16.3.6 Log below). This wait can be omitted for stand_alone switches.

IOS-XE 16.3.6 Log

21:54:16.002 PDT: %IOSXE_REDUNDANCY-6-PEER: Active detected switch 2 as standby.
...
21:56:00.711 PDT: %HA_CONFIG_SYNC-6-BULK_CFGSYNC_SUCCEED: Bulk Sync succeeded
21:56:01.735 PDT: %RF-5-RF_TERMINAL_STATE: Terminal state reached for (SSO)

Process/Explanation

  1. Switch is powered up connected to the provisioning network and initiates smart-install. Switch requests an upgrade first but no image is downloaded since image download is disabled in FreeZTP.
  2. Switch requests config, FreeZTP gives a (merged) config containing the required config.
  3. Switch applies configuration with Vlan1 configured for DHCP addressing.
    • post_ztp_1 is triggered by Vlan1 obtaining a DHCP address.
  4. [EEM Applet post_ztp_1 loaded to memory.]
    1. Applet deletes itself from running config, downloads the bin file, cleans up temp configs and writes the startup-config.
    2. Switch then runs the software install command, answers y to reload prompt from install command.
  5. Switch reloads.
    • post_ztp_2 is triggered by a syslog command specific to IOS-XE 16.x (%IOSXE_REDUNDANCY-6-PEER).

      This trigger may need to be changed! I’m unsure if this syslog message is present when provisioning a stand-alone switch as I never tested this scenario. I will update if I get an opportunity to test this on a stand-alone switch.

  6. [EEM applet post_ztp_2 loaded into memory.]
    1. Applet deletes itself from running configuration and writes the startup-config.
    2. (Optional) Add action sequences for any IOS-XE 16.x specific commands.
    3. Applet runs the package clean process to delete the old .pkg and packages.conf file(s).

Required Config

  • Disable FreeZTP image downloads, replace <SCOPE> with the name of your configured DHCP scope.

    ztp set dhcpd <SCOPE> imagediscoveryfile-option disable && ztp request dhcpd-commit && ztp service restart

  • Allocate a provisioning interface; i.e. the interface connected to the provisioning network.

  • Modify the variables in the template config snippet below to suit network/needs, then add the whole snippet to the J2 switch template. These four variables can be defined in the keystore or left in the template.

    Variable Description
    tftp_addr Address of TFTP server, typically FreeZTP.
    image_bin Name of the image file to download.
    access_vlan Vlan to configure on the provisioning interface (Gi1/0/48) after upgrade/reload is complete.
    prov_int Interface to be used for provisioning; e.g. Te1/0/48 (3850-12X48U-S interfaces 37-48 are TenGigabitEthernet.)

Template Config Snippet

!-- Variables (keys) statically defined within the template.
!{% set tftp_addr = "172.17.251.251" %}
!{% set image_bin = "cat3k_caa-universalk9.16.03.06.SPA.bin"%}
!{% set access_vlan = "501" %}
!{% set prov_int = "Te1/0/48" %}

!-- Required for EEM applet to function as intended.
logging buffered 20480 debugging
file prompt quiet
ip tftp blocksize 8192

!-- Required for TFTP transfers from FreeZTP (or other reachable TFTP server; i.e. `tftp_addr`).
interface Vlan1
  ip address dhcp
  no shutdown

!-- Interface that is connected to the provisioning network, must remain on Vlan1 for TFTP download.
interface GigabitEthernet1/0/48
  description TMP//PROVISION:Omit config; updated with post_ztp_2 EEM applet.
  switchport
  switchport mode access
  switchport nonegotiate
  switchport access vlan 1
  spanning-tree portfast
  no shutdown

!-- POST_ZTP_1 EEM applet to download and install the image, clean up config, then reload.
event manager applet post_ztp_1
  event syslog occurs 1 pattern "%DHCP-6-ADDRESS_ASSIGN: Interface Vlan1 assigned DHCP address" maxrun 900
  action 01.00 syslog msg "\n     ##Switch/stack is ready, downloading and installing image in 120s."
  action 01.01 wait 120
  action 01.02 cli command "enable"
  action 01.03 cli command "debug event man act cli"
  action 02.00 cli command "conf t"
  action 03.00 cli command "no event man app post_ztp_1"
  action 04.00 cli command "do copy tftp://{{tftp_addr}}/{{image_bin}} flash:"
  action 05.00 cli command "int vlan 1"
  action 05.01 cli command   "no ip addr"
  action 05.02 cli command   "shut"
  action 06.00 cli command "int {{prov_int}}"
  action 06.01 cli command   "no desc"
  action 06.02 cli command   "switchp acc vl {{access_vlan}}
  action 07.00 cli command "end"
  action 08.00 cli command "write mem" pattern "confirm|#"
  action 08.01 cli command ""
  action 09.00 cli command "software install file flash:{{image_bin}} new force" pattern "proceed|#"
  action 09.01 cli command "y"
  action 10.00 syslog msg "\n     ## Installation complete, reloading for upgrade."
  action 10.01 cli command "undebug all"

!-- (Optional) POST_ZTP_2 applet to run package clean and add any config commands that were previously incompatible.
!-- (Optional) Add any desired configs between actions 03.00 and 04.00.
event manager applet post_ztp_2
  event syslog occurs 1 pattern "%IOSXE_REDUNDANCY-6-PEER" maxrun 600
  action 01.00 syslog msg "\n     ## Switch/stack reloaded on new image, running 'post_ztp_2' EEM applet in 120s."
  action 01.01 wait 120
  action 01.02 cli command "enable"
  action 01.03 cli command "debug event man act cli"
  action 02.00 cli command "conf t"
  action 03.00 cli command "no event man app post_ztp_2"
  !-- action 03.01 cli command "" {# Use actions 3.01 - 3.99 for configurations specific to later IOS-XE version(s). #}
  action 04.00 cli command "end"
  action 05.00 cli command "write mem" pattern "confirm|#"
  action 05.01 cli command ""
  action 06.00 cli command "req plat soft pack clean sw all" pattern "proceed|#"
  action 06.01 cli command "y"
  action 07.00 syslog msg "\n     ## Any unused .bin or .pkg files have been deleted.\n     ## Switch is ready for deployment, OK to power off."
  action 07.01 cli command "undebug all"

Tags: , , , , , , , , ,

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.