Palo Alto Firewalls HA Pair Upgrade

I upgraded my Palo Alto firewalls from version 6.1.12 to 7.1.1 recently. It is pretty much a straight forward upgrade but might as well write about it. Also, this is for the firewalls that do not have Internet access via their management (oob) interface.

I am going to do this via CLI since I have never used the GUI. To get started, do the following

  • Download all the PANOS you will need to upgrade your PAN firewalls
  • Check the checksum of the firmware
  • Backup your config
  • Schedule a network maintenance window. There should not be a downtime if you have an HA pair, but it is always better to have a dedicated maintenance window
  • If you still have a support contract, make sure that you let your PAN engineer to be on standby because calling their number is a pain

Since I am upgrading to version 7.1.1 from 6.1.12, I would need to follow an upgrade path. The upgrade path for PAN goes like this: If I were to go to 6.1.14 from 6.1.12, then I can go upgrade to 6.1.13 directly because 6.1.14 is in the same base firmware as 6.1.12 – the base version 6.1.0; however, if I were to upgrade to 7.0.10, then I would need to upgrade to the base firmware which is 7.0.1 (there is no 7.0.0) then upgrade directly to 7.0.10.

The thing is when you upgrade, you would need to restart the firewall (just like any firewall) for the upgrade to take effect. The cool thing about the PAN, once you installed the base firmware, you can install the next version right away without restarting the firewall. This is only true if you’re going to the version that is on the same base firmware. Otherwise, you will need to restart the firewall.

To make this simple here is a visual representation of how PAN upgrade is done. The red ones are the base firmware for a specific version.
From 6.1.12
To 6.1.14
6.1.12 > 6.1.14 then restart > done

From 6.1.12
To 7.1.1
6.1.12 > 7.0.1 then restart > 7.1.0 > 7.1.1 then restart > done

From 5.0.20
To 7.1.5
5.0.20 > 6.0.0 then restart 6.1.0 then restart 7.0.1 then restart 7.1.0 > 7.1.5 then restart > done

You get the idea.

Before we begin, here are some recommended commands that need to be disabled when upgrading an HA pair. These commands are done in configuration mode.

  • set deviceconfig setting session tcp-reject-non-syn no
  • set deviceconfig high-availability group <value> election-option preemptive no

Starting from here, all the commands below are done in operational mode.

Let’s get started. The first thing is ssh or console into the active device.You can do ssh or console into the passive device, but to test the failover, you would need to start at the active device. Once you are connected to the active device, the device needs to be suspended first. The device status will change from being active (or passive) to suspended as shown below

admin@PAN01(active)> request high-availability state suspeneded

Successfully changed HA state to suspended
admin@PAN01(suspended)>

Once the firewall in suspended mode, you can import the firmware. In my case, it will import the 7.0.1 base firmware via scp. The command syntax is

scp import software from <username>@<server-ip>:<path>
admin@PAN01(suspended)> scp import software from root@10.0.5.10:/root/pan/PanOS_3000-7.0.1 
The authenticity of host '10.0.5.10 (10.0.5.10)' can't be established.
RSA key fingerprint is ba:ee:8c:a4:75:90:b4:c4:81:ed:36:37:9c:32:46:3e.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '10.0.5.10' (RSA) to the list of known hosts.
root@10.0.5.10's password: 
PanOS_3000-7.0.1 100%                                                     602MB 22.3MB/s 00:27

PanOS_3000-7.0.1 saved

Once the import is completed, the next step is to install the firmware.

admin@PAN01(suspended)> request system software install file PanOS_3000-7.0.1 
Executing this command will install a new version of software. It will not take effect until system is restarted. Do you want to continue? (y or n)

Software install job enqueued with jobid 163. Run 'show jobs id 163' to monitor its status. Please reboot the device after the installation is done.
163

admin@PAN01(suspended)>

As you can see, we can use the command show jobs id <job-id> to view the installation process. The installation takes about a few minutes.

admin@PAN01(suspended)> show jobs id 163

Enqueued                     ID             Type   Status Result Completed
--------------------------------------------------------------------------
2016/10/28 22:01:35         163       SWInstall       ACT   PEND       66%
Warnings:
Details:

admin@PAN01(suspended)> show jobs id 163

Enqueued                     ID             Type   Status Result Completed
--------------------------------------------------------------------------
2016/10/28 22:01:35         163       SWInstall       ACT   PEND       99%
Warnings:
Details:

admin@PAN01(suspended)> show jobs id 163

Enqueued                     ID             Type   Status Result Completed
--------------------------------------------------------------------------
2016/10/28 22:01:35         163       SWInstall       FIN     OK 22:06:16
Warnings:
Details:Software installation successfully completed. Please reboot to switch to the new version.

admin@PAN01(suspended)>

Since I am upgrading to 7.1.1, and I just installed 7.0.1, I would need to restart the device.

admin@PAN01(suspended)> request restart system 
Executing this command will disconnect the current session. Do you want to continue? (y or n)

Broadcast message from root (pts/0) (Fri Oct 28 22:08:15 2016):

The system is going down for reboot NOW!

Connection closed by foreign host.

Disconnected from remote host(PAN01) at 18:15:14.

Once the device boots up, check the software version

admin@PAN01(passive)> show system info | match sw
sw-version: 7.0.1

Once you confirmed the version, import the another base firmware

admin@PAN01(passive)> scp import software from root@10.0.5.10:/root/pan/PanOS_3000-7.1
The authenticity of host '10.0.5.10 (10.0.5.10)' can't be established.
ECDSA key fingerprint is 88:8f:77:fc:88:b0:dc:ef:34:31:24:68:ef:0a:72:b4.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '10.0.5.10' (ECDSA) to the list of known hosts.
root@10.0.5.10's password: 
PanOS_3000-7.1.0 100% 699MB 25.9MB/s 00:27

PanOS_3000-7.1.0 saved

Then install the firmware after importing the base firmware. However, this time the syntax has changed. The file option is not available anymore.

admin@PAN01(passive)> request system software install 
+ load-config Configuration to use for booting new software
> version     Upgrade to a software package by version

admin@PAN01(passive)> request system software install version 
7.1.0   7.1.0
<value> Upgrade to a software package by version

admin@PAN01(passive)> request system software install version 7.1.0 
Executing this command will install a new version of software. It will not take effect until system is restarted. Do you want to continue? (y or n) 

Software install job enqueued with jobid 2. Run 'show jobs id 2' to monitor its status. Please reboot the device after the installation is done.

Always check the status of the installation

admin@PAN01(passive)> show jobs id 2

Enqueued                     ID             Type   Status Result Completed
--------------------------------------------------------------------------
2016/10/28 22:23:02           2       SWInstall       ACT   PEND       84%
Warnings:
Details:Loading into software manager
Succesfully loaded image into software manager

admin@PAN01(passive)> show jobs id 2

Enqueued                     ID            Type   Status Result Completed
--------------------------------------------------------------------------
2016/10/28 22:23:02           2       SWInstall       FIN     OK 22:27:34
Warnings:
Details:Loading into software manager
Succesfully loaded image into software manager
Software installation successfully completed. Please reboot to switch to the new version.

admin@PAN01(passive)>

At this point, since we are going to 7.1.1 and we just installed 7.1.0, we do not need to restart the device. We can install the 7.1.1 first then we can restart.

admin@PAN01(passive)> scp import software from root@10.0.5.10:/root/pan/PanOS_3000-7.1.1
root@10.0.5.10's password: 
PanOS_3000-7.1.1 100% 252MB 28.0MB/s 00:09

PanOS_3000-7.1.1 saved

Install the 7.1.1

admin@PAN01(passive)> request system software install version 7.1.1

Executing this command will install a new version of software. It will not take effect until system is restarted. Do you want to continue? (y or n)
Software install job enqueued with jobid 3. Run 'show jobs id 3' to monitor its status. Please reboot the device after the installation is done.
3

admin@PAN01(passive)> show jobs id 3

Enqueued                     ID             Type   Status Result Completed
--------------------------------------------------------------------------
2016/10/28 22:31:46           3       SWInstall       ACT   PEND       81%
Warnings:
Details:Loading into software manager
Succesfully loaded image into software manager

admin@PAN01(passive)> show jobs id 3

Enqueued                     ID             Type   Status Result Completed
--------------------------------------------------------------------------
2016/10/28 22:31:46           3       SWInstall       FIN     OK 22:35:48
Warnings:
Details:Loading into software manager
Succesfully loaded image into software manager
Software installation successfully completed. Please reboot to switch to the new version.

admin@PAN01(passive)>

Once the installation is done, restart the device

admin@PAN01(passive)> request restart system
Executing this command will disconnect the current session. Do you want to continue? (y or n)

Broadcast message from root (pts/0) (Fri Oct 28 22:36:48 2016):

The system is going down for reboot NOW!

Connection closed by foreign host.

Disconnected from remote host(PAN01) at 18:36:51.

Once the firewall restarted, the device will be stuck in suspended mode. The reason is, the current active firewall version is in 6.1.12, and the upgraded firewall is on 7.1.1. The two version are not compatible with regards to high-availability pairing.

Let’s work with the second firewall. Here is a caveat. Since the newer version is not compatible, the one I upgraded will not become either active or passive until its pair gets to the compatible version. 

However, if the software is compatible, the HA state will turn to <passive>. It may take ~1 minute or 2, so I would recommend to wait for it. If the software base version is the same, then it they are compatible; therefore, wait for the HA to reestablish. 

Now, if you are upgrading this via SSH, I would highly recommend that keep the current active device as active because once you lost your connection, that device is not going to become active until you use the command request high-availability state functional. If you are connected via SSH, you won’t be able to execute this command. Therefore, we want the active firewall to boot up as an active device not suspended.

I am doing this via SSH, so I am not going to suspend the firewall. Just like the first firewall, I am going to import the firmware.

admin@PAN02(active)> scp import software from root@10.0.5.10:/root/pan/PanOS_3000-7.0.1
The authenticity of host '10.0.5.10 (10.0.5.10)' can't be established.
RSA key fingerprint is ba:ee:8c:a4:75:90:1c:c4:81:ed:36:cb:9c:32:78:3e.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '10.0.5.10' (RSA) to the list of known hosts.
root@10.0.5,10's password:
PanOS_3000-7.0.1                                             100% 602MB 11.2MB/s   00:54  

PanOS_3000-7.0.1 saved

admin@PAN02(active)>

Once imported, install the firmware.

admin@PAN02(active)> request system software install file PanOS_3000-7.0.1
Executing this command will install a new version of software. It will not take effect until system is restarted. Do you want to continue? (y or n)

Software install job enqueued with jobid 144. Run 'show jobs id 144' to monitor its status. Please reboot the device after the installation is done.
144

admin@PAN02(active)> show jobs id 144

Enqueued                     ID             Type   Status Result Completed
--------------------------------------------------------------------------
2016/10/28 23:44:34         144       SWInstall       ACT   PEND       86%
Warnings:
Details:

port22@P07101EF01(active)> show jobs id 144

Enqueued                     ID             Type   Status Result Completed
--------------------------------------------------------------------------
2016/10/28 23:44:34         144       SWInstall       FIN     OK 23:48:28
Warnings:
Details:Software installation successfully completed. Please reboot to switch to the new version.

admin@PAN02(active)>

Restart the device

admin@PAN02(active)> request restart system 
Executing this command will disconnect the current session. Do you want to continue? (y or n)

Connection closed by foreign host.

Disconnected from remote host(PAN02) at 19:35:23.

After the reboot, verify the version of the firmware

admin@PAN02(suspended)> show system info | match sw
sw-version: 7.0.1

Now, import the base firmware then install it

admin@PAN02(suspended)> scp import software from root@10.0.5.10:/root/pan/PanOS_3000-7.1.0
The authenticity of host '10.0.5.10 (10.0.5.10)' can't be established.
ECDSA key fingerprint is 88:8f:09:3c:88:b0:dc:11:34:31:24:68:ef:0a:9d:b4.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '10.0.5.10' (ECDSA) to the list of known hosts.
root@10.0.5.10's password: 
PanOS_3000-7.1.0 100%                                                       699MB 11.1MB/s 01:03

PanOS_3000-7.1.0 saved

Install the firmware

admin@PAN02(suspended)> request system software install version 7.1.0 
Executing this command will install a new version of software. It will not take effect until system is restarted. Do you want to continue? (y or n)

Software install job enqueued with jobid 3. Run 'show jobs id 3' to monitor its status. Please reboot the device after the installation is done.
3

admin@PAN02(suspended)>

Once the firmware is installed, do not restart the firewall yet. We will need to install the desire firmware. Import the firmware then install.

admin@PAN02(suspended)> scp import software from root@10.0.5.10:/root/pan/PanOS_3000-7.1.1
root@10.0.5.10's password: 
PanOS_3000-7.1.1                                                     100% 252MB 28.0MB/s 00:09

PanOS_3000-7.1.1 saved

admin@PAN02(suspended)> request system software install version 7.1.1
Executing this command will install a new version of software. It will not take effect until system is restarted. Do you want to continue? (y or n)

Software install job enqueued with jobid 4. Run 'show jobs id 4' to monitor its status. Please reboot the device after the installation is done.
4

admin@PAN02(suspended)>

 

Restart the device

admin@PAN02(suspended)> request restart system 
Executing this command will disconnect the current session. Do you want to continue? (y or n)

Broadcast message from root (pts/0) (Fri Oct 28 23:58:03 2016):

The system is going down for reboot NOW!

Connection closed by foreign host.

Disconnected from remote host(PA02) at 19:58:06.

After it reboots, login again and check  the system software version

port22@P07101EF01> show system info | match sw
sw-version: 7.1.1

Also, login to both device and check if the device is in suspend mode. If either one of them is in suspended mode, then use this command

request high-availability state functional

That should do it.

 

Cheers!

Advertisements

About networkshinobi

This blog is about the things I learned about computers and networking to help me to remember them as I push further my studies. I created this blog to help myself to continue my education; and if you find this blog helpful for your studies, that is great. That is one of the reasons why I made this blog, to share my interest and knowledge. Also, all the entries/posts I made are based on my views, opinion and for educational purposes only. If you see some mistakes, feel free to drop some comments. I would appreciate all the helpful comments. Thanks
This entry was posted in Misc.. Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s