Palo Alto Panorama “Scheduled Config Export” Error Code 67

We are using the “Scheduled Config Export” feature of Panorama to backup the configuration to an FTP server internally. However when I run the backup I get the following error:

‘Failed to export config bundle file PANORAMA_20220812.tgz to host XXXXXX.XXXXXXX.XXXXXXX port 21 user PanoramaBackupUser passive-mode yes, error code 67’

We were not able to determine the issue and had to create a case with Palo Alto support. They confirmed that this is an issue with version 10.2.2 of Panorama and has been resolved in version 10.2.3. I upgraded our appliance this morning and now this is working as expected.

UPLOAD SAML IDP fAILED – Failed to parse IDP Metadata on pALO aLTO fIREWALL

When setting up SAML IDP on Palo Alto firewall (version 10.0.6) we are importing the XML file provided by our SAML vendor.

However, when importing it we get the following error message:

Upload SAML IDP Failed
Failed to parse IDP Metadata.

The problem is that the “Profile Name” field is limited to 31 characters, but it isn’t validated by the firewall. From the validation when making a new SAML Identity Provider, only alphanumeric characters, underscore ‘_’, hyphen ‘-‘, dot ‘.’ or spaces are permitted.

If you decrease the length of the name, it will import the metadata correctly.

Deploying Palo Alto FIrewall in Azure – Maintenance Mode

I deployed a Palo Alto VM firewall into Azure recently. Every time I deployed it from the Azure template from the Marketplace or using bootstrap (which still uses the Azure template to get started) the firewall would take about 20-30 minutes and then wind up in maintenance mode without a usable IP address, and no management GUI.

Errors on the serial console were “Entry Reason: System Startup error.” and the Maintenance Entry Reason was “System start failed multiple times. Caused by service: mgmtsrvr”. I deployed the latest version of Palo Alto firewall (version 9.1.3 as of this writing).

Eventually I was able to solve the problem by trying a different password. Even though the template has the following requirements for passwords:

Our original auto generated password that broke the firewall was “wQCoPb7E7T9c5844FbbA@r5iVFQu8V2S” (no quotes). I don’t know if the @ (asterisk) symbol broke the firewall or there was a length issue, but after we changed the password the firewall deployed quickly and easily into Azure. So if you are immediately kicked into maintenance mode with your Palo Alto firewall, try a different password.

This CSR was created with an invalid algorithm – GoDaddy and ECC / ECDSA Certificates

We are working on testing the Microsoft’s Always On VPN  solution.  It is recommended to use the ECC certificates (Elliptic-curve cryptography) for performance and security reasons (256-bit ECC versus 2028 bit RSA key I’m assuming improves the performance, but I’m not sure how it is more secure).

After generating the CSR for the certificate using these instructions we pasted the CSR into GoDaddy to generate the certificate.  However after submitting the CSR we got the following error message:

“This CSR was created with an invalid algorithm”

After a call with GoDaddy Support, they confirmed that they do not currently support ECC or ECDSA or DSA keys on certificates and only support RSA keys.  If you use GoDaddy and need an ECC certificate, please give GoDaddy Support a call to ask for these keys to be supported going forward.  Hopefully enough people will contact them that they add this feature in the future.

A New Tesla in My Driveway

Tesla PowerWall

We took delivery of a new Tesla yesterday, and here’s proof of it sitting in the driveway.  You might notice that it is a litte small, missing some tires, a steering wheel and doors, that’s because it is a Tesla Powerwall.  This is probably as close as I’ll come to owning a Tesla car in the near future.

The Tesla Powerwall is a large battery pack that will work with our previously installed solar photovoltaic system that we installed on the house last year.  The power from the panels will be stored in the Tesla Powerwall during the day, and then we can use it in the house at night.

Right now, given the current cost of electricity this is about a 10 year ROI (Return on Investment), but I’m guessing that electricity will keep getting more expensive, helping to make this cost effective more quickly.

We are working with Duncan Renewables again as they did our photovoltaic panel installation last year.  Installation is happening right now, so more information after installation.  I’m happy to answer any questions if this is something you are considering as well.

Citrix NetScaler and Blank AGESSO.JSP Page

We have recently been locking down systems to protect from the DROWN Attack vulnerability. This process involves removing the insecure SSL protocols and SSL ciphers and has generally been straightforward.

Today when locking down a Citrix NetScaler 11.x VPX appliance with a locally installed Web Interface we kept running into the issue with the blank AGESSO.JSP file after logging into the web site. This is a reasonably common issue, with lots of solutions, none of which worked today. It was eventually narrowed down to missing the following cipher on the NetScaler Gateway Virtual Server: SSL3-DES-CBC3-SHA. With this cipher missing, the Java Web Interface running on the Citrix NetScaler appliance didn’t trust the installed NetScaler Gateway.

The output from SSL Labs showed the following differences:

Without the SSL3-DES-CBC3-SHA cipher included:


With the SSL3-DES-CBC3-SHA cipher included:


So it sounds like the NetScaler Web Interface is still running Java 6.x that doesn’t trust the Access Gateway.

So my new best practice for the SSL ciphers on a NetScaler Gateway VPX (running version NetScaler Build 11.0-65.31 or later) are the following:

bind ssl cipher vpx-cipher-list
bind ssl cipher vpx-cipher-list -cipherName TLS1.2-ECDHE-RSA-AES256-GCM-SHA384
bind ssl cipher vpx-cipher-list -cipherName TLS1.2-ECDHE-RSA-AES128-GCM-SHA256
bind ssl cipher vpx-cipher-list -cipherName TLS1.2-ECDHE-RSA-AES-256-SHA384
bind ssl cipher vpx-cipher-list -cipherName TLS1.2-ECDHE-RSA-AES-128-SHA256
bind ssl cipher vpx-cipher-list -cipherName TLS1-ECDHE-RSA-AES256-SHA
bind ssl cipher vpx-cipher-list -cipherName TLS1-ECDHE-RSA-AES128-SHA
bind ssl cipher vpx-cipher-list -cipherName TLS1.2-AES256-GCM-SHA384
bind ssl cipher vpx-cipher-list -cipherName TLS1.2-AES128-GCM-SHA256
bind ssl cipher vpx-cipher-list -cipherName SSL3-DES-CBC3-SHA


Citrix XenMobile 10.3 Setup of Android for Work: Enterprise Service Account Key is not found

I’m working to setup Citrix XenMobile 10.3 with Android for Work in the office.  I’ve been working through this set of documentation from Citrix here.  After Citrix enabled their side of the configuration and it correctly shows in my Google Admin console, when I try to add the Android For Work settings to XenMobile there are number of things that are strange:

  1. There are only prompts for the Domain Name, Domain Admin Account, and Service Account ID.  There is no option for the Binding Token as shown in the screenshots.
  2. Once I enter this information I get the following error message “Enterprise Service Account Key is not found. Please check the configuration values.”

There is currently no XenMobile 10.3 specific information, so I don’t know if this is a change specific to 10.3 or a bug in 10.3.

Are you seeing this same error message?  How did you get around it?

UPDATE 1/13/2016: 

We have a case open with Citrix Support.  They are currently trying to duplicate the problem on a clean system, so hopefully I should know more later today if it is a bug with the newest version of software or something we did incorrectly with the setup.  Since the Google web pages have changed the Citrix documentation is out of date, so there might be some steps missing now that Google’s process has changed.

UPDATE 1/13/2016 4:37 PM UTC:

From Citrix Support: “We have been able to confirm the errant behavior and are currently engaged with engineering on a fix. We believe that the issue centers around the changes implemented on the Google Api site, as it does not seem to generate the service account details in the same format. I hope to have an idea by EOB if this is actually the case, or if we can come up with a workaround for the problem. Thanks!”

UPDATE 2/4/2016 11:28 PM UTC

Unfortunately I don’t have a fix that I can provide directly for this.  Based on the case we had open with Citrix support it sounds like Google’s API changed breaking this functionality.  Through a combination of additional undocumented steps and changes on the Citrix back end they were able to get this working.

Useful Tools: Notepad++ and regex to the rescue

Notepad++ is one of my favorite text editors and I tend to install it on any computer I’m working with. While the built-in Windows Notepad only gets you so far, the free GPL Licensed Notepad++ has some big advantages:

1. Can handle the large text files (think 100MB log files) with ease.

2. There is a great Compare plugin in the Plugin Manager that lets you easily compare two text files.  I use this all the time when comparing firewall or switch configurations to see what has changed (or what needs to change when doing firewall migrations to new hardware.

3. You can open up multiple text files in tabs in Notepad++ and search across all the documents at the same time.

4. It is language specific.  Open up an HTML file in Notepad++ and it knows about HTML formatting and changes the text color appropriately.  Same thing with other formats like XML, PowerShell, VBScript, etc.

Yesterday  I was asked by a colleague to parse a Cisco switch configuration to find ports that are configured differently froman example port on a switch with several hundred ports on it.  The example text would look something like the following:

interface GigabitEthernet2/0/27
switchport access vlan 232
spanning-tree portfast

interface GigabitEthernet2/0/28
description I’m different
switchport access vlan 232
spanning-tree portfast

interface GigabitEthernet2/0/29
switchport access vlan 232
spanning-tree portfast

Ideally, we’d just flag port 2/0/28 as different in the example above, since it doesn’t match the other two ports.

While a regular Find/Replace operation would find the particular lines to be deleted, it wouldn’t be able to handle the changes in the port numbers.  That’s where the regular expression (regex) feature of Notepad++ can help.  Regex is a very powerful method for handling text files but can be very confusing.

There’s a great regex testing tool here: so you can experiment with building your own regex expressions.

Let’s go through this example:

Our main problem is that we need to search through all the ports and ignore the port numbers in the example above, so that we can find all the ports regardless of number: “interface GigabitEthernet2/0/27”.  To do this we need to search for the following:

interface GigabitEthernet\d*\/\d*\/\d*

Where the search terms are the following:

\d = any single numeric digit
* = repeat the previous token (the \d in this case from 0 to infinite times).
\/ = This is the escape character “\” followed by the forward slash character “/” which searches for a forward slash “/”

By combining those together we have a string that will search for all the ports in the configuration.  The only remaining issue now is the carriage returns in the file.  These can be found using \r for carriage return and \n for new line.  So the full search string we are looking for is now:

interface GigabitEthernet\d*\/\d*\/\d*\r\n switchport access vlan 232\r\n spanning-tree portfast\r\n

Now just do a replace in Notepad++ (make sure to choose the regular expression radio box) and replace that text with nothing so that the matching ports are removed from the text configuration.

Notepad++ Regular Expression Search showing Regular Expression Radio Box Checked
Notepad++ Regular Expression Search

Congratulations, now the only ports that should be left in the configuration are ports that are “different” and would need to be handled separately.

Do you have a favorite feature of Notepad++?  Do you have some favorite regex expressions that you use all the time?  Do you have some other favorite, must have tool that you’d like to share?  Please let me know in the comments.


Note: This article has also been published on the Kraft & Kennedy website

My Favorite Baklava Recipe

Since people ask for my baklava recipe I thought I would post it here to make it easier to share. Remember I’ll need to have a sample to confirm that you have made it correctly. Several pieces actually would be nice.

This version uses honey rather than a sugar syrup, so differs from many of the other baklava recipes that I’ve tried.  Both are good, but I just tend to like this version.


8″ x 12″ baking pan
Pastry brush (or I guess you could use a good paint brush (without paint))

Ingredients (most amounts are approximate):

2 lb (2 boxes) Filo dough (in the freezer or dairy section of your supermarket)
1/2 lb butter (226 g)
3 cups chopped walnuts
2 tsp cinnamon
1/2 tsp ground nutmeg
1/2 tsp ground cloves
1 cup sugar
1 1/2 cup honey (somewhere between 1 and 2 cups should be fine)


  1. Preheat oven to 350 F (180 C)
  2. Defrost the filo dough according to directions. If you don’t do this and are working with frozen filo dough, you will not be happy.
  3. Melt the butter in a bowl in the microwave. Melt additional butter as required.
  4. Brush the bottom of the pan with the butter.
  5. Lay down a single sheet of filo dough on the bottom of the pan and brush the top with melted butter.
  6. Continue laying down more filo sheets (buttering the tops every layer) until you have about 6-8 layers. Don’t worry if the filo breaks into pieces smaller than full sheets. Just patch a layer together from small sheets.
  7. Combine the chopped nuts, cinnamon, cloves, sugar, and nutmeg in a bowl. Spread about 1/4 of the mixture over the filo dough. Make sure you have spread the mixture evenly (don’t forget the edges and corners)
  8. Lay down another 4-6 sheets of filo (buttering each layer).
  9. Lay down 1/4 of the chopped nuts mixture.
  10. Lay down another 4-6 sheets of filo (buttering each layer).
    Lay down 1/4 of the chopped nuts mixture.
  11. Lay down another 4-6 sheets of filo (buttering each layer).
    Lay down remaining 1/4 of the chopped nuts mixture.
  12. Lay down remaining filo sheets (do the math ahead of time if it makes you happy, you need about 5 layers, with the top and bottom layer thicker than the in between layers). Butter between sheets.
  13. Butter the top of the filo with remaining butter.
  14. Cut the filo (before baking) into whatever shapes you want (typically diamond or triangle shapes). Make sure you cut all the way through to the bottom.
  15. Bake for approximately 45 minutes to 1 hour at 350F (180C).
  16. Immediately after removing baklava from oven pour honey over the top while hot. Make sure to cover baklava completely with honey.
  17. Let cool and serve.

Black-headed Seagull – My Google Voicemail Stalker

I was one of the early adopters of Google Voice where you could have an additional telephone number for free that could be rerouted to another phone or go straight to voice mail.

Starting on May 16, 2011 at 9:15 PM I received the first strange voice mail from this phone number: +49223482433

“Black-headed seagull. Black-headed seagull”

Then on June 3, 2011 I received the next voice mail at 7:35 PM:

“22nd of May 1972, London. Police control West End one way street. 9th to the 23rd, Saturday. Police control West End one way street in London, 22nd of May 1972, 9th to the 23rd.”

That was it, no other information. I’m taking the best guess at the transcriptions here, so please let me know if you have any suggested corrections.

The next voice mail I received was on August 10, 2012 (over a year later) at 2:58 AM again from the same phone number:

“May 1972 in London. Spain could have been involved. Spain could have been involved. Tenerife. May 1972 in London Centre Room Club West End. Centre Room Club West End. May 1972. Spain could be involved. Tenerife.”

Next voice mail didn’t arrive until January 18, 2014 at 1:52 PM again from the same phone number:

“I refer to my previous message. Oui sans unique.”

The next voice mail was on May 5, 2014 at 2:43 AM:

“621 631 and 641”

And the last time I received a voice mail was on September 4, 2014 at 6:31 PM:

“Tutankhamun exhibition 1972 in London and San Francisco. Tutankhamun exhibition. CPU Africanos the African pupil. German criminal story set here. Also Baron Sootso”

So, who is this lady? A search indicates that it might be a Helga Keuler from Pulheim, Germany, but I don’t know this person.

Why is she calling me?

What is she talking about? Most of the stories don’t seem to exist.

Got a theory? Any more clues? I’m interested to hear your theories.