Archive for July, 2015

Palo Alto and the power of an API

We recently bought Palo Alto PA-200 firewalls for our retail locations to replace our aging CheckPoint UTMs.  I didn’t investigate their API at all during the time we were looking at CheckPoint competitors.  I knew it had one, but hadn’t really given it a lot of thought.  Now that we have a massive roll-out ahead of us, I’ve started scripting parts of the process.  I must say that I love the flexibility that their API gives us.

In the past, for any major roll-out, I’ve scripted the process using telnet / SSH / HTTP (for web scraping), basically whatever interface the vendor allowed.  My goal is to make the installation fast and easy to support, while reducing the chance of human error as much as possible.  The hassle with CLI scripting for remote devices is always the parsing.  While it’s possible to do a good job parsing things manually, it’s time consuming and prone to error.  With an API, it’s faster and easier to code and you get data back in a predictable format.

If what you want to do can be done via SSH, Palo Alto has included a “Secret Decoder Ring” to help you figure out the API…  The secret is that the WebGUI and CLI both use the API whenever you do most anything.  So, in the CLI you can simply turn on “debug cli on”, and get most of the XML you need to pass to issue your API call by watching what the CLI does.  For example, if I do a “show jobs all”, I get this XML back:

<request cmd=”op” cookie=”8856737959639002″ uid=”500″><operations><show><jobs><all/></jobs></show></operations></request>

To do an API call to get the status of all your jobs, add in the blue and red portions from above appropriately:

http(s)://hostname/api/?type=op&cmd=<show><jobs><all/></jobs></show>&key=[Your API Key]

To reboot your firewall via the API:

http(s)://hostname/api/?type=op&cmd=<request><restart><system></system></restart></request>&key=[Your API Key]

Granted, there are some things I’ve not been able to figure out how to do via the API, like checking for the existence of an imported config file.  Via the CLI, just enter “show config saved ” and hit TAB after the last space.  The auto-complete feature of the PA CLI will show you a directory listing of saved config files.  If you do this with debugging turned on, you’ll note that you don’t see any “debug” info, so the autocomplete function must not use the API (or debugging autocomplete is disabled for readability purposes).

I expect that everything I need to do relative to the installation process can be handled via the API:

1. Import a pre-generated configuration file
2. Load the imported configuration file
3. Issue a local Commit
4. Check the status of the Commit
5. Read the Serial Number of the remote device being installed
6. In Panorama move the device from the “Pending” device group to the “Production” device group
7. Issue a Panorama commit for this device (by Serial Number)

If you have any need to programmatically interact with a Palo Alto firewall, I encourage you to dig into the API.  There’s tons of very good data, just waiting to be accessed.  Very easily.


July 23, 2015 at 7:33 pm Leave a comment

F5 GTM iRule to enforce Google Safe Search

There are ton’s of tools you can use to enable Google Safe Search…  Essentially, you need to serve a custom record for that’s a CNAME pointing to

Anyhow, for our Customer Wifi, we want to take some steps to limit the visibility of adult results to our customers (both for liability and PR reasons).  Since we have a large number of retail locations, all running through a central data center, we run a high performance DNS cache using our F5.  While I’m sure there are lots of ways to solve this issue, we created an iRule to handle it:

when DNS_REQUEST {  
if { [DNS::question name] == "" } {   
set lookup "[RESOLV::lookup @[RESOLVING DNS SERVER HERE] -a ""]"
set ip [getfield $lookup " " 1]
DNS::answer insert " 300 IN CNAME"
DNS::answer insert " 300 IN A $ip"
if { [DNS::question name] ends_with "" } {
DNS::answer clear
DNS::header rcode NXDOMAIN

Just replace the text “[RESOLVING DNS SERVER HERE]” with the IP address of a server capable of resolving the forcesafesearch DNS query.  If you are using Route Domains, don’t forget to include it on the end of your DNS server IP.

As a bonus, this iRule also blocks, the domain that Bing uses to display thumbnails/videos for explicit content.

July 15, 2015 at 5:59 pm Leave a comment

Making administrative web apps in PHP

What do I mean by administrative web apps?  Basically, a web app that lets you update a series of database tables,

Let’s take my most recent one as an example.  We are about to deploy LTE capable routers (Referred to as Modems in some places in this blog entry) to our remote locations.  800+ of them.  Each remote site will have an LTE router with modem and two SIM cards.  One of those SIMs will be active and the other will be there so we could switch to it, in the event we have trouble with the primary SIM (Vendor A has trouble, Vendor B might work better).  Each Vendor requires certain data when activating a SIM.  One wants the modem IMEI, another wants the modem MEID.  They all want to know which SIM ID is involved.  Oh, and each SIM is associated with a static private IP address.

The easiest way to manage such a thing is probably via a database table that can be edited via a web interface.  If it’s just you doing it all, perhaps you can edit the tables “in the raw” using something like PhpMyAdmin.  But, if you have a team involved, you might want to dial back the control a bit.  This is where you’d want to built a website to manage these database tables intelligently.

Rule #1:  Use a Web Framework!

A couple years back I learned about Bootstrap, the framework that lets you easily create clean, professional looking websites.  Using one of the frameworks will make your web apps look clean and professional.  Just pick one, and stick with it.

Rule #2:  Authenticate!

Whenever you are creating any sort of administration web app, make sure authentication plays a part.  You want to be sure that the people using your web apps are supposed to be using it.  While this is a “duh” statement for anyone writing publicly available apps, it also holds very true if it’s a private app that’s only visible to company employees.  In my case, I had a rudimentary system in place for a very important page, but many others were wide open.  While revamping the system using Bootstrap, I created a simple radius-based authorization include file to add to all the pages I wanted to secure.  I used a radius class I found online, I think this one.  I actually love it that I’m not very familiar with the class, as I’ve had to do so little with it.  I pretty much dropped it in, and it’s been working great ever since.  Since it’s just a single “include” line, securing other pages is drop-dead simple.

Rule #3:  Use Editor

This is an amazingly smart library of code.  With it, you can build a modern AJAX enabled web interface to manage your database tables very easily.  You may have to pick up a little bit of Javascript knowledge, but the back-end is PHP, and if you are half decent at PHP, you should be all set.

They have example code galore.  The majority of what you probably want to know how to do is right there.

If what you want to do isn’t listed in the examples, just ask!  Support on their forum is very good.  Most of my problems seem to stem from a lack of knowledge on the Javascript side of things.


Ok, time to talk about the elephant… No, Editor isn’t free.  It’s $119 if you are a solo developer, and goes up from there depending on the size of your team.  But trust me, it is so worth it.  If you were to try to write your own class library to do all the things Editor does, you’d spend many, many hours doing it, making the price tag a bargain.

Rule #4: Make an Audit Trail!

Any time you build a web interface that allows users to edit database tables for anything important, you should include code to audit the database tables, so you’ll see exactly who made what changes.  If you’ve used an authentication include, as I suggest, you can probably grab the logged-in user so you can write that into the audit trail as well.  I’m not suggesting this so you can beat up on the guy who made the mistake.  It’s so that you can quickly look back, see what has changed, so you can quickly fix the mistake.  It also allows you to do a little remedial training with whoever made the mistake, so they won’t make it again.

Now, an audit trail when using Editor is a bit of a challenge.  Unfortunately, the PHP code for Editor doesn’t include a smart audit capability or anything similar.  If you are serious about this, though, you can find the driver file for the database type you are using and modify it to create your audit trail.  Heck, a timestamped log file including the username and the UPDATE, DELETE, and CREATE SQL queries is probably all you need.  I actually parsed the SQL and wrote it out in an audit table so the NOC team can look through it (in another tab in the web interface) and figure out what happened, but that’s probably just me.

Rule #5:  Consider the Work Flow!

If you are writing this for someone else to use, it may be tempting to quickly write it in the fastest way you can, and then move on.  Don’t do it.  In my case, my users will be using this to keep 800+ sites straight.  That’s a big job, so I’m trying to make it as easy as possible.

Consider the things the users will need to do with the application.  When adding a new modem to my database, they’ll first scan in the IMEI (using a barcode scanner), then the MEID, so those are the first two fields on the “Create Modem” page.  The next item is the location number that will get this modem, then they can select two SIMs, and then a Vendor dropdown to indicate which SIM is active.  Design with the workflow in mind.

Build in logic to keep errors from happening.  In my case, I’m repopulating the SIM dropdown lists to only include SIMs that have not been selected before, since a SIM can only be in one modem at a time.  Similarly, if the user has selected AT&T and Verizon SIMs, don’t let them select Sprint as the Active Vendor.

Handle “special things” in the web app.  In my case, I’m going to have some LTE Survey kits that I’m giving “fake” location numbers.  I’ve added logic to the tool to prevent records marked with “fake” location numbers from being edited by the user.

July 4, 2015 at 6:40 pm Leave a comment

CradlePoint and NTP

CradlePoint routers have one interesting habit that I’ve noticed.  If they can’t connect to the WAN, they won’t try to sync with the NTP server it is configured for.

On the surface, this seems smart.  Don’t bother trying to sync time unless you have a path to the Internet, right?

But what if you have a local time server?  In that light, this is a poor decision.

Perhaps a bit more complexity to their code would satisfy everyone.  If the NTP server that is configured is an IP address and it’s one that falls into RFC1918 (private IPs), then go ahead and try to sync as soon as you boot up.  If it’s a DNS name, or a public IP, then wait until you get a WAN connection.

Anyhow, I noticed this some time back, but didn’t think too much of it.  As it turns out, it can cause an issue in our particular use-case.

In my previous post, I outlined that we are using the API to perform speed tests at remote sites, using what I call an “LTE Survey Kit”, to gather data about all three of the major carriers at once at each remote location.  We noticed an anomaly in the speed test results.  Most of the time it works fine, but occasionally we get back a completed speed test with a speed of 0.00.  That’s right.  It told us that it FINISHED the test in the 60 seconds we provided, but the speed was calculated to be 0.00.

How can that be?  If it took the full 60 seconds to download 1 MB of data…  Well, let’s do some simple math:  1000 KB / 60 seconds = 16.67 KBps.

So, if it was as slow as possible, while still completing, I’d expect a result greater than 0.00.

A bit of testing today discovered that on the few occasions we see this anomalous result, the test started before an NTP sync was done, and finishing after it completed.  The speed was calculated as if it took from sometime during 1969 until present day 2015 to complete that speed test, hence the speed of 0.00 Kbps.

So, how to fix?  Well, we could simply remove the NTP server config so it doesn’t sync, but I thought perhaps adding a wait loop would be better.  So now, when this test runs, after the CBA850s come up on the individual carriers, I check to see if NTP is synced (another API call),  Once all the devices with active WAN links have synced with NTP, then we can start the test.  As a side benefit, this gives each of the devices a little more time on the carrier prior to kicking off a test.

Also, I found out today that if you query the API (/api/status/rf I believe) you get back an array of all the results of the SINR.  This is polled every 18 seconds by the CradlePoint and could be very useful in checking for stability.  Having the extra wait time for NTP sync means we can collect data a little longer.

July 2, 2015 at 8:01 am Leave a comment

Can’t trust Cell coverage maps? Make your own!

At my day job, I’m responsible for the network connectivity for hundreds of remote locations.  We use 3G today as a backup to our T1 circuits.  One thing most IT people who have tried to put in a cellular network can probably attest to:  Vendor provided cellular coverage maps absolutely suck.  Even if you give the vendors a list of your addresses, you’ll get a far rosier picture of the coverage they can provide than exists in this reality.

So, what do you do when the vendor provided data isn’t any good?  

You make your own!


We’ve taken a small plastic tub, a consumer grade 5 port Ethernet switch, three CradlePoint CBA850’s, and a Quirky power strip to handle the huge power bricks the CBA850s need.  We’ve mounted the power strip to the bottom of the tub,  the switch to the side of the tub on one end, and the three CBA850s to the sides, so that their antennas fold down just below the top.  The CBA850s are pre-wired to the Ethernet switch, along with an extra CAT5 cable about 10 feet long.  The CBAs and the switch are plugged into the power strip.  The CBA850s are configured for AT&T, Sprint, and Verizon, so we can get a good picture of the coverage of all three carriers.  They are configured with static IPs not used elsewhere in our locations.  When our technician arrives at a location, he removes the top of the case, folds up the antennas, finds a power outlet and a free Ethernet port and plugs in.  A quick call to our NOC to let them know which location he’s at and which port to turn up is all that’s left for the tech to do.  At that point, our NOC staff can kick off a script which updates the router config for that location to NAT the CBA850s to addresses specific to that store, allowing them to be reached from headquarters.

Then, using the API magic I mentioned in my last post, the script validates that all three CBA850s are reachable, then it checks in with them to see if the WAN is connected, waiting around 5 minutes for any stragglers.  Once they are all up, or the expiration time has passed, it kicks off a series of speed tests, both upload and download, gathering the results of the tests along with other diagnostic info (SINR, signal strength, etc).  Drop that data into a database table, and there’s our “map”.

That’s no MAP!  That’s just a bunch of numbers!

No, our “map” won’t look like a map, but it will have data telling us which of the three main cellular providers is the best at every one of our locations that we’ve tested.  From the perspective of our management, that’s really all that matters.

July 1, 2015 at 8:39 am Leave a comment


July 2015

Posts by Month

Posts by Category