Posts filed under ‘Networking’

Wifi woes

I switched a few months ago to an AC router that’s sort of pro-sumer grade.  It was actually one I got on kickstarter that I was really excited about, as it dealt with home automation as well.  When I got it, I tried to use it, but it did not work very well.  After perhaps a year, I found that it was fairly usable.  I get decent coverage almost everywhere in the house.  Almost.  Also, sometimes my wife’s iPad has trouble (while I, a few feet away with my iPhone, don’t)…  The other day I ended up rebooted the router to get everything to recover, as my iPhone wasn’t working on wifi either.

Anyhow, a few months ago, I pre-ordered a Luma system to resolve my Wifi woes.  Around mid-May I believe, I got an email from Amazon saying my Luma would arrive by 7-20.  I’ve been watching the last few weeks with interest, but was very unhappy to find that it still had not shipped on Monday… Or Tuesday.  Today, I got an email from Amazon basically saying they don’t know when it will ship.  The word on Luma’s Facebook page is that Amazon pre-orders should be delivered by 8-26.

I’m guessing someone at Luma messed up, or perhaps the problem is Best Buy.  Back in June, I think, Luma announced that Best Buy would be selling their product in-store.   So, they are diverting some stock (they say 5%) which would otherwise be going to fulfill pre-orders to Best Buy.  I’m not so sure I believe them.

Today, I decided to vote with my wallet.  I cancelled my Luma pre-order, and ordered an Eero instead.  Yes, I’m paying a significant premium over the Luma, but it will be here tomorrow.  I know, I could have ordered a Luma from Best Buy, or possibly walked into a store and found one.  But…

From some reading I’ve done, the Luma doesn’t quite live up to their advertising.  It seems like the features aren’t all there as shown in their introductory video.  Will it get there?

Probably, eventually.  I imagine it will be months of growing pains, waiting for new firmware and such to be released to get everything fixed, and the missing features in place.  I’ve grown tired of that (with the other unnamed router I talk about in paragraph 3, above).

I was giving them a chance, waiting since the end of April for this product.  But the shipping date was missed, and Amazon isn’t saying when it would ship…   Social media says it’s over a month away.

I’m moving on to a more mature product, one that probably won’t give me trouble right out of the box.

 

July 20, 2016 at 7:14 pm Leave a comment

Meraki AP Syslog to Palo Alto firewall for User ID

I recently got a Meraki AP as a demo unit. Using Palo Alto’s Syslog listener, you can get user-id info from these units, if you are doing 802.1X authentication.

Just follow the instructions here, with some adjustments…

Navigate to the Device tab, User Identification menu item, then the User Mapping tab. There, select the gear icon, and on the following pop-up screen, select Syslog Filters.
Add a new filter, with these properties:
Profile Name: Meraki AP v1.0.0
Type: Regex Identifier
Event Regex: 8021x_eap_success
Username Regex: identity='([a-zA-Z0-9\\\._]+)
Address Regex: client_ip='([0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3})

Then, use your newly created filter for your Syslog Listener.

In my experience, it looks like the Meraki only logs authentication events every so often. Perhaps it is caching them? At any rate, set the Cache timeout value to something greater than the default 45 minutes. I set mine to 480, though this may need tuning, depending on the environment.

Also, be aware that the first time you authenticate after setting this up, you’ll probably show up in the ip-user-mapping with no IP address. That’s because when you initially authenticate, the first Syslog message from the Meraki shows an IP of 0.0.0.0. Subsequent authentication attempts have your IP address in them. Not sure how this works out in the long term.

I wouldn’t say this is quite production ready, but it is definitely worth playing with, if you happen to have both a PA firewall and a Meraki AP.

March 2, 2016 at 7:04 pm Leave a comment

Get Enterprise Wireless security at home for free!

A week or so ago, I had a bit of a scare with my NAS. In addition to storing my files, my Synology NAS also serves as a radius server for my wireless network.

So, while I was trying to fix my NAS, my wireless network was basically down. When it looked like my NAS wasn’t going to be back in operation for a few days/weeks, I started looking around very quickly for a hosted radius solution and found a great looking free option for a home user with a single AP. IronWifi.com.

It’s a pretty basic web interface, but it provides all you need. Sign up for the free level of service, add a network, configure your AP to point to the provided radius server using the random radius secret they provide. Try to connect with a device (it won’t yet, but that will register your AP with them). Next, approve your AP, then add your users in their web interface. After that, just connect to it with your wireless devices. For IOS devices (probably other device also), you’ll be prompted to accept their certificate, and asked for your username and password (which you set up in the ironwifi.com interface).

It’s possible I left out a step or two, but it really was very, very easy to get running. I did have issues using the admin web interface with Safari, but using Chrome it worked fine.

If all goes as well as it did for me, you’ll have your devices up and running in minutes and you can sleep secure in the knowledge that you are protecting your wireless network with much better security than using the simple PSK method 99%+ of home users are using. (Ok, I made that stat up, but it’s probably pretty close)

Oh, if you are wondering, my NAS came back to life. I’ve still not switched back from IronWifi though.

February 15, 2016 at 10:38 pm Leave a comment

Importance of Traffic Logs even for the home network

My little firewall logs just about everything that goes on. Blocked? Log it. Allowed? Log it. Most of the time, these logs roll over and I never even see the contents. However, every once in a while they come in very handy.

My wife usually spends a little while on Sunday evenings preparing attendance sheets for CCD (think Sunday school, but for Catholics). Our parish takes it very seriously, and they have given her a remote login to their data software, so she can update the attendance on-line, and they’ll have accurate records. This software appears to be SaaS (Software as a Service). Unfortunately, it’s not a web-based service. It is hosted on some remote system, and they provide her with something akin to a Citrix login to access the data. This software is PDS (Parish Data System) by ACS Technologies.

Recently the UPS on her computer started acting up. We had a quick blip tonight and her computer rebooted. When it came back up, she proceeded to connect back to this software, and was prompted with a small box asking for the Host. We don’t recall this being asked previously, as it usually just pops up a login box.

So, we checked the support website to see if they had any hints. A quick look around there seems to show that to get to any real support info, you need a Site code and a PIN, and my wife doesn’t know that. Their Live Chat support didn’t work. The only other options are Email (which also seems to require site details) and a toll free number, but they apparently don’t work weekends.

Thinking about the situation logically, I concluded that somehow this system “forgot” the remote hostname to which it normally connects. That’s what it’s prompting for connection details with a “Host” prompt.

It struck me that I might be able to find it in the logs, so off to my firewall I went. I filtered by my wife’s IP address, and tried filtering for the application “Citrix”. Zilch. Next, I started filtering out ports and applications that I knew it wouldn’t be, and told the firewall to lookup hostnames. Finally, after filtering out port 80, Facebook-base, Facebook-chat, iCloud-base, Twitter-base, and port 993 (secure Gmail in this case), I jumped from page 1 to page 10 (to get to a more appropriate time, prior to the power outage), and there it was. I recognized the name “spr.connections.ondemand.acstechnologies.com”, so I tried that as the host. I believe at that point, I got a different error. So, we closed and restart the application, and it popped up and worked just fine.

So, if you have lots of logging going on with your firewall at the house, don’t bother trying to weed it down, just let it go. One day, it just might save you lots of time.

September 26, 2015 at 9:34 pm Leave a comment

Monitoring a network with EIGRP

Most network monitoring involves polling.

So, you have a server (or farm of them) going out across the WAN every minute or so, talking to every remote device to ensure that they are up and running.

There are a number of products out there that do this, but what if you can do it smarter?

At my day job, we have hundreds of remote sites connected via T1 and they have an alternate link, soon to be LTE across the company.  We run EIGRP across our links so our routers know which links are available for traffic.  Yes, even our LTE links.  They all terminate on GRE tunnels on one router.  We set the EIGRP Hello time to 20 seconds and the Hold time to 60 seconds.  If 60 seconds pass without seeing a Hello, the link gets marked down.

I wrote a PHP program to handle this monitoring in a very efficient way.  Every minute, it performs an SSH into this router and runs a “show ip eigrp neighbors” command to get a list of all active neighbors.  This tells me that each of those neighbors are active at the time I performed the command.  I log this info to a database table.  I also run a command like “show ip route | inc Tu”.  Due to our database, my program knows which EIGRP neighbor is each location and which route belongs to each location.  If I see a connected route to any Tunnel, I know we are actively running traffic across the LTE link to that location.  Since this is done every minute, I’m logging each time that a remote device has an EIGRP connection to headquarters.  I track the state of all the locations and send SNMP traps to our central manager to create alarms when I see that an EIGRP connection that should be there is missing and when a route exists (meaning the LTE link is being actively used).

This database is tracking the total number of polls and the number of successful polls.  This lets me calculate an “Availability” number for that GRE Tunnel.  Note, this isn’t a real “Availability” number for the LTE link.  It’s an Availability number for the Tunnel, meaning it can easily be worse than the LTE link availability (if the remote router is down, perhaps).

If you described this to me as a monitoring solution, I wouldn’t expect it to work well.  The fact is that we’ve been running with this sort of solution for several years.  The difference now is that I’ve reduced the polling cycle from every 5 minutes to every minute to give me better granularity.  And it still works great, even with 150+ sites.  The beauty of this system is that adding more sites doesn’t really add more time (technically, it does, but it’s such a small number that it’s pretty much irrelevant).

September 18, 2015 at 9:59 pm Leave a comment

Best Cell Carrier coverage in the SouthEast US

Where I work, we wanted to put in LTE backup at all of our retail locations to handle communications in the event that our T1 circuit fails.  There are around 800 locations stretching from Louisiana, south to Key West, all the way to North Carolina.  We have relationships with the big three carriers, so we build survey boxes housing three CradlePoint cellular broadband adapters, one configured for each of the carriers, then took them around to our locations and ran a battery of Netperf tests to get real results for each location which were logged into a database.

Armed with that database of over 7000 test results, we selected the best carrier at each location by looking at the raw data.  My general criteria?  Look for the carrier with the best SINR (Signal to Interference + Noise Ratio), along with the best speed.  We are less concerned with cost, since they are all under $30 a month for our limited, pooled data plan.  Our goal is that we have a reliable backup that is at least as fast as the T1 circuit it would be “covering for” in the event of a T1 outage.  Most T1 outages would be measured in hours, so it needs to be available when we need it, first and foremost.  That said, we want better than 1.5 Mbps in both directions so that it can be a true T1 backup.  Looking at the data and making the selection was sometimes difficult, but we made our best guess in those cases.

I only have the actual numbers for the first 155 locations we have installed, which break down as follows:

AT&T was selected 50.9% of the time.
Verizon was selected 30.9% of the time.
Sprint was selected just over 18% of the time.

From the numbers I have seen (in passing), this pattern is pretty representative of the overall totals.

Now, I’m not much of an AT&T fan, but this is pretty impressive.

 

September 18, 2015 at 9:37 pm Leave a comment

Mass upgrading Palo Alto firewalls

My company just bought 900 PA-200 firewalls.  Unfortunately, they all are pre-loaded with firmware version 5.0.6.  The current version is 7.0.1.  To get from 5.0.6 to 7.0.1, you must install a newer content version, then upgrade to version 6.0, then to 6.1, and finally to 7.0.1. Oh, and we want to install A/V as well, in preparation for shipping them to the stores.

They have a product called Panorama that manages their firewalls (can’t manage hundreds of firewalls without, if you ask me).  It can perform upgrades from one version to another, but isn’t smart enough to know what steps must be taken to get from 5.0.6 to 7.0.1.  Someone would need to know the process, and direct Panorama to do it, each step of the way.  Since I have 900 of them to upgrade, I needed to come up with a better way!  Waiting until they were at the store connected via a T1 circuit is not a good option either, as the content, A/V, and all the firmware upgrades would be over 1.1 GB in size.

A great feature for Panorama would be to have a “base” template you set for each Device Group.  That “base” template would include things like what Content and A/V versions, and what firmware for all the devices in the group.  Whenever devices are added to this device group, Panorama should automatically set them to the proper content, A/V, and firmware versions.

But, since Panorama isn’t that smart yet, the Palo Alto API and scripting magic to the rescue.

Since I’ve been writing a script to handle our installation process, I written a Palo Alto class to handle all the communications to the PA-200s and to Panorama.  I did have to add a few more routines to the Palo Alto class to handle everything that I needed, but it now works.

Our process works this way:
1.  A tech unpacks 10 PA-200 firewalls and attaches their Management port to a subnet on our corporate network.
2.  The tech scans the serial number bar codes on the back of the PA-200s, adding them to Panorama as “Managed Devices”.
3.  The tech adds them to the appropriate Template and a special device group that exists just for the upgrade process.
4.  The tech sets an IP address, Mask, and Gateway on each unit, pointing them to DNS servers and the Panorama server, then commits the change.  (This is a copy/paste process where the IP is different for each of the 10 units being upgraded.)
5. Finally, the tech performs a Commit in Panorama.
6.  The tech then gets back to other work, waiting for an email that will be sent once all the devices are upgraded.  This should happen about 1:35 to 1:45 minutes after the Panorama commit is done.

The real work gets done in a script that runs every 5 minutes.  This script:
1.  Gets a list of all the devices in the special device group.
2.  Attempts to create an object of my custom PA class for each device.  If it can’t communicate to it, that one is discarded for now, since this script will retry in a few minutes.
3.  Panorama is checked to make sure there are no active jobs for this serial number.  If so, it’s removed from further checks.
4.  Each firewall is checked to make sure there are no active jobs.  If so, it’s removed from further checks.
5.  The content version is checked for each PA-200.  If one isn’t found, it’s serial number is added to the Content queue and it’s removed from further checks.
6.  The anti-virus version is checked for each PA-200.  If one isn’t found, it’s serial number is added to the Anti-Virus queue and it’s removed from further checks.
7.  If the firmware starts with “5”, it’s serial number is added to the 6.0 upgrade queue and it’s removed from further checks.
8.  If the firmware starts with “6.0”, it’s serial number is added to the 6.1 upgrade queue and it’s removed from further checks.
9.  If the firmware starts with “6.1”, it’s serial number is added to the 7.0.1 upgrade queue and it’s removed from further checks.
10.  If 7.0.1 is installed, it sets the IP address back to the default and issues a commit.
11.  Finally, if 7.0.1 has been installed, and the box is unreachable (because the commit has taken effect), the device is removed from the special device group and moved to a Pending group.
12. All the various “queues” I mentioned get kicked off, with the serial numbers of the devices that need that step performed passed to Panorama via the XML API.  There’s additional logic to send emails when all the devices are out of the device group.

In practice, this is taking about 1:35 to fully upgrade 10 firewalls, though I suspect we could ramp this up to 20 or more, and it would likely take very close to the same time, since Panorama is upgrading all the devices in parallel.

This will have to do until Palo Alto upgrades Panorama to do it for me.

August 9, 2015 at 5:08 pm Leave a comment

Palo Alto and the power of an API

We recently bought Palo Alto PA-200 firewalls for our retail locations to replace our aging CheckPoint UTMs.  I didn’t investigate their API at all during the time we were looking at CheckPoint competitors.  I knew it had one, but hadn’t really given it a lot of thought.  Now that we have a massive roll-out ahead of us, I’ve started scripting parts of the process.  I must say that I love the flexibility that their API gives us.

In the past, for any major roll-out, I’ve scripted the process using telnet / SSH / HTTP (for web scraping), basically whatever interface the vendor allowed.  My goal is to make the installation fast and easy to support, while reducing the chance of human error as much as possible.  The hassle with CLI scripting for remote devices is always the parsing.  While it’s possible to do a good job parsing things manually, it’s time consuming and prone to error.  With an API, it’s faster and easier to code and you get data back in a predictable format.

If what you want to do can be done via SSH, Palo Alto has included a “Secret Decoder Ring” to help you figure out the API…  The secret is that the WebGUI and CLI both use the API whenever you do most anything.  So, in the CLI you can simply turn on “debug cli on”, and get most of the XML you need to pass to issue your API call by watching what the CLI does.  For example, if I do a “show jobs all”, I get this XML back:

<request cmd=”op” cookie=”8856737959639002″ uid=”500″><operations><show><jobs><all/></jobs></show></operations></request>

To do an API call to get the status of all your jobs, add in the blue and red portions from above appropriately:

http(s)://hostname/api/?type=op&cmd=<show><jobs><all/></jobs></show>&key=[Your API Key]

To reboot your firewall via the API:

http(s)://hostname/api/?type=op&cmd=<request><restart><system></system></restart></request>&key=[Your API Key]

Granted, there are some things I’ve not been able to figure out how to do via the API, like checking for the existence of an imported config file.  Via the CLI, just enter “show config saved ” and hit TAB after the last space.  The auto-complete feature of the PA CLI will show you a directory listing of saved config files.  If you do this with debugging turned on, you’ll note that you don’t see any “debug” info, so the autocomplete function must not use the API (or debugging autocomplete is disabled for readability purposes).

I expect that everything I need to do relative to the installation process can be handled via the API:

1. Import a pre-generated configuration file
2. Load the imported configuration file
3. Issue a local Commit
4. Check the status of the Commit
5. Read the Serial Number of the remote device being installed
6. In Panorama move the device from the “Pending” device group to the “Production” device group
7. Issue a Panorama commit for this device (by Serial Number)

If you have any need to programmatically interact with a Palo Alto firewall, I encourage you to dig into the API.  There’s tons of very good data, just waiting to be accessed.  Very easily.

 

July 23, 2015 at 7:33 pm Leave a comment

F5 GTM iRule to enforce Google Safe Search

There are ton’s of tools you can use to enable Google Safe Search…  Essentially, you need to serve a custom record for http://www.google.com that’s a CNAME pointing to forcesafesearch.google.com.

Anyhow, for our Customer Wifi, we want to take some steps to limit the visibility of adult results to our customers (both for liability and PR reasons).  Since we have a large number of retail locations, all running through a central data center, we run a high performance DNS cache using our F5.  While I’m sure there are lots of ways to solve this issue, we created an iRule to handle it:

when DNS_REQUEST {  
if { [DNS::question name] == "www.google.com" } {   
set lookup "[RESOLV::lookup @[RESOLVING DNS SERVER HERE] -a "forcesafesearch.google.com"]"
set ip [getfield $lookup " " 1]
DNS::answer insert "www.google.com. 300 IN CNAME forcesafesearch.google.com"
DNS::answer insert "forcesafesearch.google.com. 300 IN A $ip"
DNS::return
}
if { [DNS::question name] ends_with "explicit.bing.net" } {
DNS::answer clear
DNS::header rcode NXDOMAIN
DNS::return
}
} 

Just replace the text “[RESOLVING DNS SERVER HERE]” with the IP address of a server capable of resolving the forcesafesearch DNS query.  If you are using Route Domains, don’t forget to include it on the end of your DNS server IP.

As a bonus, this iRule also blocks explicit.bing.net, the domain that Bing uses to display thumbnails/videos for explicit content.

July 15, 2015 at 5:59 pm Leave a comment

Can’t trust Cell coverage maps? Make your own!

At my day job, I’m responsible for the network connectivity for hundreds of remote locations.  We use 3G today as a backup to our T1 circuits.  One thing most IT people who have tried to put in a cellular network can probably attest to:  Vendor provided cellular coverage maps absolutely suck.  Even if you give the vendors a list of your addresses, you’ll get a far rosier picture of the coverage they can provide than exists in this reality.

So, what do you do when the vendor provided data isn’t any good?  

You make your own!

 

We’ve taken a small plastic tub, a consumer grade 5 port Ethernet switch, three CradlePoint CBA850’s, and a Quirky power strip to handle the huge power bricks the CBA850s need.  We’ve mounted the power strip to the bottom of the tub,  the switch to the side of the tub on one end, and the three CBA850s to the sides, so that their antennas fold down just below the top.  The CBA850s are pre-wired to the Ethernet switch, along with an extra CAT5 cable about 10 feet long.  The CBAs and the switch are plugged into the power strip.  The CBA850s are configured for AT&T, Sprint, and Verizon, so we can get a good picture of the coverage of all three carriers.  They are configured with static IPs not used elsewhere in our locations.  When our technician arrives at a location, he removes the top of the case, folds up the antennas, finds a power outlet and a free Ethernet port and plugs in.  A quick call to our NOC to let them know which location he’s at and which port to turn up is all that’s left for the tech to do.  At that point, our NOC staff can kick off a script which updates the router config for that location to NAT the CBA850s to addresses specific to that store, allowing them to be reached from headquarters.

Then, using the API magic I mentioned in my last post, the script validates that all three CBA850s are reachable, then it checks in with them to see if the WAN is connected, waiting around 5 minutes for any stragglers.  Once they are all up, or the expiration time has passed, it kicks off a series of speed tests, both upload and download, gathering the results of the tests along with other diagnostic info (SINR, signal strength, etc).  Drop that data into a database table, and there’s our “map”.

That’s no MAP!  That’s just a bunch of numbers!

No, our “map” won’t look like a map, but it will have data telling us which of the three main cellular providers is the best at every one of our locations that we’ve tested.  From the perspective of our management, that’s really all that matters.

July 1, 2015 at 8:39 am Leave a comment

Older Posts Newer Posts


Calendar

January 2022
S M T W T F S
 1
2345678
9101112131415
16171819202122
23242526272829
3031  

Posts by Month

Posts by Category