CheckPoint/Sofaware FlashForward

October 30, 2010 at 10:22 pm 1 comment

Tonight, 10-30-2010, at 09:57 PM EST, it appears that all CheckPoint Sofaware based UTM boxes worldwide running at least 8.0 firmware rebooted. (Much like the plot to the TV series Flashforward, where practically every human worldwide blacked out at the same time.)

This hardware includes the CheckPoint Safe@Office series (probably Safe@Home too), the UTM-1 series, the ZoneAlarm Z100G, along with at least one re-branded product from D-Link (the DFL-CPG310) and perhaps others.

My company has hundreds of UTM-1 Edge units that are centrally managed.  We started having very unusual problems sometime before 8:30 PM EST. We were unable to communicate via SNMP or via the WebGUI to the majority of our UTMs.  They seemed to be intermittently dropping traffic, but mostly passing data as usual.  Oddly, while these problems were happening, we did find at least one that seemed unaffected.  It was still reachable via WebGUI and SNMP.

Of all our corporate UTMs, one isn’t managed by a central management server.  Oddly enough, this UTM was exhibiting the same problems as the rest.  Just after 10PM, we saw that the problem appeared to have cleared up.  Checking the logs on our UTMs, we saw that they all appeared to reboot at about 9:57 PM EST.  I even checked the UTM that seemed to have been unaffected by the problem.  It had also rebooted.  We initially thought it was an issue with the central management server, but then I looked at the location with the UTM that isn’t centrally managed.  It also rebooted.  At just about the SAME TIME.  I was completely floored at this finding, as I couldn’t see how this could have happened.

But wait, there’s more.  Myself and another technician were trying to troubleshoot this issue from our homes.  Both of us experienced issues connecting to our corporate desktops.  What do we have in common?  We both use UTMs like those in our corporate locations for our home connections to the Internet.  Rebooted at about the same time!

Further, I have another, older model CheckPoint Safe@Office unit (small business version of the UTM hardware) that I use essentially as an access point.  Guess what?  It rebooted too, just a bit before 10 PM!

We contacted CheckPoint support, and they stated that they have reports trickling in from other customers with the same symptoms.

Let me remind everyone reading – I really, really like these little firewalls.  They are just packed with features and are (normally) very, very reliable boxes.

As I mentioned, everything seems to be back to normal now.  But, this really has us unnerved, as just like the characters from the TV series, we don’t know if or when another FlashForward event will occur.

Update 1: This problem is confirmed on the latest firmware 8.1.46, as well as on 8.1.37, and 8.0.39.  I have access to UTMs running these firmware revisions, and they all had the problem.  The CheckPoint support technician we spoke with stated that, based on initial reports, it wasn’t tied to any particular firmware.

Update 2: Gave a bit more detail above.

Update 3: In the case of my company, our hundreds of UTM-1 Edge boxes do not directly connect to the Internet.  This pretty much rules out any sort of public worm causing this issue.

Update 4: According to other sources, this appears to have been related to a time change event that took place in some parts of the world yesterday.  Here’s to hoping that this isn’t a preview of next weekend, when we move our clocks here in the U.S.

Update 5: Final word from CheckPoint sounds like a timer overflowed.  They said it won’t happen again for 13.6 years.

Advertisements

Entry filed under: Networking.

NexentaStor Community Edition Great price on a Drobo-FS

1 Comment Add your own

Leave a Reply

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Trackback this post  |  Subscribe to the comments via RSS Feed


Calendar

October 2010
S M T W T F S
« Sep   Dec »
 12
3456789
10111213141516
17181920212223
24252627282930
31  

Most Recent Posts


%d bloggers like this: