Legion World Forums LEGION WORLD Science Police Headquarters Problems getting worse...

Today was NIGHTMARE-DAY! Globat.com just emerged from a major outage - the worst in company history and everybody - customers and staff alike - still feel extremely beaten up. Here's what happened:

At approximately 5:00am Pacific Time on Thursday, February 21, 2008 we suffered a major network outage, which effected nearly all Globat.com
customers, our own Web sites and service infrastructure as well as our phone systems.

Our primary network switch in our main datacenter in downtown Los Angeles failed completely and despite tremendous efforts by our technical emergency response team could not be brought back online. This switch is a major brand name piece of equipment and as such it contains internal redundancy. We were able to switch over to the back up circuitry and begin bringing the system
back online at which time, for yet unknown reasons, the entire devise, including backup system, failed completely.

After a short deliberation we decided to rush a new switch from our vendor's nearest warehouse to our data center. While waiting for the replacement we initiated a work around to at least get email service re-established. After
several attempts all email systems finally went live and back to normal at approximately 10:00am Pacific Time.

When the new switch arrived we went through the installation and testing process and as a result were able to bring all our services back online, but not before we had suffered a 5-hour outage for e-mail services and an 8-hour outage for Web related services. In Globat's entire 6 year history, we have never experienced anything even remotely like this! The outage was so severe
that it affected our own Web sites and all but one of our telephone lines so we were unable to communicate with our customers to update anybody about the status of the incident. We decided to ask some of our staff members to post updates on well known Internet forums, which helped a little in disseminating the information.

Now that everything is pretty much back to normal I would like to tell you that we ABSOLUTELY KNOW you count on Globat.com to maintain a high degree of uptime to host your Web sites and that we take this trust very seriously. Today, however, my team and I let you down and we all feel terrible because of that. This has never happened before and we will do whatever it takes to never let it happen again! I deeply and sincerely apologize for this outage and the inconvenience and problems it may have caused. We promise to learn
from the events of this dreadful day (today was also my wife's birthday, on top of everything).

We are now taking the following steps to prevent a recurrence of this type of failure. This switch is the only piece of equipment at Globat.com that
could possibly be a single point of failure. This equipment is usually highly stable and has built-in redundancy. In additional it is being vendor monitored 24/7. We will now keep a second switch, with its own built-in redundancy, in our data center to prevent a prolonged outage in the unlikely event of another switch failure. In addition, we are working with our phone vendor to create an automated failover system so calls will be routed directly to remote support sites in the case of a failure of our primary phone system.

While nobody can completely prevent any form of failures in the future, with these changes we can certainly minimize the effects of such failures and reduce the impact on you, our valued customers.

At the time of me writing this message all of our systems appear to be back up and running and I can assure you that our network operations team will continue to investigate this unfortunate event.

In closing I would like to thank the entire Globat.com team (especially Chris, Lou, Tim, Don and Rene) for working nonstop, without sleep or
interruption to resolve this issue and, of course, our customers for their many encouraging comments and especially their incredible patience. You are certainly very much appreciated!

Ben R. Neumann
Chief Executive Officer
www.Globat.com