FLASHBACK: Assembling the Team to Take on Hurricane Katrina; Part I
Post by Sigmund Solares
Chief Executive Officer
Intercosmos Media Group, Inc.

Editors note: After more requests for retelling our story than I can count, in these new Flashback posts, I am going to provide additional and corrected details of what happened before, during and after Hurricane Katrina hit. Today I am going back to Saturday, August 27, 2005, just prior to Hurricane Katrina hitting.

There were some ominous signs about Hurricane Katrina from early on. Many seemed to “just have this feeling.” Two days before the storm hit, Michael Barnett said in this blog with the only post of the day, “Hmm. This could actually be a nasty storm.” (see
http://www.livejournal.com/users/interdictor/2005/08/27/ )

Many were becoming visibly nervous. The thought that this would be a very nasty storm was starting to predominate people’s thoughts and discussions in New Orleans. By mid-day, many employees had already evacuated, and almost all employees planned to evacuate.

It was the last part that made me think twice – everyone planned to evacuate. New Orleanians don’t evacuate, they throw hurricane parties. Every year we seem bombarded by warnings of hurricanes getting bigger and worse, and of small storms that “might” turn to monsters. I guess we all got a bit numb.

Hurricane Katrina was a wake up call before it even hit land.

---

In the many storms over the past 5 or 6 years, I never required an employee to stay behind to run our operations when a hurricane was approaching the city. Though I somehow knew this time we probably needed the help, I was not about to start a new policy and require people to take such a risk. We have always left the decision to the employees, and luckily we have always had enough employees who volunteered to stay that we were able to continue to operate the data center. Worst case scenario I knew that I could hire additional help, if necessary. As uneasy as the approaching storm made me on the inside, and as the chief guardian of 400,000 clients’ Internet communications, I still felt very confident of our abilities that day.

I made mental checklists of all of the advantages on our side. For starters, the weathermen have nearly always been wrong on big storms in this area and/or the storms weakened before hitting land. This probably won’t be the “Big One”, I rationally assured myself, after considering previous experience and calculating the odds in my head.

But, I wanted to be sure. Much of the equipment in the data center can be operated from a remote location and most of the work for running Intercosmos can be performed remotely. So I knew the datacenter could be maintained with a minimal staff staying behind.

Consequently, it was logically vital to our weathering any sort of disaster to have people both at the office and working safely away from the office. Letting the employees choose where they wanted to be, so far, had always yielded a happy mixture. Furthermore, we had enough diesel in the primary tank and redundancy built in so that employees could evacuate during a storm, and then re-enter a few days later, or so I thought.

Another factor in our favor was the redundant bandwidth pipes providing us huge entry to the Internet through multiple, major carriers. Our generators and backup system were ready and tested.

So there it was. Ready. Set. Wait.

Soon it would be game time.

---

On a professional sports team, having a good bench to draw from when your starting players are taken out of the game is very important. It can mean the difference between winning and losing. Likewise, in running a business, it is important to have a good bench. If you lose most of your starting players permanently or temporarily, can you continue to operate your business with a winning strategy?

Saturday evening before Katrina hit, I was finalizing who would be staying in the data facility when the storm hit. “If it hits” was now wishful thinking and had left everyone’s vocabulary, right along with the tens of thousands of people who had already evacuated the New Orleans area.

It was around 5 p.m. when I received a call from Paul C., who operates our Data Center. He asked for an update on the plan for operating the data center during Katrina. I told him I was staying at the 10th floor facility, along with any others who still wanted to stay -- and that it was up to him whether he stayed or whether he evacuated and worked remotely – there were benefits to both. He decided to evacuate. I had him take my small laptop with wireless access because I “knew” that if we had problems I would be at the office on a desktop, and I wanted to make sure he was online to help resolve technical problems from a remote location. Giving him back up internet access seemed like a good plan and added to my sense of overall confidence.

I then talked with our Vice President, Donny Simonton, to see if he planned to evacuate. His wife had already left to visit relatives in Tennessee, and in the days before he stated that he was thinking of riding out the storm at the office. Donny said that he was staying. With him staying here, I knew that I was close to having all of our bases covered. Donny is a workaholic with vast technical knowledge and he is very resourceful and, with time, can resolve any problem.

Next up was Michael Brunson, the former data center manager who saw us through the take-over of the facility from the beginning. He had more experience with the data center than anyone other than possibly Paul C., and it would be good to have him here if Paul was going to be working off site. Bingo! Michael said he planned to stay at the office during the storm.

Last but not least, Jim Macullum, a long-time customer support and network operations employee, said that he would be staying as well.

With four starting line-up employees planning to stay, I felt very comfortable that we could operate the facility even in the face of the growing storm. However, I wanted at least one more person at the facility as a backup. One of the owners mentioned that my best friend Michael Barnett was planning on riding out the storm, and that it probably would be a good idea for him and his fiancé to ride out the storm at the office, as opposed to their house. Me and Barnett… We had been through a lot in the past, and I knew that we could work well in a crisis. I knew that his background could come in handy. With just a quick call to Michael Barnett he committed to staying at the office.

Now we had a solid team to take on Katrina. Feeling confident, I finally went to sleep. Early in the morning, however, I was awakened by a phone call. A team member from Network Operations, Michael Brunson, was calling to let me know he needed to evacuate.

The storm was barreling toward us. Can we really handle it? Would the other team members decide to evacuate? Who should I get to replace Michael Brunson and other tema members who might evacuate? It was gut check time.

I figured we were prepared even for the worst case of being down without power for a week to 10 days, and that didn’t seem too realistic. We’d probably be out for a few days at most, I thought.

Little did we know.