Looking for a New Year's Resolution? Why not resolve to be ready for the next disaster?

Post by Sigmund Solares
Chief Executive Officer
Intercosmos Media Group, Inc.
http://directnic.com
http://intercosmos.com


FLASHBACK: Stockpiling Secrets for Operating a Data Center during and after a Catastrophe

Editors note: After uncountable requests for retelling our story, I am sharing the details of our experiences of what happened before, during and after Hurricane Katrina hit in this series of Flashback posts. Today are some candid suggestions for data center operators on surviving a catastrophe.


Supplies on Hand Are Key To Surviving

Just a couple weeks before Hurricane Katrina hit, my girlfriend said, “I am never taking you shopping again. Any time you go to the store you have to buy 25 of everything.”

There is some truth in what she said. I will buy 25 two-liters of diet coke at once for example. Of course, maintaining a stockpile of essential supplies (like Diet Coke) can be very helpful when there is a catastrophe

At Intercosmos Media Group, Inc.’s downtown New Orleans offices, we had everything that we NEEDED to survive Hurricane Katrina and to keep directNIC.com and Zipa.com operating.

However, with a little better planning the experience could have been less painful if we would have stockpiled more of what we WANTED. If you have to go through a catastrophe, don’t ignore common comforts. They are critical to good morale and keeping one’s cool, sometimes literally.



When Hurricane Katrina was headed our way with 175 mile per hour winds, I confirmed with the manager of our data center that our diesel tank was full and that we had four extra barrels of diesel. I knew everything else (in terms of essentials) was in order, so that was the only additional planning that I did. In hindsight, I should also have sought to stockpile some convenience items as we had done for some prior hurricanes.

Below, I will list the main supplies that we had for Hurricane Katrina, explaining which of the supplies came in handy and which did not; and then I will explain what procedural changes we are making after Hurricane Katrina to prepare in the event of another catastrophe, including what additional items we are stockpiling. Note that the following is a list of supplies that we had on hand to support a skeletal crew of only five individuals who were maintaining our data center:

Intercosmos’ Initial Disaster Checklist

Water:
1. 90 gallons of bottled water. 90 gallons for 5 employees will provide 3 gallons a day per person for 6 days. We used hundreds of gallons of bottled water as a result of Hurricane Katrina.
2. 35 cases of soft drinks.

Food:
3. 20 pounds of cold cuts
4. 360 slices of cheese (five packs of 72 slices each)
5. 25 loaves of bread
6. Over 100 boxes of Little Debbies – these lasted us until 2 months after Hurricane Katrina hit.
7. 18 inch by 36 inch by 36 inch cabinet full of assorted canned goods
8. 30 pounds of frozen hot dogs
9. 20 frozen hamburger patties
10. 10 frozen pizzas
11. 40 corn dogs
12. Vitamins

Heating, Cooking and Refrigeration:
13. Full size George Forman grill and small George Forman grill.
14. Large convection oven
15. Hot dog machine
16. Popcorn machine
17. 6 refrigerators
18. 1 freezer

Utensils:
19. 1000 paper plates
20. 1000 plastic forks, 1000 plastic knives, 1000 plastic spoons
21. Assorted dishes

Light:
22. 20 flashlights. Keychain flashlight came in very handy.
23. Batteries – 40 AA and 40 D
24. 200 foot of Christmas tree lights
25. 10 standing lamps

Power Generation:
26. 750 KW Onan Generator
27. Full 1500 gallons diesel tank
28. Four 55 gallon barrels of diesel
29. 20 empty 55 gallon barrels
30. 1 hand truck for moving 55 gallon barrels of diesel
31. 2 diesel pumps

Communication: Mail was not delivered for more than a month, Federal Express, UPS and other services were unavailable. Standard phone services were out.
32. Cell phones. After Hurricane Katrina hit cell phone service was sporadic at best for over a month. The only reliable form of communication over the cell phones was using the text paging service. Having a cell from issued from an out-of-state provider also was useful in many instances.
33. Radios.
34. IP phone system.
35. Websites: We had a large audience from directNIC.com. When the storm hit we placed a note on our main website which is visited by tens of thousands of individuals each day.
36. 4 redundant OC3s of bandwidth (3 of the 4 went down as a result of Hurricane Katrina)
37. Computers: Hundreds of servers in our data center and approximately 80 desktops. Through use of the computers we communicated by: email, ICQ, IRC, the blog, and a video feed.
38. Nikon D1X Digital camera with 2 extra batteries a 70 mm to 300 mm lens

Equipment: With a 20,000 square foot office at 650 Poydras we had much equipment that we did not use. I am only going to list the equipment that came in handy during or after the hurricane hit:
39. Dozens of very long extension cords
40. Duck tape
41. Two wet vacuums
42. 10 all purpose tool boxes
43. Circular saw
44. 8 foot ladder and 6 foot ladder
45. 20 pairs of yellow rubber gloves – these gloves proved to be completely inadequate for dealing with diesel.

Sanitation and hygiene:
46. Soap
47. Shampoo
48. Tooth paste
49. Four, 64-ounce bottles of hand sanitizer as well as many small bottles. We did not come close to running out of hand sanitizer. However, we have even more now.
50. 20 rolls of paper towels
51. 100 garbage bags
52. 10 bottles of bleach
53. Hand wipes – we had a small supply. Buy lots of baby wipes – they are nice sized for cleaning and very cheap.
54. Brooms
55. Vacuums

Safety:
56. 2000 disposable earplugs
57. 3 first aid kits
58. Gloves

Security:
59. Two 45 caliber hand guns with hollow point rounds
60. Flack jacket

Clothing: This is clothing that was stockpiled not including any brought by anyone staying for the hurricane.
61. 2000 ISPCreator.com t-shirts
62. 50 pairs of shorts

Shelter:
63. 27 story office building
64. 4 air mattresses – I brought additional better air mattresses to the office after the hurricanes. Amazingly we had better air mattresses at another location where we also had the pump to blow up the air mattresses.

Transportation:
65. Truck
66. Cars
67. Six battery-powered Segways

Money:
68. Cash – critical when credit card machines go out
69. Credit cards


Restocking

At some point you will be able to restock after a catastrophe. Keep in mind that it might take longer than you would normally expect.

Additional supplies that we had brought in after the hurricane:
1. Hundreds of gallons of water
2. 20 rolls of paper towels
3. Extra 500 gallon diesel tank full of diesel
4. 4 wheeler
5. Small generator
6. Several cases of diet coke
7. Bottled water
8. 409 cleaning – 4 bottles
9. Additional first aid kits
10. Lysol disinfectant spray
Junk food including:
11. Oreos
12. Pop tarts

Supplies that were brought in that we failed to stock ahead of time:
13. Work boots
14. Protective masks – to protect from mold when cleaning
15. Potato chips
16. Great Stuff for plugging holes
17. Sun shower – Shower used for camping. You can take a shower with two gallons of water
18. Toilet cleaning tablets


Don’t Keep All Your Supplies in One Basket: Compromised Supplies

Of course, sometimes supplies are compromised. It is very important to make sure that your supplies are both maintained in a safe location and that they are also accessible to you as well. Some mistakes that I made in preparing:
1. Our large truck was far away. So we could not get to it.
2. 5 refrigerators and a freezer were on the 11th floor where we lost power. So we had to move multiple refrigerators down a floor.
3. 2000 t-shirts were on the 11th floor and we needed them on the 10th floor.
4. 30 of the 35 cases of soft drinks were on the 11th floor and we needed them all on the 10th floor.
5. The cabinet of canned goods was on the 11th floor and we needed it on the 10th floor.

Furthermore, any supply that was not at 650 Poydras Street was not available to us because we locked ourselves in the building. A few days after the hurricane, we went to one of the other buildings to obtain an additional firearm and additional ammo.

Keep in mind not to keep all of your eggs in one basket. If you keep all of your supplies on the 1st floor of a building and the first floor floods, then you can lose everything. If you have everything on the top floor of a building and the building loses its roof, then you could still lose everything.

In planning for a catastrophe there are many areas where you can purchase cheap supplies. However, there are also items that you should spend a few extra dollars on. Spend extra dollars on:
1. Excellent work gloves
2. Excellent work shoes
3. A few tasty meals – to give you something to look forward to in between average meals.


How directNIC.com is preparing for the Next Big One:

Procedural changes:
1. All employees must be able to work remotely.
2. Paperless office. Less than a month before Katrina hit we had just purchased a super scanner to allow us to scan in documents on a large scale.
3. Better able to provide coverage of a catastrophe
4. More methods to contact displaced employees.
5. Ability to have some power from the generator on the 10th and 11th floors instead of just powering the 10th floor.

Transportation issues:
1. A helicopter would be nice; but, that is not going to happen any time soon.
2. A bigger truck – our mechanic evacuated WITH the large truck and left his small truck behind. The big truck could move 500 gallons of diesel at one time. The small truck could only handle 165 gallons at a time.
3. Better identification – when the City of New Orleans was locked down we ran into problems bringing employees and vendors into the city.

Quality of life and leisure issues:
1. Will have better sleeping options for the future. Don’t skimp when buying an air mattress. Make sure to have a pump.
2. For prior hurricanes I occasionally splurged buying more expensive food. I should have done this some prior to Hurricane Katrina.
3. Air freshener
4. 25 boxes of baking soda.

Better tools and supplies purchased:
1. Nikon d2x, fish eye lens, wide angle lens, Canon EOS 1Ds Mark II SLR, Canon EOS 1D Mark II N, with lenses and additional spare batteries.
2. Dehumidifiers
3. 20 additional 55 gallon diesel barrels full
4. 20 rolls of duck tape
5. Better video feed equipment

Better tools and supplies to purchase:
6. Lock cutting tool
7. Blow torch
8. Iodine or other water purification tablets.
9. Wheelbarrow
10. Rope

Personal supplies: List of additional items that I would bring in the future:
1. A belt
2. A full tank of gas (vehicle was just over ½ full)

What we will do differently for our buildings outside of 650 Poydras:
1. Plywood with interesting advertisement placed on them for boarding up windows.
2. Close storm shutters.
3. Make sure flashings are secure on roofs.
4. Make sure roof is in good shape.
5. If repairing a roof with a blue tarp make sure to nail the blue tarp over roof using strips of wood on the tarp.

Communication changes in the future:
1. Know how to use our Canon video camera, which we never took out of the box.
2. Keep track of the flash for the digital camera. The lens was misplaced after power went out on all floors that were not on generator power. We found the flash two weeks after Katrina hit when power was back on in the building.
3. Know how to use all of the security cameras that we had at the office.
4. Have a content management system in place for publishing information.
5. Archive video. We streamed everything and did not archive anything.
6. Have more small laptops.
7. Have photo software that would allow the pictures to be easily uploaded and allow individuals to comment on the pictures easily.

How to amass supplies for a catastrophe on a budget:
1. Buy in bulk from sales.
2. Buy stuff that you will use anyway and just keep a supply on hand that you will use up before it perishes.
3. Make sure to use perishable supplies and restock.