Preparing for Disaster: The Katrina Story

Sigmund Solares and Donny Simonton told ISPCON attendees the story of how they survived Katrina while operating a data center.

by Alex Goldman ISP-Planet Managing Editor [May 23, 2006]

As hurricane Katrina approached (see Katrina timeline), none of those about to face the disaster knew what was coming.

Solares told the audience he forgot his belt that day. He didn't fill up his car's gas tank "because the gas lines were too long."

The blog entry for Saturday August 27th says, "Hmm. This could actually be a nasty storm."

Sigmund Solares, the CEO of Intercosmos Media Group (and the owner of the data center on the ninth floor of an office building in New Orleans); and Donny Simonton, senior vice president of software development, opened their ISPCON session by asking how many had faced a power outage of a few days. About one third had. None had faced an outage the length of Katrina: ten days.

The storm hits
"We woke up to noise," said Solares. "We lost power. We had a balcony on the ninth floor."

"Pieces of metal flew off the roof of the building across the street, from the 27th floor. When they hit our windows on the ninth and tenth floors, they had power. Windows were breaking very fast. Our windows were double layered, but that just meant that it took two pieces of metal to break the glass instead of one."

"One the windows were broken," Simonton added, "the blinds flapped up and down and broke out the rest of the glass."

"We got lucky," Solares said. "We only lost one of three panes of glass in the data center. We moved some book cases there in case the inner layer broke. It was all that was left of the remaining two panes."

As for the open window pane, Solares related, the company improvised. "We had 2,000 tee-shirts from a recent failed promotion, and we kind of built our own levee with them. We had mops, buckets, box fans—we were high tech! While we were doing this, we didn't think it would work, but it did. We ended up with a couple of thousand dirty tee shirts, but the data center stayed high and dry."

Overall, Solares said, the building they were in lost 180 windows, and the last window to go was in Simonton's office, while he was in it. Simonton was not hurt.

The upstream wasn't doing too well, either. "We lost one of our four OC-3s during the storm, and lost two more when [BellSouth] ran out diesel."

The company had a deal with their largest consumer of bandwidth that, in an emergency, that customer would be cut off. Solares pulled the plug on them, temporarily.

The storm subsides
Immediately after the storm, the team decided to get the word out. "We had a video system with 18 cameras, but we didn't know how to use it," Solaris admitted. "We also had a high end camera, and didn't know how to use that, either."

"We found an Intel camera from 2001, their first color camera, and put that up. We put a note on the our registry site, directNIC, explaining why we could not answer the phone, linking to the video feed."

One supply that wasn't used: beer. "I brought 2 twelve packs of beer and three half gallons of liquor. During the ten days, I drank two beers," Simonton said.

In order to get on a list of companies to which the National Guard had to bring diesel, they were told to bring a router to "the colonel." Simonton says he walked through a street flooded waist deep carrying the router on his head, with Michael Bennett, ex-Special Forces and a friend of Solares, who was carrying a gun.

They found "the colonel" and after he plugged in the router, they learned it was powering the only working phone in the area, a video phone.

Meanwhile, the building was dangerous to be in, Solares said. "Anyone could have walked off the building, because all the glass was gone. You really had to look where you were going."

The fire department was pumping flood water, or driving back and forth between a fire and the river, because the plumbing was without power.

"The cavalry arrived September third," said Solares, and that's when the looting stopped.

The first priority of the Guard was to restore order. That done, they started delivering water. "Trucks would drive around the corner and throw bottled water out the window," Simonton said.

Diesel
On September third, the Guard delivered diesel to the company. Getting the diesel into the generator, however, was not easy.

"You want gloves, you want shoes, not flip flops, and you need belts when you're opening barrels," Solares said. "I had no belt, so I used an Ethernet cable."

Then BellSouth needed diesel. "We had to provide our internet provider with diesel," said Simonton.

"When they came back online," said Solares, "we got two more OC-3s back."

Lack of sleep
During the storm, Solares said, he walked downstairs to turn off the building's generator, which was about to run out of fuel. He fell asleep on the fourth floor and woke to a building without power.

Similarly, he made an error while pumping fuel. He was pumping fuel out of the emergency storage tank into one barrel with one fuel pump while Bennett was pumping fuel out of another barrel with the other fuel pump. "I'd told everyone not to plug them into the same circuit," Solares said, but that's exactly what he did. Then he wondered why they weren't working. That's what exhaustion will do to you.

Conclusion
The technology held up well during Katrina. "One server failed, and one power strip failed," said Simonton. "We had to service the building generator after it ran out of fuel."

Simonton's last piece of advice to those who might be flooded was this: "A rope ladder could be useful if you need to get out of a window into a boat."

Luckily for Solares and Simonton, their street was not flooded.

— End