A Data Center Migration

Here's the conundrum: you have to be prepared for the unexpected.

by Alex Goldman ISP-Planet Managing Editor [March 27, 2008]

"Circumstances expedited our timeline," says Ryan Ellege, COO of Overland Park, Kan.-based Aplus.Net. "Fortunately, we had done months of planning in advance. We had explained to people what they had to do, in conference calls held every other day over a period of weeks. The only people not involved in the planning were temp agency workers in Phoenix who we hired to do things like plug in power cords and patch cables."

That's no ordinary temp agency? "It's tech oriented, and it's expensive, but we found good success with the firm we've utilized."

In addition, the company flew staff into Phoenix from its other data center in San Diego.

The company was moving its servers from an old data center in Phoenix to a new one. The new data center had been built exactly according to the demanding specifications of colo.com, which was acquired by IPOWERWEB. "We took over the lease from IPOWERWEB."

The new facility has multiple fiber providers (the old facility had one) and even a dual power grid. It has strong physical security and the entire building has a regular security patrol. The new facility has redundant everything: HVAC, UPS, PDU, etc.

The old facility had been a data center for a bank. It was secure, but the equipment in it was old, some dating back to the 1980s. It had been maintained well, but was not ready for the heavy use of a webhost like Aplus.net.

So everyone knew what to do? "All equipment for one customer, even if not located adjacent, has to move at the same time. So we planned in advance which servers to grab. We had colored labels for server groups."

The servers moved first were those generating the most traffic.

There's been a change in plan
At least, that was the plan until the old facility ran out of bandwidth.

"We had no long term bandwidth commitments at the old facility because we knew we were vacating it. We had bandwidth issues there that were unforeseen, with tremendous packet loss. They would have had to put in a new circuit quickly."

So it was better to move to the new facility, which is superior in every way, but do the move faster than originally planned.

"We already had bandwidth in the new facility. We were getting the racks set up. Four days after we finished the actual move, we had planned to do a test move."

The team had experience with moves. Elledge and members of his team had worked for Communitech.net and the company that acquired it, Interland. "We migrated two customer bases that Interland bought. That was an over the wire migration."

Moving the physical servers is different. "It goes quicker, but it's tougher. With over the wire migrations, we have that process down to virtually no downtime. You cannot avoid downtime on a physical migration. So you hope to have the least intrusive down times possible."

You might assume that the best time would be between midnight and 4 AM, and the team worked during those hours, but Aplus.Net has some international customers who needed to be moved at other times.

Luck is earned by hard work
The move was a success.

"We were moving 3.5 miles. We utilize carts on moves, and each cart holds 15 servers with adequate padding. We had completed weeks of testing on dummy servers. You can plan, but you always run into unforeseen things. We lost some power supplies in the move, but we had inventory on hand at the new facility. Generally, we found that vibration caused fan damage during the move."

"On the software side, the biggest issue we ran into is that some servers hadn't been rebooted for a long time (we're not a managed hosting provider) and some updates take effect only on reboot, even in Linux. We had savvy techs troubleshooting those issues."

"It took thee nights to move the first batches of servers. We had electricians and cablers working around the clock and over the weekend to move 300 to 500 more servers."

1,500 servers in four days is not shabby at all.

As you can see, even if the plan changes, it pays to plan ahead.

—End