General

Drool Boxes on the Show Floor

Now playing in Phoenix, Arizona: A trade show on whose show floor is a greater amount of bandwidth than is available to many of the nations of the world.

by Alex Goldman ISP-Planet Associate Editor [November 17, 2003]

The International Conference for High Performance Computing and Communications opened on Saturday in Phoenix, Ariz. this year. It runs through the end of the week. The conference brings together the latest in high-performance computing and bandwidth for those who might use it: national laboratories, the Department of Defense (DoD), Department of Energy (DoE), several secret organizations, research universities, cities, and even some nations.

The conference has been run once each year since 1988. It boasts about 8,000 attendees. "These are people who are exploring the development environment of multi-million dollar supercomputer systems from companies including Cray, H-P, and IBM," explains Jim Rogers of Huntsville, Ala.-based Torch Technologies, who is serving as SC03 (SCinet 2003) Network Chair.

SCinet is the collection of very high performance wired and wireless networks that are built to support the high bandwidth requirements of the exhibitors. For 2003, Rogers is leading the team building these interconnected networks. SCinet is bringing in three OC-192c WAN pipes, and delivering this bandwidth to exhibitors and special competition sites, generally as GigE or 10GigE pipes. The show floor boasts multiple OC-192c SONET rings with self-healing properties. The 802.11 wireless network provides 802.11a and 802.11b, and is designed to handle 1,500 simultaneous users.

"It takes a whole year to plan," says Rogers. "Then we build it in a week, run it for a week, and tear it down in a few days." The people who build it are volunteers from Federal agencies, research institutions, leading universities, and participating manufacturers.

None of this could happen without the manufacturers and carriers. At every stage in the network, donated equipment and bandwidth are key, and the size of the donations is staggering. Rogers estimates that the hardware alone is worth more than $16 million.

Vendors participate because the attendees are important people. Rogers says, "this is a decision making conference. No dancing girls here."

Riding the light to the show floor
It starts at Level 3's Phoenix gateway. The company brings in 3 OC-192 pipes (roughly 10 Gbps each) from Equinix peering exchanges in Los Angeles. There, Qwest provides connections to Internet2, ESnet (the Energy and Science Network), and Teragrid. The Corporation for Education Network Initiatives in California (CENIC) provides a connection to the Stanford Linear Accelerator Center (SLAC).

Qwest in Phoenix works with MCI to provide a connection to the DoD's High Performance Computing Modernization Program.

The equipment manufacturers have sent the latest in computing and networking equipment for the buildout in the Phoenix Convention Center. It's big iron and it's hot stuff.

A Gigamux from Sorrento Networks sits in Level 3's gateway in Phoenix and at the convention center. The two provide multiple point-to-point fiber connection using WDM. Rogers says, "Sorrento recently purchased Luxn, which made a small WDM device that's easy to set up and deploy. We used it to light Qwest fiber from the Level 3 POP to the convention center."

The wide area network connections are terminated in the core using Juniper T640 and T320 routers. The core routers feed 10 Gigabit Ethernet connections to other high end equipment via multiplexed 10 Gig waves. The 10 Gig E connections are also being tapped for analysis by an intrusion detection system.

Each of the two fiber rings uses equipment from a different manufacturer. The first ring uses Ciena ONLINE Metro boxes. Ciena just obtained approval for USDA funding to help rural LECs buy them, and may be hoping to impress municipalities of all sizes with the equipment.

The second ring uses Cisco ONS 15454 boxes. Cisco says its box is the most popular, with 30,000 systems deployed worldwide to over 600 customers.

The networks use several different boxes to drop off bandwidth at different places on the show floor. There are three Cisco Catalyst 6509 boxes, each boasting four 10GigE interfaces, forty-eight 1GigE interfaces, twenty-four Fast Ethernet interfaces, and twenty-four 10/100 Ethernet interfaces. There are two E600 boxes from Force10 Networks, each with a different configuration. There is also one BigIron 15000 from Foundry networks with two 10GigE interfaces, forty-eight 1GigE interfaces, twenty-four Fast Ethernet interfaces, and forty-eight 10/100 Ethernet interfaces.

At the core of the network (which is powered by Marconi) are the niftiest (i.e., biggest) boxes of them all. The core is a mesh of four massive routers.

The Juniper T640 offers seven 10GigE interfaces, four OC-192 POS interfaces, ten 1GigE interfaces, plus L2TP tunneling. The Juniper T320, a slightly smaller box, offers seven 10GigE interfaces, three OC-192 POS interfaces, ten 1GigE interfaces, an OC-12 interface, plus L2TP tunneling. The OC-12 passes status data to the Network Diagnostic Tool (NDT) for the mesh. Juniper says its T-series routers are designed to provide "any service, any port, any time at multi-terabit rates with predictable service performance."

The E1200 from Force10 Networks holds sixteen 10GigE interfaces and four OC-192 POS interfaces. The company designed it for the five environments that provide the trade show's attendees: data centers and server farms, networks supporting cluster or grid computing, campus backbones, metro Ethernet networks, and Internet eXchanges (IXs).

Rogers is particularly excited about the "brand spankin' new" 8812 router from Procket Networks (he's seen the others before and is drooling over them too, but he's never seen this particular box before, and neither has the rest of the world). The company says simply that it is "the highest performing router available today." On the show floor it serves up twenty-four 10GigE interfaces, eight OC-192 POS interfaces, and twenty 1GigE interfaces. That's an aggregate of about 340 Gbps of bandwidth.

The aggregate bandwidth available to exhibitors is an astronomical number, well over a Terabit. The actual deployed bandwidth will certainly exceed 150Gbps. Rogers says, "if you want eight 10GigE pipes, we'll give you eight. Maybe you want to tie three or four supercomputers in labs scattered around the world and pump the results into the exhibit."

The 802.11 wireless network supports 1,500 simultaneous users. "Last year, we supported 1,000," explains Rogers. Maybe next year, they'll support 2,250 simultaneous users. The network design calls for sixty-seven access points powered by Cisco Catalyst 2950 routers and the Cisco Aironet 1200 Series Access Point in an architecture similar to that of Purdue University.

The supercomputer that goes where you go
The idea is to allow scientists at national laboratories and any other supercomputer user to log into their network as if they were still in their lab. For uses involving public networks (such as the Web itself) speeds may not be the same as those delivered over Internet2 or other research-only networks.

This "more than you could need" architecture is also designed to support a special Qwest project. Rogers explains, "we initiated something several years ago with Qwest called the high performance bandwidth challenge. There's a real prize, so for the engineers, there's pride and real money on the line. It has to be a real application, not a trumped up thing that burns bandwidth."

Last year, the challenge handed out several awards. The most bandwidth used by one application was 16.8 Gbps. This year the SCinet crew expects participants will squeeze every drop out of the 30 Gbps pipe.

The conference is also supporting IPv6, a task supervised by the SC03 Vice Chair, Chuck Fisher of the Oak Ridge National Laboratory (he will be Network Chair for SC04). The network supports IPv4 and IPv6 simultaneously (the U.S. is behind the rest of the world in deployment of IPv6, even at the government level).

Fisher says that the SCinet architecture enables an unusual application. "Participants have used access grid technology to set up audio-visual (AV) nodes all over the place (Europe, Australia, Asia) so people can participate in conference sessions over network multicast. There are over 60 nodes participating around the world in this conference." If you cannot make it to Phoenix, just look for the access node nearest you.

In addition to the manufacturers mentioned here, the conference is sponsored by ACM and the IEEE. The Federal Government and carriers bring in the manufacturers. "The carriers—AT&T, MCI, Level 3, and Qwest—pay very close attention to what's going on here, as do the DoD labs and DoE labs and other labs," notes Rogers.

"We've always tried to deploy an architecture that someone might try to implement on a campus or metro scale. People can take information from that design and apply it to their organizations."

So that's what SCinet's about. It's how to light up your city—in a week.

—End