Data centers, shipping containers, and the future of access networks

An announcement today by three companies—semiconductor vendor Fulcrum Microsystems, 10G Ethernet switch vendor Arista Networks, and network test vendor Ixia—serves to illustrate the ferment bubbling away inside the very concrete-and-steel data centers that are the reality behind the cloud computing nebulosity. The three vendors teamed up to make some performance measurements on an emulated 10Gbit Ethernet data-center backbone.

The driving force behind the work, according to Fulcrum director of product marketing Gary Lee, is an architectural shift in the networks within the giant data centers. Given the scale of deployment forecast for coming years, system operators can’t afford to wire together individual servers, or even racks of blades. They need a forklift-assembled structure like a giant Lego toy that almost literally snaps together. And in fact, server vendors are offering pods—shipping containers pre-packed with servers, networks, power, and cooling, ready to drop into place in the proverbial empty warehouse, connect to the power and network cables, and turn on. This snap-together mentality puts just as much premium on simplicity as on computing power or energy efficiency.

That drive for simplicity, in turn, is leading to the new network architecture. The normal practice in data centers, according to Lee, has been to operate three parallel network planes: an Infiniband plane for control and clustering, a FibreChannel plane for the storage network, and a fast Ethernet plane for data movement. These three, Lee says, are converging into a single 10G Ethernet. This converged fabric will bear the responsibility for the latency-critical interprocessor communications, the guaranteed-lossless, guaranteed-bandwidth storage traffic, and the massive background data flows.

And this, in turn, means the converged network is no longer a simple flat Ethernet. In order to meet the divergent needs that formerly lived on three different kinds of networks, the converged network requires the same sorts of packet classification, virtual connection, and quality-of-service capabilities that are now appearing in the carrier access networks.

Not that 10GE is the only contender for the converged network. For one, interchip interconnect standard Hypertransport is working to offer an alternative based on its consistent programming model across systems, very low latency, and many new capabilities now being added to the standard. "We expect the industry to become increasingly focused on the data-center problem," said Hypertransport Consortium general manager Mario Cavalli during last summer’s Hot Chips conference. "We want the same programming model across the elements in the hierarchy," added Jose Duato, professor of computer science and engineering at the Polytechnic University of Valencia. "This means a shared coherent memory rather than a message-passing architecture."

But Hypertransport has a lot to prove to scale up from interconnecting a few CPU chips to serving a data center, even with planned massive increases in speed and features. In that regard, 10GE, now being widely deployed in fiber networks, has an inside track. In fact, Lee says, some network equipment vendors are providing their access, aggregator, and core switches for data centers.

Such an approach isn’t optimized for data centers, Lee maintains, but the big issue is latency. General networking applications are either latency-tolerant or, in the case of problem connections like Voice over IP, have latency sensitivities in the hundreds of microseconds. In inter-CPU communications, tens of microseconds accumulate into user-visible delays. So, Lee argues, the data center’s converged network needs to rest on a new kind of switching fabric.

This argument is convenient for Fulcrum, of course, because a new kind of fabric is exactly what they offer: non-blocking, wirespeed switch chips that use a central shared memory rather than the conventional approach that surrounds a central fast crossbar with deep input and output buffers. Without having to store packets at ingress and again—perhaps multiple times—at egress, Fulcrum can claim very low and essentially fixed latency.

This latter point—latency jitter—is an issue because of one interesting footnote to the whole cloud-computing phenomenon. One of the richest and most specialized buyers of massive data centers is the investment banking community—the very folks whose huge leverage and speculative trading brought you the Great Recession. These kindly folks use the data centers to run trading algorithms that not only figure out what and when to trade, but actually execute the trades without human involvement. In some cases, microseconds make a difference in financial yield on these flash-trading transactions. So the investment bankers and their trading adversaries care a great deal about network latency within their data centers. And since the algorithms may be adjusted to compensate for network latencies, they also care about the variations in latency.

Anyway, the point of the announcement today is that Arista has built a high-end 10GE switch family, the 7100 series, using Fulcrum chips. Ixia has used their latest test system and software to put Arista’s high-end box, the 7148 SX, in an emulated data center environment and to measure aggregate network performance. One particularly valuable function of the Ixia IxNetwork system in this case, according to Ixia’s product manager Henry He, is its ability to create and test both unicast and multicast packet flows simultaneously. This not only duplicates a particular issue for converged data center networks, but it should be a great test for blocked routes, port contention issues, and egress latency jitter.

Unsurprisingly, the results Ixia reported match Fulcrum’s expectations. Maximum throughput was independent of packet size for both unicast and mixed unicast/multicast loads. And with a "real world" financial data stream, latency ranged only from 600ns to 1500ns.

As the market for massive data centers grows, expect the unique requirements of converged data-center Ethernet to have a growing influence on the broader networking community. At the end of this path may be a uniform network architecture, at least at the virtual level, between the insides of pods and the global Internet.