10GBase-T: Is it really coming this time?
If silicon designers employ advanced process nodes to lower power, and if they can overcome the signal-integrity issues, data-center managers could bypass both power and cost issues of NICs by purchasing motherboards with LAN-on-motherboard chips on them, providing 10Base-T connectivity directly off the motherboard.
By Ron Wilson, Executive Editor -- EDN, August 24, 2009
The arguments for 10GBase-T as primary interconnect in data centers seem unassailable. Offering 10 Gbits/s raw bit rate, and Ethernet protocol at up to 100 m over Cat6A or Cat7 copper cable, 10GBase-T proposes lower cost than optical interconnect, greater wiring flexibility than is possible with the 10m range of direct-attach copper, and the architectural elegance of having one tcp/ip interconnect scheme from the blade all the way up to the WAN leaving the building complex. This last point seems to grow in importance as virtualization runs rampant through data centers.
From these points it would seem an obvious move for blade vendors to start slapping 10GBase-T silicon onto their motherboards, and for data centers to migrate quickly to the faster Ethernet standard. And that is just what industry analysts expected—two and a half years ago. Yet this year the total market for 10GBase-T NICs appears to be only about $150 million, according to industry research. That's about the price of one mid-range superyacht, just to keep things in perspective.
If the deployment seems slow compared to the value proposition, it's not for want of energy on the parts of silicon vendors. Back in early 2007 Teranetics—which still claims to be leading the charge—and Solarflare Communications jumped on the bandwagon with monolithic PHY chips. Since then Teranetics, Broadcom, and Solarflare, to name a few, have pushed the level of integration for PHY chips from one port to two, while wrestling down power and cost.
But still the market remains small, and the forecast still looks like a hockey stick. The explanation may be the very issues the chip vendors are fighting: power and cost. At 5 to 10W for the PHY and another perhaps 5 to 10W for the controller, the first couple of generations of silicon were far from being an insignificant load on a data center's power or cooling resources. And with cost for a single-port NIC starting out a couple of years ago at $2,000, drifting down into the $1,000 range for today's server-class boards, one can understand a certain reticence on the part of data center managers.
That could be where LoM (LAN-on-motherboard) chips come to the rescue. If silicon designers employ advanced process nodes to lower power, and if they can overcome the signal-integrity issues of putting a PHY, a MAC, and a PCIe interface all on one die, data-center managers could bypass both power and cost issues of NICs by purchasing motherboards with LoM chips on them, providing 10Base-T connectivity directly off the motherboard.
That's a great scenario for new build-outs. For existing centers, LoM chips would need backward compatibility—the ability to negotiate down to a slower standard like 1000Base-T. Even then, a mixed cable environment that combined legacy Cat5/6 cable with 6A and 7 wiring, while functional with backward-negotiating LoM chips, would be an architect's and a maintenance manager's nightmare. And the full benefit of 10GBase-T wouldn't accrue until there were enough upgraded server boards to create some useful-sized fast partition in the data center. So LoM chips aren't necessarily going to ignite the market either. But they can't hurt.
This isn't lost on the vendors. Last November, systems vendor Mellanox introduced a single-chip 10GBase-T LoM by integrating their own controller with the Teranetics PHY. And this morning, Solarflare followed, with a family of one- and two-port chips of its own, based on the Solarflare PHY and the controller technology acquired through its merger with Level-5 Networks. One presumes that Broadcom, always masters of both integration and leading-edge process adoption, isn't far behind.
The Solarflare chips appear aimed at opening the market. On the power side, the company is quoting 8 W and 13 W for one- and two-port LoM chips, respectively. That compares to rather vague claims of about 10 W per port on the Mellanox Web site for their LoM offering. That puts the LoM chips in the range of half the power of recent NICs, and with careful integration the Solarflare parts can meet Energy Star guidelines of 8 W per port for a blade mezzanine card, according to Solarflare VP of product marketing Mike Smith. That power level is also coming into line with previously reported remarks by the company's CTO George Zimmerman, suggesting that the PHY needs to get below 5 W in order for data center managers to stop worrying about its power on the switch side.
Price may remain an issue for a while, until yield-learning can work its magic on these advanced-process and heavily mixed-signal chips. In principle a fabless semi company should be able to sell a PHY/MAC combination for less than the cost of an entire NIC. But if there are a lot of mature technology and good levels of integration on the NIC, that's not guaranteed at the beginning of the chip's product life. Suggestively, at this point neither Mellanox nor Solarflare is publishing prices.
A couple of interesting points remain, neither of which is well-addressed in the marketing literature. One is the degree to which hardware-based off-loads in the LoM chips can impact both network performance and network energy consumption in large data centers. Controller architects appear to be all over the place in their implementation of off-loads, and while obviously leaving more work for the server CPU allows you to reduce power in the LoM, it is not clear what impact these choices will have at the system level. Tossing a few more small but asynchronous tasks onto modern CPUs, with their highly complex active power management, may prevent the processor from using its best low-energy modes, and totally change the CPU's energy profile.
Another point is virtualization. Virtualization of network channels and of NICs may be transparent to applications, but it's not transparent to the hardware. Issues such as the availability and size of hardware translation tables in the controller, and the lengths of latencies in protocol processing may turn out to be very significant issues when a server is using a LoM as 64 virtual network interfaces instead of one physical interface. So even with power and price starting to come into line, there may yet be some very significant terrain in the competitive landscape. Further research, as we are always saying, will be required.
