For several years, next-generation Ethernet capable of 10-Gbps speeds has been on the brink of entering the mainstream. Some areas of the network have for some time been using optical technology, and demand is increasing for 10-GbE (gigabit-Ethernet) data rates as traffic increases. But vastly more complex technology than in previous generations of Ethernet is necessary to run 10-GbE over 100m lengths of copper. That requirement has resulted in expensive MAC (media-access-controller) and switch silicon and costly and inconvenient PHY (physical)-layer-interface chips, limiting the speedy LAN to high-performance applications covering shorter distances within data centers.

This picture is beginning to change, but widespread deployment of 10-GbE is not likely for at least two years (Figure 1). In June 2006, the IEEE finalized the 10GBaseT P802.3an spec for operation over 100m copper. Switch and controller chips that meet the spec are now available, but, along with PHY chips, they are expensive and consume too much power. These problems have been especially severe with 10GBaseT PHY-layer chips.

Silicon power consumption must be less than 5W/port for use in commercially available equipment in chips such as switches and NICs (network-interface controllers). “That’s not the case with most of these chips now,” says Alan Weckel, senior analyst for Dell’Oro Group. “In contrast, optical pluggable [transceiver] modules consume less than 1W.” Most 10GBaseT PHY chips currently consume 8 to 10W per port.

Although per-port prices have for some time been falling, lower component prices alone don’t necessarily create demand. “It will take time to ramp up volumes in the next couple of years,” says Jag Bolaria, senior analyst for The Linley Group. Meanwhile, the cost of optical fiber is decreasing, “and the early stages of 10GBaseT could be the next kicker in the road in getting volumes up.”

Analysts expect a fairly rapid ramp for 10-GbE technology over the next four to five years. Drivers include sheer bandwidth demands because of server-performance increases and other factors, such as virtualization (see sidebar “Virtualization and 10-GbE”). The demand for 10-GbE over copper, including 10GBaseT, comes from the fact that copper costs less and is easier to install than fiber, even though 10GBaseT requires unshielded Category 6A or shielded Category 7 cable to meet the spec’s maximum distance of 100m. Although fiber will still find use for longer runs, the shorter enterprise and data-center runs need less expensive, 10-Gbps links.

During the next five years, 10-GbE should replace 1-GbE in two major applications that will drive significant port growth, according to Dell’Oro Group. In wiring-closet-switch uplinks, most 10-GbE fixed ports will be uplinks on 24- and 48-port 1-GbE switches, and the remaining 10-GbE ports will be 10-GbE-only switches, those finding use at the top of the stack for aggregation. The other major application area is in direct server connections.

Currently, copper finds use mainly between switches and PCs and between switches and servers. The uplinks from wiring-closet switches to data centers that are currently optical links will remain optical and will have significant potential for volume, says Weckel. Because enterprises are undergoing a major wiring-closet upgrade, the market for switching is increasing, as well. In switch-to-switch connections, the medium for 1-GbE is now optical and will remain optical at 10-GbE. In direct-server connections, the medium is copper, and it will remain copper at the higher speed.

As the number of 1-GbE ports increases, aggregation of those ports becomes a major reason to begin using 10-GbE technology. Without it, the uplink bandwidth is less than the bandwidth of the downstream ports, and blocking occurs. Tier 1 OEMs are selling 1-GbE gear for around $100/port, says Kamal Dalmia, vice president of marketing for Teranetics. For 10GBaseT switches at their introduction, the price will be approximately $500/port, a per-gigabit cost of roughly half that of 1-GbE equipment. Another main driver is the need to connect high-performance computing-blade servers in data centers with bandwidth commensurate with their processing speeds. Those servers include eight or 16 processors, each of which is increasingly likely to contain four or even eight cores, ramping up power requirements and speed.

Aside from the combined power-consumption and technology issues in implementing the technology, another major issue could hinder the rate of 10-GbE deployments, says The Linley Group’s Bolaria. If the cost of implementing one 10-GbE port is too high, it may make more sense to aggregate links by combining two to four 1-GbE ports; more than that number would be expensive and cumbersome.

The first 10-GbE standard, IEEE 802.3ae, which originated in 2002, specified several PHY interfaces for optical-transmission media. In 2004, the IEEE 802.3ak-2004 short-reach amendment allowed 10-GbE over 15m coaxial cabling using 10GbaseCX4 PHY-layer chips. Compared with 1000BaseT, 10GBaseT data rates require highly complex digital-signal processing in silicon to deal with echo cancellation and crosstalk cancellation, as well as more sophisticated analog circuits. All of this processing to clean up the received signal must take place on links that are 10 times faster than the 1000BaseT spec. The 1000BaseT spec left a wide margin for compensation, but 10GBaseT does not, says Brad Booth, who acted as chairman of the IEEE 10GBaseT task force and is chairman of the Ethernet Alliance.

The 10GBaseT spec includes a subset for a short-reach PHY interface over a maximum 30m of Category 6 four-wire UTP (unshielded-twisted-pair) cable as a means of going to a lower-power operation mode. When the spec’s developers wrote the draft, the concern was that most devices for a 100m range would consume 10 to 12W of power, and those levels of power consumption limit 10-GbE deployment in switch applications, says Booth. In a shorter-reach application with the 30m option, the PHY-layer device could drop to lower power, which helps eliminate a lot of noise along with many cancellation circuits. The spec also allows the use of Category 6 cable for distances as long as 55m, because 70% of data centers’ reach is 55m or less. However, operation over this distance doesn’t buy lower power consumption.

Additional PHY-silicon issues are the need for multiple ports to pare down the cost per port, and the inclusion of multispeed ports for backward compatibility with 1-Gbps and even 100-Mbps Ethernet. An alternative method of connecting 10-GbE over copper may be possible using a new form factor in hot-pluggable optical-transceiver modules. The SFP+ (small-form-factor-pluggable-plus) module has a smaller footprint than the previous SFP form factor and consumes less power, allowing greater module density on a line card and offering lower per-port costs. Manufacturers are also developing direct-attachment SFP+ copper-twinaxial cables for distances of approximately 10 to 15m, which are adequate for connections within a data center.

The process technology for most chips now implementing 10-GbE over copper is 130 or 90 nm. The next generation may go to 65 nm, which is the next-lowest-cost node at which volumes may increase, says The Linley Group’s Bolaria. Some in the industry believe that it will take a 45-nm process to reach the lower power consumption required to drive volumes and are trying to integrate some of the analog circuitry necessary for 10GBaseT at that process node. But the high amount of analog circuitry could be a problem in such smaller geometries, says the Ethernet Alliance’s Booth. “One alternative may be multichip modules with analog front ends running in one process technology and digital back ends,” he says.

Definitions of “high volume” can vary. Huge volumes are on the order of 10 million and 20 million ports, says Bolaria. In 2007, volumes for 10-GbE-switch ports with 10W PHY interfaces were approximately 640,000, and they contained 10W PHY-layer chips. “The next tier would be less than 5W, which will enable [manufacturers to ship] a few million ports,” he says. “But, to get to 20 million in port volume, PHY silicon will probably need to consume less than 2W.” As ports per chip increase, chip volumes will decline somewhat.

The power budget of a typical first-generation 10-GbE endpoint NIC is approximately 25W, but designers usually want to stay closer to 15W, says Blaine Kohl, vice president of Tehuti Networks. You can build a single-port-10-GbE NIC with a 7W PHY chip, but you can tweak the power allocation even more in some designs. Manufacturers will most likely build single- and dual-port adapters for endpoints using this year’s generation of 10GBaseT PHY silicon. However, switches need PHY chips closer to 3W. By 2010, 3 to 4W 10GBaseT PHY chips will probably be available. At that point, with 2 to 3W controllers, you may be able to integrate a controller chip and a PHY chip in one package. Switches could then adopt a 10GBaseT PHY chip, endpoints could adopt the 10GBaseT package, and manufacturers could be shipping 10GBaseT equipment in volume.

After offering samples of 10-GbE-controller and 10GBaseT PHY chips for a year and a half, Solarflare Communications has just introduced second-generation, 65-nm parts. The SFT9000 10GBaseT PHY chip consumes less than 6W and features multispeed autonegotiation at speeds as low as 100 Mbps. The SFC4500 10-GbE controller chip consumes 2.2W and features virtualization acceleration. Solarflare expects next year to make available a single-chip LAN-on-motherboard device that will integrate 10GBaseT-PHY and 10-GbE-controller silicon, according to Bruce Tolley, vice president of marketing.

Teranetics’ first-generation all-CMOS TN1010 10GBaseT multirate PHY chip supports 1-GbE and 100-Mbps Ethernet. The company’s next-generation 65-nm chip will be available this year, with 30 to 40% less power consumption than the current total of about 10W, says CEO Matt Rhodes. Fulcrum Microsystems’ single-chip, 24-port FM4000 10-GbE IP (Internet Protocol) Version 4/6 switch/router chips target use in data-center-switching platforms in high-performance-computing, server-, and storage-host interconnection, and data-center-aggregation applications. The Layers 2/3/4 chips have full line-rate performance on all ports with a total throughput of 360 million packets/sec. Their 300-nsec latency provides a highly responsive network fabric that exceeds the performance of specialty fabrics, such as InfiniBand and Fibre Channel, and suits high-performance-clustered-computing applications, says Bob Nunn, Fulcrum Microsystems’ president and CEO.

Broadcom makes single-chip, 10-GbE switch and controller silicon and optical transceivers. The 65-nm process technology for these chips requires a significant effort, especially for the switches, because they are large, highly integrated chips with both analog and digital components, says Eric Hayes, director of marketing for the network-switching line of business. The 65-nm BCM56820 switch chip’s power consumption is 1W per 10-GbE port—a drastic reduction from the previous 130-nm switch chip’s 11W per port (Figure 2). “We’re seeing a clear requirement to go to 10-GbE with all of the features and capabilities that were available at 1-GbE, such as security, strong Layer 4 classification for QOS [quality of service], and Layer 3 routing,” he says.

Tehuti Networks based its new SFP+-adapter-reference designs, which include the single-port TN7587-S and dual-port TN7587-D NICs for optical 10-GbE, on the company’s single-chip TN3016 10-GbE single- or dual-port controllers. The reference designs include AMCC’s (Applied Micro Circuits Corp’s) 10-GbE SFP+ QT2025 PHY chip. When you populate the NIC with two SFP+ optical modules, the TN7587-D’s power dissipation is 15W. The single- and dual-port controllers dissipate 6 and 7W, respectively.

Many enterprises have just begun to consider 40- and 100-GbE for aggregating 10-Gbps links in data centers. “Even telephone companies are requesting 100-Gbit products,” says The Linley Group’s Bolaria. “There’s demand for it now in the core of the network.” Because of video-on-demand and Web-based applications with millions of simultaneous users, such as Facebook, Netflix, and YouTube, some industry observers predict that the bandwidth-scaling needs of carriers and ISPs (Internet-service providers) may bypass 40-GbE.

For the first time, however, no aggregation technology is in place for the upcoming lower speed of Ethernet, which could hinder the market, says Dell’Oro Group’s Weckel. For the lower speed to be successful, manufacturers must be shipping products using the higher speed, even if not in high volumes. But with 10-GbE to the server, no 40- or 100-GbE line cards exist to provide uplinks to data centers on the aggregation side. “The higher-speed uplink technology must be present for there to be truly widespread adoption,” he says.

The pressure will increase even more when servers begin to appear with 10-GbE interfaces on the motherboard, which may occur with the next generation of Intel server CPUs. Ultimately, 10-GbE may go all the way to the desktop. For many, Ethernet has become the fabric of choice in the network, and that situation is also driving 10-GbE deployment.