For 10G interconnects, the RJ45 once again will dominate

GUEST OPINION: 10GBASE-T will be adopted as the primary 10GE interconnect for data-center switches and servers for many of the same reasons that RJ45 and twisted-pair cabling already dominate at gigabit speeds.

By Bill Woodruff, Aquantia -- EDN, 5/5/2009

The IEEE cabling standard 802.3an-2006, also known as 10GBASE-T, governs 10 gigabit/second (10GE) connections over unshielded or shielded twisted-pair cables, using the venerable RJ45 connector, for distances up to 100m. 10GBASE-T will be adopted as the primary 10GE interconnect for data-center switches and servers for many of the same reasons that the standard RJ45 and twisted-pair cabling already dominate at gigabit speeds. As 10GBASE-T further matures, its traditional Ethernet attributes of low cost, ease of use, and flexibility become very compelling in the adoption decision. The newer benefits of network convergence, virtualization, and green engineering provide additional push in the direction of 10GBASE-T.

The RJ45 is currently the dominant Ethernet connector. However, to project what networking at the edge will look like in the near term, we need to examine requirements at the server, at the aggregation switch, and for the cabling interconnecting the two. Let's look at their roadmaps and the implications for 10GBASE-T adoption.

Intel and AMD chip sets with PCIe Gen1 initially constrained 10GE adoption because they provided less than 20Gb/s coming into the network interface card (NIC) and thus throttled both inbound and outbound bandwidth. Intel's Core i7 and AMD's Shanghai chip sets and new adaptor silicon with PCIe Gen2 now offer up to 40Gb/s of bandwidth per PCIe card to enable dual-port NICs. On the PHY side, 10GBASE-T suppliers now offer second-generation PHYs at less than 6W per port. Combined with advances at the MAC layer, dual-port 10GE NICs are now viable and very appealing in a high-performance server.

Servers already come with gigabit LAN on Motherboard (LOM) RJ45s as a baseline configuration embedded within the baseline acquisition cost. High-bandwidth data ports can be added by plugging in 10GE NIC cards, which can be 10GBASE-T, SFP+, or CX4. Additional ports to support storage are gained by adding another adapter, either an HBA for a Fibre Channel connection or a Converged Network Adapter (CNA) for storage over Ethernet.

Servers are now primed to enable convergence, because the new generation of chips, CNAs, and supporting operating systems has moved the converged network from concept to reality. The first stage of deployment of the CNA has the following attributes:

• 10GE performance. While individual applications may not all justify 10GE bandwidth, the convergence of the data path and storage increases the need for the higher bandwidth.
• Dual port. Path redundancy is a critical part of network design.

The converged network changes things in a number of ways. Supporting three networks is a mess, requiring three times the cables and three times the number of aggregation switches. With its need for fewer parts, network convergence also supports the trend toward reduced parts inventories.

When the LOM supports 10GBASE-T, the baseline configuration of the server will have the bandwidth to support the converged network, removing the requirement for the add-in adapter card. For this to happen, 10GBASE-T needs to achieve its next breakthrough in power dissipation.

Second-generation 10GBASE-T parts available today dissipate under 6W. While this power level works on NICs and switches, it is hard for a high-volume, mainstream server to handle this level of power for the PHY, given thermal limitations on the motherboard. Moore's Law suggests that when vendors are able to leverage the next process node, this power will drop significantly. In fact, upcoming third-generation 10GBASE-T PHYs will provide the power and density envelope that enables high-volume deployment of 10GBASE-T LOM.

Conventional wisdom is that 10GBASE-T is at a power disadvantage. How can it be called "green"? It has much to do with how power management has evolved. Power management permits the power consumed in a server to decrease as a function of the workload. However, as the load approaches zero, power for the server only decreases to about 65% of full load power.¹ The ultimate in low-power operation is for an underutilized machine to be put into a sleep state. Techniques developed in the '90s permit machines that idle overnight be put into a sleep state, and woken up via a network command. In such a sleep state, the processor, memory, chip set, and disk drives can be powered off. Wake on LAN (WoL) permits adapters to monitor the network and "wake" the server based on a network command.

Virtualization creates more power-saving opportunities. It increases data-center efficiency as multiple applications run on a single machine. Similarly, virtualization can move applications off of lightly loaded servers, permitting those servers to enter a sleep state. This capability has been featured by a number of entities, including VMWare, Microsoft, Cisco, and The Green Grid, a global consortium driving energy efficiency in data centers and business computing ecosystems. Green Grid promotes the concept of dynamic consolidation, encouraging IT managers to reap large incremental savings from transitioning a lightly loaded server into a sleep state.

The green-enabling WoL feature has traditionally been implemented on twisted-pair links. When an idle server using a 1000BASE-T link is ready to enter a sleep state, the link first auto-negotiates down to 100BASE-T to minimize power. Once that lower-speed link is established, the adapter monitors the link for the "magic packet." The PCI-SIG (Peripheral Component Interconnect Special Interest Group) has defined an auxiliary power supply, which provides just over 1W for this activity. When the magic packet is received, the server wakes up and the link returns to its highest speed.

Moving WoL to 10GBASE-T changes the MAC-PHY landscape slightly. No longer a simple MAC, adapters today strive to enable a converged network. The 10GE MAC itself consumes from just under 5W to well over 10W. It is possible for a modern 10GBASE-T PHY to monitor MAC-level functions, including watching for the magic packet. This permits the PHY to work within the V_aux power limit, and for the balance of the adapter card as well as the server to enter its sleep state. The benefit of entering a sleep state can be significant. The power of a server at idle will still be 50 to 80% of the maximum power. The power in a sleep state can save most of that 50 to 80%.

Interconnect continues to evolve as we transition from GE to 10GE. What will emerge as the dominant interconnect at 10GE? Twisted pair and the RJ45 dominate today at GE. Although SFP+ (small form factor) has initial inroads at 10GE, twisted pair and the RJ45 are asserting themselves and are likely to dominate again at 10GE because of the following factors, both historical and current:

10GE began as an interswitch link, an optical connection to the core. These optical links will continue to be an important part of the interconnect environment both for short reach within the building and for long reach, including for transport. As 10GE matured, the industry created the SFP+, which reduced the size and cost of 10GE optical interconnect.

As a new class of hardware called the Top of Rack switch was developed, the SFP+ presented itself as an attractive interconnect candidate. With downlinks to servers typically just a few meters in length, we saw the development of the direct-attach copper alternative. This two-pair twinax cable is shipped with pre-terminated SFP+ connections, supporting lengths up to 10m. Note: This technology is a point-to-point connection, not compatible with structured cabling or patch panels.

Flexibility and ambiguity are the two sides of the SFP+ coin: Flexibility, in that the empty SFP+ cage can accommodate any connection, from direct-attach copper to optics (short-range, long-range, long-reach multimode, and extended range), and ambiguity, in that just because an SFP+ module fits, there are so many choices and configurations that the correct choice may not have been made.

Network topologies are another important consideration. The Top of Rack box has downlinks to the server, which may be short, may be 10GE, or may be GE. If the downlink is SFP+ and is within the distance limitation of the direct-attach copper, a twinax connection can be made. Otherwise, for longer links, an optical module must be used, over either multimode or single-mode fiber. If the downlink is an RJ45, just plug it in and it works.

On the other hand, End of Row modular switches employ structured cabling with a patch panel. If SFP+ is selected, the option of using direct-attach copper is eliminated, requiring optical options. If the downlink is RJ45, just plug it in and it works.

Cost and power advantages are frequently cited by advocates of the SFP+; however, competitiveness must be measured as technology evolves. SFP+ is primarily dependant on optics, metal bending, and labor. 10GBASE-T is primarily dependant on Moore's Law, and thus over time shows increasing price/performance benefits.

In conclusion, the perennial "survival of the fittest" contest in the interconnect domain seems once again to be tilting in favor of the RJ45 and twisted-pair cabling for 10GBASE-T high-performance interconnect applications. And as 10GBASE-T PHYs enjoy the benefit of continuing process shrinks, the industry will see the optimal intersection of power, density, and cost to serve the most demanding applications in the data center and high-performance computing infrastructure.