High-speed signals jump over PCB traces
Every time serial data rates increase, they expose problems masked at slower speeds. Many of those problems occur because of decreased signal integrity from losses that occur in PCB traces, vias, and connectors. There are many possible solutions to signal-integrity problems, but each one comes with a downside. One such response to the problem is to eliminate PCB traces using cable assemblies.
Here's the problem. Boards used in servers and switches are often quite large and they typically have "out of the box" I/O copper or fiber) on one end and a backplane connection on the other. To minimize the overall distance that a signal travels, PCB designers will place the switching IC (FPGAs or ASICs) in roughly the center of the board. Sure, you can minimize signal losses between the outside I/O and the switch devices by placing them close together, but that only worsens signal integrity from the ASIC or FPGA to the backplane.
With signal fundamental frequencies currently at 14 GHz, losses caused by skin effect, weave effects, surface roughness, vias, and connectors are often too much for reliable communications. Even though receivers can detect incoming signals as low as 38 dB below the signal amplitude at the transmitter, there can frequently still be too much signal degradation.
A 28 Gbit/s non-return-to-zero signal has a clock rate of 14 GHz. To help with signal losses at 56 Gbits/s, many engineers are moving to 4-level pulse-amplitude modulation (PAM4), which doubles the data rate for a given signaling rate versus NRZ. Thus, a 14-GHz clock can deliver 56 Gbits/s using PAM4, but there's a problem. The signal amplitude (eye height) of a PAM4 signal is one-third that of an NRZ signal (Figure 1). That creates a tradeoff between signal loss and sensitivity to noise.
Figure 1. The eye opening of a PAM4 signal (right) is about one-third the height of an NRZ signal, which makes PAM4 signal more prone to errors from noise.
Scott McMorrow, CTO of Samtec's Signal Integrity Group, has developed a spreadsheet that shows, in tabular and graphical form, the differences in loss (dB) for lengths of for Megtron6 PCB materials and cable sizes from 28 AWG to 36 AWG at frequencies from 1 GHz to 50 GHz. Distances cover 1 in., 12 in., and 1 m. You can use the data to estimate signal losses in your designs.
To alleviate the PCB loss problem, several connector companies, including Molex, Samtec, and TE Connectivity have developed interconnects that let signals bypass PCB traces. The cable assemblies come in several flavors, depending on the application. For example, they connect I/O (often optical modules such as QSFP) to ASIC, ASIC to backplane, board to board in place of a rigid backplane and chip-to-chip on the same board. Figure 2 shows an example of a jump-over cable, this from a demonstration at DesignCon 2017.
Figure 2. Jump-over cable assemblies let signals bypass PCBs between I/O connectors and ASICs or FPGAs. They are available with straight or right-angle connectors. Source: Samtec
"Using cables for 56 Gbit/s signals cut losses by about half over using PCB traces," said Nathan Tracy, Technologist, System Architecture Team and Manager of Industry Standards at TE Connectivity. The plot in Figure 3 compares losses for several PCB materials and for cables of two sizes as a function of frequency. As you might expect, the heavier 30 AWG wire has lower loss than the 34 AWG cable. But, lower loss comes with a tradeoff, Greg Walz, Advanced Technical Marketing Manager at Molex, warns that lower loss cables also don't dampen reflections as much as those with higher loss. The increased reflections could increase the noise floor, which is critical to PAM4 encoding, especially on shorter length cables. Thus, you still need to balance loss with reflections and noise.
Figure 3. Use this table to compare losses of PCB materials and wires as a function of frequency.