| |
|
April 24, 1997
Serial
backplanes supplant parallel buses
in fast systems
Patrick van Eijk, Vitesse Semiconductor
Corp
In data-communication subsystems and
clustered multiprocessing systems, skyrocketing data
rates complicate internal, parallel-bus designs, but
point-to-point serial links borrowed from
industry-standard storage interfaces, such as Fibre
Channel, provide an apt substitute.
Burgeoning
digital-video and other bandwidth-intensive multimedia
applications require turbocharged LAN and
wide-area-network (WAN) data rates. Because of these rich
data streams, designers working on routers, switches, and
other data-communication devices likely find that
bandwidth on internal data highways has been stretched to
the limit of parallel-bus technology. Designers working
on multiprocessing clusters have probably encountered the
same roadblock. Transmission-line and skew problems
prevent faster clocking of parallel buses, and logistic
issues, such as connector cost and pc-board real-estate
requirements, stand in the way of wider buses. Designers
facing these problems can turn to serial interfaces and
leverage technology, such as 1-Gbps transceivers
developed for the Fibre Channel disk-drive industry.
Symbiotic usage in the disk industry should guarantee
widespread availability and reasonable cost for
components, and the serial links match the reliability
and bandwidth requirements of multiprocessing and
data-communication systems.
 To understand the limitations of a
parallel backplane, consider a data-communications
example. A typical Ethernet, asynchronous-transfer-mode
(ATM) or Transmission Control Protocol/Internet Protocol
(TCP/IP) switch or router consists of M input ports and N
output ports, a switch fabric, and a control system. The
control system handles output-port arbitration, data-flow
scheduling, error recovery, maintenance, and
administration. For simplicity, assume that M=N resulting
in line cards that have one input and one output port. In
a traditional architecture, these line cards interface to
one or more switch cards through a dedicated parallel bus
running for example at 50 MHz in a 622-Mbps ATM switch.
Generally, vendors support multiple protocols, such as
Ethernet, ATM, and Frame Relay. Therefore, you can view a
line card as having two distinct logic sections: a front
end, which comprises the appropriate logic to interface
to user protocols, and a back end, which provides the
appropriate data-formatting, buffering-logic, and
backplane-interconnect circuitry. Figure 1 shows a typical switch or router
organization.
Switch
shows parallel limits
In a typical
16-port Gigabit Ethernet switch, for example, each I/O
port pair resides on one line card, supporting either one
true full-duplex Gigabit Ethernet or 10 100BaseT
full-duplex links. A high-bandwidth bus needs to deliver
the data across the backplane from and to the switch
card. At a full-duplex data rate of 1 Gbps, a parallel
bus operating at 50 MHz requires 40-bit width, and a
16-port system requires a 640-bit backplane. Accounting
for extra bandwidth for clock, error-monitoring, and
control signals, designers find that a parallel backplane
can easily stretch to 800 or even 1000 bits wide.
Designers
needing to support more ports or faster data rates could
accommodate the increased bandwidth requirement by
widening the bus or increasing the bus frequency.
Although upping the bus frequency is usually not an
option because of transmission-line effects due to
impedance mismatches, crosstalk, and signal skew,
widening the bus results in higher connector pin counts,
increasing cost and insertion force (the force required
to insert a board into the backplane connector).
Fibre
Channel coding scheme
In contrast,
a serial point-to-point interconnect offers some
significant advantages over a parallel backplane. For
starters, serial backplanes can use data-coding schemes,
such as the 8B/10B coding developed for Fibre Channel,
which embeds a clock signal with the data stream, thereby
eliminating the extra clock signals a parallel bus uses.
The 8B/10B encoders and decoders can also handle error
detection based on disparity errors and invalid 8B/10B
codes. A serial-based design can still maintain control
signals for output-port arbitration and scheduling in a
parallel format because those signals require relatively
low clock rates. Logistically, meanwhile, the significant
reduction in the number of signals required to carry the
data serially results in an easily extensible system
design.
A
well-terminated serial point-to-point connection can
easily support data rates of 1 Gbps and higher. In the
above example, you can replace each 40-bit parallel
backplane bus with a serial point-to-point connection
using a 1.25-Gbps transceiver chip on each line card and
16 transceivers on the switch card. These chips have a
750-mW power dissipation, thus requiring no heat sink;
occupy 10×10 mm of board space in a 64-pin PQFP; and
will be available as a commodity product at $10 to $15.
Serial
link simplifies line card
 Surprisingly, the move to a serial
architecture results in a relatively simple
line-card-design problem. In the 16-port Gigabit Ethernet
example, the front-end logic of each line card requires a
Gigabit Ethernet transceiver and a port-controller ASIC (Figure 2). The back end of the line card comprises a
single backplane-transceiver IC. In the case of a
1000BaseT Gigabit Ethernet design, the line card uses
identical front- and back-end 1.25-Gbps transceivers
because the Ethernet community also adopted the Fibre
Channel physical-layer technology.
 In a typical Gigabit transceiver
architecture, the transmitting side of the IC accepts
10-bit 8B/10B encoded data at 125 Mbps, latches the data
in on the rising edge of the reference clock, and
serializes the data onto the TX± pseudo-ECL (PECL)
differential outputs at a baud rate 10 times the
reference-clock frequency (Figure 3). The 8B/10B encoding guarantees a
minimum transition density in the serial signal to
perform a correct clock and data recovery at the
transceiver differential input. The transceiver samples
serial-receiver data on the RX± PECL differential
inputs, recovers clock and data, deserializes the data
onto the 10-bit receiver data bus, and outputs two
recovered clocks at 1/20 the incoming baud rate.
The
port-controller ASIC provides a significant design
challenge because it incorporates an Ethernet
media-access controller; a system-management interface,
such as a PCI bus; buffering logic; memory control for an
external address-translation RAM; and 8B/10B coding
circuitry. The 8B/10B coding circuitry requires a few
thousand CMOS gates. Among other functions, the coding
circuit must insert word-alignment control characters
into the data stream that the line card transmits to the
switch. The Fibre Channel spec defines a unique
"Comma" character (0011111XXX), and a sequence
of three such characters instigates word alignment. For
the data stream from the switch to the line card, the
transceiver automatically handles word alignment,
provided that you enable comma detection (EN_COMDET is
held HIGH).
ASIC vendors
offer most port-controller functions as macrocells,
thereby simplifying ASIC design. The design requires a
state-of-the art ASIC from a semiconductor-process
perspective because of the high data rates involved. The
parallel interfaces on the Ethernet and backplane
transceivers run at 125 Mbps, so the port controller
requires two 125-Mbps interfaces. These speeds demand a
0.35-µm CMOS ASIC implementation.
The
line-card-design example also assumes that a copper
medium connects to the Gigabit Ethernet 1000BaseT
interface. Designers could add an optics module to
convert electrical signals to optical signals, thereby
adding support for a fiber-optic medium. Furthermore, the
line-card-design example supports only Gigabit Ethernet,
so the design requires no other back-end
packet-formatting logic. In a system such as a
multiprotocol router, the back-end logic of each line
card would require additional circuitry to segment and
reassemble data traveling across the backplane into
fixed-length packets. Moreover, the design would require
additional queuing logic.
Backplane
and switch design
The move to a
serial backplane also simplifies other portions of the
overall system design, including the backplane and
switch-card design. Start any such design with a robust
clock-distribution scheme to ensure a reliable result.
All of the 1.24-Gbps transceivers on the line and switch
cards must operate using the same 125-MHz reference
clock. A single reference clock prevents the loss of bits
due to frequency mismatches.
 The switch card requires 16
transceivers and a switch-fabric ASIC with 16 ports. Figure 4 shows a possible board-level implementation
of a 16-port Gigabit Ethernet switch card. The figure omits clock and control signals
for simplicity. The transceivers on the switch card
handle word alignment in each of the 16 incoming, 10-bit,
parallel-data streams. Once the transceivers detect three
sequential comma characters, the design realigns the word
boundary and the COM_DET signal notifies the appropriate
input port on the switch-fabric ASIC.
The
switch-fabric ASIC must handle word alignment on the
outgoing data streams, 8B/10B coding, and switching
functions. Designers should take care in deciding when to
add the word-alignment characters on both sides of the
switch/line-card link. Theoretically, a design requires
no additional word-alignment control once a serial
point-to-point link is up and running. Possible
signal-phase changes due to voltage fluctuations, system
temperature, and jitter, however, make it a good design
practice to insert comma characters after some arbitrary
number of cells or packets have been transmitted.
A combination
of regular word-alignment characters and innovative
buffering ensures reliable links. In a fully synchronous
switch fabric, all cell or packet boundaries must align
properly at the switch input. Although all transceivers
in the system example are frequency-locked, the
interfaces between the switch ASIC and the transceivers
on the switch card are asynchronous. A totally
synchronous architecture is unfeasible because you cannot
phase-lock the recovered clocks to the reference clocks
because of backplane trace-length variations. The
different physical routing of each incoming data stream
inevitably introduces phase differences that result in
parallel-word misalignment by plus or minus one word. A
2-byte-deep elastic buffer on each of the 16 input ports
of the switch-fabric ASIC solves this problem. The 8B/10B
coding circuit follows the buffer. The 16 coding circuits
in the switch-fabric ASIC add 30,000 to 50,000 gates of
CMOS logic to the total gate count.
Backplane-design
techniques
You probably
relegate backplane design to the final step in
Ethernet-switch development, but only proper techniques
yield a suitable result. Serial data propagates from the
3.3V differential output PECL driver on a line card
through a differential line-card trace, a
controlled-impedance backplane connector, a differential
backplane trace, another controlled-impedance backplane
connector, and a switch-card differential trace to a
differential PECL input. The high-speed PECL differential
inputs and outputs require a 75 ohms (150 ohms
differential) termination to ensure proper operation and
optimize signal quality. You should place the high-speed
differential board traces (controlled impedance) in the
same pc-board plane, separate them by a distance two to
three times the width of each trace, and sandwich them
between ground planes. Ensure that the traces of a
differential pair have the same length and minimize
stubs.
Also, specify
continuous ground planes rather than sectioning the
planes to provide isolation to various components.
Sectioning of the ground planes interferes with the
ground-return currents on the signal lines. The smaller
the ground plane, the less effective it is in reducing
ground bounce noise. If the Gigabit transceivers on the
line and switch cards are the only 3.3V devices, use a
dedicated power plane for the transceivers. If other
devices operate from 3.3V, you can share the 3.3V plane.
In either case, dedicate a 0.1-mF bypass capacitor to
each transceiver power-supply pin, and locate the
capacitor as close as possible to the pin, preferably on
the top-side board. You should also design in additional
bypass and isolation for the analog power and ground pins
because they supply power to the PLL. Noise on these pins
could result in noise on the TX± outputs and reduce
jitter tolerance on the RX± inputs. Finally, ensure that
switching power-supply noise, which usually ranges from
50 to 250 kHz, doesn't couple to the PLL power-supply
pins.
Realizable
serial-backplane advantages
Compared with
a system based on a parallel bus, the serial Ethernet
switch example adds $320 to $480 in system costs. You can
attribute these costs to serial Ethernet's 32 1.25-Gbps
transceivers. However, the technology offsets these costs
with the savings from simpler backplane connectors, a
simpler pc board with fewer layers, and increased
reliability. The serial backplane is also more reliable
because 32 full-duplex traces operating at 1.25 Gbps
replace 1000 single-ended traces operating at 80 Mbps.
Moreover, the serial approach offers an upgrade path to
handle higher data rates unavailable in a parallel
design. Designers can later migrate to 2.5-Gbps
transceivers, thereby doubling the internal data
bandwidth of the system. In a parallel environment,
doubling the line rate of 1000 signals from 50 to 100
Mbps would be virtually impossible. Furthermore, the
expected availability this year of ICs with multiple
integrated Gigabit channels will reduce the number of
devices on the switch cards. Designers will also have the
flexibility to develop line cards with one, two, or four
Gigabit Ethernet links per card by using multichannel
transceivers.
Although the
backplane interconnect example focuses on a 16-port
Gigabit Ethernet switch, the serial-backplane concept
applies equally well to ATM switches, multiprotocol
routers, and other bandwidth-intensive interconnect
applications, such as multiprocessor clusters. Increasing
demand for more network bandwidth has pushed parallel-bus
backplane architectures to the limit in networking and
other systems with high-bandwidth data-transfer
requirements. High-volume, low-cost, and small-footprint
Gigabit Ethernet transceivers enable a serial-backplane
implementation, resulting in a significant reduction in
connector pin counts and board space as well as a more
robust system design.
|
