Pump it up: PCI Express

To paraphrase Mark Twain, rumors of the PCI bus's demise have been greatly exaggerated. For years, pundits have predicted the end of traditional shared-bus architectures, such as PCI, because of problems with bandwidth, scalability, and determinism. However, when you examine today's high-performance systems, you'll find that most are still based on standard-bus technology, simply because clever designers are good at squeezing extra performance from inadequate resources. PCI Express, the latest upgrade to the aging PCI bus, promises to pump new life into legacy components and millions of lines of software.

PCI Express offers I/O designers a reprieve from the relentless escalation in processor speeds. Parallel-bus architectures suffer from a fixed maximum bandwidth that data endpoints in a system must share. As you add boards to a bus system to increase performance, you reduce the bandwidth available to each function. You cannot simply increase the shared-bus frequency to match increases in processor frequency, because users demand backward compatibility with legacy plug-in boards. PCI Express combines scalable, high-bandwidth datapaths, packetized data protocols, and compatibility with PCI hardware and driver software. The PCI Express specification is available from the PCI-SIG (PCI Special Interest Group), which maintains the spec.

In addition to bandwidth limitations, real-time problems occur with shared-bus systems. External events happen at random times, and a bus system can respond to only one event at a time. If a data transfer is in progress and a higher priority transfer needs the bus, circuitry or software must suspend the lower priority data until the high priority finishes and then retransmit the blocked data. This contention for the bus is proportional to the number of nodes in the system. The latency of suspended data may become intolerable in some real-time situations. PCI Express eliminates these problems by providing a separate interconnect path for each point-to-point data-transfer possibility so that multiple data transfers can occur simultaneously.

PCI's developers introduced the specification more than 10 years ago as a high-performance alternative to the ISA bus for desktop computers. The initial bus frequency was 33 MHz with a 32-bit-wide parallel datapath, yielding a 133-Mbyte/sec data rate (Table 1). In 1995, the PCI-SIG released Version 2.1 of the specification, which increased the bus frequency to 66 MHz and path widths to 64 bits and maintained backward compatibility with earlier hardware and software. In the late 1990s, the developers added PCI-X to the mix to initially boost clock frequencies to 133 MHz and then finally to as much as 533 MHz. Although every generation is interoperable, each device on the bus must operate at the speed of the slowest installed adapter. In addition, buses in some higher frequency PCI-X configurations have only a single slot. These problems, along with ever-increasing data-transfer rates, led designers to a develop PCI Express, a new architecture for interconnecting I/O devices.

Formerly known as 3GIO (Third Generation Input Output), PCI Express is based on LVDS (low-voltage differential signaling) for maximum bandwidth between nodes. The basic PCI Express link comprises two signal paths that use small differential voltage swings and constant-current line drivers to communicate at 2.5 Gbps in each direction. Designers expect this data rate to increase to 10 Gbps in each direction as silicon-production technology improves. Low-voltage swings deliver low-noise signals at low power consumption. You can easily increase the bandwidth of a PCI Express link by simply adding signal pairs, or "lanes," until you reach the desired performance level. The PCI Express specification supports ×1-, ×2-, ×4-, ×8-, ×16-, and ×32-lane widths. The basic 2.5-Gbps direction-transfer rate plus encoding overhead results in a single-lane performance of approximately 200 Mbytes/sec.

Although the configuration topologies are unlimited, a typical PCI Express system comprises a CPU, a host bridge, memory, and multiple I/O devices (Figure 1). The switch replaces the conventional shared bus and provides communications channels between the CPU/memory and individual I/O-device endpoints. The switch can also set up multiple, simultaneous device-to-device data transfers that do not pass through the host bridge. These independent interconnections greatly increase the effective bandwidth of the system compared with a shared-bus topology with sequential data transfers. Although the figure shows the switch as a separate block, you could incorporate it into the same silicon as the host bridge to reduce costs. You can configure device endpoints to accept PCI, PCI-X, or PCI Express adapters. An endpoint may also include a PCI Express to PCI bridge to drive a conventional, multidrop, legacy PCI bus.

The base specification defines PCI Express as having a transaction layer, a data-link layer, and a physical layer. These logical layers further divide into transmitting and receiving sides to process both inbound and outbound information (Figure 2). Although these layers do not represent a particular physical implementation, they are helpful in describing the functions of PCI Express. On the transmitting side, the transaction and data-link layers divide information into packets and append addressing, error-checking, and sequence information to ensure proper transfer through the interconnecting datapath. The physical layer includes parallel-to-serial conversion, path drivers, and impedance-matching circuitry. If an interconnection link has more than one lane width, the physical layer is responsible for distributing data evenly across all paths. On the receiving side, the three layers decompose packets to match the original data.

The basic premise of PCI Express is that the host PCI software remains compatible with and can talk to and receive data from an endpoint device without new drivers or operating-system software. PCI host software tasks include both initialization and runtime functions. During initialization, the operating system must be able to discover all of the endpoint devices and their characteristics with the same protocol a shared-bus system uses. During runtime, the PCI Express layers must automatically packetize the information, send it over the serial link, and reconstruct it on the other end without the need to modify the original software or the endpoint-adapter hardware. This transparency allows designers to upgrade high-data-rate sections of their systems for improved performance and retain other low-cost but adequate I/O adapters.

The PCI Express architecture also simplifies the implementation of other high-performance system requirements. The high-bandwidth serial links ensure low-latency transfers for isochronous delivery of audio- and video-data streams. You can independently scale individual data links to as wide as ×32 lanes for high bandwidth where necessary. PCI Express allows system designers to regulate the quality of service across the interconnection for improved signal integrity by adjusting packet size and datapaths. Hot-plug and -swap sequences are also simpler to implement with a packet-based, low-pin-count interconnection architecture.

To adapt PCI Express to the requirements of the communications and embedded-system markets, developers have ensured compatibility with the soon-to-be-released Advanced Switching specification. This new specification allows dynamic rerouting, multiple protocols, and high-availability applications. Unlike with PCI Express, Advanced Switching's developers based it on a peer-to-peer architecture, and it requires fabric-manager firmware to configure the datapaths and handle fabric-specific events or error messages. Advanced Switching uses the same physical- and data-link layers as the PCI Express architecture, but Advanced Switching's transaction layer provides specialized communications features, such as high-availability, peer-to-peer, or multicast networking; system management; scalability; and support for most networking protocols.

You can mechanically configure PCI Express hardware into form factors that look similar to those of PCI I/O adapters. A general-purpose computer motherboard could have standard PCI slots alongside PCI Express slots. The PCI Express slots has 36 pins for a ×1 lane width and an increase in the number of pins, depending on the lane width. The PCI-SIG is working on a PCI Express Mini Card specification to produce a successor to the conventional Mini PCI for wired- and wireless-communication peripherals for notebook and mobile computers. The PCMCIA Trade Association has released the Newcard specification for a PCI Express-compatible modular-expansion device to replace the PC card standard (Figure 3). As you would expect, PCI Express has also found its way into high-performance embedded systems, such as the new AdvancedTCA (Advanced Telecom Computing Architecture) from the PICMG (PCI Industrial Computer Manufacturers Group). AdvancedTCA incorporates the latest trends in high-speed interconnection technologies and next-generation processors to give users improved reliability, manageability, and serviceability.

Although PCI Express chip sets have yet to appear, several manufacturers have development projects in progress. The Intel developer network for PCI Express architecture Web site offers a catalog on which vendors can post their product descriptions. Both Texas Instruments and PLX Technology are pioneers in PCI Express-silicon development, and both expect to soon announce bridge devices. Xilinx offers PCI Express endpoint core IP (intellectual property) with ×1-lane width that supports the company's Virtex-II Pro FPGA family. In addition, its Real-PCI Express design kit contains a tested and verified PCI Express ×1 IP core, a prototyping board, reference designs, and support services. The design kit is available now for $29,000. In another project, Artisan Components is working on a PCI-Express PHY (physical-layer) IP core for 0.13-micron chip designs. The modular, eight-lane PCI Express 2.5-Gbps core acts a general-purpose I/O interconnection for adapter cards and attachment points for graphics devices or other serial interconnections, such as 1394b, USB 2.0, InfiniBand, and Ethernet. The PCI Express core creates a complete serial link including multiplexer/demultiplexer, data-encode/decode, and clock-recovery circuitry.

Because PCI Express is in the early stages of development, its designers are interested in off-the-shelf test and analysis equipment. One company that offers such equipment, Computer Access Technology, recently introduced an advanced PCI Express-verification system. The company's PETracer ML (multilane) PCI Express-analyzer system features full bidirectional decoding and capture of ×8, ×4, ×2, or ×1 PCI Express links (Figure 4). The system uses a high-impedance, nonintrusive probing technology without affecting data flow. You can tap into PCI Express fabric with a slot-interposer card or a midbus probe. PETracer ML uses the company's CATC Trace software to process, decode, and analyze captured PCI Express traffic. CATC Trace provides complete decoding of transaction-layer packets, data-link-layer packets, and all PCI Express primitives. For deeper analysis, users can display packet contents as raw 10-bit codes. For automated testing of SOC (system-on-chip) products, Agilent Technologies has integrated protocol analysis of the PCI Express interconnect standard into the popular 93000 SOC-test system.

Catalyst Enterprises offers another test device for early support for PCI Express development. The company's PCI Express Exerciser, the PX-4, is an advanced signal generator for PCI Express systems with link widths to ×4. In a PCI Express card form factor, the PX-4 acts as a pattern generator or as a PCI Express agent (Figure 5). You can also set the PX-4 can to act simultaneously as a master and as a local memory device to perform device-emulation functions. The module can act as a stand-alone development platform for testing PCI Express designs without a motherboard.

Although it has the support of Intel and dozens of other vendors, PCI Express is not the only interconnection standard vying for a place in your future designs. AMD has developed the HyperTransport I/O link architecture with a maximum data rate of 1.6 GHz on each differential wire pair and datapaths as wide as 32 bits. In addition, Motorola and Mercury Computer Systems initially sponsored RapidIO, which supports both parallel, onboard and serial, backplane connections (see sidebar "Another view on high-performance interconnections"). No matter which architecture gains the largest market share, the high-speed, serial data link should become the next step in your high-performance designs. Optical links will perhaps follow.

Author Information

You can reach Technical Editor Warren Webb at 1-858-513-3713, fax 1-858-486-3646, e-mail wwebb@edn.com.