June 18, 1998

Hot swap shrinks CompactPCI downtime

Warren Webb, Technical Editor

As CompactPCI's popularity grows, a new hot-swap specification boosts system reliability for competition in the high-availability arena. Before your next--or first--CompactPCI design, take a look at new techniques for live insertion and removal of component boards in a functioning system.

Inserting or removing a pc board with power applied usually leads to disaster. Sparks, smoke, or the silence of system shutdown signals your mistake. It may take several days of frantic troubleshooting to find and replace the damaged components. That kind of mistake might be history with a new hot-swap initiative that the PCI Industrial Computer Manufacturers Group (PICMG) just introduced for CompactPCI.

New hardware and software additions to CompactPCI not only tolerate live-board insertion and removal, but also dramatically improve system downtime by allowing hot swapping of a defective board while the remainder of the system continues to function normally.

CompactPCI's developers based the standard on the Eurocard form factor. A rugged superset of the desktop PCI bus, CompactPCI allows designers to take advantage of off-the-shelf PCI silicon and desktop software (Reference 1). This open architecture is an ideal platform for the growing telecomm and datacomm industries. Recently, according to Joe Pavlat, director of strategic planning of Motorola Computer Group, major telecomm-equipment suppliers, such as Lucent, Northern Telecom, and Mitel, have come down with "outsourcing fever."

Says Pavlat, "They've made a complete 180º flip--from doing everything internally to buying everything and concentrating on the core competencies." Following Lucent's lead, most of the telecomm-equipment suppliers have selected CompactPCI for new digital communications systems. However, standard CompactPCI does not measure up to their vision of low-cost, ultrareliable, easy-to-maintain communications hardware. To meet their system-availability goals, the telecommunications companies need to replace defective boards without shutting down the system. Hardware and software must dynamically route signals around defective components while waiting for repairs. Telecomm systems have had hot-swap capability for years, but the techniques are proprietary and cannot adapt to open architecture. Therefore, the major equipment suppliers have pressed for a hot-swap update to the basic CompactPCI specification.

Recognizing the industry's need for hot-swap capability, PICMG commissioned a working technical committee soon after releasing the 1995 CompactPCI core specification. Although the first core specification and the current revision mention hot swapping and contain some of the necessary elements, a complete picture has been unavailable until now. The technical committee completed work on the CompactPCI hot-swap specification (PICMG 2.1) in March and expects executive-committee approval soon (see box "What is PICMG anyway?").

However, the path to the new specification was not easy. The PCI Special Interest Group's (PCI SIG's) hot-plug initiative clouded PICMG's initial direction. PCI SIG is responsible for the desktop-PCI specification and needed a technique for live insertion and removal of boards for high-end servers. Compaq Computer developed an active-backplane approach that could electrically connect and disconnect any of the four or five PCI slots without disturbing the other slots. Users could insert and remove standard off-the-shelf PCI boards without modification. With its considerable influence, Compaq was able to get Microsoft, Novell, and Santa Cruz Operation to include hot-plug software in Windows NT, NetWare, and UnixWare, respectively. The hot-plug work looked like an ideal starting point for the CompactPCI hot-swap technical committee. However, after six months of debate and strong demands from the telecomm members, the committee elected to take a different approach with a completely passive-backplane. The main reason was that, if any of the components on an active backplane failed, the repair time would skyrocket. A drawback of the passive-backplane approach is that the burden of extra hot-swap circuitry falls on the adapter-board designer.

Three hot-swap levels

The new specification covers silicon, power-circuitry, backplane, and software requirements for the basic, full, and high-availability hot-swap levels. The levels range from manual to fully automated, depending on the software and PCI-interface silicon you use.

Basic hot swap is a manual approach you initiate when you enter the defective CompactPCI board number through the system console. The software then sends a signal to the board to disable PCI-bus communications and notifies you when it is OK to remove and replace the board. Basic hot swap works with PCI-bus-interface silicon and Windows NT.

Full hot swap requires PCI-interface silicon with a built-in control/status register that enables the operating software to automatically identify the defective board. With full hot swap, you remove the defective board without console interaction with the system software. As you open the card ejector, a switch on the board sends an interrupt signal, ENUM#, to the CPU to denote that a hot-swap event is in progress. The CPU then polls the PCI silicon to determine the exact board and follows with board shutdown. The board front-panel LED illuminates to tell you that the board is safe for removal. When you install the new board, the ejector switch generates another ENUM# signal to notify the CPU to enable the board.

High-availability hot swap targets systems in which users demand an absolute minimum of downtime. In addition to all the full-hot-swap capabilities, high-availability systems have a hardware-connection controller that can disable a malfunctioning board and replace it with a standby board already plugged into the system. This level is currently only a vision, but the goal is to achieve fault-tolerant systems and unattended repair.

A typical full-hot-swap adapter card contains a PCI-interface chip, power circuitry, a precharge network, a front-panel LED, and an ejector switch (Figure 1). The power circuitry accepts early power from the longest staged pins on the backplane connector in a controlled manner to prevent power-voltage sag. The power circuitry also isolates the system-reset line from local reset. When you plug a hot-swap board into the backplane, you need a local reset that is independent of the rest of the system. The resistor network provides a 1V-dc precharge bias voltage to the PCI-bus signals before contacting the backplane. PCI-silicon manufacturers plan to eventually include the precharge circuitry within the PCI chip. The hot-swap PCI silicon is electrically equivalent to a standard bus-interface chip with the addition of a control-and-status register accessible by the host CPU. The PCI silicon also interfaces to an ejector switch and a front-panel LED. The ejector switch warns the software of an imminent board-extraction event, and the front-panel LED signals when it is OK to remove the board.

Staged connector pins

One of the essential technologies of hot swap is the physical connection between the board and the backplane. Making a direct connection between the two components would certainly disrupt the other boards on the bus, unless you carefully consider power-supply inrush current and connection to the backplane PCI signals. CompactPCI uses staged pins of three lengths to control the physical connection to the backplane. Card guides ensure that board insertion is perpendicular to the backplane. The longest pins are the first to mate and supply power and ground to precharge the PCI-bus signals to 1V dc. Series resistance limits the power-supply current surge. The medium-length pins connect to the PCI-bus signals that are in a precharged, high-impedance, or disabled state. The shortest pins enable bus communications. Although the original CompactPCI standard specified staged pins on the backplane connector, the developers had to make some changes to implement the passive-backplane approach that the PICMG technical committee selected. Older Compact-PCI backplane connectors do not work with the new specification, and users must retrofit them with the new staged-pin connectors before hot-swap operation. Amp and AVX already offer 2-mm modular connectors for hot swap.

Hot swap also needs another important technology: PCI-bus-interface silicon. The manually initiated basic hot swap works with PCI Version 2.1 silicon that can tolerate V_CC from early power, asynchronous release of reset, and the precharge voltage. However, for full hot swap and high availability, the PCI silicon must include a control-and-status register (Figure 2). The operating software uses the status bits to determine which adapter board in the system is driving the ENUM# signal and to distinguish between board insertion and removal. The control bits in the register clear the ENUM# bit and control the front-panel hot-swap LED. PLX Technology has announced two PCI-silicon devices that are compatible with CompactPCI full hot swap. The $24 (100) PCI 9054 I/O accelerator targets CompactPCI bus-master adapter cards. The $55 (100) IOP 480 includes a 66-MHz, 32-bit PowerPC 401 RISC CPU core in addition to the PCI interface. Samples of both products will be available in the third quarter of this year, and production quantities will be available in the last quarter. PLX has also announced a full-hot-swap reference-design kit incorporating the PCI 9054 on a 6U CompactPCI board.

The most critical technology necessary for hot swap is the software. The operating system must dynamically allocate and deallocate system re-sources, including memory address, I/O address, and interrupt assignment, during a hot-swap event while the remainder of the system functions normally. The hot-swap technical committee elected to use the PCI SIG's hot-plug-initiative software (Figure 3). Hot-plug software operates like basic hot swap. You must request a board change through the system console and wait for the software to deallocate system resources. The device driver for the board must be hot-swap-aware. To support full hot swap and high availability, you must add a common software driver to the operating system to automatically detect the ENUM# signal that precedes a live insertion or removal. The driver then polls the CompactPCI boards in the system to determine which board caused the event, communicates with operating software to initiate the resource changes, and updates the board-status register to complete the hot-swap event. PICMG has sponsored a development effort to produce the CompactPCI common software driver for Windows NT 5. PICMG anticipates that Windows NT 5 will include the required software, but, if not, it will be available free from PICMG or the CompactPCI-board vendors.

Real-time hot swap?

Real-time-operating-system vendors are starting to support hot swap for CompactPCI. For example, Lynx Real-Time Systems Inc has announced the $5995-per-seat Lynx/HA, an option to the company's basic OS that supports high-availability functions. QNX Software Systems also claims to support hot swap with its current microkernel OS.

In addition to the new ENUM# signal, hot swap imposes some capacitance restrictions on the backplane to ensure that the connection and disconnection do not disrupt the PCI signals. Most backplanes meet the capacitance requirements, but it is worth checking if you plan to retrofit a system to meet the new hot-swap specifications. Bustronic and Amp supply hot-swap-compatible backplanes.

One new requirement for the CPU board in a hot-swap environment is that the CPU board must recognize the ENUM# signal to respond to hot-swap requests. This signal appears on a backplane trace that earlier CompactPCI specifications did not use. Most CPU manufacturers have been following the progress of the hot-swap committee and have updated their CPU boards to include the ENUM# input signal. For the record, hot  swap does not work with CPU boards. CPU failures require complete system shutdown for replacement. CompactPCI CPU boards are available from a number of vendors, including Motorola Computer Group, Ziatech, Sun Microsystems, Teknor Industrial Computers, Force Computers, Pep Modular Computers, and OR Industrial Computers.

So, can you buy a hot-swap-compatible CompactPCI board today? Yes, a few specialized boards are available for telecomm applications. Most of the boards that require hot swap are proprietary communications adapter boards that telecomm-equipment suppliers build for their customers and are unavailable on the open market. However, you should consider designing hot-swap features into your Compact-PCI boards. The additional board requirements are minimal if you choose the right PCI silicon. Hot-swap-compatible boards will simplify system maintenance and may put you in the lead when your customer demands that fault-tolerant, self-repairing, zero-downtime supersystem.

Reference

1. Quinnell, Richard A, "CPCI passes potholes, enters the on-ramp," EDN, Oct 23, 1997, pg 91.

Acknowledgments

Thanks to Joe Pavlat of Motorola and PICMG; Mark Easley, Kelly Ambriz, and Rob Robinson of PLX Technology; and Jim Medeiros of Ziatech for their valuable insight.

| EDN Access | Feedback | Table of Contents |