A new embedded architecture: Bigger better boards

The AdvancedTCA (Advanced Telecom Computing Architecture) is a new board, backplane, and software specification for the next generation of telecommunications equipment. With a larger form factor, high-availability features, and high-speed interconnections, AdvancedTCA promises to be a viable off-the-shelf alternative to today's proprietary equipment. This new architecture arrives when the telecom market is facing both business and technical challenges. As competition increases and prices decline, service providers have been forced to reduce capital expenditures. Meanwhile, higher data rates and increased volume have put a strain on the existing, largely proprietary telecom equipment.

To prepare for escalating data rates and the expected convergence of voice and data, telecom-equipment vendors began in 2001 looking at standardized architectures. Architectures such as VME and CompactPCI were a poor match for the unique telecom-equipment requirements. Board area and spacing were too small for the high-density, high-speed circuitry that advanced telecom applications require. Board and rack cooling and power distribution were also inadequate for emerging processors and support silicon. Backplane-legacy-bus and data-handling capacities were other limitations of these standards. A range of system-management approaches also made it difficult to meet telecom's high-availability goals. With these problems in mind, the equipment vendors created a new standard for telecom to take advantage of the cost savings of off-the-shelf devices. As many as 100 companies participated in developing the AdvancedTCA specification under the auspices of the PICMG (PCI Industrial Computer Manufacturers Group). The group early last year released the spec as PICMG 3.0; a short-form version of the AdvancedTCA specification is available at www.picmg.org.

High-speed serial-data links and switch-fabric technology are the cornerstones of AdvancedTCA technology. With redundant-star or full-mesh data transport and several switched-fabric alternatives, an AdvancedTCA backplane can scale to 2.5 Tbps. The extra-large board area supports all of the latest silicon and provides input power and cooling for as much as 200W per slot. The AdvancedTCA specification provides hot-swap capability for all boards and active modules, allowing systems to achieve and even exceed the elusive 99.999%, or "five-nines," availability. A sophisticated management feature monitors the health, power, cooling, and even keying of plug-in modules to ensure that systems are operating efficiently. Modules get power from redundant –48V-dc power feeds and data from redundant control and data planes to prevent a single failure from bringing down an entire chassis.

Mechanically, AdvancedTCA elements are organized into chassis or shelf structures (Figure 1). The basic 12U, 21-in.-high shelf comprises power input, management electronics, a backplane, card guides, and provisions for cooling. The maximum number of plug-in card locations varies, depending on the rack size: 14 slots for a 19-in. rack and 16 slots for the 600-mm (approximately 24 in.) ETSI (European Telecommunications Standardisation Institute) rack. A 1.2-in. board spacing allows for heat sinks and rear surface-mount components. Forced air within the shelf provides cooling for as much as 3200W. Each backplane slot accommodates an 8U×280-mm front board and an optional 8U×70-mm, rear-mounted transition module for back-of-the-rack I/O.

An AdvancedTCA backplane has zones to handle power distribution, system management, data transport, and rear I/O (Figure 2). The heavy-duty Zone 1 connectors along the bottom of the backplane carry power-management and low-data-rate-management signals to each slot. Because a low-voltage power-distribution scheme would need to deliver hundreds of amperes across the backplane, the standard's developers chose –48V dc, already a telecom standard, as the power source. The chassis provides filtering and delivers dual –48V-dc feeds to each board. The power connector handles voltage sequencing when users remove or insert boards during hot-swap. Commercially available dc/dc converters on each board provide the actual voltages that local circuitry requires.

A second backplane zone provides for high-speed, interboard data exchange. PICMG selected the Erni-Tyco ZD connector for use as the high-speed data-transport connector for AdvancedTCA (Figure 3). This high-density connector supports differential signals at rates as high as 5 GHz. This second zone allows for four interfaces between boards in a shelf. A base interface of 10/100/1000 BaseT Ethernet interconnections arranged in a dual-star configuration provides redundant paths across the backplane. The dual-star topology requires card positions one and two to be dedicated hub slots. Each hub slot has a channel connection to each node slot in the backplane and to the other hub slot. The base interface replaces the parallel, multidrop bus of earlier architectures. A channel-update interface connects adjacent boards so that redundant devices can communicate privately. The synchronization-clock interface delivers a redundant set of clocks to each board for synchronous-timing applications.

The fabric interface, the most interesting part of the data-exchange zone, provides a full mesh interconnection at which each slot has a direct connection to every other slot. Because all paths can simultaneously transfer data, a 16-slot, full-mesh backplane has an aggregate available bandwidth of 2.4 Tbps. To satisfy diverse opinions in the industry, the base AdvancedTCA specification specifies no fabric technology for use for data transport. Instead, a series of subsidiary specifications, PICMG 3.1 through 3.5, define backplane details for Ethernet, Fibre Channel, InfiniBand, StarFabric, PCI Express, and RapidIO. Although competing fabric technologies are not interoperable, this approach prevents the specification from stalling until the marketplace selects a winner.

Zone 3 is reserved for routing signals to the rear of the AdvancedTCA rack. The specification defines no connectors for this zone, leaving that specification to the board vendor, depending on the application. Rear-transition modules, similar to CompactPCI, are optional in AdvancedTCA. The major difference is that front-board, Zone 3 connectors plug directly into mating rear-transition-module connectors and are not routed through the backplane. In fact, the Zone 3 area in Figure 2 is simply a reserved open space for front and rear boards to mate.

Unlike its predecessors, each AdvancedTCA chassis requires a "shelf manager," a sophisticated management system, to ensure that all components work together. The shelf-management system monitors all active subsystems, reports anomalies, and controls the operation of each installed device. Electronic keying, an example of shelf management, ensures that modules are installed correctly and that the shelf has adequate power and cooling resources before activation. Shelf management also prevents mismatched fabric technologies from attempting to communicate. The shelf manager can also activate redundant circuitry; preside over a hot-swap event; or collect part numbers, serial numbers, revision letters, and software versions to adapt application software to changing configurations. The PICMG adapted the AdvancedTCA-management scheme from the CompactPCI management bus and the IPMI (Intelligent Platform Management Interface) specification.

Compatibility with mezzanine cards can greatly extend the capabilities of a new standard, such as AdvancedTCA. Hundreds of PMC (PCI-mezzanine-card) modules available today support a variety of computing and I/O functions. You can easily incorporate these PMC cards into early AdvancedTCA board designs to gain access to off-the-shelf designs. The large AdvancedTCA front-panel area can support as many as four PMC modules. With a proper interface on the board, other mezzanine-card standards would work equally well with AdvancedTCA. However, all mezzanine modules have limitations when you compare them with the new requirements of AdvancedTCA. For example, higher speed interfaces and expanded power requirements of next-generation silicon would require adjustment to PMC specifications. Likewise, the AdvancedTCA availability goals require remote-management and hot-swap capabilities, neither of which is available on existing mezzanines.

PMC's shortcomings plus the fact that the larger AdvancedTCA board will allow larger mezzanine cards prompted PICMG to sponsor a new mezzanine card. The proposed AMC (Advanced Mezzanine Card) specification, PICMG AMC.0, describes hot-swappable, field-replaceable mezzanine cards that target use in packet-based, high-availability telecom systems. AMC cards communicate with the carrier or base card via a packet-based serial interface over a variety of protocols, including Ethernet, PCI Express, Rapid I/O, and InfiniBand. The specification supports half-height, single width; half-height, double-width; and full-height, single- and double-width modules. The modules have escalating power limits starting at 20W for the smallest module and increasing to 60W for the largest. Figure 4 shows prototype examples of various-sized AMC modules.

Although the AdvancedTCA specification is fairly new, the PICMG Web site lists more than 20 vendors with related products available or nearing completion. For example, a 14-slot AdvancedTCA shelf system from Schroff targets telecom and networking applications including wireless IP (Internet Protocol), telephony, and optical switches (Figure 1). The system's backplane is available in a variety of topologies, including full mesh, dual star, and dual dual star. With three independent and redundant fan trays, the system provides adequate cooling to all slots, even in the event of a fan failure. The fan trays are low cost, field-replaceable units, reducing labor and maintenance costs. The system also incorporates two integrated shelf-manager/fan controllers, based on Pigeon Point software technology, to provide redundancy as well as cost-effective thermal and system management by combining these functions on one board. Prices for the new 14-slot system begin at $4850, and delivery time is four to eight weeks.

Force Computers last year introduced one of the first AdvancedTCA single-board computers (Figure 5). The ATCA-710 features a 1.8-GHz Mobile Intel Pentium 4 Processor and supports control and data-plane-processing applications including 2.5G/3G wireless and broadband-wire-line networks. To maximize flexibility, the board includes an optional daughtercard with quad PMC slots for additional processing or I/O expansion. An available, onboard, 12-port Gigabit Ethernet switch provides direct network contact between PMCs, the main CPU, and any base- or fabric-backplane interfaces. The ATCA-710 provides 1.6-Gbyte/sec memory bandwidth and a full-duplex, 3.2-Gbyte/sec daughtercard interface along with as much as 2 Gbytes of 266-MHz, double-data-rate SDRAM. It also includes an onboard IPMI controller and as much as 8 Mbytes of user flash and 4 Mbytes of boot flash. The ATCA-710 supports carrier-grade Linux and sells for $3900 (single units).

The Intel NetStructure MPCBL0001 single-board computer complies with AdvancedTCA specifications and includes single or dual 2-GHz Xeon processors, a 400-MHz system bus, and support for more than 4 Gbytes of SDRAM (Figure 6). Occupying a single AdvancedTCA slot, the hot-swappable board includes backplane connections for dual Gigabit Ethernet and optional support for dual Fibre Channel. A single PMC socket supports 64-bit transfers at 33 or 66 MHz. A built-in intelligent platform-management controller monitors, controls, and performs diagnostic functions. User-configurable embedded BIOS in onboard flash memory loads an operating system from a variety of boot devices. Testers have validated the board for use with the Linux operating system. The Intel NetStructure MPCBL0001 comes with Fibre Channel and dual 1.6-GHz Xeon processors and costs $3266.

Another clever approach to salvage the investment in a previous generation of technology comes from GNP Computers. Its AdvancedTCA carrier board provides a simple approach to integrating existing and legacy CompactPCI boards in an AdvancedTCA shelf (Figure 7). The carrier board includes a 1-Gbit Ethernet switch that allows the PICMG 2.16-compliant CompactPCI card to connect to the AdvancedTCA fabric interface. A full-mesh implementation uses the onboard switch to allow the CompactPCI card to talk directly to every other card in a PICMG 3.1-compliant shelf system. Prices for the base interface and full-mesh carrier cards begin at $2195 and $3945, respectively.

Although PICMG has yet to ratify the specification, Artesyn Communication Products has announced an AdvancedTCA-compatible AMC module (Figure 8). The hot-swappable, single-width, full-height AM7501 AMC card features a 1.6-GHz Pentium M processor with a 400-MHz, 3.2-Gbyte/sec front-side bus; 1 Mbyte of Level 2 cache; and 2 Gbytes of SDRAM. A 64-bit PCI-X bridge provides high-speed access to two Gigabit Ethernet channels, as much as 128 Mbytes of flash memory, a USB interface, an I²C system-management interface, and a front-panel 10/100BaseT management interface. The AM7501 costs $1500.

Although AdvancedTCA is mainly a hardware specification, telecom equipment manufacturers want to extend the cost savings of an open system to the software operating system. Toward this end, a group of hardware and software vendors formed the Carrier Grade Linux Working Group, a part of the OSDL (Open Source Development Lab). Members hope to create a requirements specification that will allow Linux to compete with Unix, the leading telecom operating system. The latest version of the carrier- grade Linux requirements definition is available for downloading at www.osdl.org. The specification adds functions for telecommunications and data communications with requirements for high availability, fault management and real-time performance. The document includes standards such as the LSB (Linux Standards Base), IPv6 (Internet Protocol Version 6), SNMP (Simple Network Management Protocol) support, and POSIX (Portable Operating System Interface) interfaces for timers, signals, message queues, semaphores, event logging, and threads. Members are responsible for devising their own tests for validating compliance with carrier-grade Linux requirements. MontaVista Software offers a fully supported Linux CGE (Carrier Grade Edition) that is compatible with Version 1.1 of the OSDL's specification. MontaVista's CGE also includes unique technologies, such as online debugging and patching of deployed applications.

With today's business environment, the outlook for AdvancedTCA looks bright. Equipment vendors are looking for the low-cost, off-the-shelf products possible with an open standard such as AdvancedTCA. The market for telecom equipment is worth more than $20 billion per year in shipments, and that market has constantly changing requirements and technology. Because most of the current equipment is custom or proprietary, designers must seriously consider the AdvancedTCA platform for upgrades or new products.

You can reach Technical Editor Warren Webb at 1-858-513-3713, fax 1-858-486-3646, e-mail wwebb@edn.com.