July 3, 1997

ATM switch chips switch on Net reliability

STEPHEN KEMPAINEN, TECHNICAL EDITOR

Transferring Internet, intranet, and extranet traffic increasingly depends on reliable ATM switches, because a single fault in the backbone can spiral into huge losses. To build reliable ATM switches, redundancy is key: Chips and system architectures must collaborate on switches that can survive multiple simultaneous faults.

Most businesses are now using the Internet, intranets, and extranets--almost as much as their telephones--to send and receive information. As a result, users expect the Net to be just as reliable as the phone system. Achieving that reliability is a challenge to the asynchronous-transfer-mode (ATM) switches deep within the Net: The switches must lose as little data as possible while working 24 hours a day, 365 days a year--at maximum throughput. A common approach to reaching reliability goals is to add redundancy to a design. Core switch chip sets are now available on which to build a reliable, high-performance ATM switch.

The first job of any switch is to quickly transfer blocks of data to their destinations. Traditionally, switches used a best-effort approach to transfer data blocks whether the blocks comprised ATM cells or Internet-protocol packets. "Best effort" means that the switch discards cells or packets from a block to alleviate congestion. Then, upon missing the block, the destination notifies the data source to resend that block. However, at ATM's 622-Mbps speed, using this approach would overwhelm the network resources with retransmissions of entire data blocks, in turn causing more congestion that would balloon into more lost cells.

Therefore, minimizing cell loss is essential for ensuring that the switch can reliably keep the traffic moving. Toward that end, some switch chips use nonblocking architectures in which all ports can concurrently receive from and transmit to different ports. However, congestion still occurs when multiple ingress ports want to use the same egress port. Consequently, even nonblocking switches need integrated traffic control to avoid congestion.

Reliability also means that the switch must offer high availability, measured in downtime per year. The telephone-company benchmark for availability is 99.999%, or "five-nines," availability: 3 minutes of downtime per year over rolling--continuously overlapping--six-month intervals. However, a new benchmark for ATM switches aims to achieve 99.9999%, or "six-nines," availability: no more than 30 sec of downtime per year.

No matter how good the chips are, though, the software is critical to reliability. Meeting the six-nines level means that the software must also provide fault tolerance, maintenance, and fault recovery. The software and hardware must collaborate to achieve an architecture in which even multiple simultaneous faults cannot bring down the entire switch. Using standards-based software, such as the Telecommunications Management Network, the Simple Network Management Protocol, and the Remote Monitoring modules, minimizes reliability problems. Such standards help by applying a rigorous methodology to writing and licensing software.

Redundancy, redundancy

The first requirement for reliable design is redundant components. How much redundancy you need depends on the number of simultaneous faults that you want your switch to survive. Duplicating power supplies, controllers, switch fabrics, management processors, and line cards lets your switch endure multiple faults. Figure 1 shows redundant links between eight line cards and two 8×8 (eight input ports and eight output ports) switch cores. Duplicating the links between redundant components lets a switch tolerate even more faults.

Taking redundancy a step further is to have component pairs run in parallel, or lock step: duplicating the traffic on each component, so that if one should fail the other is already carrying the traffic. For example, two line cards operating in lock step would both be carrying the same link traffic. Other components that should run in lock step include the monitoring, maintenance, and fault-recovery systems. The more redundant components a switch has, the more likely it is to survive simultaneous faults.

Core switch chips can support redundancy with proper partitioning and features (Table 1). For example, Lucent Technologies' Atlanta chip set directly supports redundancy with chip partitioning and clock distribution (Figure 2). The ATM buffer manager (ABM) supports simultaneous links to dual switch fabrics by duplicating the interface to the ATM switch element (ASX). In addition, the system-timing and -clocking schemes allow for independent line-card clocks and arbitrary clock skews across a backplane. This approach eases duplicating the clock distribution by eliminating clock trees and minimizing skew effects.

Another, often-overlooked, component of reliable switches is a redundant bus for test and maintenance (TM) . In normal operation, the data bus carries TM information. But, if the data bus should, for example, lose termination power, the TM bus gives alternate access to component modules. When choosing a TM bus, you should place higher priority on availability than on high performance. TM buses include the IEEE-1394, or Firewire, bus and the IEEE-1149.5-1995 IEEE Standard for Module Test and Maintenance Bus.

Along with a redundant bus, failure compartmentalization--stopping faults in one component from affecting others--is key to a reliable design. For example, a clock-distribution scheme can limit the effect of a faulty clock source. Redundancy and fault compartmentalization can be expensive, but a dead switch is even more expensive in lost revenue and reputation.

No surprises

There is more to a reliable design than just redundancy. You cannot let network events take your switch by surprise. Therefore, you must monitor and maintain operations under normal conditions, so that abnormal conditions produce alarm signals before catastrophic failures. For example, running scheduled diagnostic routines on alternate redundant components maintains their readiness. To let you observe operations and perform maintenance, the ITU standard I.610, B-ISDN Operations and Maintenance Principles and Functions, specifies the ATM operations, administrative, and maintenance (OAM) layer. It functions at both the network and the internal-switch levels. The standard reflects ATM's telephony origins and thus incorporates the robustness that has prevailed there for decades.

The OAM specification covers both hardware and software components. It also works internally and externally for connections within and between switches. OAM provides the tools to monitor and probe the system for warning signs of trouble; it also provides remote-diagnostics and -configuration tools (see box, "OAM mechanisms in ATM").

OAM depends on accurate statistics gathering. However, the 622-Mbps traffic on the switch ports overwhelms traditional statistics-gathering techniques. To gather statistics, the hardware cannot just sample traffic flow. It must continuously check each virtual connection (VC) to catch problems before they mushroom. Therefore, chips such as Lucent's ATM-layer user-network-interface (UNI) manager (ALM) support statistics gathering. ALM maintains a variety of optional, per-connection, 31-bit statistics counters in external memory. They are large enough to track lots of ATM cells until the management processor can collect the statistics and clear the counters. Also, to aid in statistics reporting, the ALM captures available-bit-rate (ABR) resource- and performance-management OAM cells at ingress. It then routes them to any VC for egress or to the processor port for endpoint processing. The management processor can then insert OAM and resource-management cells back through the processor port.

The most efficient way to handle an OAM cell is to avoid interrupting the management processor. The hardware can perform all real-time OAM chores and let the software perform diagnostics and processing. For example, the RCMP-800 ingress port controller from PMC-Sierra automatically receives and generates alarm indications and remote-defect-indication cells. It also generates backward-performance-reporting cells. When OAM cells, such as those for activating and deactivating VC monitoring, require processing, the RCMP-800 routes them to the processor port. In addition, the RCMP-800 identifies loopback cells for direct extraction and return to the loopback source by the egress.

Nonstop repair strategy

For full reliability, a switch must not only detect, but also recover from and repair faults. Recovery and repair start with rerouting traffic around component faults to maintain switch availability. Next, the switch should recover as many statistics as possible. Recovering statistics that occurred before the fault occurred helps find its cause. OAM and a TM bus are useful in collecting data to troubleshoot the problem. Once the diagnostic processor identifies the problem, the maintenance processor should plan a repair incorporating hot-swapping--keeping the switch operating while replacing the problem component. Hot-swapping also helps to avoid delays incurred when powering down and from reset and initialization after power-up. Some loss of throughput may occur while you perform hot-swap maintenance, but that situation is better than having the switch go out of service.

Hot-swapping without losing cells is a big challenge for recovery software. Software must not only alert the system to complete outstanding transactions and save important information before shutting down the faulty component, but also manage the switch to redundant components. It then must synchronize reconfiguration of the replacement component into the system.

Both hardware components and software modules need hot-swapping or "hot-upgrading." Software upgrades for bug fixes and feature enhancements should cause minimal disturbance to the switch. All software modules need the ability to deactivate for upgrade by replacement modules on a scheduled or unscheduled basis. This feature facilitates upgrading software and hot-swapping hardware components.

To perform hot-swapping, hardware can have no daisy-chained signals, such as clocks, between components. Thus, if a component fails, it does not bring down other components. The other hardware requirement is that no electrical transients cause data loss or component failure. (Inserting the capacitive load of a plug-in board into an active bus can cause such transients.) The recovery software is responsible for avoiding data loss. However, if your design subjects a board-edge chip's I/O pins to these transients, you should check with the chip manufacturer to ensure that the I/O buffers include transient protection.

In addition to nonstop maintenance and repair, a reliable switch needs congestion control to provide high throughput and avoid losing cells. Traditionally, data-networking switches discarded cells when traffic backed up at popular egress ports and storage buffers overflowed. Managing congestion by dropping cells worked in these cases because the end destination would miss sequentially numbered cells and initiate a retransmission. Retransmissions are no longer acceptable because of the network speeds and time-critical traffic the network carries, such as video and audio. Now, to achieve lossless congestion control, your design must use ATM flow-control techniques that force data sources to shape the traffic. "Traffic shaping" means that the source doesn't send cells into the network until the source has reasonable assurance they will get to the destination on time without being dropped.

Traffic shaping begins at the cell source and continues through the network at each intermediary switch until the destination. A traffic scheduler at each switch shapes traffic by determining when to launch cells from an egress port. The scheduler considers priorities, quality of service (QoS), and network feedback while determining which VC's cells to launch downstream and when. The VC setup protocol establishes the priority level and QoS parameters for the VC. Each scheduler in the connection records the parameters.

Traffic schedulers, such as that in MMC's ATMS2000 switch engine, are critical to congestion control. The switch shares a scheduler that schedules traffic on a per-VC basis. This approach saves the expense of having a scheduler at each port. On the other hand, Lucent Technologies' Atlanta chip set includes scheduling logic on the ALM and ABM at each port card. However, the Atlanta lacks the per-VC queuing ability, instead using programmable, weighted, round-robin algorithms in scheduling four priority queues at each port.

The ATM-cell stream flows in a VC that could pass through many switches before reaching the destination. Ideally, each switch would maintain the shaping parameters on a per-VC basis. Any chips you select for core switching should be able to manage enough VC descriptors for the converging VCs at the targeted application, be it deep in the core or at the edge of the network. For instance, the Lucent Atlanta ALM supports as many as 64,000 VCs at ingress and at egress for each port--enough to handle many converging VCs at the core of a large network.

Traffic shaping relies on sophisticated queuing and buffer management to get a VC's cells to the output ports in time to meet launch schedules. One VC cannot block another. This fact is especially true for switches at the core of the Net where thousands of connections with many QoS requirements and priorities converge. The switch chips should not allow head-of-line blocking, in which cells become stuck in FIFO buffers behind cells with lower priority. The MMC Switch Engine and XStream chip set solves this blocking problem by eliminating the use of FIFO buffers (Figure 3). The port interface stores the cell payload in a common data memory and sends the cell header to the switch-controller chips. The controllers use pointer management instead of FIFO buffers to implement nonblocking queues.

Another way to control traffic through a switch is to implement feedback paths that prohibit buffers from overflowing. The Lucent Atlanta chip set supports a nonblocking, back-pressure scheme that carries buffer-full messages among the ATM crossbar element, the ATM switch element, and the ABM to stop buffer overflow. It provides for lossless storage throughout the switch. The Atlanta ABM handles full-duplex traffic at 622 Mbps and queues as many as 32,000 ATM cells in each direction. The MMC chip set, on the other hand, uses one controller and no FIFO buffers for all traffic through the switch; therefore, it requires no separate feedback connections.

Continuous traffic control also requires per-stream traffic policing. To implement this feature, some chip sets, such as MMC's XStream, include usage-parameter control, which checks the VCs to ensure that they conform to the contract for QoS that the VC negotiated with the switch at call setup. During the negotiation, the switch assesses its resources and reserves those that VCs require. If a VC transfers more traffic than the resources allocated to it can handle, then that VC violates its contract and interferes with the resources allocated to another VC.

The best way to control traffic through the switch is to implement per-VC queuing. In the MMC chip set, per-VC queuing uses three switch-interface chips plus memory to hold the queues and monitor the fill levels. The Queuing Controller and Scheduler process the VC QoS parameters to schedule outputs. Each VC has its own queue and associated priority level. The queues are logical rather than physical and can therefore be dynamically sized. This approach allows only the active VCs to actually use memory resources. The queues also have programmable thresholds to maximize memory usage.

References

1. Anderson, Jon, Patrice Lamy, Laurent Hue, and Luc Le Beller, "Operations standards for global ATM networks: network element view," IEEE Communications, December 1996, pg 72.

| EDN Access | Feedback | Table of Contents |