Design Feature: August 1, 1996

Despite pledges of unlimited bandwidth and quality of service (QoS) that would make asynchronous-transfer-mode (ATM) networks the communications technology of the future, these networks have yet to fulfill their potential. The reason? These networks do not make full use of available bandwidth, primarily due to a lack of efficient flow control. Hope is on the horizon, however, because the ATM Forum has implemented an available-bit-rate (ABR) standard, TM4.0, that enables ATM networks to exploit the full available bandwidth by significantly changing flow- and congestion-control algorithms in ATM chips. ABR provides an efficient end-to-end flow-control mechanism that avoids congestion and lost packets in an ATM network.

Meanwhile, network-hardware vendors are pursuing hardware-implementation strategies, including chip partitioning, upgrades, and integrated RISC cores, to handle the still-untested flow-control mechanism in large networks.

In early implementations, cell loss due to congestion at switch bottlenecks and the subsequent retries at sending the data resulted in inefficient bandwidth usage. When a network switch becomes congested with data traffic, it drops data cells from frames, resulting in the retransmission of entire frames, thus wasting bandwidth. The early ATM implementations used "open-loop" flow control, in which sources launch a specified amount of traffic into the network in a specified time interval. This technique relieved ATM sources of complex flow control but pushed congestion problems into the network. Because an effective way to police renegade sources was nonexistent, nonconforming sources could overwhelm the network. In essence, controlling congestion by dropping cells and then retransmitting served not users but equipment manufacturers. Users don't want a network to drop data just to ease circuit design at the source and the switch.

In contrast, the "closed-loop," or "feedback," method, which the TM4.0 standard specifies, pushes flow-control responsibility to the source of the traffic, pushing congestion management to the traffic source and, thus, avoiding cell loss and minimizing long network queues. Feedback from the network allows the source to adjust transmission rate to a level at which users have assurance that the traffic does not become lost deep within the network.

ABR exploits bandwidth

The ABR standard stipulates explicit-rate (ER)-control service, which uses the bandwidth that higher priority traffic is allocated but not using. ER uses fields in the resource-management (RM) cell, which specifies the cell rate a user should employ for transmission over a virtual connection (VC). (VC also stands for virtual circuit and virtual channel; see box, "ATM Forum earns 'most-acronyms award'"). The RM provides a feedback mechanism for the network to notify the source that it must adjust transmission rates to avoid congestion and cell loss.

The physical medium dictates the maximum bandwidth that each loop segment can sustain—25, 155, or 622 Mbps. Ideally, the traffic manager dynamically allocates this bandwidth to the types of service, including ABR, constant bit rate (CBR), unspecified bit rate (UBR), and variable bit rate (VBR), to provide 100% usage. The ABR standard first allocates link bandwidth to the CBR traffic up to peak cell rate (PCR) and ABR up to minimum cell rate (MCR). TM4.0 then gives VBR some capacity, depending on performance parameters, such as minimum, average, and maximum cell-transmission rates and delay variations. Then, ABR uses what it needs up to the PCR. The UBR uses any leftover bandwidth up to the PCR (Figure 1).

The closed-loop link simultaneously sustains many virtual circuits (VCs). A VC provides a connection between end-user points and chooses a fixed route for the entire session and dynamically allocates bandwidth. Call setup uses a process that requires significant bandwidth overhead just to establish VCs. To minimize this overhead, you can maintain ABR VCs with no MCR. The VC is then ready to use when you need it. This technique allows ABR channels to send data when a lull in VBR traffic occurs. Users pre-establish ABR VC setup, thus using available bandwidth to carry less time- and delay-critical traffic.

The ATM Forum accomplishes its goal for ABR—to provide source-to-destination flow control—using an in-band control mechanism that uses cells to distribute flow and congestion information throughout a system. The RM cell circulates end-to-end to deliver flow information to sources (Figure 2). At specified intervals, the forward RM carries information to the destination through the network, which digests it and returns rate-adjustment information to the source. Intermediary switches can modify a few RM cell-congestion information bits to short cut the loop. As soon as intermediary switch congestion develops, the intermediary switch locates and marks an RM cell on the way back to the source (backward RM). This backward RM notifies the source to lower the transmission rate, easing congestion at the intermediary switch. Ideally, ABR uses or sacrifices the bandwidth as VBR relinquishes or demands it. This approach works well when the round-trip delay of the RM cell takes less time than the VBR VC needs to change bandwidth usage.

However, if the feedback mechanism is too slow, the ABR adjustment does not match VBR requirements. This scenario would occur, for example, if the source gets a rate-increase message for ABR just as the VBR has again started using bandwidth (because VBR has not been transmitting for some time). Then, both ABR and VBR are transmitting at full throttle, and congestion may occur. To avoid this situation, keep the feedback loop as short as possible.

Virtual source (VS) and virtual destination (VD) shorten the feedback loop and provide network-isolation boundaries. You can segment an end-to-end VC to localize the feedback to individual links (Figure 3). This approach provides fine-grained flow control but entails complex switch elements. VS/VD also isolates network segments. A typical location for VS/VD would be at the LAN/wide-area-network (WAN) interface at which different traffic characteristics intersect. Segmenting these boundaries typically enhances billing, traffic policing, and maintenance.

A drawback of the feedback method is that it pushes the cost to the end station. That end station can be a user's desktop, which can ill-afford such a cost. VS/VD also adds to policing and management capabilities at segment-isolation boundaries. However, the more segmented a network becomes, the more complex and expensive VS/VD that network contains. Another drawback is that ABR requires a lot of memory to maintain traffic-flow parameters on a per-VC basis. Because it takes a long time to set up ABR service, you should maintain these connections with no MCR. However, this approach does cost memory.

Even though the ATM Forum has finalized the ABR specification, options allow some design freedom. For example, you can base intermediary switch-congestion-feedback choices on RM content or local-cell measurements. This flexibility in implementation lets you experiment with algorithms to find the best one. However, short-term interoperability between intermediary switches and ATM-network edge devices may be a problem. Also, ATM hardware must meet basic performance parameters, such as 155 Mbps, for its silicon to comply with ATM's physical-layer standards. The hardware designer must weigh the cost-vs-performance trade-off. You also must consider memory requirements to support standard performance for certain designs.

Because the new standard pushes flow control to the traffic source, you implement much of the ABR function in the same location as the segmentation-and-reassembly (SAR) function—at the ATM network's edge device (Figure 4). The ATM edge device now must schedule time-varying traffic on a per-VC basis and generate and digest RM cells and the associated rate-computation functions. In addition, maintaining the ABR parameters requires more per-VC and -port memory. This requirement, in turn, forces the memory interface to keep up with the VC. The network edge device has only 2.7 µsec to transmit a cell at 155 Mbps. During that time, the edge-device scheduler must decide which of the possibly thousands of established VCs should next transmit. The scheduler bases this decision on updated VC parameters, previous transmissions, and availability of segmented data.

End-to-end flow control requires intelligence throughout the network. At a minimum, the switch adds logic for RM cell monitoring and congestion updating. The VS/VD requires switches to handle additional ABR source and destination behavior plus scheduling. The switch also requires more logic, scheduling, and memory and a high-speed interface to handle the per-VC ABR parameters and the traffic control at the network nodes.

Strategies for implementation

You can implement the ABR function in hardware by simply adding the hardware state machines to an upgraded version of a current SAR chip, by embedding a RISC processor core into the upgraded version of the SAR, or by partitioning the function as a separate chip that works closely with the SAR or switch chip. The trade-offs among these strategies depend on the application.

A straightforward way to provide the ABR function is to add it to an SAR chip. Since the introduction of SAR chips, manufacturers have recognized possible improvements, which can lead to revising the SAR's silicon. The revision provides an opportunity to open the design and integrate the ABR function. Implementing a hardware state machine is likely the most cost-effective way to achieve maximum performance. Such an implementation limits flexibility, however. Adding the ABR can be as simple as adding the logic to handle source and destination RM functions and to schedule service, along with adding an efficient, high-performance, memory-access method to track ABR parameters. You can perform these revisions using just one chip, as several companies demonstrate with products they will unveil late this year and early in 1997.

For example, Integrated Device Technology (IDT) officials believe that the ABR specification is now stable enough to warrant hardware implementation. This move is largely due to the ATM Forum's implementation of the Anchorage Accord, which ensures that ATM products and services will interoperate and will not require numerous upgrades (see box, "ATM Forum converges on the 'Anchorage Accord'"). IDT's implementation, a low-cost, high-performance part, demonstrates that a low-cost SAR is possible.

Brooktree designed its current SAR product, the Bt8230B, with an ABR traffic-management upgrade path in mind. The Bt8230B device already individually manages and dynamically adjusts transmission rates on a per-VC basis. The upgrade to support ABR will remain pin-for-pin compatible with the current part and work in the system-evaluation kit that Brooktree offers. The company is working on the memory interface to make the Bt8320B upgrade efficient for handling ABR descriptors.

With a new family of SAR and switch-processor chips, IgT shows that you can use hardware implementation across a variety of products. The company offers four application-specific SAR chips that implement ER in a hardware state machine. IgT plans to introduce a quad routing table that also implements the ABR specification. The current routing table supports ABR's explicit forward-congestion-indication (EFCI) algorithm.

Embed a RISC core in the SAR

Rather than employ hardware implementations, some vendors, such as LSI Logic and TranSwitch, embed a RISC core into the same device as the SAR to implement ABR. This strategy assumes that the ATM Forum will modify the ABR specification after field trials. These vendors' products let you change the traffic-control algorithm, even after a product ships, thus adapting to an evolving standard. Users, thus, gain the freedom to experiment with algorithms that the specification leaves open. However, when datapath behavior depends on software intervention, multiple simultaneous scheduling and rate-adjustment calculations will, at some point, overload processing capabilities.

LSI Logic's ATMizer II provides flexible ABR functionality. An embedded RISC processor lets you test various traffic-scheduling techniques at a transmission source. You can change the ATMizer II's fairness and priority algorithm in the field. The ATMizer II uses an embedded MIPS-II compatible, RISC µP core to manage operations and avoid the datapath. The processor enlists hardware accelerators to perform functions such as SAR and scheduling. The ATMizer II also incorporates PCI- and secondary-bus ports to flexibly manage descriptors and payloads in host or local memory. The chip provides 4 kbytes of memory for handling a few active VCs.

TranSwitch embeds a RISC processor in the SARA-II SAR device, which integrates the company's earlier SARA-S and -R. In addition to using the processor to control SAR and ABR functions, SARA-II handles many other ATM functions, including address conversion, routing control, frame-relay internetworking, and operations-and-maintenance (OAM) messaging. OAM is a set of administrative and supervisory actions regarding network-performance monitoring, failure detection, and system protection. TranSwitch works with partners to develop customized software elements to meet other applications.

Partitioning into multiple chips

The third strategy vendors use to implement ABR, partitioning it into multiple chips, combines elements of the other two. This technique, from such vendors as MMC Networks, PMC/Sierra, and Texas Instruments, offers flexibility for quick upgrades and hardware implementation for performance. The partitioning can separate the well-known and -tested functions, such as SAR, from less-tested traffic-management functions. This strategy lets you put ABR logic into a chip that you can later upgrade as the standard evolves. For example, you could begin with an ASIC and turn it into a standard part. This approach lets vendors include software control in the partitioned device and to supply a variety of ASICs. The drawbacks of this approach are that it entails the transmission-line effects of pc boards and increased delays due to drive and receive buffers for signals' getting off and on packages, thus decreasing performance.

Texas Instruments' ThunderCell architecture uses a scheduler chip to work with SAR. This type of partitioning provides traffic-management flexibility performance from a hardware SAR. The scheduler schedules cells for transmission, implements the ABR source and destination behavior, and generates and receives RM cells. You can adapt both the SAR and the scheduler to fit applications. In addition to working with a SAR at the network termination, the scheduler partitioning can operate in a switch that works as a VS/VD.

MMC Networks focuses its silicon efforts on the ATM switch specification. The company uses the partitioning strategy to provide ABR functions to the ATMS2000 switch chip set. Because the switch architecture uses a common memory with a central controller, you need not add ABR logic on a per-port basis. The chip set includes a resource-management controller (RMC), which identifies and manages RM cells, and a RISC or DSP algorithm processor that updates the parameter table and adjusts rates. MMC assists switch OEMs that implement the RMC into ASICs. You can modify the switch algorithm's software control to perform a VS/VD function. MMC and OEMs can eventually implement both parts in standard products.

PMC/Sierra provides another flexible approach with the RCMP-800/200 switch ingress device, which works with a user-specified ASIC as the egress device. The RCMP (Routing Control, Monitoring and Policing) handles address translation, cell appending, cell-rate policing, counting, and OAM functions. It also identifies and tags RM cells for the egress device to manipulate. This approach provides flexible egress-behavior choices, depending on the application requirements of the switch. You can implement the egress function in an ASIC, programmable logic, or a RISC processor. You can use this approach to monitor and modify RM cells or to accommodate a complete VS/VD.

Manufacturers may find some opportunities to improve the complex ABR flow and congestion control, once they implement it in a large-scale network. How drastic these improvements are and how they will change the hardware are anyone's guess. In the best case, only minor flaws will emerge, and the first wave of silicon will lead to widespread adoption of ATM. Alternatively, if the flaws are severe, the first silicon's usefulness could be short-lived, and the outcome depends on how committed vendors are to upgrading their products when necessary. A vendor's investment in support facilities indicates how serious the vendor is about sticking with the product through the next phase of ATM-product adoption.

With this in mind, vendors should devise flexible development tools to adapt to the maturing standard. The sophistication of development tools they offer will be critical to the overall success of customer support. As Table 1 shows, many are offering evaluation or demonstration systems, reference designs, and source code to help with system development.

A few examples stand out. Brooktree offers an evaluation system that provides hardware and software to explore various applications, such as an ATM Ethernet router or ATM desktop multimedia network. The same system will incorporate the ABR upgrade to the current product. IDT plans to follow its $99, 25-Mbps ATM, PCI, network-interface-card reference design with something similar for ABR parts. Additionally, PMC/Sierra is publishing bug lists and software on a World Wide Web page, thus offering a unique form of up-to-date support.

The ATM Forum working group has put much effort into making the ABR specification only as constraining as need be. The degree of freedom allows for creative designs to compete in the market for the best flow-control implementation. Time will tell if creative freedom leads to an efficient infrastructure or an interoperability nightmare.

Acknowledgment

Thanks to Warner Andrews of Brooktree Corp for his comments and insight.

References

1. Quinnell, Richard A, "ATM networking," EDN, March 3, 1994, pg 66.
2. Conference Proceedings, "ATM Year 96," Technology Transfer Institute, (800) 200-4884.
3. "The ATM Glossary," Technology Transfer Institute and The ATM Forum.