Software: the Achilles' heel of network processors?

Some network processors support proprietary languages, which may also be specific to a task, such as classification. A vendor may claim that one line of such a language can represent tens of lines of C and hundreds of lines of assembly. These claims may not tell the entire story. Such a language necessarily limits what you can do. For example, how many ways can you configure a single line of a high-level language versus hundreds of lines of assembly? Consider the following analogy: Even with different tenses, suffixes, and prefixes, the amount of detail you can impart with a single word cannot match the variety of expressions you could impart with 100 letters.

The question to ask is how much detail do you need to solve a task? Some packet-processing tasks, for example, are so well-defined and limited in scope that gross instructions may be sufficient and actually simplify the programming task. On the other hand, the simplicity of the language may be an illusion. Each instruction may have so many parameters and options that it takes you longer to write a single line of code than it would to have written the equivalent C code. Some instructions are more like fixed functions with long parameter lists, which are fine for a well-defined task but a barrier if you have a unique implementation for an algorithm and lack the necessary detail to outline it. Interestingly enough, these vendors still talk about how designers will want to optimize their code. Yet, with such limits of language, how creative and clever can your algorithms be? Then again, just how clever do they need to be? The specialized language should implicitly promote the creation of optimized code if it was designed to be intimate with the hardware.

In some respects, it's unclear why the vendors think that programming in assembly is so difficult. Some of the languages and C implementations the vendors tout as panaceas to programming seem less than simple. One purpose of higher level languages is to promote readability and understanding of code. However, the syntax necessary to cram enough parameters into a single line of code to accomplish the work of 100 lines of assembly requires abbreviations and tight spacing to create a wholly unreadable listing. Three lines of code would be easier to read. In some respects, the one-line metric—how many functions you can cram in a single line of code—is a race toward incomprehensibility.

Many network-processor vendors claim that abstracting code—that is, using C instead of microcode—saves a designer from having to understand the underlying hardware architecture. In almost every case, this claim is patently absurd.

Consider the issue of resource contention. In a multiprocessor or multithread architecture, two executing tasks often want to use the same resource, such as the memory, the coprocessor, or the on-chip accelerator. Resolving such contentions requires buffering and arbitration. One task gets priority over the others, and the others have to wait for the resource to free up. The processor can execute code that does not depend on the interaction with the resource during waiting, but most programming is linear, and nonlinear processing can go only so far. Chances are, you can't avoid a stall.

Deterministic environments have a maximum time in which the processor can grant access to a resource. The larger the maximum time, however, the less processing that can occur at this stage. Only a designer with full knowledge of the architecture can determine this maximum time. Increase the complexity of the possible contention, and you also increase the complexity of determining the maximum latency.

No clear answer exists for how multithread/multiprocessor architectures can efficiently resolve resource contention. Several packets in a pipeline, while at different stages of processing, may each require access to a look-up engine. Perhaps an arbitrator buffers and resolves such conflicts. However, without understanding how these stages impact each other, a designer may inadvertently design a system that calls upon a resource in bursts as opposed to spread out over time, thus creating an unnecessary system bottleneck.

There's also the problem of nondeterministic resources such as encryption engines. Loads to an encryption blade are difficult to predict, and a thread could be seriously delayed based on traffic in another part of the system altogether. The worst-case for encryption can be so bad that you may find it better to throw off such packets as exceptions.

A compiler or a hardware arbitrator can resolve resource-contention issues, but how efficiently can they do so? There's no simple benchmark; efficiency depends on the designers' code and coding style.

Another common marketing untruth is the quality of compiled code. If you press them, most vendors admit that you will want to hand-optimize compiled code because the compiler often does not understand critical system issues that affect optimization. Remember, too, that the vendor may be giving the compiler an advantage in that it can easily perform certain tricks, such as loop unrolling, performing inline instructions, executing code out of order, and collapsing contiguous memory accesses into a one access, or collocation, of data. The handwritten code they compare the compiled code against may not have taken advantage of any such optimizations. Also, compiled code sometimes takes advantage of previously hand-coded blocks of code for specific functions. The vendor demo code will follow form enough for the compile to recognize such blocks and substitute the optimized code. However, will the code you write accomplish this?

Several vendors claim that you can write the code for IPv4 (Internet Protocol Version 4) routing in 10 minutes. This claim makes no sense. It takes more than 10 minutes to figure out what you want to design, more than 10 minutes to load the software, more than 10 minutes to learn to use the tools. These claims suggest that the vendor doesn't understand the complexity of these designs. Few designers would use such code anyway; you don't have much of a design if you could do it all in a few minutes or a few days. It pays to carefully probe such vendors' software claims and understand clearly what it will take for you to alter such software so that it can be useful to you.

Another marketing claim that is hard to verify unless you've committed to undertaking a serious evaluation of the hardware is the quality of multiprocessor/multithread compilers. The DSP industry, which has for years used multiprocessor/multithreaded designs, has yet to develop excellent tools for handling debugging, profiling, and optimizing such designs. The network-processing industry and its tools are relatively new. These problems are difficult to solve. Even the control path is starting to go multiprocessor as data rates increase and single processors provide insufficient control management.

Various multiprocessor/multithreaded network processors take different approaches to partitioning processing. Two common methods are dedicating processors/threads to specific pieces of packet processing and having each processor/thread handle the entire processing of a packet. In the first case, breaking processing into pieces adds the complexity of having to stitch the pieces back together and manage the data flow between processors/threads. The internal communication architecture is like a tiny network in itself, with buffers and flow control. Such an architecture also needs to be able to change dynamically, based on actual traffic patterns. How do you account for such factors during compilation?

In the second case, the processors/threads act independently and create resource contention, depending on how different types and sizes of packets happen to fall. Packet processing is an asynchronous operation, and various processors/threads quickly fall out of synch with each other. A compiler may be able to handle contention issues for well-understood operations, but high-speed operations working with a variety of protocols with different levels of servicing and processing become difficult to define as predictable and, hence, understandable. It is a fundamental truth of network traffic that it is neither deterministic nor predictable.

Using multiple chips adds complexity, meaning that you need to share flow-control/back-pressure information and resolve resource contention. A network processor and coprocessor using a shared memory can significantly reduce the effective memory bandwidth. Using multiple chips or even multiple dies in a multichip package adds additional potential system bottlenecks. Cache coherency becomes an issue for chips that share cache: When one component is using a part of memory, another chip may be unable to access that part until the part is free, causing latencies.

Nevertheless, many of the tools vendors offer are leading-edge products. However, they may not necessarily be exactly what you need. Optimization still requires a human perspective that can consider all the factors you can't describe to a compiler. As good as the tools are, this still isn't a task for inexperienced programmers.

An important consideration is how well a development environment handles migration of code and parallel variants. In effect, your first design may be a basic product from which you want to spin off variations for different markets. For example, you can modify port configurations, the number of cards allowed, and other features. How well does the development environment support your need to manage a single base design and migrate changes across all the variations?

Several vendors are trying to target a range of markets from OC-12 to OC-192. You may be able to reuse much of the same code across such diverse product lines if they have a common instruction set. In this case, C has a distinct advantage over assembly or microcode. Maintaining development at an abstract level reduces the need to learn a new architecture and new tools and carries code to next-generation devices and across different markets.

In the upcoming generation of devices, some vendors are adopting new architectures to realize performance advantages. These architectures will result in the loss of the code base you are currently developing. Even if the architecture is much the same, new features will affect your design. For example, an additional acceleration engine could replace some of your optimized code; perhaps it is worth focusing your optimization efforts on blocks of code that won’t receive a hardware boost. Also, your optimization choices will change as the internal structure of the device changes—say, with additional interprocessor communication buses or increased memory access. You also need to keep your coprocessor choices in mind. Forwarding and classification engines are taking on more tasks than just searching tables, freeing up some resources, and necessitating code rewrites. If the architecture will differ substantially, how does the vendor plan to move your code to the new architecture?

In this way, you can be developing code that may actually run on a future generation of the device. In some cases, waiting for the next generation will give you greater performance gains than writing better code. One issue to consider, however, is that network processors tend to advance in fairly large quantum steps: from OC-12, to OC-48, to OC-192, and beyond. It probably makes little financial sense, for example, to use an OC-192 chip for a high-performance OC-48 application. Therefore, a vendor focusing on building an OC-192 part won’t really help your OC-48 design.

Choosing to use a device in your design means developing a close relationship with that device’s vendor. It pays to understand a vendor’s agenda and its impact on this relationship. Before committing yourself, ask the following questions:

• After spending a lot of money on creating a fantastic product, does the vendor have any money left over to support you through your design phases?
• Where do you fall on the list of priorities of a vendor that has 100 design wins?
• If your vendor is a start-up, how long will its venture-capital funding hold up?
• Did the vendor decide whether you would be one of its design wins, or did you decide whether the vendor would get your design win?
• How experienced is the vendor with network processing?
• How well do you trust your vendor with your intellectual property?

Remember that the vendor prequalifies you, just as you prequalify it. Getting a design win involves more than just a promise to purchase a product. The vendor must support you and has enough resources to support only so many customers. Also, the vendor’s reputation is on the line when it accepts a design win. If the prospect of working with you seems too risky, the vendor may back off.

Many vendors are glad to tell you how many design wins they have lined up. If possible, find out how many customers the software vendor has actually shipped its product to. In other words, is the vendor counting only those customers who have committed to purchasing the product or including customers who are still evaluating the software before committing? A world of difference exists between these two types of customers.

Looking at a road map of new parts is great, but consider the status of existing devices. Ask for details about the road map, including the full timeline—from preliminary data sheets to simulation models and from reference designs to samples. See whether the vendor has met previous deadlines and promises.

Some vendors let you purchase protocols and extensions that they promise to deliver later. You may want to make this investment because you could get a lower price now than later, and this early purchase gets you into the loop early enough to drive the development.

Find out what market your vendor is playing in and whether the vendor will treat you as a fringe customer or part of what the vendor considers a key market. Some vendors count on making substantial additional revenue from offering professional design services. If you are not interested in purchasing these services, does the vendor provide the development tools and support you could get from a vendor that would prefer to let you do all your own design? Also, find out whether the vendor is willing to license back from you code that you have ported or added extensions to.

You should also try to figure out your options if a vendor goes belly-up and whether your design investment is protected. Many small vendors want a larger company to purchase them. These sales can have far-reaching effects, such as a blockage in the information flow as the larger company tightens its policies, the loss of attention a single-product company can offer its customers, and the severing of relationships with now competitive partners.

Further, what kind of program has the vendor developed for software development, including for third parties? Look at the program to see what kind of support you can hope to receive from these vendors based on the type of support they receive from the hardware vendor. Also, which third parties are part of the program, and how committed are they? And when a vendor has a relationship with third parties, does the vendor or the third party take responsibility to fix bugs?

Remember, vendors do want you to be successful. After all, they start making money only when your product enters volume production. The question is: How much faith does the vendor have in you to be one of its designs wins to show a profit?

The data plane of network processing includes all wire-speed processing; that is, all the necessary processing to send a packet in the handful of nanoseconds the processor allots each packet. The control plane includes that processing that does not immediately need to send a packet from a port. Such processing includes forwarding table updates and processing exception packets, which are those packets that are too “unique” to pass through the data-plane engine.

The NPF (Network Processor Forum) is now trying to develop standard hardware and software interfaces to abstract processing elements. The software APIs it is developing attempt to abstract network-processor elements or functions to a control processor. One purpose of the APIs is to let you create application and middleware code that will allow control processors and network processors to interface with each other. The API hides the hardware particulars of a task. For example, the NPF will in the second quarter vote on the IPv4 (Internet Protocol Version 4) API. This version defines such functions as “add route” or “delete route.” For those applications that need to implement beyond plain-vanilla IPv4, designers can drill to a lower level to such functions as “modify table entry.” These still-abstract functions provide access to specialized and unique features of various network processors. An example of an application-based API is classification; that is, the application understands just that certain functions are available, not whether the processor implements classification in software, an acceleration engine, or a coprocessor.

One promised result of this API work is to ease the challenge third-party vendors face in designing application software. Currently, if a vendor wants to write an IPv6 (Internet Protocol Version 6) stack, it has to write code to an API that is unique to the network processor in use. Thus, a vendor’s resources for porting the software to other network processors limits the ability of the vendor to leverage software to a larger market. With the NPF API, the vendor should have to write the code only once to the API. To run the code on a network processor, the vendor needs to write a “shim” layer that interfaces the API to the architecture and the API of the network processor. This approach promises to open the market for third parties, making it worthwhile for them to write APIs because they’ll be able to sell enough of them to make it worthwhile and to enhance the software offering of any network processor with a shim for the API.

Many third-party vendors have developed their own shims to interface their APIs with specific network-processor APIs. By doing so, they can use an API that exposes details of both the application and the network processor to optimize the interface between the two. General interfaces are typically less efficient and more cumbersome than specific interfaces are. Thus, some vendors may continue to use their own architectures and support the NPF APIs to maintain a competitive edge. For example, it makes no difference to the application how you implement classification. However, once the application is running, the implementation can have a tremendous impact on system-level performance, such as how fast it completes a table update.

The NPF APIs focus on the control plane. With the variety of network-processor architectures and the vast differences in coding the network-processor engines themselves, it’s unclear just how useful third-party network-processor software is. In most cases, less than a few thousand lines of data-plane code run on the network processor itself. How much a third party would have to charge for such code is unclear. Additionally, this area is still one in which network-processor vendors and designers add significant design value and customization.

Candidates for future work within the NPF include multicast IPv4, MPLS, IPv6, and other protocols, which the NPF membership will determine. The NPF is also working on streaming interfaces, in which all data passes through each coprocessor or component, and look-aside streaming, in which only some data passes to the coprocessor and back into the stream. Many niches exist for vendors to develop coprocessors for encryption, classification, and traffic management. The goal is to create common interfaces to solve the current problem that coprocessors work with only one or a few network processors without using an FPGA bridge. The creation of this approach would allow designers to more easily mix and match devices.

At press time, the NPF projected that it would ratify a streaming interface by March. The first part of the look-aside interface, LA1, was also due in March. Devices that logically look like memory, including memory, content-addressable memory, and look-up engines, drove the interface. LA2, which does not have a target introduction date, will serve for devices with a request/response architecture. With such interfaces, you will be able to interface a network processor with a coprocessor without using complex software or an FPGA bridge. However, APIs generalize problems as they abstract them, creating inefficiencies and blocking access to nonstandard features/acceleration engines on various chips. Thus, vendors will continue to offer their own APIs for high-performance applications that cannot suffer the inefficiencies. Also note that defining the control and data planes becomes less critical for lower speeds, and, in some cases, one chip can handle the entire processing task, possibly eliminating layers of abstraction. In any case, vendors will probably continue to support proprietary APIs to preserve the code base of existing designs. However, note which API the software truly supports. In other words, did the vendor rebuild the software architecture for optimization or merely add an extra shim layer to bridge the proprietary API with the standard API, thus adding another hidden layer of inefficiency?

The NPF is also working on various benchmarking standards. The need for benchmarks is undeniable. Yet even benchmarking from a black-box perspective with set traffic streams fails to account for the specialized functions of each combination of network-processing components.

Many vendors believe in the design that the NPF proposes. However, even if the NPF cannot reach consensus on low-level design issues, these vendors say, much good for the industry will have come from it. Many others feel that the act of standardization is absurd in a market in which every architecture is unique. In any case, it may be some time before you see wide implementation of the work of the NPF. Many vendors have spent lots of money and need to see some return on that investment before they make another upgrade. Pockets are only so deep, and many start-ups are discovering that it’s expensive to support a customer after the design win.

Profilers monitor the performance of a system over time. At a basic level, a profiler shows you how close your design is to processing packets within wire-speed limitations based on different traffic conditions. You need to know how much and how efficiently you’re using the overhead you have for additional processing and services. The profiler can also tell you where your system hits the most congestion. The challenge is realizing when you’ve reached a point at which you approach the maximum capacity of the hardware.

Profilers work by capturing data during simulation. When you or a breakpoint stop the simulation, the profiler can then sort through all the information it has captured. You should be able to search this data to see when a register was within a range or pull up a graph comparing values over time.

Some profilers allow you to track the path of, changes in, and duration of operations on a packet as you worked on it. You can track the number of packets crossing an interface. You can track the time it takes to process individual packets over time, viewing the minimum and maximum loads on the system and viewing events or functions. You can also monitor table management and confirm that the system efficiently maintains the entries. You can monitor all of these functions by picking the appropriate registers to track, but a more useful profiler tracks such measurements on a functional level by, for example, letting you click on the “track-packets” option.

For multiprocessor/multithread architectures, you should be able to track the idle time the system spent within each processor/thread so that you can determine how to get that processor/thread to do more. You can also determine worst-case values for pipeline architectures or for processor/thread loading. The ability to track the use of both internal and external resources, such as memory or look-up engines, is also important. However, for this information to be most useful, you need to be able to easily reallocate processor/thread loading so you can quickly test new balances.

Profilers seriously impact simulation speed. A good profiler lets you define what data to capture and how often—that is, how many cycles—to capture it. By limiting the data you capture, you can significantly increase simulation performance. This increase reduces the simulation time it takes to ferret out the results of corner cases or exceptions or to see how fast sustained bursts bring the system to its knees.

Reference designs are a checkbox item on every network-processor vendor’s marketing sheet. When a vendor offers a reference design, it has built a board with a device, written some software, and can get the little LED on the corner of the board to blink on and off. Some reference designs do no more than prove that the silicon won’t explode when you apply current to it. Most claim to show you how much headroom is left beyond a basic application. Others, the vendors claim, are swell enough to put a logo on and sell as-is. In any case, silicon vendors generally focus on selling hardware, not software.

Will a reference design help you ship your product? In other words, vendors often use the term “reference design” for “demo” or “tutorial,” meaning that you have work to do even if you use the reference design as a baseline or foundation. The question is: How much?

In some industries, a general application reference platform can go a long way. In network processing, however, the varied and myriad architectures prevent general reference designs from having as much value as they might otherwise. A general network-processor design, for example, lets you add a variety of coprocessors. However, such a design is probably too general to take full advantage of the unique capabilities of the individual coprocessors, meaning that you must customize the code. For example, a reference design probably has different port configurations from the configuration you want for your product. However, you may be unable to scale the code in this fashion; after all, the vendor may have written it as a one-shot deal to prove the silicon. Also, the reference design may offer only basic functions without giving you enough pieces to make a shippable product. The claims that a vendor can supply 90% of the software don’t tell you much.

How do vendors determine what reference designs to create and support? Many talk of a push-and-pull dynamic. They ask their customers what they are designing and watch market demand so that they can guess the location of the sweet spot. A promise of a purchase order also helps.

Look at how many reference designs and for how many applications the vendor offers. If programming the network processor or coprocessor is as easy as the vendors claim, then why don’t they offer more software? Examine the reference design to see how flexible the code is and how straightforward the process will be for you to carry their code over to a design of your own. Challenge the vendor to prove the extensibility of the code to you by asking for a simple change in the code. See what such a change entails. Also check what language the vendor wrote the reference code in. The vendor may tell you that you can write code in C but then write code for the reference design in assembly or microcode.

If you wait until silicon is ready before you start evaluating a chip, you may have waited too long. Network-processor vendors typically offer simulation models of their chips months before samples of silicon will be available. In some cases, vendors announce chips a year before projected sampling. As a result, you run the double-edged-sword relationship of an alpha partner. On the one hand, you can influence the vendor and offer feedback that will facilitate easier design for you later. On the other hand, the simulation models the vendor provides you with will most certainly change as the vendor works out bugs and architectural kinks. Think about asking to see the revision specs for the simulation models. These specs might reveal something about how many and what kind of major architectural changes or minor tweaks the models have gone through, as well as how the vendor facilitated working these revisions into the customer design cycle.

Your design cycle will cross the simulation/silicon line, so many vendors design the rest of the tool chain such that these tools can’t tell the difference between simulated chip data and actual chip data. In other words, you can use the same code, debugger, and profiler that you will when working with actual silicon. A configuration tool that handles all the details between simulation and emulation is especially nice. The only real difference you should see, then, between a simulation and an emulation using a chip on an evaluation or prototype board is the speed of analysis. Because emulation employs hardware, it executes magnitudes of order more instructions per second than simulation does. On the other hand, simulators give you much greater access to internal data, code, and registers. However, you don’t have much choice but to use the simulator until vendors offer either a physical sample or Verilog that you can download into an FPGA. You don’t have much choice but to use the simulator.

Simulation speeds vary, depending on how you simulate the system. Verilog simulation runs at hundreds of cycles per second. Modular Verilog, in which you simulate only subsystems of the system in Verilog, can run at thousands of cycles per second. Full C-model simulation runs at tens of thousands of cycles per second. The numbers vendors quote, however, are typically higher than those you’ll see in reality. As soon as you simulate on a system level, say by adding a look-up coprocessor or connecting to a simulation/emulation of the control plane, you significantly reduce performance.

An important question to ask about the simulator is what the vendor means by a “cycle.” Some simulators are instruction-accurate, versus cycle/pipeline-accurate, meaning that the simulator may not simulate system bottlenecks, such as resource blocking, cache flushes, contentious memory access, and so on. Simulating bottlenecks gives you a more accurate picture of the system but takes longer. For example, simulating a 333-MHz network processor even at 10,000 cycles/sec requires 9.25 hours for one second of traffic. How long will it take for a corner case to occur or a buffer to overflow?

You can take many approaches to speeding simulation. For corner cases, create data and forwarding tables to test extremes. You can also reduce the size or prepack overflow buffers. Disable breakpoints and turn off those parts of the trace capture that you don’t need to reduce non-code-simulation processing. Adjust the sample rate of the trace from every 10 to 100 to 1000 cycles. Find out whether the vendor has additional hardware to accelerate portions of a simulation. Also, run no other applications, including sending or receiving e-mail, while running a simulation.

Many real-world stresses, such as long, sustained burst of packets, are difficult to profile in a simulation environment. These tasks require the use of a hardware emulation system. Simulation works well for testing basic functions and system efficiency, but you still need to test the extreme but expected cases to see whether the configuration will fall apart under real-world stress.

Another important aspect of simulators is their extensibility. Does the simulator support a model of the coprocessor you plan to use, and does it support multiple models or multiple coprocessors? You can limit simulation to a component level, but one of the significant challenges of designing systems with such components is getting them to work together efficiently. What will porting a model yourself entail? Also consider how monitoring the simulation environment takes place. It can be useful to have hooks into the simulator, allowing you or a third-party vendor to extend the simulation environment, by either adding analysis tools or mining simulation data that the vendor has not yet made available.

Third-party network-processor-software vendors face a challenge in developing software for a market in which almost every design has some “special” functions. Code that manages the data plane, usually running on the network processor, is often less than a few thousand lines, and developers ply it with unique implementations depending on the application, silicon, and design team to differentiate their products. Thus, no large market exists for third parties targeting niche applications on the data plane.

On the control plane, however, some software vendors claim to offer complete software for complex applications. But are there enough design wins to keep these companies in business? With a modular approach, they can reuse large amounts of their code base across several niche applications. By also offering design services, they hope to customize code for you.

Other vendors recognize that each customer has an interpretation of its application that is unique to its architecture and target market. In other words, each customer has its own twist to add. These vendors offer prepackaged applications with the caveat that the code will get you, say, 60% of the way there. They avoid uncommon or nonstandard functions, because they expect you to make changes at these points to yield system-level optimizations.

Many software vendors have developed their own “standard” APIs for their code and are working toward industry adoption of these APIs. The question is: Should this adoption be a criterion? The API may become the standard for the industry, but the vendor may then pursue its now-proprietary API and shun the accepted standard.

Few vendors are willing to abandon their proprietary APIs as groups, such as the Network Processor Forum, introduce standards. Most will support both so that their customers’ code base remains intact. Regardless of whether a vendor runs its API on the standard API or runs the standard API on its own API, you end up with an extra “shim” of code that may cause a slight loss of performance or functions.

Third-party vendors want you to think about your added value and hope that it isn’t in the code they have to offer you. To some degree, they are right: Off-the-shelf code is becoming a commodity. In this light, is such code a safe choice to focus on as “special”?