Feature
Debugging FPGA designs may be harder than you expect
Bugs can originate at every stage in the FPGA design flow; debugging success depends on using the right tools and methods.
By Chris Schalick, GateRocket -- EDN, 10/22/2009
|
In the not-so-distant past, the question used to be, “Can you do that task in an FPGA?” With the advent of modern FPGA devices, however, the question has become, “Why wouldn’t you use an FPGA?” Modern FPGAs’ complexity rivals that of ASICs. The chips contain hundreds of thousands of flip-flops, multimegabits of RAM, thousands of DSP slices, multiple soft or hard processor cores, multiple SERDES (serializer/deserializer) channels, and more. Integration of FPGA designs no longer takes place with one hardware engineer working in relative isolation and designing from the ground up. Instead, modern projects require team-based design, substantial quantities of third-party IP (intellectual-property) cores, and complex hardware/software interactions. The ability to debug such an FPGA design determines time-to-market—and time-to-profit—success. However, debugging one of today’s multimillion-gate FPGA designs is not a trivial task.
Consider a generic, high-level view of the conventional FPGA design flow (Figure 1). As an engineer, you could draw this figure in various ways—adding and rearranging the various blocks and arguing about minutiae. Newcomers who come to the design with faith in various aspects of the FPGA design flow soon discover how unfounded this faith can be. For example, it’s common in FPGA IP to use two representations—one for high-level simulation and the other for the actual implementation. Many design-and-verification engineers assume that these two representations are guaranteed functionally identical. But sometimes they are not. It is also reasonable to assume that synthesis and place-and-route tools are robust and will not introduce errors in the design. But they sometimes do. When you move to FPGAs from ASICs, you may quickly discover that bugs can appear at any stage of the FPGA design flow.
Most techies occasionally use design-related forums and sites. On these sites, real-world engineers post some cases spanning the FPGA design flow. The following examples illustrate the types of issues FPGA designers can encounter. The user, vendor, and tool names in this article are not the actual names of the participants. These examples involve a number of design issues: the quality of RTL (register-transfer-level) code, the quality of the IP, the quality of results from the synthesis engine, and the quality of results from the place-and-route engines.
Case 1: RTL qualityIt isn’t surprising to hear that you’re going to get better or worse results depending on how you write your RTL code. Consider an example from opencores.org (Figure 2). This engineer’s well-intentioned desire was to produce a better gate-level implementation netlist. Unfortunately, this code behaves differently in synthesis from the way it behaves in simulation, simply because the simulator ignores the synthesis pragmas.
Here, the pragma tells the synthesis tool that it can make arbitrary decisions about unspecified choices. This statement is contrary to what happens in the simulator. The end result of this difference is that inadvertently writing to an unspecified address actually overwrites a real register. One approach is to use linting tools, but most FPGA designers don’t use them. Linting tools are common in ASIC environments, but design engineers who gained their experience in a traditional FPGA shop tend to have no ASIC tools lying around. Managers don’t want to spend money buying such tools, designers don’t want to spend time learning to use them, and so registers are overwritten.
Case 2: IP qualityIn the case of ASIC designs, third-party IP vendors typically deliver their cores in the form of RTL, which may be encrypted, obfuscated, or unencrypted. Therefore, the same RTL representations of the IP blocks that you use during initial software simulation are the representations that you subsequently synthesize, place, and route along with the rest of the design. This commonality provides a high level of confidence that the RTL and gate-level representations are functionally equivalent.
The situation is different with FPGAs. IP vendors often supply two models—one at a high level of abstraction and one at the gate level. So the IP models you use for simulation may differ in significant ways from the corresponding models the place-and-route software uses. Often, the high-level simulation model contains behavioral constructs to make the software simulations run faster. Synthesis tools cannot handle these constructs, however. Moreover, subtle differences often exist between the behavioral- and gate-level representations that manifest themselves only when you deploy the FPGA design in its target system.
Consider an example from an FPGA vendor’s user-forum site (Figure 3). Running the simulation on the gate-level netlist from the IP vendor shows different results from those of the behavioral model from the same vendor. In this case, the gate-level representation was correct and the behavioral model wasn’t. In many cases, it’s the other way around.
Case 3: Synthesis problemsPeople believe that synthesis tools are more robust than they are. Even though some synthesis tools have been around for years, users are still logging bugs against them. Consider a case from fpga-faq.com (Figure 4). This example involves a bizarre set of circumstances. First, the synthesis tool makes an odd choice. Then, critical warnings come up that are tremendously informative, but a deluge of messages buries them, causing the designer to overlook them.
In this case, the tool acknowledges that it’s making an odd choice and is telling the user that it’s worried. In many cases, the tool thinks that it’s doing the right thing and doesn’t say a word. The problem stems from aggressive optimizations on the tool’s part. Today’s designs are large and their corresponding synthesis can take a long time, so synthesis-engine developers take short cuts. Whenever the synthesis-engine developer cuts a corner, however, he must account for an enormous set of possible conditions. Assumptions can turn into errors.
Case 4: Placement-and-routing problemsAs a final example, consider a power-up initialization problem from fpga-faq.com (Figure 5). In this case, the place-and-route tool doesn’t know what to do; it decides that a register must have some state, so it makes an arbitrary—and unfortunate—choice that causes the silicon to do something odd and unexpected. The engineer is coming up with solutions but is unsure whether they will work because visibility into the root cause is so low.
|
These examples are teasers; you occasionally hear more complex stories. In one such story, a design team lost weeks going back and forth with IP vendors trying to work out a problem. In one case, a place-and-route tool optimized some core functions out of the vendor’s own IP. Nevertheless, you can do astounding things with a modern high-end FPGA and associated design tools. You can—in one day and on your desktop—synthesize a design that’s equivalent in complexity to a Pentium CPU, run place-and-route tools, and generate the corresponding FPGA-configuration file. Just five years ago in the ASIC world, you needed a design center for one to three months to come up with a workable place-and-route strategy.
Despite these advantages, the downside is that errors can creep into the design at every stage in the process, and these errors often don’t manifest themselves until the design is in a real board in the laboratory, by which time they can be difficult and time-consuming to detect, isolate, identify, and resolve. Software simulation provides a high level of visibility into the design, but it runs relatively slowly even at the RTL. By comparison, running the design in real FPGA hardware, such as a development or prototype board, provides hardware speeds, so any problems quickly manifest themselves by crashing and burning. Due to lack of visibility into the chip, it can be difficult to determine what’s gone wrong. Is there an error with the RTL source code that causes it to behave differently with the simulator from the way it behaves with the synthesis engine? Are there functional differences between one or more of your behavioral third-party IP blocks and their gate-level equivalents? Is it perhaps a case in which your RTL and IP are functionally correct, but the synthesis engine, place-and-route engine, or both have introduced errors? Or do you have a mixture of all of these situations?
Answering all these questions requires a different approach. Software simulations offer visibility into the design but are slow. Verifying the design in hardware is fast but provides limited visibility into the internals of the system. So the answer is to create an environment in which software and hardware representations of the design can coexist. In other words, each functional block in the design should be able to reside in the software domain, the hardware realm, or both.
For example, consider a design comprising, say, 100 functional blocks. Some of these blocks could be your own internally developed, proven IP from previous projects; some could be IP from third-party vendors; and some would be your new “secret sauce” to differentiate your design from all others. Now, suppose that you could immediately move the gate-level representations of any known-good blocks, such as your internally developed IP and trusted third-party IP cores, for example, into the same type of FPGA you are targeting for your real-world design. Also assume that you could now verify these blocks in conjunction with the rest of the design running in your software simulator of choice. Right from the start, you have dramatically speeded your verification runs.
Now, as you verify each of the new blocks at the RTL or behavioral level in the context of the full-chip design, you could move the synthesized/gate-level equivalent of each verified block into the physical FPGA. As soon as any problems manifest themselves, you could repeat the verification run with the RTL version of the suspicious block, resident in the simulation world, running in parallel with the gate-level version in the physical FPGA. A software application could—on the fly—compare the signals from the peripheries of these blocks, along with any designated signals inside the blocks. This combination of conventional simulation with physical hardware and an appropriate debugging environment would make it possible to detect, isolate, identify, and resolve bugs, no matter where they originated in the design flow.
Leading FPGA vendors are continuing to innovate and produce larger, more complex chips that are quickly landing in applications that would have been ASIC-only just a few years ago. This fact means that lots of designers are on a steep FPGA-design learning curve, probably with limited resources and time. For FPGA vendors and tool suppliers, this situation presents a great opportunity to simplify and streamline development, thereby converting occasional users into lifelong customers.
| Author Information |
Chris Schalick is the founder and chief technology officer of GateRocket, which specializes in FPGA-design and -debugging technology. He has 20 years of experience in modular-system design and behavioral-system modeling for consumer and industrial equipment. Schalick has held senior engineering positions for raster-imaging, data-networking, and semiconductor-test-equipment companies and was instrumental in delivering products to market at Teradyne, Tenor Networks, Packet Engines, and Cabletron Systems. He holds a bachelor’s degree in electrical engineering from the Massachusetts Institute of Technology (Cambridge, MA). |















Chris Schalick is the founder and chief technology officer of GateRocket, which specializes in FPGA-design and -debugging technology. He has 20 years of experience in modular-system design and behavioral-system modeling for consumer and industrial equipment. Schalick has held senior engineering positions for raster-imaging, data-networking, and semiconductor-test-equipment companies and was instrumental in delivering products to market at Teradyne, Tenor Networks, Packet Engines, and Cabletron Systems. He holds a bachelor’s degree in electrical engineering from the Massachusetts Institute of Technology (Cambridge, MA).
