Save time and money with smart debugging method

Tools alone don't solve the problem; you must couple good tools with an effective debugging strategy.

Ron Choi, Eve -- EDN, April 22, 2010

The design of modern SOCs (systems on chips) requires increasingly thorough verification to ensure that manufacturers don't spend millions on mask revisions when first silicon has problems. A widely bandied number suggests that designers spend 70% of their time in verification. Companies spend big money purchasing verification tools; savings in silicon-spin costs mitigate the cost of those tools. That is, the avoidance of other costs reduces the effective cost of ownership of the tool that generated the savings.

Even though verification takes the lion's share of the design time, it's generally an activity that designers can plan and, as long as things go well, is more or less predictable. But debugging is an activity you can't plan in detail because you can't plan for an unexpected bug when you don't know of its location and nature. The amount of time it takes to debug a problem adds directly to the cost of the project. Conversely, any expenditures on tools or other means of reducing debugging efforts reduce the cost of ownership of those tools. Tools alone don't solve the problem, however. You must couple good tools with an effective debugging strategy to reap the full potential of the tools.

Emulators provide an effective means of reducing debugging time by allowing faster execution of a target design-especially one that's running embedded software-than is possible through simulation. Today's designs are enormous and complex, however. Perversely, the better job a designer has done, the more likely it will be that any bugs reside deep in unsuspected corners of the design and are hard to find. As often as not, you'll discover some pathological case after long execution. It can take billions of clock cycles before you encounter the bug.

Other system considerations can further lengthen the debugging process. Even if the bug occurs relatively early in the normal operation of the system, simple system initialization can take a nontrivial number of clock cycles, making repeated runs expensive. Randomly poking around is ineffective if the bug isn't deterministic; even if you find the bug once, it won't happen in the same way the next time. Poor planning may even necessitate multiple recompilation cycles to provide the necessary observability of the design, further lengthening the debugging process.

Replicating a bug requires deterministically and repeatedly re-executing many clock cycles until you can identify the problem. Therefore, it's essential that a debugging strategy makes the most effective use of those cycles, allowing "debugging convergence" as quickly as possible. Even though debugging seems more chaotic than verification in that you cannot plan for debugging, a modicum of forethought and a disciplined debugging process combine with the right tools to dramatically reduce debugging time and minimize the frightening unknowns of elusive bugs.


Preparation before compilation Depending on the tools at your disposal, you can furnish a DUT (design under test) with triggers and probes that can give you access to and control over the inner workings of the design. But indiscriminately adding debugging constructs into the DUT in the hope that you'll close all means of escape for any bug will quickly chew up logic, memory, and pins. You must strike a balance between specificity and flexibility.

Triggers monitor activity for an event and then activate some signals when that event occurs. If the event is a function of any RTL (register-transfer-level) signals in the DUT, then a trigger comprises some arbitrary logic between those signals, creating a static trigger signal. This trigger involves logic that you can't change without recompiling the design. If the event involves comparison of some internal state to a constant value, then you can obtain greater flexibility by creating a register to hold the comparison value and allowing that register to load during debugging. This dynamic trigger allows you to modify the triggering condition without recompiling the design.


Triggers and assertions give you information about when but not why an event occurs. You must probe deeply into the DUT to understand what's going on when the DUT misbehaves, and you have choices and trade-offs. The basic characteristics of a probe are how it accesses the internal nodes and how it stores its data for waveform generation.

The obvious way to probe a DUT is simply to create a signal that taps some internal critical node. The benefit is the ability to observe the DUT at high speed; the cost is in interconnect, meaning that you can't randomly probe everything. You must compile in a fixed connection, meaning that you can't change it without recompiling.

To add more flexibility, given a limited means of extracting probe information, you can identify broad groups of related signals that you may want to probe and then, when debugging, select one of the groups at a given probe output to view without recompiling. If you employ an emulator, the technology underlying the emulator may provide yet another way to access signals. In the case of an FPGA-based emulator, you may have access to a back door that allows interrogation of all of the nodes in the DUT. For example, Xilinx Virtex FPGAs have a read-back feature that provides debugging access to all points of the design, including memory. Because this mechanism uses a built-in FPGA, it requires no FPGA resources to implement and, hence, no compilation. The trade-off is that such read-back has a dramatic impact on clock speed, so you can't use it indiscriminately as a way to capture a bug.

How the probed values are stored also affects execution time and flexibility. The highest-clock-speed option is for the values to be stored in memory modules on the emulator boards themselves. The challenge of using onboard memory modules is that space is limited: You can store only so many signals within a limited trace window. You get greater flexibility by streaming the data off the board and letting the host PC store it on disk. The trade-off is that the bandwidth of the hard disk limits clock speed.

Read more In-Depth Technical Features

Given this selection of triggers, assertions, and probes, the preparation phase then consists of judiciously instrumenting the DUT before verification. The catch is that you want to have the tools available without having them encumber the DUT when you're not debugging. Therefore, you must minimize resource usage and make sure that you experience any impact from debugging execution only when debugging. The ability to enable or disable instrumented tools at runtime allows you to compile as much as possible up front without sacrificing verification time.

The investigation phase The goal of the investigation phase is to narrow down the scope of study to a manageable range. You want to eliminate the vast numbers of cycles that are unrelated to the bug or at least get them out of the way as fast as possible so that you can focus attention where it matters. If you do this investigation properly, this phase not only identifies when the problem happens, but also helps to locate the offending function or module.

Rapidly exercising broad swaths of execution requires abstracted technologies, such as TLM (transaction-level modeling). You don't want to exercise every gate in detail; you want to run quickly to see where things fail. Cycle-accurate transactors provide high performance and reproducibility and allow you to connect the DUT to your test bench and software debugger for analysis at the hardware or the software level.

Well-planned triggers and assertions pay off in this case. They give a high-level indication of when things are working and when they aren't. You can selectively turn them on and off to keep the DUT running at a high speed as you home in on the problem. Because you're not so concerned about detailed waveforms at this point and you are concerned about performance, static and flexible probes on main buses and interfaces are the workhorses for observing behavior. Solid upfront preparation provides a wide selection of probes available to you without recompiling.

The ability to save and restore the state of the DUT can also eliminate long initialization sequences. By saving the state of the system, you can return to that state directly to rerun the system, bypassing initialization. An emulator-runtime environment that provides such access to the detailed state of the system, including memory contents, can be of enormous value in reducing debugging time.

Detailed debugging Once you identify the range in which you need to dig deeper, you enter the detailed debugging phase, in which you examine detailed waveforms of portions of the DUT during a window of operation. TLM technology and state restoration are critical to quickly getting you to the window. Once in the window, static and flexible probes can rapidly generate traces, but dynamic probing can give you further access without recompiling when you need to check signals for which there is no precompiled probe-at the expense of clock speed. By limiting the use of dynamic probes to this phase, most debugging occurs at higher clock rates. Flexible and dynamic probes also let you generate waveforms over a practically unlimited window because they directly store their data on disk, allowing a gradual narrowing of investigation without fussing about a fixed window size.

To measure the impact of a well-planned debugging strategy and advanced debugging tools, compare traditional debugging and emulation techniques with a "smart"-debugging strategy using the ZeBu-XXL emulator on a 50 million-gate design. For a bug occurring after 12 billion cycles, the traditional approach takes slightly more than four hours plus data-transfer time. The smart approach on ZeBu takes slightly less than 30 minutes plus data-transfer time because you can delay the slowest aspects of debugging until the end, making the most efficient use of the emulator clock cycles.

The difference is more dramatic for another bug whose exact timing, sometime after 6 billion cycles, is unknown. Depending on assumptions about how many triggering attempts might be necessary to find the correct window in the traditional case, it takes four to 11 hours to complete traditional debugging and slightly more than 30 minutes for smart debugging on ZeBu.

These results confirm that the judicious use of a broad range of modern verification and debugging capabilities in a carefully planned process on an emulation platform that supports a wide range of tools can have a dramatic impact on debugging time. This method not only helps to reduce time lost to debugging, but also removes some of the chaos and unpredictability from the overall development cycle. The ability to more effectively manage these unknowns ultimately reduces the effective cost of ownership of the emulation tools, increases the revenue potential for the system, and makes for a less stressful experience for designers and managers alike.

---

Author Information

Ron Choi, product-marketing director at Eve (San Jose, CA), has a range of verification experience. He is knowledgeable about analog and digital simulation, hardware-verification languages, layered verification methodologies, emulation and system-level validation, and hardware/software co-verification. Choi has worked in EDA for the past eight years, first as a senior application consultant with Synopsys and then as a senior field-application engineer for Eve.