Shedding light on embedded debugging

Embedded debugging gets a lot of attention for being a schedule and resource hog, but there may be more to it than just fixing bad software.

By Robert Cravotta, Technical Editor -- EDN, September 4, 2008

For each year of Embedded Systems Design’s annual market survey of embedded-system developers, the single most requested area of improvement for design activities is debugging tools (Reference 1). The percentage of respondents making this request has remained steady at around 32% throughout the three years of the survey. In contrast, the percentage of respondents seeking improved programming tools has dropped from a high of 25% to 10%. Determining why the evolution of modern debugging tools is failing to hit the mark as well as software-programming tools do is worth exploring—especially when the surveys each year also confirm that the testing-and-debugging phase continues to be the one that consumes the largest amount—24%—of the project schedule. The third standout request for improvement in all three years’ surveys is the project-management function of scheduling (see sidebar “COCOMO and evidence-based scheduling”).

One explicit aspect of engineering is to create systems that perform and deliver a practical approach to a problem. Software-programming tools focus on the creation side of software engineering. The survey results suggest that programming tools are on the right path toward improving productivity for the creation of system code to solve problems. But the failure of debugging tools to trend downward as a primary concern alongside programming tools suggests that software-debugging tools are not merely extensions of programming tools that help developers correct mistakes or incorrect coding.

There is a less obvious, almost implicit aspect of engineering—that designers must not only design systems that perform some desirable function, but also eliminate or mitigate undesirable behaviors that may result from uncertainty and variability in the environment, so that the system behaves consistently across a range of operating conditions. This hidden side of engineering potentially provides insight into the challenges facing software-debugging tools, especially for embedded-system designers. In addition to dealing with processor-architecture practical constraints for performance, function, communication, latency, and power consumption, embedded systems often have to deal with real-world interfaces that may exhibit behaviors that are more difficult to predict or characterize completely across the whole range of usage scenarios.

If debugging were only about finding and correcting software-logic errors, an instruction-set simulator in conjunction with cycle-accurate simulators might provide enough visibility into the behavior of embedded systems to support debugging. Such simulators are available for most processor architectures and software-development-tool suites. Simulators can also stop the system and examine any parts of the simulated system. Unfortunately, these types of simulators usually cannot provide complete visibility into the exact interaction and latency of the memory, bus architectures, peripherals, sensors, and actuators. Such fidelity would make the simulators operate even slower than they already do.

System-level simulators, such as the virtual-system-prototype tools from Vast and the Simics virtual platform from Virtutech, have the potential to go beyond the software-execution engine and simulate the interactions of the other parts of the system. These types of development tools can enable software developers to work on a target before the physical hardware is available; they also can assist developers with system integration and testing efforts by supporting system-level fault injection and incremental integration in parallel to other development activities. These types of systems can act as precursors or support to hardware-in-the-loop simulations for high-end, complex systems. These tools support prebuilt systems and assembling a system from prebuilt parts or blocks. With additional tooling, they offer the ability to build new components to integrate into the system.

One hurdle system-level simulators face is cost, which can exceed by many thousands of dollars the price of available processor-centric simulators. It is possible that debugging tools are not following the downward trend in the surveys with programming tools not because they are failing to meet the functional needs of embedded-system developers, but rather because the higher-end debugging tools with the needed functional support still exceed some critical cost threshold. It is interesting to note that this cost threshold is lower than that for hardware-design tools, even though most software-development tools will likely support more of the additional complexity in new systems.

The last decade has seen serious erosion in royalty-based operating-system- and development-tool-license models. The growing success of Linux as an operating system in embedded systems is due largely to the cost advantage of using open-source software. Additionally, many embedded-system tools from silicon providers have adopted the open-source Eclipse platform to host their development tools, which substantially reduces the cost of building these tools, simplifies configuring their tools by the end user, and allows them to focus their engineering effort on the features of the tools rather than the look and feel of the host environment. This observation is not to say that debugging tools have not followed the same downward trend in pricing. At the extreme, many processor vendors offer small evaluation kits that allow developers to experiment with the systems for much less than $100. Indeed, many development kits that cost hundreds of dollars today include features you would find a decade ago only in much costlier tool sets.

As on-chip-debugging circuitry expands on contemporary processor architectures, the industry may continue to see the higher-end-debugging functions finding their way into lower-cost development-tool kits. Many processors, including small, 8-bit processors, contain some proprietary on-chip-debugging circuitry. “The on-chip-debug system is one of the most complex circuits in the chip because it has to non-intrusively interconnect with all of the subsystems,” says Dag Arne Braend, AVR-development-tool director at Atmel. “And it has been difficult to justify incurring the extra cost for this complexity for something that many systems will never use in field devices.”

Real-time trace appears to be the next emerging on-chip-debugging capability moving down the processor hierarchy. Processors using ARM cores with an ETM (embedded-trace macrocell) enable the downloading of instruction and data traces from the processor. The Cortex-M3 core supports a new real-time-trace capability. Trace enables reverse-order instruction execution, which more simulators are supporting. The Green Hills Software Multi Time Machine debugging suite enables developers to swap between on-chip debugging and simulator debugging to support simulated reverse-order execution.

IEEE-ISTO 5001-2003, the Nexus 5001 Forum open industry standard for a global embedded-processor-debugging method, provides a general-purpose interface for the software development and debugging of embedded processors. The initial focus of the Nexus 5001 Forum was automotive power-train applications, but its result has evolved to become a general-purpose standard. The Nexus 5001 Forum membership spans the semiconductor, development-tool, and automotive-electronics industries. As the cost of silicon continues to drop and on-chip-debugging interfaces and functions become standard, processor providers will likely flow high-end on-chip-debugging capabilities from high-end processors to lower-end processors to provide even more on-chip visibility. This step will become necessary to gain design wins.

Design aid

John Lambert, chief executive officer of Virtutech, offers an immediate possible mitigating factor for the higher costs in favor of system-level-simulation tools. “A development team usually acquires our platform to support either the front or the back end of its current project, but once the team uses the tool, it more fully realizes its value and uses it at both ends of the design cycle in future projects,” he says.

This theory exposes another possible disconnect in what embedded-system designers mean when they say they need better debugging tools. A primary function of debugging tools is to provide visibility into the state of the system during runtime. When a system includes complex interfaces, sensors, and actuators, a debugger may be the most expedient tool available for examining these external systems’ functions because accurate signal generation for all of the tightly coupled subsystems is a major challenge. Therefore, one reason you might call them debugging tools instead of design aids is that the capabilities of the tools are unusable until the system-integration phase of a project, when debugging is well under way.

Looking at debugging tools as design aids provides a valuable perspective, especially when working with closed-loop-control systems where the outputs affect the inputs of a system. It can be challenging to verify the behavior of closed-loop-control systems. I have used end-system software and sensor hardware to characterize and validate how an automatic-gain-control algorithm worked with a sensor across a range of expected circumstances (Reference 2). There was no way to simulate this condition or even test for something like it in a pure software-debugging environment because there is no accurate model for the sensor. As a result, I discovered an unknown characteristic of the sensor before it was too late.

A second example involved taking the system-integration-test data and running it through a spreadsheet program to analyze whether the closed-loop-control algorithm was performing properly. Again, no amount of software simulation or design effort could account for actually hooking the equipment together and collecting the data; it was more cost-effective to run the tests during integration than to try to create one-time-accurate models of the prototype system. However, I did use the information I learned about the system behavior to build simulation models for the next-generation-development work.

Another characteristic of debugging embedded systems is that problems are not necessarily solely from software errors. However, because it is much more cost- and schedule-effective to fix most problems in software, designers usually implement a software fix to resolve problems. However, making and recording a fix as a software change belies the amount of cross-discipline cooperation necessary to find, diagnose, and determine an acceptable resolution. In fact, many embedded-system-debugging problems are cross-disciplinary, requiring understanding and expertise of the system hardware, software, and domain-specific constraints. This situation does not pose a problem if the person doing the system-integration testing happens to be an expert in all three of these disciplines, but those types of people are rare indeed.

Because of the cross-disciplinary nature of debugging embedded systems, debugging tools and engineering services have an opportunity to fill a void. Just because a debugging tool provides a specific type of visibility into the system does not mean that developers will use that feature. David Kleidermacher, chief technology officer at Green Hills Software, agrees. “Too many options in tools require a larger learning curve for developers, [which] results in a low adoption rate of many of those features,” he says. Developers tend to use the simplest features due to time pressure, and developers adopt these advanced features more as part of a lesson they learned from previous projects. Partly in response to this realization, Green Hills added an “always-on” trace capability to its Multi Time Machine debugging tool; the trace has become more useful to developers because the default condition costs the developer practically nothing, requires no learning curve to use, and is more effective than explicitly turning on the trace-capture function.

Another challenge facing developers debugging embedded systems is setup and configuration of the testbench. This issue is a low-cost, high-value concern that many tool providers are trying to address in their tool sets. The power supply for many evaluation boards now comes through the USB connection, which greatly simplifies the power setup for using those boards. The interfaces to send commands and collect data between the embedded system and the host system are now simpler to use, and there is talk of using wireless interfaces for debugging systems in the near future. Many development-tool sets provide a preconfigured setup and allow developers to test the board to confirm that it and its tools are working properly in a known circumstance. Reid Tatge, a technical fellow at Texas Instruments, explains that a goal of the company’s development tools is to make embedded development look like the development of “wintel” systems—those based on Intel microprocessors and Microsoft Windows operating systems—to ease the learning curve for designers to use their embedded processors.

Micrium’s µC/Probe aims to make it easier for developers to quickly visualize what is going on in their systems. The tool consists of two pieces of software—one that runs on the host system and performs the data analysis and display and one that acts as a code stub that you load on the target system. The stub manages I/O and resource queries as well as communication with the host system. Although it is an intrusive form of instrumenting the embedded-system code, it provides runtime access to the system without halting your system. The tool is available for around $1000.

National Instruments’ LabView provides strong visualization support and includes a graphical-programming capability. It also includes built-in measurement and analysis functions that it organizes around application domains so that developers can select which capabilities to use. The debugging visualizations are well-suited for data-flow execution models. Prices for tool sets that integrate hardware, software, and domain-specific functions start at around $1000 but can exceed $10,000 if you need to add on a heavy amount of domain-specific tools.

This type of pricing flexibility illustrates a challenge for many tool providers; customers want to pay for only the right level of visibility based on different touch points. For example, some developers target an operating system and do not concern themselves with the details of the underlying processor architecture. Other developers target the instruction-set architecture. Still others worry about lower-level interactions, such as those in the architecture layer, which connects all of the resources in the system.

Even when the tools provide the visibility a designer needs, knowing what to look for is another hurdle because the communication link between the on-chip system and the host system limits the amount and type of captured data that can make it to the host system for real-time and post-runtime analysis. Additionally, processor vendors do not always expose features to the public that they implement on a chip; they may choose to hide those features because the features are experimental and vendors are not ready to provide robust, production-level support for them. Internal tools and expert field-application-engineering services, however, can take advantage of those hidden resources. But it is a far cry from having a tool that an expert can use in a limited situation to making that tool robust and usable across a wide range of scenarios and expertise levels. Unfortunately, engineering services are labor-intensive and expensive—far exceeding the cost threshold of most software-development budgets except in the most dire of troubleshooting scenarios.

To continue evolving, debugging tools require correlated input from hardware, software, and domain experts. In some situations, there is a negative incentive for customers of processor or tool providers to share their hard-learned lessons. If they share the key lessons with tool developers, they may shorten the learning curve necessary for competitors and avoid incurring the scheduling and engineering-expertise cost of learning how to make the embedded system work in a certain way. The experience of field-application engineers can help give tool developers the necessary insight to make debugging tools more useful in these cases.

Embedded systems have for decades been implementing multiprocessor designs, but new opportunities for tightly coupled multicore and multiprocessor systems continue to emerge. Additionally, a growing number of embedded systems are using even more sensors in their systems so that they can adequately take on and manage more complex behaviors. These trends in emerging embedded systems will require debugging tools to deliver even stronger coherency and related visualization features because the interactions between the subsystems will continue to become more complex than ever before.

Talkback

We would love your feedback!

Post a comment

» View All Talkback Threads

• Resource Center
• Browse Categories
• Browse Companies

Featured Company

• Rohde & Schwarz

  

  Rohde & Schwarz is an independent group of companies specializing in electronics. It is a leading supplier of solutions in the fields of test and measurement, broadcasting, radiomonitoring and radiolocation, as well as secure communications... more

• Agilent Technologies

  Agilent Technologies Inc. (NYSE: A) is the world's premier measurement company and a technology leader in communications, electronics, life sciences and chemical analysis... more

---

Most Recent Resources

• Atmel's QTouch Library 2.0
• The Impact of Digital Oscilloscope Blind Time on Your Measurements
• LTE-Advanced Technology Introduction
• FREE Webcast: Demystifying TD-LTE: A separation from LTE FDD
• Near Field Communication (NFC) Technology and Measurements