Embedded Processing

In a previous post I proposed that the differences between developing for embedded versus application-level systems are poorly distinguished and that, as an industry, we are paying an increasingly visible price for this lack of clarity. I personally came to understand some of the differences between the two types of development when I transitioned into the embedded world many years ago.

I did not think much of those differences as a public discussion until someone asked me why more embedded developers are not attending multicore conferences. That question made me realize that there are at least two very different perspectives on what multicore or multiprocessor means, and the typical embedded care-abouts are not the same as what I have seen as the major focus of contemporary muticore conferences (topic for the future).

I was moved closer to pointing out that there are different needs when I found independent observations noted that even after decades of opportunity, college curriculums still are not adequately preparing graduates for the embedded-design community. The tipping point for me though, was that across several years of embedded-designer surveys, "better debugging tools" is still the No. 1 concern among those surveyed, and no one seems to understand why. I smell a disconnect.

I will share my experience of transitioning into the embedded world as a starting point to making the needs of the embedded-software developer more visible. One of my goals is to get you to think back on how you entered the embedded-design world and what transition you had to go through. I am working on a feature article for Sept. 4 that I hope will be able to point out what is going well and what needs work for system-debugging tools to meet the needs of embedded designers. I ask that you post here or email me with any insights you can share. This is an opportunity to take those survey results and add another layer of information to help the toolmakers help you.

Making the transition to embedded design required profound changes in how I viewed software and hardware. Prior to making the transition, I had 10 years of experience developing software for many types of platforms—from time-share mainframes down to 8-bit computers. I developed assemblers, compilers, operating systems, interpreters, database engines, automation tools, and even computer games. In hindsight, a common description for all of this software is that it is all application-level or end-product software—software that the user is aware of and works with in some direct or indirect interface.

One of the first things I had to internalize when transitioning to embedded design was that my software was invisible in the end system. The end user had no idea it was there—nor did they ever need to. I believe this an essential component of what makes something an embedded system. This had a significant impact on how I defined my worth and my ability to tell people what I did for a living. I laughingly adopted the philosophy of "You know you're an embedded designer when you have to oversimplify your job description for 'normal' people". I found myself just telling people I worked on the Space Shuttle or aircraft because it was too frustrating to try to explain the invisible portion of the system that I actually worked on in those types of systems.

Working on the invisible portion of the system has a profound consequence on design constraints because it is difficult to extract a premium from an end customer for something they cannot directly see, but that is required for the product to do what it does. I believe this is a fundamental reason why embedded systems are built as cheaply as possible. Cheap and small implementations do not make it embedded—rather, the fact that you cannot extract extra revenue out of it drives the need to do it cheaply.

Another immediate difference was that I had to learn about real-world errors. In the nonembedded world, errors were pretty straightforward—you wrote code that queried an API or a register for a logical or discrete indication of the success of an operation. For many embedded designs, I had to learn about uncertainty, ambiguity, hysteresis, nonlinearity, and how to apply many types of filtering to address these design challenges. In closed-loop control systems, correct and incorrect operation is not always easy or obvious to discern.

As an example, I spent the better part of a year in the desert doing prototype research for autonomous vehicles. When we ran a test, it often took an additional week or more of intensive data analysis in the lab to determine if the vehicle behaved correctly for the proper reasons. It was not sufficient just to observe the overall system behavior because in the production version, there would be no human intervention available to help the system resolve ambiguities and uncertainties. The tool of choice for that data analysis was a spreadsheet program coupled with homegrown system-level software simulators.

I believe this experience begins to speak to a possible reason why embedded designers keep asking for better debugging tools. While a better and faster software simulator of the target processor is a nice thing to have, it is wholly insufficient to capture and visualize the complex relationships in a system where outputs affect inputs, such as in closed-loop control systems. Because we had no such tools, other than the ones we built for ourselves, we tested with iterative approximations using open-loop data, and if we were lucky enough to have the budget and time, we got to build hardware-in-the-loop simulations. I expect that this situation is still prevalent today.

I suspect "better debugging tools" means not only better visibility into the processor resources, but better visibility and visualization of the entire system, especially real-world interfaces that experience all sorts of physical anomalies such as drift, hysteresis, stuck values, and nonlinear responses. This leads me to perhaps the most profound shift of my transition into embedded design.

When I first started doing embedded design work, we invariably ran into design issues. Even though the software I was responsible for (usually) did exactly what the specification called for it to do, almost every single design problem was labeled as a software bug. At first this irked me to no end because the source of most of the problems was most definitely hardware in nature. By the end of my transition, I became a full convert to the following philosophy: If the software can detect, compensate, avoid, or correct an anomalous condition in the system, it is, by definition, a software problem—regardless of the root cause. In the long run, for most classes of problems, it is much cheaper to fix it in the software than in the hardware.

This approach makes the task of debugging software messy because it requires cross-discipline skill in the designer or cross-discipline cooperation between the hardware and software developers that is usually not necessary for end-system-level, or application-level software development. Before I entered the embedded-design world, I had never used an in-circuit emulator, a logic probe, or an oscilloscope. These became valuable tools for embedded design, but alas, they were standalone from the software-debugging tools. So in addition to having a cross-discipline problem, debugging embedded designs includes coordinating a mish-mash of tools that do not integrate with each other very well.

I hope that sharing my experiences triggers some thoughts for you that you can share here or in an email to me. The further I follow this line of reasoning, the more convinced I am that we are not directly addressing the needs of embedded developers, but rather, we are indirectly addressing them by directly addressing the needs of application developers. Unfortunately, embedded development includes a whole layer of additional cross-discipline issues that go beyond pure software debugging. Help me to help you by sharing your pet peeves about what is missing (and working) in today's embedded tools. I suspect some patterns will become evident if enough people respond.

As mentioned, I am working on a September article on embedded debugging that I hope can shed light on your issues. I am also planning a hands-on project for a 2009 article that will tackle the challenges in porting software to different processor architectures. I suspect both of these articles will be opportunities to give your observations more weight.