From causes to effect

Two characteristics distinguish a good designer from a really good engineer. The first is the ability to produce a design despite numerous and tight constraints, such as those on power, size, cost, or compatibility. The second is the ability to debug. Engineers develop both skills through experience, mentoring, and mistakes.

Debugging is dirty, unpleasant work, and it doesn't hold the same kind of glamour as design itself in an engineer's life. Yet every project has a debugging process, and debugging thoroughness and speed make the difference between a properly released product and one that is late, provokes long-term headaches, and reflects poorly on your design team.

One of the reasons that debugging is difficult is that you are usually working with multiple unknowns, overlapping problems, uncertain hardware and software, and test and data-collection challenges. Unlike the popular press, which blithely and effortlessly announces "The stock market was up (or down) today because of XYZ," we know that the real world doesn't have such a simple, well-defined, or even knowable mapping and link between a single problem and its tangible appearance.

Debugging requires a special mindset. Unlike some aspects of design, in which the amount of progress your make is roughly proportional to the time you invest, debugging often has periods of no apparent progress despite your hard work. You have to be able to think simultaneously about the big picture, the small picture, possibly incorrect assumptions, things that may obscure or hide what's really happening, tests and data that may reveal or mislead, and more.

Then there are those bugs that appear only when several factors come together in just the right way or are hard to consistently re-create. Many times, one bug sets the stage for a second bug, which in turn becomes the problem you see, so you have to drill down to find the real source. You spend lots of time devising tests that you hope will let you separate and independently verify what is happening versus what you think is happening. Often, the tests you run yield contradictory or inexplicable results.

Yet somehow, you peel back the layers of confusion—which act as a bodyguard for problems, misconceptions, and misunderstandings, preventing you from discovering the bug—and a real, shippable product emerges. Of course, you also have to fix the bug, and, in many cases, that's almost as difficult as finding it in the first place—but that's another story.

So here's a challenge to EDN readers: Tell us about some of the most challenging, frustrating bugs you've had to root out and why they were so. We don't mean bugs that were caused by basic foolishness, such as when you spent a day racking your brain and observing confusing data on a circuit that you forgot to power up. (Who, me? Impossible!) And we don't mean the ones due to metric versus nonmetric misunderstanding. We want to hear about bugs that took hard work and insight to understand and reach their inner core. Tell us how, while you unraveled the intertwined circumstances that made the bug your nemesis and showed it who was the better player of the two of you, you learned lessons that helped in your future efforts and made you a better, more savvy real-world engineer.

If we get enough reader examples, we'll run them as a few pages in print or the Web version of EDN. This way, you can both learn from others and see the commonality as well as the differences of the engineering experience. You can e-mail them directly to me at bschweber@edn.com.