Jack Ganssle, Embedded-Systems Contributing Editor
Columnist:October 27, 1994
There he sits, the engineering guru of the organization, perhaps respected, but feared by all because of his arcane knowledge. His desk is a foot deep in paper, and the lab bench is a mess of old food containers and smoldering solder drippings. Tools and resistor clippings threaten to short out any test system carelessly placed on the bench. Wires crisscross every square inch of tabletop--scope probes, clip leads, RS-232C cables--all going somewhere, though perhaps no one really knows their destination.
Ask the guru for a piece of paper, and he burrows frantically through the mess. Usually the paper never comes to light because it's lost. Don't worry, he can re-create it for you as soon as he has a chance. The PAL equations that he comes up with are probably right. But if they're not, no problem: He's already de-bugged that circuit twice and is quite the expert.
I'm a reformed lab pig. My 12-step recovery program involved living in tiny places--a VW microbus and many boats--and forced me to become organized to deal with the lack of space. Fortunately, my personal quest for organization spread into the lab when I discovered how much time I saved by putting things away.
Quite simply: Mess and clutter decrease productivity. Those few minutes a day spent putting things away save hours of searching. Sweep the solder drippings off the bench occasionally, and your incidence of catastrophic failures can plunge dramatically.
An organized lab promotes correctness. How many times have you seen engineering changes that never made it into production because someone forgot to write them down? Or, the notation was made on the corner of a napkin that was accidentally used to wipe up a spill.
When starting to debug a new project, remove everything from the bench and sweep it. A quick wipe with a damp cloth removes those accumulated coffee stains. Then, put everything not absolutely needed back on the shelves. This is your chance to remove the clutter, so be relentless.
Be sure you can easily reach the computer's frequently used connectors. If two devices must share an RS-232C port, buy a switch box ($10 from Jameco at (800) 831-4242) and reduce the wear and tear on connectors--and your back.
Don't work with unacceptable power distributions. Too many of us spend half our lives swapping power plugs, so you should consider buying an outlet strip (Techni-Tool (215) 941-2400) sells strips of 15 for $64).
Many years ago, Miles and Beryl Smeaton sailed their aging boat around Cape Horn with expert boat builder John Guzzwell as crew. When the boat flipped in 30-ft seas and the hull cracked open, Guzzwell was shocked to discover that all of the Smeaton's tools were rusty and dull. As water poured in, he carefully sharpened and cleaned the tools before undertaking the repairs that eventually saved their lives.
The moral is to buy good tools and take care of them. You can live with those dikes and needle-nose pliers for weeks on end. Buy cheap stuff and watch your blood pressure skyrocket every time you are unable to clip a lead close to the board. Keep your tools organized--buy a little toolbox to keep them from falling on the floor and getting lost.
How is your soldering equipment? A vacuum desolderer is great for making large-scale changes. However, during prototyping I find it's often easier to hack away at the board, mounting chips on top of chips and using plenty of blue wire.
During the first days (or weeks) of bringing up a new embedded system, I often find myself making lots of little modifications to the system. A hot iron always at hand is critical. After things begin to work, I start testing I/O interfaces by writing low-level drivers and exercising the code, making software and hardware changes as needed in parallel. The code changes much faster than the wiring, so it seems wasteful to keep an iron hot all the time. (JDR (800) 538-5005) sells a neat $30 cordless-soldering iron that heats in seconds, ideal for those infrequent changes.)
Being an immensely stupid person, I require vast quantities of clip leads. Most of my ideas are wrong, so I save a tremendous amount of time by using a clip lead to try a design change and see what happens.
Clip leads have a short lifetime in a development lab. Accidentally connect Vto ground, and the plastic tip melts horribly. I hate it when that happens. We used to send a runner to Radio Shack occasionally to replenish our supply, but we found that "the Shack" couldn't keep up with our needs.
We now occasionally buy 100 clips and have a high-school kid solder up 50 leads. It gives us an infinite supply for a time, and may help a fledgling engineer find his true vocation. (Bring a part-time worker in from your local high school to help maintain the lab. The cost is minuscule, the lab is better off for it, and you show one more kid an alternative to slinging burgers.)
Beware of clip leads in high-frequency applications. Connecting a switching signal to ground with even a short lead often doesn't work. The lead looks like a transmission line and presents a rather high impedance at the frequency of your circuit--so high that it may not have an effect.
An embedded system's hardware is never far from the software. At the very least, the person responsible for designing the hardware should provide low-level software drivers to the software team. It's unfair for a person to dump a barely documented complex system onto a firmware engineer's desk and say, "Now it's your problem."
Make sure your lab area is set up for serious software development! Clearly, your computer must include the properly installed compilers and assemblers needed for the project. The debuggers, make utilities, and other software resources are as important as quality hand tools to quickly and painlessly write, compile, and test the code.
Set up the environment with a make utility so you can compile or assemble without twiddling compiler switches. Only amateurs develop by tediously hand-assembling instructions into the debugger. You should never make patches in the debugger and upload these to a file for later use. Change a source file, so you can alter the comments as required to write drivers that are beautiful to behold and a joy to use.
Develop your code with the environment the firmware folks use so they can immediately integrate your test routines. It drives me crazy to see the lousy planning so endemic in this industry. Too often laziness or stupidity reigns, so one engineer does his work under a particularly obsolete version of a compiler. Meanwhile, the rest of the team must painfully massage the code into something compatible with the version they use. Why work so hard?
Hardware design requires as much software support as does the firmware. PALs, PLDs, and FPGAs let you create much of the hardware design late in the game. Make sure your bench is set up with all of the tools you need to edit and compile these devices.
Unfortunately, too many of the tools are not happy running under a standard DOS or Windows environment. You may need a special boot sequence to free memory. Our PAL compiler isn't very happy with Microsoft's EMM386 for some reason, our PC layout software won't run from a Windows DOS shell, and the FPGA compiler requires every resource each system has--and then some. One solution is to use a boot scheme that asks the software you'll be running and then invokes the proper AUTOEXEC.BAT and CONFIG.SYS. Another strategy is to wait for all of this stuff to be reasonably available under Windows.
Often, I speak with a sad engineer trying to debug a complex system with a 286 running Windows in 1 Mbyte of RAM. Decent computers are so cheap, and modern tools demand so many resources that it's simply false economy to make do with yesterday's technology. Yet, no company can toss out all of the computers every year to follow the latest offering. Clearly, the minimum engineering machine today is a 486 with 8 Mbytes of RAM and plenty of disk space.
If you are saddled with an antique machine, measure the time spent waiting for compiles. Make a persuasive argument to the boss based on your salary plus overhead. Let him know just how much money he'll save with a $2000 to $3000 investment.
In one of my previous columns in EDN, I wrote about setting up a drawing system. The drawing system is only as good as the information that goes into it. All too often, the frenetic pace of debugging hardware tempts us to be less than careful about writing down changes.
Resist this temptation. Your company is paying you to debug a prototype for one reason only: so the prototype can be turned into a working production system. If you carelessly forget to document modifications, the company will need at least one additional pc-board revision, which you'll have to debug all over again. This is a terrible waste of money. A wise manager of such a documentation-free engineer either retrains or fires the individual.
Avoid scraps of paper to list things you've learned. The best solution is a meticulously maintained engineering notebook. Write everything down, clearly and concisely. The good sisters of my grammar school practically committed suicide over their failed attempts to teach me penmanship. So, writing clearly is a particular headache for me. I've learned to slow down and print because, most of the time, I can't read my own script.
Use one set of schematics to record changes. This is your master-development drawing set. Staple the schematics together and clearly label them as your masters.
In the movie "Jurassic Park," one brilliant slob controls a sea of workstations. When he disappears in his quest for the big score, we see his desk covered with papers and books scattered helter-skelter, old food remnants decomposing into new forms of life, and McDonald's wrappers blocking computer ventilation ports. Despite a nervous laugh, why is this image such an accepted icon of our industry?