Smack attack

Terry Staler, Specialty Concepts Inc - April 19, 2012

After graduating from college in 1976 with my brand-new bachelor’s degree in electrical engineering, I went to work for a small southern California company that produced telephone-usage-monitoring equipment. My responsibilities included those of a junior engineer, technician, purchasing agent, and assemblyman. (The company was really small.) Our first product connected to phone lines in large corporate phone systems and recorded the number of calls placed and total off-hook time. This monitoring allowed the accounting department to compare phone bills with actual usage and to catch any overcharges or abuse of phone usage.

Each unit was in a plastic box that measured about 8×12×3 in. and contained a handful of ICs; a PCB; a transformer; a 5V-dc supply; and a wheeled counter and timer, both 120V ac. A circular connector at the top of the unit mated with a phone-company-installed socket. We could view the counter and timer’s wheels through slots in an aluminum front panel. During testing, these units performed flawlessly, both in our lab and at a few smaller beta test sites. When we started to perform larger installations, however, we received some complaints. The counter would show two, three, or four calls when only one had actually occurred. We confirmed this information by comparing timer information with the number of calls; the average call was just too short.

I received the unenviable job of trying to solve the problem on-site. I drove to the company, which had more than 10,000 employees; checked in at the front desk; and was escorted to the phone-equipment room. This hot and humid room contained what looked like a million punch blocks and an infinite amount of wire covering every square inch. Our units—all clicking merrily away—were hanging from several hundred sockets. At random, I watched one unit, and, sure enough, I saw the counter advance two or three times in quick succession. I disconnected this unit and brought it back to our lab for troubleshooting. Yet, try as I might, I could never get it to fail, despite subjecting the unit to heat, humidity, and occasional sharp smacks with the palm of my hand. Talk about abuse!

Reinstalling the unit produced the expected result: random counts. Switching this unit with another may or may not solve the problem. In other words, this problem was random and intermittent—every engineer’s nightmare.

So the burning of the midnight oil began. Because we could not replicate the failure in our lab, we reinstalled the units at the customer’s site. At one point, we had 20 more bypass capacitors than ICs on the board. We redesigned the 5V power supply. Nothing worked.

One day, staring at the PCB with bloodshot eyes, I noticed that one of the PCB traces for the counter ran under one of the ICs. This line carried 120V ac. Having this line run under the IC was clearly not good design practice. I removed the trace and replaced it with a piece of wire routed away from the PCB. We reinstalled the unit, and—voilà—it worked.

As the junior engineer, I was assigned the task of sitting in the hot and humid equipment room with my X-acto knife and soldering iron, making repairs at the site. A week later, all of the units were working perfectly.

We never determined why the units failed at the site but not in our lab, but we suspected that the ac spike from the counter’s firing coupled into the IC was just below the threshold at which a false signal would occur. Something at the site had pushed the signal to the limit, perhaps because of the phenomenal number of wires, with their attendant radiated noise, covering the entire facility. Proper and careful PCB layout could have averted this problem.

Terry Staler is president of Specialty Concepts Inc (Chatsworth, CA). He treats newbie engineers with great respect.