It was a quiet day when my manager came to me with an urgent request. Our most recent quality numbers showed a significant regression, causing the failure rates of our control boards to more than triple in three months! As this fact became better known, the number of fingers pointing at the engineers to fix this issue increased exponentially. Management assigned a team of engineers to look for anything that could have changed. Did we make any component or process changes? Was the regression within normal statistical variation? Was there a change in the reporting structures? The answer to all these questions was no.

With nothing to show for our first investigation, we ordered many “failed” controls from the service department. While waiting for controls from the service department to analyze, we called customers and service technicians to get more info on why they were replacing these controls. Many customers would say only that the control boards failed intermittently, would not respond to key presses, or would randomly just stop. Service techs were not much help, either, often preferring a “swaptronics” approach to troubleshooting, rather than detailed root-cause analysis.

Finally, the first 10 boards arrived, and, as engineers, we eagerly attempted to uncover the issue, fix it, and put this whole thing behind us. Strangely, all 10 boards worked as we expected with no issues. The same scenario occurred with the next 10, the 10 after that, and so on until we had analyzed more than 50 boards and still had no idea what was causing the problem.

With the work becoming more and more hopeless, a batch of controls came in for review. I went about analyzing the boards, and the first few checked out fine. Then, I finally got one board that actually was malfunctioning.

I isolated the issue to a tactile switch that was not working as expected. Upon further inspection, I noticed some contamination on a hidden shelf between the opposite-polarity leads of the switch. Examination of another failing board confirmed the presence of this foreign material, which looked like flux contamination. The mixture of this contamination with the metals, voltage bias, and local humidity in that area had caused electrochemical migration, creating dendrites of the local metals. These dendrites, in turn, had slowly shorted out the switch. When I brought another engineer into the lab, though, all the failing boards were functioning perfectly fine.

After some good-natured ribbing from the other engineer, I again looked into the boards. A board had started to fail again. Taking another board and breathing across a suspicious switch caused it to also start failing. The film of condensed water molecules from my breath, in conjunction with the lower resistive path that the dendrites had caused, made the overall resistive path small enough to affect the circuit’s performance. Those previous controls tested OK because they had dried out before I tested them.

It turns out that, about the time of the regression, the company had contracted with a new supplier for the switch, but no one ever notified the engineers of the change. The new part was more susceptible to contamination due to spacing and construction, which caused the increased field failures.

Despite the good-natured ridicule I’d received for the “breath test,” it nonetheless was an effective way to run a quick environmental-condensation test, stressing controls and finding issues.