Where there's smoke
Smoke billows out of a test station; operator error, or some other mysterious phenomenon?
Chris Fazekas, Lockheed Martin Missiles and Fire Control -- EDN, November 17, 2011
I received an e-mail one day, requesting that I investigate
an automated test station that I had brought online several
months prior. The support engineer explained that the system
had inexplicably shut down during some UUT (unitunder-
test) troubleshooting and that the engineer running
the test smelled smoke in the area. I visited the test area and
asked the engineer to walk me through his exact procedure for
troubleshooting.All of my assumptions about what had happened involved operator error. If the wrong relays had been commanded to close, they could have shorted out the power supplies and triggered their overcurrent-protection mode. The engineer showed me five times exactly what he had done. I didn’t believe him the first four times, but I soon realized that it was not operator error!
I decided I’d try finding the source of the smoke. When I saw no obvious signs of burned resistors or scorched wires in the test adapter, I got out my tools and started pulling apart the test station one card at a time. The cross-point-relay matrix had a burned adapter card in front of it. The damage was so extensive that it left scorch marks on a nearby card.
A relay problem had obviously
shorted the power, but how could that
be the case when the software never
commanded the wrong relays to close?
How could the damage have been so
extensive when the power supplies are
current-limited?The only way to figure out what had happened was to examine the burned card and trace back the shorted traces to the cause of the problem. The burned lines connected a 26V-ac output of the UUT to the digitizer in the test station. With a 1-MΩ impedance on the digitizer, the voltage and current did not even come close to the maximum limits of the relays. To make it even more perplexing, the engineer never ran those tests during his troubleshooting.
Now, with more questions than I had before my trip to the test area, I packed up my stuff and headed back to my desk to pore over all of the information, hoping to spark some theories. First, I drew the entire schematic, including the relays from the cross-point-matrix card, the test adapter, and the portion of the UUT involved. I checked the current rating of the adapter card that had burned. I then stepped through each line of code in the software and started reading data sheets, line by line, for hours. I could find no reason that the card had burned.
When reading through the crosspoint-matrix data sheet, I noticed a
warning below the voltage and current
ratings of the relays. It stated that
operators must protect the relays from
voltage transients. As soon as I read
that warning, it hit me: The UUT
component that the engineer was measuring
was the output of a transformer.
An inductive device, such as a transformer,
generates a voltage transient
whenever you remove power because of
the collapsing magnetic field. This transient
could sometimes reach thousands
of volts, far exceeding, for short periods,
the voltage limit of the relays.I concluded that, over several months of properly running the UUT, the cross-point matrix was receiving damaging voltage spikes with each power cycle until a relay finally failed and shorted the circuit. To make matters even worse, the shorted transformer could supply more than 10A, explaining why the adapter card was so badly damaged. I learned a valuable lesson about designing with reed relays that day.
First, I assured the engineer who had performed the troubleshooting that he was not at fault. I then added an MOV (metal-oxide varistor) to suppress the voltage transients and a fuse to prevent the MOV from shorting out at the end of its life. I then included a voltage-divider circuit to further isolate the relays from the transformer in case another short occurred.
Chris Fazekas is a systems test engineer at Lockheed Martin Missiles and Fire Control (Orlando, FL).
Talkback
