Feature
Lightning strikes sewage setup
Tales From The Cube: Mysterious, random chip failures leave SCADA-system engineer wondering whether isolation from ground could be the cause.
By Jacob Brodsky, Washington Suburban Sanitary Commission -- EDN, 9/17/2009
In early 1988, we had finished the start-up of a brand-new SCADA (supervisory-control-and-data-acquisition) system at the water-and-sewer utility where I work. It was summer, and we were busy repairing the damage from frequent thunderstorms. A co-worker, George, was supposed to provide postmortem repair and analysis of the RTU (remote-telemetry-unit) processor boards. Most of the 1-foot-square, multilayer processor boards had obvious scorch marks from lightning damage. We'd typically see a burned switching-power-supply module, PCB (printed-circuit-board) burns at the phone-line interface, or burns on an analog-input card. We'd then take a closer look at each of the sites where these boards originated and try to improve the situation through more careful grounding, better surge suppressors, or another means.
One morning, I noticed that George had set aside a few RTU processor cards. When I asked about it, he told me that they were malfunctioning but that he'd been having a hard time with figuring out what was wrong, so he'd set them aside to work on when he had time. "I just repaired this one over here," he said. "It was a single blown 74HC00 gate; the other three in the same chip were just fine." I asked where another similar card had come from. "You're going to love this," he said, "All of these problem cards, except for one, are from the Cabin John interceptor." I walked away, befuddled.
I made a detour in my travels later that week to take a look at the Cabin John interceptor, a waste-water metering vault. Resembling a small brick outhouse, it sat at the bottom of a valley in the woods. The RTU processor's only purpose at that site was to gather flow data from a large venturi tube, which measures fluid pressures and velocities, in the vault below the building. This flow meter was important because it was a change-of-custody billing meter. The saying "sewage flows downhill" isn't just an expression; it's also a fact of life. The sewage in this area was headed for a plant in another jurisdiction.
|
The only two points of data coming from the site were flow and a loss-of-ac-power alarm. The RTU processor was in a robust fiberglass box. At the back of the box, all the PCBs mounted flat on a grounded steel plate with ¼-in. standoffs and isolated screws. The setup also included an I/O card, the processor card, a battery-charge controller board, some terminal strips, an ac circuit breaker, and the 928/952-MHz telemetry radio. A 100-foot tower stood next to the building to get the antenna above the tree line.
The installation gathered the flow data from a 4- to 20-mA-current-loop pressure transmitter next to the cabinet. The pressure-transducer electronics, batteries, and PCBs were isolated from ground. Sure that George must have been wrong about something, I went back to his shop. He had set aside a few more cards, all with different failures.
The next morning, as I stumbled into the shower, I had a flash of insight: Maybe lightning was striking the tower and causing the ground potential of the backplate to rise. The board was quite close to the backplate. Perhaps a static charge was forming between the grounded metal backplate and the isolated RTU processor. A small arc could be destroying the weakest chips.
I got to work early and snagged some 1-in.-long nylon screws and standoffs from the maintenance shop. Replacing the ¼-in. screws with the longer ones protected the isolated board from random chip damage, fixing the problem, and we haven't seen failures of this sort since.
| Author Information |
| Jacob Brodsky is a control-system engineer with the Washington Suburban Sanitary Commission (Laurel, MD). |
















