Mismating mix-up makes for monstrous military mission
Tales From The Cube: A technician mates together two mismatched connectors, damaging an aircraft and yielding an impossible task.
By Bob Mason, Schneider Electric -- EDN, April 23, 2009
My first job out of college was for a military-electronics company. We built rectangular metal boxes full of electronics that go into aircraft and space vehicles. Both plug and socket types of keyed circular connectors covered the boxes. We knew that bad things could happen if we mismated the connectors, so we went to great lengths to make sure to plug together only those connectors that were intended for each other. We varied the sizes of the shells and also varied whether a given connector was a plug or a socket. We avoided having two plugs or sockets of the same diameter. If we absolutely had to have two shells of the same size and type, we made sure that each one had unique keying configurations that were incompatible with any other connectors on the box.
So, in accordance with Murphy’s law, an aircraft-maintenance technician somehow managed to mate two plug-type connectors together, despite all of the size, keying, and other preventive measures that we had designed in. The result of this configuration was a large flash and loud boom when he powered up the aircraft. The electronics unit was destroyed, as were other circuits in the aircraft.
Although we were impressed with the technician’s ingenuity and perseverance, the officers in his chain of command were not amused. The general in charge of the air base got directly involved.
He first ordered us to determine what would occur if any connector signal on the box were to be connected to any other connector signal on the box, regardless of whether it was physically possible. After determining that outcome, he instructed us to make whatever circuit changes were necessary to make such a mismating innocuous. He didn’t go as far as to tell us to make it work in that configuration, but he did insist that no damage occur.
Our managers, duly chastened after their visit to the general’s office, immediately set to work. They put together a task force to start the cross-connection analysis. The task facing us was enormous. The box had 20 connectors, each with about 30 pins. When we began, it took several hours to analyze the effect of shorting a pair of signals together. As we got used to the work, that time dropped to about one hour per pair. This phase was only for analysis, however; it didn’t include redesign. After several days, we saw how little progress we were making. We decided to determine the scope of the task.
The math wasn’t hard: 20 connectors of 30 pins each totals 600 pins. The combination of 600 items, two at a time, is 179,700 combinations. At an hour each, that job would take 179,700 man-hours, or 86.4 man-years. The four people on the team could count on spending the next 21+ years doing these analyses.
We presented our figures to our management. They didn’t say much, but we saw some eyebrows rise. Management took our figures to the general. We never heard what the general said, but the next week we were taken off the connector analysis and returned to our interrupted projects.
The lessons I learned from this experience were that you should always define the scope of a job in the beginning, and, whenever you think you have made something idiot-proof, remember, along will come a better idiot.
-
A good story for those concerned with complexity. If you have 20 things, say issues of EDN, and arrange them in every possible order, that is 19! (19-factorial, ie 19x18x17x16....etc) and that is more seconds than the universe has existed since the Big Bang. Now you know why software with thousands of branches is untestable. I worked for a project manager at Teledyne that had been a flight-line chief in the Air Force in Germany in the 60s. He said they had one set of radar boxes that got condensation, so they drilled holes in the bottom to let the water out. After that, he said everything, every single problems keeping those F-16s flight-worthy was cables and connectors.
Paul Rako - 2009-5-5 19:10:00 PDT -
Brilliant Anecdote. Thanks for sharing this.
Not many people will realise, but the generals reaction
is usually the first reaction an informed and
experienced personnel will take, to make it idiot
proof, as they call it, besides the actions you had
already taken
And not just in the military, but in every engineering
management firm. But the task was just too humongous
with the number of connectors you''''ve got.
A classic case of everything is possible but not always
feasible
Harry - 2009-27-4 23:43:00 PDT -
A friend of mine worked at Kennedy Space Center in the 70''s. They were having some intermittent telemetry problems from one of the launch pads and he assigned a summer intern to investigate.
The intern pulled the prints for the cabling from the pad to the bunker and carefully traced each one from end to end. He discovered that in the transition from one drawing to the next, the cable in question had two male connectors. "Well, that can''''t be right!" So they got in a Jeep and drove to the manhole where the (supposedly) two male connectors resided. Sure enough, the print was correct, and the installer had carefully mated the two male connectors as best he could, and then wrapped it all in a great wad of electrical tape.
The cables were built to print and had passed inspection.
Dave Thomson - 2009-23-4 10:40:00 PDT
