Murphy's Law applies even under water

While working on a hydroelectric feed tunnel through the Andes in Peru, one engineer learns that Murphy’s Law is relevant when it comes to a manipulator and a mountain.

David Jeffrey, Marine Applied Research and Exploration -- EDN, November 11, 2011

Amongst the accessories was a manipulator arm. It was controlled by an RS232 serial link. Because of a drop-dead development time, we arrived in Peru before the final testing of the manipulator. However, over a test cable it had performed fine and we didn’t anticipate any problems. Classic Murphy’s Law. I now know better—get Murphy underwater and he goes on steroids.

The only difference (we thought) between the test cable and the umbilical was that in the latter the serial link was carried over an optic fiber. The main ROV control used an identical link which performed flawlessly during trials. However, when we tried the manipulator, it refused to work. I could see the RS232 signals as they left the optical converter in the ROV, but the arm didn’t respond. Looking at the manipulator electronics, I found that the TTL level output from the RS232 level shifter didn’t move from ground. Clearly the level shifter chip was faulty.

This Tale is a runner-up in EDN’s Tales from the Cube: Tell Us Your Tale Contest, sponsored by Tektronix. Read the other finalists’ entries here.

So I replaced it—but with the same results. The output seemed nailed to ground. I checked for a short in the PCB, but the trace was fine. In desperation, I shut the entire system down so I could watch it come to life. I found a curious phenomenon. As the first commands were sent to the manipulator, I could see a few good pulses on the “bad” pin, but after that, nothing. This suggested another explanation. Most microcontroller I/O pins can be set as input or output. Suppose the microcontroller was changing the pin in question to output mode and setting it low? That would explain what we were seeing. When two TTL logic signals argue, the low side always wins.

The manipulator was designed and manufactured in Scotland. I was on top of a mountain in Peru, with a very shaky telephone link and a significant time difference. But desperate times require desperate measures, so I called the Scottish company and, as luck would have it, managed to speak to one of the engineers familiar with the code. I asked if he could provide a small software patch that would force the “bad” pin to input mode on every program cycle, about 10 times a second. After a few minutes he called back with a single line of code for us. Fortunately I had an EPROM eraser and programmer with me, and was able to insert the code with some difficulty. (I did not have a compiler or other aid—we had to work editing the raw hex.)


We never did find out what caused the problem in the first place. It was probably an RS232 timing issue between the fiber optic output and the microcontroller input, and we didn’t have time to do any digging. But we had a solution and that was enough.

---

David Jeffrey is an electronics engineer who has specialized in applications where water plays an important role, from wave power devices to robotic submarines.