Stopped in our tracks
The minicomputer was halting at random times, but why?
Ray Hill, Systemation Technology Inc -- EDN, July 15, 2010
| View as PDF |
Back in the ’70s, I worked for a medical-electronics
company that made patient-monitoring equipment.
My group developed a minicomputer-based system
that accepted bedside data and provided patient status
and charting information on a couple of CRTs at
a central nurses’ station. The equipment comprised
a 19-in. rack of interface electronics and a minicomputer. Most
of our installations went into hospitals, but it was sometimes difficult
to find a place in the hospital for this rather large rack of
equipment. Most of the time, we put the equipment into an outof-
the-way closet.We were feeling good about the performance of this new system for the first year or two, and we had systems in many hospitals around the country. However, we then installed a system in a large, well-known hospital in Chicago, inside “the loop.” The equipment was located in a closet on the hospital’s third floor, close to the nurses’ station. It was not a very comfortable place, but nevertheless, the closet was cozy and had a view of the Chicago skyline.
Minicomputers were then large, and they consumed a lot of power. You had to treat them kindly and pay attention to what kind of environment in which you wanted them to operate. If the minicomputer was unhappy, it would usually just halt. There was no reboot and no watchdog; it just stopped executing code. For this reason, we made sure that the minicomputer rack in our assigned closet had adequate cooling, conditioned power, and other amenities.
The new installation in Chicago would run for days and then randomly stop—apparently without reason. The time, day, date, weather, and phase of the moon could not explain why this would happen. Our service people changed out boards, memory, I/O, cabling, and even the entire minicomputer, but the problem persisted. After a couple of months, we were all baffled, and the hospital employees were getting upset.
| Read More Tales from the Cube |
A couple of days went by with no problem. Then it happened! The minicomputer just stopped running, the CRT screens froze, and the keyboard went dead. Nothing seemed to have changed. I restarted the system and waited again. Several hours later, it stopped again. This time, I noticed a small flash in the window. I went to the window in time to notice the “el” train going by. The elevated track ran beside the building at about the level of our floor. After that discovery, I monitored the situation for 24 hours and noticed that the minicomputer stopped only when a train went by.
We demonstrated this problem to the hospital and immediately began grounding and shielding both the equipment and the closet. We installed a wire mesh across the window and the wall and connected it to a newly installed ground that the hospital provided. This approach fixed the problem. Evidently, the train was generating enough EMI (electromagnetic-interference) to stop the minicomputer.
I called my boss and blamed the whole thing on the train. I then asked him whether I could come home.
Talkback
-
This one's second-hand, related by an engineer I once worked for, but it must set a record for long-distance train effects. He was working on a military project out in the Nevada desert somewhere. The project was a giant magnetometer intended to detect something secret and military. No ferrous metals were allowed anywhere near it. The frame of the machine was built of wood held together with machined fiberglass bolts and nuts. After machining, the bolts and nuts were boiled in nitric acid to remove any traces of iron left from the cutting tools. They were serious about this. The machine worked as intended most of the time, except every day at a certain time, a large amount of spurious noise would appear. They finally traced it down to a train that made a run every day at that hour fifty miles away. That's a sensitive magnetometer! He never did tell me what they detected besides trains.
David Sherman - 2010-5-8 01:27:04 PDT -
Quote:
In the 80s's I was working for the Goodyear Tire and Rubber Co at the Tire Plant in Lawton, Oklahoma. They built a Medical Center and ran it with a TI MiniComputer. They didn't provide conditioned power and in the afternoon when they started the Banbury Motors to mix rubber it would go down. We installed a 1KW SOLA Xfmr and solved the problem. Our Chief Electrical Engineer couldn't believe that was the problem, but it was...
Small world. My last Banbury control system install was in your plant in 1988.
Those Sola minicomputer isolation transformers can be a real problem solver. I installed a batch control system at a glass plant in Stockton, CA in the mid '80s. The plant had split power (grounded, center tap 120VAC) and we were concerned about what that might do to our signal processing equipment. I installed a pair of 1KW Sola transformers as a precaution before installing the new control system. We were near the end of the installation process when the plant experianced a power blip. All the maintance techs were used to the old system going haywire after power blips so they all piled into the control room. They looked at me, and looked at the perfectly functioning system. After that, the head of plant engineering wanted to know how big those Sola transformers came. This is the same guy who poo-pooed putting in the transformers in the first place.
Phil Ouellette - 2010-27-7 08:22:26 PDT -
In the early 80's I was playing with my third computer, after building my first two, i broke down and bought a Radis Shack TRS-80. Everytime it was windy outside, the computer would lockup ramdomly. This only ocured during windy periods, so I went looking outside. I noticed that I was able to simulate the problem if I hit the 120/240 service drop to my house. Breaking out my trusty oscillscope, ona nice windy day I was actually able to see 200+ volt spikes sitting on top of the 120 vac. These were only a few a few microseconds wide. The cheap power supply would capacitively coupy these pulses to the 5 volt supply, randomly change a bit, and lokup the computer.
The really interesting part of this story was convining the local utility (whom I worked for) that there was a problem. Eventually they found that the old style bolted secondary connection has some fractured wires. This was replaced with a compression fitting, and now 30+ years later as the same house all is fine.
Dave Bassett - 2010-22-7 18:56:49 PDT -
Ray,
In the 80s's I was working for the Goodyear Tire and Rubber Co at the Tire Plant in Lawton, Oklahoma. They built a Medical Center and ran it with a TI MiniComputer. They didn't provide conditioned power and in the afternoon when they started the Banbury Motors to mix rubber it would go down. We installed a 1KW SOLA Xfmr and solved the problem. Our Chief Electrical Engineer couldn't believe that was the problem, but it was...
John w Lynch - 2010-22-7 10:38:34 PDT -
Hay Ray
I'm sure we must have met at some point. I too was in Biomed in the late 70s. I too had a halting minicomputer. Turned out to be a bumper plating shop about half a mile down the road. They had a deal with the electric company to hold off their plating operation until after peak load, about 10 PM. Sure enough, about 10 PM every night my beeper went off announcing system halt. A quick check found the problem. The plating operation pulled the AC line down just below 100VAC! I'm surprised that the mini was the only thing that was having problems.
Mike Dingus - 2010-20-7 21:38:43 PDT
