EDN logo


Columnist: November 23, 1994

The Case of the Crashing 68000

Jack Ganssle ,
embedded-systems contributing editor

It was a dark and stormy night in the port of Baltimore. Angel, my leggy receptionist, bleeped me on the squawk box: "Spike's here, Jake. And my paycheck bounced again. I've quit worse jobs than this!" They call me Jake in the sleazy end of town, where the streets of the dirty city tell stories of pathos and passion. Edgar Allen Poe lived and died a dirty tramp in a gutter not far away. Old rotting buildings still testify to the port's seamier history, a history no amount of urban renewal can--or should--remove.

"Cool it, Angel. Show the creep in and leave me alone." Spike stumbled into the old armchair that looked like it went four rounds with an angry bear. A bottle of Captain Morgan fell out of his pocket and shattered on the floor, creating a spicy odor that overwhelmed the scent of mildew.

"It's bad, Jake," he mumbled, scratching the gray stubble on his face. "The Kid just can't make it work. If he doesn't get that thing fixed, the customers will off him for sure." Great. Just after I duct-taped the windows shattered in the last round of tommy-gun fire. At least they only nicked my monitor; once a round went right into the hard disk and ricocheted off a file of backup tapes. I poured him a shot. He tossed it back, fast, his hands shaking as he slammed the empty glass on the table. "Jake, this system is killing me. The last set of pc boards were no good. Now we're almost ready to ship it, but every few hours the 68000 crashes. I just don't know what we're going to do." As he said this, Spike hung his head down, elbows on his knees--a defeated man in a dirty business where the rules change daily and the spec sheets are mostly written in Japanese.

"Tell me more, Spike. When does this little bastard crash?"

"Never in the main loop, Jake," he replied. "Only when the command interpreter goes out to service a packet, and then only sporadically. We can execute millions of cycles without a problem, then boom--it's history."

"Ya plan to execute the thing, huh?" I asked[el1]and then realized he didn't mean it quite so literally. "I think it's time we call in Bruno."

"No! Never! I swore I'd never work with him again!" he screamed in a thin whine. "I hate those damn consultants! They always make us look so stupid!" I placed the call. "Four thousand dollars a day, plus expenses--in advance," was the gruff reply before the phone was slammed down. We waited for his white limo. Bruno muscled his bulk though the door, a door wide enough for the usual slob but one that seemed pitifully narrow now. Bruno's face was a mass of scars, scars I knew he had won in a score of battles. The big one on his forehead was from IBM on the ATC project. That slash on his cheek came from the abandoned Sergeant York. He wore these scars proudly. His leather briefcase banged on the floor.

"Hargmpfh!" Bruno was never much for words, but put him in front of a keyboard and his thick fingers were as graceful as Angel doing ballet.

"Well, Bruno, you know the score. The system fails once in a while and the engineers have no idea why. It's a new design, but it looks pretty good for an outfit like this." Bruno's hand crashed down on my desk, making the clip pop out of my .45. "Where's my money?" he rumbled. I scribbled a check, which he held to the light for a minute before pocketing it and clumping off to the lab. We followed--Spike, slouching and mumbling, me with both hands on my wallet. The door to the lab creaked open, and we descended to the bowels of the building. Arcs of electricity flowed up Jacob's ladders, while over in one corner a team was trying to reanimate Elvis's corpse. We ignored the stench and moved to a low-slung table where a solitary engineer was working feverishly over a small computer system.

"What's dis?" Bruno asked, pointing to a 68000-based factory controller connected via flat-ribbon cables to a cage full of STD bus cards. The Kid stared up at Bruno, fascinated and immobilized by fright, like a deer caught in the headlights of my Packard. His neck let out a sharp click as he looked down again and told Bruno that this was the source of all of his trouble. For three months, the Kid had been debugging the hardware and software. He knew he was under the gun--we had hired him to replace an older engineer who made too much money. I saw that outcast just last week, hustling quarters in the street, and barely recognized him. His sign "Can you spare a dime? I know calculus!" gave him away as one of those poor fools who had not learned to quit engineering by age 25.

Trying to be helpful, I asked the Kid what he though the problem might be. "I dunno. I've looked at everything. The timing is perfect, the voltage levels are fine. It doesn't make sense!"

With a hand that looked like a side of ham, Bruno grabbed the scope probe from the Kid's trembling fingers and started looking at different test points. "You ain't got no decent gear," Bruno complained. "How am I gonna work with dis lousy 100-MHz scope? It ain't even digital!" I made a quick note to have Spike hit the local test-equipment store that night after it closed. He'd have to be careful, because the cops were starting to get wise to us. They knew there was a pattern to the rash of test-equipment capers on the south side. Bruno pulled out a flip phone and ordered his driver to bring in the 1-GHz four-channel beauty I knew he had stashed in the limo. Then I started listening to Bruno mutterings. At $4000 a day, plus expenses, we had to learn a lot from the big brute.

"Them low-frequency scopes with 100-MHz probes can't really see what's on a high-speed bus. I bet dis here sucker is oscillating, like real fast. Real fast," he muttered over and over.

Bruno propped his scope on the back of the chauffeur and probed the address lines. They looked pretty good to me, tristating as always as a processor enters a bus hold mode. The data lines were awful, but data always is a mix of ones, zeros, and tristate conditions as the memories decode chip selects and output enables. Only the control lines--read, write, and address strobe--seemed to present solid levels all of the time. Still, in a lifetime spent in the grungiest labs in the meanest cities, this all looked pretty familiar to me. "What do you think?" I queried.

Bruno's head slowly revolved around to glare at me while the rest of his huge body remained motionless. "Don't rush an expert. Gimme time to think."

"You have that scope set all wrong," the kid whined. "'I always set the vertical channel on 2V/cm. You've got it on 1V/cm!"

"Shaddup!" Bruno said as he shoved the Kid back onto his stool. "Yoose guys gotta understand that digital circuits are really analog. Ya can't see good enough on a 2V/cm setting to tell if a signal is above the legal minimum one level or below the legal maximum zero. Hey, I betcha don't even know what voltage a logic one is, anyway."

"CMOS or TTL, HCT or HC?" shot back the Kid.

"Good answer, Kid. Each logic family has its own levels, and ya gotta make sure ya obey the rules. With the scope set to 1V/cm, I can see in an instant if any of these levels fall outside of the legal range. Ya know how I hate being illegal. Besides, dat's just the sort of problem dat will cause dis kind of intermittent." Spike spoke up. "If it's intermittent then maybe it's a broken track or something."

"Yeah. Lemme see." Bruno lifted the board and flexed it gently, all the time watching the monitor. The system kept running. Then he turned the board over and ran his fingers gently over all of the pins. I stared, fascinated at the dozens of tiny scars on his fingertips.

"It's dem sharp through-hole leads. Dey cut me up, bad, like Roscoe da Razor did that time in Vegas." Later, I learned that Roscoe was in the joint for passing counterfeit 486s. I also learned that Bruno was searching for unconnected input pins that may have drifted to the right state through luck, or 'cause someone up there watches over drunks, sailors, and swill like me.

"Da impedance of any digital output is pretty low. My paws shouldn't mess up the circuit at all," Bruno went on. Despite his almost sensual stroking of each pin, the circuit continued to run without fail.

Now Bruno hooked the 1-GHz scope to one data line and carefully connected the probe's very short ground lead to the IC's ground. He made sure the bandwidth limiter was off, triggered on the read line, and tuned the scope's trigger controls like an old-time ham pulling in a very weak signal. A low growl alerted me that something was up. He checked another data line, and then another.

"It's oscillating!" Bruno roared. "What, yoose got amateurs designing the stuff now? Watch. I triggered da scope on the read pulse. Look at the data line after da read pulse goes away."

"No one does that," snapped the Kid. I didn't know what Bruno was getting at, but, having seen him in a rage once before, I kept mum. I know for a fact that the pub owner now wishes he had, too.

Bruno's neck started turning red. I backed slowly away. "Of course da signal's OK during read, you louse. After read, when da bus tristates, it's oscillating at about 450 MHz. Dat'll crash your system. Pull-up the data bus with some resistors in a SIP."

"No way, Bruno. The data books only talk about the data bus during read. Who cares what happens during a tri-state condition?" asked the Kid.

"Add da SIPs," Bruno demanded. The Kid refused again. Bruno whipped out a 9mm and held it to the Kid's temple. "Da last idiot dat messed with my ideas is pushin' up daisies" he rumbled. I rushed forward to solder the SIPs in myself. I didn't want Bruno offing the Kid.

"Now we go to your office and wait," Bruno said. "I've set da system up to run all night to see if it's fixed. Look--the oscillations are gone, and I'll bet you a pair of brass knuckles it's OK now."

For 24 hours we sat, facing each other across the beat-up oak conference table. Bruno's eyes bored into mine. He said nothing; I replied in kind. Each twitch escalated into a near shootout as we nervously watched each other's movements, my heater poised under the table, his likewise close at hand. All the time I wondered whether this outsider was worth his $4000 a day, plus expenses.

The new day dawned. The squawk box reported success with the SIPs; a full report was on its way up.

Minutes later, carrying a stack of engineering notebooks, Elvis strode into the room.


PICTURE

Jack Ganssle is the president of Softaid, a vendor of emulators and other embedded-systems tools. His idea of heaven is sailing across oceans, although the ugly face of common sense precludes these dreams too often. He can be contacted via Compuserve at "76366,3333," or via Internet at "76366,3333@compuserve.com". For those users of the Pony Express send mail c/o Softaid, 8310 Guilford Rd, Columbia, MD 21046.


| EDN Access | feedback | subscribe to EDN! |
| design features | design ideas | columnist |


Copyright © 1995 EDN Magazine. EDN is a registered trademark of Reed Properties Inc, used under license.