Toyota, drive by wire, and our failure to learn from experience
There simply is no systematic approach to ensuring the quality of an integrated hardware/software system.
By Ron Wilson, Executive Editor -- EDN, March 4, 2010
I see from the morning news that Toyota’s adventure into the world of embedded software is going badly. The company’s second attempt to find a quick fix for unintended acceleration in its conventionally powered vehicles is barely under way, and evidence is already emerging that the underlying problem is likely in the engine controller, not in the pedal’s mechanical assembly. Now we hear from Japan that the Prius, Toyota’s golden child, has a problem with its brake-by-wire control system.
Decades ago, Audi accidentally introduced drive by wire with its advanced cruise control on the Audi 5000. The cars were allegedly subject to spontaneous acceleration, a problem the company blamed on operator error. At the time, researchers at another European high-end auto company claimed to have uncovered a problem in Audi’s engine-control firmware and reproduced the acceleration without requiring a driver to mistake the gas pedal for the brake. The ensuing liability litigation, however, extinguished all hope of diagnosing and documenting the problem so that the rest of the real-time-software community could avoid it.
All this came to mind when I attended a panel on achieving quality closure at last month’s DesignCon in Santa Clara, CA. Despite the subject of the panel—achieving quality closure—the issue of software sat like an elephant in the corner of the room, awaiting notice. One of the panelists pointed out that the most serious quality problem in IC designs now is not the quality closure on the hardware but the integrity of the firmware and software that will run on the chip. There simply is no systematic approach to ensuring the quality of an integrated hardware/software system.
|
This situation is a tragedy. Work was well under way 30 years ago on the problem of formally proving software correctness. One company had designed a completely deterministic microprocessor that made it possible to mathematically prove all of the possible trajectories of a code set. Computer scientists such as Edsger Dijkstra were making strides in a method for creating formally proven software. But along came C, Unix, and the cult of the hobby programmer, and the entire notion of formal correctness vanished under a smokescreen of hacking.
Now, after decades invested in metrics-driven verification, formal verification, and methodology management, designers find that their chips don’t work as expected because the software is still being “verified” by feeding it test cases until the schedule expires. Consumers find that their cars run into these problems for the same reason, and the press blames the problem on “electronics.”
Once again, as in Audi’s day, it is safe to conclude that a gag order as part of a class-action settlement will screen whatever accurate diagnostic work takes place on the Toyota problems so that no one in the industry can benefit from what Toyota engineers learn. In that way, we can repeat the situation with the next generation of software-governed systems, a new set of executives can avoid blame for the tragedies, and a new set of lawyers can make their fortunes from the resulting litigation.
The only parties in this little tragedy with an interest in improving the state of the art are the engineers, whom no one will consult, and the victims, whom the lawyers will silence. It would be better for everyone if it were a principle of civil law that, when a failure inflicts damage, the vendor and independent parties must place all of the diagnostic information they find into the public domain, and the courts may not use this information to assess or assign damages. Such a notion might somewhat restrict the income opportunities of litigators, but it would unquestionably assist the engineering community in learning from its mistakes.
Contact me at ronald.wilson@reedbusiness.com.
-
In my 35 years of mechanical engineering I've worked in two specialties; brake & military reliability.
Although it is not published, it is quite possible to calculate braking horsepower from stop time, speed & vehicle weight. The result? The horsepower capacity of a properly functioning vehicle brake system typically runs about 200% of engine horsepower. This is codified, in a back door sort of way, in the Federal Motor Vehicle Safety Standards (FMVSS) which apply to brakes. Braking horsepower is not specified but a vehicle cannot meet the standard without the capacity mentioned.
From my experience in military RMS, I've reached a number of conclusions.
First, any sufficiently complex system can be expected to behave in an unanticipated manner; whether electronic or software or mechanical. The number of these unanticipated operating increases with complexity but the ability, the likelihood, of detecting all these modes decreases. Methodologies like DFMEAs and FTAs help but are still limited by the education and experience of the person or people doing the analysis.
Second, software reliability is problematic for two reasons. Fault states disappear when the power is cycled and the amount of time required to completely test large programs approaches infinity. Early fly-by-wire aircraft & smart weapons had some truly amusing failures.
Third, the auto auto companies have done a passable job dealing with heat stress and EMI after early failures with the first computer controlled engines. These issues were easier to discover and fix.
Finally, get over the idea of greedy auto companies. Drive-by-wire is expensive and I believe that it would not be on the road today but was forced into early release by federal standards on pollution and fuel economy. Don't get me wrong. Reducing pollution and increasing fuel economy are both worthy goals. We've come a long way. Still, bloated bureaucracies have repeatedly shown themselves unable to make good, timely judgments as to what is possible, practical and affordable.
Gerald "Jerry" Dreisewerd - 2010-6-6 06:52:57 PDT -
re: "There simply is no systematic approach to ensuring the quality of an integrated hardware/software system." If you believe this quote, I suggest you stop flying in modern commercial aircraft. Absolute perfection may be unattainable, but there are some fairly rigorous, systematic processes for verifying systems. They aren't cheap and you can't go from "adolescent fantasy car concept" to reliably functioning product in a few weeks. It's the rush to get to the market with the most horsepower and "bling" and the lowest cost to produce that get the auto manufacturers in trouble. I think the bulk of the new car consuming public really doesn't care anyway -- after all, we have been conditioned to treat cars as toys and ignore the obvious risk of "playing" on the highway (look at the commercials) which every reasonably intelligent person knows is riskier than Combat. Product defects are good business for the ambulance chasers and the opportunists, anyway, so where is the incentive to really understand/prevent them?
William L. Thrush, P.E. - 2010-17-4 13:40:00 PDT -
Ever wonder who is driving the automated rail trains in Germany and the USA?
How about testing your code. When lives are at stake, good enough isn't good enough!
Jim Black - 2010-14-4 15:19:00 PDT -
There may or may no be a software problem but there definitely is an HMI problem.
I have a 2009 Lexus. One thing I noticed is that the brake and gas pedals are too close together and are at almost the same height. In fact, with my foot pointing forward, my foot can bridge between the two pedals. If you're thinking I must have big feet, you're wrong; my shoe size is 9.
I tried an experiment: with my foot bridging the two pedals, I pressed down. What happened? The car accelerated. The gas pedal is more sensitive than the brake pedal so each incremental push results in much more power than braking.
I suspect that many of these accidents are related to the proximity of the gas pedal to the brake pedal.
Stuart R. Michaels - 2010-9-4 15:49:00 PDT -
Before the auto guys jump to the aero guys DO178B level A software design practices and the associated cost increases, I suggest that a Safety Hazard Assessment (similar to the aerospace SAE ARP 4761) be applied. This will identify the critical faults and associated consequences that need to be mitigated. Mitigation can be via "178B" or independant hardware redundancy or maintance, etc. An equivalent FAA/EASA/CAAC organization can be the clearing house to ensure the effort is completed and protect company proprietary design practices. Further, the 4761 process, identifies redundancy violators such as (a) Single Point Failures, (b) Latent Failures, (c) Combinations of Failures With Excessively High Probability, (d) Installation Problems, and (e) Design Errors.
This solution adds only needed costs to the design. I don't think most of us can afford a multi-million dollar car as is the airline baseline.
Glen High - 2010-8-4 11:30:00 PDT
