The Dreamliner saga: When your solution is more than just a software patch
We looked at the Boeing 787 Dreamliner lithium-battery issue a few months back ("Dreamlining Boeing and batteries"), and the good news is that Boeing has come up with a solution (aka "fix") that is approved by the FAA and is being implemented on both in-process and delivered aircraft.
But unlike the convenient world of software patches and downloadable upgrades, this is the real world with serious mechanical, electrical, and thermal consequences. I won’t detail the totality of the fix here - you can Google it yourself or read this well-written piece "How Boeing Rescued the 787" from The Wall Street Journal (sorry, it may be behind their pay wall).
In short, the solution requires lots of work, as you’d expect. Among the elements are better inter-cell insulation; a stronger, heat/flame resistant containment box; potential venting if needed to outside "just in case;" plus various new sensors and software modifications.
I've been following this situation closely, since it simultaneously combines so many engineering and non-engineering disciplines: raw electrical power; power management; safety evaluation; mechanical, thermal, and chemical engineering; aerospace design; politics, media visibility ... the list goes on. It has all the elements of a dramatic, reality-based movie in which engineers would be heroes (except, IMO, with today's public disdain for engineering, the story would likely focus on the problem rather than solving it, and with the engineers portrayed as negligent practitioners, sad to say).
I won't even begin to pretend to know what the real problem is, nor what Boeing should be doing. Unlike the many so-called "pundits" I have read and who admit they know little or nothing about batteries, but are nonetheless comfortable speculating and presuming from the commanding heights and isolation of their PC screens and keyboards, I know that I don't know.
Nor will I say it's all because Boeing farmed out so much of the design and instead focused on integration for the 787, rather than do the design in-house; that's easy to say, but with the many complexities of today's technologies, maybe you have little choice but to rely on the expertise of outsiders. I just don’t know.
(Personally, I'm guessing it's an internal battery malfunction, probably due to a subtle manufacturing defect, rather than a power-management/control/charger problem. Or perhaps it's an unanticipated combination of "small" problems. In general, serious failure analysis and experience shows that most failures do not have a single cause, but rather several problems adding together in a nasty scenario, and then cascading to something bigger. But that's just a gut feel, and I may be very wrong.)
Whatever it is, I do know this: thousands of engineers put in zillions of hours trying to understand what was going on, and what could be done about it. Whatever was going to be done would be orders of magnitude more complex and involved than just downloading a clever patch, or maybe changing a few resistor values somewhere.
In the real world of power and aerospace, the fix to many problems involves significant design changes; having to do them in the field adds to the challenge. Just try to imagine the constraints of developing a solution to the problem that can has to be retrofitted to in aircraft already in the field (admittedly by well-trained teams), as well as those on the assembly line. Think about the engineering change orders (ECOs) and verification needed of the plan itself, the procedure to make the changes, and the process to verify that it has been done right. It truly makes my head spin.
Yes, we've all heard the management-consultant mantra that "change is hard." In my view, they have no idea how hard it can be, when you are trying to understand, solve, and implement a fix for a problem of this type, on a program this big and visible, with the consequences so large.
Have you ever been involved in a production and field-fix situation that required more than just relatively minor changes to component values? What was it like?