Embedded Processing

Technical Editor Robert Cravotta explores processor and software-processing architectures and the impact they have on system and software development. Relevant architectures include microprocessors, microcontrollers, digital signal processors (DSPs), multiprocessor architectures, processor fabrics, coprocessors, and accelerators, plus embedded cores in FPGAs, SOCs, and ASICs.

Advertisement

Profile

About Me

RSS Feed

Add this blog to your RSS newsreader!

By Category

Processor-based Design Articles

Blog

Embedded Processing

« Are embedded reference designs... | Main | Updated: EDN DSP Directory... »

Thursday, March 29, 2007

The consequences of redesigning a complicated embedded design from scratch

Mar 29 2007 8:45AM | Permalink |Email this|Comments (21) |

All too frequently I hear how some group of people feel the only way to get their system to work properly is to start over from scratch. The most recent example of this was from the council for a large school district near where I live. While starting over from scratch may sound like a great idea, it is usually a poor choice for any system that has been operating for a period of time. I learned this first-hand early in my career while taking care of the interface and software that operated between the ground and flight equipment of an IR&D project. I shared my story in the current issue's "Tales from the Cube" article, "The smallest change necessary: Wise boss imparts important design principle," which touts the value of making only the smallest change necessary to accomplish your goal of changing your system's behavior.

The idea that doing a redesign from scratch is much better than using the existing implementation is flawed. The existing implementation has been tested under field conditions and contains fixes to real-life conditions that the system has actually had to operate in. How does starting over preserve those lessons learned?

Besides the very high chance of having to repeat hard-won lessons learned that are already captured in the existing system, the effort spent starting over from scratch is effort not being put into making the current system work better, adding new capabilities to the existing system, and keeping your product viable against your competitor's alternatives—especially if they have decided not to start over from scratch. Netscape's fall as the preeminent Web browser is partially attributable to the company's decision to rewrite the browser from scratch while its competitor, Microsoft, incrementally improved its browser.

The decision to start over from scratch is basically a decision to throw away many man-years of effort and lessons learned that the current system implementation encompasses. Aside from the different incentives between politicians and commercially competitive teams for wanting to start over from scratch, why would a design team spend more than a few moments even consider starting over?

One reason is that some designers may claim the legacy system configuration/code is messy because it is hard to figure out why the system is implemented the way it is—especially when you do not have the historical perspective behind all of the decisions that led to the current implementation. Taking this to the next step, how does rebuilding a system without completely understanding why the current configuration is the way it is result in a better configuration?

Here's a description of what engineering is that I use: Engineering is the discipline that enables a system to behave consistently despite having to operate over a range of conditions. When we design a system from scratch, we by necessity (to contain the amount of complexity we have to consider at one time) design the system to the most likely conditions. As the system operates successfully under those conditions, we allow the user of the system to try a wider range of conditions, and this is where and why the system configuration or code gets "messy"—to accommodate those variable conditions that are not compatible with the general conditions.

Other possible incentives for this bias to want to start over from scratch may be that our culture rewards the maverick that blazes a new trail and makes new ways of doing things possible. Another reason is that the technical labs in college usually give projects to students that start from a blank slate. Are we biasing our future design engineers to develop thought patterns that propagate working from a clean slate rather than making small changes to and improving on legacy structures?

Reader Comments

at 3/29/2007 1:17:22 PM, Ted said:
There are two classes of complex systems: Those that evolved from simple systems, and the ones that don't work.

at 3/29/2007 1:54:30 PM, Martin said:
Fred Brooks says in his Mythical Man Month classic "build one to throw away - you have to anyhow". Every time you build a system you have to know that you have to trash the first version. This should be part of planning. The problem is that many managers see the prototype and turn this messy hack into the final product. If you plan to redo then you can avoid having to start over from scratch - because you already did.

at 3/29/2007 11:57:20 PM, Tom said:
We had a system who's core design did not reflect the product's basic process and where some 50 parallel and undocumented tasks interfered. In addition the compliers were poor and unreliable. After 4 years of debugging the software manager quit and we could rewrite the code from scratch. Now we are successful again.

at 3/30/2007 8:09:22 AM, Rich said:
I agree with much of what you say, but redesigning a system from scratch can sometimes yield tremendous gains. It is vitally important to make sure first that you understand all of the existing system and why things were done that way (the historical perspective). What decisions were made because of technology limitations, what ''hacks'' were really due to previously unforeseen conditions, what decisions were made because of the capabilities of, or constraints put on, the original designers or the support personnel? Sometimes designs are perpetuated simply because everyone (or certain ''experts'') understand the quirks and redesigning may create chaos. If redesign is beneficial, build a platform that embraces future improvements and add-ons, don''t just create different quirks.

at 4/4/2007 1:19:15 PM, Laz said:
How was God able to create the entire world in only 6 days? He didn't need to be backward compatible. It can be risky to start over, but the rewards can be exponential instead of incremental.

at 4/4/2007 1:34:27 PM, David said:
Is far too simplistic to say, "make only the smallest change necessary." Sometimes the "smallest change" requires one to put everything on the table with an open mind willing to evaluate, keep, or discard anything. Often the existing design has to be discarded before one can discover what is possible. One has to discard Windows to find out what is possible in MacOS or Unix. The old design is never a total loss. Is still worthwhile as a learning experience. The world would be a poorer place if we never did anything that wasn''t based on the classes we took in school. Sometimes best to write them off as a learning experience and do something different. Use a technique that was not taught in school.

at 4/4/2007 1:50:04 PM, Gideon said:
I was tasked with maintaining and adding features to software that someone else wrote. After struggling with it occasionally for about three years, I finally convinced myself, my manager and our client that it would be best to rewrite it from scratch. The rewrite took about three weeks. I reduced the code size by a factor 9, reduced 40+ threads to just 1, and improved the performance by a factor > 2. It became much quicker to add new features and fix problems. Had I not rewritten the code, I would have wasted much more than three weeks struggling to fix bugs and add features to the old code. Reasons that the old code was so bad in this case: The roadmap was not clear to the original developer when he started; there were bad design decisions; the design was far more complex than required. Sometimes there are technical reasons for complex code, but other times the guy who wrote the code is just not very good at writing code or has done a very bad job. Reasons that the rewrite was so quick and successful: I had spent a lot of time trying to fix the old code, which meant that I knew the application very well; I had spent a lot of time considering the weaknesses of the old architecture and thinking about how I could have done better; and the new design was much simpler than the old.

at 4/4/2007 3:22:47 PM, Kris Heidenstrom said:
I agree with Gideon who had to rewrite the program from scratch because it was a mess. Robert Cravotta assumes that the original design and implementation were good. This is usually the case, which is why most development _is_ incremental and the developer recognises that incremental changes are best. That''s not always the case though. Recently I have rewritten from scratch two firmware projects that were originally written by a recently graduated programmer. He was inexperienced, and unfamiliar with real-time concepts, restrictions and techniques. The original implementation was unreliable and inconsistent - problems would show up, but would not reoccur when tested for specifically, and so were disregarded - and its behaviour was not documented. The original code was poorly structured, inefficient, minimally commented (most importantly, the basic design and structure was not documented at all), and very difficult to modify without breaking something (somewhere). In this case, most of the prior experience and bug fixes didn''t represent valuable insights into the nature of the features and behaviour needed

at 4/4/2007 3:27:25 PM, Kris Heidenstrom said:
... (continuing my comment which was snipped by the web site...) needed; rather they were patches applied to hold the whole mess together when we found that it didn't work as it should in some combinations of circumstances, due to poor initial design. I agree that rewriting from scratch is usually inappropriate, but it's also not usually even _considered_ by developers, so I think the article states the obvious. If a developer actually _recommends_ a complete rewrite, and not for job-political reasons, the recommendation should be taken seriously.

at 4/4/2007 10:22:03 PM, Sujit Liddle said:
Sometimes one is asked to add a major feature that is really difficult to implement using the band-aid approach. The more efficient and elegant your code is, the easier it will be to debug and maintain. The decision whether or not to redesign should not be taken lightly.

at 4/4/2007 11:15:58 PM, Jay Abel said:
Depends on the size of the system, small systems are usually better off when rewritten from scratch. As for large systems, I mostly agree; but I take exception to the notion that lessons are captured [only] in the product: Why in the hell aren't those lessons documented? If they are not documented, who in their right mind would want to use such a system as a starting point? Engineering is NOT about building one that works. It is about building thousands that work and being able to explain why they work to anyone who needs to know.

at 4/4/2007 11:46:52 PM, Shreshtha Kumar said:
This is one of the best article and discussion related to design I have gone through. I agree with the view of Robert Cravotta with assumption that the present system is designed an experienced designer. If the design is not good, than any patch on it breaks a block, get in between and holds using a feeble thread which can any time collapse. I feel whole discussion points to, design the system keeping in mind - “let this system evolve.” There has to be a long foresight to design a good system which itself is flexible and with lots of placeholder and nodes to attach future additions and changes seamlessly. Initially that design may seem bulky and un-optimized but that will help it to sustain worst practical adjustments. Had Gideon not understood the system, its flaws, and many hidden merits, I am sure the new system scratch would not have been so good. After all, he is comparing with the old system which whose merits have become obvious. As Rich said “redesigning a system from scratch can sometimes yield tremendous gains”, that may be true only when the new design encompass all the fixes and changes earlier system has gone under field conditions and that can happen when you understand the existing system completely. Its job of designer and code maintainer to add appropriate comments for any concept added. Any missed comment in the code is going to lessen - the life of that code and “concept reuse” in future design and even existing design. There cannot be a single instance, howsoever strong it may be, to support or flaw the topic. I think all the pros and cons should be considered in future design and better utilize the work from scratch for other system innovation rather than rewriting the existing system. shreshthakumar@gmail.com

at 4/22/2007 12:33:28 PM, Mohammed Hamed said:
One great insight I get from this article is the fact that colleges do not teach but starting from scratch. I think reverse engineering and the ability to gain an insight into an existing system are important skills for the engineer/developer and many professionals spend a great deal of their career doing only that. On the software side, I thin the open source movement is one great way to learn in that regard as new comers to the project usually have to spend some time to understanding the existing system. I certainly encourage college students to participate in the development of these projects.

at 4/29/2007 8:52:09 AM, Marri Srinivas Reddy said:
This is an important article for the designer/developer. I think the reverse engineering approch is good for learning phase and understanding the system. But not to start from the scratch. Offcoure may be required to develop the system from the scratch for time to market requirements,which you may not able achieve with legacy systems.

at 4/30/2007 11:05:34 AM, Jay Abel said:
Cheers to Mohammed Hamed, I think this point is very insightful and bears repeating. Colleges don't teach engineers how to systematically investigate a system to extract the valuable lessons encoded in its evolution. A phrase from Weinberg's 'Secrets of Consulting' is applicable to software development: Things are the way they are because they got that way. It means that when you see something that looks weird, don't just assume the programmer didn't know any better. It might be a signpost to otherwise invisible emergent complexity. Sometimes it's just lousy code, but unless you research it you might miss something important.

at 5/1/2007 1:11:21 PM, J said:
Making any generalizations on this topic is a bad idea - all generalizations are false. :) I've had to take over some designs that made me say "Wow! This is brillant!", and many more that made me say "This is nonsense!". In all cases, judgement needs to be exercised before a decision can be made about whether a design should be built upon or torn-down and rebuilt. (Generally) managers will prefer the former, and (generally) engineers will prefer the latter. Neither is going to be correct 100% of the time.

at 5/7/2007 10:13:43 AM, Tony said:
One approach is to try to improve the existing code dramatically - basically, an in-place rewrite. The buzzword for this is refactoring (e.g. Martin Fowler''s book Refactoring: Improving the Design of Existing Code). I suspect a lot of times it is a happy medium between rewrite from scratch and doing the minimum. It''s important to document existing behavior. One way to do that is through a well designed test system (unit tests and functional tests). Then the tests document the system - and makes refactoring much easier.

at 8/27/2007 9:29:43 AM, Bijoy Das said:
The basic issue here is re-engineering vs redesigning of an existing system. It all depends on actual situation how the system was designed. Documentation is one very important factor which is required to understand the design considerations of an existing product including its real life fixes. In most of the cases lack of proper documentation leads to a lot of confusions for which people find redesigning to be better than fixing the bug. After studying the existing system, one will have to make an estimate of the size and complexity of the change and accordingly take a decision for re-engineering or redesigning. Here again, the capability of the individual to understand the existing design is also important. Thus the whole issue is very subjective and cannot be taken as a verdict.

at 10/8/2007 1:50:25 AM, Willsey said:
Stop saying scratch.... =)

at 1/9/2008 5:02:24 PM, Marcos Bitara said:
How can we start a design if it already scratch, A good designer/architech not measure in term of scratch but in how she/he start from the clean paper to produce a best result. like what we read in genesis 1:1-31 In [the] beginning God created the heavens and the earth. See.. The designer in this holy book was not used any scratch.

at 4/30/2009 6:56:33 PM, Payday Loans said:
I found lots of interesting information on www.edn.com. The post was professionally written and I feel like the author has extensive knowledge in the subject. www.edn.com keep it that way.

Embedded Processing

Profile

RSS Feed

Recent Posts

Recent Comments

Most Commented On

Archives