Post a CommentFeature
Avoiding pitfalls in managing embedded systems projects
Budgeting, scheduling, design, and implementation are a few of the issues that the project manager must successfully control to insure success on any embedded development tasks.
By Timothy Stapko, Digi International -- EDN, 2/12/2008
Project Management has become so important across a multitude of disciplines that it has achieved the status of a discipline in and of itself. Some of the problems encountered in project management are common to any organization regardless of the project being executed, but because the project manager is closely involved in all aspects of the project it can really help if he or she has some mastery of the relevant technologies and processes involved in the actual implementation of the project. This becomes especially relevant in technology-based organizations as projects often involve the development of new products, and it is very difficult to keep accurate track of progress without at least a working knowledge of the pertinent subject material. In our case, we will be discussing embedded systems projects and some common pitfalls to identify, plan for, avoid or fix. Specifically, we will be discussing projects in an industrial, private-sector setting where project management is often loosely defined. Some organizations may have more specific rules about how projects are to be managed, either by company policy or to meet certain regulations. However, some of the pitfalls we discuss will be applicable to any embedded systems project, so even project managers with strict management requirements may benefit from the material.
So just what is an "embedded system"? For our purposes, we will assume that an embedded system is any technological product that includes some type of microprocessor-based "smarts" dedicated to a particular task. Some examples include irrigation systems controls, building access control systems, missile guidance systems or even components of larger systems like processor modules or networking cards. We will assume that these systems include both hardware, and software/firmware implementations.
Phase 1: Requirements Gathering
The first part of any project is the requirements gathering phase. This is the most important and probably most overlooked phase of any project. Simple design mistakes can cost thousands of dollars toward the end of the project, and ambiguous requirements can derail a project entirely. When in the requirements gathering stage, it is vital that the project manager take ownership of the project right away. This ownership includes the gathering of stakeholders to hash out the expectations for the final product. Gathering requirements is the single most important activity at this step, and it cannot be overlooked. Most useful is a single document, often called the project charter, product specification, or requirements document that includes all of the expectations and requirements for the end product (not to be confused with the design document, which comes later and describes the actual implementation). In order to assure a smooth project, every detail that is important to a stakeholder must be recorded in the requirements document in unambiguous language.
One easy way to make sure the requirements in the specification are unambiguous is to use simple language that is purposefully defined within the same document. Strong, simple words like must and should are useful for this purpose. As an example, the specification could define "must" to mean "all requirementsincluding the word 'must' cannot be modified or omitted without explicit consent from all stakeholders." Defining this language up front and using it consistently will help to remove ambiguity in the requirements document (which admittedly cannot be removed entirely since everything is up for interpretation). A real-world example of this method in practice is evident in the Request For Comments (RFCs) documents that are used as standard implementation guidelines for Internet and networking protocols (see http://www.ietf.org/rfc.html for examples).
So for an embedded systems project, what requirements are important? Well, there are the obvious bullet points of "what does it do" and "what is the cost target" but there are some that may not be so obvious. These non-obvious requirements may include things like the placement of connectors, the orientation of parts on a circuit board or even the color of the printed circuit board. Marketing professionals and customers can come up with some outrageous requirements that can be deal-breakers, so be sure to include marketing and customer representatives in the requirements gathering at some level. For the software or firmware, an API should be included, not with any great detail, but at least an overview of the functions that will be included and a brief description of each. Any novel or complex algorithms that are identified as being necessary for the product should also be included as well as an overview of the desired functionality for the software in general.
Once you have the requirements document, you may think that the design is the next step, but it is important to get a final signoff on the requirements. Get all the stakeholders in a room (preferably only for an hour or so at a time, but as frequently as possible until you are done), and go through the requirements document. You will know when the requirements are good to go when everyone starts complaining about your continual meetings – this means that there is nothing interesting happening anymore so no changes should be necessary. The last thing is to get all the project stakeholders to sign off on the requirements document, formally or informally. This way, any issues that arise later can be solved by showing the appropriate person where they signed off on the feature in question. Changes to the requirements document should always go through a formal review with all stakeholders and the revised document should be signed off as in the beginning.
Ideally, the requirements should be signed off before the initial design, and the schedule and cost estimates for the project should come after the initial design work (roughly 10 -15 percent of the project cost is a sufficient amount for the initial design for a lot of projects). However, this approach is often at odds with the wishes of many stakeholders who would like to know the scope of the investment required for the project. This leads to another basic pitfall of managing technology projects in general – the development time and cost for any sufficiently complex system is nearly impossible to predict without doing a significant portion of the work. If the stakeholders will not allow a project to move forward without "accurate" up-front estimates, either try to get a research project approved (30% of the total initial estimate is a good rule of thumb) or warn them that the project costs are likely to be very different than your initial estimates. As an example, it is highly unlikely that you can accurately estimate a $500,000 project with only $50,000 (which is probably enough for the initial design), but with $150,000 you will likely be far enough into implementation to have identified the most significant risks and the problem areas with regard to the design of the product and the staffing of the project. At that point, if you have been diligent in tracking issues and effort, your estimates should be significantly better, and you should work with the stakeholders to revise the project goals with your new numbers. This may cause the project to be cancelled, but it is better to cancel a project that is not worth it as soon as possible rather than keep working on it until the money runs out. Always keep in mind that any project may be subject to cancellation, and do not let a doomed project continue. Too often a project will have a powerful backer in your organization that may not realize the project may simply not be worth it, so keep an open mind and try not to get too emotionally attached to whatever the project is – your job as a project manager should be to point out all the flaws that might make the project a poor investment and let the stakeholders decide whether or not to continue. If you are both the manager and a stakeholder, try to be objective when looking at the numbers. It is important to believe in the project, but it is equally important to realize when the project cannot succeed and be willing to end it (this is really hard when you have to disappoint a large number of people which makes it even more important to develop accurate estimates as soon as possible).
Now we move on to the meat of the project: design and implementation. For a sufficiently complex product, there will likely be multiple teams concentrating on different aspects of the implementation, and the project manager is responsible for making sure those teams communicate often. Perhaps the single most dangerous piece of managing an embedded systems project is lack of communication, especially between hardware teams (luckily, though it is more expensive to develop, software is much easier to adapt and fix in many cases). The project manager should schedule regular meetings with the entire team (not long, maybe half an hour but at least once a week), and with any functional managers that may be involved. The most important tool other than the facilitation of inter-team communication is the design document. Unlike the requirements document, the design document should include every tiny detail of the implementation that is necessary for functionality. The design document should outline the placement of components in a circuit (including those that are specified in the requirements and those that are not), the relevant values or tolerances of those components, descriptions of software API, lists of source file names and organization, and compatible hardware and software needs such as driver support. Hardware and software should be described in separate documents, and each of those may consist of multiple documents themselves. For a circuit, the design document may be as simple as a folder of CAD files showing the circuit schematic and component placement. Software design documents should include relevant source files, special coding requirements, the language or languages to be used, and code snippets for particularly complex implementation details. The design documents should be kept in a centrally-located, easy-to-access location and should be kept up-to-date as the implementation continues (a wiki on a local server is a good idea). Not only will the document help keep your teams on task and help to prevent mistakes, but it can also serve as a primary reference for later maintenance.
During the design process it is a good idea to develop mock-ups and prototypes of any of the tricky parts of the hardware and software components to verify that the concepts are feasible. A big pitfall (and a temptation for many engineers) is to do all the design on paper and assume that the implementation will follow suit. However, since the final product must function outside the virtual world of abstract design, there are some features that may not work as expected and certain designs that may be physically impossible. Nailing down these difficult and novel parts of the design early will be of tremendous help during implementation, and it will make estimating the project expenses and schedule a lot easier. This prototyping is part of the reason so much money needs to be spent up front to determine the estimates within a reasonable certainty because each engineering project is (basically by definition) a new and unique experience. If you are building widgets on an assembly line that are all identical, it becomes almost trivial to measure the amount of time each widget takes to manufacture, but engineering a new product from the ground up is never identical to the last project. This inherent property of engineering and technology means that we can never achieve truly accurate estimates without significant work – the best we can hope for is a decent approximation of expenditures by comparing similarities between the new project and existing projects. If you don't have prior experience with any similar projects, you may as well roll some dice to determine expenses and schedule if you don't do enough design research up front. For this reason it is of vital importance for a project manager to track each project as precisely as possible with as much detail as possible so that future projects may be better estimated with the library of collected data. Without this vital tool, any project manager will be up the proverbial creek without a paddle.
Phase 5: Revising the Schedule and Budget
Once you have the design documents mostly complete (and hopefully a number of past projects to measure against) you can finally revise your schedule and cost estimates. This is where being objective can really help a project. Early successes or failures in the design process can give a false sense of how the project may go in the future, and this must be taken into account. If you have prior project data, then you can check your estimates against similar circumstances for other projects, or you can use the experiences in the prototyping and design phase of the project to gauge how long something will take (and therefore how much it will cost). Engineers tend to be a fairly optimistic bunch when it comes to their own abilities, so a healthy dose of reality must be used when analyzing estimates from your team and real-world data provides that insight. For example, if implementing a particular circuit took four weeks in a previous project, but an engineer tells you he can build a similarly complex circuit in two weeks, then it may be tempting to trust the engineer's judgment and go with the more aggressive estimate. The problem is that there are always problems that no one anticipates and simply assuming the engineer is right could lead to a major problem later. Using four weeks in this case is probably a better choice in order to account for enthusiastic team members. However, knowing your team can help in this situation as well. If you have information about the accuracy of each team member's estimates on previous tasks, you can determine whether or not each person is capable of achieving what is promised. The tendency to over-promise and under-deliver is prevalent in the fast-changing technology industry and it is important to get a clear picture of the project to make sure the investment will pay off in the end. Tracking how individuals perform is a useful tool for estimation since it is highly unlikely that an engineer's performance will improve significantly between projects so you can adjust your estimates accordingly. Having this data will also make it easier to convince your stakeholders that your estimates are the best you can provide since a large cost and long schedule can easily be interpreted as an attempt to sandbag the project to make yourself look good. While being over budget and over schedule is a major issue, another peril of estimation is when stakeholders think certain tasks should not take as long as you suggest and accuse you of inflating numbers to make your own life easier. Having data from both previous projects and from the design and prototype phase will make your argument stronger and help to convince naysayers that what you provide is the most accurate estimate possible.
At the point when the design documents have been completed, the prototypes verified, and the stakeholders provided with your most accurate estimates for cost and schedule, you can finally start on the implementation. Unlike the design phase, where the team has loose goals and a rough plan, the implementation phase has detailed instructions on how to proceed with the project. At this point, staffing the project takes on significant importance since it will determine the cost and schedule. If there are too many people on the team, communication and overlapping jobs will kill your budget. Too few and the project will take forever. You have to use an assumption of resources in the initial schedule, but you may not get what you ask for, so keeping an eye on how your team is doing is very important. Weekly status meetings (keep them short!) and detailed tracking will allow you to see when you get off schedule.
Pitfalls of Project Management
Technical challenges can derail a project faster than just about anything, depending on their severity. Most of the major design issues can be avoided with sufficient prototyping and design review, but some things don't show up until you try to make the whole system work since the problem is in how the various subsystems work. When you are dealing with components for the hardware, make sure at least one team member becomes an expert (or is already an expert) on the specifications and requirements for each component. Some components may have tricky requirements that may not be well documented – the author has experienced this personally where a vital component requirement was documented in a single footnote of the part's specification. In another case, a vital step for initializing a component was not documented anywhere and was illustrated only in a small timing diagram. In both situations, these minor issues delayed the release of a product by several weeks or months. This is precisely the type of situation that is impossible to predict, and why it is vital to do as much prototyping in the design phase as possible. Other than that, an obsessive-compulsive attention to detail (combined with ample experience) is the only thing that can save you from these types of problems. Acknowledging these issues exist, keeping an eye out for them, and identifying and resolving them quickly will make the project easier and will be a great help in keeping the project on schedule.
Other issues with hardware have to do with the fact we are building products that will exist in the real world. Certain components (i.e. anything to do with analog) are sensitive to seemingly innocuous board features. In one project the author was involved in, the use of a circuit auto-routing program caused some traces from a high-speed SRAM component to route near to the inputs of an analog-to-digital converter. The interference from the SRAM lines caused the ADC to be practically unusable. A tool used to save time actually resulted in a massive delay in a product release, leading to a mandate to always route analog circuitry by hand instead of letting a program do it. Some engineers may be aware of such issues and be able to account for them, but others (particularly those fresh out of school) may not have the experience to identify those types of problems. For this reason, always solicit advice and guidance from your organization's most experienced employees – a simple bit of advice early on can save you tremendous headaches later.
Software and firmware in a project provide additional challenges above and beyond the hardware. Software being "soft" is inherently more difficult to specify than hardware. With a hardware design, an engineer specifies the values of every component and has complete control over the placement of components in a circuit. Sure, there are challenges in dealing with physical limitations, but overall the requirements for hardware are often much simpler than the requirements for software. The inherent problem in developing software or firmware is that there are no physical limits as there are in hardware. With the wide-open nature of software development, it is vital to have a detailed plan and define the desired functionality as precisely as possible, even for a small firmware application. In the software design document, the entire API, along with expected limits on all inputs and outputs, should be defined and broken down in a logical manner. Breaking up software into multiple functional components is always a good idea, and diagrams and flowcharts showing the organization and control flow of the program can help immensely.
As for the implementation of software or firmware application, there are a number of established models for software development. Studying the different models can help a project manager in determining the best course of action for proceeding on any project. For example, an application that has stringent quality requirements (such as for telecom or military products) might benefit from the "waterfall" method of software engineering, where design and implementation "fall" logically from the requirements and any problems result in moving back up – this method takes a long time but helps to guarantee that all the bases are covered in a design. For an application that has a tight time restraint (such as with consumer devices), a different method will likely be better. The so-called "extreme programming" methods usually involve the development of test programs early on and pair up developers to catch and deal with problems quickly.
The implementation of software or firmware is necessarily accompanied by testing. One big pitfall in development is when a software engineer implements and unit tests a feature in isolation. In the confines of the environment created by the implementation engineer, the feature may perform flawlessly, but when integrated into a larger system or executed on actual hardware, the feature may cause failures or unexpected behavior. Given the fact that defects typically cost considerably more if they make it into the testers' hands, it is vital that software engineers work together and have accurate hardware on which to test their work. Integration of software components and hand-testing of software on "real" hardware before handing the application off to test should be vital components of any embedded systems project.
Testing is obviously the final piece of the project, but this may involve any number of actual activities. There is the obvious testing of the product for functionality, but with embedded systems there are often a lot more details that need to be covered. Temperature and environmental testing are of vital importance for applications that will live in harsh industrial or outdoor locations. There may be regulations that require certain certifications in order for the product to be salable. It can be even trickier if the application includes a wireless communications element, since intentional radiators must often adhere to strict governmental regulations (enforced by agencies such as the FCC in the US and CE in Europe). Some regulations may result in design changes (such as the strict wireless requirements for Japanese certification) and must be accounted for in the initial design. For this reason, it is of vital importance to identify any and all potential markets for your product at the outset and include all design considerations in the initial design and requirements documents. Even if there is no plan to sell the product into a particular market, it cannot hurt to keep regulations in mind since you never know what might happen later. Anticipating the possibilities that sales and marketing representitives might throw at you later can make you into a hero and ensure that your project is a success.
In conclusion, embedded systems project management is a difficult and detail-oriented process that is different every single day. This article has covered a few of the major issues in developing an embedded systems product, but there are obviously hundreds of issues that can be discussed that are far beyond the scope of this project. Only through being involved and learning everything (and I mean everything) about a project can a project manager truly own his or her project. With embedded systems, it helps to have a background in both hardware and software – you don't need to have college degrees in order to be successful, but furthering your education in those subjects will definitely make your life easier. If you can learn to identify problems independently from your engineering team, you can really make a difference. As a final example and overall summary, the outline below shows a basic breakdown that should work for many embedded systems projects. You don't have to use the numbers as shown, but having this type of structure in mind will definitely help in keeping your project on track – if you find that any single part is taking more than its share of cost or time, then you know it is time to rethink your estimates. Remember that vigilance and attention to detail are your most important weapons in combating project failure – good luck.
A sample project breakdown:
5 percent - preparation and requirements gathering
10 - 15 percent - design work (may go up if the product is conceptually difficult)
50 percent - implementation (may be less if design is harder)
30 percent - testing (may go up or down depending on quality requirements)
| Author Information |
 Timothy Stapko is a lead software engineer and project manager for Digi International with focus on the Rabbit line of embedded products. He earned a bachelor's degree in Computer Science and Engineering and a master's degree in Computer Science from the University of California, Davis. Stapko has more than 8 years software industry experience and is the author of "Practical Embedded Security." You can contact him at timothy.stapko@digi.com |
