Beware this integration nightmare

-January 12, 2016

Embedded software has come a long way in last 15 years let alone the last 40. While there are still many areas that developers and teams can improve in, firmware has for the most part moved away from single module monolithic programs with goto statements sprinkled everywhere. Code bases have become more organized and they are starting to take the shape of general purpose and object oriented software. Despite these advances, however, firmware integration may still be one of the biggest challenges facing software engineers today.

Most engineers attempt to reuse what has already been invented. If a library module exists, why rewrite it? Simply integrate it into the code base and have one less feature to worry about.

The problem is that the assumption many developers make -- that libraries or open source code follow interface and software design similar to their own code -- can be very far from the truth. Rather than speeding up development, then, integrating existing components could have the exact opposite effect and lengthen the development cycle instead.

Unfortunately, I recently encountered such a circumstance while developing a bootloader.

The ability to update a microcontroller's firmware in the field through a bootloader is a common and often critical feature of an embedded system. But it shouldn’t be an entire development cycle unto itself. Over the past five years I have written dozens of bootloaders for nearly every microcontroller and interface imaginable. All of those development efforts have resulted in a modular bootloader framework with well-defined interfaces and documented pre- and post conditions. Writing a bootloader for a new device has become, for me, extremely fast and efficient -- exactly what a developer would want in a common system feature.

But not this time.

As with all development nightmares, it all started with a simple request: integrate the client's OS and use their serial driver and protocol to speed up development. Normally the code space of a bootloader should be minimized but in this case code size wasn’t a primary factor. So, the request appeared to be no big deal. A quick examination of the serial driver gave the impression that it should be easy enough to integrate the driver into the existing bootloader framework.

Nothing could have been further from the truth.

The development effort started off being perfect. By leveraging existing templates and process, the effort jumped out ahead of schedule and was moving along at an amazing pace. Then we put the OS and serial driver into place and they seemed functional. The next step was to integrate the bootloader serial protocols with the existing serial driver. That is where the first integration issues arose.

The serial driver wasn’t really a driver. Instead, it was a driver with a serial protocol already built into it. But the bootloader framework already used a (different) serial protocol. So, Integrating the bootloader protocol into the customer's existing serial "driver" quickly became non-trivial.

In order to preserve the customer's existing protocol -- because it was, of course, a "must have" -- the serial driver now had to be modified. It needed to be able to decide whether the bootloader stream was being received or the original protocol was being received. This seems like a minor update and not a big deal, except that as it turned out the base OS code was meticulously balanced such that minor changes to the code resulted in the system crashing. Before we could perform the integration of a simple serial protocol, an entire code base now had to be debugged to determine why seemingly minor changes caused the system to throw a major fit. This is not a trivial endeavor when the developer's knowledge about the code base is minimal because the code was supposed to just work off-the-shelf.

Once we resolved the base code issues, we updated the serial "driver" to handle the bootloader protocol packets as well. One might think that the project would then move smoothly, but they would be mistaken. The well-tuned and oiled bootloader framework now required its own modifications to make its round construction fit into a square hole.

The integration of each bootloader feature ended up requiring a minor code change, but one that in turn resulted in an unexpected behavior or bug that had to be carefully tracked and resolved. Eventually the two code bases were integrated, but it took nearly two weeks longer than originally anticipated and generated more issues than one might wish to recall.

On the surface, the use of an off-the-shelf or open-source code base can bring a developer the anticipation of a stress free and shorter development cycle. But beware. Embedded software development that uses outside contributions is riddled with assumptions based on incomplete information and unfounded expectations about the quality and design of the received software and hardware. The result is that while most such development efforts at first seem trivial, the devil and the delays will be found in the details.

Is using that open source library really saving you time and reducing cost?

Jacob Beningo is principal consultant at Beningo Engineering, an embedded software consulting company. Jacob has experience developing, reviewing and critiquing drivers, frameworks and application code for companies requiring robust and scalable firmware. Jacob is actively involved in improving the general understanding of embedded software development through workshops, webinars and blogging. Feel free to contact him at, at his website, and sign-up for his monthly Embedded Bytes Newsletter here.

Loading comments...

Write a Comment

To comment please Log In