Forge ahead?
Targeting a preproduction processor for your application design may mean forging your own tools because you need them sooner than they will be available.
By Robert Cravotta, Technical Editor -- EDN, March 6, 2003
|
I have seen many presentations in which a vendor claims that its platform is ideal for handheld-application development that includes audio and video capabilities. During questioning, the conversation explores the vendor's development tools and support infrastructure, including third-party integrators and library providers. I have often wondered whether it is even half as easy as I am told it is to implement these capabilities on these platforms. Eventually, such wondering became the seed concept for a hands-on project: Would these development-support infrastructures allow someone without an expert knowledge of audio and video to include them in a design?
I approached Texas Instruments about using the Innovator development kit for the OMAP (Open Mobile Application Processor) platform to implement some audio and video functions from the perspective of an application developer. The OMAP5910 device, one of the processors that the Innovator development kit supports, is the first OMAP with general availability. It integrates an ARM925 microcontroller core, a TMS320C55x DSP core, and shared on-chip resources and peripherals into a single device. Texas Instruments claims that ARM developers can use the device with the operating systems, software-development tools, and standard APIs (application-programming interfaces) that they are familiar with, without an intimate knowledge of DSP programming.
One key assumption that aims to allow application developers to develop with a DSP without learning the DSP internals is that someone else writes the DSP code and provides the interface code to interconnect that code into their preferred APIs. Looking at Texas Instruments' network of independent OTCs (OMAP Technology Centers), which provide system-integration expertise; the third-party-network catalog offerings for compliant algorithms; and the way the DSP/BIOS Bridge works suggests that this goal may be realizable. The DSP/BIOS bridge is the interconnection mechanism between the cores for OMAPs, and it places the DSP in a secondary or coprocessor capacity (see sidebar "Bridging between processors").
Why consider a dual-heterogeneous core device for audio and video when the same capabilities exist in single-core microprocessor systems, and programming for multiple cores, especially heterogeneous ones, is more complicated? Power consumption for handheld applications is one reason. An application that partitions well between the cores can allow lower clock rates, take advantage of the DSP's computational efficiency, and allow support for more features than comparable single-core systems. However, developers tend to choose devices they are familiar with for new designs (Reference 1). Texas Instruments' development-support infrastructure is one strategy that makes OMAPs more accessible to traditional microprocessor developers.
A primary goal of this project was to put this development-support infrastructure—which includes the OTC, third-party networks, and Web resources—through its paces. Another goal was to determine how much effort adding recording and playback support for audio and video to an application would require—especially because I have little direct technical experience with these capabilities. If adding them proved as easy as, say, implementing them in a single line or small block of code, I would try more interesting tasks, such as merging and overlaying audio over a recorded movie.
A significant detail changed this project from a pure exercise in implementing audio and video capabilities to what I have dubbed an early-adopter exercise: The OMAP5910 and tools were still in the sampling stage and in preproduction testing during the development of this article (see sidebar "Sampling versus production"). Therefore, some components, such as the video driver, might be unavailable before the project's conclusion. Nonetheless, I believe that the proposed project, even with a smaller set of technical goals, would still be valuable to you.
The setup
WinCE .NET (WinCE 4.1) made the most sense as the operating system to accommodate the small window for this project, because it was probably the easiest operating system for me to get up to speed with and because of its support for audio and video. Selecting the audio and video codecs, AMR (adaptive-multirate) and MPEG-4, was somewhat arbitrary, because my main goal was not to implement for maximum performance, compression, or quality, but to explore how to implement audio and video manipulations through the DSP without detailed experience.
Because I would be working with members of Texas Instruments' OTC companies and third-party-algorithm providers, a key element for moving this project forward was establishing a working relationship with them. Stellcom, an OTC company, acted as an integrator, providing consulting support for development and application questions that would arise during the project and integrating the operating system, board-support package, and DSP/BIOS bridge that would run on the Innovator. Stellcom also built and provided the software-development kit necessary for the ARM microprocessor core. Emuzed, a third-party algorithm provider, contributed the DSP algorithms that interfaced with the DSP/BIOS bridge for encoding and decoding AMR and MPEG-4.
The Innovator development kit is a handheld, expandable development platform that comprises a processor module, an interface module, an expansion module, a camera module, and an LCD/touchscreen panel (Figure 1). A breakout board accommodates these modules and provides Ethernet, keyboard, and mouse support (Figure 2). Besides the diagnostics bootable kernel, the kit includes two bootable user-flash areas. The boot sequence for this project began with Microsoft's boot loader in one user-flash area and completed by loading the WinCE .NET image from the other user-flash area. I updated both flash areas several times throughout the project because each version integrated more capabilities into the system image, through a USB link with a host-boot utility on the desktop.
I did not need to use Texas Instruments' Code Composer Studio because I was not developing any DSP code; the DSP code was coming from Emuzed. I would be able to load the DSP code via cexec, a WinCE console application that is available on the microprocessor core as part of the DSP/BIOS Bridge component. I did not need to use Microsoft's Platform Builder, because Stellcom was using it to roll a software-development kit for the Innovator development kit. Instead, I needed to use only Microsoft's EVC/C++ 4.0 (Embedded Visual C/C++), along with the software-development kit from Stellcom, to develop the application code.
Putting it all together
Because it was a work in progress, the entire development environment did not arrive in one box. It came in pieces and from different sources. I received the Innovator kit from Stellcom after the company had loaded the boot kernel and initial operating-system image. The CD that came with the Innovator kit included the boot-host utility for subsequent updates to the user-flash areas. I downloaded the ActiveSync and EVC/C++ tools from Microsoft's Web site. ActiveSync allowed me to work directly with the Innovator kit, including single-stepping through the code, from the EVC/C++ environment through the USB port. It would be sometime later that I would receive the DSP algorithms from Emuzed via an FTP site that Stellcom set up to facilitate transferring the latest version of the software-development kit and operating-system image.
Unfortunately, EVC/C++ requires Windows XP or Windows 2000, so I had to upgrade my workstation operating system. It took two operating-system-upgrade attempts to get my workstation fully functional. The reinstallation was predicated by a noncooperative virus-protection package that apparently did not completely uninstall itself before the upgrade. Fortunately, uninstalling the operating system restored the workstation to its former condition, and I was able to find the manual procedure for completely removing the virus-protection package. If you ever find yourself performing an operating-system upgrade, check the Web site for any applications, such as virus-protection software, that might have a tight coupling with the operating system. Simply uninstalling the software via the add/remove programs icon in the control panel may not be enough.
Products go through a preproduction phase for a reason. There are probably many functions that developers have yet to completely implement, interface drivers that do not yet include robust error trapping and recovery, or even capabilities that developers have yet to test. I did not always know where to find the documentation, and I sometimes did not receive some of the documentation right away, specifically for the DSP/BIOS bridge. Initially, this lack of documentation was a source of frustration. As I quickly gained experience with the system, however, my frustration lessened, because I better understood how the system operated. Going to a workshop or class for the platform was unnecessary but would have been a good idea.
We quickly determined that e-mail was not a good way to transfer file updates, because the files were large—some as large as 50 Mbytes. Even when the files were small, if they resembled executable files, the antivirus scanner software on the e-mail server stripped them out. Instead, we used an FTP site. I began having problems accessing the FTP site when I tried to download the second update. It took a few tries and a little longer than a week to resolve things. Fortunately, we were able to rely on overnight "sneaker-net" to move the update files around on a CD.
Connecting over the USB port did not work for the first system image, but the second system image addressed it. Because I was not working directly with any of the hardware, I did not need to use the breakout board. Assembled, the Innovator kit looks like a PDA, so I made the mistake of thinking it would function like a consumer product. This assumption bit me when I tried to make my first USB connection with ActiveSync. I connected the USB cable to the Innovator kit and turned on the power, just like I might do with any consumer product. Instead, I needed to power up the Innovator kit and let the operating system complete loading before connecting the USB cable. The development power-up sequence differed from a production power-up sequence in that it relied on the operator to manually ensure that everything initialized properly and in the correct order before linking the two systems together.
When you are working with other teams, it is a good idea to know what software, including the version, each is working with. I was trying to use the latest version of the boot-host utility on the CD in the kit to update the user-flash areas, but it was not working. After speaking with Stellcom representatives, I found out that the company was using an earlier version. After I received that version, I was able to update the flash. From then on, I checked the time stamps on all files to identify discrepancies and avoid version incompatibility. Disconnecting another tool involved installing the software-development kit for EVC/C++. Because I was using EVC/C++ instead of Platform Builder to build my applications, I did not have all the files that Platform Builder installs. When I received the first software-development-kit-installation file, it did not properly link with the EVC/C++ environment, because it assumed that certain files were already on my system. When building or receiving software-development-kit-installation files, determine which assumptions about the target system the install wizard uses, so that you deliver or receive all the necessary files.
When I received the first iteration of the DSP algorithms from Emuzed, I found a capability that had not yet been tested—namely, the DSP/BIOS bridge. The system image had so far not included it. This discovery caused some heartache. Adding the bridge required the removal of certain software-development-kit components, because the system image was already straining the available flash storage. The final version of the user-flash image was almost 15 Mbytes, and the runtime footprint was just less than 2 Mbytes.
The ideal picture of DSP transparency in OMAP-platform development for using multimedia in a WinCE .NET environment is that the application developer has to work only at the high-level DirectX API. Therefore, any DSP code must also include applicable filters or wrapper filters that reside on the microprocessor side of the device. It must also register the filters and DSP codecs to the operating system, allowing application developers to use the DSP codecs in the same as way as they would use any normal codec installed on the microprocessor. Because this hands-on project used tools that were still works in progress, we could not realize this scenario.
Only the audio driver became available during the project window; the video- camera driver would not be available until after the project concluded. Therefore, video capture would have to wait until the driver was available, or I would have to write directly to the low-level control for the camera from the application code. Once I realized that synchronized video and audio would be such a complicated task, I chose to focus on implementing audio. Unless a desire to be the first to market drives your project schedule to conflict with driver availability, there is probably little return on the resources you'll invest to directly drive the camera interface. You will not likely have to make the same trade-off when using production devices and mature development tools.
I was however, able to use the audio driver. Implementing .wav audio was a straightforward process. The Waveform Audio API recognized .wav files, because the filters for .wav files were part of building the operating-system. Using the AMR encoder and decoder was a different story, because the filter support that would normally be available with a production package was not yet ready (see sidebar "Filters"). When the filter support is available, you can use the simpler, synchronous, high-level, multimedia streaming APIs to manage the resources. However, in this case, the lack of filter support meant that to implement AMR support, I would need to create my own filters or more directly manipulate the data in the application code.
Using the Waveform Audio API was straightforward and allowed me to insert DSP/BIOS-bridge calls as necessary. I needed to add a few functions calls to create, delete, and execute the DSP nodes for encoding and decoding. The application code handled capturing the data, routing it to the encoder, storing the file, routing the stored file to the decoder, and, finally, playing it back (Figure 3). Other than the DSP-node create- and delete-function calls, there was nothing to indicate that the DSP and not the microprocessor executed these functions. Effectively, I manually implemented a custom filter graph and filter-graph manager in the application code. Taking the next step to convert the code into a set of filters that could register themselves with the operating system would allow other applications to use the new media format.
Reusing software across many projects continues to grow as a strategy for completing more complex software development in less time. This reuse is more complicated for multiprocessor applications and even more so for heterogeneous-core designs. The DSP/BIOS bridge reduces the interconnect variability that third-party vendors must support when they offer algorithms for such devices—increasing the opportunity for developers to use these algorithms across many projects. On a whim, I called an ARM-tool provider that was not listed in the Texas Instruments marketing material. The company had recently begun exploring how to support the bridge with its operating system and tools, because its customers were asking it to.
If you do build your own tools and drivers because they are not available when you need them, be sure that you build them so that you can use them across multiple applications or that you can easily port to the production instantiations when they are finally available. Implementing the peripheral control and filter graphs as application code is quick and dirty and may support quick turnaround for your current project, but it may complicate your effort during the next-generation development. Unless you want to be in the business of updating your drivers, you will want to know how to port to the production-supported drivers after they become available.
This hands-on project was made more challenging because it was a part-time activity and I am not an expert Windows programmer. I could not apply my own expert knowledge to develop missing or late drivers or filters as easily as someone doing a true development project using preproduction silicon and tools might be able to. However, this project did illustrate some of the wait-for-or-build-it-yourself trade-offs that design teams must make when choosing to use preproduction products and support.
I can see how a microprocessor-biased application developer can be spared of having to learn the intimate details of DSP programming when using a production-supported OMAP. I was able to demonstrate how to offload the audio encoding and decoding to the DSP while working entirely with the development tools that a Windows-application developer would normally use. Unfortunately, due to time and complexity constraints, I did not get to implement the more interesting data streaming, transformations, and synchronization that I had hoped to, because all of the support software necessary to simplify these capabilities was not yet available.
Implementing audio and video support in an embedded system is a less mature process than implementing them in a desktop environments. Embedded systems, by design, are more resource-constrained than desktop systems, complicating the implementation and support of audio and video. The operating systems for these systems generally cannot afford to include the same rich support for audio and video that desktop systems enjoy. However, I do see processor vendors and their development-support infrastructures making headway to bring these capabilities to handheld systems. This project would have been very different had we done it six to nine months later, but then, those few months can mean the difference between being the first to market and playing catch-up.
| Reference |
|
Author Information
 Before undertaking this hands-on project, Technical Editor Robert Cravotta had never implemented multimedia support in an application. You can meet him this April 22 to 26, 2003 at the Embedded Systems Conference in San Francisco or reach him at 1-661-296-5096, fax 1-661-296-1087, e-mail rcravotta@edn.com. |
For more information...
When you contact any of the following manufacturers directly, please let them know you read about their products in EDN.
Accelerated Technologies
1-251-661-5770
www.acceleratedtechnology.com
ARM
1-408-579-2200
www.arm.com
Emuzed
1-510-252-9637
www.emuzed.com
Green Hills Software
1-805-965-6044
www.ghs.com
Microsoft
1-425-882-8080
www.microsoft.com
MontaVista
1-408-328-9200
www.mvista.com
Productivity Systems
1-972-479-9484
www.prodsys.com
Stellcom
1-888-554-2024
www.stellcom.com
Texas Instruments
1-800-336-5236
www.omap.com
Wind River
1-800-872-4977
www.windriver.com
