December 18, 1997

EEMBC: on the road to delivering
unbiased benchmarks

In the four years that I've been EDN's microprocessor editor, I've yet to see believable benchmark results for an embedded processor. Sometimes I wonder if I'm just being narrow-minded, but then, what processor company will admit that it had to manipulate the benchmark results to come out on top? The bottom line is that the embedded industry needs unbiased and meaningful benchmarks to evaluate the plethora of microprocessors and DSPs.

In the words of Jim Turley, senior editor of Microprocessor Report, "Designers need [benchmarks] to evaluate competing claims, vendors want them to gauge their own strengths and weaknesses, and analysts need them to draw accurate conclusions." Because of the lack of other benchmarks, designers, vendors, analysts, and editors (excluding me) rely on Dhrystone MIPS for a microprocessor's performance results. Although MIPS measures a system's integer and string-handling capability, it is a synthetic and extremely abused benchmark. When a vendor presents its processor's MIPS rating, you can always assume that the vendor performed the test under the best conditions. For example, I used a software simulator to run the Dhrystone MIPS benchmark on an anonymous processor. I discovered that with zero and three memory wait states, the processor delivered 159 and 109 MIPS, respectively. What number does the vendor report?

In April 1997, I organized the EDN Embedded Microprocessor Benchmark Consortium (EEMBC), with the primary goal of developing real-world benchmarks with precise rules for result reporting. These benchmarks, currently under development, consist of suites of tests. From a high-level perspective, these benchmark suites encompass applications in the automotive/industrial, consumer, networking, office-automation, and telecommunication industries. Within each suite, individual tests measure one or more processor functions, allowing you to determine which functions are appropriate for your application. For each test, vendors must report runtime characteristics that include compiler versions and switches, processor clock and bus speed, wait states, and cache size. Furthermore, the vendor must clearly document any code changes that it implemented to improve the benchmark performance; this documentation ensures that the exact test is repeatable and unbiased.

Unbiased benchmark tests are EEMBC's mission, and the consortium's membership of 21 vigorously competing microprocessor companies helps ensure unbiased tests. Some concern has arisen, however, that EEMBC's exclusion of tool vendors and developers from membership will lead to benchmark bias. For example, Lindsey Vereen, editor in chief of Embedded Systems Programming, worries in his November editorial that, without tool vendors to provide balance, microprocessor companies "can lobby to push through benchmarks that make their architectures look good and defend against those that make them look bad." Admission of tool vendors to the consortium, Vereen claims, would be a fair way of reducing this possibility for bias.

Although Vereen's viewpoint is perfectly understandable, it presumes a problem that doesn't exist. Considering that the consortium consists of 21 member companies, all of which have diverse opinions and equal voting rights, the benchmarks can hardly be biased. Furthermore, the admission of tool vendors or embedded-system developers to membership, especially with voting rights, might itself introduce bias by skewing results toward architectures with larger numbers of commercial adopters.

However, it turns out that this possibility wouldn't have been an issue anyway. EEMBC members are not fussing over architectural issues; instead, they have concentrated on defining benchmark suites that target the most popular embedded applications. In fact, it's already obvious that each architecture will have some winning and some losing benchmarks. Furthermore, the independent contractors that EEMBC is hiring to develop the benchmark code are absolutely required to generate architecture-independent code. RISC or CISC, big- or little-endian, the benchmark code will have no bias.

EEMBC strongly encourages tool vendors and developers to begin interacting with the consortium members. Give them your input. If you contact me at markus.levy@worldnet.att.net, I'll put you in touch with the proper person from the microprocessor company of your choice. Here's your opportunity to help influence the benchmark standard that will revolutionize the embedded industry's methods of evaluating microprocessors and DSPs.

EDN Access | Feedback | Table of Contents |