Benchmarking Digital Signal Processors, do FPGAs cost more?

(This post is a sidebar to my 04/21/2011 EDN cover story: “DSPs Power the Race to 4G“)

When you speak to vendors about DSPs, they may refer you to their BDTI benchmarks and certification. BDTI (Berkeley Design Technology) founder and President Jeff Bier explains that the company provides independent, detailed hands-on technical evaluations and engineering services, focusing on DSP-intensive applications in embedded processing systems. Jeff emphasizes that BDTI does not focus just on the DSP processor, and that they evaluate the entire system of components; including FPGAs, CPUs, etc. Processors do make up one major category that BDTI evaluates, along with the associated tools and software.

BDTI’s quantitative evaluation techniques have resulted in the development of seven benchmark suites for DSP applications, with the mostly widely used being the BDTI kernel benchmark; a set of 12 DSP processing kernels for functions such as FFT, Viterbi decoder, and FIR (finite impulse response) filter. Companies that license BDTI’s benchmarks get access to the reference code, documentation, and test vectors to implement on their processor.

Jeff points out that DSP software is always optimized, unlike general-purpose software, such as for PCs. For results to be meaningful, benchmarks must be optimized in the same was as any DSP application code. You can’t just compile the code and run it like a PC benchmark. Vendors invest their own engineering resources to optimize the benchmark code, within BDTI’s specifications. Vendors will typically compare their results to competing processors, and often gain insight into where their designs need to be improved.

When a DSP manufacturer is satisfied with their benchmark results, the next step is to come back to BDTI for certification, which Jeff says is really an audit. BDTI will take delivery of a vendor’s tools, hardware, cycle accurate simulation model and documentation, etc., and examine their code line-by-line. They will make sure they can consistently reproduce the vendor’s results, and check that the code does not violate the BDTI spec. Vendors have the flexibility to optimize the benchmark code, but not to modify it to manipulate the results in their favor.

BDTI also looks at how well optimized the DSP code is, so as to avoid benchmarking the programmer more than the processor. Vendors use BDTI’s feedback to improve their tests, and when the process is done the results finally become BDTI-certified and can be published. BDTI distills down the 12 benchmarks into a single aggregated score; the BDTImark-2000. Cost and energy figures of merit are derived by calculating performance/dollar or per watt. For DSP silicon IP the value benchmark is based on performance per mm² of silicon area. For baseband applications, BDTI also has a communications benchmark based on a complete functional OFDMA receiver. The company is also working on a LTE baseband processing benchmark that they will release later this year.

BDTI has done benchmarks comparing high-end FPGAs to high-end DSP processors with the OFDMA benchmark. This was last done in 2007, and also in 2001. For FPGAs, BDTI had the chip companies do their own optimized designs, to “cram as many channels as we can into a single chip“, according to Jeff. Through this method, BDTI derives a cost/channel. Results were consistent from 2001 to 2007, showing that FPGAs have much higher capacities than DSPs, on the order of 30-50X.  Though a high-end FPGA is much more expensive than an ASSP, Jeff emphasizes that in infrastructure applications it’s not the cost of the chip but the cost per channel that is important. FPGAs came out way ahead on that score, he says, with much higher performance/dollar when the cost/channel is taken into account.

Jeff acknowledges that a disadvantage of FPGAs is that they are more difficulty to use than ASSPs. By requiring designers to work with HDLs (hardware-description languages), the total design effort is much higher. This is why vendors offer hardened IP and targeted design platforms to ease the design effort. In 2009, BDTI evaluated HLS (high-level synthesis) tools for FPGAs, and found that under the right kind of application you can get very good results. While you will still need to do some RTL/HDL design, for data paths the HLS tools worked well.

Here are links to some of BDTI’s recent benchmark studies:

1. Inside DSP article on BDTI Benchmarks results for TI ‘C66x DSP, TI’s newest processor with support for both fixed- and floating-point arithmetic, targeting LTE baseband processing. http://www.bdti.com/InsideDSP/2010/11/18/Ti
2. Inside DSP article on BDTI Benchmarks results for Freescale processors targeting 4G baseband processing. http://www.bdti.com/InsideDSP/2010/12/16/Freescale
3. BDTI’s analysis of a new CEVA core targeting 4G wireless applications. http://www.bdti.com/InsideDSP/2010/11/18/Ceva
4. Case study points out the way to meaningfully benchmark processors. http://www.bdti.com/InsideDSP/2010/10/19/CaseStudy
5. Description of BDTI’s DSP Kernel Benchmarks, most widely used benchmarks for comparison of processor performance on DSP applications. http://www.bdti.com/Services/Benchmarks/DKB
6. Description of BDTI’s OFDM Receiver Benchmark, a multichannel benchmark for evaluating a wide range of architectures for communications applications. http://www.bdti.com/Services/Benchmarks/OFDM
7. Results for BDTI’s OFDM Receiver Benchmark, showing comparison of DSPs, multicore and massively parallel processors, and FPGAs. http://www.bdti.com/Resources/BenchmarkResults/OFDM