"The outrageous untruths are gone since PREP benchmarks came out. It took 11/2 years to get everyone to agree on how FPGAs would be measured. They're not perfect, but they did the job," says John East, CEO of the world's third largest FPGA manufacturer, Actel.
East has reason to be proud of how his industry group, PREP, a nonprofit consortium of 14 companies, mostly FPGA houses, managed to hammer out its intracompany differences to create an honorable FPGA benchmark.
Sure, marketing managers from the FPGA companies, armed with the techniques satirized in that MBA-school, must-read classic, How to Lie with Statistics, still try their best at manipulating the publicly available PREP benchmarks. But, the truly near-criminal FPGA marketing lies/advertisements have stopped.
This forced the poor FPGA vendors to concentrate on actually providing measurable value to keep their customers. It's been painful, but, as a result, FPGA customers are continually offered better FPGAs that have higher speed, more capacity, and low prices.
Encouraged by this success, PREP decided to create a special subcommittee called Working Group 4 (WG-4) chartered with the controversial task of benchmarking the synthesis tools used to design these newly available high-capacity FPGAs. Little did they know what resistance they'd get!
EDA-vendor reservations
"They're trying to get at synthesis benchmarking on unrealistically small, contrived designs that aren't representative of the top-down design process," says Alain Hanover, CEO of ViewLogic, a PC-oriented EDA vendor. "In place of using single, tiny data-path, state-machine, and combinational-logic subdesigns as a basis of synthesis benchmarking [which was what WG-4 considered], real-world designs are actually far larger combinations of these subdesigns. Synthesis is a very subjective thing to try to measure; it depends on your approach. Instead, I'd like to see benchmarking of top-down design processes where they measure results a designer gets working from a specification, writing Verilog or VHDL, and then going through a suite of tools to get to his final design."
Facing this resistance from the EDA-synthesis vendors, WG-4 originally wimped out by disclosing that it did not plan to run FPGA-synthesis benchmarks or to publish any results. Instead, WG-4 planned to mimic the benchmarking cop-out that ACM's SIGDA offered some time ago: Present a collection of publicly accessible designs and let the customers perform their own "at- home" benchmarks. (That is, don't report any results.)
The WG-4 designs were originally only in VHDL. WG-4 would also provide test benches so users could modify the VHDL to fit the vagaries of each brand of FPGA-synthesis tools being tested. (VHDL is not one hardware-description language, but a collection of very closely related hardware-description languages.) The FPGA-synthesis vendors liked this compromise/capitulation offered by WG-4 because there would be no benchmark reporting and the user would be forced to look at each FPGA-synthesis vendor's product--enabling the vendor's sales staff to sweet-talk a sale. It was business as usual.
Then the press had its turn. Some of the PREP representatives from Synopsys, Exemplar, Xilinx, and Altera tried to defend the "synthesis-is-too-complicated-so-let's-not-benchmark-it" stance. But, EDA troublemakers Mike Dini, Sean Murphy, and I chanted a collective "Wimp! Wimp! Wimp!" in the EE Times article that announced the WG-4 cop-out. This, plus quite a few other people's behind-the-scenes actions encouraged WG-4 to successfully brainstorm ways to solve their FPGA-synthesis benchmarking issues.
WG-4 revitalized
In place of the using the simple step-and-repeat of simple circuits that didn't give many FPGA-synthesis tools a chance to shine, John Birkner, chairman of WG-4 and vice president of Quicklogic, reported that WG-4 was thinking of offering a series of benchmark designs that became gradually larger. This would enable the user to determine how effectively the FPGA-synthesis tool utilized FPGA space.
The designs, available on Internet, would range from 1000 to 10,000 gates from a cross section of functionality and market segments of real-world designs recruited by the 10 FPGA vendors from their customers. More important, EDA vendors, FPGA vendors, and users could benchmark these designs and publicly report their results plus the exact synthesis-tool settings they used on the same Internet site.
Talk of having standardized place-and-route settings was kicked around. Also, realizing it's bad business to alienate ASIC designers who are primarily Verilog users (plus finding out that it's not hard to translate from synthesizable Verilog to synthesizable VHDL and vice versa), WG-4 will offer designs and test benches in Verilog.
Many of the WG-4 members voiced their preferred strategy of embarrassing the 10 FPGA vendors and 10 FPGA-synthesis vendors into partaking in the benchmarks. That is, if Synopsys refused to have their tools or AT&T refused to have their FPGAs involved in this synthesis benchmark, their absences would be rather conspicuous.
"Instead of playing Big Brother, WG-4 plans to use the 'carrot-but-no-stick' approach to get industry cooperation," says Steve Sharp, Xilinx's WG-4 representative.
Uncle John wants YOU
Keep in mind that WG-4 is still in the formative stage for much of this, and Birkner wants your ideas and help. (He can be contacted at "john@qlogic.com" or (408) 987-2020.) Exactly how will benchmarks be reported? What if Minc benchmarked Data I/O and reported suspicious results, making Data I/O look bad? What if Innovative Synthesis Technologies reported unrealistically good results with its tools that no one else could duplicate? Should there be a public record of who tried to report what results that were deemed questionable by the WG-4 and why they were rejected? How can they get lots of users to be a part of this undertaking instead of just companies defending their vested interests?
No matter how you answer these questions, you'll still have those who "lose" in the benchmarks howling that the benchmarks were somehow flawed in the first place.
"Yes, everyone wants a benchmark that makes their own silicon or synthesis product look good," says East, "but I think an imperfect benchmark is better than none."