Qualcomm's MSM8660: The Benchmark's The Thing Wherein I'll Crown The King

Continued from ‘Qualcomm’s MSM8660: Dual-Core ARM With Adreno 220 Graphics For The Performance-Hungry‘…

What benchmark did I employ in this study? After much consideration and no shortage of gnashing of teeth, I reluctantly settled on Aurora Softworks’ Quadrant Advanced, one of the utilities which came pre-installed on the MDP. Quadrant is seemingly by far the most popular Android-focused benchmarking suite of the moment, a status assisted in no small part by the availability of a gratis standard edition. However, as AnandTech also noted, Aurora Softworks is quite secretive about just what Quadrant tests, and how. If you’d like to join me in striving to sort out just who Aurora Softworks is and what the company’s up to, here are some additional resources I’ve come across; the company’s blog, its Twitter feed, and an associated Google discussion group.

Here’s what I’ve been able to decipher from watching Quadrant in operation, along with logging its feedback messages:

• Total: A numerical average of the subsequently listed specific tests’ results
• CPU: A consolidation of twelve sub-tests:
  • Branching (?, the message so quickly flashes on-screen, then disappears, that I’m not 100% certain on this one)
  • Arithmetics - int
  • Arithmetics - long int
  • Arithmetics - short int
  • Arithmetics - byte
  • Arithmetics - floating point
  • Arithmetics - double precision
  • Checksum
  • Compression
  • XML parsing
  • Video decoding - H.264
  • Video decoding - AAC audio
• Memory
  • Throughput
• I/O
  • File system writes
  • File system reads
  • Database writes
  • Database reads
• 2D
• 3D

Quadrant’s free Standard Edition only delivers the Total benchmark score and is also advertising-supported. The $2.99 Advanced Edition dispenses with the ads and also provides per-test breakdown scores, while the $24.99 Professional Edition is a feature clone of Advanced Edition with licensing terms that enable commercial use. However, instead of going the conventional route of leveraging the Google Android Market and/or Amazon Appstore for Android, Aurora Softworks has for some unknown reason chosen to distributed Quadrant Advanced and Professional via the obscure SlideME, whose purchase terms involve unpalatable first-time verification charges and whose SAM (SlideME Android Application Manager) is reportedly buggy and incompatible with some Android-based hardware and software. As such, for my multi-platform testing purposes, I decided to leverage the already-resident APK for Quadrant Advanced v1.1.1 on the MDP (which I extracted using the Android SDK’s ADB) versus purchasing the more current Quadrant Advanced or Professional v1.1.7.

I’m hopeful that although Quadrant’s benchmark algorithm specifics are undocumented, thereby making absolute-sense analysis of results data of dubious-at-best value, comparative number sets from different hardware/software testing candidates will still be meaningful in a relative sense. Quadrant Advanced includes a far more in-depth version of the earlier-shown ‘About Phone’ facility built into Android. Here are some screenshots for you:

What hardware/software alternatives did I benchmark the MDP against? Regular readers may already recall that I have two Android handsets in my possession, a Motorola Droid upgraded to latest-and-greatest (for it, at least) Android v2.2.2, and a Google Nexus One running Android v2.3.4. I also have three Android-based tablets; a 7″ widescreen Barnes & Noble NOOKcolor and 10″ widescreen ViewSonic gTablet, both running CyanogenMod 7.0.3 (aka Android 2.3.3), and a 10″ widescreen Motorola Xoom which I recently updated to Android 3.1. The latter two tablets, like the MDP, are based on a dual-core ARM SoC, Nvidia’s 1 GHz Tegra 2.

Keep in mind that all three tablets have much larger screens (therefore pixel-count resolutions) than do any of the handsets represented in this study; this discrepancy will hamper the tablets’ comparative graphics benchmark results. Note, too, that Quadrant Advanced contains built-in reference benchmark results for the Nexus One running both initial Android 2.1 and the subsequent Android 2.2 upgrade, along with the Motorola Droid running an unspecified earlier Android 2.x build. I’ve included these pre-defined results data sets in the summary table which follows, along with the results I determined in my testing using up-to-date Android variants.

I ran each Quadrant Advanced benchmark suite five times, choosing a representative average data result set to publish here. Run-to-run variation was in general minimal (no more than a few percent), with greater variance in some tests versus others, and with some hardware versus the rest. And since I was able to selectively control Vsync (via ADB-delivered commands) on the MDP, I captured benchmark data with the setting both enabled and disabled. Vsync, when ‘on’, caps the upper-end graphics frame rate at 60 fps. Here’s what Qualcomm’s documentation advises, which will also be familiar to anyone who’s ever done computer-based graphics chip/card benchmarking (note that for some unknown reason, my particular MDP arrived with Vsync ‘on’ versus ‘off’, which supposedly was the default):

Vsync “on” is the default setting for all Snapdragon devices. With Vsync “on” there is a limitation on frame rate that provides stability and improved screen display quality on Android devices. However, with Vsync on, the full performance of the GPU cannot be achieved since it is frame rate limited. Vsync is recommended to be turned “off” when benchmarking GPU performance and “on” for CPU and Web testing.

Continue reading ‘Qualcomm’s MSM8660: Does Dual-Core Hype Translate Into Practical-Benefit Reality?‘…