Brian's Brain

The topic of general-purpose applications for GPUs (graphics processors) beyond traditional 2D- and 3D-graphics acceleration is seemingly at the perpetual forefront of my professional mind. Do a web search on my past editorial output and you'll see what I mean. Back in mid-June, I updated you on a plethora of GPGPU (general-purpose GPU) applications; this time I'd like to focus in on still and video image editing tasks such as decoding (i.e. decompression), encoding (compression), transcoding and other processing functions.

I've been planning this writeup for a long time, and two recent events provoked me to pull the trigger now versus later. First is the (likely true) rumor of recent days that Apple's leveraging the video processing circuitry in the latest MacBook and MacBook Pros' Nvidia GPUs to hardware-assist MPEG-4 video decoding. The second, as I alluded to last week, is last month's launch of Adobe's GPU-accelerated CS4 (Creative Suite 4).

To the first data point, all I can muster is a duh. Don't get me wrong...considering the pitiful state of GPU exploit in OS X that existed just three and a half years ago, I'm delighted by these latest developments. But as I pointed out three years ago, Microsoft's DirectX VA API for exploiting (both by Microsoft- and third party-authored apps) GPUs' video decoding function blocks has existed since late 2000. It'd be embarrassing for Apple to not be in a comparable position eight years later.

The graphics core-inclusive Intel chipsets used in previous MacBooks didn't hardware-accelerate MPEG-4, so the dramatic CPU utilization plunge isn't a surprise. What's a little surprising to me, though, is that Apple waited until now to implement such support. MacBook Pros, along with desktop-targeted iMacs and Mac Pros, have included MPEG-4-supportive AMD/ATI and Nvidia GPUs for years. Note, too, that the MPEG-4 decode acceleration support currently implemented leverages QuickTime...general-purpose third-party support doesn't yet seem to be in place. So it's a good first step, and about time, but Apple still hasn't caught up with where Microsoft has been since the decade's beginning.

Now let's look at that second data point. Adobe's been progressively implementing GPU acceleration support in its products for years. Again, I pointed out three years ago that the company's Premier video editing family and After Effects graphics rendering application already leveraged GPUs, and at that time senior product manager Giles Baker also strongly 'telegraphed' the company's near-future intentions to GPU-accelerate various Photoshop family variants. Acrobat has also implemented GPU acceleration since v8. What's interesting to me about the GPU embrace in Creative Suite 4 is how it's implemented, how limited it is compared to what theoretically could have been GPU-accelerated, and why.

Apple's 3.5-year old OS 10.4 added the Core Image API, which enabled tapping into GPU resources for image editing performance boosts. Apple's own applications harness Core Image, as do a number of third-party programs. But Creative Suite 4 doesn't seem to be on the Core Image list...various company officials consistently told me that Adobe was using the OpenGL API. Fundamentally, I suspect that this is because CS4 is cross-platform, intended to run both on OS X and Windows (arguably, in fact, the Windows version of CS4 is more advanced than its OS X peer, due to lack of 64-bit support on the Apple platform). However, because OpenGL is fundamentally a graphics API, not an image editing API, this choice results in a practical limit in the degree to which GPU leverage can occur.

Short-term API restrictions aside, my personal opinion is that we'll long-term still see only a partial leverage of the GPU for still and video image editing functions. Simple math, I think, clarifies the reasons why. A 512 MByte dedicated graphics frame buffer might seem massive, even when you stack it up against a 10 Mpixel still image (at, don't forget, 24 uncompressed color bits per pixel, or 48 bits per pixel in the HDR aka high definition rendering era). But if I've done my math right, a single 24-bit 1920x1080 pixel video frame consumes more than 6 MBytes of memory. I'll let you do the math to figure out how diminutive a 24-, 30- or 60-fps clip will still gobble up all the available frame buffer space...and don't forget about incremental scratchpad RAM requirements.

Frame buffer consumption isn't the only issue to consider. Three years ago, I know that I raved about PCI Express's bidirectional bandwidth...realize that this was a relative analysis versus the PCI and AGP predecessors. Most processing functions that might be GPU-accelerated are intermediary; the data flows to the GPU over PCI Express, gets crunched by the graphics processor, then flows back to main memory and the CPU over that same PCI Express link. Is it any surprise, therefore, that the few GPU-boosted tasks in Photoshop CS4 and Bridge CS4 are predominantly one-way i.e. display-only? Pixels head to the graphics subsystem for panning, zooming and canvas-rotation, then continue out the DisplayPort, DVI, HDMI or VGA connector to the LCD.

Next, expand your mindset beyond still images and consider video processing. Lossy compression, for example, while it might at first glance seem a natural fit for the massively parallel processing capabilities of GPUs, loses some of its luster upon closer inspection. Modern video codecs leverage CPUs' increasingly faster and increasingly higher core counts to iteratively run compressed clip candidates against the original version of the frame series. Via PSNR or other analysis metric algorithms, they assess the degree of resultant artifacts and, if necessary, automatically re-run the compression routine using different settings in an attempt to improve the results.

Couple the limited amount of frame buffer memory on graphics cards with the back-and-forth non-infinite bandwidth of PCI Express (which would negate any GPU-induced performance boost) and you can see why GPU-accelerated video utilities don't leverage iterative compression techniques...and then you can see why GPU-enabled Badaboom didn't even stack up against open-source Handbrake (more from AnandTech), far from commercial products. And don't forget that whereas most PC software (including codec code) is today targeted for x86 CPUs, AMD/ATI and Nvidia's GPUs are definitely not x86 implementations.

Yes, I know that most of that code is C-based, and yes I know that AMD/ATI's Stream and Nvidia's CUDA will both suck in C. But as anyone who's ported high-level language software from one processor to another will concur, the migration still isn't remotely a slam-dunk. This is in no small part because the GPU toolsets have a unique learning curve versus leveraging existing x86 toolset expertise...is it any wonder that if CPU handoff to a media coprocessor is going to happen, I think x86-based Larrabee is a compelling candidate (if Intel can achieve competive cost and performance metrics, that is)?

So welcome to the party, S3, but don't assume that the cake will be particularly sweet, that the punch will be especially thirst-quenching, or that the candles will burn forever. Is the picture clear, folks? If not, as always, I welcome your thoughts.