Cameras in hand: Image quality takes on new meaning for chip architects

By Ron Wilson, Executive Editor -- 1/12/2007 12:39:00 PM

Behind the temporary walls of the briefing rooms at CES, a new picture of the handheld camera was gradually emerging this week. As the megapixel counts of sensors take on more importance as a numbers game than as a significant contributor to image quality, the search for differential advantage is branching out in other directions. This is changing everything from the design of image sensors to the architecture of handsets and video cameras.

Today it is arguable that the key differentiator in retail camera sales is still pixel count, even though this number has become about as relevant as, say, clock frequency in personal-computer sales. Customers are not well equipped to the compare image quality of competing cameras, and of course online sales never even give them the chance. So they go by numbers and reputation.

But reputation is formed in part by the experience of existing users. And this presents a problem. Users tend to judge their results against a standard set by the professionally produced images in magazines and on the Web and not against the results they themselves achieved with last year's 3-megapixel camera. And in fact, casual users can produce good images with these cameras—if the get their prints from a professional printing service that devotes arrays of Pentiums to image post-processing to correct white balance, recognize and enhance object edges and chrominance, correct focus, remove artifacts, and so forth. A few billion instructions of image processing can do nearly what an experienced photographer can do.

This effect is even more critical for video cameras, which are starting across the boundary into high definition. Here, the user's expectation is set not by another user with a video camera, but by a film crew with a lighting staff, makeup people, set designers, and a sound crew—the kind of team that produces commercials and programming for HDTV broadcast.

In both these cases, consumers will unconsciously judge their results against inappropriate models. And they will, of course, be disappointed if the image they see doesn't measure up. This puts pressure on the camera architects to cram as much image quality into the acquisition process, and as much post-processing capability into the platform, as possible.

The effort begins with the image sensors. Ironically, the pursuit of higher megapixel figures is, with the exception of certain image types, destructive rather than beneficial to image quality. This is because for a given sensor size—and hence price range—increasing the pixel count means decreasing the active area of the photodiode, and hence lowering the sensitivity and/or the signal-to-noise ratio. Already in the 6-megapixel range, sensitivity has been reduced enough to seriously compromise image quality at low light levels.

One possible solution comes from Toshiba, which is working on a way to get more light to the active area. Today the individual microlenses that are bonded onto the surface of the image-sensor array are circular—so the lens doesn't collect light from the whole rectangular area of the pixel cell, only from a circular area that fits inside the rectangle. Toshiba researchers are working on a rectangular microlens that would cover the entire pixel area, substantially increasing light-gathering efficiency.

Toshiba is also working on the system-architecture problem. In many cell-phone handset designs today, even the pixel-level post-processing to remove noise and bias from the raw CMOS sensor data is done on the handset's baseband or applications processor. But, according to Toshiba vice president Andrew Burt, while that solution is attractive to platform developers for cost reasons, it is losing its appeal with handset manufacturers, who find that the design team that created the sensor should remain in control of the pixel-level processing. This is tending not only to put the pixel-level signal-processing hardware back on the sensor die, but also to enlarge the definition of what needs to happen at the pixel level. "As resolutions approach 5 to 8 megapixels, we see image-processing applications migrating onto the pixel processor," Burt said.

At the same time, scene-level post-processing is being driven toward photo-kiosk levels, but without the banks of processors necessary to support that kind of computing load. Digital still cameras long ago started down this path with software red-eye removal, image stabilization, automatic white-balance, and scene-specific control over focus and exposure. The image-improvement team instantly gobbles up any memory and DSP power the architects give them.

It's obvious that a handset, with its highly constrained environment, must fear that path. Yet handset vendors can't protect their products from being judged by the same standards consumers apply to other digital-imaging products—why doesn't this look as good as the pictures in National Geographic, or People? Ironically, the small area available in the handset for the image sensor—and hence the lower megapixel count for a given sensor technology node—means the demand for image post-processing is actually higher for handset images than for images from dedicated cameras. And of course consumers would rather all that work happened transparently inside the device.

Handset architects have few degrees of freedom in responding to this problem. But radical algorithm development and large-scale parallel processing—the only architectural technique that can offer increased number-crunching power without ruinous power demands—are possible directions for development.

Finally, at least one insider sees video cameras heading down the same road. Cameras are moving to true HD: 1080-line, progressive-scan imaging. That means the consumer is going to be looking at the output of a handheld camera on an HD screen in high resolution. The consumer is not going to be happy.

Part of the problem is refresh rate. "Hollywood can make movies at 24 frames per second," observes Didier LeGall, executive vice president at Ambarella. "But they have professional cinematographers and specifications written right into the script about how fast a pan or zoom will be, how a camera will track, how fast the hero will run across the scene, and so on.

A novice recording HD sequences with a handheld camera, swinging it around, zooming in and out, is going to create flicker and motion artifacts at anything like a 24 fps refresh rate. That is one of the reasons the industry is moving rapidly toward 120 Hz.

But the problems go beyond the mechanical. "Can consumers even make 'viewable' content?" LeGall asks. "Normally in an hour of recorded video, an average user gets maybe one or two minutes that are good enough to watch." LeGall suggests that post-processing tools, ranging from simple things like indexing and—in effect—the ability to semi-automatically extract a two-minute trailer from that hour of bad video, will be necessary for consumers to have a positive experience with their new cameras.

Editing software, and even image-processing software to insert computer-generated repairs and edits into the captured video, may become important too, just as image-manipulation software is now important to a segment of the still-camera market. "At least we have one major advantage over film," LeGall muses. "When you make video a digital data type instead of a separate medium, you have already solved half the problem."

© 2009, Reed Business Information, a division of Reed Elsevier Inc. All Rights Reserved.