December 18, 1997

Digital photography CLICKS

Brian Dipert, Technical Editor

Digital cameras contain several highly interdependent subsystems and rely on other hardware and software to complete the picture. When everything clicks, the results are impressive.

Digital imaging is still in its infancy compared with traditional silver-halide-based photography. This new technology has both advocates and detractors in the electronics and photography communities, but many companies have a strong interest in  seeing digital imaging succeed. The circuits, mechanics, and software within digital cameras are intriguing, as are the trade-offs and partnerships that manufacturers make in image capture and PC-based image processing, transmission, archiving, and output. These companies must balance several camera requirements, including:

• base cost;
• cost of accessories, such as memory cards, cables, and power adapters;
• image resolution;
• sharpness;
• color depth and density;
• power consumption;
• ease of use with both the camera's user interface and any bundled software;
• ergonomics;
• stand-alone operation vs PC dependency;
• upgradability;
• compatibility with existing camera equipment;
• performance, in both the delay from shutter press until the camera takes the picture and the delay between pictures; and
• the number of pictures that you can take before filling the storage media.

In a new industry in which no standards exist and end customer's real and perceived needs are unclear, decision making is especially difficult. Success requires fast product-development cycles, close coordination within and among development teams (as well as among companies), and a commitment to take educated risks with little or no short-term payoff. Success also requires knowing when to stay with an idea in the face of initial skepticism along with a willingness to change course when a "better mousetrap" clearly emerges. The core of engineering is this balancing act between what is theoretically possible and what is practically achievable.

Representative cameras from Eastman Kodak illustrate products priced at less than $500, less than $1000, and more than $1000 (Figure 1). Kodak was one of the first companies to heavily invest in digital imaging and today has one of the broadest product lines. Cameras from other manufacturers in each price range often have comparable features.

Good pictures need good optics

A digital camera is only as good as its weakest link, and the image-capture process begins with the lens. The fundamental trade-off that engineers make is between plastic and glass lenses. Plastic lenses are less expensive, and under controlled conditions they can deliver surprisingly better results than glass alternatives. However, plastic is far more sensitive to temperature variations, scratches more easily, and is more susceptible to light-caused flare effects than glass.

Kodak's DVC300 is a USB-tethered camera for teleconferencing and low-resolution still-image capture. It was designed for office use with adequate diffused lighting, stable temperatures, and minimal handling, so Kodak chose plastic lenses. The DC120 and DC210 target users of portable 35-mm cameras and need to handle rougher treatment and wide variations in temperature and lighting. Thus, they use higher quality, coated-glass lenses. At the high end, the DCS460 takes advantage of the assortment of Nikon optics. (Digital cameras using Canon (Tokyo) and Minolta (Tokyo) 35-mm single-lens-reflex equipment are also available.) However, because the CCD sensor is smaller than the dimensions of a 35-mm frame, the effective focal length of a given lens increases. To a point, lens quality improves with a larger number of lens elements. Similar to µP caching, however, greater complexity eventually nets diminishing returns.

What about focus? Lenses exhibit a phenomenon called the "hyperfocal distance," a function of the lens-element placement, aperture size, and lens focal length that defines the in-focus range. All objects from half the hyperfocal distance to infinity are in-focus. The DVC300 lens in its default setting is designed for installation above the PC monitor with the PC user a few feet away. Using the PC monitor as feedback, the DVC300 also has a manual-focus mode for still-image close-ups. This capability is useful, for example, if you want to show a picture of some equipment you have in front of you to the person with whom you are teleconferencing.

The DC210, because of its point-and-shoot target market, has a fixed-focus lens. Interestingly, the DC120, a camera of roughly the same price but targeting the more serious photographer, has an autofocus lens with a macro mode. Autofocus adds cost in the form of lens-element movement motors, infrared focus sensors, control-processor complexity, and other circuits, but the DC210 spends its feature-set budget in other areas. Minolta's Dimage V has a unique lens design: The optics and sensor connect to the remainder of the camera using a 3-ft cable and can detach to capture otherwise-inaccessible shots or to achieve unique camera angles.

The DVC300 and DC120/DC210 offer a digital zoom and an optical zoom, respectively. A digital zoom doesn't alter the orientation of the lens elements. Depending on the digital-zoom setting, the camera discards a portion of the pixel information that the image sensor captures. The camera then enlarges the remainder to fill the expected image-file size. In some cases, the camera replicates the same pixel information to multiple output-file bytes, which can cause jagged image edges. In other cases, the camera creates intermediate pixel information using nearest-neighbor approximation or more complex gradient-calculation techniques; this process is called "interpolation" (Figure 2). Interpolation's chief disadvantage is that it blurs edges compared with those of the same image captured using a higher resolution sensor.

With optical zooms, the chief trade-off is between manual and motor-assisted zoom control. The latter incurs additional cost, but camera users often prefer it for its easier operation. The DC120 and DC210 lenses have a threaded front ring that enables the use of filters, wide/telephoto converters, and other accessories.

Behind the lens and in front of the image sensor, many cameras employ an optical prefilter--a piece of quartz that selectively blurs the image. This prefilter conceptually serves the same purpose as a lowpass audio filter. Because the image sensor contains fixed spacing between pixels, light wavelengths shorter than twice this distance can produce aliasing distortion if they strike the sensor. (Notice the similarity to the Nyquist audio-sampling frequency?) A similar type of distortion comes from taking a picture containing edge transitions that are too close together for the sensor to accurately resolve them. This distortion often manifests itself as color fringes around an edge (Figure 3) or as a series of color rings known as a moiré pattern. The DCS460, because it uses the optical subsystem of a standard 35-mm camera, has no prefilter, but its higher quality lens and image sensor inherently minimize aliasing effects.

Almost all digital cameras contain a viewfinder that aids in framing the picture. If the camera provides zoom, the viewfinder angle of view and magnification often adjust accordingly. Many cameras use a range-finder configuration, in which the viewfinder has a different set of optics (and, therefore, a slightly different viewpoint) from that of the lens used to capture the image. Viewfinder marks that delineate the lens-view borders partially correct this difference, or "parallax error." At extreme close-ups, the LCD gives the most accurate framing representation of your picture. Olympus' (Tokyo) D-500L and D-600L digital cameras share the same single-lens-reflex architecture as the company's 35-mm and Advanced Photo System cameras. Because you compose the picture through the same lens that takes it, there is no parallax error, but such a camera requires a mirror, a shutter, and other mechanics to redirect the light to the viewfinder prism. If the camera is a digitized version of a standard 35-mm or 21/4-in. single-lens reflex, such as the Kodak DCS460, these mechanics already exist.

Editing needs sensing first

Many camera manufacturers, when talking about digital "film," are referring to the flash memory or other image-storage media. These companies are only half right. The other film function--capture--occurs in the sensing subsystem, which comprises the CCD or CMOS sensor, analog processing circuits, and ADC. Almost all of today's digital cameras use CCDs. Toshiba (Tokyo), Umax (Fremont, CA), and Vivitar (Tokyo) all sell digital cameras with CMOS sensors (Sound Vision designs and builds the last two cameras), and you can expect more in the near future (Reference 1).

The ADC primarily determines a camera's color depth or precision (number of bits per pixel), although back-end processing can artificially increase this precision. A camera's color density, or dynamic range, which is its ability to capture image detail in light ranging from dark shadows to bright highlights, is also a function of the sensor sensitivity. Sensitivity and color depth improve with larger cell size; the bigger the cell, the more electrons available to react to light photons and the wider the range of light values the sensor can resolve. Cell size must balance with the desired number of cells (resolution) and the percentage of the CCD devoted to cells vs other circuits (area efficiency, or fill factor). As with TVs, PC monitors, and DRAMs, sensor cost exponentially increases as sensor area increases because of lower yield and other technical and economic factors.

A digital camera uses one of three kinds of CCDs (Figure 4). Linear sensors, which you can also find in digital copiers, scanners, and fax machines, offer the best combination of low cost and high resolution, but cameras using them sequentially sense and transfer each pixel row of the image to an on-chip buffer. For this reason, linear-sensor-based cameras have long exposure times and are usable only with still-life subjects. Full-frame-area sensors, such as those in Kodak's DCS460, have high area efficiency and simultaneously capture all of the image pixels. However, full-frame-area sensors require a separate mechanical shutter to block light when you're not taking a picture and at the conclusion of an exposure, when the camera transfers each cell's stored charge to the ADC.

The third and most common CCD type is the interline-area sensor. Also used in camcorders, the interline-area sensor contains both charge-accumulation and corresponding light-blocked, charge-storage elements for each cell. Separate charge-storage elements remove the need for a costly mechanical shutter and also enable slow-frame-rate video display on the camera's LCD. However, the area efficiency is low, causing a decrease in sensitivity, resolution, or both for a given sensor size. Also, a portion of the light striking the sensor doesn't actually enter a cell unless the CCD contains microlenses (Figure 4c).

Regardless of the type, still-image sensors have far more stringent requirements than their video alternatives. Video includes motion, which draws your attention away from low image resolution, inaccurate color balance, limited dynamic range, and other shortcomings. With still images, these errors are immediately apparent. Video scanning, similar to television display, is interlaced, but still-image scanning is ideally progressive. Interlaced scanning with still-image photography can result in pixel rows with image information shifted relative to each other. This shifting is due to subject motion, a phenomenon more noticeable in still images than in video.

Cell dimensions are another fundamental difference between still and video applications. Camcorder sensor cells are rectangular, often with 2-to-1 horizontal-to-vertical ratios, corresponding to television- and movie-screen dimensions. Still pictures look best with square pixels, analogous to film "grain." Ironically, Kodak's DC20, DC25, and DC120 cameras, as well as Polaroid's PDC-2000, use rectangular cell sensors that benefit from the high-volume cost savings of the videocamera market. As part of the DC120 postsensing image-processing step, Kodak converts each two rectangular cells' information into approximately three square image pixels. This type of interpolation suffers greater loss of resolution in the horizontal direction than in the vertical but otherwise produces good results.

Although low-end cameras may not produce 35-mm-compatible images if you blow up the photographs to 5×7 in. or larger, camera manufacturers carefully consider their target customers' usage when making feature decisions. PCs (including Macs and workstations) have monitor resolutions on the order of 72 lines/in., and many Web-site-published and e-mailed photos take up only a percentage of the display and use a limited color palette. Color ink-jet printers, even if they advertise 600, 720, or 14,400 dpi, require image resolutions roughly one-third (or less) that stringent. Depending on the printer's ink-control capability, it may use multiple groupings of cyan, magenta, yellow, and black (CMYK) dots to represent a broader color range for each image pixel. This process is called "half-toning"; some printers use as many as seven ink colors for additional color accuracy. The image resolution needs to match the declared printer resolution only when the output is black and white or when you use a dye sublimation or other, higher quality color printer. Finally, consider that more than 90% of all of today's traditionally developed photos are 4×6-in. or smaller. However, if you might enlarge an image or a portion of that image, you want to invest in one of the million-pixel or higher resolution cameras.

The DVC300, targeting VGA monitors, has a 640×480 (307,200)-cell sensor, and most of the new cameras for the less-than-$500 mass consumer market specify similar resolutions. 640×480-pixel resolution also produces good-quality, snapshot-sized ink-jet prints. The DC120 contains a 850×984-cell sensor (836,400 rectangular cells), which interpolates to 1280×960 (1,228,800)-square-pixel counterparts. Kodak's DC210 uses a square-cell sensor measuring 1160×872 (1,011,520) cells. Polaroid's PDC-2000 includes a 1656×620 (1,026,720)-rectangular-cell sensor, which back-end processing converts to a 800×600 (480,000, SVGA)- or 1600×1200 (1,920,000)-square-pixel image. Kodak's DCS460 contains a 3072×2048 (6,291,456)-square-cell CCD sensor. The resultant 24-bit color file is almost 20 Mbytes!

A sensor, normally a monochrome device, cannot extract specific color information if it is exposed to a full-color spectrum, so the sensor requires some prefiltering. The three most common methods of controlling the light frequencies reaching individual pixels are to use multiple sensors, a rotating multicolor filter, or a color-filter array (Figure 5). In each case, the most popular filter palette is the red, green, blue (RGB) additive set, which color displays also use. The RGB additive set is so named because these three colors are added to an all-black base to form all possible colors, including, ultimately, white.

The subtractive color set of cyan-magenta-yellow is another filtering option (starting with a white base, such as paper, subtractive colors combine to form black). The biggest advantage of subtractive filtration is that each filter color filters through a portion of two additive colors (yellow filters allow both green and red light to pass through them, for example). For this reason, cyan-magenta-yellow filters give better low-light sensitivity, an ideal characteristic for camcorders. However, the filtered results must subsequently convert to RGB for display. Lost color information and various artifacts introduced during conversion can produce nonideal still-image results. Still cameras, unlike camcorders, can also easily supplement available light with a flash.

The multisensor color approach, with three separate filters and sensors, produces accurate results but is also costly (Figure 5a). A color-sequential- rotating filter (Figure 5b) requires three separate exposures and, therefore, suits only still-life photography. A variation of this second technique that uses a tricolor LCD, the liquid-crystal tunable filter, promises much shorter exposure times, but only very expensive cameras offer this filter. The third and most common approach, an integral color-filter array, places an individual red, green, or blue (or cyan, magenta, or yellow) filter above each sensor pixel, relying on back-end image processing to approximate the remainder of each pixel's light-spectrum information from nearest neighbors.

Note that in Figure 5c, there are twice as many green pixels as there are red or blue. This structure, called a Bayer pattern after Kodak scientist Bryce Bayer, results from the observation that the human eye is more sensitive to green than to red or blue, so accuracy is most important in the green portion of the color spectrum. Variations of the Bayer pattern are common but not universal. Polaroid's PDC-2000 uses alternating red-, blue- and green-filtered pixel columns, and the filters are pastel or muted in color, thereby passing at least a small percentage of multiple primary-color details for each pixel. Sound Vision's CMOS-sensor-based cameras use red, green, blue, and teal (a blue-green mix) filters.

High-end digital cameras offer variable sensitivity, akin to an adjustable ISO rating for traditional film. In some cases, summing multiple sensor pixels' worth of information to create one image pixel accomplishes this adjustment; however, other cameras use an analog amplifier to boost the signal strength between the sensor and ADC, which can distort and add noise. In either case, the result is the appearance of increased grain at high-sensitivity settings, similar to that of high-ISO silver-halide film.

Although most PC software and graphics cards do not support pixel color values larger than 24 bits (8 bits per primary color), why do you often find 10-bit, 12-bit, and even larger ADCs in digital cameras? The human eye notices quantization errors in the shadows, or dark areas, of a photograph more than in the highlight, or light, sections. Greater-than-8-bit ADC precision allows the back-end image processor to selectively retain the most important 8 bits of image information for transfer to the PC. According to Kodak's Ken Parulski, chief architect for digital cameras, less-than-8-bit ADCs make little sense; these converters would provide little system-cost savings, and the color quality would be acceptable only on a 256-color display.

Image processing: black magic

Digital cameras' hardware designs are rather straightforward and in many cases benefit from experience gained with today's traditional film cameras and video equipment. Image processing, on the other hand, is the most important feature of a camera (your eye and brain can quickly discern between "good" and "bad" prints). It is also the area in which camera manufacturers have the greatest opportunity to differentiate themselves and in which they have the least overall control. Image quality depends highly on lighting and other subject characteristics. Not only software and hardware inside the PC (if it exists) can degrade the camera output, but also the printer or other output equipment. Because capture and display devices have different color-spectrum-response characteristics, they ideally should calibrate to a common reference point, so that they automatically adjust a digital image passed to them by other hardware and software to produce optimum results. Thus, several industry standards and working groups have sprung up, the latest being the Digital Imaging Group (http://www.digitalimaging.org/).

The biggest trade-off in the image-and-control-processor subsystem is the percentage of image processing that takes place in the camera vs in a PC. Until the DC210, Kodak's digital cameras completed most, if not all, image processing in the PC after transferring the image files out of the camera. For the DVC300, processing is completely PC-based; the camera contains little more than a sensor, an ADC, an Altera (San Jose, CA) CPLD for simple state-machine control, and an Intel USB peripheral controller. Other Kodak cameras optionally compress the CCD output and perform simple processing to construct a low-resolution and minimum-color tagged-image-format-file (TIFF) image "thumbnail," used by the LCD (if the camera has one) and by the PC's image-editing software. This approach has several advantages: The camera's processor can be low-perform-ance and low-cost, and minimal between-picture processing means you can take the next picture faster. The files are smaller than their fully finished lossless alternatives, such as TIFF, so you can take more pictures before "reloading." Also, you lose no image detail or color quality inside the camera because of the conversion to an RGB or other color gamut or to a lossy file format, such as JPEG. Intel, with its Portable PC Camera '98 Design Guidelines (www.intel.com/imaging), strongly recommends a PC-based-processing approach. Intel's 971 PC Camera, including an Intel-developed 768×576-cell CMOS sensor, also relies on the PC for most image-processing tasks.

The alternative approach to image processing is to complete all operations within the camera, which then outputs pictures in one of several finished formats, such as JPEG, TIFF, and FlashPix. Notice that many digital-camera manufacturers also make photo-quality printers. Although these companies are not precluding a PC as an intermediate image-editing and-archiving device, they also want to target the roughly 50% of households that don't currently own PCs by providing a means of directly connecting the camera to a printer. If the camera outputs a partially finished and proprietary file format, it puts an added burden on the camera manufacturer or application developer to create PC-based software to complete the process and to support multiple PC operating systems. Finally, nonstandard film formats limit the camera user's ability to share images with others (e-mailing your kids' pictures to relatives, for example), unless they also have the proprietary software on their PCs.

Regardless of where the image processing occurs, it contains several steps, according to Kodak's Parulski. First, if the sensor uses a Bayer or other selective color-filtering technique, interpolation reconstructs 8 or more bits each of red, blue, and green information for each pixel. Second, processing modifies the color values to adjust for differences in how the sensor responds to light compared with how the eye responds (and what the brain expects). This conversion is analogous to modifying a microphone's output to match the sensitivity of the human ear and to a speaker's frequency-response pattern. Color modification can also adjust to variable-lighting conditions; daylight, incandescent illumination, and fluorescent illumination all have different spectral frequency patterns, for example.

Processing can also increase the saturation, or intensity, of portions of the color spectrum, modifying the strictly accurate reproduction of a scene to match what humans like to see. Camera manufacturers call this approach the "psychophysics model." It's an inexact science, because color preferences highly depend on the user's cultural background and geographic location (people who live in forests like to see more green, and those who live in deserts prefer more yellows and browns). The characteristics of the photographed scene also complicate this adjustment; adding red to the image might produce a more pleasing autumn-leaf picture but would give skin tones an undesirable sunburnt look. For this reason, some cameras, when you press the shutter once, actually capture multiple images at different exposure and color settings, sampling each and selecting the one they "think" you'll prefer. After turning on the DVC300, you won't see the first few frames, because during that time the camera and PC communicate with each other over USB, calibrating for the best possible results.

After color adjustment, processing can also sharpen the image. Simplistically, the sharpening algorithm compares and increases the color differences between adjacent pixels. However, to minimize jagged output and other noise artifacts, this increase factor varies and occurs only beyond a specific differential threshold, implying an edge in the original image. Compared with standard 35-mm cameras, you may find it difficult to create shallow depth of field with digital cameras; this characteristic is a function of both the optics differences and the back-end sharpening. In many situations, though, focusing improvements are valuable features that increase the number of usable pictures. The final processing steps are image-data compression and file formatting. The compression is either lossless, such as the Lempel-Zif-Welsh compression in TIFF, or lossy (JPEG or variants). The compressed-file size depends on the amount of pixel data variability in the original image. If the camera offers a fixed file size for each camera setting, it probably varies the compression ratio on a per-image basis to maximize quality and minimize the delay needed between successive shots.

Image processing can also partially correct nonlinearities and other defects in the lens and sensor; this approach is how NASA obtained usable images from the Hubble space telescope until its optics system was upgraded. Some cameras also take a second exposure after closing the shutter, then subtract it from the original image to remove sensor noise, such as dark-current effects seen at long exposure times.

Processing power fundamentally derives from the desired image resolution, the color depth, and the maximum-tolerated delay between successive shots. For example, Polaroid's PDC-2000 processes all images internally in the camera's high-resolution mode but relies on the host PC for its superhigh-resolution mode. Many processing steps, such as interpolation and sharpening, involve not only each target pixel's characteristics but also a weighted average of a group of surrounding pixels (a 5×5 matrix, for example). This involvement contrasts with pixel-by-pixel operations, such as bulk-image color shifts. Image-compression techniques also make frequent use of discrete cosine transforms (DCTs) and other multiply-accumulate convolution operations. For these reasons, fast on-processor hardware-multiply circuits are desirable, as are many on-CPU registers to hold multiple matrix-multiplication coefficient sets.

If the image processor has spare bandwidth and many I/O pins, it can also serve double duty as the control processor--running the autofocus and zoom motors and flash, responding to user inputs, and driving the LCD and interface buses. Abundant I/O pins also enable selective shutdown of camera subsystems when they are not in use--an important attribute in extending battery life (along with frugal use of the LCD). The DVC300 draws all power solely from the USB connector, making low power consumption especially critical.

Sierra Imaging has just announced its Raptor coprocessor for Fujitsu's SPARClite embedded processor, offering fast on-chip hardware multiply and interface flexibility to many front-end CCD and CMOS sensors and back-end displays and transfer buses. Following the math coprocessor model, you can easily imagine imaging coprocessors and CPU cores integrated on a common ASIC within a short time. Sierra provides several digital-photography products: coprocessors, full camera reference designs, and host-based editing software. Whereas the company's first reference design based on SPARClite executed roughly 100,000 instructions/image in approximately 8 sec, the coprocessor, along with CPU-perform-ance improvements, will enable completion of more than 500,000 instructions/image in less than 1 sec.

Kodak's DC120 uses the Hitachi SH7034 for both control and simple image-processing functions, plus a separate ASIC for DCT-based compression. The DC210, on the other hand, outputs JPEG and FlashPix finished files; therefore, its processing requirements are more rigorous. The DC210 uses Hitachi's new SH-DSP processor, a single-chip combination of the 32-bit SH-2 and a 16-bit imaging-optimized DSP. Kodak's DCS460 includes an Intel 80C196 for control and voice annotation of the image files, plus two Xilinx (San Jose, CA) FPGAs for fast, reconfigurable, hardware-based image processing, enabling multiple-shot bursts. Kodak plans to use Motorola's (Schaumburg, IL) MPC800 embedded PowerPC processors for cameras in development, part of the two companies' recently announced joint technology agreement that also includes CMOS-sensor development. Atmel (San Jose, CA) and Polaroid also have a joint development relationship, with a similar future vision of the single-chip (sensor and processor) digital camera. Finally, LSI Logic just announced its DCAM-101 image processor, based on a MIPS (Mountain View, CA) R3000 core (Figure 6).

References

1. Kempainen, Stephen, "CMOS image sensors: eclipsing CCDs in visual information?" EDN, Oct 9, 1997, pg 101.

2. Digital Camera Companion (ISBN 1-57610-097-9), Coriolis Group Books, Scottsdale, AZ, 1997.

3. Internet news groups: http://www.alt.comp.periphs.scanners/, http://www.comp.graphics.apps.photoshop/, http://www.comp.periphs.printers/, http://www.comp.periphs.scanners/, and http://www.rec.photo.digital/.

Acknowledgments

I'm grateful to Kodak representatives Andrea Taft, Ken Parulski, and Michael McCreary for their time, effort, information, and insights. Thanks also to Intel's Don Verner, Maarten DeHaan at Polaroid, Max Baron at Fujitsu, and the folks at Sierra Imaging.

| EDN Access | Feedback | Table of Contents |