Technological advancements of recent years have enabled new video capabilities in end equipment ranging from cell phones and MP3 players to video walls and billboards. Because of the strong demand and level of innovation that characterize digital-video markets, system manufacturers must consider what form of video to include in their new products. One of the major issues is whether to provide for HD (high-definition) displays, which offer much better image quality than viewers were accustomed to in the past. Given the market's forward impetus toward bigger and better equipment, the answer would seem to be a no-brainer: yes.

But it's important that video-system developers be clear about what HD involves to avoid the impression that everything must be HD. You can call a variety of display formats "high-definition," and some or all of these may be inappropriate in certain applications. In addition, you must weigh the quality of the display against the level of signal compression. And, in any system, there is an issue of cost. The real question, then, is not so much whether an application requires HD, but how to achieve the best possible quality given the system's display, bandwidth, and storage constraints, as well as consumers' price expectations. Once you determine this issue, you can select an underlying processing technology that meets the general system requirements for performance, flexibility, and cost efficiency.

Looking for an all-inclusive description of HD reveals how slippery the term can be. For most people and in most uses, "HD" refers to HDTV, with its wider screen and higher resolution than traditional SDTV (standard-definition TV). Although the width-to-height aspect ratio for SDTV is 1.33-to-1 (4-to-3), HDTV at 1.78-to-1 (16-to-9) is closer to the dimensions of most cinema releases, so there is less need for cropping or letterboxing to fit film images onto the screen. The top resolution that manufacturers define for HDTV offers six times the visual information of SD, making details appear correspondingly crisper and colors more intense. Because of the digital transmission of HDTV, pictures either come through, or they do not: The snowy, washed-out images and vertical rolling of analog broadcasts are things of the past. A digital source also enables multiple images to simultaneously fit onto the same screen for easier channel surfing or reference to programming schedules and other information. When multispeaker surround sound adds to these visual advantages, the result can be an astonishing new experience in home entertainment.

The appeal of large, brilliant screens that closely mimic the movie-theater experience is the driving force when it comes to marketing the new digital-transmission technology to consumers. But HDTV is not synonymous with DTV (digital TV), which has a broader definition, though public perception may have confused the two terms. Manufacturers have been phasing in DTV in the United States over the last several years, with the governing ATSC (Advanced Television Standards Committee) defining a number of commonly used digital-broadcast formats. DVB (Digital Video Broadcasting), the equivalent international organization, has also defined formats that overlap with the ATSC formats.

Table 1 lists some of the common display formats for DTV, showing that there are a number of HDTV formats. Almost all HD displays are now progressive-scan, and HDTV sets with 1080p (1920×1080-pixel resolution, progressive scan) have only recently become available. In countries that transmit PAL (phase-alternation-line)- or SECAM (séquentiel couleur avec mémoire)-based analog television, refresh rates of 25 and 50 frames/sec find use, rather than the NTSC (National Television System Committee) rates of 30 and 60 frames/sec in the United States. The 480i60 (720×480-pixel resolution at 60 frames/sec, interlaced-scan) SD format provides a digital approximation of NTSC analog-TV reception, whereas 576i60 is a digital approximation of PAL and SECAM. HDTV screens can also display these SD formats, though not at the full screen width. Manufacturers sometimes market the 480p60 and 576p60 formats, as well as some resolutions that standards do not list, such as 1080×720 pixels, under the label EDTV (extended-definition TV). For compatibility with film, standards define 24 frames/sec for use in video production, though it is not a broadcast-transmission rate.

Digital video also has low-end resolutions, which some refer to as LDTV (low-definition TV), which often finds use in Internet streaming and low-end video. In these low-end formats, the basic unit is CIF (Common Intermediate Format), which at 352×288 pixels is roughly one-quarter of the SD-screen resolutions. CIF and its subdivisions, such as QCIF (quarter CIF, or 176×144 pixels), are familiar in computer streaming video and provide the basis for divided-screen applications on DTVs. CIF also has its multiples: 4CIF (4 times CIF, or 704×576 pixels), 9CIF (9 times CIF, or 1056×864 pixels), and 16CIF (16 times CIF, or 14081×152 pixels). These higher end CIFs overlap in scale with but do not match the HDTV formats, so you can use LDTV to build HD formats that do not correlate with HDTV resolutions.

At the high end, new formats, such as the lab-demonstrated UHDV (ultra-high-definition video), are appearing that even further push display technology. UHDV provides 16 times the pixels of a 1920×1080-pixel image. Even without going to these spectacular lengths, you can find equipment that goes beyond HDTV resolutions in digital-cinema and commercial-video production. In addition, a range of HD-computer-graphics formats sometimes finds use in video display. Although these computer-graphics formats traditionally exhibit the 1.33-to-1 SD aspect ratio of CRTs, some of the recent variants incorporate support for wide-screen LCDs that can handle HD-digital video, among other applications. These wider formats include WXGA (wide extended graphics adapter) at 1280×800 pixels, WSXGA+ (wide super XGA+) at 1680×1050 pixels, and WUXGA (wide ultra XGA) at 1920×1200 pixels, all with aspect ratios of 1.6-to-1. Refresh rates for these formats are almost always progressive-scan and much faster than DTV rates. Although these formats differ from the HDTV-standard formats, computer- and DTV-display technology is so closely linked today that most HDTV sets actually use VESA (Video Electronics Standards Association) resolutions, such as WXGA, with the TV converting the video from the broadcast or recorded format to the actual display format.

Clearly, a developer had better understand the range of display requirements for a system before undertaking an HD design. Just as important as the technical requirements, though, are the perceptual ones: How an end user employs the system can make an even bigger difference than the format of the display. First, HD's effects are perceptible only with large displays: At less than 40 inches diagonally, the display largely loses the extra resolution. Obviously, HD would be pointless for a handheld display on a cell phone. But even in a small display for, say, viewing in the back seat of a car, a viewer cannot tell the difference between HD and SD.

Second, the effect of surround sound is even more dramatic than that of a high-resolution display. In other words, great sound will make a so-so image seem better, whereas inadequate sound will diminish the effect of a great image. Manufacturers should look to improve audio along with, or even before, video.

The third factor affecting image quality is video compression and decompression. Normally, the system compresses digital video to reduce the enormous bandwidth it requires, which, uncompressed, would exceed 124 Mbps for the SDTV broadcast formats and approach 750 Mbps for 1080i60. Storage also is a factor, because single-layer DVDs can hold approximately 4.7 Gbytes of data—enough for only short clips of uncompressed video. Double-layer HD DVD and Blu-ray discs extend storage to approximately 30 and 50 Gbytes, respectively, but they still require a huge amount of compression to hold hours' worth of video content.

The Main Profile of MPEG-2 (Moving Pictures Experts Group 2), the best established and most widely used standard for video compression, normally provides high-quality, source-dependent compression at ratios of approximately 30-to-1 to 50-to-1, using 4:2:0 color sampling. Because H.264, also known as MPEG-4 AVC (Advanced Video Coding) and as MPEG-4 Part 10 Main Profile, roughly doubles this level of compression, the video-broadcast and -recording industry will be moving to the new standard during the next few years. All of the ITU (International Telecommunications Union)/MPEG standards are lossy, however, so the played-back decompressed image is by nature less well-defined than the original image before compression. Because the images are in motion and because the standards' developers based them on a great deal of study about how people perceive images, the technologies conceal the loss of image definition so well that it is generally unnoticeable. However, pushing this loss beyond the approximately 60-to-1 to 100-to-1 compression ratio of H.264's High Profile risks revealing flaws in the image; these flaws show up much better with HD displays.

Decompressed 1080i60 video requires six times the amount of data that decompressed SD video requires, and 720p60 more than 5.3 times as much. In raw terms, therefore, the system must provide six times the processing throughput and memory for HD as it would for SD. Moreover, because the more advanced codecs achieve greater compression by employing more memory and processing, the system requirements become correspondingly higher. For instance, the memory requirement for an MPEG-4 Simple Profile 480i30 SD decoder for reference-frame data is approximately 500 kbytes, whereas the minimum requirement for an H.264 High Profile decoder for 1080i60 HD is about 9 Mbytes—18 times higher. This increase comes not only from the greater HD pixel resolution, but also from the fact that MPEG-4 Simple Profile requires only one frame for its frame-prediction algorithm whereas H.264 Level 4.0 High Profile requires five frames. For HD decoding, the processor therefore must have 18 times the available memory for internal reference-frame buffers. It also requires additional memory for display buffering and other decoder functions, such as stream buffers and tables (Figure 1).

Table 2 summarizes the general system requirements of three representative and widely varying HD-codec applications. The first, broadcast HDTV, outputs the highest bit rate and operates in real time using either MPEG-2 or H.264 for compression. Using today's technology, broadcast systems that compress HDTV programs for transmission require numerous high-frequency DSPs (digital-signal processors) and FPGAs (field-programmable gate arrays), supported by several gigabytes of memory. A number of these devices, in a "blade" arrangement, may operate as a parallel-processing farm to provide multiple channels of compressed-HDTV output.

Another application, videoconferencing, needs to scale to accommodate the low-megabit-range bandwidths available through many WAN (wide-area-networking) links yet still be able to provide an HD image that can appear on a large display. Latency in the video-coding and -decoding process must be minimal to not interfere with conversation. Given the transmission requirements and assuming that you are using the H.264 Baseline Profile for compression, the 720p30 HD format—somewhat lower in resolution than the HDTV example but still high end—is practical. Today's technology requires several DSPs and hundreds of megabytes of memory to satisfy this application.

A common consumer application for video compression is in DSCs (digital still cameras), which can capture short video clips for viewing on HD displays. DSC systems have to be easy-to-use and inexpensive, both factors leading to the use of MPEG-4 Simple Profile to minimize processing and memory requirements. A single highly integrated DSP-based SOC (system on chip) for portable-system applications can perform the codec requirements of this application (Table 3).

Obviously, the requirements of HD systems vary widely, depending on bandwidth and compression levels, as well as display formats. Video clips stored using a DSC would look primitive in a broadcast application, for example, whereas an HD-broadcast bit stream would overwhelm a videoconferencing system with data. But, at any level of application, HD formats will have considerably greater requirements for memory and processing than SD formats do. These requirements translate into higher component costs, which, like almost all semiconductor costs, will diminish predictably over time. For the system manufacturer, then, the fundamental question may be whether to build in HD support now at today's cost or stay with SD support for the next year or few years until component costs are lower and HD demand has increased.

Manufacturers must also consider the versatility of their designs, because virtually every digital-video system today has to take into account the continual introduction of, and improvement in, codecs. The influence of H.264, whether for DTV broadcasts, IPTV (Internet Protocol TV), videoconferencing, or other applications, will be significant in the next few years. Competing standards, such as WMV9 (Windows Media Version 9)/VC-1 and China's AVS (Audio Video Coding), and the ITU/MPEG standards all offer variations in implementation. Systems such as set-top boxes may have to dynamically deal with a number of standards and variations, interface with entertainment and gaming consoles, and support home-computer networks and, eventually, videophones.

It may be important for such a system to not only decode, but also transcode and transrate, video streams to support different displays and handle application and control software. Even an application such as video surveillance needs the ability to upgrade its codec and add features such as object analysis and recognition. When you add variability of video-input-stream formats and end applications to the variety of potential HD outputs, the need for system flexibility becomes apparent. Designers must bear this need in mind as they select an enabling technology for their video systems.

To support the high-throughput, multiple-application requirements of an HD-video system, a processor must provide both performance and versatility at a reasonable cost. By design, DSPs supply a high level of performance for handling real-time algorithms, such as audio/video codecs and HD-rate data streams. Processors that integrate both DSP and RISC (reduced-instruction-set-computer) cores have the additional advantage of being able to partition performance between the DSP for signal processing and the RISC for control, communications, and applications software. Multicore DSPs for audio/video applications also include VICPs (video-image coprocessors) that provide hardware acceleration for operations that video codecs frequently use; they also provide additional on-chip hardware, such as video scaling and blending of graphics and video for creating the on-screen display to further offload video-display processing.

Programmable DSPs provide the flexibility to support a variety of codec and display standards, and they also allow quick system adaptation to accommodate new functions. In addition, you can readily reprogram the same basic design to meet requirements for different market segments and regions. DSPs have in recent years also become more user-friendly by offering a comprehensive, open-software platform with audio/video APIs (application-programming interfaces) that make the DSP transparent, so that the developer has only to program the RISC using C and standard development tools. Finally, DSPs that feature SOC integration with a memory subsystem and peripherals for video can help minimize system costs even in systems with HD and other advanced video features.