EDN hands-on story: Video characterization creates hands-on headaches, part two
How well do video codecs decrease bit rates and still preserve quality? And can you ascertain their credentials without recruiting an army of test subjects? The answer: a qualified "yes."
By Brian Dipert, Technical Editor -- EDN, August 8, 2002
Last issue in part 1, I described my motivation for this video-compression project, discussed background info to the analysis tools I'd be using and codecs I'll be testing, and detailed the problems I encountered and surmounted during the testing (EDN, July 25, pg 53). Read on for the results and my future plans, to appear on the Web-site addendum to the article series.
As you analyze the results in Table 1 (pg 84), keep in mind the mammoth file sizes of the source clips I used. Comparing them with the compressed file sizes in the table, you should be impressed with how well the codecs retain a semblance of the original image at such significantly slimmed bit rates. I resist drawing any definitive conclusions from the table because, depending on what clip or combinations of clips represent the kind of video media you typically encode and whether encoding speed, resulting file size, image quality, or some combination of these variables in an application-specific priority ordering is important to you, you'll likely reach a different conclusion from other readers. As I put on various applications' "sets of eyes," I consequently interpreted the data in unique ways.
The most important question to ask up-front is whether JNDmetrix results correlate to the quality differences I observed. I am confident that JNDmetrix is doing all that it claims to do and that its results are meaningful beyond its MPEG heritage. The codec-versus-codec numbers matched what my eyes and brain interpreted, and it was fascinating to peruse the report files (available for download on the EDNWeb-site addendum to this article) and see how the frame-to-frame JNDmetrix number variances matched changes in scene complexity and transitions between key and delta frames.
The JNDmetrix reports give per-frame results for both luma and chroma; combine them, placing a higher priority on luma, as does the human visual system; and then average the per-frame results to give a clip result. I had originally intended to put not only the clip average, but also the "best" and "worst" frame results in the table. However, Dixon convinced me that this approach would unfairly represent the codecs, because one or two "bad" frames in an otherwise-solid clip would be acceptable to the eyes. I think that standard-deviation numbers in the reports would be useful, however. Options I didn't enable further break down the numbers into various color channels, and you can even obtain per-frame bit maps to display the pixel-versus-pixel variances from reference to test clip. As you compare the JND numbers for the codecs and settings, the following guidelines from Sarnoff's documentation may be of benefit:
"A 1 JND rating indicates a 75% probability of seeing a difference between two images (vs. 50% guess). Two image sequences below 1 JND apart cannot be distinguished, even when the exact nature and location of differences are known in advance. Two image sequences below 3 JNDs apart are not obviously different. Differences are visible, but only when an observer knows exactly where to look. At 5 JNDs, differences are readily apparent. In probability terms, a 2 JND difference corresponds to a 93.75% probability of discrimination (0.75+0.75×(1–0.75)), and a 3 JND difference corresponds to a 98.44% probability. Although probability of discrimination asymptotes quickly as a function of JNDs, the units are useful because they correspond to roughly linear magnitudes of subjective visual difference."
One of the first things you might notice is that the codecs only rarely achieved an encoded bit rate that was near, at, or below the 300- or 800-kbps target—a disappointing fact. Admittedly, I intentionally chose many challenging clips. But when I tell an encoder that I want a certain average bit rate, regardless of whether I'm encoding with VBR or CBR, the encoder should hit that bit rate without my having to do iterative work. When you specify CBR encoding in WMV8, you might expect the Windows Media Encoder to deliver a bit stream with a constant number of bits per video frame.
Look closely at the WMV encoding report files (also available at the Web addendum), though, and you discover that the Windows Media Encoder delivers a constant transport rate stream with bounded bit-rate variance on a frame-by-frame basis. Note the term "bounded." The encoder defines an intermediate buffer, whose size the encoded bit stream communicates to the decoder. Streaming CBR clips from a server results in an initial buffer-filling delay before playback begins. However, a frame's or a sequence of frames' encoded bits are never large enough to exceed the available playback-buffer size.
WMV's VBR mode follows the customary unbounded per-frame bit-rate definition the industry uses. One-pass VBR mode, according to Microsoft documentation, prioritizes the specified quality over the specified bit rate, whereas two-pass VBR flips these variables' priorities. A comparison of the one- and two-pass numbers by and large validate these suppositions, though a few baffling contradictions existed. More generally, look at whether the longer encoding times of two-pass mode in both CBR and VBR encoding result in higher enough JNDmetrix quality measures for the types of media you're interested in.
The Ligos CBR encoder, like all modern codecs that support a mix of key and delta frames, is also a bounded-bit-rate VBR technology. Analysis of the MPEG-4 bit streams in Interra's M4 software verifies the presence of a mix of I and—theoretically much smaller—P frames, correlating to the encoder defaults during compression. Note that the MPEG-4 Simple Profile Level 3 decode buffers are much smaller than those that Windows Media Video requires. This memory-density difference may be important in cost-conscious consumer-electronics devices.
JNDmetrix-analysis comparisons of MPEG-4 and WMV8 bear out what my eyes saw. For MPEG-4 files smaller than WMV8 files, large smeared areas appeared in MPEG-4's images, whereas the WMV8's images contained detail. More important, with comparably sized files, MPEG-4 contained more large "blocky" artifacts and other irregularities, such as significant loss of facial details. The poor quality extends to MPEG-4 files significantly larger than those of WMV8. But keep in mind that those artifacts would be far less disagreeable on a tiny cell- phone screen display than on the large monitors I viewed them on. And, because I couldn't measure processor load during decoding, MPEG-4 Simple Profile may—and from anecdotal information, probably does—have an advantage in this regard over WMV8.
Comparing MPEG-2 with WMV8, like MPEG-4 versus WMV8, provides an intriguing industry standard-versus-proprietary trade-off that was one of my fundamental motivations for this study. I also hoped to infer some measure of the viability of advanced video codecs in conjunction with mature red-laser technology as an alternative to blue lasers for high-definition DVD. WMV8 generally achieved image quality moderately lower than to comparable to MPEG-2 with slightly to significantly smaller files than MPEG-2. But it did so with the trade-off of much slower encoding. Keep in mind that WMV8 encoding ran as a bare-bones DOS command-line utility, whereas the MPEG-2 and -4 encoders ran as plug-ins under Adobe Premiere. MPEG-encoding speeds in a simpler configuration might be even faster. And keep in mind, too, that WMV8 is months' old technology lacking bit-stream backward-compatibility constraints that other codecs have; MPEG-2 is nearing its 10th birthday and has bit-stream-format constraints.
Until recently, I thought that the reason JNDMetrix supported only the AVI and MPEG formats was that it integrated AVI and MPEG file-wrapper parsers. (It also inputs sequences of still image files at one per frame.) I was half-right. Actually, JNDmetrix hands off any file with an MPG extension to the PC's DirectShow filter stack for parsing. So, I theoretically may not need to decode a compressed file to AVI but could simply rename its file extension as MPG, although this technique doesn't currently seem to be working with MPEG-4.
I'm concerned, too, given my quality-degrading attempt to transcode MPEG-2 to lossless AVI, that the Vegas Video WMV decoder might be doing more quality damage than simply tacking on redundant frames, adversely and unfairly affecting the WMV scores. I'll try WMV-to-MPG renaming after Sarnoff adds support in JNDmetrix for DirectShow decoders that implement seeking via not only frame-stamping, which it currently supports, but also time-stamping. Microsoft has also promised me a more reliable WMV-to-AVI transcoder. I'll communicate any difference in the results to you via the Web-site addendum.
I'd like to obtain JNDmetrix results for MPEG-4 Advanced Simple Profile, for both the CIF- and D1-resolution reference clips. The MPEG-4 encoder in Apple Computer's QuickTime Version 6 Public Preview supports both Simple and Advanced Simple profiles and saves in both MP4 and Premiere-friendly MOV file formats. Unfortunately, it significantly prioritizes per-frame quality, offers limited adjustment of quality settings, and consequently drops frames unless I encode to high bit rates. For example, I had to encode the mobile_cif clip to 800 kbps to preserve frames, making comparisons with both Ligos' Simple Profile and Microsoft's Windows Media Video 8 meaningless. Dicas' encoder, from my limited experimentation thus far, offers more quality options and should produce more comparable bit rates.
I'd like to test other codecs via transcoding to AVI or through the DirectShow rename-to-MPG loophole. WMV9, code-named Corona, is due out this year and on my evaluation wish list. Other candidates include the open-source VP3 codec from On2 Technologies, which the company is offering to the Internet Streaming Media Alliance as an alternative to license-fee-burdened MPEG-4, and the company's more advanced, proprietary VP5 codec. DivX 5 is on my list, too. Apple's embrace of MPEG-4, coupled with its current legal quagmire with former partner Sorenson, has diminished the importance of evaluating Sorenson Video. I might also again try deinterlacing the VQEG clips with a more robust Cleaner, Adobe AfterEffects, Canopus ProCoder, or VirtualDub algorithm, but this task is low-priority.
Although I'd like to look at RealVideo, RealNetworks isn't cooperating, despite numerous discussions it conducted during the past few months, both with me and with Sarnoff. RealNetworks isn't confident in JNDmetrix's analysis results, and the company's programs don't supply a DirectShow filter that I can tap into. Third-party work-arounds, such as Tinra and TechSmith's Camtasia, exist, but RealNetworks can easily dismiss their transcoding accuracy. As a result, I depend on obtaining software from RealNetworks that decodes RealVideo to AVI, a series of still images, or some other format that JNDmetrix accepts.
I'd also like to assess how much my choice of computing hardware and operating system influenced my results. My motherboard supports the addition of a second PIII-800 CPU, and I also plan to install the multitasking Windows 2000 OS in a separate hard-drive partition. I also have a Pentium 4-based system that will be useful in case I determine that any of the encoders support and can performance-benefit from the CPU's SSE2 instruction set.
In all cases, after I, as a mainstream user, complete my work, I'll hand the reference AVIs to the codec vendors, as more advanced users, for optimized encoding. Numerous optimization options are possible, including processing the video before encoding it and as part of the subsequent decode; adjusting intermediate buffer sizes; altering the I, P, and B frame proportions and patterns—going so far, perhaps, as to manually insert key frames; and tweaking the quality and other encoder settings. In all cases, though, I'll insist that the vendors provide me in response 300-kbps CBR and 800-kbps VBR clips, the original targets of this project.
After my and the vendors' work is complete, my goal is to have a foundation that you can build on with your own analysis. How do the results differ at bit rates other than the ones I chose? Will slight tweaks to the encoder settings enable MPEG bit rates to more closely approximate my original 300- and 800-kbps targets with little to no loss in quality? Can the WMV encoder, through altered settings, accelerate compression while matching or even exceeding the quality levels I saw? I look forward to hearing from you and to adding your data to this article's Web-site addendum.
| Acknowledgments | ||
| Thanks to Doug Dixon from Sarnoff and Stewart Worrall from the University of Surrey for their support. They, along with Olivier Amato from Canalweb, Robert Saint James from Ligos, Amir Majidimehr and Ming-Chieh Lee from Microsoft, Rob Koenen from the MPEG-4 Industry Forum, and consultants Ben Waggoner and Brandon Wirtz, helped immeasurably. | ||
Acknowledgments
Thanks to Doug Dixon from Sarnoff and Stewart Worrall from the University of Surrey for their support. They, along with Olivier Amato from Canalweb, Robert Saint James from Ligos, Amir Majidimehr and Ming-Chieh Lee from Microsoft, Rob Koenen from the MPEG-4 Industry Forum, and consultants Ben Waggoner and Brandon Wirtz, helped immeasurably.
| Author Information |
 You can reach Technical Editor Brian Dipert at 1-916-454-5242, e-mail bdipert@edn.com. |
