Image and Video Compression for Multimedia Engineering

Transcription

1 Image and Video Compression for Multimedia Engineering Fundamentals, Algorithms, and Standards Second Edition Yun Q. Shi New Jersey Institute of Technology Netvark, New Jersey, USA Huifang Sun Mitsubishi Electric Research Laboratories Cambridge, Massachusetts, USA (g) Boca Raton London CRC Press Taylor & Francis Group NewYork CRC Press is an imprint of the Taylor & Francis Group, an informa business

2 Contents Preface to the Second Edition Preface to the First Edition Content and Organization of the Book Authors xxi xxiii xxvii xxxi Part I Fundamentals Chapter 1 Introduction 1.1 Practical Needs for Image and Video Compression Feasibility of Image and Video Compression Statistical Redundancy Spatial Redundancy Temporal Redundancy Coding Redundancy Psychovisual Redundancy Luminance Masking Texture Masking Frequency Masking Temporal Masking Color Masking Color Masking and Its Application in Video Compression Summary: Differential Sensitivity Visual Quality Measurement Subjective Quality Measurement Objective Quality Measurement Signal to Noise Ratio An Objective Quality Measure Based on Human Visual Perception Information Theory Results Entropy Information Measure Average Information per Symbol Shannon's Noiseless Source Coding Theorem Shannon's Noisy Channel Coding Theorem Shannon's Source Coding Theorem Information Transmission Theorem Summary 30 Exercises 30 References 31 Chapter 2 Quantization 2.1 Quantization and the Source Encoder Uniform Quantization Basics 36 vii

3 viii Definitions Quantization Distortion Quantizer Design Optimum Uniform Quantizer Uniform Quantizer with Uniformly Distributed Input Conditions of Optimum Quantization Optimum Uniform Quantizer with Different Input Distributions Nonuniform Quantization Optimum (Nonuniform) Quantization Companding Quantization Adaptive Quantization Forward Adaptive Quantization Backward Adaptive Quantization Adaptive Quantization with a One-Word Memory Switched Quantization Pulse Code Modulation Summary 56 Exercises 57 References 58 Chapter 3 Differential Coding 3.1 Introduction to DPCM Simple Pixel-to-Pixel DPCM General DPCM Systems Optimum Linear Prediction Formulation Orthogonality Condition and Minimum Mean Square Error Solution to Yule-Walker Equations Some Issues in the Implementation of DPCM Optimum DPCM System D, 2-D, and 3-D DPCM Order of Predictor Adaptive Prediction Effect of Transmission Errors Delta Modulation Interframe Differential Coding Conditional Replenishment D DPCM Motion Compensated Predictive Coding Information-Preserving Differential Coding Summary 77 Exercises 78 References 78 Chapter 4 Transform Coding 4.1 Introduction Hotelling Transform Statistical Interpretation Geometrical Interpretation 84

4 ix Basis Vector Interpretation Procedures of Transform Coding Linear Transforms D Image Transformation Kernel Separability Symmetry Matrix Form Orthogonality Basis Image Interpretation Subimage Size Selection Transforms of Particular Interest Discrete Fourier Transform Discrete Walsh Transform Discrete Hadamard Transform Discrete Cosine Transform Background Transformation Kernel Relationship with DFT Performance Comparison Energy Compaction Mean Square Reconstruction Error Computational Complexity Summary Bit Allocation Zonal Coding Threshold Coding Thresholding and Shifting Normalization and Roundoff Zigzag Scan Huffman Coding Special Code Words Rate Buffer Feedback and Equalization Some Issues Effect of Transmission Error Reconstruction Error Sources Comparison between DPCM and TC Hybrid Coding Summary 110 Exercises 112 References 112 Chapter 5 Variable-Length Coding: Information Theory Results (II) 5.1 Some Fundamental Results Coding an Information Source Some Desired Characteristics Block Code Uniquely Decodable Code Instantaneous Codes Compact Code Discrete Memoryless Sources 120

5 X Extensions of a Discrete Memoryless Source Definition Entropy Noiseless Source Coding Theorem Huffman Codes Required Rules for Optimum Instantaneous Codes Huffman Coding Algorithm Procedures Comments Applications Modified Huffman Codes Motivation Algorithm Codebook Memory Requirement Bounds on Average Code Word Length Arithmetic Codes Limitations of Huffman Coding The Principle of Arithmetic Coding Dividing Interval [0,1) into Subintervals Encoding Decoding Observations Implementation Issues Incremental Implementation Finite Precision Other Issues History Applications Summary 137 Exercises 138 References 139 Chapter 6 Run-Length and Dictionary Coding: Information Theory Results (III) 6.1 Markov Source Model Discrete Markov Source Extensions of a Discrete Markov Source Definition Entropy Autoregressive Model Run-Length Coding D Run-Length Coding D Run-Length Coding Five Changing Pixels Three Coding Modes Effect of Transmission Error and Uncompressed Mode Error Effect in the 1-D RLC Case Error Effect in the 2-D RLC Case Uncompressed Mode 149

6 xi 6.3 Digital Facsimile Coding Standards Dictionary Coding Formulation of Dictionary Coding Categorization of Dictionary-Based Coding Techniques Static Dictionary Coding Adaptive Dictionary Coding Parsing Strategy Sliding Window (LZ77) Algorithms Introduction Encoding and Decoding Summary of the LZ77 Approach LZ78 Algorithms Introduction Encoding and Decoding LZWAlgorithm Summary Applications International Standards for Lossless Still Image Compression Lossless Bilevel Still Image Compression Algorithms Performance Comparison Lossless Multilevel Still Image Compression Algorithms Performance Comparison Summary 162 Exercises 163 References 164 Part II Still Image Compression Chapter 7 Still Image Coding: Standard JPEG 7.1 Introduction Sequential DCT-Based Encoding Algorithm Progressive DCT-Based Encoding Algorithm Lossless Coding Mode Hierarchical Coding Mode Summary 178 Exercises 178 References 178 Chapter 8 Wavelet Transform for Image Coding: JPEG A Review of Wavelet Transform Definition and Comparison with Short-Time Fourier Transform Discrete Wavelet Transform Lifting Scheme Three Steps in Forward Wavelet Transform Inverse Transform 186

7 xii Lifting Version of CDF (2,2) A Demonstration Example (5,3) Integer Wavelet Transform A Demonstration Example of (5,3) IWT Summary Digital Wavelet Transform for Image Compression Basic Concept of Image Wavelet Transform Coding Embedded Image Wavelet Transform Coding Algorithms Early Wavelet Image Coding Algorithms and Their Drawbacks Modern Wavelet Image Coding Embedded Zerotree Wavelet Coding Set Partitioning in Hierarchical Trees Coding Wavelet Transform for JPEG Introduction of JPEG Requirements of JPEG Parts of JPEG Verification Model of JPEG An Example of Performance Comparison between JPEG andjpeg Summary 200 Exercises 200 References 202 Chapter 9 Nonstandard Still Image Coding 9.1 Introduction Vector Quantization Basic Principle of Vector Quantization Vector Formation Training Set Generation Codebook Generation Quantization Several Image Coding Schemes with Vector Quantization Residual VQ ClassifiedVQ Transform Domain VQ PredictiveVQ Block Truncation Coding Lattice VQ for Image Coding Fractal Image Coding Mathematical Foundation IFS-Based Fractal Image Coding Other Fractal Image Coding Methods Model-Based Coding Basic Concept Image Modeling Summary 217 Exercises 217 References 217

8 X1H Part III Motion Estimation and Compensation Chapter 10 Motion Analysis and Motion Compensation 10.1 Image Sequences Interframe Correlation Frame Replenishment Motion Compensated Coding Motion Analysis Biological Vision Perspective Computer Vision Perspective Signal Processing Perspective Motion Compensation for Image Sequence Processing Motion Compensated Interpolation Motion Compensated Enhancement Motion Compensated Restoration Motion Compensated Down-Conversion Summary 236 Exercises 237 References 238 Chapter 11 Block Matching 11.1 Nonoverlapped, Equally Spaced, Fixed Size, Small Rectangular Block Matching Matching Criteria Searching Procedures FullSearch D Logarithm Search Coarse-Fine Three-Step Search Conjugate Direction Search Subsampling in the Correlation Window Multiresolution Block Matching Thresholding Multiresolution Block Matching Algorithm Threshold Determination Thresholding Experiments Matching Accuracy Limitations with Block Matching Techniques New Improvements Hierarchical Block Matching Multigrid Block Matching Thresholding Multigrid Block Matching Optimal Multigrid Block Matching Predictive Motion Field Segmentation Overlapped Block Matching Summary 267 Exercises 269 References 269

9 xiv Chapter 12 Pel Recursive Technique 12.1 Problem Formulation Descent Methods First-Order Necessary Conditions Second-Order Sufficient Conditions Underlying Strategy Convergence Speed Order of Convergence Linear Convergence Steepest Descent Method Formulae Convergence Speed Selection of Step Size Newton-Raphson's Method Formulae Convergence Speed Generalization and Improvements Other Methods Netravali-Robbins' Pel Recursive Algorithm Inclusion of a Neighborhood Area Interpolation Simplification Performance Other Pel Recursive Algorithms Bergmann's Algorithm (1982) Bergmann's Algorithm (1984) Cafforio and Rocca's Algorithm Walker and Rao's Algorithm Performance Comparison Summary 283 Exercises 284 References 284 Chapter 13 Optical Flow 13.1 Fundamentals D Motion and Optical Flow Aperture Problem III-Posed Problem Classification of Optical Flow Techniques Gradient-Based Approach Hörn and Schunck's Method Brightness Invariance Equation Smoothness Constraint Minimization Iterative Algorithm Modified Hörn and Schunck Method Lucas and Kanade's Method Nagel's Method Uras, Girosi, Verri, and Torre's Method 297

10 XV 13.3 Correlation-Based Approach Anandan's Method Singh's Method Conservation Information Neighborhood Information Minimization and Iterative Algorithm Pan, Shi, and Shu's Method Proposed Framework Implementation and Experiments Discussion and Conclusion Multiple Attributes for Conservation Information Weng, Ahuja, and Huang's Method Xia and Shi's Method Multiple Image Attributes Conservation Stage Propagation Stage Outline of Algorithm Experimental Results Discussion and Conclusion Summary 321 Exercises 323 References 324 Chapter 14 Further Discussion and Summary on 2-D Motion Estimation 14.1 General Characterization Aperture Problem Ill-Posed Inverse Problem Conservation Information and Neighborhood Information Occlusion and Disocclusion Rigid and Nonrigid Motion Different Classifications Deterministic Methods versus Stochastic Methods Spatial Domain Methods versus Frequency Domain Methods Optical Flow Determination Using Gabor Energy Filters Region-Based Approaches versus Gradient-Based Approaches Forward versus Backward Motion Estimation Performance Comparison between Three Major Approaches Three Representatives Algorithm Parameters Experimental Results and Observations New Trends DCT-Based Motion Estimation DCT and DST Pseudophases Sinusoidal Orthogonal Principle Performance Comparison Summary 340 Exercises 341 References 341

11 XVI Part IV Video Compression Chapter 15 Fundamentals of Digital Video Coding 15.1 Digital Video Representation Information Theory Results: Rate Distortion Function of Video Signal Digital Video Formats Digital Video Color Systems Progressive and Interlaced Video Signals Video Formats Used by Video Industry ITU-R Source Input Format Common Intermediate Format ATSC Digital Television Format Current Status of Digital Video/Image Coding Standards JPEG Standard JPEG MPEG MPEG MPEG H H.263, H.263 Version 2 (H.263+), H.263++, and H.26L MPEG-4 Part 10 Advanced Video Coding or H.264/AVC VC RealVideo Summary 356 Exercises 356 References 357 Chapter 16 Digital Video Coding Standards: MPEG-1/2 Video 16.1 Introduction Features of MPEG-1/2 Video Coding MPEG-1 Features Introduction Layered Structure Based on Group of Pictures Encoder Structure Structure of the Compressed Bitstream Decoding Process MPEG-2 Enhancements Field/Frame Prediction Mode Field/Frame DCT Coding Syntax Downloadable Quantization Matrix and Alternative Scan Order Pan and Scan Concealment Motion Vector Scalability MPEG-2 Video Encoding Introduction Preprocessing 373

12 xvii Motion Estimation and Motion Compensation Matching Criterion Searching Algorithm Advanced Motion Estimation RateControl Introduction of Rate Control Rate Control of Test Model 5 for MPEG Optimum Mode Decision Problem Formation Procedure for Obtaining the Optimal Mode Optimal Solution Near-Optimal Greedy Solution Practical Solution with New Criteria for the Selection of Coding Mode Statistical Multiplexing Operations on Multiple Program Encoding Background of Statistical Multiplexing Operation VBR Encoders in StatMux Research Topics of StatMux Summary 393 Exercises 393 References 394 Chapter 17 Application Issues of MPEG-1/2 Video Coding 17.1 Introduction ATSC DTV Standards A Brief History Technical Overview of ATSC Systems PictureLayer Compression Layer Transport Layer Transmission Layer Transcoding with Bitstream Scaling Background Basic Principles of Bitstream Scaling Architectures of Bitstream Scaling Architecture 1: Cutting AC Coefficients Architecture 2: Increasing Quantization Step Architecture 3: Re-Encoding with Old Motion Vectors and Old Decisions Architecture 4: Re-Encoding with Old Motion Vectors and New Decisions Comparison of Bistream Scaling Methods MPEG-2 to MPEG-4 Transcoding Down-Conversion Decoder Background Frequency Synthesis Down-Conversion Low-Resolution Motion Compensation Three-Layer Scalable Decoder Summary of Down-Conversion Decoder 418

13 xvm 17.5 Error Concealment Background Error Concealment Algorithms Code Word Domain Error Concealment Spatio-Temporal Error Concealment Algorithm Enhancements Directional Interpolation I-Picture Motion Vectors Spatial Scalable Error Concealment Summary of Error Concealment Summary 431 Exercises 432 References 432 Chapter 18 MPEG-4 Video Standard: Content-Based Video Coding 18.1 Introduction MPEG-4 Requirements and Functionalities Content-Based Interactivity Content-Based Manipulation and Bitstream Editing Synthetic and Natural Hybrid Coding Improved Temporal Random Access Content-Based Efficient Compression Improved Coding Efficiency Coding of Multiple Concurrent Data Streams Universal Access Robustness in Error-Prone Environments Content-Based Scalability Summary of MPEG-4 Features Technical Description of MPEG-4 Video Overview of MPEG-4 Video Motion Estimation and Compensation Adaptive Selection of 16 X 16 Block or Four 8X8 Blocks Overlapped Motion Compensation Texture Coding INTRA DC and AC Prediction Motion Estimation/Compensation of Arbitrary ShapedVOP Texture Coding of Arbitrary Shaped VOP Shape Coding Binary Shape Coding with CAE Algorithm Gray-Scale Shape Coding Sprite Coding Interlaced Video Coding Wavelet-Based Texture Coding Decomposition of the Texture Information Quantization of Wavelet Coefficients Coding of Wavelet Coefficients of Low-Low Band and Other Bands Adaptive Arithmetic Coder 450

14 xix Generalized Spatial and Temporal Scalability Error Resilience MPEG-4 Visual Bitstream Syntax and Semantics MPEG-4 Visual Profiles and Levels MPEG-4 Video Verification Model VOP-Based Encoding and Decoding Process Video Encoder Video Segmentation Intra/Inter Mode Decision Off-Line Sprite Generation Multiple VO Rate Control Video Decoder Summary 460 Exercises 461 References 461 Chapter 19 ITU-T Video Coding Standards H.261 and H Introduction H.261 Video Coding Standard Overview of H.261 Video Coding Standard Technical Detail of H Syntax Description PictureLayer Group of Blocks Layer Macroblock Layer Block Layer H.263 Video Coding Standard Overview of H.263 Video Coding Technical Features of H Half-Pixel Accuracy Unrestricted Motion Vector Mode Advanced Prediction Mode Syntax-Based Arithmetic Coding PB-Frames H.263 Video Coding Standard Version Overview of H.263 Version New Features of H.263 Version Scalability Improved PB-Frames Advanced Intracoding Deblocking Filter Slice-Structured Mode Reference Picture Selection Independent Segmentation Decoding Reference Picture Resampling Reduced-Resolution Update Alternative Inter VLC and Modified Quantization Supplemental Enhancement Information H Video Coding and H.26L 481

15 XX 19.6 Summary 481 Exercises 481 References 482 Chapter 20 A New Video Coding Standard: H.264/AVC 20.1 Introduction Overview of H.264/AVC Codec Structure Technical Description of H.264/AVC Coding Tools Instantaneous Decoding Refresh Picture Switching I-Slices and Switching P-Slices Transform and Quantization Intraframe Coding with Directional Spatial Prediction Adaptive Block Size Motion Compensation Motion Compensation with Multiple References Entropy Coding Loop Filter Error-Resilience Tools Profiles and Levels of H.264/AVC Profiles of H.264/AVC Levels of H.264/AVC Summary 505 Exercises 505 References 505 Chapter 21 MPEG System: Video, Audio, and Data Multiplexing 21.1 Introduction MPEG-2 System Major Technical Definitions in MPEG-2 System Document Transport Streams Structure of Transport Streams Transport Stream Syntax Transport Streams Splicing Program Streams Timing Model and Synchronization MPEG-4 System Overview and Architecture Systems Decoder Model Scene Description Object Description Framework Summary 526 Exercises 526 References 527 Index 529