MULTIDIMENSIONAL SIGNAL, IMAGE, AND VIDEO PROCESSING AND CODING JOHN W. WOODS Rensselaer Polytechnic Institute Troy, New York»iBllfllfiii.. i. ELSEVIER AMSTERDAM BOSTON HEIDELBERG LONDON NEW YORK OXFORD PARIS SAN DIEGO SAN FRANCISCO SINGAPORE SYDNEY TOKYO Academic Press is an imprint of Elsevier
CONTENTS Preface xiii Acknowledgments xvii v 1 TWO-DIMENSIONAL SiGNALS AND SYSTEMS 1 1.1 Two-Dimensional Signals 2 1.1.1 Separable Signals 6 1.1.2 Periodic signals 7 1.1.3 2-D Discrete-Space Systems 9 1.1.4 Two-Dimensional Convolution 11 1.1.5 Stability of 2-D Systems 13 1.2 2-D Discrete-Space Fourier Transform 14 1.2.1 Inverse 2-D Fourier Transform 18 1.2.2 Fourier Transform of 2-D or Spatial Convolution 19 1.2.3 Symmetry Properties of Fourier Transform 26 1.2.4 Continuous-Space Fourier Transform 28 1.3 Conclusions 31 1.4 Problems 31 References 33 2 SAMPLING IN TWO DIMENSIONS 35 2.1 Sampling Theorem Rectangular Case 36 2.1.1 Reconstruction Formula 40 2.1.2 Ideal Rectangular Sampling 43 2.2 Sampling Theorem General Regulär Case 48 2.2.1 Hexagonal Reconstruction Formula 52 2.3 Change of Sample Rate 57 2.3.1 Downsampling by Integers Mi x M2 57 2.3.2 Ideal Decimation 58 2.3.3 Upsampling by Integers L\ x Lj 61 2.3.4 Ideal Interpolation 62 2.4 Sample-Rate Change General Case 64 2.4.1 General Downsampling 64
VI CONTENTS 2.5 Conclusions 66 1.6 Problems 66 References 70 3 TwO-DiMENSIONAL SYSTEMS AND Z-TRANSFORMS 71 3.1 Linear Spatial or 2-D Systems 72 3.2 Z-Transforms 16 3.3 Regions of Convergence 79 3.3.1 More General Case 82 3.4 Some Z-Transform Properties 83 3.4.1 Linear Mapping of Variables 84 3.4.2 Inverse Z-Transform 85 3.5 2-D Filter Stability 89 3.5.1 First-Quadrant Support 91 3.5.2 Second-Quadrant Support 91 3.5.3 Root Maps 96 3.5.4 Stability Criteria for NSHP Support Filters 98 3.6 Conclusions 100 3.7 Problems 101 References 103 4 TwO-DiMENSIONAL DiSCRETE TRANSFORMS 105 4.1 Discrete Fourier Series 106 4.1.1 Properties of the DFS Transform 109 4.1.2 Periodic Convolution 111 4.1.3 Shifting or Delay Property 112 4.2 Discrete Fourier Transform 113 4.2.1 DFT Properties 115 4.2.2 Relation of DFT to Fourier Transform 120 4.2.3 Effect of Sampling in Frequency 121 4.2.4 Interpolating the DFT 122 4.3 2-D Discrete Cosine Transform 123 4.3.1 Review of 1-D DCT 125 4.3.2 Some 1-D DCT Properties 128 4.3.3 Symmetrie Extension in 2-D DCT 131 4.4 Subband/Wavelet Transform (SWT) 132 4.4.1 Ideal Filter Case 132 4.4.2 1-D SWT with Finite-Order Filter 135 4.4.3 2-D SWT with FIR Filters 137 4.4.4 Relation of SWT to DCT 138 4.4.5 Relation of SWT to Wavelets 138
CONTENTS vi/ 4.5 Fast Transform Algorithms 140 4.5.1 Fast DFT Algorithm 140 4.5.2 Fast DCT Methods 141 4.6 Sectioned Convolution Methods 142 4.7 Conclusions 143 4.8 Problems 144 References 147 5 TWO-DIMENSIONAL FiLTER DESIGN 149 5.1 FIR Filter Design 150 5.1.1 FIR Window Function Design 150 5.1.2 Design by Transformation of 1-D Filter 156 5.1.3 Projection-Onto-Convex-Sets Method 161 5.2 HR Filter Design 165 5.2.1 2-D Recursive Filter Design 165 5.2.2 Fully Recursive Filter Design 171 5.3 Subband/Wavelet Filter Design 174 5.3.1 Wavelet (Biorthogonal) Filter Design Method 178 5.4 Conclusions 182 5.5 Problems 182 References 187 6 INTRODUCTORY IMAGE PROCESSING 189 6.1 Light and Luminance 190 6.2 Still Image Visual Properties 194 6.2.1 Weber's Law 195 6.2.2 Contrast Sensitivity Function 196 6.2.3 Local Contrast Adaptation 198 6.3 Time-Variant Human Visual System Properties 199 6.4 Image Sensors 201 6.4.1 Electronic 201 6.4.2 Film 203 6.5 Image and Video Display 204 6.5.1 Gamma 205 6.6 Simple Image Processing Filters 206 6.6.1 Box Filter 206 6.6.2 Gaussian Filter 207 6.6.3 Prewitt Operator 208 6.6.4 Sobel Operator 208 6.6.5 Laplacian Filter 209
vüi CONTENTS 6.7 Conclusions 211 6.8 Problems 211 References 213 7 IMAGE ESTIMATION AND RESTORATION 215 7.1 2-D Random Fields 216 7.1.1 Filtering a 2-D Random Field 218 7.1.2 Autoregressive Random Signal Models 222 7.2 Estimation for Random Fields 224 7.2.1 Infinite Observation Domain 225 7.3 2-D Recursive Estimation 229 7.3.1 1-D Kaiman Filter 229 7.3.2 2-D Kaiman Filtering 233 7.3.3 Reduced Update Kaiman Filter 235 7.3.4 Approximate RUKF 236 7.3.5 Steady-State RUKF 236 7.3.6 LSI Estimation and Restoration Examples with RUKF 237 7.4 Inhomogeneous Gaussian Estimation 241 7.4.1 Inhomogeneous Estimation with RUKF 243 7.5 Estimation in the Subband/Wavelet Domain 244 7.6 Bayesian and MAP Estimation 248 7.6.1 Gauss Markov Image Models 249 7.6.2 Simulated Annealing 253 7.7 Image Identification and Restoration 257 7.7.1 Expectation-Maximization Algorithm Approach 258 7.7.2 EM Method in the Subband/Wavelet Domain 262 7.8 Color Image Processing 263 7.9 Conclusions 263 7.10 Problems 263 References 266 8 DIGITAL IMAGE COMPRESSION 269 8.1 Introduction 270 8.2 Transformation 272 8.2.1 DCT 272 8.2.2 SWT 274 8.2.3 DPCM 275 8.3 Quantization 276 8.3.1 Uniform Quantization 278 8.3.2 Optimal MSE Quantization 278
CONTENTS ix 8.3.3 Vector Quantization 280 8.3.4 LBG Algorithm [7] 282 8.4 Entropy Coding 284 8.4.1 Huffman Coding 285 8.4.2 Arithmetic Coding 286 8.4.3 ECSQandECVQ 287 8.5 DCT Coder 289 8.6 SWT Coder 292 8.6.1 Multiresolution SWT Coding 298 8.6.2 Nondyadic SWT Decompositions 300 8.6.3 Fully Embedded SWT Coders 300 8.6.4 Embedded Zero-Tree Wavelet (EZW) Coder 301 8.6.5 Set Partitioning in Hierarchical Trees (SPIHT) Coder 304 8.6.6 Embedded Zero Block Coder (EZBC) 306 8.7 JPEG 2000 308 8.8 Color Image Coding 309 8.8.1 Scalable Coder Results Comparison 311 8.9 Robustness Considerations 311 8.10 Conclusions 312 8.11 Problems 312 References 315 9 THREE-DIMENSIONAL AND SPATIOTEMPORAL PROCESSING 317 9.1 3-D Signals and Systems 318 9.1.1 Properties of 3-D Fourier Transform 320 9.1.2 3-D Filters 321 9.2 3-D Sampling and Reconstruction 321 9.2.1 General 3-D Sampling 323 9.3 Spatiotemporal Signal Processing 325 9.3.1 Spatiotemporal Sampling 325 9.3.2 Spatiotemporal Filters 326 9.3.3 Intraframe Filtering 328 9.3.4 Intraframe Wiener Filter 328 9.3.5 Interframe Filtering 330 9.3.6 Interframe Wiener Filter 331 9.4 Spatiotemporal Markov Models 332 9.4.1 Causal and Semicausal 3-D Field Sequences 333 9.4.2 Reduced Update Spatiotemporal Kaiman Filter 335 9.5 Conclusions 338 9.6 Problems 338 References 339
x CONTENTS 10 DIGITAL VIDEO PROCESSING 341 10.1 Interframe Processing 342 10.2 Motion Estimation and Motion Compensation 348 10.2.1 Block Matching Method 350 10.2.2 Hierarchical Block Matching 353 10.2.3 Overlapped Block Motion Compensation 354 10.2.4 Pel-Recursive Motion Estimation 355 10.2.5 Optical flow methods 356 10.3 Motion-Compensated Filtering 358 10.3.1 MC-Wiener Filter 358 10.3.2 MC-Kalman Filter 360 10.3.3 Frame-Rate Conversion 363 10.3.4 Deinterlacing 365 10.4 Bayesian Method for Estimating Motion 371 10.4.1 Joint Motion Estimation and Segmentation 373 10.5 Conclusions 377 10.6 Problems 378 References 379 10.7 Appendix: Digital Video Formats 380 SIF 381 CIF 381 ITU 601 Digital TV (aka SMPTE Dl and D5) 381 ATSC Formats 382 11 DIGITAL VIDEO COMPRESSION 385 11.1 Intraframe Coding 387 11.1.1 M-JPEG Pseudo Algorithm 388 11.1.2 DV Codec 391 11.1.3 Intraframe SWT Coding 392 11.1.4 M-JPEG 2000 394 11.2 Interframe Coding 395 11.2.1 Generalizing 1-D DPCM to Interframe Coding 396 11.2.2 MC Spatiotemporal Prediction 397 11.3 Interframe Coding Standards 398 11.3.1 MPEG 1 399 11.3.2 MPEG 2 "a Generic Standard" 401 11.3.3 The Missing MPEG 3 High-Definition Television 403 11.3.4 MPEG 4 Natural and Synthetic Combined 403 11.3.5 Video Processing of MPEG-Coded Bitstreams 404 11.3.6 H.263 Coder for Visual Conferencing 405 11.3.7 H.264/AVC 405 11.3.8 Video Coder Mode Control 408 11.3.9 Network Adaptation 410
CONTENTS XI 11.4 Interframe SWT Coders 410 11.4.1 Motion-Compensated SWT Hybrid Coding 412 11.4.2 3-D or Spatiotemporal Transform Coding 413 11.5 Scalable Video Coders 417 11.5.1 MoreonMCTF 420 11.5.2 Detection of Covered Pixels 421 11.5.3 Bidirectional MCTF 423 11.6 Obj ect-based Video Coding 426 11.7 Comments on the Sensitivity of Compressed Video 428 11.8 Conclusions 429 11.9 Problems 430 References 431 12 VIDEO TRANSMISSION OVER NETWORKS 435 12.1 Video on IP Networks 436 12.1.1 Overview of IP Networks 437 12.1.2 Error-Resilient Coding 440 12.1.3 Transport-Level Error Control 442 12.1.4 Wireless Networks 443 12.1.5 Joint Source-Channel Coding 444 12.1.6 Error Concealment 446 12.2 Robust SWT Video Coding (Bajic) 447 12.2.1 Dispersive Packetization 447 12.2.2 Multiple Description FEC 453 12.3 Error-Resilience Features of H.264/AVC 458 12.3.1 Syntax 458 12.3.2 Data Partitioning 459 12.3.3 Slice Interleaving and Flexible Macroblock Ordering 459 12.3.4 Switching Frames 459 12.3.5 Reference Frame Selection 461 12.3.6 Intrablock Refreshing 461 12.3.7 Error Concealment in H.264/AVC 461 12.4 Joint Source-Network Coding 463 12.4.1 Digital Item Adaptation (DIA) in MPEG 21 463 12.4.2 Fine-Grain Adaptive FEC 464 12.5 Conclusions 469 12.6 Problems 469 References 471 Index 477