The Essential Guide to Video Processing

Transcription

1 The Essential Guide to Video Processing Second Edition EDITOR Al Bovik Department of Electrical and Computer Engineering The University of Texas at Austin Austin, Texas AMSTERDAM BOSTON HEIDELBERG LONDON NEW YORK OXFORD PARIS SAN DIEGO SAN FRANCISCO SINGAPORE SYDNEY TOKYO HI ллг,vi ГЛгч Academic Press is an imprint of Elsevier

2 Preface About the Author xvii xix CHAPTER 1 Introduction to Digital Video Processing Sampled Video Video Transmission Objectives of this Guide Organization of the Guide 8 CHAPTER 2 Video Sampling and Interpolation и 2.1 Introduction Spatiotemporal Sampling Structures Sampling and Reconstruction of Continuous Time-Varying Imagery Sampling Structure Conversion Frame-rate Conversion Spatiotemporal Sampling Structure Conversion Conclusion 28 References 28 Further Information 29 CHAPTER 3 Motion Detection and Estimation Introduction Notation and Preliminaries Binary Hypothesis Testing Markov Random Fields MAP Estimation Variational Formulations Motion Detection Hypothesis Testing with Fixed Threshold Hypothesis Testing with Adaptive Threshold MAP MRF Formulation MAP Variational Formulation Experimental Comparison of Motion Detection Methods Motion Estimation Motion Models Estimation Criteria Search Strategies 54 V

3 vi Contents 3.5 Practical Motion Estimation Algorithms Global Motion Estimation Block Matching Phase Correlation Optical Flow via Regularization MAP Estimation of Dense Motion Experimental Comparison of Motion Estimation Methods Perspectives Acknowledgments 66 References 66 CHAPTER 4 Video Enhancement and Restoration Introduction, Spatiotemporal Noise Filtering Linear Filters Order-Statistic Filters Multiresolution Filters Coding Artifact Reduction Artifact Reduction in the Spatial Domain Artifact Reduction in the Frequency Domain Blotch Detection and Removal Blotch Detection Motion Vector Repair and Interpolating Corrupted Intensities Video Inpainting Restoration in Conditions of Difficult Object Motion Vinegar Syndrome Removal Intensity Flicker Correction Flicker Parameter Estimation Estimation on Sequences with Motion Kinescope Moire Removal Scratch Removal Conclusions 104 Acknowledgements 105 References 105 CHAPTER 5 Video Stabilization and Mosaicing Introduction Video Stabilization Outline 110

4 vii 5.2 Biological Motivation: Insect Navigation Ill Centering Behavior and Collision Avoidance Ill Control of Flight Speed and Stabilization Measuring Distance by Integrating Optical Flow Camera Model and Image Motion Model Camera Model Effect of Camera Motion Image Features Structure from Motion Feature based Algorithms Flow-Based Approaches Global Flow Models Flow-Based Algorithm Stabilization and Mosaicing Video Mosaicing Stabilization and Mosaicing with Additional Information VIVID Metadata Stabilization with Metadata Inertial Measurements Stabilization with Inertial Measurements Motion Segmentation Motion Super-resolution Three-dimensional Stabilization Summary 137 Acknowledgements 138 References 138 CHAPTER 6 Video Segmentation Introduction Scene Change Detection Spatiotemporal Change Detection Spatial Change Detection Using Two Frames Temporal Integration Combination with Spatial Segmentation Motion Segmentation Dominant Motion Segmentation Multiple Motion Segmentation Simultaneous Motion Estimation and Segmentation Motion-Field Model and MAP Framework Two-Step Iteration Algorithm Semantic Video Object Segmentation Chroma-Keying Semiautomatic Segmentation 164

5 viii Contents 6.7 Examples Performance Evaluation of Video Segmentation 170 Acknowledgements 170 References 170 CHAPTER 7 Motion Tracking in Video Introduction Rigid Object Tracking D Rigid Object Tracking D Rigid Object Tracking Articulated Object Tracking D Articulated Object Tracking D Articulated Object Tracking 219 References 220 CHAPTER 8 Basic Transform Video Coding Introduction to Video Compression Video Compression Application Requirements Digital Video Signals and Formats Sampling of Analog Video Signals Digital Video Formats Video Compression Techniques Entropy and Predictive Coding Block Transform Coding The DCT Quantization MC and Estimation Transform Coding: Introduction to the Video Encoding Standards Transform Coding Standard Example: The H.261 Video Encoder Closing Remarks 265 References 265 CHAPTER 9 MPEG-1 and MPEG-2 Video Standards MPEG-1 Video Coding Standard Introduction MPEG-1 Video Coding versus H MPEG-1 Video Structure Summary of the Major Differences between MPEG-1 Video and H Simulation Model MPEG-1 Video Bit-Stream Structures Summary 277

6 ix 9.2 MPEG-2 Video Coding Standard Introduction MPEG-2 Profiles and Levels MPEG-2 Video Input Resolutions and Formats MPEG-2 Video Coding Standard Compared to MPEG Scalable Coding Data Partitioning Other Tools for Error Resilience Test Model MPEG-2 Video and System Bit-Stream Structures Summary 291 References 292 CHAPTER 1 0 MPEG-4 Visual and H.264/AVC: Standards for Modern Digital Video Introduction Terminology MPEG-4 Part Object-based Representation Video Object Coding Mesh Object Coding Model-based Coding Still Texture Coding Scalability Error Resilience MPEG-4 Part 2 Profiles MPEG-4 Part 10: H.264/AVC H.264/AVC Video Coding Layer: Technical Overview Profiles MPEG-4 Compression Performance MPEG-4 Part MPEG-4 Part 10: H.264/AVC MPEG-4 Video Applications Conclusions and Outlook 327 Acknowledgements 328 References 328 CHAPTER 11 Interframe Subband/Wavelet Scalable Video Coding Introduction Motion Estimation and Compensation for MCTF Connected and Unconnected Blocks 335

7 Using Chroma for Motion Estimation Improving the Haar MCTF New Haar MCTF Overlapped Block Motion Compensation Scalable Motion Vector Coding EZBC Coder Coding Process Context Modeling Scalability Packetization Frequency Roll-Off Extension to LeGall and Tabatabai 5/3 Filtering Objective and Visual Comparisons Some Visual Results Multiple Adaptations Related Coders Conclusions 363 References 363 CHAPTER 12 Digital Video Transcoding Introduction Video Transcoding for Bit Rate Reduction Transcoding of Intracoded Frame Transcoding of Intercoded Frame Fast Video-Transcoding Architectures DCTDomainlMC Heterogeneous Video Transcoding MV Estimation for Spatial Resolution Reduction MV Estimation for Temporal Resolution Reduction Spatial Resolution Reduction Macro-block-Coding Type Decision Bit Rate Control in Video Transcoding Error-Resilient Video Transcoding Concluding Remarks 385 References 386 CHAPTER 13 Embedded Video Codecs Introduction Block-Based Video Coding Embedded Video Codec Requirements and Constraints Embedded Video Codec Design Flow Understanding the Chip Architecture Understanding the Codec Algorithms Modularity and APIs Definitions 401

8 xi Reference Codec Software Development in Golden С Platform-Specific Development and Porting Kernel Optimization and Integration Concurrent Processing Overall Optimization Stress and Conformance Testing New Trends Summary 413 References 414 CHAPTER 14 Video Quality Assessment Introduction HVS Modeling Based Methods Feature Based Methods VQM Motion Modeling Based Methods Performance Conclusions 433 References 434 CHAPTER 15 A Unified Framework for Video Indexing, Summarization, Browsing, and Retrieval Introduction Content Categories Storage and Compression Terminology Image and Video Features Statistical Features Compressed-Domain Features Content-Based Features Video Analysis Shot Boundary Detection Key-Frame Extraction Play/Break Segmentation Audio Marker Detection Video Marker Detection Video Representation Video Representation for Scripted Content Video Representation for Unscripted Content Video Browsing Video Browsing Using ToC-Based Summary Video Browsing Using Highlights-Based Summary

9 xii Contents 15.6 Video Retrieval Feature-Based Retrieval (Statistical and Compressed) Content-Based Retrieval Relevance Feedback Query-Concept Learner Efficient Annotation through Active Learning Considerations in Multimedia Databases A Unified Framework for Indexing, Summarization, Browsing, and Retrieval Conclusions and Promising Research Directions 469 Acknowledgements 470 References 470 CHAPTER 16 Video Communication Networks Introduction Video Compression Standards Introduction Overview MPEG-2 Video Compression Standard MPEG-2 Systems Standard Video Communication Networks Introduction Hybrid Fiber-Coax Networks Digital Subscriber Loop Wireless Networks Fiber Optics Integrated Services Digital Network ATM Networks Internet Protocol Networks Introduction Multicast Backbone Real-Time Transport Protocol Real-Time Transport Control Protocol Real-Time Transport Streaming Protocol H Session Initiation Protocol Integrated Services Resource Reservation Protocol Differentiated Services DiffServ Summary 525 References 525

10 xiii CHAPTER 17 Video Security and Protection Introduction Video Encryption Candidate Domains for Encrypting Multimedia Building Blocks for Media Encryption Security Evaluation of Media Encryption Video Encryption System Design Video Authentication Background Content Level Authentication Stream Level Authentication Video Fingerprinting for Traitor Tracing The Background Coded Fingerprinting Experimental Results of Video Fingerprinting Intravideo Collusion 566 References 566 CHAPTER 18 Wireless Video Streaming Introduction On Joint Source-Channel Coding Rate-Distortion Theory Operational Rate-Distortion Theory Practical Constraints in Video Communications Illustration Video Compression and Transmission Video Transmission System Video Compression Basics Channel Models End-to-End Distortion Error Resilient Source Coding Channel Coding Forward Error Correction Retransmission Joint Source-Channel Coding Problem Formulation Internet Video Transmission Wireless Video Transmission Distributed Multimedia Communications Video Streaming over Multiuser Networks Mobile TV Standards Peer-to-Peer Internet Video Broadcasting Video Streaming over Multihop Wireless Networks

11 xiv Contents 18.7 Discussion 612 References 613 CHAPTER 19 Video Surveillance Introduction Categorizing Applications, Target Scenes, and Video Analytics Video Surveillance Applications Video Surveillance Target Scenes Video Analytics for Video Surveillance Review of Video Analytic Algorithms Motion and Change Detection Object Detection Object Tracking Behavioral Analysis Tools Gait Recognition Face Recognition Conclusion 648 References 649 CHAPTER 20 Face Recognition From Video Introduction Properties and Literature Review Set of Observations Temporal Continuity/Dynamics D Model A General Framework of Probabilistic Identity Characterization Recognition Setting and Issues Instances of Probabilistic Identity Characterization ER From a Group of Still Images ER From a Video Sequence A System Identification Approach TheARMAModel Framework for Recognition Experiments, Results, and Discussion Conclusions 685 References 685 CHAPTER 21 Audiovisual Speech Processing Introduction Analysis of Visual Signals Face Detection, Mouth, and Lip Tracking Visual Features Two Visual Feature Extraction Systems 698

12 xv 21.3 Audiovisual Information Fusion Speech Classes in Audiovisual Integration Classifiers in Speech Applications Feature and Classifier Fusion Audiovisual Automatic Speech Recognition Bimodal Corpora for ASR Experimental Results Audiovisual Speech Synthesis Coarticulation Modeling Facial Animation Visual Text-to-Speech Speech-to-Video Synthesis Visual Speech Synthesis Evaluation Audiovisual Speaker Recognition Summary and Discussion 729 References 731 Index 739