Fundamentals of Multimedia

Size: px

Start display at page:

Download "Fundamentals of Multimedia"

Ralph Pope
6 years ago
Views:

1 Ze-Nian Li Mark S. Drew Jiangchuan Liu Fundamentals of Multimedia Second Edition Springer

Contents Part I Introduction and Multimedia Data Representations 1 Introduction to Multimedia 3 1.1 What is Multimedia? 3 1.1.1 Components of Multimedia 4 1.2 Multimedia: Past and Present 5 1.2.1 Early History of Multimedia 5 1.

2 Contents Part I Introduction and Multimedia Data Representations 1 Introduction to Multimedia What is Multimedia? Components of Multimedia Multimedia: Past and Present Early History of Multimedia Hypermedia, WWW, and Internet Multimedia in the New Millennium Multimedia Software Tools: A Quick Scan Music Sequencing and Notation Digital Audio Graphics and Image Editing Video Editing Animation Multimedia Authoring Multimedia in the Future Exercises 22 References 23 2 A Taste of Multimedia Multimedia Tasks and Concerns Multimedia Presentation Data Compression Multimedia Production Multimedia Sharing and Distribution Some Useful Editing and Authoring Tools Adobe Premiere Adobe Director Adobe Flash Exercises 52 References 56 xiii

xiv Contents 3 Graphics and Image Data Representations 57 3.1 Graphics/Image Data Types 57 3.1.1 1-Bit Images 57 3.1.2 8-Bit Gray-Level Images 58 3.1.3 Image Data Types 62 3.1.4 24-Bit Color Images 62 3.

3 xiv Contents 3 Graphics and Image Data Representations Graphics/Image Data Types Bit Images Bit Gray-Level Images Image Data Types Bit Color Images Higher Bit-Depth Images Bit Color Images Color Lookup Tables Popular File Formats GIF JPEG PNG TIFF Windows BMP Windows WMF Netpbm Format EXIF PS and PDF PTM Exercises 78 References 80 4 Color in Image and Video Color Science Light and Spectra Human Vision Spectral Sensitivity of the Eye Image Formation Camera Systems Gamma Correction Color-Matching Functions CIE Chromaticity Diagram Color Monitor Specifications Out-of-Gamut Colors White Point Correction XYZ to RGB Transform Transform with Gamma Correction L*a*b* (CIELAB) Color Model More Color Coordinate Schemes Munsell Color Naming System 99

Contents xv 4.2 Color Models in Images 99 4.2.1 RGB Color Model for Displays 100 4.2.2 Multisensor Cameras 100 4.2.3 Camera-Dependent Color 100 4.2.4 Subtractive Color: CMY Color Model 102 4.2.5 Transformation from RGB to CMY 102 4.

4 Contents xv 4.2 Color Models in Images RGB Color Model for Displays Multisensor Cameras Camera-Dependent Color Subtractive Color: CMY Color Model Transformation from RGB to CMY Undercolor Removal: CMYK System Printer Gamuts Multi-ink Printers Color Models in Video Video Color Transforms YUV Color Model YIQ Color Model YCbCr Color Model Exercises 110 References Fundamental Concepts in Video Analog Video, NTSC Video PAL Video SECAM Video Digital Video Chroma Subsampling CCIR and ITU-R Standards for Digital Video High-Definition TV Ultra High Definition TV (UHDTV) Video Display Interfaces Analog Display Interfaces Digital Display Interfaces D Video and TV Cues for 3D Percept D Camera Models D Movie and TV Based on Stereo Vision The Vergence-Accommodation Conflict Autostereoscopic (Glasses-Free) Display Devices Disparity Manipulation in 3D Content Creation Exercises 137 References 138

xvi Contents 6 Basics of Digital Audio 139 6.1 Digitization of Sound 139 6.1.1 What is Sound? 139 6.1.2 Digitization 140 6.1.3 Nyquist Theorem 142 6.1.4 Signal-to-Noise Ratio (SNR) 144 6.1.5 Signal-to-Quantization-Noise Ratio (SQNR) 145 6.

5 xvi Contents 6 Basics of Digital Audio Digitization of Sound What is Sound? Digitization Nyquist Theorem Signal-to-Noise Ratio (SNR) Signal-to-Quantization-Noise Ratio (SQNR) Linear and Nonlinear Quantization Audio Filtering Audio Quality Versus Data Rate Synthetic Sounds MIDI: Musical Instrument Digital Interface MIDI Overview Hardware Aspects of MIDI Structure of MIDI Messages General MIDI MIDI-to-WAV Conversion Quantization and Transmission of Audio Coding of Audio Pulse Code Modulation Differential Coding of Audio Lossless Predictive Coding DPCM DM ADPCM Exercises 177 References 180 Part II Multimedia Data Compression 7 Lossless Compression Algorithms Introduction Basics of Information Theory Run-Length Coding Variable-Length Coding Shannon-Fano Algorithm Huffman Coding Adaptive Huffman Coding Dictionary-Based Coding Arithmetic Coding Basic Arithmetic Coding Algorithm Scaling and Incremental Coding 210

xvii 7.6.3 Integer Implementation 214 7.6.4 Binary Arithmetic Coding 214 7.6.5 Adaptive Arithmetic Coding 215 7.7 Lossless Image Compression 218 7.7.1 Differential Coding of Images 218 7.7.2 Lossless JPEG 219 7.

6 xvii Integer Implementation Binary Arithmetic Coding Adaptive Arithmetic Coding Lossless Image Compression Differential Coding of Images Lossless JPEG Exercises 221 References 223 Lossy Compression Algorithms Introduction Distortion Measures The Rate-Distortion Theory Quantization Uniform Scalar Quantization Nonuniform Scalar Quantization Vector Quantization Transform Coding Discrete Cosine Transform (DCT) Karhunen-Loeve Transform* Wavelet-Based Coding Introduction Continuous Wavelet Transform* Discrete Wavelet Transform* Wavelet Packets Embedded Zerotree of Wavelet Coefficients The Zerotree Data Structure Successive Approximation Quantization EZW Example Set Partitioning in Hierarchical Trees (SPIHT) Exercises 277 References 280 Image Compression Standards The JPEG Standard Main Steps in JPEG Image Compression JPEG Modes A Glance at the JPEG Bitstream The JPEG2000 Standard Main Steps of JPEG2000 Image Compression* Adapting EBCOT to JPEG Region-of-Interest Coding Comparison of JPEG and JPEG2000 Performance 304

xviii Contents 9.3 The JPEG-LS Standard 305 9.3.1 Prediction 308 9.3.2 Context Determination 308 9.3.3 Residual Coding 309 9.3.4 Near-Lossless Mode 309 9.4 Bi-level Image Compression Standards 309 9.

7 xviii Contents 9.3 The JPEG-LS Standard Prediction Context Determination Residual Coding Near-Lossless Mode Bi-level Image Compression Standards The JBIG Standard The JBIG2 Standard Exercises 313 References Basic Video Compression Techniques Introduction to Video Compression Video Compression Based on Motion Compensation Search for Motion Vectors Sequential Search D Logarithmic Search Hierarchical Search H Intra-Frame (I-Frame) Coding Inter-Frame (P-Frame) Predictive Coding Quantization in H H.261 Encoder and Decoder A Glance at the H.261 Video Bitstream Syntax H Motion Compensation in H Optional H.263 Coding Modes H.263+ and H Exercises 337 References MPEG Video Coding: MPEG-1, 2, 4, and Overview MPEG Motion Compensation in MPEG Other Major Differences from H MPEG-1 Video Bitstream MPEG Supporting Interlaced Video MPEG-2 Scalabilities Other Major Differences from MPEG-1 358

xix 11.4 MPEG-4 359 11.4.1 Overview of MPEG-4 359 11.4.2 Video Object-Based Coding in MPEG-4 362 11.4.3 Synthetic Object Coding in MPEG-4 375 11.4.4 MPEG-4 Parts, Profiles and Levels 383 11.

8 xix 11.4 MPEG Overview of MPEG Video Object-Based Coding in MPEG Synthetic Object Coding in MPEG MPEG-4 Parts, Profiles and Levels MPEG Descriptor (D) Description Scheme (DS) Description Definition Language (DDL) Exercises 391 References 392 New Video Coding Standards: H.264 and H H Motion Compensation Integer Transform Quantization and Scaling Examples of H.264 Integer Transform and Quantization Intra Coding In-Loop Deblocking Filtering Entropy Coding Context-Adaptive Variable Length Coding (CAVLC) Context-Adaptive Binary Arithmetic Coding (CABAC) H.264 Profiles H.264 Scalable Video Coding H.264 Multiview Video Coding H Motion Compensation Integer Transform Quantization and Scaling Intra Coding Discrete Sine Transform In-Loop Filtering Entropy Coding Special Coding Modes H.265 Profiles Comparisons of Video Coding Efficiency Objective Assessment Subjective Assessment Exercises 431 References 433

xx Contents 13 Basic Audio Compression Techniques 435 13.1 ADPCM in Speech Coding 436 13.1.1 ADPCM 436 13.2 G.726 ADPCM, G.727-9 437 13.3 Vocoders 439 13.3.1 Phase Insensitivity 439 13.3.2 Channel Vocoder 439 13.

9 xx Contents 13 Basic Audio Compression Techniques ADPCM in Speech Coding ADPCM G.726 ADPCM, G Vocoders Phase Insensitivity Channel Vocoder Formant Vocoder Linear Predictive Coding (LPC) Code Excited Linear Prediction (CELP) Hybrid Excitation Vocoders* Exercises 453 References MPEG Audio Compression Psychoacoustics Equal-Loudness Relations Frequency Masking Temporal Masking MPEG Audio MPEG Layers MPEG Audio Strategy MPEG Audio Compression Algorithm MPEG-2 AAC (Advanced Audio Coding) MPEG-4 Audio Other Audio Codecs Ogg Vorbis MPEG-7 Audio and Beyond Further Exploration Exercises 480 References 481 Part III Multimedia Communications and Networking 15 Network Services and Protocols for Multimedia Communications Protocol Layers of Computer Communication Networks Local Area Network and Access Networks LAN Standards Ethernet Technology Access Network Technologies 489

Contents xxi 15.3 Internet Technologies and Protocols 494 15.3.1 Network Layer: IP 495 15.3.2 Transport Layer: TCP and UDP 496 15.3.3 Network Address Translation and Firewall 501 15.

10 Contents xxi 15.3 Internet Technologies and Protocols Network Layer: IP Transport Layer: TCP and UDP Network Address Translation and Firewall Multicast Extension Router-Based Architectures: IP Multicast Non Router-Based Multicast Architectures Quality-of-Service for Multimedia Communications Quality of Service Internet QoS Rate Control and Buffer Management Protocols for Multimedia Transmission and Interaction HyperText Transfer Protocol Real-Time Transport Protocol RTP Control Protocol Real-Time Streaming Protocol Case Study: Internet Telephony Signaling Protocols: H.323 and Session Initiation Protocol Further Exploration Exercises 526 References Internet Multimedia Content Distribution Proxy Caching Sliding-Interval Caching Prefix Caching and Segment Caching Rate-Split Caching and Work-Ahead Smoothing Summary and Comparison Content Distribution Networks (CDNs) Representative: Akamai Streaming CDN Broadcast/Multicast Video-on-Demand Smart TV and Set-Top Box (STB) Scalable Multicast/Broadcast VoD Broadcast/Multicast for Heterogeneous Users Stream Replication Layered Multicast Application-Layer Multicast Representative: End-System Multicast (ESM) Multi-tree Structure Peer-to-Peer Video Streaming with Mesh Overlays Representative: CoolStreaming Hybrid Tree and Mesh Overlay 562

xxii Contents 16.7 HTTP-Based Media Streaming 563 16.7.1 HTTP for Streaming 564 16.7.2 Dynamic Adaptive Streaming Over HTTP (DASH) 565 16.

11 xxii Contents 16.7 HTTP-Based Media Streaming HTTP for Streaming Dynamic Adaptive Streaming Over HTTP (DASH) Exercises 567 References Multimedia Over Wireless and Mobile Networks Characteristics of Wireless Channels Path Loss Multipath Fading Wireless Networking Technologies IG Cellular Analog Wireless Networks G Cellular Networks: GSM and Narrowband CDMA G Cellular Networks: Wideband CDMA G Cellular Networks and Beyond Wireless Local Area Networks Bluetooth and Short-Range Technologies Multimedia Over Wireless Channels Error Detection Error Correction Error-Resilient Coding Error Concealment Mobility Management Network Layer Mobile IP Link-Layer Handoff Management Further Exploration Exercises 610 References 612 Part IV Multimedia Information Sharing and Retrieval 18 Social Media Sharing Representative Social Media Services User-Generated Content Sharing Online Social Networking User-Generated Media Content Sharing YouTube Video Format and Meta-data Characteristics of YouTube Video Small-World in YouTube Videos YouTube from a Partner's View Enhancing UGC Video Sharing 628

Contents xxiii 18.3 Media Propagation in Online Social Networks 632 18.3.1 Sharing Patterns of Individual Users 633 18.3.2 Video Propagation Structure and Model 634 18.3.3 Video Watching and Sharing Behaviors 637 18.

12 Contents xxiii 18.3 Media Propagation in Online Social Networks Sharing Patterns of Individual Users Video Propagation Structure and Model Video Watching and Sharing Behaviors Coordinating Live Streaming and Online Storage Further Exploration Exercises 640 References Cloud Computing for Multimedia Services Cloud Computing Overview Representative Storage Service: Amazon S Representative Computation Service: Amazon EC Multimedia Cloud Computing Cloud-Assisted Media Sharing Impact of Globalization Case Study: Netflix Computation Offloading for Multimedia Services Requirements for Computation Offloading Service Partitioning for Video Coding Case Study: Cloud-Assisted Motion Estimation Interactive Cloud Gaming Issues and Challenges of Cloud Gaming Real-World Implementation Further Exploration Exercises 671 References Content-Based Retrieval in Digital Libraries How Should We Retrieve Images? Synopsis of Early CBIR Systems C-BIRD: A Case Study Color Histogram Color Density and Color Layout Texture Layout Texture Analysis Details Search by Illumination Invariance Search by Object Model 686

xxiv Contents 20.4 Quantifying Search Results 688 20.5 Key Technologies in Current CBIR Systems 692 20.5.1 Robust Image Features and Their Representation 692 20.5.2 Relevance Feedback 694 20.5.3 Other Post-processing Techniques 695 20.

13 xxiv Contents 20.4 Quantifying Search Results Key Technologies in Current CBIR Systems Robust Image Features and Their Representation Relevance Feedback Other Post-processing Techniques Visual Concept Search The Role of Users in Interactive CBIR Systems Querying on Videos Querying on Videos Based on Human Activity Modeling Human Activity Structures Experimental Results Quality-A ware Mobile Visual Search Related Work Quality-Aware Method Experimental Results Exercises 710 References 711 Index 715

Preface. I Introduction and Multimedia Data Representations 1

Preface. I Introduction and Multimedia Data Representations 1 Contents Preface x I Introduction and Multimedia Data Representations 1 1 Introduction to Multimedia 2 1.1 What is Multimedia?.... 2 1.1.1 Components of Multimedia.... 2 1.2 Multimedia: Past and Present....