Concepts of Image & Video Coding

Transcription

1 Concepts of Image & Video Coding Applications of Digital Video Multimedia Digital TV, HDTV VCD (Video CD), DVD (Digital Versatile/Video Disc), Digital camcoder, Digital VCR,... Video on demand Videophone and video conferencing Distance learning Remote collaboration Surveillance... 1

2 Video Coding Related Products Software Video codec Streaming video server/player Video authoring/editing image/video database Video authentication Digital watermarking Video games Remote collaboration tool Surveillance system Distance learning Computer vision Digital library Multimedia application software... Hardware MPEG-2 chip MPEG-4 chip H.264 chip Media processor DVD player Set-top box Game station Internet appliance Mobile terminal Videophone Video conference terminal Digital camera, VCR, TV HDTV... Digital Image/Video Good quality for storage and transmission Interactivity Variable-rate transmission on demand Easy software conversion from one standard to another Integration of various video applications Editing capabilities, such as cutting and pasting, zooming, removal of noise and blur 2

3 Digital Image/Video Robustness to channel noise and ease of encryption Trend of DSP VLSI Problem with a digital video system Huge amount of data Solution: Image/Video Compression Image (frame) pel Frames to be digitized are first partitioned into a large number of picture element or pels for short, which are typically arranged in row and columns. A row of pels is called a scan line. 3

4 Images and Video time... Pixel or Pel Line Frame or Picture Sequence Needs for Video Compression Without compression JPEG Test Pictures: 720(pels)x576(lines)x1.5bytes=4.977Mbits = 77.8s at 64kbits/s Visual telephony (e.g. 15 frames/s): 352 (pels) x 288 (lines) x 15 (frames/s) x 1.5 bytes = Mbits/s Digital TV (CCIR frames/s): 720 (pels) x 480 (lines) x 30 (frames/s) x 1.5 bytes = Mbits/s HDTV (e.g. 1280x720 pels 60 frames/s): 1280 (pels) x 720 (lines) x 60 (frames/s) x 1.5 bytes = Mbits/s 4

5 Needs for Video Compression Without compression A television-quality digital video signal 216Mbits /sec 2-hour movie (uncompressed) Over 194Gbytes 42 DVDs or 304 CD-ROMs Compression results in lower bit-rates which enables new applications and lower transmission and storage cost Some Standards for Image/Video Compression JPEG : Joint Photographic Experts Group ITU-T H.261 :Video codec for audiovisual services at px64kbps ITU-T H.263 :Experts group on very low bit-rate video telephony MPEG-1 <Moving Picture Experts Group> :Digital storage media, up to 1.5 Mbps MPEG-2 :Generic coding of moving pictures and associated audio MPEG-4 :multiple bit-rate audiovisual coding, visual object coding 5

6 Data Compression the process of reducing the amount of data required to represent a given quantity of information More efficient representation of digital images is necessary A compression system consists of two distinct structure of blocks: - encoder(compressor) and decoder (decompressor). Compression/Coding Reducing the size of image data files Retain necessary information Lossless: -- No data are lost -- The original image can be recreated exactly Lossy: -- Allow a loss in the actual image data -- The original uncompressed image cannot be exactly created from the compressed file 6

7 Lossless No data are lost The original image can be recreated exactly Image Compression ratio 2:1 to 3:1 Lossy Allow a loss in the actual image data The original uncompressed image cannot be created exactly from the compressed file Image Compression ratio 10:1 to 20:1 7

8 Compression Ratio (Cr) Cr= = = Source coder input size Source coder output size Uncompressed File Size Compressed File Size SIZEu SIZEc Compression Ratio. (cont) Example: Image 256X256 pixels, 256 level grayscale, compressed file size 6554 byte. Original Image Size = 256X256(pixels)X1(byte/pixel) = bytes Compression Ratio =

9 Bits Per Pixel (bpp) BPP= Number of Bits Number of Pixels = (8)(Number of Bytes) N x N Bits Per Pixel. (cont) Example: Image 256X256 pixels, 256 level grayscale compressed file size 6554 byte. Original Image Size = 256X256(pixels) = pixel Compressed file = 6554(bytes) X8(byte/pixels) = bits Bits Per Pixel =

10 Fidelity Measure Fidelity Measure is used to measure information loss Objective fidelity measure - Measured mathematically about the amount of error in the reconstructed image Subjective fidelity measure - Measured by human observation Objective Fidelity Measure I( r, c) = ^ I( r, c) = The original image The decompressed image Error(r,c)= I( r, c) PSNR = 10 Objective measures: ^ log 10 1 N I( r, c) N 1 N 1 2 r = 0 C = 0 ( L 1) ^ I ( r, c ) I ( r, c ) Easy to generate Not necessarily correlated to our perception Seemingly unbiased

11 Tradeoffs in Lossless Compression Coding Efficiency -Compression Ratio? Coder Complexity Memory requirements? Power requirements? Operations per second? Coding Delay Tradeoffs in Lossy Compression Bit error probability? Signal/Noise? Mean opinion score? Coding Efficiency -Compression Ratio? Coder Complexity Memory requirements? Power (mw)? Operations per second? Coding Delay 11

12 Rate-Distortion of Lossy Algorithms Distortion D R Rate Distortion Measure: PSNR (Peak Signal-to-Noise Ratio) Subjective evaluation Compression/Coding To provide efficient data storage and transmission Remove redundancy Interpixel redundancy Spatial redundancy (intra coding) Temporal redundancy (inter coding) Statistical redundancy (Coding redundancy) Perceptual redundancy (Psychovisual redundancy) 12

13 Compression Technology Theorem Spatial redundancy DPCM, DCT, Subband, wavelet Temporal redundancy DPCM, MC/ME Statistical redundancy RLC, VLC Perceptual redundancy SQ, VQ, fractal Compression Technology Statistical redundancy Lossless Depend on the probabilistic characterization of signal Perceptual redundancy Lossy, irreversible Complex, depends on context and application 13

14 A Typical End-To-End Digital Video System Analog video in Video capture, Format conversion, Pre-processing Source coder (lossy + lossless) Channel coder Channel Analog video out Post-processing, Format conversion Source decoder (lossy + lossless) Channel decoder Source Coding Reduction of data redundancy Temporal redundancy Motion Estimation/Compensation (ME/MC) Spatial redundancy Discrete Cosine Transform (DCT) Statistical redundancy Variable Length Coding (VLC) Perceptual redundancy Quantization (Q) 14

15 Standard Video Encoder Video in _ + DCT Q IQ VLC Compressed video out IDCT Motion estimation (ME) Frame memory Motion vector + DCT: Discrete Cosine Transform IDCT: Inverse DCT Q: Quantization IQ: Inverse Quantization VLC: Variable Length Coding Block-Based Coding time 15

16 Sequence Block Based Coding Picture Block GOB Y Y C B C R Y Y Macroblock (MB) Encoder Image Block Intra Coding: Transform Coding T Transform Transform Coefficients Q Quantization Entropy Coding Bitstream Decoder Bitstream Entropy Decoding T 1 Inverse Transform Reconstructed Image Block 16

17 Discrete Cosine Transform (DCT) DC coefficient AC coefficients Horizontal freq. DCT 8 x 8 image block DCT coefficients Vertical freq. Motion compensated difference block 1 DCT Effect of DCT and Quantization 2 Q 3 IQ IDCT 4 Z 3 1 DCT QUANTIZATION (Step size = 30) IQ, IDCT 4 Mean Square Error =

18 DCT Formula DCT IDCT DCT & IDCT 18

19 Average 19

20 Comments By zeroing the last 4 values the maximum error is 7 out of 255 Which is 2.7% By compressing the data 50% we introduce an error of 2.7% Quality acceptable YCbCr Color Mode Y = 0.299*R *G *B Cb = *R *G *B Cr = 0.701*R *G *B 20

21 JPEG Conversion Algorithm Convert RGB to YCbCr Divide image into 8x8 macro block Compute the DCT of each macroblock Quantization Divide the DCT values such that many values in the DCT becomes zero 21

22 Quantization Table YQ50 Quantization Table CbCrQ50 22

23 Image Macroblock DCT 23

24 DCT Coefficients Divided by Quantization Table Discussion A achieve higher compression at the cost of lower image quality a different table can be used Table will contain higher numbers 24

25 Temporal Redundancy: Inter Coding Conditional replenishment Transmit only the changing blocks Previous Frame Current Frame Inter Coding : Motion Compensation Motion Previous Frame Pel-based motion Block-based motion Object-based motion Current Frame 25

26 Block-Based Motion Compensation Block matching to estimate motion Motion vectors and residue are transmitted Previous Picture Current Picture Block-Based Motion Compensation (Cont.) Offset: motion vector Differential coding in x Differential coding in y Residue: prediction error Coded as in intra coding 26

27 Applications Multimedia Content Server Intermediate Servers Notebooks Workstations Network Network Smart Phone Home PC Web TV Multimedia Content PDA Transcoder TRANSCODER Deliver Deliver Networks Networks ADSL H.26L Pre-encoded MPEG-2 Video Digital Cable Mobile MPEG-4 SP MPEG2 Video MPEG-4 SP 27

28 JPEG2000 Markets and Applications JPEG at bpp 28

29 JPEG2000 at bpp ROI: Some visual results 29

30 DEMO (Rate Control) at 24k bits/s Origin(10fps,6s) at 128k bits/s 遠距教學 (Distance Learning) 即時群播教學系統 虛擬教室教學系統 課程隨選教學系統 30

31 Video Phone ITU (International Telecommunication Union) - ) H.324 國際標準 面對面的通訊 內建 33.6K bps Modem 高品質影像和聲音 簡單操作 電源關掉電話亦能使用 Video Conference 影像由 Camera 擷取 影像的顯示裝置可為 TV,PC Monitor, LCD 即時影像和聲音的傳輸 31

32 Desktop Video Conference 視訊軟體 Netmeeting 32

33 工研院開發的四人視訊會議系統 Virtual Video Conference 33

34 34

35 Virtual Video Conference Remote Control 台北市即時路況 路況影像資料每分鐘更新一次, 且每隔三十秒自動重新讀取影像檔 35

36 36

37 台北市 基隆河大佳段 太魯閣 墾丁 37

38 MPEG-4 4 VIDEO OBJECT PLANE (VOP) VOP1 VOP2 VOP3 型號 : J-SH53 大小 :50 X 98 X24 (mm) 重量 : 約 115 公克擁有 100 萬像素 CCD 攝影機最大 1140X858 的畫面攝影內面液晶螢幕為 26 萬色 (2.4QVGA) 外面液晶為 6 萬 5536 色 (1.2GF) 使用 MPEG-4 壓縮格式 :80X60 最大攝錄時間為 10 秒,128x96 為五秒 ( 均含影像與聲音 ) 支援 Java 程式 38

39 In the Future By NTT DoCoMo 39