A glimpse into video processing

Size: px

Start display at page:

Download "A glimpse into video processing"

Merry Parker
6 years ago
Views:

1 A glimpse into video processing Pierre Gronlier - pierre.gronlier@gmail.com - pierre.gronlier@skype.net Software developer in Test - Microsoft Skype division October Paris

2 1 Introduction The Goal What is Video? 2 What do we see? The eye Colors 3 2D - Audio Back to Basics 4 3D - Video Temporal masking Frequencyl masking Frames Codecs 5 Video transmission Overview Research vs Development vs Testing 6 References 7 Software development methods Agile 8 Intership

3 The Goal The Goal Compress understandable data into a form that minimizes the space required to store or transmit it and being able to restore compressed informations to an understandable form.

4 The Goal The Goal Compress understandable data into a form that minimizes the space required to store or transmit it and being able to restore compressed informations to an understandable form. Lossless Compression Binary Data : Deflate, Run-Length-Encoding, Lempel-Ziv-Welch Audio : FLAC, WMA 9 Lossless,... Image : BMP, JPEG-LS, JPEG-2000, PNG, TIFF,... Video : FFV1, Huffyuv, MSU Lossless,...

5 The Goal The Goal Compress understandable data into a form that minimizes the space required to store or transmit it and being able to restore compressed informations to an understandable form. Lossless Compression Binary Data : Deflate, Run-Length-Encoding, Lempel-Ziv-Welch Audio : FLAC, WMA 9 Lossless,... Image : BMP, JPEG-LS, JPEG-2000, PNG, TIFF,... Video : FFV1, Huffyuv, MSU Lossless,... Lossy Compression Audio : MP3, ACC,... Image : JPEG, JPEG-2000, PNG, GIF,... Video : MPEG-1/2/4, VP8, WebM, H.264, DiVX,...

6 What is Video? Analog vs Digital What is the best way to connect equipment? HDMI (digital YCbCr) HDMI (digital RGB) Analog YPbPr Analog RGB Analog S-Video (4 wires) Analog Composite (2 wires) (a) HDMI (b) S-Video (c) Composite

976 - MPEG-1 Full-D1 : 720x576i25 / 720x480i29.97/23.

7 What is Video? Resolution SD : PAL/NTSC VCD : 352x288p25 / 352x240p29.9/ MPEG-1 Full-D1 : 720x576i25 / 720x480i29.97/ MPEG-2 HDTV HD-Ready : 1280x720p 1980x1080i Full-HD : 1080p25/30 - DVB/ATSC - MPEG p50/60 - DVB-S2 - MPEG-4

8 What is Video? Interlaced vs Progressive For the same rate : Interlaced :Better for static scene. has a better framerate has a lower definition Figure: Exemple of artefact Progressive :Better for high motion scene. has a better definition has a lower framerate

9 The eye The Eye (a) Our receptor (b) Density of the rod and cone cells Figure: 4.5 million cone cells - 90 million rod cells [Curcio ]

10 The eye The Eye (a) Fovea (b) Periphery Cone cells : allow color perception peceive finer detail fast rapid changes detection Rod cells : high sensitivity to light

11 sensitivity The eye The Eye Overall sensitivity green red blue (x10) wavelength (nm) Figure: Responsivity spectra of human cone cells Rem : trichromats vs tetrachromatic (zebrafish, human, mice,...)?

12 Colors Color Spaces What is the gamut?

13 Colors Color Spaces srgb : RGB colors with gamma correction applied : BMP, PNG,... YUV : Y Cb Cr = Kr (1 Kr Kb) Kb 0.5 (B Y )/(1 Kb) 0.5 (R Y )/(1 Kr) R G B Requirements : channels decorrelation fast transformation reversible transformation easy to implement http: // page=transformations#ycbcr

14 Colors Color Spaces (a) RGB (b) Y (c) Cb (d) Cr

Colors Color Spaces ITU-R BT-601 vs ITU-R BT-709 vs SMPTE 240M : BT 601 = 0.256788 0.504129 0.097906 0.144914 0.284497 0.429412 0.429412 0.359579 0.069833 R G B, Kb = 0.114, Kr = 0.299 BT 709 = 0.

15 Colors Color Spaces ITU-R BT-601 vs ITU-R BT-709 vs SMPTE 240M : BT 601 = R G B, Kb = 0.114, Kr = BT 709 = R G B, Kb = , Kr = SMPTE 240M = vs : R G B, Kb = 0.087, Kr = Demo : conversion errors (a) (b)

16 Colors Color Spaces Color Transform Coding gain db Optimal (Karhunen-Loève transform) 4.97 ITU-R BT.709 [1] 3.79 FCC [4] 3.91 ITU-R BT [5] 3.92 SMPTE 170M [6] 3.92 SMPTE 240M [7] 3.83 JPEG2000 RCT 4.31 YCoCg 4.54 Figure: Theoretical coding gain YCoCg : YCoCg = R G B = 1/4 1/2 1/4 1/2 0 1/2 1/4 1/2 1/ R G B Y Co Cg YCoCg-R : A Color Space with RGB Reversibility and Low Dynamic Range Joint Video Team (JVT) of ISO/IEC MPEG & ITU-T VCEG

17 Colors Color Subsampling (a) 444p : IYU2,... (b) 422p : IUYV, UYVY,... (c) 411p (d) 420p : YV12, NV12,...

18 Colors Color Subsampling (e) 444p : IYU2,... (f) 422p : IUYV, UYVY,... (g) 411p (h) 420p : YV12, NV12,... If Y, U and V are 8-bit numbers, theirs values are in the range [0..255] In average : 444p 24bits 422p 16bits 410p, 420p 12bits NB : the pixel subsampling format does not specify the color space or the memory layout of the data ie : BT.601 vs BT.709 and planar vs packed http: //

19 Back to Basics Temporal vs Frequency (i) Frequency masking (j) Temporal masking

20 Back to Basics Temporal vs Frequency (k) Encoder (l) Decoder

21 Temporal masking Temporal reduction Macroblocks : Figure: Motion vectors / Optical Flow

22 Frequencyl masking HVS - CSF Contrast Sensitivity Function :

23 Frequencyl masking HVS - CSF Contrast Sensitivity Function : (a) Temporal and Frequency masking (b) Csf function

24 Frequencyl masking DCT DCT : Figure: DCT basis function Figure: One example of HVS matrix

Frequencyl masking DCT DCT : 16 16 16 16 17 18 21 24 16 16 16 16 17 19 22 25 16 16 17 18 20 22 25 29 16 16 18 21 24 27 31 36 17 17 20 24 30 35 41 47 18 19 22 27 35 44 54 65

25 Frequencyl masking DCT DCT : Figure: DCT basis function Figure: One example of HVS matrix Wavelets : JPEG-2000 based Codec Dirac :

26 Frames IPB frames (a) I frames

27 Frames IPB frames (e) I frames (f) IP frames

28 Frames IPB frames (i) I frames (j) IP frames (k) PB frames

29 Frames IPB frames (m) I frames (n) IP frames (o) PB frames

30 Frames Intra/Inter Figure: The big picture

31 Frames How to mesure distortions? PSNR MSE = 1 W.H W 1 x=0 H 1 I (x, y) J(x, y) 2 y=0 ( MAX 2 ) PSNR = 10 log 10 MSE

32 Frames How to mesure distortions? SSIM Luminance + Contrast + Structural changes

33 Codecs Codecs vs Containers Physical Support DVD, CD, BluRay,... Container AVI, MPG, MKV, MOV, OGM, FLV,... Video Stream Audio Stream Audio Stream Subtitle Subtitle Data MP3 FLAC AC3 OGG... MPEG-2 MPEG-4 Xvid Divx H Each container has its specifications. streaming DVD menus several audio tracks chapter support error resilient Figure: container :.flv,.mkv,.avi,.mp4 Example : MP4 vs MPEG-4

34 Codecs How to do codecs comparison? Only one of those can be compared at once Framerate, Resolution, Quality, Bitrate, Encoding speed, Decoding speed, (Rate controller : VBR, CBR, CRF, CQ)

35 Codecs How to do codecs comparison? Only one of those can be compared at once Framerate, Resolution, Quality, Bitrate, Encoding speed, Decoding speed, (Rate controller : VBR, CBR, CRF, CQ) Commons errors : color spaces biased measurement tools small pool of input files (cartoon, movies, CCTV,...)

36 Codecs How to do codecs comparison? Only one of those can be compared at once Framerate, Resolution, Quality, Bitrate, Encoding speed, Decoding speed, (Rate controller : VBR, CBR, CRF, CQ) Commons errors : color spaces biased measurement tools small pool of input files (cartoon, movies, CCTV,...) Every codec has its features : WebM x Famous benchmark by the MSU group : video/codec_comparison/introduction_en.html

37 Codecs H.264 AVC Profiles Constrained Baseline Profile (CBP), Baseline Profile (BP), Main Profile (MP), High Profile (HiP) Levels 1, 1.1, 2.1, 3, 3.1, 4

38 Codecs H.264 AVC Profiles Constrained Baseline Profile (CBP), Baseline Profile (BP), Main Profile (MP), High Profile (HiP) Levels 1, 1.1, 2.1, 3, 3.1, 4 Figure: Archos 70it

39 Codecs H.264 SVC : Resolution Framerate Quality Figure: Scalable stream

40 Overview The big loop Capturer Encoder Sender Raw Frame Raw Frame Renderer Network Enc Frame Enc Frame Decoder N Packets M Packets Receiver Figure: VoIP

41 Research vs Development vs Testing Step by Step Capturer Encoder Sender Raw Frame Raw Frame Renderer Network Enc Frame Enc Frame Decoder N Packets M Packets Receiver Figure: VoIP Capture : webcam (front/back + orientation on mobile) format (fourcc, resolution, fps) capture card (PAL/NTSC), screen sharing

42 Research vs Development vs Testing Step by Step Capturer Encoder Sender Raw Frame Raw Frame Renderer Network Enc Frame Enc Frame Decoder N Packets M Packets Receiver Figure: VoIP Encoder : speed optimization : DCT, motion vectors, slices perceptual quality : HVS matrix rate-controller : VBR, CBR, CFR...

43 Research vs Development vs Testing Step by Step Capturer Encoder Sender Raw Frame Raw Frame Renderer Network Enc Frame Enc Frame Decoder N Packets M Packets Receiver Figure: VoIP Sender : frame packetization, packet size, MTU bitrate pacer FEC, NAL

44 Research vs Development vs Testing Step by Step Capturer Encoder Sender Raw Frame Raw Frame Renderer Network Enc Frame Enc Frame Decoder N Packets M Packets Receiver Figure: VoIP Receiver : packet loss, duplication, reordering feedback frame reconstruction

45 Research vs Development vs Testing Step by Step Capturer Encoder Sender Raw Frame Raw Frame Renderer Network Enc Frame Enc Frame Decoder N Packets M Packets Receiver Figure: VoIP Decoder : concealment speed optimization frame pacer

46 Research vs Development vs Testing Step by Step Capturer Encoder Sender Raw Frame Raw Frame Renderer Network Enc Frame Enc Frame Decoder N Packets M Packets Receiver Figure: VoIP Renderer : aspect ration, cropping color space conversion orientation, scaling

47 Research vs Development vs Testing Step by Step Capturer Encoder Sender Raw Frame Raw Frame Renderer Network Enc Frame Enc Frame Decoder N Packets M Packets Receiver Figure: VoIP What is missing here? : Media layer : profile negotiation (RTP/RTSP), Audio, Conferencing Transport layer : NAT transversal, STUN

48 References Figure: Digital Video and Hdtv Algorithms and Interfaces : Algorithms and Interfaces by Charles Poynton

49 References Figure: Digital Video Quality : Vision Models And Metrics by Stefan Winkler

50 References Figure: Video Demystified : A Handbook for the Digital Engineer by Keith Jack

References Figure: http://citeseerx.ist.psu.edu/ Figure: http://www.

org/ IRC : http://freenode.net/ Softwares : AviSynth http://avisynth.

51 References Figure: Figure: Figure: IRC : Softwares : AviSynth VirtualDub Avidemux berlios.de/index.html

52 Different thinkings Waterfall Agile (Scrum, Continuous Integration, Test Driven Development) Lean (Kanban, Product Management) V-model

53 Agile Scrum Figure: Scrum

54 Agile What is a job title? Technical VS Managerial ladder Software developer Software developer in test Product Owner Project Manager

55 Out of the topic slide Skype :

56 Out of the topic slide Skype : Where Exhibition centers, conferences (MWC, IBC, CNIT,...) VIE Alumi Yellow Pages

57 Out of the topic slide Skype : Where Exhibition centers, conferences (MWC, IBC, CNIT,...) VIE Alumi To train : Yellow Pages Project Euler TopCodec

58 Out of the topic slide Skype : Where Exhibition centers, conferences (MWC, IBC, CNIT,...) VIE Alumi To train : Yellow Pages Project Euler TopCodec Moving aboard France vs others : Management & Technical career medical cover pension compare only comparable things (income taxes, net, brut/gross)

59 Questions 1 Introduction The Goal What is Video? 2 What do we see? The eye Colors 3 2D - Audio Back to Basics 4 3D - Video Temporal masking Frequencyl masking Frames Codecs 5 Video transmission Overview Research vs Development vs Testing 6 References 7 Software development methods Agile Questions?

MULTIMEDIA AND CODING

07 MULTIMEDIA AND CODING WHAT MEDIA TYPES WE KNOW? TEXTS IMAGES SOUNDS MUSIC VIDEO INTERACTIVE CONTENT Games Virtual reality EXAMPLES OF MULTIMEDIA MOVIE audio + video COMPUTER GAME audio + video + interactive