Lec 21 Multimedia Communication Summary Part II Multimedia Transport

Transcription

1 CS/EE 5590 / ENG 401 Special Topics (17804, 17815, 17803) Lec 21 Multimedia Communication Summary Part II Multimedia Transport Zhu Li Course Web: Z. Li, Multimedia Communciation, 2016 Spring p.1

2 Outline Multimedia Communication Summary Advanced Multimedia Communication Quiz-2 Review MPEG Systems: File Format (FF) and DASH QoS and QoE Multimedia Transport Systems Media Transport Congestion Models and Control Rateless Erasure Correction Network and Storage Coding Z. Li, Multimedia Communciation, 2016 Spring p.2

3 Part I: Video Coding Summary Visual Signal Processing & Compression 2D video signal processing: color space conversion, transforms, motion compensation Signal Compression: entropy, conditional entropy, arithmetic coding, context adaptive arithmetic coding Video coding tools: intra-prediction, interprediction, Fast Motion Estimation, R-D Optimization: Lagrangian methods for optimization Software tools: HEVC open source, Python Tool HARP. Z. Li, Multimedia Communciation, 2016 Spring p.3

4 Part II: Multimedia Transport Summary Video Communication MPEG system: FF (File Formats) o abstracting/api, agnostic of compression technology. MPEG system: DASH/MMT: o DASH: streaming solution. MPD, adaptation via client pulled HTTP GET. o MMT: for broadcasting, works over IP networks QoE metrics: o Subjective QoE metrics: MOS score, Objective QoE: PSNR, e.g., Perceptual QoE metric: SSIM Media Transport: o RTP/RTCP/RTSP, works over IP networks, o HTTP/WebSocket, defacto Internet Transport o WebRTC, QUIC, new development, non TCP transport over the Internet Congestion Models and Control o TCP Type: AIMD, Slow Recovery, Fast Re-transmit. Delay based TCP o RMCAT: GCC: Delay + Loss based Congestion Control FEC o Multicasting Coding: LT code o Network Coding o Storage Coding MOS JPEG images JPEG2000 images Fitting with Logistic Function MSSIM (Gaussian window, K1 = 0.01, K2 = 0.03) Z. Li, Multimedia Communciation, 2016 Spring p.4

5 Outline Multimedia Communication Summary Advanced Multimedia Communication Quiz-2 Review MPEG Systems: File Format (FF) and DASH QoS and QoE Multimedia Transport Systems Media Transport Congestion Models and Control Rateless Erasure Correction Network and Storage Coding Z. Li, Multimedia Communciation, 2016 Spring p.5

6 Special Topics Class: Advanced Multimedia Communication EE/CS 5590 Special Topics Class for Winter Session, 2016 Advanced Multimedia Communication Course Syllabus Prepare MS/PhD students for cutting edge technology and research in the advanced multimedia computing and communication area Format: o Lectures, Paper Presentation, Project Evaluation: o Homework: 30%, Paper Presentation: 20%, Project 50%. Prerequisite: o Have taken Multimedia Communication class, or consent from instructor. Special Thanks to Samsung Research America, for donating devices for this class! Z. Li, Multimedia Communciation, 2016 Spring p.6

7 3D Multimedia Computing & Communication 360 Video Panoramic (stereo) video capture 360 video compression tools 360 video system research Object tracking and identification in 360 video Light Field Light field capture and compression (e.g, modified HEVC intra based) Super resolution from Light Field JPEG XS: low complexity low delay compression of light field Point Cloud Point Cloud Geometry Compression o Intra and inter prediction scheme o OctTree vs KdTree methods o Poisson surface method Point Cloud Color Attribute Compression o Graph signal processing approach Z. Li, Multimedia Communciation, 2016 Spring p.7

8 Transport Optimization: Instantaneous Streaming New Compression Schemes Dictionary Learning and Super-resolution Deep Learning as a compression technique Spatio-Temporal QoE metric and Quantization Design New Transport Study QUIC/WebRTC system simulation over NS-3 and GENI New congestion model from machine learning Joint Congestion Avoidance and FEC Content Identification & Network De-Duplication Robust rate agnostic content identification De-duplication and delivery acceleration IRTF Info Centric Networking CDN and Cache HTTP Cache system Robust cache de-duplication and re-direction Source-Channel coding for Cache and Storage P2P and Network Coding for CDN Z. Li, Multimedia Communciation, 2016 Spring p.8

9 Outline Multimedia Communication Summary Advanced Multimedia Communication Quiz-2 Review MPEG Systems: File Format (FF) and DASH QoS and QoE Multimedia Transport Systems Media Transport Congestion Models and Control Rateless Erasure Correction Network and Storage Coding Z. Li, Multimedia Communciation, 2016 Spring p.9

10 MPEG File Format (FF) Chocolate box approach Abstract the content into a hierarchical structure, meta data, media data Hiding the compression tech details The Solution ISOBMFF ISO Based Media File Format aka, Mp4 moov mdat moof hint Z. Li Multimedia Communciation, 2016 Spring p.10

11 ISOBMFF file types Plain File: Simple recording of plain media data, data first, header last mdat box, then moov box, e.g, foreman_320kbps_nf120.mp4 Progressive File: For progressive download or streaming Header first then media data Interleaved Chunks Fragmented File: Multiple moof segments followed by moov box Good for continous recording Segmented File: Self contained and playable fragments in signle file or in separate files For HTTP streaming (DASH) Tools: segment file, indexing Z. Li Multimedia Communciation, 2016 Spring p.11

12 DASH in a Nutshell Dynamic, Adaptive, Streaming over HTTP: An OTT solution HTTP De-facto Internet Transport Infrastructure Z. Li Multimedia Communciation, 2016 Spring p.12

13 Media Presentation Data Model MPD: a manifest of content available on HTTP server Accessible segments and their timing As a.xml file to be retrieved by clients at the start of DASH session Credits to figures on following slides: Christian and Ali, Over the Top Content Delivery: State of the Art and Challenges Ahead, ICME 2015 Tutorial Z. Li Multimedia Communciation, 2016 Spring p.13

14 DASH FDH/SAND DASH FDH Full Duplex HTTP with HTTP 2.0 and WebSocket implementation Define sub-protocols and messages for push based operations. DASH SAND Parameters Enhancing Delivery (PED) messages that are exchanged between DANEs, Parameters Enhancing Reception (PER) messages that are sent from DANEs to DASH clients, Metrics and Status messages that are sent from DASH clients to DANEs. Z. Li Multimedia Communciation, 2016 Spring p.14

15 QoS QoS Quality of Service A network centric metric Measuring the delay, loss, throughput,..etc Does not directly translate into user experiences Typically characterized by the packet arrival and departure curves Buffer size: b(a,d, t), delay, d(a,d, t) Z. Li, Multimedia Communciation, 2016 Spring p.15

16 Subjective and Objective QoE MOS Scores User Study based PSNR most widely used, good correlation with MOS Weigthed PSNR (from different channels) Weighted MSE WPSNR = w 1 PSNR 1 + w 2 PSNR 2 + w 3 PSNR 3 WPSNR_MSE 10log 10 (w MSE 1 1 (2 1) w MSE 2 B 2 2 w 3 MSE 3 ) Z. Li Multimedia Communciation, 2016 Spring p.16

17 Perceptive QoE: SSIM Structural Similarity Measure (SSIM) current state of art General factorized form of power a, b, c:, = [,,, ] Typically used: a=b=c=1, =, let = = Then:, = (2 + )(2 + ) ( + + )( + + ) Matlab Implementation: [val, map]=ssim(ref_im, im); Z. Li Multimedia Communciation, 2016 Spring p.17

18 Validation with MOS Scores SSIM is a better predictor than PSNR Dataset JP2(1) JP2(2) JPG(1) JPG(2) Noise Blur Error # of images PSNR SSIM JPEG images JPEG2000 images Fitting with Logistic Function JPEG images JPEG2000 images Fitting with Logistic Function MOS 50 MOS PSNR MOS(PSNR) MSSIM (Gaussian window, K1 = 0.01, K2 = 0.03) MOS(MSSIM) Z. Li Multimedia Communciation, 2016 Spring p.18

19 RTP Header Payload type RTP Header Incremented by one for each RTP PDU: PDU loss detection Restore PDU sequence Identifies synchronization source Identifies contributing sources (used by mixers) Z. Li, Multimedia Communciation, 2016 Spring p.19

20 SPDY / HTTP 2.0/WebSocket Work Still works on top of a TCP connection Slow start (mitigated by changing init cwnd size to 16) Head of Line (HOL) blocking: o Another disadvantage of SPDY is that an out-of-order packet delivery for TCP induces head of line blocking for all the SPDY streams multiplexed on that TCP connection. Connection Latency: 3 RTT to establish a secure link Under utilization of link capacity by TCP Rate Control no loss, in order delivery, not that a big deal for media data (we have CTS/DTS) Z. Li, Multimedia Communciation, 2016 Spring p.20

21 WebRTC Motivation: native browser support for real time communication for a variety of applications Non TCP based connectivity, over RTP instead. Javascript API for HTML (W3C) Signalling & NAT traversal, Security (IETF RTCWEB) Congestion control (IETF RMCAT) Z. Li, Multimedia Communciation, 2016 Spring p.21

22 QUIC Quick UDP Internet Connection Main QUIC Features/Design Goals: Connection establishment latency Improved congestion control more suited for media QoE Multiplexing without head-of-line blocking Forward Error Correction (FEC) reduce delay. Connection migration: native support for multipath via CID (Connection ID) Z. Li, Multimedia Communciation, 2016 Spring p.22

23 TCP Congestion Control AIMD- Additive Increase Multiplicative Decrease, congestion window based control. When receiving ACK, increase cwnd by one, when packet loss, halve the cwnd Slow Start (TCP Tahoe): Introducing a threshold control When below thres, double cwnd per ACK If above thres, additive increase If loss, halve thres, cwnd=1 Fast retransmit: After receiving 3 duplicate ACK Resend first packet in window. o Try to avoid waiting for timeout Fast recovery: After retransmission do not enter slowstart. Threshold = cwnd/2 Congwin = 3 + Congwin/2 Each duplicate ACK received Congwin++ After new ACK o Congwin = Threshold o return to congestion avoidance Z. Li, Multimedia Communciation, 2016 Spring p.23

24 TCP Rate at steady state Segment size: MSS Round Trip Delay: RTT Prob of packet loss: p TCP Throughput = ( ) Observation Reducing RTT is the key! Indeed, AKAMAI, Netflix,,etc, use RTT as the KPI for deploying and provisioning CDN edge servers. Prob of loss is due to congestion, mostly. For wireless networks, loss due to PHY layer has wrong interpretation in TCP control! Z. Li, Multimedia Communciation, 2016 Spring p.24

25 RAMCAT - GCC Google Congestion Control (GCC) implemented in Chrome and Firefox to support WebRTC Utilizes RTP and RTCP for media data transport and control Has sender side control, which is loss based, probe the available BW as sending rate A s. Receiver side control is delay based, computes REMB, Receiver Estimated Maximum Bitrate, A r to limit the sending rate A s = max, 1 0.5, > ( + 1, < 0.02, Z. Li, Multimedia Communciation, 2016 Spring p.25

26 RMCAT: Receiver Side Delay Based Congestion Model Receiver side update A r (t i ) according to the congestion state estimation Packet Arrival Stats based link usage state estimation, Packet delay variation: Where, {t i } {T i } are time stamps of ith video packet sending and recving time. C is the link capacity, L is the video packet size. Queuing delay variation: m(t i ) = t i t i-1 (T i -T i-1 ) Network jitter noise, n(t i ), Z. Li, Multimedia Communciation, 2016 Spring p.26

27 RMCAT: Link Overuse Detection Observe arrival filter signal m(t): Z. Li, Multimedia Communciation, 2016 Spring p.27

28 FEC: Fountain Codes Randomized Coding scheme that generates a continuous stream of bits Erasure adaptation at the receiver Original content Encoded packets Users reconstruct Original content as soon as they receive enough packets Encoding Engine Transmission Z. Li, Multimedia Communciation, 2016 Spring p.28

29 FEC: LT Code Encoding Draw a degree distribution at coding iteration n, d(n), from Soliton distribution d,,,,, K K( K 1) Then random draw d(n) symbols from uniform distribution to generate xor output bits Z. Li, Multimedia Communciation, 2016 Spring p.29

30 Decoding Algorithm FEC: LT Code Decoding Performance Prob of decoding K source symbols from K symbols, (1- ) The overhead is, ln ( / ) Z. Li, Multimedia Communciation, 2016 Spring p.30

31 FEC: LT Code Decoding Example Iteratively removing degree one nodes, until there s no edge left and all source bits recovered Find degree 1 checks t 1 Set s 1 =t 1 Update all checks connected by s 1, {t 2, t 4 } Remove edges to s 1. Find degree 1 checks t 4 Set s 2 =t 4, update checks connected by s 2, {t 2,t 3 } Remove check edges connected by s 2, {t 2,t 3 } Decode s 3 = t 3. Z. Li, Multimedia Communciation, 2016 Spring p.31

32 Network Coding Liquid interpretation of network flows The total cost of communication is the sum of costs of sending all info bits from source to destination Circuits Switch (Analog Networks): Create an electric circuits that connects the end points Similar to connecting pipes (still doing this in oil pipelines) Exclusive: pipelines at any given time can only serve one flow Packet Switch (Internet) Info is packetized Each packet is sent independently Packets can share the same pipeline Congestion is taken care of by Routing. Can we do better? Network Coding! Z. Li, Multimedia Communciation, 2016 Spring p.32

33 Network Coding: A Simple Use Case A and B are sending info via node S: Traditional method: involves 6 packets Network Coding: involves 4 packets: at A: a+(a+b)=b, B: (a+b)+b = a Gain of 50% efficiency replacing routing with network coding, or mixing info via network coding. Z. Li, Multimedia Communciation, 2016 Spring p.33

34 Network Coding: Butterfly Network Example Network coding improves the throughput At the receiver t 1, receiving a, directly, recover b= (a+b)+a At the receiver t 2, receiving b directly, recover a = (a+b) +b t 1 S t 2 Z. Li, Multimedia Communciation, 2016 Spring p.34

35 Network Coding Summary A powerful idea with many great potentials Main difficulties: High complexity in inverting G t, network encoding matrix Overhead due to carrying encoding vector at each node Linear dependency in encoding at nodes, resulting in lack of innovation for inverting Gt. Applications Gossip algorithms via Network Coding: sending n packets to n users, randomly receiving and mixing, O(n) instead of O(nlogn) P2P file sharing: MS Avalanche Passive Network Tomography: Find out the edge failures in network via network coding Z. Li, Multimedia Communciation, 2016 Spring p.35

36 Coding for storage Storage Coding: Erasure Protection Z. Li, Multimedia Communciation, 2016 Spring p.36

37 Storage Coding: Coding for Cloud Storage Storage erasure resilience For n-k MDS code, we can recover info from any combination of m disk failures Z. Li, Multimedia Communciation, 2016 Spring p.37

38 Repetition: Erasure tolerance: 1 Overhead: 2 Repetition vs Coding in Storage Coding Tolerance: 1 Overhead: 1.5 Z. Li, Multimedia Communciation, 2016 Spring p.38

39 Loss recovery in distributed storage Failed node recovery: Node a failed Node e is online How much communication is necessary for e to recover info on a? Z. Li, Multimedia Communciation, 2016 Spring p.39

40 A Naïve Solution Download 2mb data Node e download (a+b) and (a+2b) Can recover both a and b by inverting [1 1; 1 2] matrix To recover 1mb data we incurred 2mb traffic. Z. Li, Multimedia Communciation, 2016 Spring p.40

41 Network Code Design to reduce repair bandwidth Split the 1mb data packet into 500kb parts Encode parts and send only 1.5mb data that can be decoded. Similar to the network butterfly example Z. Li, Multimedia Communciation, 2016 Spring p.41

42 The Min Storage and Bandwidth Tradeoff If we allow each node to store some extra info, then we can improve repair bandwidth efficiency Min storage Min repair bandwidth Z. Li, Multimedia Communciation, 2016 Spring p.42

43 Summary MPEG Systems File Format offers an abstraction of underlying compression technology and provide API to applications, open source: MP4Box tool DASH: HTTP based video transport, key innovation: MPD, pull based operation. New development: FDH, SAND. Media Transport RTP/RTCP/RTSP HTTP/SPDY/WebSocket WebRTC QUIC Coding for Transport/Storage Fountain Coding: for multicast, draw from 2 distributions, decoding by BP Network Coding: mix data instead of buffer and routing. High promise, still quite some challenges. Storage Coding: find optimal tradeoff between storage efficiency and loss reconstruction overhead. Z. Li, Multimedia Communciation, 2016 Spring p.43