Salsify: Low-Latency Network Video Through Tighter Integration Between a Video Codec and a Transport Protocol

Transcription

1 Salsify: Low-Latency Network Video Through Tighter Integration Between a Video Codec and a Transport Protocol Sadjad Fouladi, John Emmons, Emre Orbay, Catherine Wu, Riad S. Wahby, Keith Winstein

2 Internet video delivery Sender network Receiver 2

3 Internet video delivery Different classes of Internet video delivery applications: Video streaming (previous class) Live video Real-time video (today s class) 3

4 Each of these classes has a different latency target Latency target: the amount of time between when something happens you see it and 4

5 Class 1: Video streaming Latency target: tens of minutes BBA [T.-Y. Huang et al., SIGCOMM 14] MPC [X. Yin et al., SIGCOMM 15] Pensieve [H. Mao et al., SIGCOMM 17] Oboe [Z. Akhtar, SIGCOMM 18] 5

6 Class 2: Live video Latency target: 5 30 seconds Vic [S. McCanne et al., MULTIMEDIA 95] VDN [M. K. Mukerjee et al., SIGCOMM 15] 6

7 Class 3: Real-time Video Latency target: tens of milliseconds Why would an application need that? 7

8 Real-time video systems transmit video with low latency Sender network Receiver 8

9 to maintain the interactivity of the application. Interaction Sender network Receiver 9

10 Cloud Video Gaming

11 CLOUD EDITION Remote Surgery

12 MIT INTERNET CO. Teleoperation of Robots and Vehicles

13 Video Conferencing

14 Video Conferencing

15 Video Conferencing (reality)

16 Watch the video at: WebRTC (Chrome 65)

17 Current systems do not react fast enough to network variations, end up congesting the network, causing stalls and glitches.

18 Researchers already knew about this K. Winstein, A. Sivaraman, H. Balakrishnan, Stochastic Forecasts Achieve High Throughput and Low Delay over Cellular Networks, NSDI Skype throughput (kbps) delay (ms) time (s) 18

19 and designed better throughput prediction algorithms K. Winstein, A. Sivaraman, H. Balakrishnan, Stochastic Forecasts Achieve High Throughput and Low Delay over Cellular Networks, NSDI Sprout throughput (kbps) delay (ms) time (s) 19

20 But better congestion control alone couldn t save the day.

21 Conventional design: two control loops at arm s length video codec transport protocol 21

22 Conventional design: two control loops at arm s length video codec transport protocol 22

23 Transport estimates the network throughput and communicates that to the codec target bit rate video codec transport protocol 23

24 Codec produces compressed frames, targeting that bit rate video codec transport protocol compressed frames 24

25 Transport occasionally updates that estimate new target bit rate video codec transport protocol compressed frames 25

26 Codec updates frame rate and quality accordingly video codec transport protocol compressed frames 26

27 The problem: codec and transport are too decoupled The codec can only respond to changes in target bit rate over coarse time intervals. Individual frames may cause packet loss/queueing. The transport has little control over what codec produces. The resulting system is slow to react to network variations. 27

28 Decades of research and development on these components MPEG-1 Sprout NADA VC-1video MPEG-2 GCC transport BBR LEDBAT H.265 codec H.264 H.263 protocol CDG RemyCC VP8 VP9 AV1 FBRA 28

29 Salsify explores a more tightly-integrated design transport protocol & video codec

30 Salsify, a new heart from old parts Individual component of Salsify are not exactly new: The transport protocol is inspired by packet pair and Sprout-EWMA. The video format, VP8, was finalized in The functional video codec was introduced in [Fouladi et al., NSDI 17]. Salsify is a new architecture for real-time video that integrates these components in a way that responds quickly to network variations.

31 Salsify s architecture: Video-aware transport protocol transport protocol & video codec 31

32 Video-aware transport protocol What should be the size of the next frame? * without causing excessive delay There s no notion of bit rate, only the next frame size! Inspired by packet pair and Sprout-EWMA, transport uses packet inter-arrival time, reported by the receiver. 32

33 Receiver keeps a moving average of packet inter-arrival times frame i frame i+1 Receiver Sender OLD IDEA! t₁ t₂ t₃ t₄ t₅ T α t i + (1 α) T T : average packet inter-arrival time 33

34 Sender does not transmit continuously frame i frame i+1 Receiver Sender t₁ t₂ grace t₃ t₄ t₅ period T α (t i g i ) + (1 α) T 34

35 What should be the size of the next frame? The receiver sends back an acknowledgement for every received packet, containing the latest inter-arrival time. The sender calculates the next frame size as follows: T : average inter-arrival time d : target delay N : number of packets in flights frame size = ( d T N ) packets 35

36 Salsify s architecture: Functional video codec transport protocol & video codec

37 The encoder can only know the output size after the fact. It s challenging for any codec to choose the appropriate quality settings upfront to meet a target size they tend to over-/undershoot the target. 37

38 Video codec A piece of software or hardware that compresses and decompresses digital video. Encoder Decoder 38

39 Video codecs provide a simple interface encode([,,...]) [frames...] decode([frames...]) [,,...] 39

40 First frame in the sequence is stored in its entirety keyframe 40

41 The encoder exploits the similarities between frames to compress the video keyframe 41

42 The encoder exploits the similarities between frames to compress the video keyframe 42

43 The encoder exploits the similarities between frames to compress the video keyframe "previous frame, offset (a, b)" frame offset (a, b) interframe 43

44 Predictions are not always perfect current block 44

45 Predictions are not always perfect current block closest block in previous frame 45

46 The encoder also needs to store the residues = current block closest block in previous frame residue 46

47 Decode process frame offset (a, b) + = prediction residue actual block 47

48 What frames can be referenced? The codec keeps a list of references that can be used by frames. Each frame can replace one of the reference slots with its output. 48

49 Decode process slot offset (a, b) + = prediction residue actual block 49

50 The encoder can only achieve the bit rate on average frame size (KB) 12 6 target bit rate (2 Mbps) frame number 50

51 Why real-time video is a different problem 1. Video streaming Latency target: tens of minutes 2. Live video Latency target: 5 30 seconds Encoder timescale to achieve target bit rate: 1-2 seconds How often a keyframe can be inserted: 2 seconds 3. Real-time video Latency target: tens of milliseconds 51

52 The challenge: Getting an accurate frame out of an inaccurate codec Trial and error SOUNDS GOOD, DOESN T WORK! Encode with different quality settings, pick the one that fits. 52

53 The video codec is stateful Frames are depending on references that live in the mind of the decoder. Every frame can change this set of references, and future frames are counting on that. 53

54 Video codec prob tables source state 54

55 Video codec output prob tables source state frame 55

56 Video codec output prob tables source state frame target state prob tables 56

57 Video codec is an automaton keyframe interframe interframe interframe 57

58 There s no way to undo an encoded frame in current codecs encode([,,...]) [frames...] decode([frames...]) [,,...] The state is internal to the encoder no way to save/restore the state. 58

59 Functional video codec to the rescue encode(state, ) state, frame decode(state, frame) state, Salsify s functional video codec exposes the state that can be saved/restored. Described in Encoding, Fast and Slow: Low-Latency Video Processing Using Thousands of Tiny Threads, NSDI 17 59

60 Order two, pick the one that fits! Salsify s functional video codec can explore different execution paths without committing to them. For each frame, codec presents the transport with three options: A slightly-higher-quality version, A slightly-lower-quality version, 50 KB better Discarding the frame. worse 10 KB 60

61 Salsify s architecture: Unified control loop transport protocol & video codec 61

62 Codec Transport Here s two versions of the current frame. 50 KB better worse 25 KB target frame size 30 KB 62

63 Transport Codec I picked option 2. Base the next frame on its exiting state. 25 KB target frame size 30 KB 63

64 Codec Transport Here s two versions of the latest frame. 50 KB better worse 25 KB target frame size 55 KB 64

65 Transport Codec I picked option 1. Base the next frame on its exiting state. 50 KB target frame size 55 KB 65

66 Codec Transport Here s two versions of the latest frame. 70 KB better worse KB target frame size 5 KB 66

67 Transport Codec I cannot send any frames right now. Sorry, but discard them. target frame size 5 KB 67

68 Codec Transport Fine. Here s two versions of the latest frame. 45 KB better worse 20 KB target frame size 50 KB 68

69 Transport Codec I picked option 1. Base the next frame on its exiting state. 45 KB target frame size 50 KB 69

70 There s no notion of frame rate or bit rate in the system. Frames are sent when the network can accommodate them.

71 Loss recovery Option 1: Ignore dropped packets. Option 2: Retransmit dropped packets. Option 3: Restart the video stream with a keyframe or a key-slice. 71

72 Loss corrupts the current frame... frame corrupted frame 72

73 Loss corrupts the current frame... and the rest! frame corrupted frame frame frame 73

74 Loss recovery Option 1: Ignore dropped packets. Option 2: Retransmit dropped packets. Option 3: Restart the video stream with a keyframe or a key-slice. 74

75 Option 5, Salsify s way: Jump back to the last correct state 75

76 Measurement Testbed

77 Goals for the measurement testbed A system with reproducible input video and reproducible network traces that runs unmodified version of the system-under-test. Target QoE metrics: image quality and video delay. 77

78 Goals for the measurement testbed A system with reproducible input video and reproducible network traces that runs unmodified version of the system-under-test. Target QoE metrics: image quality and video delay. 78

79 Goals for the measurement testbed A system with reproducible input video and reproducible network traces that runs unmodified version of the system-under-test. Target QoE metrics: image quality and video delay. 79

80 Video delay!!!!!!! sender receiver time 80

81 Sender Receiver Network Simulator AV.io Video System Measurement System

82 82

83 emulated network receiver HDMI output barcoded video video in/out (HDMI) HDMI to USB camera

84 emulated network receiver HDMI output barcoded video video in/out (HDMI) HDMI to USB camera

85 emulated network barcoded video video in/out (HDMI) HDMI to USB camera receiver HDMI output

86 emulated network receiver HDMI output barcoded video video in/out (HDMI) HDMI to USB camera

87 emulated network receiver HDMI output barcoded video video in/out (HDMI) HDMI to USB camera

88 emulated network receiver HDMI output barcoded video video in/out (HDMI) HDMI to USB camera

89 emulated network receiver HDMI output barcoded video video in/out (HDMI) HDMI to USB camera

90 Sent Image Timestamp: T+0.000s Received Image Timestamp: T+0.765s Quality: 9.76 db SSIM

91 Evaluation of Salsify

92 Evaluation results: Verizon LTE Trace 18 Video Quality (SSIM db) Skype Better WebRTC (VP9-SVC) WebRTC FaceTime Hangouts Video Delay (95th percentile ms)

93 Evaluation results: Verizon LTE Trace 18 Video Quality (SSIM db) Skype WebRTC (VP9-SVC) WebRTC Status Quo (conventional transport and codec) FaceTime Hangouts Video Delay (95th percentile ms)

94 Evaluation results: Verizon LTE Trace 18 Video Quality (SSIM db) Skype WebRTC (VP9-SVC) WebRTC Status Quo (conventional transport and codec) FaceTime Salsify (conventional codec) Hangouts Video Delay (95th percentile ms)

95 Evaluation results: Verizon LTE Trace 18 Video Quality (SSIM db) Skype WebRTC (VP9-SVC) WebRTC Status Quo (conventional transport and codec) FaceTime Salsify Salsify (conventional codec) Hangouts Video Delay (95th percentile ms)

96 Evaluation results: Grace Period 18 Video Quality (SSIM db) Skype WebRTC (VP9-SVC) WebRTC Status Quo (conventional transport and codec) FaceTime Salsify Salsify (conventional codec) Hangouts Salsify (no grace period) Video Delay (95th percentile ms)

97 Evaluation results: AT&T LTE Trace 16 Video Quality (SSIM db) WebRTC (VP9-SVC) WebRTC Hangouts Better FaceTime Skype Salsify Video Delay (95th percentile ms)

98 Evaluation results: T-Mobile UMTS Trace 14 Salsify Video Quality (SSIM db) WebRTC (VP9-SVC) Skype WebRTC Better FaceTime Hangouts Video Delay (95th percentile ms)

99 Network Variations Watch the video at: WebRTC (Google Chrome 65.0 dev) Salsify

100 Network Outages Watch the video at: WebRTC (Google Chrome 65.0 dev) Salsify

101 Final Remarks

102 Where Salsify is not a good fit Long latency budgets Non-variable links Low-power devices 102

103 Where Salsify is not a good fit Long latency budgets Non-variable links Low-power devices 103

104 Where Salsify is not a good fit: non-variable links 12 WebRTC (VP9-SVC) Video Quality (SSIM db) Better FaceTime WebRTC Hangouts Salsify Skype Video Delay (95th percentile ms)

105 Where Salsify is not a good fit Long latency budgets Non-variable links Low-power devices 105

106 Codecs have been treated as black boxes in video systems for a long time.

107 Takeaways Salsify is a new architecture for real-time Internet video. Salsify tightly integrates a video-aware transport protocol, with a functional video codec, allowing it to respond quickly to changing network conditions. Changes to the architecture of systems can sometimes yield significant benefits. More info at: snr.stanford.edu/salsify Thank you: NSF, DARPA, Google, Dropbox, VMware, Huawei, Facebook, Stanford Platform Lab, and James.

108

109

110