網路串流技術 (III) 鄭憲宗 資訊工程學系 國立成功大學

Transcription

1 網路串流技術 (III) 鄭憲宗 資訊工程學系 國立成功大學

2 Outline 5. Wireless Streaming Techniques: 5.1. Flow Control and Error Recovery 5.2. Buffer Management 6. Applications and Conclusions 6.1. 影音同步議題 6.2. 多媒體串流技術 : 可調式壓縮 (scalable coding) 6.3. flash streaming

3 視訊串流系統在 Wireless LAN 的 應用及限制

4 Outline IEEE 802 Committees Spread Spectrum Modulation Schemes Ease Address Problems Basic Service Set and Extended Service Set End-to-End QoS Video Networking over Wireless Application Conclusions

5 IEEE 802 Committees

6

7

8 Spread Spectrum Modulation Schemes Ease Address Problems DSSS Direct Sequence Spread Spectrum OFDM Orthogonal Frequency Division Multiplexing FSSS Frequency Hopping Spread Spectrum

9 DSSS- Direct Sequence Spread Spectrum (1/3)

10 DSSS- Direct Sequence Spread Spectrum (2/3)

11 DSSS- Direct Sequence Spread Spectrum (3/3)

12 OFDM- Orthogonal Frequency Division Multiplexing

13 Basic Service Set (BSS)

14 Independent Basic Service Set (IBSS)

15 Extended Service Set (ESS) ESS: one or more BSSs interconnected by a Distribution System (DS) Traffic always flows via access point Allows clients to seamlessly roam between APs

16 Distributed System (DS) A thin layer in each AP Embodied as part of the bridge function Keeps track of AP-NN associations Delivers frames between APs Three types: Integrated: A single AP in a standalone network Wired: Using cable to interconnect the APs Wireless: Using wireless to interconnect the APs

17 ESS: Single BSS (with integrated DS)

18 ESS: BSS s with Wired Distribution System (DS)

19 ESS: BSS s with Wireless Distribution System (DS)

20 End-to-End QoS Video Networking over Wireless

21 Adaptive to Wireless Heterogeneous Networks

22 Moving Toward All-IP Wireless Broadband

23 Perfect Synergy of WLAN/Wi-Fi and WiMAX

24 Ad-Hoc & Infrastructure Modes of WLAN

25 Serving Multiple Video Streams in A WLAN

26 Distributed Camera Networks (DCNs)

27 Tracking Across Ad-Hoc Camera Networks

28 Application to Vehicular Wireless Ad-Hoc Networks

29 One-Hop or Multi-Hop Broadcasting within DCNs

30 Modeling Backoff Mechanism using

31 Modeling the Dynamics of Multiple Nodes

32 Metrics for Performance Evaluation

33 Throughput Maximization

34 Future internet: Conclusion content + service + management Interactive, ubiquitous, personalized, secure, aware Video networking and IPTV are killer applications for the next generation wireless broadband

35 References Jenq-Neng Hwang, QoS Challenges for Video over Wireless Broadband: WLAN, Wireless Ad Hoc and WiMAX Networks, Department of Electrical Engineering University of Washington, Seattle WA.

36 Wireless Video Transmission Techniques on MPEG-4 Streaming System 36

37 Outline Introduction Background and Related Work Streaming System Process of Adding Hint Tracks Design and Implementation of Video Packet Interleaving Strategy Conclusion & Future Work 37

38 Introduction Two main problems of Wireless Streaming: Low bandwidth Adopt High Compression Ratio Error Prone degrade the playback quality (especially in video) FEC (Forward Error Correction), Re- Transmission, Interleaving, Redundant Packets, etc. 38

39 Outline Introduction Background and Related Work Streaming System Process of Adding Hint Tracks Design and Implementation of Video Packet Interleaving Strategy Conclusion & Future Work 39

40 Background and Related Work Streaming Techniques Introduction to MPEG-4 MPEG-4 overview Video Coding Concepts Video Error Resilient Tools MPEG-4 File Format(.mp4) Related Protocols RTP/RTCP RTSP SDP 40

41 Streaming Techniques Features: A media content is segmented into small pieces and they are sent to client one-by-one. Need small buffer on client: Save Storage Viewing the media content immediately: Save Time Two Model: Pull Model Push Model 41

42 Two Model Pull Model of Streaming e.g. HTTP streaming Push Model of Streaming e.g. RTP streaming 42

43 HTTP RTP Streaming approach Pull Push Predominance Client Server Transport Layer TCP UDP Reliability High Low Delay sensitivity High Low Implementation difficulty Low High Server type Web server Dedicated server 43

44 Background and Related Work Streaming Techniques Introduction to MPEG-4 MPEG-4 overview Video Coding Concepts MPEG-4 File Format(.mp4) Related Protocols RTP/RTCP RTSP SDP 44

45 MPEG-4 overview MPEG-4 : New standard of multimedia coding Object-based coding scheme Support error-resilience in error-prone environment Resync Mark Data partition RVLC Systems architecture Object-oriented scene presentation Interaction Tools BIFS, OD Framework, Delivery layer - DMIF Independent of applications 45

46 Video Coding Concepts(1/2) MPEG-4 provides three modes for encoding input VOP. Intra frames (I-frame) A frame encoded independently (like JPEG) forward Predicted frames (P-frame) A frame that is predicted based on previous frames. Bidirectional predicted frames (B-frame) A frame that is predicted based on past as well as future frames. 46

47 Video Coding Concepts(2/2) Example of GOP pattern 47

48 MPEG-4 File Format(.mp4) The file is composed of the smallest units - atom in a hierarchical manner. Three Different Types of Track: audio track video track hint track Easy to Parse and Extract the RTP hint samples for streaming server Stored the SDP information in each hint track 48

49 Atom Structure Layout SIZE section (e.g. n) 4 bytes TYPE section (e.g. moov, trak) 4 bytes Data/ Child Atoms n-8 bytes 49

50 Simple File Format Layout mp4 file moov trak (video) mdat Audio Samples trak (audio) Video Samples Other atoms Atoms of Meta-data trak (hint) RTP Hint Samples Atom that contains Access Units 50

51 Related Protocols(1) RTP: Real Time Transport Protocol (RFC 1889) Based on UDP (additional 12 bytes header) Header seq., payload type, ssrc RTCP: Real Time Control Protocol Packet type: SR,RR,SDES,APP,BYE feedback information between source & destination Functions: Synchronization Detect loss, round trip time, jitter 51

52 Related Protocols(2) RTSP: Real Time Streaming Protocol (RFC 2326) RTSP acts as a "network remote control" for multimedia servers. The protocol is similar in syntax and operation to HTTP. SDP: Session Description Protocol (RFC 2327) Actually, the SDP is just the text information, not a really protocol. It is intended for describing multimedia sessions for the purpose of session announcement, session invitation. 52

53 Outline Introduction Background and Related Work Streaming System Design and Implementation of Video Packet Interleaving Strategy Conclusion & Future Work 53

54 Streaming System Testbed: ITRI/CCL media box streaming server Apply IETF based Streaming Solutions Reference MPEG4IP, Apple Darwin Streaming Server Network Environment: IEEE b 54

55 Streaming System Gen playlist & config file fork mproc CGI program HTTP SDP Generator RTSP server Web DESCRIBEBrowser SDP info SETUP,PLAY Hinted mp4 file RTPTransmitter Player MP4 RTP Inetrleaving De-Inetrleaving Reader Module Module RTCP Process of Adding Hint Tracks 55 Server Side Client Side

56 Outline Introduction Background and Related Work Streaming System Process of Adding Hint Tracks Design and Implementation of Video Packet Interleaving Strategy Conclusion & Future Work 56

57 Process of Adding Hint Tracks(1) Step 1: get and parse the un-hinted mp4 file ftyp mdat video bitstream moov mvhd iods trak... Step 2:copy the moov atom to Temp file ftyp mdat video bitstream moov mvhd iods trak... copy Temp File 57

58 Process of Adding Hint Tracks(2) Step 3: get each sample size with stsz atom, and fragment the media samples into RTP packets with the maximum payload size :1460 bytes. Add partial RTP header information Step 4: write the RTPHintSamples to file ftyp mdat video bitstream mdat RTPHintSamples 58

59 Process of Adding Hint Tracks(3) Step 5: write the original moov to file and modify its size section (p.s. add the size of hint track to the size of moov ) ftyp mdat video bitstream mdat RTPHintSamples moov... Temp file Step 6: write the hint track atoms to files (include SDP information) ftyp mdat... mdat... moov trak... udta... 59

60 Process of Adding Hint Tracks(4) moov trak tkhd tref mdia payload type udta hnti sdp hinf m=video 0 RTP/AVP 96 a=rtpmap:96 MP4V-ES/90000 a=control:trackid=2 a=mpeg4-esid:1 a=fmtp:96 profile-level-id=1; config=000001b b c d0a ; 60 timescale

61 Outline Introduction Background and Related Work Streaming System Process of Adding Hint Tracks Design and Implementation of Video Packet Interleaving Strategy Conclusion & Future Work 61

62 Design and Implementation of Video Packet Interleaving Strategy Idea and concepts Algorithm Implementation Details Evaluation 62

63 Idea and concepts Adopt the Interleaving scheme with GOP (Group of Pictures) concepts. Two priorities: I-frame > Non-I-frame Method: Insert the Non-I-packets into I-packets Distance of successive packets: I-packets > Non-I-packets Benefits: Reduce the Impact of Bursty Packet Losses in I-frame Drawbacks: Require small Interleaving buffer on server (small delay) Require small De-interleaving buffer on client (small delay) 63

64 Algorithm Server Side (Interleaving) Client Side (De-Interleaving) 64

65 Server Side(1) I B B P B B P B B GOP 1 GOP 2 INBUFF i 1 i 2 i 3 i 4 N i =4 TEM P b 1 b 2 p 1,1 p 1,2 b 3 b 4 p 2,1 p 2,2 b 5 b 6 65 N pb =10

66 Server Side(2) q = N pb ( N 1) r = N % N 1 pb i ( ) i INBUFF i 1 i 2 i 3 i 4 b 1 b 2 p 1,1 p 1,2 b 3 b 4 p 2,1 p 2,2 b 5 b 6 66 q=3 q=3 q=3 r=1

67 Time Complexity of Server The additional merge Time Complexity of INBUFF:O( N M ) N denotes the length of INBUFF M denotes the length of TEMP 67

68 Client Side RTP packet Get RTP Timestamp T NOW if (T NOW -T INIT <=D GOP ){ DEINTBUFF RTP packet RTP packet RTP packet }else{ } 68 T 1 <= T 2 RTP packet T 3 <= T NOW <= Flush DEINTBUFF to DECOBUFF (decoding buffer) T INT =T INT +D GOP ;

69 Time Complexity of Client the additional time complexity of de-interleaving on the client :O(N 2 ) N denotes the total number of RTP packets in one GOP 69

70 Implementation Details(1) Server Side (mproc) ForwardData(Audio/Video Samples) MP4Reader RTPTransmitter mp4v2 JRTPLib Interleaving Module 70

71 Implementation Details(2) Client Side (wmp4player) GUI (Win32) Display (SDL.dll) Player System (libmpplayer.lib) Xvid(Xvid_plugin.dll) AAC(aac_plugin.dll) MP3(mp3_plugin.dll) De-Interleaving Module 71 UCL RTP Lib (uclmm.lib)

72 Evaluation Packet Loss Characterization Two State Markov Chain (Gilbert Model) Peak Signal to Noise Ratio (PSNR) Method Results 72

73 Two State Markov Chain (Gilbert Model)(1) p p (GOOD) (BAD) 1-q q 73

74 Two State Markov Chain (Gilbert Model)(2) p=p(1 0)= P (packet n is lost packet n-1 is received) q=p(0 1)= P (packet n is received packet n-1 is lost) P(X=0) Avg. Success Prob. P(X=1) Avg. Loss Prob. 1 p q PX ( = 0) PX ( = 0) = p 1 q P( X = 1) P( X = 1)..(1) PX ( = 0) + PX ( = 1) = 1..(2) p From (1) and (2): PX ( = 1) = p + 74 q

75 Two State Markov Chain (Gilbert Model)(3) Equation (3) gives the probability of having a loss episode with length k (k consecutively lost packets) given that the flow had enter the loss state X=1. p = (1 q) k 1 q k.(3) Consequently, the lengths of loss episodes (in the Gilbert model) are geometrically distributed and thus the average length of a bursty loss equals 1/q 75

76 Peak Signal to Noise Ratio (PSNR) PSNR = 20log [ f (, i j) f '(, i j) ] ( N N ) 1 2 MSE (Mean Square Error) 76

77 Method of Experiment Original bitstream Classic LossTable modified bitstream Source bitstream: mobile.m4v decoder 300 frames (352*288) GOP size 15 Raw 1 Mbits/s Avg. Video PSNR Original bitstream modified bitstream decoder Raw Video Avg. PSNR Interleaving Loss Table 77

78 PSNR Result(1) Avg. Burst Length : 4 classic interleaving 78 Avg. PSNR(db) Avg. Loss Rate

79 PSNR Result(2) Avg. Burst Length : 6 classic interleaving Avg. PSNR(db) Avg. Loss Rate 79

80 PSNR Result(3) Avg. Burst Length:8 classic interleaving Avg. PSNR(db) Avg. Loss Rate 80

81 Snapshot Result(1) Comparison of frame 31 81

82 Snapshot Result(2) Comparison of frame 61 82

83 Snapshot Result(3) Comparison of frame

84 Outline Introduction Background and Related Work Streaming System Process of Adding Hint Tracks Design and Implementation of Video Packet Interleaving Strategy Conclusion & Future Work 84

85 Conclusion & Future Work Integration of other codecs error resilient supported codecs H.264 codec Combine other error control schemes FEC, Re-Transmission, Duplicated Packets, etc. Support other file format Extend to support 3gp file format Porting the player to other platform PDA, Smart Phone 85

86 Thanks! 86