Πολυμεσικό Υλικό στο Internet: Συγχρονισμός, Επεξεργασία και Διακίνηση

Πολυμεσικό Υλικό στο Internet: Συγχρονισμός, Επεξεργασία και Διακίνηση Διακίνηση Video με χρήση του HTTP Β. Μάγκλαρης <maglaris@netmode.ntua.gr> Μ. Γραμματικού <mary@netmode.ntua.gr> Δ. Καλογεράς <dkalo@noc.ntua.ge> www.netmode.ntua.gr

Outline Streaming over Internet 1 st Generation Progressive 2 nd Generation UDP based Real-Time Protocols (RTP, RTCP, RTSP) 3 rd Generation TCP based Proprietary Adobe Adaptive HTTP based

Challenges TCP/UDP/IP, best-effort, with no guarantees on expectation or variance of packet delay Streaming applications delay of 5 to 10 seconds is typical and has been acceptable, but performance deteriorate if links are congested Real-Time Interactive requirements on delay and its jitter have been satisfied by overprovisioning (providing plenty of bandwidth), what will happen when the load increases?...

On-Demand Streaming Important and growing application due to reduction of storage costs, increase in high speed net access from homes, enhancements to caching Interactive control by user (but often with long response time) Ubiquitous on the web: YouTube, Netflix, Vimeo Television networks, Hollywood, etc. Most local radio & TV stations Virtually everywhere on websites 4

First Generation: HTTP Download A simple architecture is to have the Browser request the object(s) and after their reception pass them to the player for display No pipelining

Helper Application i.e. media player Displays content, which is typically requested via a Web browser; typical functions: Decompression Jitter removal Error correction: use redundant packets to be used for reconstruction of original stream GUI for user control Examples: RealPlayer Adobe Flash Player Windows Media Player QuickTime DivX Web Player

First Gen: HTTP Progressive Download (2) Alternative: set up connection between server and player; player takes over Web browser requests and receives a Meta File (a file describing the object) instead of receiving the file itself; Browser launches the appropriate Player and passes it the Meta File; Player sets up a TCP connection with Web Server and downloads or streams the file

Meta file requests 8

HTTP Progressive Download With helper application doing the download, playback can start immediately... Or after sufficient bytes are buffered Sender sends at maximum possible rate under TCP; retransmit when error is encountered; Player uses a much larger buffer to smooth delivery rate of TCP

HTTP Progressive Download

Buffering Continuous Media Jitter = variation from ideal timing Media delivery must have very low jitter Video frames every 30ms or so Audio: ultimately samples need < 128ms jitter But network packets have much more jitter that that! Solution: buffers Fill them with best effort Drain them via low-latency, local access

Max Buffer Size File Position Streaming, Buffers and Timing Max Buffer Duration "Bad": Buffer overrflows = allowable jitter "Good" Region: smooth playback Buffer Duration Buffer Size "Bad": Buffer underflows and playback stops Buffer almost empty Time

The myth of SBR Any single bitrate you choose for a video is by definition the wrong video for 99% of users. Average Throughput for your Site Visitors 100 90 80 70 These users will rebuffer These have lower quality than they could sustain 60 50 40 30 20 10 0 100 300 500 700 900 1100 1300 1500 1700 1900 2100 2300 2500 2700 2900 3100 3300 3500 3700 3900 4100 4300 4500 4700 4900 5100

Drawbacks of HTTP Progressive Download (2) HTTP connection keeps data flowing as fast as possible to user's local buffer May download lots of extra data if you do not watch the video TCP file transfer can use more bandwidth than necessary Mismatch between whole file transfer and stop/start/seek playback controls. However: use file range requests to seek to video position Cannot change video quality (bit rate) to adapt to network congestion

2nd Generation: Real-Time Streaming This gets us around HTTP, allows a choice of UDP vs. TCP and the application layer protocol can be better tailored to Streaming; many enhancements options are possible

Legacy Video Streaming Approaches RTSP and RTP

Example: Real Time Streaming Protocol (RTSP) For user to control display: rewind, fast forward, pause, resume, etc Out-of-band protocol (uses two connections, one for control messages (Port 554) and one for media stream) RFC 2326 permits use of either TCP or UDP for the control messages connection, sometimes called the RTSP Channel As before, meta file is communicated to web browser which then launches the Player; Player sets up an RTSP connection for control messages in addition to the connection for the streaming media

RTSP Operation 18

RTSP Media Stream Stateful Server keeps track of client's state Client issues Play, Pause,..., Close Steady stream of packets UDP - lower latency TCP - may get through more firewalls, reliable Credit: some content adapted from Alex Zambelli 19

RTSP Exchange Example C: SETUP rtsp://audio.example.com/xena/audio RTSP/1.0 Transport: rtp/udp; compression; port=3056; mode=play S: RTSP/1.0 200 1 OK Session 4231 C: PLAY rtsp://audio.example.com/xena/audio.en/lofi RTSP/1.0 Session: 4231 Range: npt=0 (npt = normal play time) C: PAUSE rtsp://audio.example.com/xena/audio.en/lofi RTSP/1.0 Session: 4231 Range: npt=37 C: TEARDOWN rtsp://audio.example.com/xena/audio.en/lofi RTSP/1.0 Session: 4231 S: 200 3 OK 20

Example 2: RTMP - Real-Time Messaging Protocol Proprietary Adobe protocol Runs over TCP Manages audio, video, and other Multiplex multiple streams over TCP connection 21

Flash Streaming Server Flash Streaming Server communicates with its clients using the Adobe patented RTMP over TCP, which manages a two-way connection, allowing the server to send and receive,video, audio and data between client and server Typically used with FLV encoded files

RTMP- Variations of RTMP Standard,unencrypted RTMP. The default port is 1935. RTMPT- RTMP tunneled over HTTP The default port is 80 RTMPS- RTMP sent over an SSL. SSL enables secure TCP/IP connections. The default port is 443 RTMPE- Enhanced and encrypted version of RTMP. RTMPTE- Encrypts the communication channel, tunneling over HTTP

Drawbacks of RTSP, RTMP Web downloads are typically cheaper than streaming services offered by CDNs and hosting providers Streaming often blocked by routers UDP itself often blocked by firewalls HTTP delivery can use ordinary proxies and caches Conclusion: rather than adapt Internet to streaming, adapt media delivery to the Internet 24

Bandwidth is never constant Even preselecting bitrates will lead to rebuffering problems as bandwidth fluctuates over the duration of the vide, especially for long form content Ideally we would like to switch bitrates constantly to always give the user the highest quality they can sustain at any point in time. Buffering

Bandwidth is never constant Even preselecting bitrates will lead to rebuffering problems as bandwidth fluctuates over the duration of the vide, especially for long form content Ideally we would like to switch bitrates constantly to always give the user the highest quality they can sustain at any point in time. Adaptive delivery 1500kbps 1000kbps 500kbps Buffering

3rd Generation: HTTP Streaming Other terms for similar concepts: Adaptive Streaming, Smooth Streaming, HTTP Chunking Client-centric architecture with stateful client and stateless server Standard server: Web servers Standard Protocol: HTTP Session state and logic maintained at server Video is broken into multiple chunks Chunks begin with keyframe so independent of other chunks A series of HTTP progressive downloads of chunks Playing chunks in sequence gives seamless video

Adaptive Bit Rate with HTTP Streaming Encode video at different levels of quality/bandwidth Client can adapt by requesting different sized chunks Chunks of different bit rates must be synchronized All encodings have the same chunk boundaries and all chunks start with keyframes, so you can make smooth splices to chunks of higher or lower bit rates

How does segmentation work? 1 Incoming video 3 into multiple short blocks. Each block holds the same section of video, encoded at a different size and bitrate. 500 kbps 1000 kbps 2 2000 kbps is split by an encoder

How does adaptive delivery work? The segmented video is stored on a server, along with a text file which describes the names of each segment. This text file is called a manifest. A player downloads the manifest and then begins requesting individual segments of video. It makes its choice based on bandwidth conditions, grabbing the best quality it can at the time. SERVER CLOUD PLAYER

Media Source Extension the future of browser video A W3C standard - https://dvcs.w3.org/hg/html-media/rawfile/tip/media-source/media-source.html Supported today by Chrome, IE11 and Opera MSE allows JavaScript to - fetch and parse a manifest - retrieve video, audio and text segments using XHR - feed them to the <video> tag to be decoded and rendered. MSE gives developers incredible control over playback, buffering and adaption logic of in-page video.

Pareto principle in video popularity Also known as the 80-20 (or 90-10) rule [http://en.wikipedia.org/wiki/pareto_principle] Associated with highly skewed distributions Caused by positive-feedback effects ( the rich get richer ) Does non-ugc content (say movies from NetFlix) follow a similar pattern? A practical implication of this skewness: caching applies!

Caching works Let s build CDNs Take advantage of locality and relativity A practical implication of this skewness: caching works! Akamai You tube Netflix based on Amazon infrastracture

Adaptive HTTP Streaming System (Protocol) Server Can be standard web server Media segment can be prepared in-line or offline Client Sends series of HTTP GET segment requests and receives segments Performs rate adaptation before sending a new GET segment request

Advantages of HTTP Streaming Easy to deploy: it's just HTTP, work with existing caches/proxies/cdn/firewall Fast startup by downloading lowest quality/smallest chunk Bitrate switching is seamless Many small files Small with respect to the movie size Large with respect to TCP 5-10 seconds of 1Mbps 3Mbps 0.5MB 4MB per chunk 35

Which adaptive streaming formats exist today? HTTP Live Streaming (HLS) - Controlled by Apple - Known as HLS, it is supported well by ios, Safari and half-heartedly by Android. HTTP Dynamic Streaming (HDS) - Controlled by Adobe - Known as HDS, it is played back only by Flash clients with custom apps. Smooth Streaming (SMOOTH) - Controlled by Microsoft - Played back via Silverlight clients, Xbox, MPEG-DASH (DASH) - An ISO standard - Playback via MSE in browsers, native apps on ios, Android and WInOS.

Simple HLS example <video id="player" controls poster="sintel.jpg" preload="none" style="width:480px;height:204px"> <source src="http://multiplatformf.akamaihd.net/i/multi/april11/sintel/sintelhd_,512x288_450_b,640x360_700_b,768x432_1000_b,.mp4. csmil/master.m3u8" type='application/x-mpegurl' /> <source src="http://akamaicorppmd.edgesuite.net/flashvod/video/mp4/sintel.mp4" type='video/mp4'/> <source src="http://akamaicorppmd.edgesuite.net/flashvod/video/mp4/sintel.webm" type='video/webm' /> </video>

Example of HLS Meta Data

RTMP Other Video Streaming Approaches

Adaptive video streaming over HTTP Several players support it Several commercial content providers use it Microsoft Smooth Streaming Player Netflix Player Adobe OSMF/Zeri Player Apple Player

HTTP Adaptive Video Streaming

HTTP Adaptive Video Streaming: Manifest File and Fragments

Smooth Streaming Player

Smooth Streaming Player Sample HTTP Request: GET /mediadl/iisnet/smoothmedia/experience/bigbuckbunny7 20p.ism/QualityLevels(2040000)/Fragments(video=400000 000) HTTP/1.1

Adobe OSMF/Zeri Player

Adobe OSMF Player Sample HTTP Request: GET /content/inouteditmbr/inoutedit\_h264\_3000seg1-frag5 HTTP/1.1

What Makes Netflix Interesting? Commercial, feature-length movies and TV shows and not free; subscription-based Nonetheless, Netflix is huge! 25 million subscribers and ~20,000 titles (and growing) consumes 30% of peak-time downstream bandwidth in North America A prime example of cloud-sourced architecture Maintains only a small in-house facility for key functions e.g., subscriber management (account creation, payment, ) Majority of functions are sourced to Amazon cloud (EC2/S3) user authentication, video search, video storage, DNS service is sourced to UltraDNS Leverage multiple CDNs (content-distribution networks) for video delivery Akamai, Level 3 and Limelight

Netflix Architecture Netflix has its own data center for certain crucial operations (e.g., user registration, billing, ) Most web-based user-video interaction, computation/storage operations are cloud-sourced to Amazon AWS Video delivery is out/cloud-sourced to 3 CDNs Users need to use MS Silverlight player for video streaming

Netflix Videos and Video Chunks Netflix uses a numeric ID to identify each movie IDs are variable length (6-8 digits): 213530, 1001192, 70221086 video IDs do not seem to be evenly distributed in the ID space these video IDs are not used in playback operations Each movie is encoded in multiple quality levels, each is identified by a numeric ID (9 digits) various numeric IDs associated with the same movie appear to have no obvious relations 49

Netflix Player

Netflix Player Sample HTTP Request: GET /sa2/946/1876632946.wmv/range/2212059-2252058?token=1283923056\_d6f6112068075f1fb6 0cc48eab59ea55\&random=1799513140 HTTP/1.1

Netflix Videos and Video Chunks Videos are divided in chunks (of roughly 4 secs), specified using (byte) range/xxx-xxx? in the URL path: Limelight: http://netflix-094.vo.llnwd.net/s/stor3/384/534975384.ismv/range/0-57689?p=58&e=1311456547&h=2caca6fb4cc2c522e657006cf69d4ace Akamai: http://netflix094.as.nflximg.com.edgesuite.net/sa53/384/534975384.ismv/ range/0-57689?token=1311456547_411862e41a33dc93ee71e2e3b3fd8534 Level3: http://nflx.i.ad483241.x.lcdn.nflximg.com/384/534975384.ismv/range/0-57689?etime=20110723212907&moviehash=094&encoded=06847414df 0656e697cbd Netflix uses a version of (MPEG-)DASH for video streaming 52

Terms and Definitions of Adaptive HTTP Streaming Need Media Presentation Description (MDP) which provides metadata For requesting (GET request) media segments For rate adaptation purpose Segment which may include media data or metadata to decode Use DASH an example in the few slides

Example: DASH Client Thomas Stockhammer, Qualcomm, DASH Design Principles and Standards, Presentation at MMSys 2011

Meta Data DASH uses MPD (Media Presentation Descriptor) and Index Information as metadata for DASH Access Client Initialization and Media Segments for Media Engine Reuse of existing container format Source: Stockhammer, Qualcomm, DASH Design Principles and Standards, Presentation at MMSys 2011

Media Presentation Data Model MDP - description of accessible segments and corresponding timing Source: Stockhammer, Qualcomm, DASH Design Principles and Standards, Presentation at MMSys 2011