Performance and Scalability of WebRTC ELEC-E7320 Internet Protocols assignment 2 presentation 22 Feb 2017 Group O: Pete Lyly Antti Majakivi Olli-Mikko Ojamies Bhavya Omkarappa
Presentation outline Introduction to WebRTC performance and scalability evaluation Presentation of the analysis methods Performance analysis Scalability analysis Comparison to SIP Conclusions
Introduction What is the impact to the user experience, when connection quality changes How can we evaluate scalability limits We were adding number of sessions video: true, WebRTC permits to either minimise/maximise the video quality audio: true } Connection speed requirements depend on the selected quality level. This quality level is negotiated with Session Description Protocol SDP. { We were using standard settings for getusermedia First normal connection, connection was weakened with different parameters Additional protocols are coming soon By reducing the quality, it is possible to add more participants to conference (CPU is limiting) What are the minimum requirements?
Media processing system diagram RTCP receiver report 1/s { video: true, var vgaconstraints = { video: { mandatory: { maxwidth: 640, maxheight: 360}}} audio: true } var hdconstraints = { video: { mandatory: { minwidth: 1280, minheight: 720}}}
Analysis methods: test environment Two Linux laptop computers One with real web camera, other with virtual camera with continuous video Wireless connection to same WLAN (aalto open) Sharing a room in simple WebRTC web service (https://simplewebrtc.com/demo.html) Why not virtual environment and mininet? Qualitative analysis in addition to quantitative with real video source Important data in application layer, collected from browser Traffic control possible also in native Linux system
Analysis methods: test scenarios Performance tests: two laptops in a WebRTC session 1. 2. 3. 4. 5. 6. Initial test with unmodified environment 250ms added delay 500ms added delay 128kb/s limit of uplink speed 10% packet loss A superset of previous: 250ms delay, 128kb/s limit and 10% loss
Analysis methods: test scenarios Scalability test: increase number of clients in a session - various clients (laptops and smartphones) - data collection from the laptop first joining the video conference Similar procedure for every performance test scenario - first 60 seconds without any traffic control - enable traffic control for first client and wait 60 seconds - enable traffic control for second client and wait 60 seconds -> capture traces always 180s long and interesting data in 60s intervals - in scalability test client added once in every 60 seconds
Analysis methods: traffic control tools tc: traffic control in Linux command line - Enables kernel level traffic control Controls all outbound traffic of adapter Simple to use and test tc configuration example: # % # % add delay of 250ms sudo tc qdisc add dev wlp2s0 root netem delay 250ms loss 10% remove traffic control parameters sudo tc qdisc del dev wlp2s0 root netem
Analysis methods: data collecting WebRTC has integrated reporting tools, but how to collect and analyze the data? Built-in reporting functionality in Chrome browser chrome://webrtc-internals/ - statistics and graphs, but limited usability of static graphs - data exporting functionality (huge JSON formatted text files) Open source online tool for importing webrtc-internals dumps available - http://fippo.github.io/webrtc-dump-importer/ - good presentation of data with dynamic graphs
chrome://webrtc-internals vs. dump importer tool
Performance analysis, test #1: initial test with unmodified environments Client A, bit rate (bps) Client A - Received video 854x480px at 24 FPS - Transmitted video at ~1.7Mbps Client B - Received video 640*480px at 30 FPS - Transmitted video at 2Mbps Both clients: - Low RTT (<10ms) - Very low packet loss
Performance analysis, test #2: 250ms delays Client A, RTT (ms) & transmitted bit rate (bps) - Upstream delays initiated at 60 sec and 120 sec points Transmitted bit rate drops temporarily Some packet loss during delay initiations
Performance analysis, test #2: 250ms delays Client B, received video resolution & frame rate - Small temporary frame rate drop, recovered rapidly No change in resolution Similar results for Client B -> Client A transmission
Performance analysis, test #3: 500ms delays Client A, RTT (ms), transmitted bit rate (bps) and Client B received frame rate - Pretty much similar results as in test #2, but stronger effects
Performance analysis, test #4: 128kb/s uplink Client A, transmitted bit rate (bps) Client B, received frame rate and resolution
Performance analysis, test #5: 10% packet loss Client B, packet loss - Transmitted bit rate goes down when packet loss happens Lower bit rate dropped frame rate from 30 to 5-10 FPS
Performance analysis, test #6: Delay, limit & loss Client A, transmitted bit rate (bps) Client B, received frame rate and resolution The combo was too much: connection was lost after both sides activated the delays, rate limits and packet losses
How UI displays the dropped connection
Scalability analysis Round-trip time RTT (ms) - RTT is increased when more clients join Packet loss - Increased amounts of packet loss when new clients join
Scalability analysis Transmitted bit rate to one client - Bit rate per client drops when more clients join Bandwidth & CPU are the most likely bottlenecks
Scalability analysis Received frame rate and resolution from one client - When bit rate goes down, frame rate suffers first Resolution lowered in roughly 20 second steps
SIP Performance Analysis
Call Flow for SIP Protocol with 600 calls in one client.
RTP Data streams during the initiation of the call and during the mid of the call.
RTP Stream Analysis
Comparison Of WebRTC and SIP Protocol Factors WebRTC SIP Protocol API with associated number of Protocols Protocol, not an API Media Transport SRTP, new RTP Profiles SRTP (o), RTP Media transport path/connection Same path with all media and control Separate : audio/video RTP vs RCTP Security Model User trusts browser but not web site User trusts device and service provider Audio Codecs G.711 and Opus Mandatory Optionally others Typically G.711, G.729, G.722 Video Codecs Undefined yet likely VP8 and/or H.264 Typically H.261, H.263, H.264
Conclusions Performance of the webrtc - we see that framerate, resolution are been changed using Session Description Protocol. Our measurements showed, existing negotiation process provides good capabilities to adjust the resolution and framerate based on the capabilities. W.r.t. Scalability, WebRTC runs within a browser, which means each peer-to-peer session needs to processed separately. This means in practise more than 10 simultaneous WebRTC sessions will consume all the CPU. SIP is lower level protocol and as we measured, even hundreds of SIP connections can performed by one client. Our measurements were demonstrating how WebRTC can handle connection speeds between 128kBit/s up to almost 2 MBit/s which means WebRTC can efficiently do adaptation to the available connection speed. We were able to identify several situations reasons why and when WebRTC changes communication. E.g. when packets were dropped 10%, we noticed how connection bandwidth was almost doubled, because WebRTC started to resend packets.
Questions?