WebRTC 1.0 Real-Time Communications in the Browser

Size: px

Start display at page:

Download "WebRTC 1.0 Real-Time Communications in the Browser"

June Arnold
5 years ago
Views:

1 WebRTC 1.0 Real-Time Communications in the Browser Huib Kleinhout Product Manager, Google

4 2011

5 2018

6 >1.8B Weekly Chrome audio/video minutes, 3X from last year

7 >1300 WebRTC-based companies and projects

8 >80% Of all installed browsers have now WebRTC built-in

9 >10k Downloads of our AAR lib last 30 days

10 >6k+ Downloads of our Cocoapod last 30 days

11 WebRTC in YouTube Live WebRTC in YouTube Live Works in browser and mobile Opus + H.264 Replaces RTMP Decreased buffer rate Increased quality

12 WebRTC 1.0 Specs Well documented Compliance Tests Reliability Consistent across browsers Can be fully tested Works every time

13 Completing version 1.0 Version 0.9 Version 1.0

14 WebRTC pipeline and standards Javascript client processing coding session transport processing decoding session transport capture Javascript client STANDARDS playback MediaCapture and Streams WebRTC 1.0 API Audio & Video codecs VP8, H264, OPUS Session management Data channel Bandwidth estimation JSEP, SDP, FEC... Packetisation Security NAT transversal STUN,TURN, ICE, RTP, SCTP, DTLS...

15 Standardization of WebRTC 1.0 W3C IETF WebRTC 1.0 CR JSEP ~RFC Media Capture and Streams CR Data Channel ~RFC Identifiers for WebRTC's Statistics CR RTP Usage ~RFC Transports ~RFC Audio/Video codecs RFC Requirements RFC

16 Standards Compliance

17 Making Chrome fully compliant with the current standards WebRTC SDP based WebRTC 1.0 SDP based with some object API s ORTC Object based API s

18 SDP Munging a=fmtp:108 a=rtcp-fb:100 nack pli v=0 level-asymmetry-allowed=1;packetization-mode=1 a=fmtp:100 o= IN IP ;profile-level-id=4d0033 level-asymmetry-allowed=1;packetization-mode=1 s=a=rtpmap:109 rtx/90000 ;profile-level-id=42001f t=0 0 a=fmtp:109 apt=108 a=rtpmap:101 rtx/90000 a=group:bundle audio video data a=rtpmap:124 H264/90000 a=fmtp:101 apt=100 a=msid-semantic: WMS a=rtcp-fb:124 goog-remb a=rtpmap:102 H264/90000 m=audio 9 UDP/TLS/RTP/SAVPF a=rtcp-fb: transport-cc a=rtcp-fb:102 goog-remb c=in IP a=rtcp-fb:124 ccm fir a=rtcp-fb:102 transport-cc a=rtcp:9 IN IP a=rtcp-fb:124 nack a=rtcp-fb:102 ccm fir a=ice-ufrag:nkpr a=rtcp-fb:124 nack pli a=rtcp-fb:102 nack a=ice-pwd:ywke3pgpkiugguj4limufbcs a=fmtp:124 a=rtcp-fb:102 nack pli a=ice-options:trickle level-asymmetry-allowed=1;packetization-mode=1 a=fmtp:102 a=fingerprint:sha-256 ;profile-level-id= level-asymmetry-allowed=1;packetization-mode=0 F7:EA:F9:45:48:A6:F8:2D:DD:BF:26:E0:47:A0:A9:57:00:D6:27:A9:BA:FE:CD:C0:10:0E:DB:BA:05:92:8E:01 a=rtpmap:120 rtx/90000 ;profile-level-id=42001f a=setup:active a=fmtp:120 apt=124 a=rtpmap:122 rtx/90000 a=mid:audio a=rtpmap:123 red/90000 a=fmtp:122 apt=102 a=extmap:1 urn:ietf:params:rtp-hdrext:ssrc-audio-level a=rtpmap:119 rtx/90000 a=rtpmap:127 H264/90000 a=recvonly a=fmtp:119 apt=123 a=rtcp-fb:127 goog-remb a=rtcp-mux a=rtpmap:114 ulpfec/90000 a=rtcp-fb:127 transport-cc a=rtpmap:111 opus/48000/2 m=application 9 DTLS/SCTP 5000 a=rtcp-fb:127 ccm fir a=rtcp-fb:111 transport-cc c=in IP a=rtcp-fb:127 nack a=fmtp:111 minptime=10;useinbandfec=1m=video 9 b=as:30 UDP/TLS/RTP/SAVPF a=rtcp-fb:127 nack pli a=ice-ufrag:nkpr a=fmtp:127 c=in IP a=ice-pwd:ywke3pgpkiugguj4limufbcs level-asymmetry-allowed=1;packetization-mode=1 a=ice-options:trickle ;profile-level-id=42e01f SDP API

19 RTC Transceivers and Unified Plan in Chrome Interface Version RTCRtpReceiver and RTCRtpContributingSource support 59 RTCRtpSender (addtrack) 64 RTCRtpSender.replaceTrack 65 RTCRtpReceiver 65 Unified Plan SDP (optional) 68 RTCRtpTransceiver 69 RTCRtpEncodingParameters 70 sendencodings 71 Unified Plan default 72 (feb 2019) Current Chrome version: 70

264 SRTP NetEQ Video jitter buffer Multiplexing Echo cancellation, noise suppression, gain control

20 Apps API W3C Web APIs ios Objective-C APIs Android Java APIs WebRTC C++ API (PeerConnection) Session management / abstraction WebRTC library Voice Engine Video Engine Transport Opus, ISAC, G.711 VP8, VP9, H.264 SRTP NetEQ Video jitter buffer Multiplexing Echo cancellation, noise suppression, gain control Bandwidth Estimation, error correction P2P STUN/TURN/ICE Audio capturing/rendering Video capturing/rendering Network I/O

21 Browser support Chrome Firefox Edge Safari Library webrtc.org webrtc.org proprietary webrtc.org Version current 1-6 months behind JS layer Follows C++ layer Custom 1-8 months behind ORTC & WebRTC Follows C++ layer

22 Testing

23 Testing WebRTC - wpt.fyi Need more green! Test automation Spec compliance work

24 Web Platform Test results (Oct 2018) Chrome Firefox Edge Safari Media Capture 83% 74% 12% 80% Peer Connection 44% 55% 16% 45%

25 Testing WebRTC - Interoperability Web Platform Tests javascript javascript javascript SRTP/SCTP STUN/TURN HTTP KITE

26 Testing WebRTC - KITE W eb Dr iv er Selenium Grid KITE dashboard AppRTC tests Hardware, VMs, hosted VMs... github.com/webrtc/kite

27 Reliability

Reliability in Chrome Common reliability

28 Reliability in Chrome Common reliability problems Signals Incorrect device selection Device API s Muted, volume problems Exclusive locks on devices Audio driver instability Trackend Forwarding problems Muted Name/label/HID Events Metrics Audio energy

29 Reliability in Chrome Issue Status Version Audio glitches on Linux, CrOS Fix in Chrome POSIX layer 60 No audio from microphone on Mac OS X Workaround for Core Audio issue 63 Long delays when screen sharing Rework of pacing algorithm 63 Audio echo New AEC3 algorithm 69 Crashes in camera subsystem in OS Chrome Video Process 62 Crashes in audio subsystem in OS Chrome Audio Process 70 Audio driver hangs Hang detection & restart 71 Current Chrome version: 70

30 No mic audio on Mac OS X Audio Capture failure rate on Mac OS X {TLDR} 8.5% Deep deep investigation of MacOS CoreAudio behavior revealed that sleeping and waking the computer can cause a race condition that results in the mic being disabled. A fix for this issue was added in Chrome M63 0.4% M63

31 AEC3 Redesigned WebRTC echo canceller solution for desktop platforms Addresses fundamental existing AEC issues Experimentation with built-in AEC in Windows and Mac

32 Screen sharing latency improvements Interframe spacing on 200kbps link and 5% packet loss, 5fps

33 Chrome Audio Capture Process Fix no audio caused by system layer Solution: Audio Service Runs platform audio in a separate process Auto-restart on failure Renderer Renderer Renderer IPC Recover from audio crashes IPC IPC Problem: Browser Platform audio

34 Chrome Video Capture Process Video capture moved to separate process Driver crashes do not bring down the entire browser. Process/capture can now be restarted in JS Simplifies IPC, data ownership and sharing between processes. Released in M62

35 Coming soon

36 VP9 SVC VP9 provides ~35% better visual quality than VP8 at equivalent bitrate HD at 700kbps up/down

37 Screen Capture Different API implementations Convergence towards extension-less getdisplaymedia() Browser Current getdisplaymedia() Firefox getusermedia() Under development Edge getdisplaymedia() Supported Chrome desktopcapture/tabcapture (extension API) Under development Safari none Under development

38 WebRTC 1.0 High quality real time communication in the browser 6+ years of engineering Fully open source Open and free to use standards Implemented in all major browsers

39 WebRTC + Browser tech = fun! WebRTC + WebGL = AR, VR, shader based video processing WebRTC + TensorFlow.js = AI in communications WebRTC + Screen capture = Game streaming WebRTC + Canvas capture = Video Mixing WebRTC + PWA = Installable Web Apps + cross browser / instant updates / fully sandboxed / open standards!

40 Thank

The Frozen Mountain irtc White Paper Series

The Frozen Mountain irtc White Paper Series This white paper is the third in a series on Internet Based Real Time Communications (irtc) written by Frozen Mountain Software s CTO Anton Venema. The complete