The Frozen Mountain irtc White Paper Series This white paper is the second in a series on Internet Based Real Time Communications (irtc) written by Frozen Mountain Software s CTO Anton Venema. The complete series consists of: irtc: Introduction to Internet Real Time Communications irtc: The Role of WebRTC in Internet Real Time Communications irtc: The Role of Signaling in irtc irtc: Real Time Messaging for Chat and More irtc: Selective Forwarding irtc: Audio/Video Mixing irtc: Telephony VOIP and PSTN irtc: Content Broadcasting As the primary architect of Frozen Mountain s WebSync, IceLink and LiveSwitch products, Anton is uniquely qualified as a WebRTC expert. Internet Based Real Time Communications (irtc) is much more than just WebRTC. It s an overall architecture that includes both streaming and non-streaming data transmission defining a complete real-time communications solution.
Non-streaming irtc applications send bursts of data over guaranteed delivery networks. Common use cases include text chat, diagnostic data transmission, browser synchronization, and audio/video conference signaling. Data is typically sent through a central server to which all endpoints connect. Streaming irtc encompasses all applications where high volume data is sent, typically over unreliable networks. Common use cases include audio calling, video chat, sensor data collection, and live media broadcast. Data can be streamed directly between endpoints (peer-to-peer P2P) or through a central server that either forwards packet data (selective forwarding SFU) or mixes decoded media (multipoint control MCU). Want to learn more about irtc? Read the entire series on our website at frozenmountain.com
3 The Role of WebRTC in Internet Real Time Communications (irtc) Part 1 The Basics of WebRTC Introduction Nobody would argue that WebRTC has had a significant impact on industries that depend on real-time communications. From education to healthcare to telecommunications, WebRTC jumped into the forefront of business development plans as a cost-effective and user-friendly option for video streaming. In a world where plugins were viewed increasingly as annoying and intrusive security risks, web developers suddenly had a brand new plugin-free tool in their belt allowing them to create immersive applications that brought people together in new and exciting ways. It sounds great (and it is), but there is a reason that it took so long for the web development community to agree on a standard for peer-to-peer media streaming. It s hard - very hard. Streaming live video from a camera to another device may sound simple, but there are many moving parts in a real-time peer-driven media application. Networks are messy and complicated to navigate. Codecs are often proprietary and incompatible. Each application s requirements are as unique as the next. There is a continual drive to reduce connectivity time and improve performance. It has been years since WebRTC first surfaced, and the web community is still not yet in agreement on all points. In this series of white papers, we will examine WebRTC and ORTC in detail in the hopes of coming to a better understanding of the technology itself. We will look at the technologies used, the design decisions made, the impact on real-world applications, and how it has and is evolving into new areas. Peer Connections Before getting into WebRTC itself, let s take a step back and look at peer communication in general - what it is and why it s important. The What To start, what makes peer-to-peer communication different from traditional client-to-server communication?
4 Say we are running a web server. The server probably has a publicly accessible IP address and is listening on predefined TCP ports 80 and 443 for regular and TLS-secured traffic, respectively. Any device connected to the Internet can open a TCP connection to one of those ports and send an HTTP request. The server will accept the connection, receive the HTTP request, generate an appropriate HTTP response and send it back over the TCP connection to the waiting client, who can in turn parse the response and do something with it, like display the response content in a web browser. This basic client-to-server behaviour is what drives the world wide web. But what makes it client-to-server? Can t the client and server be viewed as peers? Say we want to create a connection between two mobile phones so they can take part in an audio call. Both participants in the call are equal in the sense that they are equally privileged. Phone #1 can initiate a connection to phone #2 just as easily as phone #2 can initiate a connection to phone #1. Neither one is a client or server, so we refer to them as peers. In the web server example, the client and server are not equally privileged. The client can initiate a connection to the server, but the server cannot initiate a connection to the client, so we don t refer to this as peer communication. But there are exceptions to every rule. Say we want to run a connection between two mobile phones, but we want to record the conversation as well. There are a few options available to us: 1. Each client records their own audio/video and uploads it to a server later for mixing/post-processing. 2. A third, silent, headless peer joins the call. This peer doesn t contribute any new media to the call (hence silent) and doesn t have any user interface associated with it (hence headless), but runs on a server somewhere and simply writes the incoming media from the other two participants to disk. 3. Each client connects to a selective forwarding unit (SFU) or multipoint control unit (MCU) rather than directly to each other. The SFU/MCU forwards the received media from one client to the other client, writing it to disk as it passes through. For now, let s focus on option #2. Option #1 is valid, but not particularly useful for our discussion here.
5 Option #3 is a more advanced setup and will be discussed in a future publication. The headless participant from option #2 would typically run on a server somewhere. Topologically, it is equally privileged in that it is just another participant in the call, albeit with different stream directions (receive-only instead of send-receive). It may have a different role in joining the call, but architecturally, the process for connecting each pair of peers is the same. So would this be considered client-to-server or peer-to-peer communication? Does running on a server automatically make it client-to-server? Let s go with no, at least for now. For our purposes, we would still consider this peer-to-peer. Consider that the headless participant could run on a mobile phone just as easily as it could run on a dedicated blade in a data center somewhere. It s participation (or lack of participation) in no way affects the call itself. The Why So why go peer-to-peer at all? Why not just stream data up to and down from a server? In many cases, this may actually be preferable, particularly when we re dealing with large number of users, but there are also several advantages to sending data directly between peers and bypassing the server. First, the pros: 1. Lower latency. This is one of the strongest technical arguments for peer-to-peer connections. All things considered equal, sending media directly between peers can cut latency down significantly. Whether or not this matters depends a lot on your use case. A two-way video call, for example, will notice the slight difference between 250ms and 500ms of latency, whereas a one-way video broadcast could perhaps tolerate up to 10s of latency without having a noticeable impact to the user experience. 2. Lower costs. Bandwidth may be relatively cheap, but it can still add up in a hurry, especially when you have a few hundred or a few thousand subscribers pushing several megabits per second of data per video call up to your server. Running the media plane between peers translates to a big reduction in server-side resources. There are cons as well: 1. Complexity. Client-to-server topologies are easy to design and test. Peer-to-peer topologies are inherently more complicated and require specialized STUN/TURN servers to negotiate firewalls and form connections. Software updates require special considerations for backwards compatibility and rollouts since neither side of the connection is under direct management. 2. Less control. By allowing the media to flow directly between peers, we inherently give up a piece
6 of control. The media sent between peers can t be inspected, controlled, or recorded without client agreement. For heavily regulated industries like healthcare and telecom, an intermediary server may be required to satisfy requirements like lawful interception on behalf of government agencies. Peer connections don t replace client-server connections, nor are they intended to. They do, however, provide a powerful tool to build applications that meet the demands for low-latency and cost-effective vreal-time streaming. Until recently, this tool was not available in a web browser without the use of third-party plugins like Flash, which used proprietary standards and required the use of expensive server software. By creating an open WebRTC standard and implementing it in their respective web browsers, Google and Mozilla have quietly ushered in a new era of web-based applications. Media Streams WebRTC is much more than just networking, though. Connections by themselves have limited usefulness if we don t have a stream of audio or video to send or capacity to receive incoming media. Inside a web browser, HTML5 gives us the ability to capture local audio and video from a connected microphone, camera, or other media device registered with the operating system. Both WebRTC and ORTC allow these local media streams to be attached to a peer connection for the purposes of streaming their contents out to remote peers, as well as getting a media stream to handle the incoming audio and video from a remote peer. The video tracks of these media streams can be configured to display in the page, while the audio tracks can be directed to play to attached audio output devices. Outside the browser, we have a lot more flexibility and can capture media from just about anywhere, including files, other network streams, or even programmatically generated tones and images. We can also render media to disk, redirect to other network endpoints, or even process the live stream in ways limited only by our imagination - speech recognition, motion tracking, sentiment analysis, etc. Codecs Before sending the media over a peer connection, it has to be compressed. Raw audio and video is simply too large to send efficiently in our current Internet infrastructure. Likewise, after receiving media over a peer connection, it has to be decompressed. A media codec (coder-decoder) does exactly this.
7 WebRTC has mandated three audio codecs and two video codecs: 1. Audio - PCMU (G.711µ) running at 8,000Hz with a single channel (mono). 2. Audio - PCMA (G.711a) running at 8,000Hz with a single channel (mono). 3. Audio - Opus running at 48,000Hz with two channels (stereo). 4. Video - VP8. 5. Video - H.264/AVC using Constrained Baseline Profile Level 1.2. Video codecs in particular are a contentious issue, with years of discussion leading up to the decision to require both VP8 and H.264. VP8 was originally the only mandatory video codec, in large part due to its relatively clean license from Google allowing royalty-free commercial use. H.264, on the other hand, is heavily patent-encumbered and requires royalty payments to MPEG-LA, which goes against the spirit of the open web. H.264 has been the dominant standard for years, though, and as such is ubiquitous in almost every modern computing device. In the end, it was determined that it was better to find a way to make H.264 free to use rather than have it lose its place in the future world of WebRTC. Cisco currently provides pre-compiled binaries for OpenH264, a software-based implementation of H.264. These binaries can be downloaded by applications at runtime and used, free of charge, until Cisco decides otherwise. For ios-based devices that prohibit dynamic linking, Apple has opened up access to the hardware H.264 encoder and decoder through the new VideoToolbox API in ios 8. Future media codecs like VP9 and H.265 could be added at some point in the future, but for now are not mandatory. Signaling The last part of WebRTC, which isn t actually part of WebRTC (more on that later), is signaling. Signaling is what allows a device to initiate a peer connection to another device. Since both sides in a peer connection typically connect to the Internet from behind a firewall or router, it is necessary for each peer to connect to a common signaling server that allows them to talk to each other, at least a little bit, prior to the establishment of a peer-to-peer connection. We will go into this in detail in our next white paper.
8 Wrap-Up We hope that this has helped you understand WebRTC and peer-based communications a bit better. As a final note and plug for our products, consider using IceLink and WebSync for your WebRTC-based applications. With a consistent, powerful API and support for popular development frameworks/platforms like.net, Java, macos, ios, Android, Windows Phone, Xamarin, Unity, and of course JavaScript, IceLink makes building WebRTC- and ORTC-compatible applications a breeze on every major platform. It s powerful media engine and codec-agnostic API allows you to do just about anything imaginable with your audio and video data. More than just a signaling system, WebSync s publish/subscribe architecture lets you deliver real-time text and binary messages to your entire client base quickly and efficiently. A number of high-performance scaling options for your servers let you create a cluster that can auto-scale simply and effectively. Stay tuned as we dive into more WebRTC fundamentals in our next publication!