RTP Protocol Transport of H.264 Video and AAC Audio

Table of Contents Using the RTP Protocol to Transport Video and Audio... 1 DMSDK Development Environment... 1 RTP A/V Streaming Client/Server Application Architecture... 1 RTP Protocol Transport of H.264 Video/AAC Audio... 2 RTP H.264 Video/AAC Audio Transport Timing... 4 RTP A/V Streaming Server Application Functional Diagram... 4 Additional Resources... 5 List of Figures Figure 1: DMSDK Development Context Diagram.... 1 Figure 2: RTP A/V Streaming Client/Server Applications.... 2 Figure 3: RTP H.264 Video Packet Structure.... 2 Figure 4: RTP AAC Audio Packet Structure.... 3 Figure 5: RTP Protocol H.264 Video/AAC Audio Transport Timing Diagram.... 4 Figure 6: RTP A/V Streaming Client/Server Application Functional Block Diagram.... 5 Document: AN-121115-1 Document Status: Released Revision: 1.0 Notice: The copyright in this document, which contains information of a proprietary nature, is vested in Cimarron Systems, LLC. The content of this document may not be used for purposes other than that for which it has be supplied and may not be reproduced, either wholly or in part, in any way whatsoever, nor may it be used by, or its content divulged to any other person whomsoever without prior written permission. For further information regarding this document, please contact Cimarron Systems, LLC at info@cimarronsystems.com. Copyright 2012 Cimarron Systems, LLC all rights reserved.!i

Using the RTP Protocol to Transport Video and Audio The present application note describes the use of the RTP Internet Protocol (IP) to simultaneously transport H.264 video and AAC audio bitstreams across the network. Specifically, the RTP A/V Streaming Client/ Server Applications available as part of the Cimarron Systems Digital Media Software Development Kits (DMSDK) implements many of the concepts outlined here. DMSDK Development Environment Figure 1 shows a context diagram of the typical development environment in which the DMSDK operates, including: the TM320DM36x DVEVM running a number of the DMSDK components, a Ubuntu Linux Host computer, an audio/video source, and a video display/audio output device. Figure 1: DMSDK Development Context Diagram. RTP A/V Streaming Client/Server Application Architecture As illustrated in Figure 2, RTP A/V Streaming Server and RTP A/V Streaming Client Applications include: a RTCP Transmitter User Agent; an A/V Encoder set for encoding video to H.264 and audio to AAC-LC; and a RTP Transmitter. The RTP A/V Streaming Client component includes: a RTP Receiver; a RTCP Receiver User Agent; and a MPEG-2 Transport Stream (TS) Encoder that multiplexes the encoded H.264 video and AAC-LC audio into a transport stream. Both the RTP A/V Streaming Server and RTP A/V Streaming Client Applications implement H.264 video and AAC audio streaming in accordance with IETF RFC 3550 RTP: A Transport Protocol for Real-Time Copyright 2012 Cimarron Systems, LLC all rights reserved.!1

Applications, RFC 6184 RTP Payload for H.264 Video, RFC 3550 RTP: A Transport Protocol for Real-Time Applications, and RFC 3551 RTP Profile for Audio and Video Conferences with Minimal. DMSDK RTP A/V Streaming Application Software Architecture DVEVM36x Ubuntu Linux Host RTP A/V Streaming Server RTP A/V Streaming Client A/V Capture H.264 Video Encoder AAC-LC Audio Encoder RTCP Tx User Agent RTP Tx TI Embedded Linux Kernel MPEG-2 TS Output / File Writer MPEG-2 TS Encoder RTCP Rx User Agent RTP Rx Ubuntu Linux Network: Ethernet, Wireless, etc. Figure 2: RTP A/V Streaming Client/Server Application Architecture. RTP Protocol Transport of H.264 Video/AAC Audio Figure 3 shows the structure of the RTP H.264 Video Packet constructed in accordance with IETF RFC 6184 RTP Payload for H.264 Video which implements a simple method to signal the start of each H.264 Network Abstraction Layer (NAL) Unit as well as its Raw Byte Sequence Payload (RBSP), i.e., the NAL Unit bitstream data bytes (as detailed below, the NAL Unit start signaling method appends a single byte to the end of the standard RTP Packet Header). RTP H.264 Video Packet Structure 0 1 2 3 4 5 6 7 0 1 2 3 4 5 6 7 0 1 2 3 4 5 6 7 0 1 2 3 4 5 6 7 V P X CC M payload type sequence number timestamp synchronization source identifier (SSRC) contributing source identifiers (CSRC) F NRI NAL unit type raw byte sequence payload (RBSP) optional RTP padding Figure 3: RTP H.264 Video Packet Structure. Figure 4 shows the structure of the RTP AAC Audio Packet constructed in accordance with IETF RFC 3550 RTP: A Transport Protocol for Real-Time Applications which implements carriage of the AAC audio Copyright 2012 Cimarron Systems, LLC all rights reserved.!2

! RTP Protocol Transport of H.264 Video and AAC Audio 1 2 bitstream via the common Audio Data Transport Stream (ADTS). RTP AAC Audio Packet Structure 0 1 2 3 4 5 6 7 0 1 2 3 4 5 6 7 0 1 2 3 4 5 6 7 0 1 2 3 4 5 6 7 V P X CC M payload type sequence number timestamp synchronization source identifier (SSRC) contributing source identifiers (CSRC) audio data transport stream (ADTS) optional RTP padding Figure 4: RTP AAC Audio Packet Structure. The syntax descriptions present below apply to both RTP H.264 Video and AAC Audio Packets:! V (2 bits): RTP Protocol Version, always equal to 2.! P (1 bit): If the Padding bit is set, the packet contains one or more additional padding octets at the end which are not part of the payload.! X (1 bit): If the Extension bit is set, the fixed header MUST be followed by exactly one Header Extension.! CC (4 bits): The CSRC Count contains the number of CSRC identifiers that follow the fixed header.! M (1 bit): The interpretation of the Marker is defined by a profile it is intended to allow significant events such as frame boundaries to be marked in the packet stream.! Payload Type (7 bits): This field identifies the format of the RTP payload and determines its interpretation by the application.! Sequence Number (16 bits): The Sequence Number increments by one for each RTP data packet sent and may be used by the receiver to detect packet loss and to restore packet sequence (the initial value of the sequence number should be random).! Timestamp (32 bits): The Timestamp reflects the sampling instant of the first octet in the RTP data packet. The sampling instant MUST be derived from a clock that increments monotonically and linearly in time to allow synchronization and jitter calculations.! SSRC (32 bits): The SSRC field identifies the Synchronization Source this identifier should be chosen randomly, with the intent that no two Synchronization Sources within the same RTP session will have the same SSRC Identifier.! CSRC (32 bits): The CSRC List identifies the contributing sources for the payload contained in this packet. The number of identifiers is given by the CC Field only 15 can be identified not used in the present implementation.! Padding (M bytes): Padding bytes added to the end of a packet in order to force an whole number of packet bytes. 1 2 See the CS365-TI DMSDK and/or CS368-TI Datasheets for details regarding Cimarron Systems DMSDK development environment, features, architecture, and capabilities. For an excellent tutorial concerning the application of RTP, please see the presentation RTP: Multimedia Streaming over IP, by Colin Perkins, USC Information Sciences Institute. Copyright 2012 Cimarron Systems, LLC all rights reserved.!3

The following syntax descriptions apply only to RTP H.264 Video Packets:! F (1 bit): A value of 0 indicates that the NAL Unit Type octet and payload should not contain bit errors or other syntax violations. A value of 1 indicates that the NAL Unit Type octet and payload may contain bit errors or other syntax violations.! NRI (2 bits): A value of 00 indicates that the content of the NAL Unit is not used to reconstruct reference pictures for inter picture prediction value greater than 00 indicate that the decoding of the NAL Unit is required to maintain the integrity of the reference pictures.! NAL Unit Type (5 bits): Interpretation of NAL Unit Type values 0-23 are the same as ITU-T H.264 recommendation see RFC 6184 Table 3 for the interpretation of values 24-29.! Raw Byte Sequence Payload (N bytes): A sequence of bytes constituting the H.264 video NAL bitstream. The following syntax descriptions apply only to RTP AAC Audio Packets:! Audio Data Transport Stream (1024 bytes): A sequence of bytes constituting the AAC audio bitstream. RTP H.264 Video/AAC Audio Transport Timing The timing diagram shown in Figure 5 illustrates the relationship between audio/video sample frame timing, packet transmission timing, and RTCP packet transmission/reception for the RTP A/V Client/Streaming Server Applications when set for H.264 (30 fps)/aac (32 ks/s). In the present example, the video timestamp clock is 90 khz while the audio timestamp is 32 khz, i.e., the audio sampling rate. RTP Transport Timing for H.264 (30 fps) and AAC (32 ks/s) 33 ms 1 15 30 3000 3015 3030 6000 6015 6030 90000 90015 90030 video_timestamp_clk (90 khz) video_rtp_timestamp_value 11 µs v_rtp_random_# + 0 v_rtp_random_# + 3000 v_rtp_random_# + 90000 video_rtp_tx 32 ms 1 5 10 1024 video_packet 0 video_packet 1 video_packet 29 1029 1034 2048 2053 2058 32768 32773 32778 audio_timestamp_clk (32 khz) 31.25 µs audio_rtp_timestamp_value audio_rtp_tx rtcp_tx rtcp_tx_packet 0 a_rtp_random_# + 0 a_rtp_random_# + 1024 a_rtp_random_# + 32768 1024 samples 1024 samples aac_packet 0 aac_packet 1 aac_packet 31 1024 samples rtcp_tx_packet m rtcp_rx rtcp_rx_packet 0 rtcp_rx_packet n Figure 5: RTP Protocol H.264 Video/AAC Audio Transport Timing Diagram. RTP A/V Streaming Server Application Functional Diagram Figure 6 shows a functional block diagram of the RTP A/V Streaming Server Application portion of the Copyright 2012 Cimarron Systems, LLC all rights reserved.!4

! RTP Protocol Transport of H.264 Video and AAC Audio software architecture illustrated in Figure 2. Note that the blue colored blocks represent the functional components that starts up/shuts down the application; reads the keyboard/mouse inputs; initializes the audio/video codecs; captures and encodes the audio/video frames; constructs and transmits the audio/video RTP packets; composes then transmits RTCP Session Messages; receives then interprets the RTCP QoS Messages; and, based upon QoS status, adjusts application performance parameters as required. RTP A/V Streaming Server Implementation Command Line main() process command line create threads pass control control() manage threads shutdown Legend: Request Response I/O Key / Mouse Input Thread Thread Thread Thread Scan Inputs capture() Capture Buffers video() audio() time() System Time Session Messages QoS Messages Request Codec Types Codec Type Data Encoded Video Encoded Audio u_interface() codecs() dv_rtp() da_rtp() rtcp_tx() rtcp_rx() Keyboard / Mouse Inputs UDP/IP Ethernet Tx UDP/IP Ethernet Tx TCP/IP Ethernet Tx TCP/IP Ethernet Rx Figure 6: RTP A/V Streaming Client/Server Application Functional Block Diagram. Additional Resources In addition to those already described, listed below are a number of resources that may be helpful: 1. RTP FAQ at Columbia University 2. RTP Tools at Columbia University 3. WireShark Capture for RTP For more information regarding this and other Cimarron Systems products, please contact us using the contact information below. Copyright 2012 Cimarron Systems, LLC all rights reserved.!5

Contact Information: Cimarron Systems, LLC Evergreen, Colorado telephone: (303) 674-9207 email: info@cimarronsystems.com www.cimarronsystems.com Revision History: Date Version Notes 11/15/2012 version 1.0 Initial version Copyright 2012 Cimarron Systems, LLC all rights reserved.!6