Technical White Paper for Huawei's Videoconferencing Network Diagnostics Tool (NLog)

Transcription

1 Technical White Paper for Huawei's Videoconferencing Network Diagnostics Tool (NLog) Huawei Technologies Co., Ltd. All rights reserved.

2 Contents 1 Videoconferencing Network Conditions Overall Description Detailed Function Description Providing Participants' Network Status Statistics Generating Active Conferences' Packet Loss Alarms Exporting Network Status Statistics on Conferences and Participants Archiving Participants' Network Status Statistics Principle in Packet Loss Statistics Collection Basic Characteristics of Packets in Videoconferencing Principle in Packet Loss Information Collection Packet Loss Ratio Calculation After Data Streams Are Forwarded Through MCUs Impact of Network Exceptions on Packet Loss Ratio Calculation Comparison Between Statistics Collected by NLog and Network Devices Comparison Between Statistics Collected by NLog and the Ping Command Summary Application Case Participants Are Frequently Disconnected Due to No Stream Reception in the XX Site Image Artifacts Due to Network Disorder in the XX Site Appendix: Acronyms and Abbreviations on NLog User Interface Huawei confidential. No spreading without permission. Page 2 of 15

3 1 Videoconferencing Network Conditions The quality of service (QoS) of the bearer network has a great impact on the videoconferencing system's performance. When packet loss occurs, the audio and video quality will suffer, and users may notice interruptions of transmission or artifacts. When the videoconferencing service quality deteriorates, system administrators must be able to quickly diagnose the packet loss issue and accordingly improve the QoS. Realizing the importance of a network diagnostics tool dedicated for the videoconferencing, Huawei developed NLog V2 for its videoconferencing products. NLog V2 is a powerful network diagnostics tool for improving system troubleshooting efficiency and reducing maintenance costs. 2 Overall Description With the flash technology, NLog V2 can display network conditions in real time, including the network packet loss (audio, video, and presentation packet loss; transmitting and sending streams in dual directions), network jitter, network delay, and cyclic redundancy check (CRC) for dedicated lines. Compared to the previous version of NLog, NLog V2 shows significant improvements in terms of user friendliness, capability, and statistics comprehensiveness. Using this tool, administrators can ascertain the current network status during conferences, or obtain the conference statistics after the conferences end to facilitate troubleshooting. Table 2-1 describes the difference between NLog V2 and its predecessor. Table 2-1 Comparison between two versions of NLog NLog V2 Previous Version of NLog Flash graphics Supported Not supported Video packet loss analysis Supported Supported Audio packet loss analysis Supported Supported Presentation packet loss analysis Supported Not supported Network jitter Supported Not supported Network delay Supported Not supported CRC for dedicated lines Supported Not supported Real-time refresh Supported Not supported I-frames statistics collection Supported Not supported Packet loss alarm for participants Supported Not supported NLog V2 is installed and used with the conference management software (ResourceManager, RM) as part of the software Huawei confidential. No spreading without permission. Page 3 of 15

4 3 Detailed Function Description 3.1 Providing Participants' Network Status Statistics During conferences, users can check the current network statistics on one or more participants. The statistics include: Audio packet loss ratio Video packet loss ratio Presentation packet loss ratio Network delay Network jitter I-frames sent to the Multipoint Control Unit (MCU) The statistics are presented in flash graphics in real time, enabling users to intuitively ascertain the current network status of participants. Figure 3-1 Page showing a participant's current network status statistics On the following pages, users can view historical network status statistics on one or more participants: Participant predefinition page Network statistics collection page Network statistics archive page On any page showing the network statistics, users can: Specify the time span between two adjacent sampling points along the x-axis Huawei confidential. No spreading without permission. Page 4 of 15

5 Select which type of statistics to display Drag the time axis 3.2 Generating Active Conferences' Packet Loss Alarms NLog V2 allows users to set the packet loss threshold, exceeding which alarms will be generated. To set the packet loss threshold, click Conference in progress in the left pane of the window, and right-click a conference in the right pane; choose System Settings > Network Loss Packet Alarm Threshold. In an active conference, if the packet loss ratio of one of the participants exceeds the predefined threshold, an alarm icon will be displayed for that participant on the active conference page. Users can then intuitively locate the participant whose network environment is poor. Figure 3-2 Setting the packet loss threshold 3.3 Exporting Network Status Statistics on Conferences and Participants NLog V2 allows users to export network status statistics on completed conferences for analysis. Content that can be exported Packet loss statistics on all participants in a conference Packet loss statistics on one of the participants in a conference Exported file format CSV format Pictures (when exporting packet loss statistics on a single participant) Huawei confidential. No spreading without permission. Page 5 of 15

6 3.4 Archiving Participants' Network Status Statistics To prevent the network statistics from getting oversized, users can configure archive settings for the collected statistics. When the size of statistics files exceeded the threshold, the system will automatically archive the files. Users can still view and export the archive. Figure 3-3 Viewing packet loss statistics in an archive 4 Principle in Packet Loss Statistics Collection This section describes NLog V2's working principle and how it obtains packet loss statistics. 4.1 Basic Characteristics of Packets in Videoconferencing Videoconferencing provides real-time communication. Ideally, for a natural communication experience, packets must be received and transmitted in real time at an equal interval. For example, to deliver 30 fps video without freezing or jumps, there must be one frame per 33 ms. Though the endpoint can cache and adjust the received packets before output (at the expense of increased delay), the network itself must be stable enough. The size and transmitting-receiving frequency of the IP packets in real-time transmission are dependent on the call bandwidth. Size of Video Packets The size of video packets depends on the actual call bandwidth. The following tables describe the general cases Huawei confidential. No spreading without permission. Page 6 of 15

7 Table 4-1 H.263 packetization Bandwidth lower than 512 kbit/s Bandwidth lower than 1 Mbit/s Bandwidth lower than 4 Mbit/s Bandwidth higher than 4 Mbit/s 500 bytes (about 128 packets at 512 kbit/s) 800 bytes (about 160 packets at 1 Mbit/s) 1000 bytes (about 480 packets at 4 Mbit/s) 1300 bytes (about 730 packets at 8 Mbit/s) Table 4-2 H.264 packetization Bandwidth lower than 768 kbit/s Bandwidth lower than 1 Mbit/s Bandwidth lower than 1.5 Mbit/s Bandwidth higher than 1.5 Mbit/s 500 bytes (about 190 packets at 768 kbit/s) 800 bytes (about 160 packets at 1 Mbit/s) 1000 bytes (about 190 packets at 1.5 Mbit/s) 1300 bytes (about 180 packets at 2 Mbit/s and 730 packets at 8 Mbit/s) Size of Audio Packets Currently, audio is packaged at an interval of 20 ms (50 packets every second). Table 4-3 describes the audio packet size. Table 4-3 Audio packet size G.711/G.722 (64 kbit/s) The audio payload is 160 bytes, and the IP header length is about 50 bytes, so the total length of the packet is about 220 bytes. G.728 (16 kbit/s) The audio payload is 40 bytes. The total length of the packet (audio payload plus IP header length) is about 100 bytes. AAC-LD (variable bit rate; 80 kbit/s is used as an example) The audio payload is 200 bytes. The total length of the packet (audio payload plus IP header length) is about 260 bytes Huawei confidential. No spreading without permission. Page 7 of 15

8 4.2 Principle in Packet Loss Information Collection Figure 4-1 Principle in packet loss information collection The MCU reports the packet loss ratios A% and B% of the endpoint A to NLog. The MCU receives the packet loss ratio A% reported by the endpoint using RTCP. The MCU collects the packet loss ratio B% during reception The endpoint collects the packet loss ratio A% during reception. A Audio and video packet loss statistics displayed on the NLog user interface are reported by MCUs. The MCUs collect audio and video packet loss statistics for each participant during packet reception and transmission, and periodically report them to SMC. NLog records the statistics and presents them on its user interface. In the statistics: Packet loss during reception: The MCU examines the Real-Time Transport Protocol (RTP) serial numbers of media streams received from endpoints to detect packet loss, and collects statistics on packet loss every second. Packet loss during transmission: The MCU obtains the packet loss ratio in the transmission direction from the peer endpoint through RTCP. The endpoint provides the statistics every 6s. The MCU collects packet loss statistics in terms of audio, video, and presentation packets, and reports them to NLog every 6s. NLog stores all statistics reported by the MCUs for users to view or query. 4.3 Packet Loss Ratio Calculation After Data Streams Are Forwarded Through MCUs During conferences, participant B may view the video of participant A. In such case, if the packet loss ratio from endpoint A to the MCU is 10%, and the packet loss ratio from the MCU to endpoint B is also 10%, then the MCU will report a 20% packet loss ratio for endpoint B during transmission, as indicated on the NLog user interface. MCUs directly forward video streams between endpoints, so the packet loss ratios before and after forwarding are added up. In the previous example, 10% is endpoint B's packet loss ratio, while 20% is its actual video packet loss. Figure 4-2 shows the process of forwarding video streams and collecting packet loss statistics Huawei confidential. No spreading without permission. Page 8 of 15

9 Figure 4-2 Process of forwarding video streams and collecting packet loss statistics NLOG Packet loss Packet loss ratio for packets received from endpoint A: 10% Packet loss ratio for packets transmitted to endpoint B: 20% Packet loss I H G F E C B A A MCU B Line 1: 10% packet loss Line 2: 10% packet loss Packet loss for total packets received by endpoint B from MCU: 20% The previous figure shows the data transmission process within 1s in a multipoint conference, where packet loss on line 1 is 10% and packet loss on line 2 is also 10%. 1. The MCU receives video streams from endpoint A, detects 10% packet loss, and reports it to NLog (for recording per second). 2. The MCU converts the data stream serial numbers (for example, from 1, 2, 3 to A, B, C ) and ensures that the serial numbers of data streams forwarded to endpoint B are continuous. 3. B receives video streams from the MCU, regards packets D and G as lost, and obtains the packet loss ratio of 20%. 4. B reports the packet loss ratio to the MCU. 5. The MCU reports the packet loss ratio of packets transmitted to endpoint B to NLog for storage and analysis. NOTE s report the average packet loss ratio to the MCU every 6s. If burst packet loss occurs on the transmission line (for example, 6% packet loss for 1s and no packet loss for the rest of the time), the average packet loss ratio reported by the endpoint and recorded by NLog is both 1%. 4.4 Impact of Network Exceptions on Packet Loss Ratio Calculation The network environment in which videoconferencing endpoints are located is complicated: MCUs and endpoints are located in different places; networks between endpoints may have multiple network devices (such as switches, routers, and firewalls) deployed; the network that bears the videoconferencing service may share the lines and bandwidth with other data services. As a result, media stream transmission between MCUs and endpoints may experience the following issues: Disorder: An ordered data segment is processed by multiple network devices during transmission. The data may pass through different routes or the network devices may introduce disorder when forwarding the data, so the segment probably arrives at the destination with disordered packets. Jitter: Normal source devices transmit data packets at equal intervals. Those data packets, because of network congestion or processing by network devices, may not arrive at the destination at equal intervals. Instead, multiple packets may arrive at the same time or await reception Huawei confidential. No spreading without permission. Page 9 of 15

10 Delay: Network delay occurs during data transmission across the network. The delay is supposed to be fixed, and does not affect the MCU's performance. However, due to network congestion or data processing, such as forwarding, the delay for some packets may be increased. To address the preceding issues, devices need to cache and adjust the received packets before using them. The cache and adjustment may also affect the actual packet loss ratio. The following are some examples. Packet loss statistics caused by disorder Figure 4-3 Packet loss statistics caused by disorder Nlog Packet loss ratio for packets received from endpoint A: 10%. A Four packets are disordered in line 1. MCU In a multipoint conference shown in the previous figure, four packets are disordered in line The MCU receives data streams from endpoint A. Assume that the MCU sorts the received packets every 60 ms (three packets). When the MCU receives packet 5, it starts to sort the packets. 3. After sorting packets 5, 6, and 7, the MCU does not find packet 4, regards that packet as lost, and starts sending packets 5, 6, and When packet 4 arrives at the MCU, it already expires and is discarded. 5. The MCU reports the packet loss ratio to the NLog. Audio and video quality affected by jitter 15 ms 1 1 receives Video 1 Audio 1 outputs Play audio 1 30 ms 2 2 receives Video 2 Audio 2 outputs Display video 2 Play audio 2 45 ms Network jitter No packets receives No output outputs Display video 2 Play audio 2 Display video 4 60 ms receives Video 3/4 Audio 3/4 outputs Play audio Huawei confidential. No spreading without permission. Page 10 of 15

11 1. At 45 ms, jitter occurs because of network congestion, and no data streams are transmitted to the endpoint. 2. No packets are available for the endpoint to decode and output, so the endpoint can only display the previously delivered audio and video. 3. At 60 ms, the delayed packets arrive at the same time. (If there are too many packets, the endpoint may fail to receive all of them, resulting in packet loss.) 4. The endpoint packets 3 and 4. Only one frame is allowed to be output at a time, so packet 3 amounts to a discarded packet, resulting in choppy audio and discontinuous video. Buffer exists on the endpoint to counteract jitter. However, if the buffer is too large, the delay will be increased and the conference quality will suffer. Audio and video quality affected by network delay 15 ms 1 Line delay sends 60 ms 30 ms 45 ms 60 ms sends sends sends Line delay 60 ms Line delay 80 ms Line delay 60 ms 75 ms 5 Line delay sends 60 ms 1 sorts outputs Display video 1 Play audio 1 2 sorts outputs Display video 2 Play audio sorts sorts outputs outputs Display video 3 Play audio 3 Display video 4 Play audio 4 As shown in the previous figure, line delay increases the total delay and affects the audio-visual interaction. 4.5 Comparison Between Statistics Collected by NLog and Network Devices Packet loss statistics collected by network devices concern only the packet loss that occurs when the devices forward the packets. These statistics reveal only the devices' packet forwarding condition. Packet loss statistics presented by NLog are collected based on the RTP packet serial numbers. These statistics reveal the end-to-end packet loss, involving all network paths and devices that the data packets pass through, as well as impact of network disorder, jitter, and delay. Therefore, the packet loss statistics collected by the network devices are available on NLog, while the packet loss statistics presented on NLog may not be available on the network devices. NLog provides a true presentation of the packet loss for each participant Huawei confidential. No spreading without permission. Page 11 of 15

12 4.6 Comparison Between Statistics Collected by NLog and the Ping Command In actual applications, this case may happen: As indicated by NLog, packet loss occurs when packets are transmitted from the MCU to the endpoint; however, if the user pings the MCU from a computer on the same LAN, no packet loss is detected. Such case exists because the differences described in Table 4-4. Table 4-4 Comparison between statistics collected by Nlog and ping command Packet transmission mechanism Response mechanism Application scenario NLog An MCU may execute hundreds of data packets every second. (For details about the number of packets, see section 4.1 "Basic Characteristics of Packets in Videoconferencing".) An MCU continuously transmits packets at equal intervals. The statistics collected by the MCUs and NLog reveal packet loss during media stream transmission and reception. Ping Command Executed by Microsoft Windows-based Computers Using the ping command, a Microsoft Windows-based computer transmits one packet every second. After transmitting a ping packet, a Microsoft Windows-based computer will not continue the transmission until it receives acknowledge from the peer end. The ping command is mainly used to test the connectivity between devices and the network delay. As described in the previous table, the ping command cannot emulate the media stream scenario in actual applications or reflect the random packet loss condition over the network. The ping command can be used as reference only when collecting statistics on media stream transmission status. 5 Summary The packet loss statistics displayed on the user interfaces of network devices and NLog are different from the actual condition on the network. The difference is relevant to the statistics' application objectives. When collecting packet loss statistics, NLog concerns the end-to-end audio-visual interaction during conferences. The packet loss statistics collected by network devices are used as a parameter when evaluating the bearer network condition. 6 Application Case 6.1 Participants Are Frequently Disconnected Due to No Stream Reception in the XX Site Symptom: In the XX site, the endpoint log before 15:00 keeps recording no stream reception, network disconnection, network recovery, and server registration failure information. An endpoint exits the conference because the endpoint does not receive streams for 20s. The MCU log records that the GK disconnects the call Huawei confidential. No spreading without permission. Page 12 of 15

13 Large packet loss is also recorded on Nlog. Cause analysis: Huawei confidential. No spreading without permission. Page 13 of 15

14 The endpoint log and Nlog record that network packet loss exists. The endpoint will be disconnected if it does not receive streams for 20s. Fault locating method: ----End Step 1 Run the TRACERT command on the MCU to trace the network nodes through which the packets travel from the MCU to the endpoint. Step 2 Check whether packet loss exists on each node (by capturing packets and viewing statistics on switches). Root cause: Through packet analysis and statistics on routers (or switches), it is detected that the packet loss on one router is large. Packet loss does not exist if the router is bypassed. 6.2 Image Artifacts Due to Network Disorder in the XX Site Polycom RMX2000 Digital cascading HUAWEI VP9650 SMC2.0 conference control platform Polycom endpoint Polycom endpoint Huawei HD endpoint TE50 Huawei HD endpoint TE50 Symptom: During the deployment test in a site, a Huawei 9650 is manually cascaded with the Polycom MCU. When you view images sent by the Polycom MCU, smearing occurs on the images in the case of large movements. Nlog detects that a few packets are lost. Cause analysis and fault locating method: The Polycom MCU and Huawei 9650 are in different networks. In addition, the network quality is poor. Through video stream packet capturing, it is found that serious packet jitter and disorder exist. The packet jitter even reaches 100 ms (default jitter compensation of the MCU: 80 ms), so the MCU cannot re-sort the packets, resulting in packet loss. The problem is more obvious in the case of large movements. Solution: Increase the jitter compensation value for the MCU Huawei confidential. No spreading without permission. Page 14 of 15

15 7 Appendix: Acronyms and Abbreviations on NLog User Interface ARL = Packet loss ratio for audio reception ASL = Packet loss ratio for audio sending VRL = Packet loss ratio for video reception VSL = Packet loss ratio for video sending XRL = H.239 presentation receive packet loss ratio XSL = H.239 presentation send packet loss ratio CRC = Cyclic redundancy check NDV = Network delay value NRJ = Network accept jitter NSJ = Network send jitter Huawei confidential. No spreading without permission. Page 15 of 15