ID B30C: IP Video Surveillance Applications Using the Renesas SH 7724 MPU Global Edge Ian Carvalho Architect Kiran Gaitonde Program Manager 12 October 2010 Version 1.0 1
Mr. Ian Carvalho System Architect, Global Edge Software Ltd. Responsible for Technology Strategy Currently Driving Development Programs for: Video Telephony Video Surveillance Wi-Fi Media Streaming Previous Experience: Embedded System Development Wi-Fi Software Development VoIP Stack & STB Application Development 2 2
Mr. Kiran Gaitonde Program Manager, Global Edge Software Ltd. Responsible for Platform & Telecom Engineering Currently Driving Development Programs for: Video Surveillance Android Linux/RTOS BSP and Drivers Device Testing Previous Experience: Embedded System Development Wi-Fi Software Development Avionics Software Development 3 3
Renesas Technology and Solution Portfolio Microcontrollers & Microprocessors #1 Market share worldwide * ASIC, ASSP & Memory Advanced and proven technologies Solutions for Innovation Analog and Power Devices #1 Market share in low-voltage MOSFET** * MCU: 31% revenue basis from Gartner "Semiconductor Applications Worldwide Annual Market Share: Database" 25 March 2010 ** Power MOSFET: 17.1% on unit basis from Marketing Eye 2009 (17.1% on unit basis). 4 In the session B30C, IP Video Surveillance on the Renesas SH7724, Ian Carvalho introduces this high level image of where the Renesas Products fit. The big picture.
Renesas Technology and Solution Portfolio Microcontrollers & Microprocessors #1 Market share worldwide * ASIC, ASSP & Memory Advanced and proven technologies Solutions for Innovation Analog and Power Devices #1 Market share in low-voltage MOSFET** * MCU: 31% revenue basis from Gartner "Semiconductor Applications Worldwide Annual Market Share: Database" 25 March 2010 ** Power MOSFET: 17.1% on unit basis from Marketing Eye 2009 (17.1% on unit basis). 5 This is where our session, B30C, IP Video Surveillance on the SH7724, is focused within the Big picture of Renesas Products Let me first introduce our rich portfolio of microcontrollers and microprocessors solution which includes 8, 16 and 32 bit CPU cores. <Click>
Microcontroller and Microprocessor Line-up Superscalar, MMU, Multimedia Up to 1200 DMIPS, 45, 65 & 90nm process Video and audio processing on Linux Server, Industrial & Automotive High Performance CPU, Low Power Up to 500 DMIPS, 150 & 90nm process 600uA/MHz, 1.5 ua standby Medical, Automotive & Industrial High Performance CPU, FPU, DSC Up to 165 DMIPS, 90nm process 500uA/MHz, 2.5 ua standby Ethernet, CAN, USB, Motor Control, TFT Display Legacy Cores Next-generation migration to RX General Purpose Up to 10 DMIPS, 130nm process 350 ua/mhz, 1uA standby Capacitive touch Ultra Low Power Up to 25 DMIPS, 150nm process 190 ua/mhz, 0.3uA standby Application-specific integration Embedded Security Up to 25 DMIPS, 180, 90nm process 1mA/MHz, 100uA standby Crypto engine, Hardware security 6 2010 Renesas Electronics America Inc. All rights reserved. Here are the MCU and MPU Product Lines, I am not going to cover any specific information on these families, but rather I want to show you where this session is focused <Click> 6
Microcontroller and Microprocessor Line-up Superscalar, MMU, Multimedia Up to 1200 DMIPS, 45, 65 & 90nm process Video and audio processing on Linux Server, Industrial & Automotive High Performance CPU, Low Power SuperH for Video Up to 500 DMIPS, 150 & 90nm process 600uA/MHz, 1.5 ua standby Medical, Automotive & Industrial High Performance CPU, FPU, DSC Up to 165 DMIPS, 90nm process 500uA/MHz, 2.5 ua standby Ethernet, CAN, USB, Motor Control, TFT Display Legacy Cores Next-generation migration to RX General Purpose Up to 10 DMIPS, 130nm process 350 ua/mhz, 1uA standby Capacitive touch Ultra Low Power Up to 25 DMIPS, 150nm process 190 ua/mhz, 0.3uA standby Application-specific integration Embedded Security Up to 25 DMIPS, 180, 90nm process 1mA/MHz, 100uA standby Crypto engine, Hardware security 7 2010 Renesas Electronics America Inc. All rights reserved. These are the products where this presentation applies <Click> 7
Innovation 8 Using the SH4 Processors new and varied applications in video processing can be performed in areas such as: High resolution HD recording for Surveillance Video Analytics for real time alerts Rear View Cameras Baby Monitoring Airport Surveillance Home Security System 8
Agenda Real Time, Linux and Surveillance Multimedia Requirements and Characteristics Surveillance Requirements SH7724 and Surveillance Trade-offs and Effects Solutions for the Effects Video Surveillance System Diagram and Components Issues Faced in Developing Surveillance Solution Advanced Features Other Related Solutions 9 We will be covering the following topics in this session: 9
Key Take Aways 10 This presentation summarizes the capabilities of the SH7724 processor for video processing particularly with respect to the performance H.264 encoder. It also explains the ease with which multimedia applications can be built on the reference platform using the Linux BSP and libraries provided by Renesas. 10
Real Time, Linux and Surveillance Real Time System Characteristics Correctness Data Integrity Computation Accuracy Timeliness Hard Real Time Soft Real Time Linux Characteristics Default OS Not Real Time No Guaranteed Interrupt Latency (but sub 10ms) Preemptive Scheduling System Mode and User Mode Surveillance Characteristics Soft Real Time Requirements 500ms Latency Acceptable Needs Guaranteed Video Frame Rate of 30 fps (33.33 ms) Needs Guaranteed Network Bandwidth 11 Correctness Data Integrity 0 Tolerance Computation Results 0 Tolerance Timeliness Hard Real Time Mission Critical Systems need not only accuracy, but also timely responsiveness with 0 tolerance. E.g: Flight Control, Missile Control Soft Real Time Critical Systems need accuracy, but minimal latency in response is acceptable Linux Scheduling Round Robin, depends on system load Interrupt Latency Overheads due to memory model, context switching, etc Pre-emption Lack of features like priority inheritance IO Operations Re-ordering of IO requests Live audio-video systems Usually soft real-time Latency in timeliness can degrade quality, but the system can continue to operate -Hard deadlines limits flexibility and adaptability, while increasing the hardware resource requirements and lowering the hardware efficiency 11
Multimedia Requirements Access with Specific Timing Interdependent Data Sets Reliable Media Delivery Synchronization Higher Computational Power Large Storage Compression 12 Access with Specific Timing Example: Video capture/playback must happen at frame rate of 24 to 30 frames per second. This is handled by using the Linux driver that can handle the capture and buffering at the time of arrival of the interrupt at the rate programmed in the camera. Interdependent Data Sets Example: Interlaced or Progressive Video, I-Frame/P-Frame/B-Frame in Compressed Video Reliable Media Delivery Must meet specific timing requirements, as well as minimum loss of data in transmission Synchronization Between streams from different interfaces on same system as well as different interfaces on different systems Higher Computational Power To meet timing constraints of capture, encode and transmission Large Storage Size of 1 second raw video data of 720p at 30 fps is 55,296,000 bytes (54 MB) Compression - Reducing the quantity of data used to represent digital multimedia data. The above mentioned 1 second of raw video data that would have occupied 54 MB of storage can be compressed to approximately 480 KB which reduces the transmission and storage requirements by over 100 times. 12
Multimedia Video Characteristics Video Capture & Display Frame Rate Resolution Latency Quality Video Codec Bandwidth Quality Computation Intensity 13 Ideal Frame Rate is > 30 FPS Resolution 720p (~1 MP), 1080p (2 MP). Digital still picture camera have much higher resolutions going beyond 15 MP Video Latency is the time taken in processing the video pipeline to get an output from the camera. These delays could be caused by capture time, compute time or transmission time. Acceptable real time video latencies are within 150 ms. Acceptable latency when video is streamed over networks where the viewer cannot see the real and the captured image simultaneously is around 500 ms. Quality of video is based on many components, like the camera, video processing engine (8bit, 16bit, 32bit), etc. Codecs MJPEG, MPEG2, MPEG4, H.264 Quality of the Codecs usually are related to the bandwidth utilized but with newer algorithm, like H.264, high quality video is possible even at low bandwidths. Codecs are usually selected based on the amount of processing power available. Today these CPU intensive tasks have been moved into specialized hardware block which perform the video encoding and decoding. The SH7724 processor is one such example where the H.264, MPEG4, WMV and JPEG codecs are implemented using hardware engines. 13
Multimedia Audio Characteristics Audio Playback & Recording Sample Size Sampling Rate Channels Latency Audio Codec Bandwidth Quality Computation Intensity 14 Sample Size is the number bits used to store each sample. This can vary from 8 bit to 32 bits. The more bits used the better the quality and the more the bandwidth required. CD quality is based on 16 bit sampling at 44.1 KHz in stereo. Sampling Rate is the number of samples that are captured per second. The higher the sampling rate the higher the frequency range that can be captured. Channels Mono, Stereo, 5.1, 5.2, 7.1, 9.1 Audio Latency Ideal < 10 ms Acceptable 10 to 50 ms Inadequate > 100 ms Various audio codecs are available for encoding and decoding various kinds of audio like speech and music. Speech Codecs are G.711, G.722, G.723.1, G.726 (ADPCM), G.729, Speex, GSM-AMR, etc. Each of these have different characteristics and are suited for different applications some of which being legacy applications. There are many Audio Codecs, the most popular being MP3 codec. Other popular codecs are AAC, Vorbis (which is a free Audio Codec), etc. The Renesas SH7724 processor is more than capable of running these codecs on the RISC CPU and DSP Processor. 14
Surveillance Requirements High Resolution Video with Audio Support Secure Access Location Independence Mobile Device Support Network Attached Storage Archiving Dynamic DNS Real Time Alerts Integration with other Applications Robustness and Availability Event Driven Video Content Analysis 15 Secure Access Authentication, Authorization, Encryption Location Independence Remote Monitoring, From Anywhere through Internet Mobile Device Support Alerts as SMS or E-mail to ubiquitous and easily accessible mobile phones and PDAs to enable prompt & swift action Network Attached Storage For secure storage, contingency against disasters & theft, increased capacity, added availability with RAID Archiving Automatic continuous archiving, either locally or remotely, for later usage or audit purposes Dynamic DNS Support for dynamic IP addresses Real Time Alerts - Based on detected events for actions like raising alarm, closing a gate, convert to continuous recording Integration with other applications Example: Access Control Systems, Fire Safety, Hazardous Material Management, Verification of disputed transaction Robustness and Availability 24 x 7, Intelligent UPS Event Driven Detection of relevant events, Tagging for efficient retrieval, On demand Video Content Analysis Video Content Analysis People Video Analytics: Counting people, intrusion, presence detection, movement tracking, face capture when in view, falling and running etc. Vehicle Video Analytics: License plate reading, stopped car, wrong way, speeding, model and make etc. Object Video Analytics: Unattended object, removed object, object moved etc. 15
SH7724 Features 32-bit Superscalar RISC CPU with 7-Stage Pipeline 900 DMIPS @ 500 MHz 3.5 GFLOP single/double precision FPU with 10 stage pipeline 32KB I-Cache & 32KB D-Cache 256KB L2 Cache MPEG4, H.264 and WMV Accelerators with 720p HD quality JPEG Accelerator 2D Graphics Accelerator LCD Controller Two camera interface, up to 5M pixel sensors Sound I/O Module Ethernet MAC (10/100Mbps) ATAPI Interface 2 x USB 2.0 High Speed Host / Function Controller emmc 4.2 NAND Interface SD/MMC and SDIO Interface I2S & I2C Interface for Audio 16 16
SH7724 and Surveillance Video Encoding Hardware VPU Engine H.264, MJPEG, MPEG4, WMV H.264 Encoding 720p @ 30 fps Tunable parameters Hardware VEU Engine Pre-processing and Post-processing Scaling, Color Space Conversion, Rotation Camera Dual Camera Support Up to 5 Mega Pixel Resolution Display LCD Interface IP Video Streaming Ethernet Interface USB / SDIO for Wireless Networking Interface 17 17
SH7724 Platform Diagram 18 18
Surveillance with Linux on SH7724 Linux 2.6.x BSP Video4Linux2 Driver for Camera ALSA Driver for Audio Library for SH-Mobile VPU (Video Codecs) Library for SH-Mobile VEU (Image Scale/Rotate) Library for SH-Mobile BEU (Image Blend) Library for SH-Mobile JPU Middleware for Hardware Engines Conflict manager for system resources (UIOMUX) DirectFB Support for Display GStreamer Support 19 SH7724 Linux BSP provides subsystems and libraries needed for surveillance applications Gstreamer plugin gst-sh-mobile-0.10.1 contain Hardware accelerated elements that perform video encode, decode, scale and color space conversion. 19
SH7724 IP Camera System Diagram H.264 UDP/RTP MJPEG TCP/HTTP Figure 1: Video Surveillance System Diagram 20 20
Trade Offs Resolution vs. Bandwidth Resolution vs. Frame Size I-Frame Interval vs. Bandwidth Software vs. Hardware Codecs TCP vs. UDP Effects of Trade-offs 21 1. Quality vs. Affordability a. Finding the right combination of image quality, frame rate and bandwidth. b. Image quality is paramount 2. The right balance, making choices about bandwidth, resolution, frame rate and compression 21
Resolution vs. Bandwidth Resolution vs. Bandwidth 5 Bit Rate in Mbps 4 3 2 1 3.87 3.96 3.90 3.97 3.89 3.97 3.89 3.94 0 720p SVGA VGA QVGA Resolution 30 FPS 20 FPS H.264 Encoder Settings I-Frame Interval = 30, Bit Rate = 4000000 22 Bandwidth requirement is almost same for different frame rates as well as resolutions when the bit rate and I- Frame interval are constant. The difference will be noticed in the quality of the image. The images viewed at 20 fps will have better quality than that viewed at 30 fps.
Resolution vs. Frame Size Resolution vs. Frame Size Frame Size in KB 30 25 20 15 10 5 0 24.45 24.46 24.45 24.42 16.30 16.30 16.28 16.28 720p SVGA VGA QVGA Resolution 30 FPS 20 FPS H.264 Encoder Settings I-Frame Interval = 30, Bit Rate = 4000000 23 The Frame size is almost the same for various resolutions but varies with frame rate. This is because the I- Frame Interval and Bit Rate are constant. The difference will be noticed in the quality of the image. The images viewed at 20 fps will have better quality than that viewed at 30 fps.
I-Frame Interval vs. Bandwidth Bandwidth vs. I Frame Interval 5 Bit Rate in Mbps 4 3 2 3.90 3.45 2.27 1.73 1.55 1.52 1.47 1.42 1 0 10 20 30 40 50 60 70 80 I Frame Interval H.264 Encoder Settings Resolution = 1280 x 720, Frame Rate = 30 FPS, Bit Rate = 4000000 with a Fixed Image 24 Bandwidth requirement reduces with increase in I Frame Interval for a fixed image. But for live video where frames vary, the encoder will improve the image quality by maintaining configured bit rate. 24
Software Codec vs. Hardware Codec Software Codec Slower CPU Intensive Any codec can be implemented Easy to modify and upgrade No dependency on manufacturer Eg : ffmpeg Hardware Codec Faster Less load on CPU Restricted to Specific Codec(s) Difficult and costly to modify Dependency on manufacturer Eg : sh4 vpu 25 Hardware codecs are preferable for surveillance solutions as it provides real time feature with better CPU usage. 25
libjpeg vs. SH7724 JPU JPEG Encoding: CPU Usage vs. Resolution CPU Usage in % 100 80 60 40 20 0 99.60 99.60 99.60 99.60 3.30 3.50 4.10 4.90 1280 x 720 800 x 600 640 x 480 320 x 240 Resolution in Pixels S/W Codec H/W Codec 26 This graph shows the advantages of having hardware codec over software codec for JPEG Encoding. It depicts CPU usage comparison of software and hardware JPEG encoders for different resolution. The input image used in this test contained rainbow stripes having VIBGYOR colours. As you can see software encoding takes almost 100% CPU time where as the dedicated hardware codec leaves CPU almost free. 26
libjpeg vs. SH7724 JPU JPEG Encoding: Frame Rate vs. Resolution Frame Rate in FPS 300 250 200 150 100 50 0 292.51 82.39 53.29 28.50 39.50 3.45 6.53 10.43 1280 x 720 800 x 600 640 x 480 320 x 240 Resolution in Pixels S/W Codec H/W Codec 27 This graph shows the advantages of having hardware codec over software codec for JPEG Encoding. Frame Rate increases as the resolution reduces. Observe the frame rates produced by software and hardware codecs. If we limit the frame rate we can reduce the CPU usage further. For lower resolutions the frame rates are high and it is the reason for increase in CPU usage. 27
libx264 vs. SH7724 VPU H.264 Encoding: CPU Usage vs. Resolution CPU Usage in % 100 80 60 40 20 0 99.60 99.60 99.60 99.60 1.10 1.00 1.00 0.90 1280 x 720 800 x 600 640 x 480 320 x 240 Resolution in Pixels S/W Codec H/W Codec 28 This graph shows the advantages of having hardware codec over software codec for H.264 Encoding. It depicts CPU usage comparison of libx264 software encoder and SH7724 VPU hardware encoder for different resolution. The input image used in this test contained rainbow stripes having VIBGYOR colours. As you can see software encoding takes almost 100% CPU time where as the dedicated hardware codec leaves CPU almost free. 28
libx264 vs. SH7724 VPU H.264 Encoding: Frame Rate vs. Resolution Frame Rate in FPS 40 30 20 10 0 28.50 29.10 29.60 29.80 6.50 0.50 1.50 3.00 1280 x 720 800 x 600 640 x 480 320 x 240 Resolution in Pixels S/W Codec H/W Codec 29 This graph shows the advantages of having hardware codec over software codec for JPEG Encoding. Frame Rate increases as the resolution reduces. Observe the frame rates produced by software and hardware codecs. If we limit the frame rate we can reduce the CPU usage further. 29
TCP vs. UDP TCP HTTP, HTTPs, FTP, Telnet, etc... Byte stream, no definite boundaries Comparatively slower Reliable and controls congestion Re-ordering of packets Connection based Delivery Guaranteed Used in non-time critical applications Error checking UDP DNS, DHCP, TFTP, RIP, VOIP, etc... Individual packets, definite boundaries Comparatively faster Lightweight, no tracking of connection No re-ordering of packets Connectionless Delivery not guaranteed Used in time critical applications No error checking 20 bytes header 8 bytes header 30 What is the best transport layer protocol for IP video surveillance? Looking into pros and cons of two well known protocols TCP and UDP, the light weight and real timeliness of UDP makes it better choice for IP surveillance applications. 30
Effects of Trade-offs Characteristic Advantage Disadvantage Increase Bandwidth Improves Quality Increases Cost Reduce Resolution Reduces Bandwidth Reduces Clarity Reduce Frame Rate Video Compression Reduces Bandwidth Balances Quality vs. Bandwidth Motion Quality is Reduced Requires Specialized Hardware and Software 31 Choosing the optimized values depending on the application and affordability is the key. Finding the right combination of image quality, frame rate and bandwidth. Image quality is paramount Highest possible resolution would make objects clearly identifiable High frame rates would provide smooth motion Compression wouldn t be necessary so the videos would always be totally unaltered Requires many gigabits of bandwidth, storage requirements The higher the resolution and frame rate, the more bandwidth is needed. The compression, helps retain higher image quality while reducing bandwidth and cost. However effective compression is, there shall still be trade-offs 31
Issues Faced CPU Usage Latest versions of the Codec Libraries CPU usage dropped from 60% to 2% Frame Rate at Higher Resolution Use of multi-threaded programming Packet Loss Use of UDP + RTP to detect packet loss RTP specific to H.264 fields used to identify Frame Types and avoid decoding of frames if I-Frame is missed 32 These are some of the issues and solutions faced while developing the surveillance application on SH7724. - Use of supporting libraries improved CPU usage in great extent. - Exploring the concurrent execution of CPU and hardware units using the multi threaded programs resulted in streaming 720p HD in real time frame rate. - Usage of RTP ensured proper decoding at the other side. Finally SH7724 is best suited platform for Surveillance applications as it can stream 720p HD with 30 fps even with the overhead of RTP and most importantly all these with less than 2% CPU usage!!! 32
Advanced Features PTZ Controls Video Analytics Engine 1080p Resolution Encryption Accelerators 33 Areas of Enhancements and Improvements: Camera module with pan-tilt-zoom support Video Analytics Engine Support for higher frame rates Encryption accelerators for security 33
Other Related Solutions Video Phones Video Conferencing Streaming Media Clients Digital Video Cameras Rear View Cameras 34 SH7724 platform can be used in variety of multimedia solutions and applications. 34
Conclusion 35 With its range of hardware accelerators for Multimedia coding, the SH7724 Processor is best suited for Multimedia applications. 35
More Information Renesas Web Site http://www.renesas.com/ SuperH Linux & Open Source @ Renesas https://oss.renesas.com/ Global Edge Software Ltd. http://www.globaledgesoft.com/ mailto:sales@globaledgesoft.com 36 36
Questions? 37 37
Questions 1. What is the performance of the SH7724 FPU? 3.5 GFLOPS 2. How many camera s does the SH7724 SoC Support? 2 3. What interfaces are available for networking? Ethernet, USB and SDIO Interfaces 4. What is the CPU Load while performing H.264 Video Encoding? 2% 5. What codecs are supported in hardware on the SH7724 SoC? H.264, MPEG4, WMV and JPEG 6. What is the functionality of VEU? Image Scaling, Color Space Conversion and Rotation 38 38
Thank You! 39 39
40