ID 730L: Getting Started with Multimedia Programming on Linux on SH7724

ID 730L: Getting Started with Multimedia Programming on Linux on SH7724 Global Edge Ian Carvalho Architect 14 October 2010 Version 1.0

Mr. Ian Carvalho System Architect, Global Edge Software Ltd. Responsible for Technology Strategy Currently Driving Development Programs for: Video Telephony Video Surveillance Wi-Fi Media Streaming Previous Experience Embedded System Development Wi-Fi Software Development VoIP Stack & STB Application Development 2

Renesas Technology and Solution Portfolio Microcontrollers & Microprocessors #1 Market share worldwide * ASIC, ASSP & Memory Advanced and proven technologies Solutions for Innovation Analog and Power Devices #1 Market share in low-voltage MOSFET** * MCU: 31% revenue basis from Gartner "Semiconductor Applications Worldwide Annual Market Share: Database" 25 March 2010 ** Power MOSFET: 17.1% on unit basis from Marketing Eye 2009 (17.1% on unit basis).

Renesas Technology and Solution Portfolio Microcontrollers & Microprocessors #1 Market share worldwide * ASIC, ASSP & Memory Advanced and proven technologies Solutions for Innovation Analog and Power Devices #1 Market share in low-voltage MOSFET** * MCU: 31% revenue basis from Gartner "Semiconductor Applications Worldwide Annual Market Share: Database" 25 March 2010 ** Power MOSFET: 17.1% on unit basis from Marketing Eye 2009 (17.1% on unit basis). 4

Microcontroller and Microprocessor Line-up Superscalar, MMU, Multimedia Up to 1200 DMIPS, 45, 65 & 90nm process Video and audio processing on Linux Server, Industrial & Automotive High Performance CPU, Low Power Up to 500 DMIPS, 150 & 90nm process 600uA/MHz, 1.5 ua standby Medical, Automotive & Industrial High Performance CPU, FPU, DSC Up to 165 DMIPS, 90nm process 500uA/MHz, 2.5 ua standby Ethernet, CAN, USB, Motor Control, TFT Display Legacy Cores Next-generation migration to RX General Purpose Up to 10 DMIPS, 130nm process 350 ua/mhz, 1uA standby Capacitive touch Ultra Low Power Up to 25 DMIPS, 150nm process 190 ua/mhz, 0.3uA standby Application-specific integration Embedded Security Up to 25 DMIPS, 180, 90nm process 1mA/MHz, 100uA standby Crypto engine, Hardware security 5 2010 Renesas Electronics America Inc. All rights reserved.

Microcontroller and Microprocessor Line-up Superscalar, MMU, Multimedia Up to 1200 DMIPS, 45, 65 & 90nm process Video and audio processing on Linux Server, Industrial & Automotive High Performance CPU, Low Power SuperH Up to 500 DMIPS, 150 & 90nm process 600uA/MHz, 1.5 ua standby Medical, Automotive & Industrial High Performance CPU, FPU, DSC Up to 165 DMIPS, 90nm process 500uA/MHz, 2.5 ua standby Ethernet, CAN, USB, Motor Control, TFT Display Legacy Cores Next-generation migration to RX General Purpose Up to 10 DMIPS, 130nm process 350 ua/mhz, 1uA standby Capacitive touch Ultra Low Power Up to 25 DMIPS, 150nm process 190 ua/mhz, 0.3uA standby Application-specific integration Embedded Security Up to 25 DMIPS, 180, 90nm process 1mA/MHz, 100uA standby Crypto engine, Hardware security 6 2010 Renesas Electronics America Inc. All rights reserved.

Innovation 7

Agenda SH7724 Platform Features SH7724 (ECOVEC) Kit Contents Included Software Multimedia on Linux Frameworks & Libraries LAB Session Setting up the Hardware Platform Setting up the Development Environment Linux Kernel Configuration and Compilation Writing a Media Application Capture, Scale and Display Demonstration using SH Codecs Video Capture, Encode, Decode, Scale and Display Conclusion Optimizations Q & A 8

Key Take Aways 9

SH7724 Block Diagram 10

Platform Features SH7724 SoC Features 32-bit Superscalar RISC CPU with 7- Stage Pipeline 900 DMIPS @ 500 MHz 3.5 GFLOP single/double precision FPU with 10 stage pipeline 32KB I-Cache & 32KB D-Cache 256KB L2 Cache MPEG4, H.264 and WMV Accelerators with 720p HD quality JPEG Accelerator 2D Graphics Accelerator LCD Controller Two camera interface, up to 5M pixel sensors Sound I/O Module Ethernet MAC (10/100Mbps) ATAPI Interface 2 x USB 2.0 High Speed Host / Function Controller emmc 4.2 NAND Interface SD/MMC and SDIO Interface Development Platform Features Memory NOR Flash: 64MB 16-bits DDR2 SDRAM: 256MB (optional 512MB) 10/100 Ethernet Interface Audio IN & OUT NTSC / PAL Video Input and Output 2 x Digital Camera Interface DVI Interface LCD Interface with 7 WVGA LCD 6 push-button switches 2 x USB 2.0 Interfaces 3-axis Linear Accelerometer 2 x SD/MMC Slots RTC JTAG Interface 11

Kit Contents Renesas SH7724 Board (Revision: R0P7724LC) 7 WVGA LCD (800x480) Camera Board SH7724 Linux BSP Bootable USB Memory Stick containing Linux Kernel Image Root File System Interface Cables Power Adapter 12

Pictures of Kit Contents Camera Module Display Module & Touch Screen Processor Board 13

Included Software GNU SH4 Compiler Tool Chain Linux BSP 2.0.0 Linux 2.6.33 Kernel DSP Libraries Shcodecs Library Libshcodecs : wrapper library that provides API for applications to make use of the VPU Middleware Shjpeg Library Libshjpeg : JPEG Encoding and Decoding Library Supporting Libraries Libuiomux : System Resource Manager including UIO Devices Libshveu : Library for controlling VEU (image format conversion, scaling and rotation) 14

Additional Libraries VPU Middleware libavcbe : H264 Encoder Core libm4vse : MPEG4 Encoder Core libm4iph : VPU4 Driver The VPU Middleware Library is provided separately by Renesas on purchasing an Evaluation Kit 15

Framework & Libraries Linux BSP Full Featured *nix compatible OS with integrated USB Stack, Networking Stack and a host of other stacks ALSA Audio recording & playback framework Video4Linux Video Capture and Playback Framework DirectFB Graphics Library with Hardware Acceleration, Input Device Handling and Abstraction 16

Software Framework Diagram Applications User Space UIOMUX SH JPEG Linux Libraries Video4Linux, ALSA, DirectFB SH Codecs VPU Middleware SH VEU Kernel Space Linux OS, Device Drivers and Sub-Systems (Video4Linux, Networking, ALSA, etc.) ECOVEC SH 7724 Platform Hardware Camera Display Ethernet USB SDIO Keypad 17

Start of Lab Session Keep your dice turned to the section of the lab you are on. (Instructions are provided in the lab handout) Please refer to the Lab Handout and let s get started 18

Checking Progress We are using the die to keep track of where everyone is in the lab. Make sure to update it as you change sections. When done with the lab, your die will have the 6 pointing up as shown here. 19

Lab Description The lab sessions covers a walk through of the installation procedure for setting up the SH7724 based ECOVEC Platform, its development environment and the development of Multimedia Applications on the Renesas SH7724 Platform using the Tools, Linux Operating System and Libraries provided. 20

Lab Objectives Setting up the Evaluation Platform Setting Up the Development Environment Installation of the BSP, Development Tools and Libraries Running the Pre-built Images Preparing and Booting with a USB Drive Development using NFS SH Linux Kernel Configuration and Compilation Renesas Specific Configuration Items Platform Execution Environment (Bootloader, etc.) Writing a Simple Media Application Capture, Scale and Display 21

Lab Materials Lab Workstation Setup Ubuntu Linux Workstation Download Directory with Renesas SH4 Linux BSP 2.0 VPU Middleware Debian Packages for DHCP Server, TFTP Server, NFS Server minicom Terminal Emulator Renesas SH7724 Target Setup SH7724 (Ecovec) Platform Display & Camera Modules USB Cable (for Console) Ethernet Cable, Power Adapter USB Drive with Linux Boot Image and Sample Code Lab Instruction Booklet 22

Setting up the Evaluation Platform 23

Start Lab Session 1 24

Lab Session 1 Complete Accomplished Setup the development platform and able to run the pre-built Linux image and the demo applications Next Setting up the Development Environment 25

Setting Up The Development Environment Ubuntu Linux as the Development Host platform Any IDE that the User is Comfortable with can be used For the purpose of the LAB we will be using Anjuta DevStudio which is the GNOME Integrated Development Environment For convenience and saving time all the required software has been downloaded and installed on the Lab machines Procedure to Setup the Development Environment Download the BSP & Tool Chain Available from https://oss.renesas.com/ Install the BSP & Tool Chain Setting up for Network Booting Booting with a Pre-built Image over the Network 26

Start Lab Session 2 27

Lab Session 2 Complete Accomplished Setting up the Development Environment by installing the SH4 Linux BSP for the SH7724 Installed and Configured the Networking Utilities that are required to boot of the Network Are able to boot of the Network using pre-built Linux images Next Session SH Linux Kernel Configuration and Compilation 28

SH Linux Kernel Configuration and Compilation In this Session we will: Configure the Linux Kernel Compile the Linux Kernel Install and execute the newly compiled Linux Kernel image 29

Start Lab Session 3 30

Lab Session 3 Complete Accomplished Configured the Linux Kernel for SH7724 Platform Built the Kernel Image Are able to boot of the our new SH Linux Kernel Image Next Session Writing a Simple Media Application 31

Writing a Simple Media Application Objective Capture 720p Frames from the Camera @ 30 fps Scale for WVGA (800x480) Display Output the frames on the Display Lab Session Code Walk Through Compilation Execution 32

Start Lab Session 4 33

Lab Session 4 Complete Accomplished Went through the code required for a simple media application that captures video from a Camera, Scales the captured image and displays the scaled image on the LCD Compiled the application on the Host Development Environment that was setup earlier Executed the program successfully 34

Optimizations Necessary to be Real Time Using buffers that are provided by the UIOMUX Library to avoid coping of data Threading is required in order achieve 720p Capture and Encode @ 30 fps 35

Conclusion Summarize the capabilities of the platform Standard Linux System Linux Programming on all platforms is the same Ease of use Porting to target is easy Many frameworks, libraries and application readily available Where to get additional information Renesas web site Renesas Open Source Web Site https://oss.renesas.com/ Video4Linux http://linux.bytesex.org/v4l2/ Ubuntu Linux Distribution http://www.ubuntu.com/ 36

Questions? 37

Question 1. What clock speed does SH7724 Processor run at? 500 MHz 2. What is the DMIPS performance of the SH7724 SoC? 900 DMIPS or 1.8 MIPS / MHz 3. How many camera s are supported on the SH7724 SoC? 2 4. What is VEU? What is it used for? Video Engine Unit It is used for video pre and post processing operations like scaling, color space conversion, etc. 5. What is VPU? What is it used for? Video Processing Unit. It is used for Encoding and Decoding video in various formats like H.264, MPEG4, WMV, etc. 6. What is the camera resolution set to in the media application presented? 720p or 1280 x 720 7. What is the CPU utilization when running the CSD application? ~4% 38

39 Thank You!

40 Appendix

Using Hardware Accelerated Codecs Demonstration of Using SH Codecs to Uses the H.264 Hardware Accelerated Codec Builds on Previous Sessions Application Capture 640x480 Frames from the Camera Encode the Video using the H.264 Encoder Display Encoded Images Properties Decode the Encoded Video Output the frames on the Display Code Walk Through of the Video Encode and Decode Process 41

Useful Libraries X Toolkits GTK+ for GUI Development Qt for GUI Development GStreamer for recording and playback of audio and video Libavcodec for encoding and decoding audio and video SDL Direct Audio / Video Playback Library DirectFB Hardware Accelerated Display Library 42

Abbreviations VPU Video Processing Unit VEU Video Engine Unit JPU JPEG Processing Unit 43