A REPORT ON MY WORK ON THE. BEAGLEBOARD XM (Angstrom OS), WHICH INCLUDES IMAGE PROCESSING, OPTIMIZATION AND WORK WITH LEOPARD CAMERA.

Transcription

1 A REPORT ON MY WORK ON THE BEAGLEBOARD XM (Angstrom OS), WHICH INCLUDES IMAGE PROCESSING, OPTIMIZATION AND WORK WITH LEOPARD CAMERA. 1 Kaushik Kanetkar

2 Contents: 1. Porting Angstrom to Beagle Xm 3 2. Getting the code to work alongwith ffmpeg on Beagle 6 3. Work on Leopard camera 7 4. My time-lapse example 9 5. Remote debugging using GDB Image processing and optimization 11 1.Neon instructions 7. Porting Ubuntu O.S on the Beagleboard 18 2

3 Porting Angstrom on to Beagle Xm: Following are the steps to load Angstrom kernel on your Beagleboard Xm: Preparing the SD card: Plug in the SD card on your host system and wait for it to mount. You can check by the command df -h. Look for the line which starts with /dev/sdxx, it will generally be mounted in /media/ Make sure the size of the SD card is enough. Now un-mount the SD card using umount /dev/sdxx For Angstrom to run you will need to have a sd-card with two partitions; one fat partition to hold the boot files and then a ext3 partition with the root file system on it. Make 2 partitions p1:fat32 (0x0c) and other p2: ext3 (0x83) partition. Make p1 bootable and keep around Mb space. To make the two partitions you can either use the GUI based application: Disk Utility. The other way would be to use the script omap-mkcard.sh in the folder by running the following commands: chmod +x omap3-mkcard.sh sudo./omap3-mkcard.sh /dev/sdx sync Remove the SD card and plug it in after 2-3 seconds. You will find two partitions /media/boot and /media/angstrom. The next step would be to generate a build. You can use the rootangstrom.tar.gz in the folder, or else you can generate your custom build at Select: BeagleBoard (the original BeagleBoard and BeagleBoard-xM both use the same build); simple; and console only. Now in the additional packages section, make sure you select the following: Toolchain Native u-boot mkimage All kernel modules Ffmpeg Bootloader Files (x-load/u-boot/scripts) in the Platform specific packages section 3

4 Now click on build me! which will generate a *.tar.gz. Download this *.tar.gz in your working directory. There are only three files that are mandatory for boot partition. The following lines will extract the files from the download build and copy those to the boot partition on the sd-card. tar xvf [your-file].tar.gz cp boot/mlo-* /media/boot/mlo cp boot/uimage-* /media/boot/uimage cp boot/u-boot-*.bin /media/boot/u-boot.bin sync Now we have to copy/extract the filesystem to your SD card. cd /media/ansgtrom sudo tar xvf [path to you download file].tar.gz sync Make it safe to remove your SD card. umount /media/boot umount /media Angstrom Remove the SD card and place it inside the card case on your Beagleboard. Connect the Serial (BeagleBoardxM) to USB(Host PC) converter and configure the minicom connection. Start minicom on your host machine in configuration mode. As root: # minicom -o -s -w A menu of configuration should appear. Use the Down-arrow key to scroll down and select the Serial port setup option, and press Enter. Verify that the listed serial port is the same one that is connected to the target board. If it is not, press A, and enter the correct device. This is /dev/ttyusb0 on most Linux distributions. Set the Bps/Par/Bits option by pressing the letter E and using the next menu toset the appropriate values. You press the key that corresponds to the value , and then press Enter. Set Hardware flow control to No using the F key. Set Software flow control to No using the G key. 4

5 Press Enter to return to the main configuration menu, and then press Esc to exit this menu. Reset the board, and wait for a moment. If you do not see output from the board, press Enter several times until you see the prompt. If you do not see any output from the board, and have verified that the serial terminal connection is setup correctly, contact your board vendor. When you boot Angstrom for the first time on the beagleboard, it will take some time. Be patient. Once it is done, you will be asked for a beagleboard login: (enter root ) Run the following command: ifconfig usb (gives an ip address to your usb) Now run the following command on your host: ifconfig eth (configures a static ip address) From your beagleboard see if you can ping your host. ping If you are able to ping, then you are all set and ready to work with Angstrom on beagleboard and to transfer files between your beagle and the host system! 5

6 Timelapse on the Beagleboard : Build the code_to_capture_frames.c code on the Beagleboard using: arm-angstrom-linux-gnueabi-gcc code_to_capture_frames.c lm pthread lrt Run the executable./a.out file The leopard camera will now capture images and store it in the YUV folder. The next step is to convert the YUV s into PPM s and encode them into a video. You have two options: 1. On the Beagleboard: Build the buildvideo.c code using gcc buildvideo.c lm Now run the./a.out output file and specify the no. of frames you want to use in forming the video../a.out <no.of frames> 2. On your host system (Ubuntu): Transfer the YUV folder from Beagleboard to the host system using scp. scp r yuvs/ hostname@host_ip_address:/home/ Build the buildvideo.c code and run it as above. Run the video on VLC player. 6

7 Leopard Camera: Refer to the following document regarding Video4Linux in Srinivas report and the link : I have made a few changes to his code to optimize the use of Leopard camera. Firstly, I will just go through some of the important ioctl s for getting images from the Leopard camera. ioctl(fd, VIDIOC_REQBUFS, &req); -This allocates the number of buffers to be allocated. ioctl(fd, VIDIOC_QUERYBUF, &buffer); -It queries the status of the buffers. You need to make sure that all the allocated buffers are queried. ioctl(fd, VIDIOC_QBUF, &buffer); -This ioctl call is used the enqueue a buffer in the driver s incoming queue. ioctl(fd, VIDIOC_DQBUF, &buffer); -This ioctl is used to dequeue a filled buffer from the driver s outgoing queue. ioctl(fd, VIDIOC_STREAMON, &type); ioctl(fd, VIDIOC_STREAMOFF, &type); -The VIDIOC_STREAMON and VIDIOC_STREAMOFF are used to start and stop the capture. Note: While using the VIDIOC_STREAMON there is a high chance that it might prompt an error that the device is busy. So make sure there is a while loop such that wait till the ioctl stops returning -1 (error value), and then only proceed. 7

8 It takes a lot of time for the VIDIOC_DQBUF, VIDIOC_QBUF calls to execute, especially if only one buffer is used. I added more buffers in the queue ( a maximum of 32 can be allocated ). Adding more buffers brought down the time for capturing and also brought the frame capturing rate up. The following was included after all the settings related to MMAP and STREAMON were done: {buffer.type = V4L2_BUF_TYPE_VIDEO_CAPTURE; buffer.memory = V4L2_MEMORY_MMAP; buffer.index = j //make sure this value of j is the value of buffer number which you are currently using. ioctl(fd, VIDIOC_DQBUF, &buffer); ----getting the yuv----- ioctl(fd, VIDIOC_QBUF, &buffer); } With this, I was able to get a better frame rate, by making use of buffers. Future Scope: While aiming to achieve a higher frame rate, it gives an error in the VIDIOC_QBUF or VIDIOC_DQBUF. If somehow, this error can be avoided by making optimum use of the maximum amount of buffers available, then, a stable and a better frame rate can be achieved. *Special packages to click screenshots* Mplayer: Angstrom has an application called mplayer which can be used on the Beagleboard, to view and click images based on the frame rate. I used this application on the console to click images. 8

9 Time-lapse example : 1. I have created a time-lapse focusing on a digital clock. The frequency is 1Hz. The ppms folder contains the images clicked by the Leopard camera at every second. 2. The other time-lapse I created was the melting of ice-cubes. 9

10 Remote debugging using GDB: Get the following cross-compiler tools (arm-none-linux-gnueabi-gcc) from Using command line: Now, for example the ip address of my system (host) is and the ip address of the target (Beagle) is With the given Angstrom kernel, you indeed have gdbserver installed on your target. To start gdbserver on the target you need to specify a port number which you will listen and then connect through and the executable to debug. The port can be anything so long as it s not a reserved port e.g. 21,80. I am using the port number gdbserver test:2345 a.out This creates the pid and starts listening on port Run the gdb command from the host system: arm-none-linux-gnueabi-gdb./a.out Once gdb starts, run the following: (gdb) target remote :

11 Image processing and optimization: I used an image from the Leopard cam (pic-5.ppm), having a resolution of 640x480. I applied the sharpen algorithm on it without any optimization or mulit-threading. This was the result I got using the arm-angstrom-linux-gnueabi-gcc compiler for ARM. Then, I used some optimizations mentioned in the following paper by Prof.Sam Siewert These are the Intel SIMD instructions for gcc optimization. It provided the following speed-up. Generally we would want the processor to speed up more. So ARM does have a number of instruction sets, one of them being the NEON instruction set. NEON is included in all Cortex-A8 devices but is optional in Cortex-A9 devices. Lets see how the NEON provides better optimization. 11

12 NEON Instructions : The Advanced SIMD extension (aka NEON or "MPE" Media Processing Engine) is a combined 64- and 128-bit single instruction multiple data (SIMD) instruction set that provides standardized acceleration for media and signal processing applications. NEON can execute MP3 audio decoding on CPUs running at 10 MHz and can run the GSM adaptive multi-rate (AMR) speech codec at no more than 13 MHz. It features a comprehensive instruction set, separate register files and independent execution hardware. NEON supports 8-, 16-, 32- and 64-bit integer and single-precision (32-bit) floating-point data and operates in SIMD operations for handling audio and video processing as well as graphics and gaming processing. In NEON, the SIMD supports up to 16 operations at the same time. Devices such as the ARM Cortex-A8 and Cortex-A9 support 128-bit vectors but will execute with just 64 bits at a time, whereas newer Cortex-A15 devices can execute 128 bits at once. NEON Instructions perform Packed SIMD processing: Registers are considered as vectors of elements of the same data type Data types can be: signed/unsigned 8-bit, 16-bit, 32-bit, 64-bit, single prec. float Instructions perform the same operation in all lanes As it can be seen, for adding R1 to R2, it performs the single instruction of adding on multiple data (four lanes) thus providing speedup. 12

13 Using the NEON optimization: -- In order to instruct the compiler to produce NEON you should use the following compile flags: -mfpu=neon -- To specify which floating-point ABI to use, permissible values are: `soft', `softfp' and `hard'. Specifying `soft' causes GCC to generate output containing library calls for floating-point operations. `softfp' allows the generation of code using hardware floatingpoint instructions, but still uses the soft-float calling conventions. `hard' allows generation of floating-point instructions and uses FPU-specific calling conventions. -mfloat-abi=name -- To specify the name of the target ARM processor. GCC uses this name to determine what kind of instructions it can emit when generating assembly code. -mcpu=cortex-a8 -- If you want the compiler to attempt to vectorize your integer / floating point code for the NEON you should add: -ftree-vectorize to your flags. -ftree-vectorize Using all these alongwith the O3 optimization, this was the result I got on the same single threaded sharpen code on the 640x480 image. The NEON instructions provided a speedup of around 3X! 13

14 Multi-threading: Multi threading does give better results especially when higher amount of processing is involved. Threading helps in speeding up the process. I have written a code to modify/process any sized image into any desired number of thread blocks, and get the necessary speedup. This was the result, when the image was sharpened using single-threading. When multiple threads are used to process the image at its different locations, the following result is observed on the same 640x480 image. (Using 4x3 threads) Multithreading using 8x6 threads: Thus, there is a speed up when 3x4 threads are used as compared to 6x8 threads. 14

15 Following are the results of some image processing done on the images captured by the Leopard camera. Some other transformations: 1. Centroid <-> Thus, the neon provides a huge speedup in performance. 15

16 2. Boost The following output shows the speedup in performance using NEON. <-> 16

17 Image upscaling: I wrote a code to convert 320x240 images to 640x480 images. The method I used was nearest neighbor averaging. Now for example, we have to transform a 2x2 to a 4x4. The new added pixel will be the average of the neighboring pixels. Once, the neighboring pixels method was carried out, I applied the averaging filter, to smoothen out edges (if any) Similarly, for a conversion from 640x480 to 1280x720, the neighboring pixels method is used, where the ratio is taken into account. 640/1280 = (1/2) 480/720 = (2/3) Every new pixel in the up scaled image corresponds to the (1/2)th location in the original images width while, to the (2/3)rd location in its height. 17

18 THIS IS A TUTORIAL TO HELP YOU PORT THE UBUNTU OPERATING SYSTEM ON THE BEAGLEBOARD xm 18

19 Porting UBUNTU on the BeagleBoard: Following are the steps to port another operating system, Ubuntu, on the BeagleBoard Xm. Steps: Make your home directory to store all the files for the Ubuntu operating system. (root files) Plug in your micro SD card onto your host laptop system. Now we need to get the armel: wget md5sum ubuntu r1-minimal-armel.tar.xz tar xvf ubuntu r1-minimal-armel.tar.xz Instead of the above commands, you can use the files in my folder Ubuntumake. Now, enter the following: cd ubuntu r1-minimal-armel.tar.xz sudo./setup_sdcard.sh --mmc /dev/sdb --uboot beagle_bx --rootfstype ext3 19

20 This process includes formatting your SD card, transferring boot files into the boot partition and the rootfilesystem onto the other partition of the memory card. 20

21 When it is done, remove the SD card, and plug it back after 2-3 seconds. You will see two partitions, as listed below. Remove the SD card from the card holder and plug it in the Beagleboard and switch on the power. Connect the Serial (BeagleBoardxM) to USB(Host PC) converter and configure the minicom connection. Start minicom on your host machine in configuration mode. As root: #minicom -o -s -w A menu of configuration should appear. Use the Down-arrow key to scroll down and select the Serial port setup option, and press Enter. Verify that the listed serial port is the same one that is connected to the target board. If it is not, press A, and enter the correct device. This is /dev/ttyusb0 on most Linux distributions. 21

22 Set the Bps/Par/Bits option by pressing the letter E and using the next menu toset the appropriate values. You press the key that corresponds to the value , and then press Enter. Set Hardware flow control to No using the F key. Set Software flow control to No using the G key. Press Enter to return to the main configuration menu, and then press Esc to exit this menu. Reset the board, and wait for a moment. If you do not see output from the board, press Enter several times until you see the prompt. If you do not see any output from the board, and have verified that the serial terminal connection is setup correctly, contact your board vendor. You will see something like this when the kernel boots. Once, it has booted, it will ask for a login, and a password. The login is ubuntu, and the password is temppwd. 22

23 It is necessary that you update your kernel using the following command. To get the Ethernet working, enter the following command. sudo dhclient eth0 You can check your assigned ip-address using ifconfig a 23

24 References: 1. Nisheeth Bhatt s IS 2. Srinivas Panchpakesan s work on the Leopard camera. 24