CSE 237B Project Report Implementing KLT Algorithm on IPAQ. 1 Implementing KLT Algorithm on ipaq Problem Statement... 2

Size: px

Start display at page:

Download "CSE 237B Project Report Implementing KLT Algorithm on IPAQ. 1 Implementing KLT Algorithm on ipaq Problem Statement... 2"

Robyn Lloyd
6 years ago
Views:

1 CSE B Project Report Implementing KLT Algorithm on IPAQ 0 Implementing KLT Algorithm on ipaq Implementing KLT Algorithm on ipaq.... Problem Statement.... Introduction.... Project Plan.... KLT Algorithm Description..... KLT Feature selection..... KLT Feature Tracking.... KLT Algorithm Implementation..... Design Considerations..... Implementation Decision.... Performance Metrics..... Impact of Image resolution on Feature selection and Feature Tracking time Feature Tracking time wrt number of features to track..... Code Optimization.... Summary.... Future Work.... References:....0 Appendix... 0

2 CSE B Project Report Implementing KLT Algorithm on IPAQ 0 0. Problem Statement The aim of this project is to accomplish following tasks Implement Kanade-Lucas-Tomasi (KLT) Feature tracking algorithm on IPAQ Integrate IPAQ with Axis Network Camera which can provide streaming data to IPAQ and use this data to track the features. To Measure the performance of KLT algorithm and optimize it.. Introduction KLT algorithm is used in Computer Vision community to track the features in an image. The KLT tracker is based on the early work of Lucas and Kanade []. There are basically two steps which have to take place during feature tracking: Selecting Good features to track in an image. Tracking the features in an image sequence. Good features are located by examining the minimum eigenvalue of each by gradient matrix, and features are tracked using a Newton-Raphson method of minimizing the difference between the two windows. Multiresolution tracking allows for even large displacements between images. When features are lost the algorithm replaces the lost features by finding other new features.. Project Plan ) Study of KLT algorithm. ) Implementation of KLT (Kanade-Lucas-Tomasi ) algorithm for feature selection / tracking on IPAQ. ) Integration of Axis web camera with IPAQ and provide feedback to the camera to move in the direction of the movement of the image. ) To measure Performance benchmarks on the KLT algorithm.

3 CSE B Project Report Implementing KLT Algorithm on IPAQ. KLT Algorithm Description.. KLT Feature selection... Introduction to feature selection The most important step before tracking is the selection of trackable features. How do we select features which are trackable? The keyword during feature selection is texture of the image. Areas with a varying texture pattern are mostly unique in an image, while uniform or linear intensity areas are often common and not unique. The aim of the KLT algorithm is to find the coordinates in the image which has a varying texture Finding a good feature A texture pattern can only exist if we look at multiple pixels in an area. The features that we are tracking are more accurately described as feature windows containing texture information. The area of such a feature window can vary, depending on the number of features which needs to be located. If we consider a feature window of size W ( say x pixels). Then the varying texture pattern can be captured by the image gradient. The image gradient is defined as: g = g g x y For example when image Fig is used to find the selectable features. The The image gradients in X and Y coordinates are shown in figures and.

CSE B Project Report Implementing KLT Algorithm on IPAQ Figure The original image for which the

gradient in Y axis gg T = gx gxg y g xg y g y If we now integrate the matrix derived above over the area

analyzing its eigenvalues we can classify the texture in the region W.

4 CSE B Project Report Implementing KLT Algorithm on IPAQ Figure The original image for which the selectable features need to calculated Fig Image gradient in X axis Now consider the product Fig Image gradient in Y axis gg T = gx gxg y g xg y g y If we now integrate the matrix derived above over the area W, we get: Z = g W gxg x x y g g y wdx g y 0 The x matrix Z contains pure texture information and by analyzing its eigenvalues we can classify the texture in the region W. Two small eigenvalues represents a roughly constant intensity pattern within W. One small and one big eigenvalue means that a linear (unidirectional)

CSE B Project Report Implementing KLT Algorithm on IPAQ pattern was detected. If Z has two large eigenvalues, then the area W contains a feature which can be tracked.

5 CSE B Project Report Implementing KLT Algorithm on IPAQ pattern was detected. If Z has two large eigenvalues, then the area W contains a feature which can be tracked. This feature can either be a corner, a so called salt-and-pepper texture or any texture pattern which has a prominent enough change in intensity in more than one direction. The equation for Z forms an intricate part of the Kanade-Lucas-Tomasi tracking algorithm.it is necessary to establish a minimum threshold for the value of the eigenvalues. In conclusion, if the two eigen values of Z are λ and λ we accept a window if min ( λ, λ ) > predefined threshold λ Figure shows the features selected by the algorithm for its first image figure. 0 Fig Shows which features are selected for feature tracking.. KLT Feature Tracking 0... Introduction to the method In a nutshell the technique can be described as follows: First of all features have to be selected which can be tracked from image to image in a video image stream. The selection of the features is based on texture (intensity information). A number of fixed-sized feature windows are selected on the first image of a sequence. These feature windows are tracked from one image to the next using the KLT method []. This method calculates the sum of squared intensity differences between a feature in the previous image and the features in the current image. The displacement of the specific feature is then defined as the displacement that minimizes the sum of differences. This is done continuously between sequential images so that all the features can be tracked.

6 CSE B Project Report Implementing KLT Algorithm on IPAQ 0... Tracking features The fundamentals of tracking can be explained by looking at two images in an image sequence. Let us assume that the first image was captured at time t and the second image at time t + τ. It is important to keep in mind that the incremental time τ depends on the frame rate (capture rate) of the video camera and should be as small as possible. A higher frame rate allows for better tracking. A grayscale image is a pattern of intensities where the intensity values range from 0 to.an image can be represented as function of variables x and y. We add the variable t as the time the image was captured. Thus, any section of an image can be defined by an intensity function I(x,y,t). If we now define a window in an image taken at time t+τ as I(x,y,t+τ ). The basic assumption of the KLT tracking algorithm is I(x,y,t+τ ) = I ( x - x, y - y, t) () From () it is clear that every point in the second window can by obtained by shifting every point in the first window by an amount (x, y). This amount can be defined as the displacement d = (x, y) and the main goal of tracking is to calculate d Calculating feature displacement The basic information has now been established to solve the displacement d of a feature from one window to the next. For simplicity, we redefine the second window as B(x ) = I(x,y,t+τ ) and the first window as A(x - d) = I(x - d) = I(x-x, y-y,t) where x = (x,y). The relationship between the two images is given by B(x) = A(x-d) + η (x) () η (x ) is a noise function caused by interference. An error function which has to be minimized in order to minimize the effect of noise: = W [A(x-d) B(x)] [A(x-d) B(x)] wdx () As we see is a quadratic equation of d. To minimize we need to differentiate with d and set the result to 0. d / dd = W [ A(x) B(x) T g d] gwda = 0 () Where A refers to area of window W. If the terms of are rearranged and if we use the fact that T ( g d)g = (g g T )d it follows that ( W g g T wda) d = W [ A(x) B(x) ] gwda ()

7 CSE B Project Report Implementing KLT Algorithm on IPAQ Equation is known as the Kanade-Lucas-Tomasi tracking equation and can be rewritten as follows: Zd = e () Where Z is a X matrix and e is a vector in two dimensions. The displacement is now simply the solution of equation. 0. KLT Algorithm Implementation The Algorithm is implemented on ipaq. The ipaq connects with a linux host over USB / Serial connection to get the images captured by the axis network camera. The layout of integration is shown in fig. The detailed configuration of ipaq, linux host and the network camera is given in the Appendix section. HTTP Protocol Converts compressed Image formats to PGM Network Linux Host Camera D Image USB/Serial KLT Algorithm On Familiar Linux. Fig Component overview of the Project Implementation.

Network camera gives the images in compressed format ( JPEG format / MJPEG format).

The IPAQ which was used for implementation doesn t have Ethernet connectivity or Blue Tooth

. Implementation Decision Considering the above mentioned challenges a linux/window host interfaces

This solved the above mentioned challanges in following ways: The network camera streams the data to

By doing this the IPAQ doesn t waste any CPU cycles in converting JPEG to PGM format.

8 CSE B Project Report Implementing KLT Algorithm on IPAQ.. Design Considerations KLT algorithm uses images of PGM ( Portable gray map ) format and the Axis Network camera gives the images in compressed format ( JPEG format / MJPEG format). Axis Network camera doesn t support serial or USB connectivity. The IPAQ which was used for implementation doesn t have Ethernet connectivity or Blue Tooth connectivity. 0.. Implementation Decision Considering the above mentioned challenges a linux/window host interfaces between the Network Camera and the IPAQ. This solved the above mentioned challanges in following ways: The network camera streams the data to the linux host which does the conversion of JPEGs to PGM format. By doing this the IPAQ doesn t waste any CPU cycles in converting JPEG to PGM format. This may not be necessary if the network camera code can modified to output PGM formats. Check Future works section. The IPAQ connects to the linux host using USB and downloads the PGM images at the rate at which it is capable of processing. The linux host downloads the MPJEG stream over HTTP interface and converts it into PGM format stream. The IPAQ gets the images from the linux host and then runs the KLT algorithm to track the features on downloaded images. The algorithm replaces the lost features in an image frame with the new set of features. ( As seen in Fig and Fig ) 0 Fig : Set of images downloaded from the server. The image is moving in the left direction. Fig: The KLT algorithm successfully tracks the frames in the subsequent set. If any features are lost it replaces it with new features for it to be tracked in the next image sequence.

9 CSE B Project Report Implementing KLT Algorithm on IPAQ 0 As it can be seen from the above figures that the KLT is able to successfully track the feature which is moving in the left direction. The KLT algorithm marks the tracked feature with respect to x,y coordinate for each frame in the image sequence. Using this data the implementation code calculates the displacement of the tracked features and hence the displacement coordinates of the actual image. The original idea of the Project was to track the features and move the camera in the direction of the image movement. Since the Model of Network camera used in the Implementation doesn t support the PTZ ( Pan Tilt Zoom) control commands to Tilt and move the camera so this part could not be implemented. In current implementation it just prints the displacement coordinates in the standard out. 0. Performance Metrics Following performance metrics are analyzed for the system.. Impact of Image resolution on Feature selection and Feature Tracking time.. Feature selection time vs Feature Tracking Time. Feature Tracking time wrt number of features to track.. Algorithm parameters optimizations. Posix method clock_gettime( clock_id, timespec) call is used to measure the performance metrics of the system. Check Appendix for more information. The configuration of the linux host, ipaq and the network camera is given in the appendix section.

10 CSE B Project Report Implementing KLT Algorithm on IPAQ.. Impact of Image resolution on Feature selection and Feature Tracking time. Table shows the performance metrics collected for different image resolution. Image Resolution Feature Selection Time in on ARM Feature Tracking Time on ARM Feature Selection Time in on x Feature Selection Time in on x ( in millisecs) (in millisecs) ( in millisecs) ( in millisecs) 0x0 0. 0x0 0 0x0 0x Time in milli secs x0 0x0 0x0 0x0 Feature Selection Time Feature Tracking Time Image Resolution Fig : shows the graphical view of performance metrics for ARM Processor as depicted in Table. 0

11 CSE B Project Report Implementing KLT Algorithm on IPAQ Time in milli secs Feature Selection Time Feature Tracking Time x0 0x0 0x0 0x0 Image Resolution Fig : shows the graphical view of performance metrics on x system as depicted in Table. This is just to compare the performance values between ARM processor and x linux host.... Analysis It seems clear from the graph that the Increasing image resolution is indirectly impact on performance time of Feature tracking algorithm. The Feature Tracking part seems to be more severely effected than the Feature selection part by increasing the image resolution. In general the tracking of features takes more time then the selection of features Explanation and Conclusion: The KLT divides the actual image into various small windows ( for example x pixels) and then calculates the intensity gradient for each window and compares it with threshold eigen value. By increasing the resolution of the image the computation is directly impacted as intensity gradient for more windows need to be calculated. The x results look very promising as the image resolution decreases. But the performance on ARM is still very dismal. The reason is because the image intensity gradient calculation is very computing intensive and uses floating point calculation. The x has a floating point processor but the IPAQ doesn t have a floating point processor. It seems evident from the KLT algorithm that the tracking time is more computing and memory intensive than the feature selection part. The reason why tracking takes more time is because for tracking, the image is sub sampled couple of times unless it reaches

12 CSE B Project Report Implementing KLT Algorithm on IPAQ a certain pyramid level. KLT algorithm sets the sub sample level to be atleast. So regardless of pyramid level the image will be sampled at least twice for tracking the features. From this graph it can be concluded that the max frame rate which an ipaq can handle with image resolution of size 0x0 is approx - frames per second... Feature Tracking time wrt number of features to track. Feature Selection Vs Feature Tracking Time on x system on a 0x0 image 0 Time in millisecs # of features to select/track Feature Selection Time Feature Tracking time 0 Time in millisecs Feature Selection Vs Feature Tracking Time on ARM on a 0x0 image # of features to select/track Feature Selection Time Feature Tracking time... Analysis ) It seems evident that the number of features to select and track doesn t affect the performance time

13 CSE B Project Report Implementing KLT Algorithm on IPAQ 0... Explanation and Conclusion As explained earlier the KLT algorithm divides the image into various small window and then calculates the image intensity gradient (eigen values) for each smaller window. It then sorts each of the probable features with the decreasing values of eigen values. The n selectable features are the top n features of this sorted list. So regardless of the number of features to select it always have to compute the intensity gradients for the whole image. But it is still logical to give some practical number for feature selection because if the KLT is configured to replace the lost features from one image to another then the probability of feature loss will be more if the number of features to track is more. This in turn will be more expensive because more often the Replace Lost Features algorithm will be called which has the same complexity as Selection Feature List Algorithm Code Optimization The KLT implementation takes various parameters. The value of these input parameters impacts the performance. The parameters which gave significant performance benefits are described as following : TrackingContext->MinDistance: The minimum distance between each feature being selected, in pixels. Used by KLTSelectGoodFeatures() and KLTReplaceLostFeatures(). If this value is big in higher resolution images then the selection and replacing of features will be faster. This value has to modified carefully as it may increase in number of lost features. TrackingContext->window_width, window_height: The size of the feature window, in pixels. KLT algorithm divides the image into various small window and then calculates the image intensity gradient (eigen values) for each smaller window. Each small window is of size window_width x window_height. Higher values gives better result. This value has to modified carefully as it may increase in number of lost features. sequentialmode If TRUE, then the previous image is saved and used later. Used by KLTTrackFeatures() and KLTReplaceLostFeatures() to speed the computation when tracking through an image sequence. writeinternalimages If TRUE, then the internal images used for feature selection and tracking, that is, the smoothed and differentiated versions of the original images, are written to files. For performance benefits this value should be set to false.

14 CSE B Project Report Implementing KLT Algorithm on IPAQ 0 nskippedpixels The number of pixels in between each pair of possible features. Used to speed up the computation of KLTSelectGoodFeatures() npyramidlevels The number of pyramid levels. The lower values reduces the tracking time. Debugging statements : Removing Debugging statements helped the performance. Compiler optimizations: Compiler Optimizations O feature enhances the performance of the system.. Summary The KLT algorithm is successfully implemented on IPAQ (Strong ARM).However its performance is really bad on IPAQ. The KLT Implementation on IPAQ is only able to successfully track images of resolution (0x0) with maximum rateof frames per seconds 0 0. Future Work ) It will be worth to try the following source forge net project which implements a library to support execution in fixed point arithmetic to allow for fast execution on systems with no floating point processor. To overcome the problems of integer overflow, the library calculates a position of the decimal point after training and guarantees that integer overflow can not occur with this decimal point. The link for this project is The following anecdote by the author seems very interesting as it seems he faced the same performance problem which we are facing. In [Nissen et al., 00] I participated in building and programming of an autonomous robot, based on an Compaq IPAQwith a camera attached to it. In [Nissen et al., 00] I participated in rebuilding this robot and adding artificial neural networks (ANN) for use in the image processing. Unfortunately the ANN library that we used [Heller, 00] was too slow and the image processing on the IPAQwas not efficient enough. The IPAQdoes not have a floating point processor and for this reason we had written a lot of the image processing using fixed point arithmetic. From this experience I have learned that rewriting code to fixed point arithmetic, makes a huge difference on the performance of programs running on the ipaq

15 CSE B Project Report Implementing KLT Algorithm on IPAQ ) Implementation of KLT Algorithm o a board which supports floating point processor ) Implementation of KLT Algorithm on ETRAX. Axis camera uses Etrax chip which runs embedded linux OS. The whole algorithm can be implemented in the network camera itself. From the specifications it seems very promising ) Some effort can be given in reducing the complexity of the KLT-Algorithm. References: [] Stan Birchfield, Derivation of Kanade-Lucas-Tomasi Tracking Equation, Unpublished, May. [] Carlo Tomasi and Takeo Kanade, Detection and Tracking of Point Features, Carnegie Mellon University Technical Report CMU-CS--, April. [] [] Appendix ) Clock Details Clock_gettime(timerId, timespec) o Clock_gettime(CLOCK_ID, timespec)gets the current value of specific clock o Clock_idsclock_realtime,clock_monotonic, clock_process_cputime,clock_thread_cputime Clock_getres( CLOCK_ID, timespec) o Gets the resolution for the specific clock_id o resolution is 0 msecs for clock_real_time

16 CSE B Project Report Implementing KLT Algorithm on IPAQ o Linux implementation on ARM supports only CLOCK_REALTIME ) ipaq Configuration details ~ # cat /proc/cpuinfo Processor : StrongARM-0 rev (vl) BogoMIPS :. Features : swp half bit fastmult CPU implementor : 0x CPU architecture: CPU variant : 0x0 CPU part : 0xb CPU revision : Hardware : HP ipaq H00 Revision : 0000 Serial : Memory Info ~ # cat /proc/meminfo total: used: free: shared: buffers: cached: Mem: Swap: MemTotal: 00 kb MemFree: kb MemShared: 0 kb Buffers: 0 kb Cached: 0 kb SwapCached: 0 kb Active: kb Inactive: kb HighTotal: 0 kb HighFree: 0 kb LowTotal: 00 kb LowFree: kb SwapTotal: 0 kb SwapFree: 0 kb ) Linux Host Configuration details: Linux ajitc-linux el # Thu Oct 0:: EDT 00 i

17 CSE B Project Report Implementing KLT Algorithm on IPAQ i i GNU/Linux ajitc-linux:/local/mnt/workspace/cseb->cat /proc/cpuinfo processor : 0 vendor_id : GenuineIntel cpu family : model : model name : Intel(R) Pentium(R) CPU.0GHz stepping : cpu MHz :. cache size : KB fdiv_bug : no hlt_bug : no f00f_bug : no coma_bug : no fpu : yes fpu_exception : yes cpuid level : wp : yes flags : fpu vme de pse tsc msr pae mce cx apic sep mtrr pge mca cmov pat pse clflush dts acpi mmx fxsr sse sse ss ht tm bogomips :. ajitc-linux:/local/mnt/workspace/cseb->cat /proc/meminfo total: used: free: shared: buffers: cached: Mem: Swap: 0 0 MemTotal: kb MemFree: kb MemShared: 0 kb Buffers: 0 kb Cached: 0 kb SwapCached: kb Active: 0 kb ActiveAnon: 0 kb ActiveCache: kb Inact_dirty: 0 kb Inact_laundry: 0 kb Inact_clean: kb Inact_target: kb HighTotal: 0 kb HighFree: 0 kb LowTotal: kb LowFree: kb

18 CSE B Project Report Implementing KLT Algorithm on IPAQ SwapTotal: 000 kb SwapFree: 0 kb HugePages_Total: 0 HugePages_Free: 0 Hugepagesize: 0 kb

Face Tracking : An implementation of the Kanade-Lucas-Tomasi Tracking algorithm

Face Tracking : An implementation of the Kanade-Lucas-Tomasi Tracking algorithm Dirk W. Wagener, Ben Herbst Department of Applied Mathematics, University of Stellenbosch, Private Bag X1, Matieland 762,