Traffic Surveillance using Aerial Video

Transcription

1 Traffic Surveillance using Aerial Video UGP Project III (EE491) Report By Anurag Prabhakar(11140) Ritesh Kumar(11602) Under the guidance of Dr. K. S. Venkatesh Professor Department of Electrical Engineering Indian Institute of Technology, Kanpur November 2014 Indian Institute of Technology, Kanpur 1

2 Acknowledgement We would like to express our deep sense of gratitude to Prof. Dr. K. S. Venkatesh, for his invaluable help and guidance during the course of the project. I am highly indebted to him for his constant encouragement us by giving his critics on my work. I am grateful to him for having given me the support and confidence. We would also like to thank Mr. Narendra and people from Flight Lab for helping us get clear videos using the Drone camera. Anurag Prabhakar, Ritesh Kumar, Department of Electrical Engineering, Indian Institute of Technology, Kanpur Indian Institute of Technology, Kanpur 2

3 Contents 1. Abstract 4 2. Introduction Parrot AR2 Drone Optical Flow Algorithms Adopted Overall Algorithm Identification Of vehicles and Speed estimation Offsetting Camera Motion Results Conclusions References...17 Indian Institute of Technology, Kanpur 3

4 Abstract Remotely sensed data using aerial video has great potential to enhance existing data sources and therefore to improve traffic management. While the use of remotely sensed data to monitor traffic flows is not new, the aim here is to examine the integration of aerial video extraction and automated image processing to improve the accuracy and cost effectiveness of the data collection process. Several different aerial platforms are under investigation for the data collection. With these platforms, a number of experiments in which aerial video was collected from one of the crossings of the IITK campus (Old SAC area) are described. A technique has been developed to process the digital imagery, in near real time, to estimate vehicle speeds directly from the video images. The algorithm is capable of detecting vehicles and measuring their speeds, using Image Registration, Edge Detection and Optical Flow techniques. When vehicles are matched between two frames, their speeds can be estimated directly. In addition, similar techniques can be used to derive other traffic flow parameters, such as densities, travel times, delays, turning counts, queue lengths, and measures of platoon dispersion. The automated image processing significantly reduces the amount of time needed to generate these traffic data. Indian Institute of Technology, Kanpur 4

5 Introduction Traffic flows in many large urban areas are monitored and managed using data from a variety of sensors. The traditional technologies for traffic sensing, including inductive loop detectors and video cameras, are positioned at fixed locations in the transportation network. While these detectors do provide useful information and data on the traffic flows at particular points, they generally do not provide useful data for traffic flows over space. It is not possible to move the detectors, and they are not capable of providing data on vehicle trajectories and paths through the network. It is also difficult to use these sensors to identify traffic flow characteristics (speeds, acceleration/deceleration, and routing information) of individual vehicles. The use of aerial platforms (in our case, Drones) for traffic data collection has appeal to supplement the traditional traffic sensors. Aerial photography and video has the properties of being both mobile, so that it may be deployed where needed to collect traffic data, and of capturing movements of individual vehicles in both time and space. When used in conjunction with traditional ground-based sensors, more comprehensive data can be collected for traffic monitoring and management. Indian Institute of Technology, Kanpur 5

6 Parrot AR2 Drone Specifications of Input Video: Resolution: 1280x780 (through horizontal camera, after getting it tilted. Initially we took videos from vertical camera but it had rather low resolution. So we got the horizontal camera tilted to face downwards with the help of Flight Lab, IITK.) Data Rate: 4189 kbps Frame Rate: 30 frames/ sec App: AR2 Free Flight (Android) Indian Institute of Technology, Kanpur 6

7 Optical Flow Optical flow is the pattern of apparent motion of image objects between two consecutive frames caused by the movemement of object or camera. It is 2D vector field where each vector is a displacement vector showing the movement of points from first frame to second. Consider the image below : It shows a ball moving in 5 consecutive frames. The arrow shows its displacement vector. Optical flow has many applications in areas like : Structure from Motion Video Compression Video Stabilization Optical flow works on several assumptions: 1. The pixel intensities of an object do not change between consecutive frames. 2. Neighbouring pixels have similar motion. Consider a pixel I(x,y,t) in first frame (Taking into account the time dimension since we re working with videos). It moves by distance (dx,dy) in next frame taken after dt time. So since those pixels are the same and intensity does not change, we can say, Indian Institute of Technology, Kanpur 7

8 Then take taylor series approximation of right-hand side, remove common terms and divide by to get the following equation: where: Above equation is called Optical Flow equation. In it, we can find fx and fy, they are image gradients. Similarly ft is the gradient along time. But (u,v) is unknown. We cannot solve this one equation with two unknown variables. So several methods are provided to solve this problem : 1. Sparse Optical Flow - Lucas-Kanade method 2. Dense Optical Flow - Farnerback method LUCAS-KANADE METHOD We have seen an assumption before, that all the neighbouring pixels will have similar motion. Lucas-Kanade method takes a 3x3 patch around the point. So all the 9 points have the same motion. We can find for these 9 points. So now our problem becomes solving 9 equations with two unknown variables which is over-determined. A better solution is obtained with least square fit method. Below is the final solution which is two equation-two unknown problem and solve to get the solution. Indian Institute of Technology, Kanpur 8

9 So from user point of view, idea is simple, we give some points to track, we receive the optical flow vectors of those points. But again there are some problems. Until now, we were dealing with small motions. So it fails when there is large motion. So again we go for pyramids. When we go up in the pyramid, small motions are removed and large motions becomes small motions. So applying Lucas-Kanade there, we get optical flow along with the scale. DENSE OPTICAL FLOW Lucas-Kanade method computes optical flow for a sparse feature set (in our example, corners detected using Shi-Tomasi algorithm). OpenCV provides another algorithm to find the dense optical flow. It computes the optical flow for all the points in the frame. It is based on Gunner Farneback s algorithm which is explained in Two-Frame Motion Estimation Based on Polynomial Expansion by Gunner Farneback in Result of Lucas-Kanade on a traffic image Indian Institute of Technology, Kanpur 9

10 Algorithms Adopted The main contribution of our technique is to compensate for the motion of the camera platform. To do this, the idea is to detect individual vehicles in a frame, to register consecutive frames, and then to employ image transformation, image subtraction, and object matching to track individual vehicles 5.1 OVERALL ALGORITHM Explanation of above flowchart: 1. Two frames of 1 or 2 sec of time shift are extracted from the video and are pre-processed by converting into grayscale to accelerate the process. Indian Institute of Technology, Kanpur 10

11 2. Standard edge detection is performed on the frames and the frames are parsed to search for the concentration of white pixels, which outline both moving and stationary objects. 3. The second frame is registered to the first one using offset of camera motion using optical flow and the two frames are overlapped and subtracted. 4. After subtraction, only vehicles should have pixel values different from zero in the overlapping portion of the two frames. However, in many cases stationary objects may be seen as moving objects (pixel values different from zero) because of limitations in georeferencing accuracy. This is solved by setting a minimum speed threshold for moving objects (e.g. only objects with a displacement of more than 20 pixels). 5.2 VEHICLE MATCHING AND SPEED ESTIMATION Matching the cars in the two frames can be done using a minimum cost matching algorithm. After matching the vehicles, vehicle trajectories can then be derived from the time and position information in order to obtain traffic parameters such speed, density, spacing, etc. The following greedy algorithm achieves vehicle identification, with its accuracy depending on a threshold and the density of traffic on the road (with smaller thresholds and lesser traffic density giving better results). 1. Take a vehicle in the first frame if it has not yet been selected and initialize a variable minimum_distance to a very high value. 2. Assume the speed at which this vehicle should travel and determine the predicted location of this vehicle in the second frame. The algorithm is fairly robust to the assumed speed, when the speeds are consistent across vehicles (e.g., on a freeway). One might expect performance to be degraded for less consistent speeds. 3. Take each vehicle of frame 2 in turn and determine the Euclidian distance between the predicted location of the vehicle in the 1 st frame and the current vehicle in the 2 nd frame. 4. If this distance is below a certain threshold limit, and it is less than minimum_distance, then record that value and the car number as the possible match for the car in frame 1. Update the value of minimum_distance to this new value. 5. After all the cars in frame 2 have been checked, and if a match is found for the car in frame 1, then make this tentative match as permanent. 6. Repeat the above steps till all the cars in frame 1 are evaluated, given a successful or unsuccessful match. Indian Institute of Technology, Kanpur 11

12 5.3 OFFSETTING CAMERA MOTION Approach: Observing that the still background will move in exactly the opposite direction of the camera but moving vehicles will move in random directions, we use a dense Optical Flow Algorithm to differentiate the foreground and the background. Thereon, the problem is reduced to regular frame difference and subsequent blob detection. From the implementation of calcopticalflowfarneback() we can get velocity vector for each pixel of the frame. This velocity vector (fxy) has two components fxy.x and fxy.y giving the displacement of each pixel in x and y direction respectively. Analysing the values of fxy.x and fxy.y for different pixel, we develop a filter that minimum pixel shift in moving vehicle should be 20 pixels in any direction, this filter out motion of some of the stationary ie., fxy.x>20 or fxy.y>20. Since, the motion of the background was very arbitrary and abrupt we were unable to filter all the background motion, so we tried using some filter on fxy.x and fxy.y combined by taking the atan(fxy.y/fxy.x) and thus the direction of motion of each pixel which passed the previous filter. We divided the 360 degree into 10 smaller division of 36 degree each and all the pixel are grouped according to their direction values. Then we calculated the direction with most numbers of pixel and subtracted that direction from the all other pixel to cancel the motion of background. Indian Institute of Technology, Kanpur 12

13 Results SETUP We set the max height to around 15m and flew the drone around the SAC area. First we had images of only the road in the frame, but it was very difficult to do background subtraction with that, so we took more videos with a greater average height, around 20m. This time, we could get the roadside into the frames as well, and the background subtractor worked well. In the earlier case, objects on the road couldn t be differentiated from moving vehicles (in random directions) because background couldn t be identified. SELECTED FRAMES Indian Institute of Technology, Kanpur 13

14 Indian Institute of Technology, Kanpur 14

15 The estimated speeds of the matched vehicles were calculated using the image scale and the time between frames. For motorized vehicles, typical vehicle speeds feel around the 30 kmph mark (slow speed due to place being a crossing). Highly erroneous speeds is a possibility when the traffic density is high and there are high chances of wrong matching of vehicles, but the same did not happen in our case as there were hardly more than 2 vehicles in a frame at a time. Such an erroneous speed could be discovered by a simple filter. Indian Institute of Technology, Kanpur 15

16 Conclusions A simple technique for vehicle identification and speed estimation is given. This technique shows promise that it may be run in real time on on-boards processors in the near future, allowing one to estimate individual vehicle speeds from aerial imagery on a more or less continuous basis. Further research can be done to analyze the performance of given algorithms in terms of: CPU (image processing) times as a function of image resolution, scale, and field of view. Communication requirements to send the results to a ground station; Detection error rates; and, Tracking error rates. Indian Institute of Technology, Kanpur 16

17 References 1. Anurag Mittal, Dan Huttenlocher; Scene Modeling for Wide Area Surveillance and Image Synthesis ; Computer Science Department; Cornell University 2. Eric Hayman and Jan-Olof Eklundh, Statistical Background Subtraction for a Mobile Observer, Computational Vision and Active Perception Laboratory (CVAP), KTH, SE Stockholm, Sweden 3. Sedat Doğan, Mahir Serhan Temiz and Sıtkı Külür, Real Time Speed Estimation of Moving Vehicles from Side View, Images from an Uncalibrated Video Camera; Department of Geodesy and Photogrammetry, Engineering Faculty, Ondokuz Mayis University. Indian Institute of Technology, Kanpur 17