ENGR3390: Robotics Fall 2009

Transcription

1 J. Gorasia Vision Lab ENGR339: Robotics ENGR339: Robotics Fall 29 Vision Lab Team Bravo J. Gorasia - 1/4/9

2 J. Gorasia Vision Lab ENGR339: Robotics Table of Contents 1.Theory and summary of background readings Description of experimental work...4 Tracking the red ball...4 Following the road Presentation of robotic software used...5 Tracking the red ball...5 Following the road Technical analysis of raw experimental results Conclusions supported by condensed results Appendix Student revised lab write up...13 Table of Figures Code to find the ball in the image and find its size...5 Block diagram of the algorithm which converts the position of the ball from 2D image space to 3D world space...6 Ball display code. The use of many different graphs helps in debugging...6 Front panel for the tracking the red ball experiment...7 Main VI. This displays the source image, the filtered image, the Hoff transform line fitting results and the final heading. It provides auxiliary information like the frame rate as well...8 Block diagram for main panel. It is a simple while loop while runs through a cycle of load images, filter images for the road, line fitting to the road, and a calculation where the direction vector is....9 Video Images.vi. This sub-vi loads the road images from the directory and sends them out in sequence...1 Filter Images.vi. The Images come in and are filtered...1 Hough Transform code. This takes an array of pixels and finds the line that best...11 Image to vector.vi. Converts the 2D array into a list of coordinates...11 Transform to lines.vi. Takes the coordinates of pixels and finds the lines that will pass through them...11 Calculate theta.vi. This sub-vi averages the gradients of the two incoming lines...11 Direction overlay.vi. Overlays an arrow over the original image...12

3 J. Gorasia Vision Lab ENGR339: Robotics Executive Summary We explored implementing machine vision within Labview with two different experiments: following the red ball, and finding the road. The two experiments revealed the sequence that most machine vision processes must execute: get image, filter image, process image to find what is desired and use those findings in a meaningful way. They also revealed how different situations will require different algorithms, as current sensors and algorithms are not as universal as the human eye in its ability to perceive images and the human brain in processing images. All in all, the lab showed the limitations in current machine capabilities, and helped explore ways to overcome them. While Labview does provide many prebuilt tools to assist the roboticist, they are not trivial to use effectively.

4 J. Gorasia Vision Lab ENGR339: Robotics 1. Theory and summary of background readings Since its emergence in the 194s, machine vision has emerged as an essential component in industrial processes as a tool for inspection. Within robotics, vision has been used as a powerful sensor for path recognition, object identification and such 1. The vision ability is a combination of cameras, algorithms and calibration. Calibration is important as current vision ability is application specific, meaning that different techniques need to be used for different situations. Since it relatively low cost and powerful, cameras and vision are important tools in the mobile roboticist's toolbox. Labview provides a vision toolbox which greatly simplifies the implementation of vision algorithms within Labview 2. In addition, Labview provides the Vision Assistant which makes it faster to prototype different algorithms with its GUI. This report details work done in two different vision experiments, tracking a ball and following a road. They reveal challenges in machine vision, and detail methods to achieve the designated tasks. Through these experiments, I hope to learn about issues currently facing robot vision, and how to work with them. 2. Description of experimental work Tracking the red ball A Point Grey 3 stereo camera is pointed at a spherical red ball which is against a plain white background. Through analysis of the image provided by the camera, the position of the ball in 3D space must be determined. This is made more challenging by: Making the ball move in circles in the horizontal plane. Changing the background image such that the ball is more difficult to locate. The goal is verified by comparing the perceived interpretation of the ball position to the actual position of the ball. Following the road A series of images from a tractor driving down a dirt road is provided. The goal is then to detect the road and propose the heading that the tractor use to stay on the road. This is challenging because: The images contain sky, grass and other artifacts which make it difficult to distinguish the road. The lighting conditions change as the tractor progresses down the road. This will make it difficult to make narrow filters to look for specific road details as the filters will not work all the time

5 J. Gorasia Vision Lab ENGR339: Robotics 3. Presentation of robotic software used Tracking the red ball A camera captures the image of the red ball against its background. Some image filtering needs to be done to find the ball and determine its size. This code is shown in Figure 1. The filter uses the find circles algorithm in Labview to find the ball. Figure 1: Code to find the ball in the image and find its size. If we assume that the ball remains in the camera plane, we can estimate the position of the ball using some trigonometry. The camera gives a 64x48 pixel image, shown in Image Out of Figure 4. If the center of the ball can be found, its offset from the center of the image can be determined. In addition, the width of the ball perceived in the image can be easily found. With knowledge of the viewing angle of the camera, you can then use the previous two pieces of information to determine the position of the ball in 3D space. This algorithm is shown in Figure 2.

6 J. Gorasia Vision Lab ENGR339: Robotics Figure 2: Block diagram of the algorithm which converts the position of the ball from 2D image space to 3D world space. With the position of the ball, the next step is to output the ball's position in an easily understood way. This is accomplished through using 3 different XY graphs and a final 3D graph to show the ball moving in 3D space. Figure 3 shows this algorithm. Figure 3: Ball display code. The use of many different graphs helps in debugging. Figure 4 shows the front panel of the ball display code. Showing all the information at the same time is useful again for debugging.

7 J. Gorasia Vision Lab ENGR339: Robotics Figure 4: Front panel for the tracking the red ball experiment

8 J. Gorasia Vision Lab ENGR339: Robotics Following the road Figure 5: Main VI. This displays the source image, the filtered image, the Hoff transform line fitting results and the final heading. It provides auxiliary information like the frame rate as well. The main VI for the road following is shown in Figure 5, while the main code structure is shown in Figure 6.

9 J. Gorasia Vision Lab ENGR339: Robotics Figure 6: Block diagram for main panel. It is a simple while loop while runs through a cycle of load images, filter images for the road, line fitting to the road, and a calculation where the direction vector is. The code is relatively simple. The road images are read from memory, filtered to identify the road, line fitting is done to find the edges of the road, and a vector is determined for the tractor to head along. One of the main things about vision in Labview is how is deals with memory allocation and data flow for images. An Imaq Create block is used to allocate a part of memory for an image, and all further processing happens to that part of memory. Therefore, if you want to maintain the image a few steps previously before some processing, you will need to allocate more memory for it. That is shown in Figure 2 where an original image copy is made using the source image for use later on in the image overlay. Next, since Labview is a data flow based language, a block will only execute if all the required inputs are wired. To make sure that the sequence of execution is correct, and to make sure images are ready for VIs to use, it is important to wire up the error inputs and outputs. Otherwise, many errors will occur. In addition the error lines give access to features like changing image palettes, pretty useful stuff. Video Images is the sub-vi that reads the files in sequence from the file directory, and is shown in Figure

10 J. Gorasia Vision Lab ENGR339: Robotics Figure 7: Video Images.vi. This sub-vi loads the road images from the directory and sends them out in sequence. The next step is image filtering. Figure 8: Filter Images.vi. The Images come in and are filtered Since all images consist of road, sky and grass, intelligent filtering needed to be done. It was useful to use the Vision Assistant as it help prototype the different filtering options very quickly. We settled on using differences in RGB values to distinguish between the sky and the other two, and HSL values to distinguish between the grass and the other two. Using a mask to look at whatever is not sky or grass, you will get mostly road. There are some artifacts in the resulting image, with some outline of the horizon and shadows on the road. This was very hard to correct robustly, as the road had different lighting as the tractor progressed through the road. After a filtered image of the road had been produced, the image is thresholded to emphasize the road more, and to convert it from a RGB32 image to a binary image. Then, the image is cropped to emphasize the area in front on the tractor, ignoring the horizon and the area too close to the tractor.

11 Figure 11: Transform to lines.vi. Takes the coordinates of pixels and finds the lines that will pass through them. J. Gorasia Vision Lab ENGR339: Robotics Next, the binary image is cleaned up, by filling up holes and removing small artifacts. As a final step, the binary blob of the road is converted to an outline of pixels. This is to make line fitting easier, as the pixels would represent edges. This is all shown in Figure 8. Now with the image full of pixels of where it think the road is, line fitting must be done to find out the lines that can determine the road. The image is converted into a 2D array of numbers to facilitate more mathematics. Figure 9: Hough Transform code. This takes an array of pixels and finds the line that best. The code in its almost all its entirety is shown in Figure 9. First, the array is converted into a 1D list of the coordinates of where the pixels are. For each coordinate, iterate through possible x intercepts, and find the gradient of the line required (in degrees) to pass through that point from that intercept. By incrementing an array of intercept and gradient every time a match is found, a distribution of the possible lines in the image can be found. By choosing the highest two matches from far enough spots of gradient in the array, the two lines can be found. A few extra details make this algorithm more effective. Firstly, not all possible x intercepts are used. Instead, increments of 1 are used, to improve speed. Next, a low past filter is implement by summing the previous 2 results to find the strongest peaks, to decrease the effect of noise. Finally, points with too high or low gradient are ignored as the road consistently is within a middling range of gradient. The next step is to use the two lines produced by the Hough transform to calculate the angle that the tractor should head at. Assuming that the tractor is heading in the right direction, the best algorithm is to simply average the gradients of the two lines (Figure 12). Figure 12: Calculate theta.vi. This sub-vi averages the gradients of the two incoming lines. Figure 1: Image to vector.vi. Converts the 2D array into a list of coordinates.

12 J. Gorasia Vision Lab ENGR339: Robotics Finally, the direction vector needs to be displayed over the original image. This is shown in Figure 13. Figure 13: Direction overlay.vi. Overlays an arrow over the original image. The arrow is overlayed using the Overlay commands which accept the pixel position of the endpoint of the lines to be displayed. Most of the code is to determine those pixel positions. The size of the arrow is based on the image size there is a block to get the image size then the length of the arrow is about 16% of the width of the image. The arrow head is an equilateral triangle with 4% of the length of the arrow width. All of these numbers empirically made and checked. 4. Technical analysis of raw experimental results To test both experiments, they were run and validated using the displays in their main front panels. Since a lot of data was displayed on the front panels, its was easy to isolate problems and fix bugs in the code. No technical analysis was done as we lacked time to determine optimal results for the road following. That would be something to do if more time was permitted. 5. Conclusions supported by condensed results Ultimately, we learn that implementing robotics vision within Labview is non trivial even with the numerous prebuilt tools that are provided. There is a wealth of algorithms to be chosen and parameters to be tweaked, which makes getting accurate results difficult. Importantly, current algorithms and filters are not robust enough to be used for all situations, making it important to choose correctly for the particular application. Nevertheless, vision is still a powerful tool for the roboticist, and is something I definitely want to learn more about in the future.

13 J. Gorasia Vision Lab ENGR339: Robotics 6. Appendix All code is attached at end of report. 7. Student revised lab write up I would add the following few paragraphs to help people understand vision within Labview better: One of the main things about vision in Labview is how is deals with memory allocation and data flow for images. An Imaq Create block is used to allocate a part of memory for an image, and all further processing happens to that part of memory. Therefore, if you want to maintain the image a few steps previously before some processing, you will need to allocate more memory for it. That is shown in Figure 2 where an original image copy is made using the source image for use later on in the image overlay. Next, since Labview is a data flow based language, a block will only execute if all the required inputs are wired. To make sure that the sequence of execution is correct, and to make sure images are ready for VIs to use, it is important to wire up the error inputs and outputs. Otherwise, many errors will occur. In addition the error lines give access to features like changing image palettes, pretty useful stuff.

14 Ball in 3D Image Out X vs time Y vs time Z vs time status code source Time (s) Time (s) Time (s) Topview px world x py world y pr world z X (mm)

15 Clean up Camera set up image RGB (U32) Contrast Recognition & Localization px py pr Image Out world x world y world z True X vs time Y vs time Z vs time 1 Time and Location Arrays Topview time array x array y array z array Ball in 3D 2 stop Running <3 Stopped! Iinfrastructural stuff 1 Plotting False No plot at time zero

16 Raw Camera (Colour) First Pass Second Pass Image Output ImageIn Best Circle Center. X. Y Radius.. Xo. Yo. Ra 16 C error_in status code source error out status code source Tab Control

17 Contrast Ballfind settings Outside to Inside Outside to Inside ImageIn error_in Tab Control ImageFirstPass Gray1 Find Circle First Pass ImageSecondPass x,y,r 2 Image Out Gray2 36 Find Circle Second Pass Best Circle ImageIn Refresh Image error out Overlay Circle on Original Image Xout Yout Radius

18 Connector Pane ball_location.vi px py pr x y z Front Panel px x py y pr z

19 Block Diagram px H. angle to ball x 64x48 pic width/2 32 y py Sensor PX width const 76 yb z Horizontal Field of View (Degrees) d pr Estimate distance along diagonal from camera to center of ball Approx. is good for range [55,8]cm

20 visionlabv.vi Source Image Final Image Filtered image Hoff transform Intensity Graph Frame STOP X intercept Angle X intercept 2 Angle X Intercept 22

21 Hoff transform Intensity Graph X intercept Angle X intercept 2 Filtered image Angle 2 Heartbeat Source Image Final Image CalculateTheta.vi 1 Frame start original Frame status stop

22 Heartbeat.vi Cycle Heartbeat Cycle Heartbeat Cycle Heartbeat 2

23 VideoImages.vi Frame In Image Out Image Out Frame In error in status source code error out status source code

24 No of frames Frame In TheRoad\TheRoad VideoFrame 4.bmp False 1622 False Image Out error out error in

25 image filter.vi Image in Error in Final image 2 error out Image in Error in status code error out status code source source Color Mode RGB Green Range Lower value Final image 2 Upper value 255 Red Range Lower value 5 Upper value 255 Blue Range Lower value Upper value 255 Operation Gradient in Crop image Image in Color Mode Operation Final image 2 Error in Rm Grass.vi Rm sky.vi Apply masks Threshold image 2 2 error out Green Range Red Range 2 Fill holes Remove small particles Find outline Blue Range

26 Rm Grass.vi Image Out error in Image Out Image Dst Out error out Image Dst Out error in error out status code status code source source 1 HSL Image Out Image Dst Out Grass Mask error in IMAQ ColorThreshold 8 IMAQ RemoveParticle PClose 1 IMAQ Morphology error out Grayscale (U8) invert image 143

27 Rm sky.vi Image Src error in (no error) Image Dst Out error out Red or Hue Range (Color Threshold 1) Lower value 86 Upper value 253 Green or Sat Range (Color Threshold 1) Lower value error in (no error) Image Src error out 129 Upper value status code status code 255 source source Blue or Luma or Val or Inten Range (Color Threshold 1) Lower value 16 Upper value 236 Image Dst Out 1 RGB Image Src Red or Hue Range (Color Threshold 1) Image Dst Out Sky mask IMAQ ColorThreshold 5 1 IMAQ RemoveParticle error in (no error) error out Invert image Grayscale (U8) Green or Sat Range (Color Threshold 1) Blue or Luma or Val or Inten Range (Color Threshold 1)

28 test2.vi Image Pixels (U8) 3 times averaged X Intercept Angle X Intercept 2 Angle 2 Angle Canceling Resolution 4 Image Pixels (U8) Xintercept Pixel Resolution 2 X Intercept Canceling Resolution 3 Intensity Graph X Intercept Angle X Intercept X Intercept 5 Angle 2 3 times averaged Vector Reformat X Intercept 5 25

29 Intensity Graph Image Pixels (U8) Angle 18 3 times averaged Vector Reformat x Y 25 y X Intercept 155 X Intercept 2 Angle 2 Angle Canceling Resolution X Intercept Angle Xintercept Pixel Resolution X Intercept Canceling Resolution Disabled Final Index Y Size X Size

30 Image_to_Vector.vi Image Pixels (U8) Final Index Y Size X Size Vector Format Image Pixels (U8) Final Index Y Size X Size Vector Format Final Index X Size 1 Vector Format 1 Image Pixels (U8) Y Size 2 1

31 Tranform_to_Lines.vi Angle Canceling Resolution Transformed Array X Size X Intercept Canceling Resol... Xintercept Pixel Resolution X Intercept 2 Angle 2 X Intercept Angle Transformed Array X Intercept 2 X Size X Intercept Canceling Resolution Xintercept Pixel Resolution Angle Canceling Resolution Angle 2 X Intercept Angle Intensity Graph X Intercept 5 X Intercept Transformed Array X Size X Intercept Canceling Resolution Xintercept Pixel Resolution 1 Angle 2 Angle Canceling Resolution 1 Angle 2 X Intercept 2 Intensity Graph

32 CalculateTheta.vi Theta1 Theta2 ThetaOut Theta1 ThetaOut Theta2 Theta1 Theta ThetaOut 18

33 DirectionOverlay.vi Image In Theta Image Out Image In X Resolution Y Resolution X Vector origin Y Vector Origin X Vector End Vector Length Y Vector End X Arrow 1 X Arrow 2 Y Arrow 1 Y Arrow 2 ArrowSideLength error out status source code Image Out Theta error in status source code

34 ArrowSideLength.25 Y Arrow 2 6 X Arrow 2 Y Arrow 1 X Arrow 1 Y Vector End Theta X Resolution.8 Vector Length X Vector origin X Vector End 5 4 Y Resolution Y Vector Origin Image In 4 Arrow Arrow Image Out error in Fill error out Overlayed