Robot vision review. Martin Jagersand

Size: px

Start display at page:

Download "Robot vision review. Martin Jagersand"

Shon Terry
5 years ago
Views:

1 Robot vision review Martin Jagersand

2 What is Computer Vision? Computer Graphics Three Related fields Image Processing: Changes 2D images into other 2D images Computer Graphics: Takes 3D models, renders 2D images Computer vision: Extracts scene information from 2D images and video e.g. Geometry, Where something is in 3D, Objects What something is What information is in a 2D image? What information do we need for 3D analysis? Computer Vision CV hard. Only special cases can be solved. Image Processing Computer Vision 90 horizon

3 Machine Vision 3D Camera vision in general environments hard Machine vision: Use engineered environment Use 2D when possible Special markers/led Can buy working system! Photogrammetry: Outdoors 3D surveying using cameras

Vision Full: Human vision We don t know how it works in detail Limited

get from a camera? 1. Location of a dot (LED/Marker) [u,v] = f(i) 2.

4 Vision Full: Human vision We don t know how it works in detail Limited vision: Machines, Robots, AI What is the most basic useful information we can get from a camera? 1. Location of a dot (LED/Marker) [u,v] = f(i) 2. Segmentation of object pixels All of these are 2D image plane measurements! What is the best camera location? Usually overhead pointing straight down Adjust cam position so pixel [u,v] = s[x,y]. s Pixel coordinates are scaled world coord X u v Y

s Pixel coordinates are scaled world coord Lower brightness so LED brighterest Put LED on robot end-effector effector

5 Tracking LED special markers v Put camera overhead pointing straight down on worktable. Adjust cam position so pixel [u,v] = s[x,y]. s Pixel coordinates are scaled world coord Lower brightness so LED brighterest Put LED on robot end-effector effector Detection algorithm: Threshold brightest pixels I(u,v)>200 Find centroid [u,v] of max pixels Variations: Blinking LED can enhance detection in ambient light. Different color LED s can be detected separately from R,G,B color video. X u Y Camera LED

6 Commercial tracking systems Polaris Vicra infra-red system (Northern Digitial Inc.) MicronTracker visible light system (Claron Technology Inc.)

7 Commercial tracking system Images acquired by the Polaris Vicra infra-red stereo system: left image right image

8 IMAGE SEGMENTATION How many objects are there in the image below? Assuming the answer is 4, what exactly defines an object? Zoom In 3/29/2016 Introduction to Machine Vision

9 8 BIT GRAYSCALE IMAGE /29/2016 Introduction to Machine Vision

10 GRAY LEVEL THRESHOLDING Objects Set threshold here 3/29/2016 Introduction to Machine Vision

11 BINARY IMAGE 3/29/2016 Introduction to Machine Vision

12 CONNECTED COMPONENT LABELING: FIRST PASS A A A A A EQUIVALENCE: B=C B B C C B B B C C B B B B B B 3/29/2016 Introduction to Machine Vision

13 CONNECTED COMPONENT LABELING: SECOND PASS A A A A A TWO OBJECTS! B B CB CB B B B CB CB B B B B B B 3/29/2016 Introduction to Machine Vision

14 IMAGE SEGMENTATION CONNECTED COMPONENT LABELING 4 Neighbor Connectivity 8 Neighbor Connectivity P i,j P i,j 3/29/2016 Introduction to Machine Vision

15 What are some examples of form parameters that would be useful in identifying the objects in the image below? 3/29/2016 Introduction to Machine Vision

16 OBJECT RECOGNITION BLOB ANALYSIS Examples of form parameters that are invariant with respect to position, orientation, and scale: Number of holes in the object Compactness or Complexity: (Perimeter) 2 /Area Moment invariants All of these parameters can be evaluated during contour following. 3/29/2016 Introduction to Machine Vision

17 GENERALIZED MOMENTS Shape features or form parameters provide a high level description of objects or regions in an image For a digital image of size n by m pixels : M ij = n m x= 1 y= 1 x i y j f ( x, y) For binary images the function f(x,y) takes a value of 1 for pixels belonging to class object and 0 for class background. 3/29/2016 Introduction to Machine Vision

18 GENERALIZED MOMENTS M ij = n m x= 1 y= 1 x i y j f ( x, y) X i j M ij Area Y Moment 64 of Inertia 93 3/29/2016 Introduction to Machine Vision

19 SOME USEFUL MOMENTS The center of mass of a region can be defined in terms of generalized moments as follows: X = M 10 Y = M 01 M 00 M 00 3/29/2016 Introduction to Machine Vision

20 SOME USEFUL MOMENTS The moments of inertia relative to the center of mass can be determined by applying the general form of the parallel axis theorem: M 02 = M 02 M M M 20 = M 20 M M M 11 = M 11 M 10 M M /29/2016 Introduction to Machine Vision

21 SOME USEFUL MOMENTS The principal axis of an object is the axis passing through the center of mass which yields the minimum moment of inertia. This axis forms an angle θ with respect to the X axis. The principal axis is useful in robotics for determining the orientation of randomly placed objects. TAN 2 θ = 2M 11 M 20 M 02 3/29/2016 Introduction to Machine Vision

22 Example X Principal Axis Center of Mass Y 3/29/2016 Introduction to Machine Vision

23 3D MACHINE VISION SYSTEM XY Table Laser Projector Digital Camera Field of View Plane of Laser Light P(x,y,z) Granite Surface Plate 3/29/2016 Introduction to Machine Vision

24 3D MACHINE VISION SYSTEM 3/29/2016 Introduction to Machine Vision

25 3D MACHINE VISION SYSTEM 3/29/2016 Introduction to Machine Vision

26 Morphological processing Erosion 26

27 Morphological processing Erosion 27

28 Dilation 28

29 Opening erosion followed by dilation, denoted Ao B = ( A B) B eliminates protrusions breaks necks smoothes contour 29

30 Opening 30

31 Opening 31

32 Opening A o B = ( A B) B Ao B = {( B) ( B) A} z z 32

33 Closing dilation followed by erosion, denoted A B = ( A B) B smooth contour fuse narrow breaks and long thin gulfs eliminate small holes fill gaps in the contour 33

34 Closing A B = ( A B) B 34

35 Image Processing Filtering: Noise suppresion Edge detection

36 Image filtering Mean filter * = Input image f Filter h Output image g

37 Animation of Convolution p1,1 p1,2 p1,3 p1,4 p1,5 p1,6 p2,1 p2,2 p2,3 p2,4 p2,5 p2,6 To accomplish convolution of the whole image, we just Slide the mask p3,1 p3,2 p3,3 p3,4 p3,5 p 3,6 p4,1 p4,2 p4,3 p4,4 p4,5 p4,6 p5,1 p5,2 p5,3 p5,4 p5,5 p5,6 p6,1 p6,2 p6,3 p6,4 p6,5 p6,6 Original Image m1,1m1,2m1,3 m2,1m2,2m2,3 m3,1m3,2m3,3 Mask C 4,2 C 4,3 C 4,4 Image after convolution

38 Gaussian filter Compute empirically * = Input image f Filter h Output image g

39 Convolution for Noise Elimination --Noise Elimination The noise is eliminated but the operation causes loss of sharp edge definition. In other words, the image becomes blurred

40 Convolution with a non-linear mask There are many masks used in Noise Elimination Median Mask is a typical one Median filtering The principle of Median Mask is to mask some sub-image, use the median of the values of the sub-image as its value in new image I=1 2 3 J= Rank: 23, 47, 64, 65, 72, 90, 120, 187, 209 median Masked Original Image

41 Median Filtering

42 Edge detection using the Sobel operator In practice, it is common to use: Magnitude: Orientation:

43 Sobel operator Original Magnitude Orientation

44 Effects of noise on edge detection / derivatives Consider a single row or column of the image Plotting intensity as a function of position gives a signal Where is the edge?

45 Solution: smooth first Where is the edge? Look for peaks in

46 Vision-Based Control (Visual Servoing) Initial Image User Desired Image

47 Vision-Based Control (Visual Servoing) : Current Image Features : Desired Image Features

48 Vision-Based Control (Visual Servoing) : Current Image Features : Desired Image Features

49 Vision-Based Control (Visual Servoing) : Current Image Features : Desired Image Features

50 Vision-Based Control (Visual Servoing) : Current Image Features : Desired Image Features

51 Vision-Based Control (Visual Servoing) : Current Image Features : Desired Image Features

52 u,v Image-Space Error v u : Current Image Features : Desired Image Features E =[ à ] One point E =[y 0 à y ã ] Pixel coord ô õ E = à E = y u y v Many points y 1. y 16 ã ô y õ ã u y v à y 1. y 16 0

53 Other (easier) solution: Image-based motion control Note: What follows will work for one or two (or 3..n) cameras. Either fixed (eye-in-hand) or on the robot. Here we will use two cam Motor-Visual function: y=f(x) Achieving 3d tasks via 2d image control

54 Vision-Based Control Image 1 Image 2

55 Vision-Based Control Image 1 Image 2

56 Vision-Based Control Image 1 Image 2

57 Vision-Based Control Image 1 Image 2

58 Vision-Based Control Image 1 Image 2

59 Image-based Visual Servoing Observed features: Motor joint angles: Local linear model: y = [ y 1 y 2... y m ] T x = [ x 1 x 2... x n ] T Δy = JΔx Visual servoing steps: 1 Solve: 2 Update: y ã à y k = JΔx x k+1 = x k + Δx

60 Find J Method 1: Test movements along basis Remember: J is unknown m by n matrix J = f 1 f x 1 ááá 1 x.. n... f m x 1 Assume movements Finite difference: Jt.. Δy. 1. f m x n.. Δy 2.. Δx 1 =[1, 0,..., 0] T Δx 2 =[0,. 1,..., 0] T. Δx n =[0, 0,..., 1] T ááá.. Δy n..

61 Find J Method 2: Secant Constraints Constraint along a line: Defines m equations Δy = JΔx Collect n arbitrary, but different measures y Solve for J ááá Δy T â 1 ááá ã ááá Δy T 2 ááá. â. = ã ááá Δy T n ááá ááá Δx T â 1 ááá ã ááá Δx T 2 ááá. â. ã ááá Δx T n ááá JT

62 Find J Method 3: Recursive Secant Constraints Based on initial J and one measure pair Adjust J s.t. Rank 1 update: Δy = J k+1 Δx Δy, Δx J ê k+1 = J ê (Δy à J k + ê kδx)δx T Δx T Δx Consider rotated coordinates: Update same as finite difference for n orthogonal moves

63 Achieving visual goals: Uncalibrated Visual Servoing 1. Solve for motion: 2. Move robot joints: 3. Read actual visual move 4. Update Jacobian: [y à y k ]=JΔx x k+1 = x k + Δx Δy J ê k+1 = J ê (Δy à J k + ê kδx)δx T Can we always guarantee when a task is achieved/achievable? Δx T Δx

64 Visually guided motion control Issues: 1. What tasks can be performed? Camera models, geometry, visual encodings 2. How to do vision guided movement? H-E E transform estimation, feedback, feedforward motion control 3. How to plan, decompose and perform whole tasks?

65 How to specify a visual task?

66 Task and Image Specifications Task function T(x) = 0 Image encoding E(y) =0 task space error task space satisfied image space error image space satisfied

67 Visual specifications Point to Point task error : E =[y 2 à y 0 ] y 0 y E = y. 1. y 16 à Why 16 elements? y. 1. y 16 0 y y 0

68 Visual specifications 2 Point to Line Line: ô E pl (y, l) = y l á l l y r á l r ô l l = y õ 3 â y 1 y 4 â y 2 õ ô y l = y õ 5 y 6 y 3 y 4 Note: y homogeneous coord. y 1 y 2

69 Parallel Composition example E (y) = wrench y 2 - y5 y 4 - y7 y (y y ) y (y y ) (plus e.p. checks)

70 Visual Specifications Additional examples: Line to line Point to conic Identify conic C from 5 pts on rim Error function ycy Image moments on segmented images Any image feature vector that encodes pose. Etc.

71 Achieving visual goals: Uncalibrated Visual Servoing 1. Solve for motion: 2. Move robot joints: 3. Read actual visual move 4. Update Jacobian: [y à y k ]=JΔx x k+1 = x k + Δx Δy J ê k+1 = J ê (Δy à J k + ê kδx)δx T Can we always guarantee when a task is achieved/achievable? Δx T Δx

Robot Vision Control of robot motion from video. M. Jagersand

Robot Vision Control of robot motion from video M. Jagersand Vision-Based Control (Visual Servoing) Initial Image User Desired Image Vision-Based Control (Visual Servoing) : Current Image Features : Desired