Robot Vision Control of robot motion from video. M. Jagersand

Transcription

1 Robot Vision Control of robot motion from video M. Jagersand

2 Vision-Based Control (Visual Servoing) Initial Image User Desired Image

3 Vision-Based Control (Visual Servoing) : Current Image Features : Desired Image Features

4 Vision-Based Control (Visual Servoing) : Current Image Features : Desired Image Features

5 Vision-Based Control (Visual Servoing) : Current Image Features : Desired Image Features

6 Vision-Based Control (Visual Servoing) : Current Image Features : Desired Image Features

7 Vision-Based Control (Visual Servoing) : Current Image Features : Desired Image Features

8 u,v Image-Space Error v u : Current Image Features : Desired Image Features E = [ ] One point E = [y y ] Pixel coord E= y u y v Many points E = y. y 6 y u y v y. y 6

9 Conventional Robotics: Motion command in base coord EE We focus on the geometric transforms

10 Problem: Lots of coordinate frames to calibrate Camera Center of projection Different models Robot Base frame End-effector frame Object

11 Image Formation is Nonlinear Camera frame

12 Camera Motion to Feature Motion Camera frame

13 Camera Motion to Feature Motion This derivation is for one point. Interaction matrix is a Jacobian. Q. How to extend to more points? Q. How many points needed?

14 Problem: Lots of coordinate frames to calibrate Camera Center of projection Different models Robot Base frame End-effector frame Object Only covered one transform!

15 Other (easier) solution: Image-based motion control Motor-Visual function: y=f(x) Achieving 3d tasks via 2d image control

16 Other (easier) solution: Image-based motion control Note: What follows will work for one or two (or 3..n) cameras. Either fixed or on the robot. Here we will use two cam Motor-Visual function: y=f(x) Achieving 3d tasks via 2d image control

17 Vision-Based Control Image Image 2

18 Vision-Based Control Image Image 2

19 Vision-Based Control Image Image 2

20 Vision-Based Control Image Image 2

21 Vision-Based Control Image Image 2

22 Image-based Visual Servoing Observed features: Motor joint angles: Local linear model: Visual servoing steps: Solve: y = [ y y 2... y m ] T x = [ x x 2... x n ] T y = J x 2 Update: y y k = J x x k+ = x k + x

23 Find J Method : Test movements along basis Remember: J is unknown m by n matrix Motor moves (scale these): Suitable J = Finite difference: f f x x.. n... f m x J \thickapprox f m x n < y<2 pixels x = [,,..., ] T. y. x 2 = [,,..., ] T.. x n = [,,..., ] T. y 2.. y n.

24 Find J Method 2: Secant Constraints Constraint along a line: Defines m equations Collect n arbitrary, but different measures y Solve for J y T y T 2.. = y T n y = J x x T x T 2.. x T n JT

25 Find J Method 3: Recursive Secant Constraints Based on initial J and one measure pair Adjust J s.t. Rank update: y = J k+ x y, x ( y J k x) x J k+ = J T k + x T x Consider rotated coordinates: Update same as finite difference for n orthogonal moves

26 Achieving visual goals: Uncalibrated Visual Servoing. Solve for motion: 2. Move robot joints: 3. Read actual visual move [y y k ] = J x x k+ = x k + x y ( y J 4. Update Jacobian: k x) x J k+ = J T Can we always guarantee when a task is k achieved/achievable? + x T x

27 Visually guided motion control Issues:. What tasks can be performed? Camera models, geometry, visual encodings 2. How to do vision guided movement? H-E transform estimation, feedback, feedforward motion control 3. How to plan, decompose and perform whole tasks?

28 How to specify a visual task?

29 Task and Image Specifications Task function T(x) = Image encoding E(y) = task space error task space satisfied image space error image space satisfied

30 Visual specifications Point to Point task error : E = [y 2 y ] y y E = y.. y 6 Why 6 elements? y.. y 6 y y

31 Visual specifications 2 Point to Line Line: E pl (y, l) = y l l l y r l r l l = y 3 y y 4 y 2 Note: y homogeneous coord. y y 2 y l = y 5 y 6 y 3 y 4

32 Parallel Composition example E (y) = wrench y 2 - y5 y 4 - y7 y (y y ) 6 2 y (y y ) (plus e.p. checks)

33 5.4. Visual specifications Image commands = Geometric constraints in image space HRI: User points/clicks on features in an image Robot controller drives visual error to zero

34 Visual Specifications Additional examples: Point to conic Identify conic C from 5 pts on rim Error function ycy Point(s) to plane Plane to plane Etc.

35 Achieving visual goals: Uncalibrated Visual Servoing. Solve for motion: 2. Move robot joints: 3. Read actual visual move [y y k ] = J x x k+ = x k + x y ( y J 4. Update Jacobian: k x) x J k+ = J T Can we always guarantee when a task is k achieved/achievable? + x T x

36 Task ambiguity Will the scissors cut the paper in the middle?

37 Task ambiguity Will the scissors cut the paper in the middle? NO!

38 Task Ambiguity Is the probe contacting the wire?

39 Task Ambiguity Is the probe contacting the wire? NO!

40 Task Ambiguity Is the probe contacting the wire?

41 Task Ambiguity Is the probe contacting the wire? NO!

42 Task decidability and invariance A task T(f)= is invariant under a group G of transformations, iff x n n f V, g G x with g(f ) V T(f )= T(g(f ))= If T(f ) = here, T(f ) must be here, if T is G invariant sim T(f ) must be here, if T is G invariant aff T(f ) must be here, if T is G proj invariant T(f ) must be here, if T is G invariant inj

43 All achievable tasks... are ALL projectively distinguishable coordinate systems One point Coincidence Noncoincidence Collinearity General Position 2pt. coincidence 3pt. coincidence α 4pt. coincidence 2 2pt. coincidence Cross Ratio General Position Coplanarity

44 Composing possible tasks Injective Projective C inj C wk T pp T pp point-coincidence T col T copl T crα collinearity coplanarity cross ratios (α) Task primitives AND OR NOT T T T 2 = T T 2 = T T 2 T = T 2 if T = otherwise Task operators

45 Result: Task toolkit wrench: view wrench: view T wrench (x..8 ) = T pp (x,x 5 ) T pp (x 3,x 7 ) T col (x 4,x 7,x 8 ) T col (x 2,x 5,x 6 )

46 Serial Composition Solving whole real tasks Task primitive/ link. Acceptable initial (visual) conditions 2. Visual or Motor constraints to be maintained 3. Final desired condition Task = A = (E init, M,E final ) AA2...A k

47 Natural primitive links. Transportation Coarse primitive for large movements <= 3DOF control of object centroid Robust to disturbances 2. Fine Manipulation For high precision control of both position and orientation 6DOF control based on several object features

48 Example: Pick and place type of movement 3. Alignment??? To match transport final to fine manipulation initial conditions

49 More primitives 4. Guarded move Move along some direction until an external contraint (e.g. contact) is satisfied. 5. Open loop movements: When object is obscured Or ballistic fast movements Note can be done based on previously estimated Jacobians

50 Solving the puzzle

51 Conclusions Visual servoing: The process of minimizing a visuallyspecified task by using visual feedback for motion control of a robot. Is it difficult? Yes. Controlling 6D pose of the end-effector from 2D image features. Nonlinear projection, degenerate features, etc. Is it important? Of course. Vision is a versatile sensor. Many applications: industrial, health, service, space, humanoids, etc.

52 What environments and tasks are of interest? Not the previous examples. Could be solved structured Want to work in everyday unstructured environments

53 Use in space applications Space station On-orbit service Planetary Exploration

54 Power line inspection A electric UAV carrying camera flown over the model Simulation: Robot arm holds camera Model landscape: Edmonton railroad society

55 Use in Power Line Inspection

56 Some References M. Spong, S. Hutchinson, M. Vidyasagar. Robot Modeling and Control. Chapter 2: Vision-Based Control. John Wiley & Sons: USA, 26. (Text) S. Hutchinson, G. Hager, P. Corke, "A tutorial on visual servo control," IEEE Trans. on Robotics and Automation, October 996, vol. 2, no 5, p (Tutorial) F. Chaumette, S. Hutchinson, "Visual Servo Control, Part I: Basic Approaches," and "Part II: Advanced Approaches," IEEE Robotics and Automation Magazine 3(4):82-9, December 26 and 4():9-8, March 27. (Tutorial) B. Espiau, F. Chaumette, P. Rives, "A new approach to visual servoing in robotics," IEEE Trans. on Robotics and Automation, June 992, vol. 8, no. 6, p (Imagebased) W. J. Wilson, C. C. Williams Hulls, G. S. Bell, "Relative End-Effector Control Using Cartesian Position Based Visual Servoing", IEEE Transactions on Robotics and Automation, Vol. 2, No. 5, October 996, pp (Position-based) E. Malis, F. Chaumette, S. Boudet, "2 /2 D visual servoing," IEEE Trans. on Robotics and Automation, April 999, vol. 5, no. 2, p (Hybrid) M. Jägersand, O. Fuentes, R. Nelson, "Experimental Evaluation of Uncalibrated Visual Servoing for Precision Manipulation," Proc. IEEE Intl. Conference on Robotics and Automation (ICRA), pp , 2-25 Apr (Uncalibrated, model-free) F. Chaumette, "Potential problems of stability and convergence in image-based and position-based visual servoing," The Confluence of Vision and Control, LNCS Series, No 237, Springer-Verlag, 998, p