Introduction to visual servoing

Transcription

1 October 17th December 16th, 2005 Introduction to visual servoing Professor IFMA LASMEA Clermont-Ferrand, France

2 Outline of the presentation Basic concepts Introduction Classification History and Bibliography Seventies and heighties Nineties Current research Modeling Computer vision Kinematic control Task function 2

3 A set of basic notation v=(v ω ) Kinematic Screw i T j Homogeneous Transformation matrix e Task Function (Error function) i t j = (t x,t y,t z ) Translation vector s Sensor signal (At each iteration) u = (u x,u y,u z ) Rotation axle s* Sensor signal (At equilibrium) θ Rotation angle L Interaction matrix i r j = θ.u Rotation vector i V j C Frame change matrix for L Combination matrix i R j =exp([ i r j ] x ) Rotation matrix 3

4 Basic History and Modeling concepts Bibliography Basic concepts : Introduction s & = L s.v CNA M I Low and intermediate level 2D Visual Features h1 ( t) h2 ( t) ==>.. hn ( t) s( t) = f ( t) 3D Visual Features ( s( t) *) e = C. s v = λ.e CNA M I Low and intermediate level h(t) Reconstruction (Model) s1( t) s ( t) 2.. sn ( t) ==> s( t) 2D/3D Visual Features CNA M I Low and intermediate level h(t) Reconstruction (Model) s1( t) s ( t) 2.. sn ( t) ==> s( t) 4

5 x z y x z y Basic concepts History and Bibliography Modeling Basic concepts : Introduction Case of embedded camera ν c Fc ds dt ( p t) s = s, δs dp = s& =. δp dt Fc Fa δs + s& δt z y Fo z x y x Interaction matrix = L. v s c δs + δt v c = ν ω ω c c c If (fixed object) s& = δs = 0 δt L s. v c 5

6 s Basic concepts History and Bibliography Basic concepts : Introduction * - + s(t) law v v = inverse Jacobian λ. q& + L s ler Modeling ( s( t) s *). Power Joint feedback IBVS Image Based Visual Servoing h(i) Features Extraction 6 I Visual feedback Sensor of 2D features Allen, Andreff, Asada, Berry, Cervera, Chaumette, Christensen, Collewet, Corke,Cowan,Cretual, Crowley, Daney, Debain, Degushi, Devy, Dornaika, Espiau, Feddema, Gangloff, Ginoux, Hager, Hamel, Hashimoto, Horaud, Hutchinson, Jagersand, Kanade, Kelly, Khosla, Khadraoui, Kragic, Iwatsuki, Lee, Malis, Marchand, Martinet, Mezouar, Motyl, Myasaki, Nelson, Hosoda, Papanikolopoulos, Piepmier, Pissard-Gibollet, Rives, Sanderson, Soueres, Swain, Urban, Weiss,.. (non exhaustive)

7 s Basic concepts History and Bibliography Basic concepts : Introduction * - + law s(t) v v = Inverse Jacobian λ. q& + L s ler Modeling ( s( t) s *). Power Joint feedback PBVS Position Based Visual Servoing Pose Estimation h(i) Features Extraction I Visual feedback Sensor of 3D features Daucher, Dhome, Grosso, Krupa, Martinet, Malis, Morel, Rizzi, Sandini, Sharifi, Siciliano, Wilson, Zanne,.. (non exhaustive) 7

8 s Basic concepts History and Bibliography Basic concepts : Introduction * - + law s(t) v v = Inverse Jacobian λ. q& + L s ler Modeling ( s( t) s *). Power Joint feedback HBVS Hybrid 2D/3D Based Visual Servoing Pose Estimation h(i) Features Extraction I Visual feedback Sensor of 3D features Andreff, Basri, Chaumette, Chesi, Daney, Degushi, Dixon, Martinet, Malis, Mahony, Morel, Ostrowski, Taylor.. (non exhaustive) 8

9 Basic concepts History and Bibliography Modeling History and Bibliography : Seventies and heighties Bolles and Paul in 73 Shirai and Inoue in 73 Fixed camera Assembly tasks «Look and move» (non exhaustive bibliography) G.J. Agin in 79 Sanderson and Weiss 87 Espiau 87 Samson, Espiau 89 Embedded camera Classification Sensor based control Task function approach Feddema 89 Feature based trajectory generation. 9

10 Basic concepts History and Bibliography 10 Modeling History and Bibliography : Nineties Sandini 96, Wilson 96 Martinet Position based visual servoing (non exhaustive bibliography) Chaumette 90, 92 Visual servoing Maru 93 Stereo Corke 94 Dynamic visual servoing Malis 97 2D+1/2 visual servoing De Schutter 97 Morel/Malis 98 Corke 93, Gangloff 96, Hashimoto 96 Bensalah 96, Piepmeir 02 Motyl Camera/laser stripe sensor Cretual 98 2D+dt visual servoing Force/vision coupling Chaumette 98 Local minima. Target tracking

11 Basic concepts History and Bibliography Modeling History and Bibliography : Current research ( ) Malis 00, Martinet/Cervera (non exhaustive bibliography) Multi-camera, Stereo Mezouar Martinet 02, Siciliano 02 Wilson 02 Malis 02 Corke, Hutchinson 02 Kragic 02 Trajectory generation Position based visual servoing Invariant visual servoing Partitioned visual servoing Robust visual servoing 11

12 Basic concepts History and Bibliography Modeling History and Bibliography : Current research ( ) (non exhaustive bibliography) Chesi 03, Corke 02, Malis 99 Mezouar 02, Morel 03 Thuilot 02 Horaud 02, Mezouar 04 Malis 04 Malis 03 Field of view Central catadioptric Cameras Efficient Second order control Stability pb in depth distribution Non exhaustive Bibliography : see the web site for a longer one 12

13 Basic History and Modeling concepts Bibliography Modeling : Computer vision Pinhole camera Normalized coordinates x n = y n = X Z Y Z 13

14 Basic History and Modeling concepts Bibliography Modeling : Computer vision Pinhole camera K f = 0 f s r 0 u0 v 1 0 f 0 Intrinsic paramaters: f : focal length s : skew r : aspect ratio u 0,v 0 : principal point 14

15 Basic History and Modeling concepts Bibliography Modeling : Computer vision Projection Matrix 15

16 Basic History and Modeling concepts Bibliography Modeling : Computer vision Camera calibration 16

17 Basic History and Modeling concepts Bibliography Modeling : Computer vision Pose estimation 17

18 Basic concepts History and Bibliography Modeling : Kinematic control Use of Kinematic model. p s * - + law Inverse Jacobian q& ler Power s(t) Joint feedback Pose Estimation h(i) Filter h I Visual feedback q & 1 = J ( q) p& J(q) represents the robot jacobian The k th line is given by : J k ( q) p = q k 1 p q k 2 p q k n 1 p q k n 18

19 Basic concepts History and Bibliography : Task function p& J 1 ( q) q& Modeling Robot Velocity control basis Sensor p Simplified modeling of the loop Integration due to the sensor p& IKM q& 1/p System DGM Modeling of the sensor by the DGM Using the current state of the robot p p& 1/p p p represents the position/orientation of the robot end effector 19

20 Basic History and concepts Bibliography : Task function Modeling Velocity control basis Simplified modeling of the loop p * + - p& e p C 1/p p Error Function Proportional control law e = p * p e & = p& * C = λ p& = λ. e = λ. ( p p) Exponential decrease of the Error Function e & = λ.e 20

21 Basic concepts History and Bibliography : Task function Samson, Espiau End of 80 The robotic task can be described as a regulation to zero of a task function Modeling Main concepts The task function characterizes the robotic task to be performed and allows to establish a virtual link between the sensor and the environment. ( q t) e, e e Some task function : * ( q, t) = q() t q () t e * ( q, t) = p( q) p ( t) ( q, t) = d( q) d * q*(t) represents a desired trajectory in the joint space p*(t) a desired trajectory in the operational space (i.e cartesian space for a robot with 6 d.o.f.) d* represents a desired distance between the object and the end effector 21

22 Basic concepts History and Bibliography Modeling : Task function Main concepts e When the sensor deliver the information s and dim(s)= k, the task function to be regulated can written like: ( * ) ( p t) = C. s( p t), s 1, m x 1 m x k k x 1 n = dim(q) m = dim(e) k = dim(s) m m k n C represents a combination matrix (dim = m x k, full rank m); It allows to take into account more sensor informations than necessary to perform the robot control. If s is well chosen, the m components of e 1 are independent and allow to control m d.o.f. (necessary to perform the robotic task) 22

23 Basic concepts History and Bibliography Modeling : Task function Task redondancy If the interaction matrix is not a full rank matrix δs δp ou δs δq dim( p) ou dim( q) Main concepts Number of independent components of s One hybrid task can be defined : - one primary task e 1 (maintain an interaction constraint during the execution of the task) - one secondary task (minimize a cost function h s ) Gradient of the Cost function h s g T s = δh δt s T 23

24 Basic concepts History and Bibliography Modeling : Task function Main concepts Global task function ( ) T I W W e = W e + γ m g 1 s A represents a full rank matrix as : ( W ) ker( ) ker = L s γ + I m W W Orthogonal projector on the kernel of W gain allowing to tune the preponderance of the primary task in regard with the secondary task + I m W W + W Represent two operators which guarantee that the induced motions due to the secondary task (included in the kernel of W) are compatible with the convergence of s to s* (primary task) 24

25 Basic concepts History and Bibliography Modeling : Task function Main concepts Without a secondary task Task function e 1 ( * ) ( r t) = C. s( r, t), s Fixed object Condition of convergence With a secondary task Task function δe 1 = δt 0 Condition of convergence ( ) T I W W +. e +. + m g 1 s e = W γ. C + = L s + = W Ls C. law law v c = λ. + L s. ( * s( r, t) s ) v c ( + ) δg = λ. e( r, t) γ. Im W. W. δt T s 25

26 Basic concepts History and Bibliography Modeling Different kind of works : Application of known methods Innovation (new methods) Application fields Real world objective (thanks to image processing improvements) A cooking approach : spherical, cylindrical, mixture A large number of robot applications Research community is growing (thanks for all of us) 26

27 October 17th December 16th, 2005 Applications in visual servoing Professor IFMA LASMEA Clermont-Ferrand, France

28 Outline of the presentation Manipulator robot Visual servoing : 2D and 3D approach, Stereo Force/Vision control Mobile Robot Automatic Guided vehicule in agricultural context. Automatic Guided vehicule in indoor context. 28

29 3D 3D Visual servoing Positioning task face to an unknown object Dhome, Jurie Positioning task 29

30 ST Stereo Visual servoing Cervera [99,01,02,03] Comparaison study : 3D point 2D Stereo point 2D image point + depth 2D image point + disparity Case study for grasping: Oriented blob 30

31 FV Force and vision coupling Force feedback LASMEA IFMA-SOCRATES Visual servoing Robot control By vision/force coupling External Position/force Hybrid Position Force External Vision/force Hybrid Vision Force Hybrid Vision Force : case study Done by M. Prats from Castellon, Spain 31

32 Outline of the presentation Manipulator robot Visual servoing : 2D and 3D approach, Stereo Force/Vision control Mobile Robot Automatic Guided vehicule in agricultural context. Automatic Guided vehicule in indoor context. 32

33 2D Automatic Guided Vehicles Debain [96] laws for agricultural machines Combine-harvester Harvesting work 33

34 2D Automatic Guided Vehicles Debain [96] Guiding on a sloping ground 34

35 2D Automatic Guided Vehicles : target tracking Clady [02] Fixed focal length camera Wide angle Our Vision sensor Sensor Decision Module Actuators PTZ led Camera 35

36 2D Automatic Guided Vehicles : target tracking Clady [02] 36

37 2D Automatic Guided Vehicles : visual memory Blanc [04] Ait Ader[04] Navigation using visual memory 37

38 2D Automatic Guided Vehicles : visual memory Blanc [04] Ait Ader[04] Navigation using visual memory 38