Control of industrial robots. Control with vision sensors

Transcription

1 Control of indstrial robots Control with ision sensors Prof. Paolo Roo Politenio di Milano Dipartimento di Elettronia Informazione e Bioingegneria

2 Visal measrements Artifiial ision deies are sefl sensors for robotis bease they mimi the hman sense of sight and allow to gather information from the enironment withot ontat. Nowadays seeral roboti ontrollers integrate ision systems. The typial se of ision in indstrial robotis is to detet an objet in the robot s sene whose position (and orientation) is then sed for online path planning in order to drie the robot to the identified objet. Online re-planning of the path an also be performed when the ision system detets some nexpeted hange in the path the robot is spposed to follow (for example a orner in a ontoring task). Alternatiely isal measrements an be sed in a real time feedbak loop in order to improe position ontrol of the end effetor: this is the onept of isal seroing.

3 Examples of se of ision a binpiking problem. Piking sasages

4 Examples of se of ision Following a orner dring a ontoring task: View from an external amera View from the onboard amera

5 Examples of se of ision A ball athing robot (example of isal seroing).

6 The hardware amera ideo data frame grabber indstrial PC zoom lens

7 The amera The amera is a deie that an measre the intensity of light onentrated by a lens on a plane the image plane.

8 How an image is stored The image plane ontains a matrix of pixels (CCD: Charge Copled Deie). The light is aptred in terms of intensity only (gray sale) intensity and spetral omponents (RGB) Gray sale: eah pixel desribes with a ertain nmber of bits the sale from white to blak. RGB : for eah pixel and for eah primary olor (red green ble) a ertain nmber of bits is sed to express the sale in sh olor. If we hae 8 bits for eah olor we an express different olors

9 2D projetion The amera performs a 2D projetion of the sene. This projetion entails a loss of depth information: eah point in the image plane orresponds to a ray in the 3D spae. In order to determine the 3D oordinates of a point orresponding to a 2D point in the plane additional information is needed: mltiple iews with a single amera mltiple ameras knowledge of harateristi relations between releant points of the framed objets

10 Perspetie projetion x p P A point P with oordinates (X Y ) in the amera frame is projeted into a point p with oordinates ( ) in the image plane expressed in pixels. z From similarity of triangles: X Y y ξ X Y Other methods exist to represent the projetion on the 2D plane (saled orthographi projetion affine projetion) : foal length (in pixel)

11 Image featres In artifiial ision we denote with image featre any harateristis that an be extrated from an image (e.g. an edge or a orner). We then define a parameter of an image featre a qantity expressed by a real nmeri ale whih an be ompted from one or more image featres. Parameters of an image featre an be gathered in a etor: ξ [ξ 1 ξ 2 ξ k ]. Examples of parameters of image featres: point oordinates length and orientation of a line onneting two points entroids and higher order moments parameters of an ellipse

12 Image featres Example: we want to extrat pixel oordinates (featres) of the orners of the top fae of a be whih is on the table. Original pitre Contor pitre (binary) Pixels are olored white orresponding to high illmination hange (gradient based edge detetion) Image featres (points)

13 Calibration The amera has to be alibrated before sage in a roboti ision system: Internal alibration: Determination of the intrinsi parameters of the amera (like the foal length ) as well as of some additional distortion parameters de to lens imperfetions and misalignments in the optial system External alibration: Determination of the extrinsi parameters of the amera like the position and the orientation of the amera with respet to a referene frame

14 3D ision 3D ameras retrn information on the depth as well Depth map: the intensity of the pixel is proportional to the inerse of the distane.

15 3D ision: tehnology The mostly adopted tehnology is based on the time of flight. Light emitter Light sensor Phase lag is proportional to trael time (whih is trn is proportional to distane)

16 Camera onfigration The first deision to be made when setting p a ision-based ontrol system is where to plae the amera. The amera an be: monted in a fixed loation in the workspae (eye-to-hand onfigration) so that it an obsere the maniplator and any objets to be maniplated attahed to the robot aboe the wrist (eye-in-hand onfigration) Eye-To-Hand Eye-In-Hand

17 Eye-to-hand onfigration Eye-To-Hand Adantages the field of iew does not hange as the maniplator moes the geometri relationship between the amera and the workspae is fixed and an be alibrated offline Disadantages as the maniplator moes throgh the workspae it an olde the amera s field of iew

18 Eye-in-hand onfigration Adantages the amera an obsere the motion of the end effetor at a fixed resoltion and withot olsion as the maniplator moes throgh the workspae Eye-In-Hand Disadantages the geometri relationship between the amera and the workspae hanges as the maniplator moes the field of iew an hange dramatially for een small motions of the maniplator

19 Control arhitetres: lassifiation Roboti ision ontrol systems an be lassified based on arios riteria. A first lassifiation is based on the following qestion: Is the ontrol strtre hierarhial with the ision system proiding set-points as inpt to robot s joint-leel ontroller or does the isal ontroller diretly ompte the joint-leel inpts? In the first ase: In the seond ase: dynami look and moe diret isal seroing Adantages of the dynami look and moe approah: the reded sampling rate of the isal signal does not ompromise the oerall performane of the position ontrol system in seeral indstrial robot ontrollers it is only allowed to operate at the position setpoints leel the robot an be seen as an ideal positioner in the Cartesian spae ths simplifying the design of the ision ontrol system

20 Control arhitetres: lassifiation A seond lassifiation is based on the following qestion: Is the error signal defined in 3D (task spae) oordinates or diretly in terms of image featres? In the first ase : In the seond ase : position based ontrol image based ontrol Position based ontrol: ision data are sed to bild a partial 3D representation of the world pose estimation algorithms are omptationally intensie (a real-time implementation is reqired) and sensitie to errors in amera alibration Image based ontrol: ses the image data diretly to ontrol the robot motion an error fntion is defined in terms of qantities that an be diretly measred in an image and a ontrol law is onstrted that maps this error diretly to robot motion

21 Position based s. Image based Two examples: Position based Image based

22 Position-based look-and-moe Camera x d + Cartesian ontrol law Joint ontrollers Atators x Pose estimation Image featre extration Video DIFFICULT

23 Image-based look-and-moe DIFFICULT + Control law in the image featre spae Joint ontrollers Atators Image featre extration Video

24 Position-based isal seroing Camera x d + Cartesian ontrol law Atators x Pose estimation Image featre extration Video DIFFICULT

25 Image-based isal seroing DIFFICULT + Control law in the image featre spae Atators Image featre extration Video

26 Image-based shemes To sole an image-based sheme we need to relate motion of the amera with motion of the featres in the image plane: O ω Linear and anglar eloities of the amera frame Linear eloity of the point featre in the image We will end p with the notion of interation matrix (and of image Jaobian)

27 Kinemati relations Consider a moing amera obsering a point fixed in spae: amera frame We hae: w w w ( t ) P ( t ) O ( t ) P R + or: P ( ) w T w w ( t ) R ( t ) P O ( t ) obsered point Differentiating wrt time: world frame P ω P O (an be easily proen )

28 Kinemati relations We hae then obtained this fndamental relation: Linear eloity of a point in the amera frame P ω P O Linear eloity of the amera frame Anglar eloity of the amera frame Position of a point in the amera frame where all etors are expressed in the amera frame.

29 Kinemati relations Define: ω ω ω z y x z y x O O O Y X O P ω The preios relation an be expressed in terms of salar eqations: z x y y z x x y z O Y X O X Y O Y X ω ω ω ω ω ω From the perspetie geometry: Y X

30 Kinemati relations By sbstittion we obtain: z x y y z x x y z O O Y O X ω ω ω ω ω ω We an also take the deriatie of the eqations of the perspetie geometry: 2 2 Y Y Y dt d X X X dt d

31 Interation matrix Combining the preios eqations we finally obtain: where the matrix: ( ) ω O L ( ) L is alled the interation matrix. It relates the linear and anglar eloities of the amera to the eloity in the image plane

32 Interation matrix The interation matrix: 1. is a 2 X 6 retanglar matrix 2. depends on the atal ales of the featres and and on the depth (merely as a sale fator) 3. an be deomposed in two sbmatries ( ) ( ) ω ω + L O L O does not depend on the depth ( ) L

33 Nll spae of the interation matrix Sine the interation matrix is 2 X 6 the nll-spae has dimensions 4 whih means that there are 4 motions of the amera that do not prode any motion of the featre in the image plane. It an be proen that the nll spae is spanned by the for etors: ( ) ( ) ( ) ( ) Motion of the amera frame along the projetion ray that ontains point P Rotation of the amera frame abot a projetion ray that ontains the point P

34 Nll spae of the interation matrix Motion of the amera frame along the projetion ray that ontains point P Rotation of the amera frame abot a projetion ray that ontains the point P

35 Mltiple featre points The definition of interation matrix an be easily extended to the ase of the oordinates of n image points. We define the featre etor ξ and the etor of depth ales as: n n n ξ The interation matrix is obtained by staking the n interation matries for the n indiidal featre points: ( ) ( ) ( ) n n n n L L L ξ

36 Image Jaobian We an now relate the motion of the featre point to the motion of the robot in joint spae: L( ) T ( q) J( q)q joint eloities robot Jaobian We an write: where the matrix: J J ( q)q I ( q) L( ) T ( q) J( q) I hange of oordinates (from base to amera frame) is alled the image Jaobian.

37 Dependene on the depth The image Jaobian J I depends on the depth : J ( q)q I This information is learly not aailable bt it an be estimated in seeral ways: sing the desired goal position sing geometrial knowledge of the sene sing the geometrial knowledge of the objet Sitable obserers an be setp

38 Kinemati ontrol law A ontrol law an be now deised based on the image Jaobian: # ( ξ ( )) ( I J J ) q d + ξd + I I 0 # q JI K ξ Minimm norm soltion Term projeted in the nllspae of the Jaobian (does not moe the featres) + Control law in the image featre spae Joint ontrollers Atators Image featre extration Video

39 Mahine Vision Toolbox The Mahine Vision Toolbox by Peter Corke proides many fntions that are sefl in mahine ision and ision-based ontrol: xes/mahine-ision-toolbox/ Combined with the Robotis Toolbox by the same athor it allows to simlate ision-based ontrol systems for robots: xes/robotis-toolbox/

40 An example of IBVS sheme The robot is an ideal positioner For points in spae are assigned: the blok amera based on the rrent position/orientation of a amera retrns the image featres of the for points These are ompared to the desired featres A blok is aailable that omptes the interation matrix. A psedo-inerse of sh matrix is then ompted

41 An example of IBVS sheme Time histories of the featre errors Time histories of the amera position oordinates