Visually Guided Microassembly Using Optical Microscopes and Active Vision Techniques

Transcription

1 Proceeding of the 1997IEEE tntemational Conference on Robotic and Automation Albuquerque,New Mexico - April 1997 Viually Guided Microaembly Uing Optical Microcope and Active Viion Technique BarmehwarVikramaditya Bradley J. Nelon Abtract Robut micromanipulation trategie are needed in order to overcome a potential technology barrier to the commercial development of complex microelectromechan ical (MEMS) device. Development of robut trategie require that two critical iue be aakireed that are not applicable to the traditional macro manipulation domain. Thee two iue include the much higher relative poitioning accuracy required, and the vatly different mechanic of manipulation. Both of thee dt~erence ugget the need to ue micromanipulation trategie guided by real-time enor feedback. In thi pape~ we preent experimental reult that invetigate the ue of high reolution optical ytem with controllable intrinic and extrinic parameter for microaembly application. Reult of two active viion trategie, depth-from-defocu and viual ervoing, uing an optical microcope are preented. Depth-from-defocu reult indicate a repeatability of 8pm out one order of magnitude greater than the wavelength of light) i achievable. Hual ervoing reult demontrate viually ervoed motion over large cale of motion with ubmicron repeatability being achieved. The abili~ to ervo over large cale i made poible by the ue of a unique formulation of the image Jacobian for an optical microcope. 1. Introduction If a microdevice mut be made of different material, ha a complicated geometry, or i manufactured uing incompatible procee, aembly of the device i required. Although automatic aembly ha been a reearch iue for over two decade, almot all of thi reearch ha been in a macro domain. Important difference between manipulation in macro and micro domain exit and mut be directly addreed in order to develop robut micromanipulation trategie. The mot obviou difference i the required poitional accuracy of automatic aembly mechanim. In the macro domain, accuracie in the range of 10-3 inche can be achieved with enor-le manipulator. In the micro domain (part dimenion within micron to millimeter Departmentof Mechanical Engineering Univerity of Illinoi at Chicago Chicago, Illinoi range), ubmicron (10-4 to 10-5 inche) accuracie are often required. Thee accuracie are beyond the calibration range of conventional open loop preciion aembly device ued in indutry [15]. The econd major difference i in the mechanic of object interaction. In a micro domain, gravity i often negligible and electrotatic, urface tenion, and Van der Waal force dominate [1][2]. Currently, the mechanic of micromanipu- Iation are poorly undertood, and thu the reult of imple enorle manipulation trategie are unpredictable. Thee two critical difference between macro and micro domain indicate that enor-baed manipulation trategie mut be developed for even the implet micromanipulation trategie, uch a the alignment tak that are prevalent in microaembly. In a macro domain, viually guided manipulation ha been hown to effectively compenate for error in the calibration of camera-len ytem, manipulator, and workpace. Unfortunately (for viual ervoing reearcher), manufacturing engineer uually prefer the cot of trongly calibrated part handling ytem to the complexity of viion ytem. However, in a micro domain precie calibration i highly dependent on preciely modeled kinematic which are ubject to thermal growth error. Two common technique for compenating for thermal error include either the ue of expenive cooling ytem, or waiting hour for the thermal equilibrium of the device to tabilize. Slocum [15] point out that thermal growth error are typically the mot difficult to control and compenate [in preciion machine deign]. Becaue thee type of factor greatly affect the cot and reliability of preciion aembly machine, real-time viion feedback can be ued effectively and economically a a component of a microaembly ytem. In addition to thermal growth, it hould alo be poible to compenate for ome characteritic of poorly modeled micromanipulation mechanic through viion feedback, a i currently done with manual microaembly. In thi paper, we invetigate active viion iue that apply to the ue of optical microcope for viually guided microaembly. A high reolution optical microcope i ued to experimentally determine the performance of two active viion trategie, depth-from-defocu and viual er / IEEE 3172

2 voing, In Section 2, we dicu previou work in micromanipulation. In Section 3, we preent our viual ervoing formulation, and in Section 4 we preent our depth-fromdefocu formulation. Section 5 decribe experimental reult from the two trategie, and Section 6 ummarize our work. 2. The Mechanic of Micromanipulation The mechanic of manipulation in the macro world are predictable, to a degree. For example, when a gripper open, force due to gravity caue the part drop. Thi predictability ha enabled the ucce of many complex enorle manipulation trategie. In the micro world, force other than gravity may dominate. For example, [2] decribe a microaembly cenario in which electrotatic force dominate and caue a part to jump into a gripper before contact actually occur. A the gripper open to place the micropart at it goal, the part may tick to the gripper finger and fail to remain at the deired location. If humidity in the room happen to be high, urface tenion effect can dominate gravitational force, and the part would alo remain tuck to the gripper [1]. It i etimated that for part with major dimenion below 100pm, gravity i negligible and other force dominate [1]. However, thi i only a rough etimate and depend on everal factor, uch a ma denity, urface roughne, humidity, part geometry, electrical grounding, etc. The more accurate tory i that micromanipulation i a very complex proce. Although particular force can be defined, their effect on the proce can only be roughly etimated. For thi reaon, the development of micromanipulation trategie mut incorporate viion feedback to correct for poorly modeled micromanipulation mechanic. Three force, electrotatic, urface tenion, and Van der Waal, tend to dominate interaction between micropart. Some of thee force may be ued to advantage in micropart manipulation. For example, controlled variation in electric potential between object can be ued for graping. Similarly, urface tenion may be controlled through microtube and microheater. Variou micromanipulation technique have been propoed [1][2][12][ 17]. Thi paper focue on alignment trategie uing viual ervoing and depth-from-defocu technique that entail the ue of an active optical microcope. 3. Viual Servoing with an Optical Microcope 3.1. Vkual Servoing Formulation In formulating the viual ervoing component of our ytem, the Jacobian mapping from tak pace to enor pace i needed. We deire a Jacobian for the camera-len ytem of the form x~ = Jv(l))xT (1) where i i a velocity vector in enor pace; J,($) i the image Jacobian matrix and i a function of the extrinic and intrinic parameter of the viion enor a well a the number of feature tracked and their location on the image plane; and XT i a velocity vector in tak pace Derivation of the Image Jacobian Pat work in viual ervoing ha aumed a pinhole camera model in deriving the image Jacobian [13][6] [10] a hown in Figure 1. A feature on a manipulated object i oberved on the image plane with repect to the camera frame {C) at CR but i controlled with repect to the manipulator tak frame {T] at P. Thi model i inappropriate for an optical microcope. Intead, the ray diagram for a typical optical microcope i hown in Figure 2.. The intermediate image i projected at a ditance g behind the poterior principal focu of the objective. By imilar triangle: h = (h, g)/fo (2) where f,, i the poterior objective focal length, and hl i the object ize in the tak frame. Figure 1. Tak frame-camera frame definition with a pinhole camera model. mechanical zoom element Figure 2. Ray diagram for the optical ytem. Thi image i ubequently viewed by the projection eyepiece and project the image after econd tage magnification onto the CCD receptor that i lying at a ditance of c behind the poterior principal focal plane of the eyepiece. Thu, the linear magnification i given by imilar triangle a: h2 = (h c)/fe (3) and the total linear magnification m = h2/h, = (gc)/(.fo Ye ) i given by where g i the optical tube length and i the ditance between the poterior principal focal plane of the objective 2 (4) 3173

3 and the anterior principal focal plane of the eyepiece. For typical microcope g i a contant 160rnm. The coordinate frame for the camera i located at the center of the CCD array and the Zc axi coincide with the optical axi of the len ytem. Then we have () gc ~ Jo fe c (5) = () Sfi c (6) where x and y are the projected coordinate in enor pace. Mapping the motion in tak pace into enor pace optical flow we obtain by differentiating (5) and (6) with repect to time to get the optical flow equation._ g~,xc. f?c x 1 gc ~d, - Sxfo fe c c+ S/eye JoY~2 e ~,2 0 o fe It can be een that c +f, i contant, and thu i = fe (7) Uing (7) and the optical flow equation x=xy-xq+- xc 1 1., fo c X 7 H ce e f. we get j = y ~ y ;+: 1., fo ~ -yf7 (9) b] e o If the camera frame undergoe tranlation and rotation with repect to the current frame and CV= [Xc~c Zc1 r and % = [ Cl)Icmy=alzc] are the repective velocitie then c Ṗ = CV CQ x CP where CP i the motion velocitie due to camera frame motion. Expanding the expreion give xc = -xc - Oyczc + OIZCYC (lo) Yc = - yc O)zcxc + (aiczc (11) Zc = - ic - coxcyc + O@Yc (12) Uing (5), (6), (10), and (11) and ubtituting into (8) and (9) we get &=_ mxc m@yczc+ o) Y Zcy -x C+f + ; -~ xjo T x x x [} e o j ol c +fe Yjo = AC+ Aczc - %cx - y ; Sy fe c f Y Y [) From the above equation the Jacobian can be formulated a T XS = Jv Cv c~ ~,), j; T [ 1 where X = ; j~, Jv repreent the Jacobian for the [1 ytem and contain ytem parameter uch a focal length, pixel pacing, optical tube length and total linear (8) magnification. Thi i imilar to the reult for an extended Jacobian propoed in [6] for a zoom len. The final image Jacobian matrix i given a Zcm Y c +f< Ty-3 x x f, [1f;c Jv = x (13) Zcm X Sx Y c +f, o-~o o. f Y [1 cf= Y Y Y Generally everal feature are tracked. Thu, for n feature point the Jacobian i of the form. where Ji(t) i the Jacobian (13). (14) matrix for each feature given by 3.3. Vkual Servoing Controller The tate equation for the viual ervoing ytem i created by dicretizing (1) and rewritingthe dicretizedequation a x(k+ 1) = x(k) + TJ (k)u(k) (15) where x(k)e R2M (M i the number of feature being tracked); T i the ampling period of the viion ytem; and u(k)= XTi ~i~ OX,WY,@z~ T i the manipulator end- [ effecter velocity. The Jacobian i written a Jv(k) in order to emphaize it time varying nature due to the changing feature coordinate on the image plane. The intrinic parameter of the camera-len ytem are contant for the experimental reult to be preented. The viual ervoing controller i formulated baed on controlled active viion [13]. The control objective of the ytem i to control end-effecter motion in order to place the image plane coordinate of feature on the target at ome deired poition. The deired image plane coordinate could be contant or changing with time. The control trategy ued to achieve the control objective i baed on the minimization of an objective function that place a cot on en-or in feature poition and a cot on providing control energy. F(k+ 1) = [x(k+ l)-xd(k+ l)] TQ[x(k+ l)-xd(k+ 1)1+UT(k)LIf(k) (16) Thi expreion i minimized with repect to the current control input u(k). The end reult yield the following expreion for the control input u(k)= -( TJvT(k)QT./v(k)+ L)-] 7 JvT(k)Q[x(k)-xD(k + 1)] (17) The weighting matrice Q and L allow the uer to place more or le emphai on the feature error and the control input. Their election effect the tability and repone of the tracking ytem. The Q matrix mut be poitive emidefinite, and L mut be poitive definite for a bounded repone. Although no tandard procedure exit for 3174

4 chooing the element of Q and L, general guideline for a macro ytem can be found in [13], along with ytem model and control derivation that account for ytem delay, modeling and control inaccuracie, and meaurement noie. The meaurement of the motion of the feature on the image plane mut be done continuouly and quickly. The method ued to meaure thi motion i baed on an optical flow technique called Sum-of-Square-Difference (SSD). A more complete decription of the algorithm and it implementation can be found in [10]. 4. Depth-From-Defocu for Optical Microcope 4.1. Depth of focu and Depth of field Depth-from-defocu ha been tudied extenively a a technique for recovering depth etimate from the limited depth-of-field exhibited by optical lene, for example [4][9][14]. In the pat, active viion reearcher have aumed that depth-of-field i determined by pixel ize. Thi i becaue magnification i relatively low for macrodomain tak. For example, in [16] the depth-of-field formulation i given a A=Da a~ p(d -fl a~+p(d-fl ) (18) where A i the depth-of-field, D i the focu ditance, a i the len aperture, ~ i the len focal length, and p i the minimum of the x and y pixel dimenion on the CCD array. The above formulation i valid for optical ytem well approximated by a pinhole camera model. However, for low aperture ytem, i.e. large magnification ytem uch a optical microcope, diffractive effect between wavelength of light dominate, and A i given a * on+~ 2A2 7mA (19) where ~. i the wavelength of light in a vacuum, n i the diffractive index of the len, A i the numerical aperture of the len ytem, and m i the magnification of the optical ytem [8]. The ignificance of thi equation i that extremely mall depth-of-field reult, i.e. A i on the order of the wavelength of light. Thi provide the opportunity to calculate depth from active microcope ytem with accuracie approaching the wavelength of light, which can be ued to advantage for microaembly Focu Meaure In order to etimate depth-from-defocu a focu meaure i needed. Many focal meaure have been propoed in the pat, and an excellent comparion of many of thee can be found in [7]. A technique baed on an analyi of the frequency content of image patche i ued in order to determine where the image i in maximum focu. Thi technique i decribed in detail in [11] and i baed on an analyi of the Fourier Tranform of image patche. 5. Experimental Reult 5.1. Hardware Setup Experiment were conducted with a Mitutoyo FS60 optical microcope and Wentworth MP950 IC probe tation hown in Figure 3. The probe tation wa retooled for motion control uing high preciion Kollmorgen bruhle DC motor. Image proceing and viual ervoing control calculation were performed with a viion ytem coniting of a digitizer and multiple TMS320C40 DSP S. The motor were controlled uing a Delta Tau PMAC-PC ervo card. Both the viion ytem and the PMAC communicate directly over a PC ISA bu. The ervomotor PID loop i cloed at 2.2KHz on the PMAC card uing encoder feedback. The viion ytem i able to track up to 5 16x 16 feature template at 30Hz. The viual ervoing loop i cloed at thi frequency. Figure 3. Experimental etup 5.2. Viual Servoing Reult The tep repone of the viual ervoing ytem with four different objective lene wa teted. Overall reult are hown in Table 1. Image of a miniature canning electron microcope electrotatic column [3] which wa ervoed by the poitioning tage of the probe tation are hown in Figure 4. Settling time for each of the four trial wa quite imilar, approximately O.1, a hown in Figure 5. Thi demontrate that the model of the microcope developed i an accurate repreentation of the optic. Optimal performance wa achieved by tuning the value of the diagonal term in the control gain matrix Q in (17). Relatively mall adjutment in Q between magni- 3175

5 fication were required in order to achieve optimal performance. Critically damped repone wa eaily achieved. It hould be noted that, although the 500X len ha a greater magnification than the 200X len, they both have identical numerical aperture. Therefore, the reolution with which object can be poitioned relative to one another i of a imilar magnitude. Thi reolution contraint wa manifeted in the deadband that wa purpoely introduced in the viion tracker in order to reduce ocillation in commanded motion due to image tracking noie me {E o 01 0,2 0, ,5 1O;x... 4,$ b -- id 3Wrn 20X / loox + 200X ~ : i : \ X 500X Figure 4. Image of miniature canning electron microcope electrotatic column [3] ued for tracking at four different magnification ; irmyl o ETM 20X loox 200X 500X NA Gain R A Step Step Rep. (pm) (~m) (pix) (~m) (w) , Table 1: Viual Servoing Reult with Four Different Objective Lene. TM:total magnification, NA:numerical aperture, Gain: Experimentally tuned value of diagonal element of Q, R:reolution, A:depthof-field, Step: commanded tep ize in pixel and micron, Rep:etimated repeatability of poitioning ytem baed on controller deadband. {= o Depth-from-Defocu Reult Reult from the depth-from-defocu trategy are hown in Figure 6. The depth wa determined by moving the probe tage along Zc in tep of 8.6pm. Thi tep ize wa Figure 5. Step repone of micro-viual ervoing ytem at four different magnification. 3176

6 determined to be the depth-of-field according to (19) when the len configuration reult in a total magnification of 100x. A frequency baed focal meaure wa ued to determine dene depth map for image of a 6 in. mil file hown below. A ingle erration on the file (approximately 220Lm in height) wa viewed under the microcope and multiple 400x6 pixel ize image were taken at varying level of focu. The depth map demontrate the applicability of thi technique for relative poitioning of object along the optical axi. Thi add another degree of freedom to the enor-baed poitioning ytem. F7xeL Rxel Figure 6. Depth map for a ingle erration (hown in the white box) of the 6 in. mil file pictured in the inet. 6. Concluion Active viion trategie applied to high reolution optical ytem how promie in overcoming a technology barrier to the automated microaembly of hybrid MEMS device. In thi paper, we have theoretically and experimentally invetigated the ue of active viion technique with an optical microcope. Viual ervoing reult demontrate robut control down to ubmicron repeatability. Thi highly precie repeatability wa achieved with viually ervoed motion at relatively high peed in the mml range. Depth-from-defocu technique demontrate that it i poible to create depth map with a repeatability of approximately 8Lm, and therefore poible to ue thi technique for aligning object along the optical axi in real-time. The goal of thi work i to develop microaembly trategie for hybrid MEMS device that compenate for modeling uncertaintie inherent in the micro-domain, uch a thermal growth, humidity effect, electrotatic force, etc. Acknowledgment Thi reearch wa upported by the National Science Foundation through Grant Number IRI and CDA We would like to thank Alan Feinerman for providing a miniature SEM aembly. Reference [1] F. Arai, D. Ando, T. Fukuda, Micro Manipulation Baed on Micro Phyic-Strategy Baed on Attractive Force Reduction and Stre Meaurement: Proc IEE.WRSJ Int. Conjl on Intelligent Robot and Sy, (IROS95), 2: , Pittburgh, Augut 5-9, [2] R.S. Fearing, Survey of Sticking Effect for Micro Part Handling, Proc IEEE/RSJ Int. Confi on Intelligent Robot and Sy. (IROS95), 2: , Pittburgh, Augut 5-9, [3] A.D. Feinerman, D.A. Crewe, D.C. Perng, SE. Sheaf, and A.V. Crewe, Sub-centimeter micromachined electron microcope: J. Vat. Sci. Technol. A, 10(4), pp , [4] P. Groman, Depth from Focu, Pattern Recognition Letter, 5, pp , [5] B.K.P. Horn, Robot Viion, MIT Pre, [6] K. Hooda, H. Moriyama, M. Aada, Viual Servoing Utilizing Zoom Mechanim, Proc IEEE Int. Con$ on Robotic and Automation, pp , [7] E. Krotkov, Focuing, Int. J. computer Viion, 1, pp , [8] L.C.Martin, The Theory of the Microcope, American Elevier, [9] S.K. Nayar, Y. Nakagawa, Shape from Focu, IEEE Tran. on Pattern Analyi and Machine Intelligence, 16, pp , [10] B. Nelon, N.P. Papanikolopoulo, and P.K. Khola, Viual ervoing for robotic aembly, Viual Servoing Real-Time Control of Robot Manipulator Baed on Viual Senory Feedback. ed. K. Hahimoto. River Edge, NJ: World Scientific Publihing Co. Pte. Ltd. pp , [11 ] B. Nelon and P.K. Khola, Integrating Senor Placement and Vkual Tracking Strategie, Experimental Robotic III: The Third International Sympoium, Kyoto, October , ed. T. Yohikawa and F. Miyazaki, Springer-Verlag London Ltd., London, pp , [12] K. Ozawa, S. Ogawa, H. Ihida, and Y. Hattori, High-peed Meauring Equipment of Fiber Core Poition of Optical Fiber Array Uing Piezo Actuator, Proc IEEE Int. Con$ on Robotic and Automation, pp , [13] N.P. Papanikolopoulo, Nelon, B. and Khola, P.K., Full 3-d tracking uing the controlled active viion paradigm, Proc IEEE Int. Symp. on Intelligent Control (ISIC-92), pp , [14] A. Pentland, A New Sene of Depth of Field, IEEE Tran.on Pattern Analyi and Machine Intelligence, 9(4), pp , [15] A.H. Slocum, Preciion Machine Deign, Prentice Hall, [16] K. Tarabani, R.Y. Tai, and P.K. Allen, Satifying the Reolution Contraint in the MVP Machine Viion Planning Sytem, in Proc. of the 1990 Darpa Image Undertanding Workhop, , [17] Y. Yamagata and T. Higuchi, A Micropoitioning Device for Preciion Automatic Aembly uing Impact Force of Piezoelectric Element, Proc IEEE Int. Con$ on Robotic andautomation, pp ,