sensors ISSN

Transcription

1 Sensors 1, 1, ; do:1.339/s OPEN ACCESS sensors ISSN Artcle Development and Implementaton of Image-based Algorthms for Measurement of Deformatons n Materal Testng Lug Barazzett and Marco Scaon * Department of Buldng Engneerng Scence and Technology, Poltecnco d Mlano, va Marco d Oggono, 18/a, Lecco, Italy; E-Mal: lug.barazzett@polm.t * Author to whom correspondence should be addressed; E-Mal: marco.scaon@polm.t; Tel.: ; Fax: Receved: 1 July 1; n revsed form: 15 July 1 / Accepted: 3 August 1 / Publshed: 1 August 1 Abstract: Ths paper presents the development and mplementaton of three mage-based methods used to detect and measure the dsplacements of a vast number of ponts n the case of laboratory testng on constructon materals. Startng from the needs of structural engneers, three ad hoc tools for crack measurement n fbre-renforced specmens and D or 3D deformaton analyss through dgtal mages were mplemented and tested. These tools make use of advanced mage processng algorthms and can ntegrate or even substtute some tradtonal sensors employed today n most laboratores. In addton, the automaton provded by the mplemented software, the lmted cost of the nstruments and the possblty to operate wth an ndefnte number of ponts offer new and more extensve analyss n the feld of materal testng. Several comparsons wth other tradtonal sensors wdely adopted nsde most laboratores were carred out n order to demonstrate the accuracy of the mplemented software. Implementaton detals, smulatons and real applcatons are reported and dscussed n ths paper. Keywords: automaton; computer vson; constructon materals; dsplacement/deformaton; mage metrology; photogrammetry; vson metrology; targets

2 Sensors 1, Introducton Deformaton measurement durng laboratory testng on constructon materals ams at determnng the ntrnsc characterstcs of the consdered object. The examnaton of the deformaton and the knowledge of the appled load (e.g., a mechancal or thermal load) allows the analyss of the mathematcal model that descrbes the behavour of a constructon element. Several nstruments can be used to measure object deformatons durng loadng tests. However, the most wdely adopted tools are lnear-varable-dfferental-transformers (LVDTs) and stran gauges [1], whch provde the magntude of the dsplacement wth the nvestgaton of the changes of electrcal resstance due to a load. These tools are consdered proven technques, wth an accuracy of ±1 μm or even less, and they gve real-tme data. On the other hand, they only provde 1D measurements lmted to the area n whch the sensor s fxed. In addton, a connecton wth a control unt s necessary and after destructve tests these knds of sensors can be damaged. Thus, LVDTs or stran gauges are not a convenent choce n the case of extensve analyss on the whole body, n whch a great number of 3D ponts wth a good spatal dstrbuton must be measured. Image-based methods can analyse the whole deformaton feld of a body by trackng a vast number of ponts dstrbuted on the object. Images contan all the nformaton to derve 3D measurements from multple D mage coordnates wth lmted cost and good accuraces. In fact, mage-based technques have been used n several applcatons whch nvolve the determnaton of the shape of a body and ts changes, wth satsfactory results n terms of completeness, precson and tme [-5]. These are also known as vson metrology applcatons. Some commercal cameras (or photogrammetrc ones), trpods, lght sources and synchronzaton devces are the components needed to obtan hgh precson 3D measurements for a large number of ponts. However, the extracton of 3D nformaton from D mages s not a smple ssue and algorthms for mage processng must be developed n order to obtan an automated elaboraton. The goal of mage-based methods n materal testng s the estmaton of accurate 3D coordnates startng from D measurements n the mages through a perspectve mathematcal formulaton between the object and ts projecton nto several mages. Some commercal software allow the analyss of the dynamc changes of several targets dstrbuted on the object n an fully automatc way, but f markerless mages are employed no commercal automatc solutons are avalable on the market. Moreover, the procedure becomes a full-feld non-contact technque only wthout targets, when the natural texture of the object s drectly used (generally after a prelmnary enhancement wth flters that modfy the local contrast of the mage). For nstance, ths knd of analyss provdes the detecton and the measurement n fluds, where LVDTs and stran gauges cannot be employed. Bascally, the precson achevable wth mage-based technques depends on the sze of the nvestgated elements [6]. For experments n a controlled envronment a standard devaton of the object coordnates n the order of 1:1, of the largest object dmenson s expected, but durng analyss n repeatable system confguratons (e.g., wth fxed cameras) a precson of 1:5, has been acheved [7,8]. In [9] a hyper redundancy network s used for the study of the deformatons of a rado telescope, wth an accuracy n the range of 1:58, to 1:67, obtanable through the use of more mages than those strctly necessary. For nstance, two mages per staton enhance the effectve angular measurement resoluton of a factor of 1.4, whle four mages per staton lead to a factor of. In

3 Sensors 1, flm-based photogrammetrc measurements of bg antennas ths dea has led to an accuracy approachng one part n a mllon [1]. As the technologcal development of commercal low-cost cameras s rapdly ncreasng, mage-based methods and low-cost software are commonly used n several sectors (e.g., archaeology [11], geology [1], medcne [13]) wth good results n terms of precson. However, photogrammetrc methods have a lmted use for materal testng n cvl engneerng. Ths s manly due to the lack of automatc processng algorthms and user-frendly software, especally n the case of markerless mages. Some low-cost dgtal cameras and targets can be a convenent soluton for the analyss of the whole surface of an object. The employed targets can be really nexpensve (a pece of whte paper wth a black mark s suffcent for many applcatons), whle n the case of more exhaustve experments they can be prnted on metal plates or can be made of retro-reflectve materals. The centre of the target can be automatcally measured wth a hgh precson (up to ±.1 pxel) n a fully automated way, mprovng the precson of the correspondng 3D coordnates. A group of targets permanently fxed on the object provdes a regular mesh for all deformaton analyses. These dense ponts can approxmate the deformaton feld of the whole body. A fundamental advantage of an mage-based method s the possblty of analysng more targets than those strctly necessary, wthout ncreasng the cost of the test and wth a lmted worsenng of the processng tme. However, n some applcatons targets cannot be employed (e.g., for flud elements) and automatc methods based on the natural texture of the body must be developed. Ths knd of analyss s more complcated, especally n the case of bad surfaces wthout detals. Ths fact lmts the use of mage-based methods nsde cvl engneerng laboratores. Ths paper presents three mage-based algorthms capable of analysng the deformaton feld of a generc object durng a loadng test. These methods work wth targets but also wth markerless mages and can determne the 3D coordnates of a huge number of ponts n an automatc way. They are currently employed n some cvl engneerng laboratores, where several buldng materals and structural elements are tested wth satsfactory results n terms of accuracy. In several applcatons these methods ntegrate or substtute tradtonal sensors and provde addtonal nformaton, whch are useful for more complete and detaled nvestgatons. We focus on the measurement of a fnte number of ponts wth a good dstrbuton, whle other exstng approaches present the extenson of the measurement problem to the whole surface of the body [14]. Our choce s motvated by the needs of structural engneers, who were nterested n the analyss of partcular ponts n crucal locatons. Another dfferent approach s presented n [15], n whch a system for modellng the nteracton behavour of real objects (ncludng deformatons) was developed. A fully automated mage-based measurement system s descrbed n [16]. The frst tool here presented allows the estmaton of crack varatons n flud fbre-renforced specmens (Secton ). Ths s a new non-conventonal applcaton for whch there are no commercal solutons. Ths task requred the development of an ad hoc sensor for mage acquston and an algorthm for the automatc dentfcaton of the crackng process. In partcular, a CMOS sensor was transformed nto a crackmeter devce. The other tools were developed for dynamc D and 3D measurements on standard structural elements (e.g., beams, pllars, foundatons, walls ) and can operate wth targets (lke commercal software) but also wthout markers (Secton 3). Ths last opton

4 Sensors 1, provdes new measurements durng tests where only lmted nformaton can be acheved wth tradtonal sensors. In order to demonstrate the potental of these photogrammetrc methods, a theoretcal explanaton and several examples are shown and dscussed, wth a check of the achevable accuracy. Moreover, the advantages (and dsadvantages) about the use of low-cost dgtal cameras are reported and dscussed, wth a comparson wth tradtonal sensors.. Crack Aperture Estmaton n Flud Specmens.1. Overvew Cracks are expected for several constructon materals durng ther servce ablty [17], especally n the case of renforced-concrete elements. However, a sgnfcant varaton of the crack aperture can lead to a progressve deteroraton of the steel renforcement rod wth a consequent worsenng of the stablty of the structure. For these reasons, crack montorng plays a fundamental role durng nspectons and laboratory probes. The study of new technques able to avod or mtgate the crackng process s a feld of research of prmary mportance n cvl engneerng. Some nnovatve solutons based on the use of fbres (n addton to exstng mxtures) demonstrated the possblty to lmt the propagaton of the cracks. Laboratory testng on fbre-renforced specmens allow one to study the effect of dfferent fbres and mxture components (water, cement, sand ), n order to determne the best compromse for real applcatons. A tradtonal analyss s based on the study of the aperture, shape, locaton and orentaton of cracks wth small specmens that smulate the behavour of the real object. To montor the aperture of a crack durng standard tests stran gauges are generally used. However, cvl engneers needed more exhaustve and specfc measurements than those achevable wth these standard sensors. In fact, stran gauges allow only one-pont and one-dmensonal measurements [18], after a stable applcaton of the sensor on the specmen. These knds of nformaton are useful n the case of tradtonal laboratory applcatons, but they are nsuffcent for exhaustve and detaled scentfc analyss focused on the development of nnovatve materals. Indeed, the needs of cvl engneers requred the development of more advanced solutons. Another ssue regards the state of the body: all measurements on the specmens begn after the castng, when the specmens s lqud, and prelmnary data about the number of expected cracks and ther postons are not avalable. For these reasons, a new soluton capable of analysng the deformatons n these partcular workng condtons was necessary. The developed tool for such measurements s composed of a mechancal arm carryng a dgtal camera (Fgure 1) and an algorthm for automatc mult-mage processng. A thermal chamber, whch contans the sensor used to photograph the specmens, allows controlled and stable testng condtons. Then, an automated postonng system moves a CMOS INFINITY 1-3 camera (3.1 megapxels) equpped wth a mm lens over the specmen. All mages are captured for several postons at dfferent epochs by plannng the trajectory of the mechancal arm and the number of shots. Each mage covers an area of 18 mm 13.8 mm roughly and ts nspecton allows the detecton of very small detals. The robotc arm moves the camera along prefxed drectons n order to photograph the whole specmens. Ths operaton can be repeated several tmes and the mages can be used for a mult-epoch

5 Sensors 1, nvestgaton to detect dynamc varatons. As a typcal test may requre several days, hundreds of mages can be collected. Therefore, a useful choce (before analysng all mages) s related to a rapd vsual check of the mages of the last epoch n order to select the mages whch contan a crack. The remanng mages can be removed from the elaboraton to speed up the whole processng. Fgure 1. The developed system for crack aperture estmaton... Crack detecton and Image Coordnate Measurement The estmaton of the crack aperture s carred out wth an automated algorthm capable of detectng a crack n each sngle mage by measurng ts border coordnates (n pxels). Then, a procedure based on smple geometrc consderatons between the camera and the specmen allows the estmaton of the crack aperture n metrc unts. Image coordnates can be automatcally measured wth the methodology proposed n [19], n whch a tool named IMCA (IMage Crack Aperture) was developed by the authors for crack measurement durng structure nspectons. The same procedure s used n these experments, but some changes were necessary to adapt the algorthm n the case of fbre-renforced specmens and repettve expermental condtons. The new procedure uses a flterng algorthm that detects the mage coordnates of a crack by means of an ntellgent reducton of the colour depth. Ths technque can be assumed as a converson of the orgnal RGB mage to a new bnary mage ( s the crack whle 1 means background). As the radometrc content of a generc mage s expressed by three functons whch correspond to the colours (red, green and blue) of Bayer s flter [] coverng the sensor, ths new strategy uses ths nformaton to automate the measurement phase. We denote these functons as R(, j), G(, j) and B(, j), n whch (, j) are the coordnated of a generc pxel. The behavour of the RGB functons along a cross-secton of a crack s qute smple: they rapdly decrease and ncrease n the crack and have a qute constant value far from the crack. Startng from these smple consderatons, a flterng algorthm was developed. It s based on the mnmum values R mn, G mn, B mn of the functons (n the mddle of the crack) and the values R A G B B C n whch the slope of the functons changes. These values can be estmated wth the analyss of some cross-secton and then used for the completon of the test. A global level L can be estmated for the crack as follows:

6 Sensors 1, R L 3 R mn A G G mn B B B mn C (1) and represents the parameter to flter the mage. For any generc pxel (, j) of the mage a local level L (, j) can be estmated by consderng ts radometrc content: 1 Rmn Gmn Bmn L' (, j) () 3 RP GP BP The creaton of the fltered mage s carred out by comparng L and L as follows: L (, j) L > L (, j) L = L (, j) L < pxel CRACK; pxel BORDER; pxel CRACK or BORDER. Ths method uses the whole RGB content of an mage, whle other exstng technques (e.g., [1,]) work wth ther combnatons and need a prelmnary converson to create a new gray-scale mage. Ths choce s motvated by the results obtaned wth the proposed methodology, although we are testng a procedure whch uses the green channel. The man advantage durng laboratory testng, wth repettve workng condtons, s the possblty of estmatng a prelmnary global level, whch can be consdered a constant for specfc applcatons. In fact, f llumnaton condtons are stable (n ths case a LED s permanently employed for all mages) the global level does not vary sgnfcantly durng the test. In addton, small errors n ths phase can be consdered systematc errors and can be removed durng the estmaton of the aperture varatons. After some tests we estmated an optmal level for fbre-renforced concrete elements equal to Crack Aperture Estmaton To estmate the crack aperture a transformaton between mage and object spaces must be employed. As the analyss starts wth a flud specmen (ts external surface s horzontal), the robotc arm was assembled n order to generate D horzontal movements. Wth ths partcular confguraton, mage and object planes (or camera sensor and specmen surface) are parallel and the camera maps the object through a smlarty transformaton where the scale s the only unknown. Ths smple soluton allows an easy computaton of object coordnates wthout usng more complex transformatons requrng the knowledge of several parameters. A more detaled descrpton about ths procedure s shown n Secton 3., because an extenson of ths transformaton s used n another tool, whle for ths partcular applcaton the scale number s the ambguty. The scale factor was estmated by measurng the sze of a pxel projected onto a reference object (a small metal plate) placed on the specmen. The sze of ths object was measured wth a callper: t s suffcent to dvde the wdth of the plate by the number of pxel pcturng the object to determne the scale factor. Wth the INFINITY camera a pxel covers an area of 9 μm 9 μm, whch s also the accuracy of the mplemented tool (see next secton for further detals). The output nterface of the tool, whch gves a graphcal and numercal vsualzaton of the crack aperture, s shown n Fgure. The procedure s qute smple: the user just has to select a crack n an mage and the dynamc analyss can

7 Sensors 1, be carred out n automatc way. Ths task s performed by consderng correspondng mages taken at dfferent epochs. Lastly, the aperture varatons can be estmated by usng the varatons detected at dfferent epochs (relatve measurements). Fgure. Some results wth the mplemented software: crack borders and the estmated aperture..4. Accuracy of the Method To check the accuracy of the mplemented method a comparson wth other sensors s mandatory. Nowadays, a system capable of measurng the aperture varatons n flud elements wth an accuracy and a densty better than the mplemented tool s not avalable. Ths means that accuracy cannot be checked wth experments on flud specmens. To overcome ths drawback we developed an alternatve soluton wth a sold object and a specal mcrometrc sledge (Fgure 3), whch s composed of two plates (the frst one s fxed whle the second one can be moved wth two mcrometrc screws n order to smulate a planar moton). Two mechancal gauges provde the magntude of the dsplacements wth an accuracy superor to ±.1 mm. The sledge allows one to smulate the aperture of a synthetc crack, where all ponts have the same dsplacement (rgd moton). Anyway, ths s suffcent to check the accuracy of the mage-based method. A Nkon D8 camera equpped wth a 9 mm lens was placed over the sledge n order to determne the smulated varaton wth the mplemented tool. The mathematcal relaton between mage and object spaces was estmated wth a specal calbraton frame, composed of ponts wth known coordnates (see Secton 3.). From a theoretcal pont of vew, the precson of object coordnates σ XY can be estmated wth a smple formula: d c XY xy (3)

8 Sensors 1, where d s the camera-object dstance, c the focal length of the camera and σ xy the mage coordnate precson. The fundamental assumpton of Equaton 3 s the parallelsm between mage and object planes. Fgure 3. The sledge used to check the accuracy of the mage-based tool. However, Equaton 3 gves a theoretcal precson that must be compared wth real data (useful for a prelmnary knowledge about the expected accuracy). In our tests we placed the camera wth a dstance d 1 equal to 6 mm, then we reduced the dstance to d = mm. Both mechancal and mage-based measurements were compared and the results showed a standard devaton of the dfferences of ±.37 mm (d 1 = 6 mm) and ±.1 mm (d = mm). Supposng that the precson of the flterng algorthm s equal to ±1 pxel, a theoretcal precson of ±.4 mm and ±.14 mm can be estmated wth the camera used n both confguratons (pxel sze s.61 mm). Ths means that the precson of the mplemented tool s equal to the GSD (Ground Samplng Dstance), whch represents the projecton of a pxel onto the object. To mprove the precson of the object coordnates the camera-object dstance can be reduced or the focal length can be ncreased. However, n both cases the angle of vew s progressvely reduced and a smaller part of the object can be maged. The best choce s a compromse between precson and maged area. Several other comparsons valdated the proposed results and confrm the expected accuracy n the case of the INFINITY camera (a pxel projected onto the object s ±.9 mm). 3. D Deformaton Measurements 3.1. Overvew of the Implemented Method Durng some tests the analyss of 3D movements s not strctly necessary. In fact, f the analyzed object s flat (e.g., the external surface of a beam), the estmaton of a D moton s more than

9 Sensors 1, suffcent for several experments. Ths fact leads to a smplfcaton of the measurement problem, wth a reducton of the degrees of freedom for a generc pont of the object. Moreover, there s an advantage n terms of cost: a sngle mage for each epoch becomes suffcent to analyze the movements of all ponts. Startng from the mage coordnates of pont (x, y ) the correspondng object coordnates (X, Y ) can be calculated by usng a D homography. Beyond the reducton of the number of cameras (a fxed camera s suffcent, thus synchronzaton devces are not mandatory), a rgorous calbraton of the camera s not needed. However, the nfluence of an uncalbrated camera on the fnal results should be carefully consdered, although the whole operaton can be carred out wthout knowng the ntrnsc parameters of the camera used. Ths could be an advantage when no nformaton about the used sensor s avalable. Ths aspect s analyzed wth more detals n Secton 3.5. The equpment ncludes a camera placed on a trpod and an algorthm able to track all mage ponts and to estmate real movements. The acquston frequency depends on several factors and vares wth the nvestgated object and the selected load. For ths reason t s not possble to fx an optmal value for every experment. Ths means that an ad-hoc samplng frequency must be estmated before the begnnng of the test by consderng several factors (e.g., load, expected deformaton, object, texture ). Probably, the best soluton s to acqure more mages than those strctly needed. Then, mages can be decmated. The mplementaton of an ad-hoc software was necessary because commercal solutons for fully automated mage processng are not avalable on the market. Some commercal packages (e.g., Australs, Wtness, PhotoModeler ) work wth targets, but f markers cannot be employed the elaboraton needs tedous nteractve measurements. In addton, these software packages generally work wth two or more cameras, whle the stuaton wth a sngle mage needs a dfferent mathematcal formulaton. 3.. Target Localzaton and Matchng Several targets dstrbuted on the object are a vald support n mage-based deformaton measurements. A regular mesh of targets allows one to analyze the whole surface of the body, whle the use of tradtonal sensors (e.g., stran gauges or LVDTs) ncreases the cost and needs complex connectons wth control unts. For these reasons, photogrammetrc targets are a cheap soluton wth a smple connecton on the analyzed body. Durng several real surveys, all targets can be prnted (e.g., a black dot wth a whte background), whle for more advanced and extensve analyss they can be made of metal. Fgure 4 shows a typcal analyss wth targets: the surface of the beam can be consdered a flat object and a regular mesh allows the measurement of ts deformaton feld. In ths case, sx LVDTs were used, but they offer few measurements along prefxed drectons. Wth ths n mnd, the target-based mage soluton s more convenent. All targets can be automatcally matched by usng a D normalzed cross-correlaton technque between a target template and the mage [3]. Bascally, the method supposes that the target template (a perfect mage of the target, whch can be easly created wth any software for mage vsualzaton or edtng) s smlar to the target used durng the survey.

10 Sensors 1, Fgure 4. A beam can be consdered a flat object. An automated search of the target(s) n the whole mage can be carred out by comparng the template wth the local content of the mage (a prelmnary converson of the orgnal RGB mage to a new grayscale one must be performed). The measurement of the centre of the target n the mage s carred out by movng the template f(x, y) n a search wndow (or n the whole mage), wth a sequence of small dsplacements (e.g., one or two pxels). For each poston the normalzed correlaton coeffcent ρ(x, y) between f(x, y) and the local content of the mage g(x, y) ( patch ) can be estmated wth the relaton: ( x, y) N f ( x' u, y' v) g ( x u, y v) N M f ( x' u, y' v) g( x u, y v) f un vm un vm M N M f un vm g 1/ g where (x, y, x, y ) are the centers of template and patch and μ f and μ g are the mean values of ntensty of f and g. The sze of the patch s (N + 1) (M + 1) pxels. The centre of the target can be assumed at max [ρ(x, y)] wth an addtonal constrant on the mnmal value (e.g.,.7). Sub-pxel precson can be acheved by estmatng the frst dervatve of ρ(x, y) [4]. Ths method s easy to mplement and fast from a computatonal pont of vew [5], but t takes nto account only two shfts between template and mage. In the case of real surveys there are several other deformtes such as scale varatons or rotatons, affne deformatons, llumnaton changes and so on. They lead to poor results wth ths basc geometrc model. In [6] a modfed cross-correlaton approach was developed to consder all these deformtes by reshapng the patch wth an affne transformaton, whch s more sutable for real surveys. However, we prefer to use the method proposed n [7] and coned Least Squares Matchng (LSM). Startng from a perfect smlarty between the template and the patch: f ( x, y) g( x, y) (5) the LSM method takes nto account a more realstc stuaton, n whch equaton 5 s not consstent and a nose e(x, y) s added: f ( x, y) e( x, y) g( x, y) (6) (4)

11 Sensors 1, To operate wth the Gauss-Markov Least Squares estmaton model g(x, y) must be lnearzed at an approxmate locaton wth a frst order Taylor s expanson: g( x, y ) g( x, y ) f ( x, y) e( x, y) g( x, y ) dx dy g gxdx g x y y dy (7) where an affne transformaton s consdered as geometrc model (a radometrc correcton s not used because llumnaton condtons are stable f lght sources are used durng experments n controlled condtons): x a a x a y y b b x b y The parameters a and b (shfts) are unknown values that ndcates the centre of the target, whle the other coeffcents can be used to adjust shape deformatons. Equaton 6 can be cast n the form: 1 1 (8) f ( x, y) e( x, y) g g y da x ( x, y) g da g db g x db g y db y y 1 x y g x da x 1 (9) Fnally, the unknown parameters can be grouped nto a vector: and the system can be wrtten as: T [ da, da1, da, db, db1, db ] n whch l k = f(x, y) g (x, y ). The soluton s gven by: x (1) Ax l e (11) T 1 T x ( A A) A l (1) In order to complete the lnearzaton wth a Taylor s expanson, a set of ntal approxmatons for the unknowns s chosen as follows: da db ; da1 db 1 ; da db1 and then the soluton s computed teratvely wth a stop crtera (e.g., on the estmated sgma-naught). In the mplemented verson of the LSM algorthm, some tests to check the determnablty of parameters (1) were ncluded; further nformaton about ths aspect can be found n [8]. The LSM method ensures hgh precson measurements (up to ±.1 pxels) and s an optmal choce n the case of targets. However, t cannot be consdered as an alternatve to cross-correlaton: cross-correlaton provdes good approxmate values about target locatons and LSM refnes center coordnates. Thus, the combned use of both these technques s strctly mandatory n order to automate the whole analyss. (13) 3.3. Computaton of Object Coordnates and Dynamcal Analyss Object coordnates can be calculated by usng mage coordnates and a transformaton between mage and object spaces. In the case of flat objects all ponts le on the same plane and the mathematcal transformaton between mage and object spaces can be descrbed wth a D homography.

12 Sensors 1, The relaton between an mage pont n homogenous coordnates (x, y, 1) T and the correspondng object coordnates (X, Y, 1) T s: or, n a compact form: X a1 Y b1 c1 a b 3 1 c c3 a3x b y 1 (14) X Hx (15) H contans the parameters of the transformaton and has a rank defcency, thus only eght elements are ndependent and an external constrant must be used (we set the last elements c 3 equal to 1). To obtan nhomogeneous coordnates t s suffcent to dvde mage and object coordnates by ther thrd coordnate. Ths leads to the nhomogeneous form of the planar homography: X a1x a y a c x c y Y b1 x b y b3 c x c y (16) whch are lnear n the elements of H. In fact, a multplcaton by the denomnator leads to: c x X c 1 1 c x Y c y X X y Y Y a x b x 1 1 b a y a y b 3 3 (17) To estmate the eght coeffcents of H some correspondences between the mage and object spaces must be known (at least four ponts). Gven m correspondng ponts, Equatons (17) provde a system of m equatons that can be solved va Least Squares [9]. The measurements of the object ponts needed to estmate H can be carred out wth several technques (e.g., total staton, calbrated frames ). Moreover, H s estmated for the frst epoch and then assumed constant durng the next phases. Indeed, f the camera s placed on a stable trpod the transformaton does not change and D dsplacements can be drectly estmated by usng mage coordnates. A better strategy to vsualze the results s based on the removal of the perspectve effect from the mages. Here, the homography H estmated for the frst mage can be appled to all mages before measurng the mage coordnates. The measurement of mage coordnates wth the rectfed mages provdes object coordnates. Fgure 5 shows some rectfed mages for the beam sequence. As can be seen, mages present a dstorton that cannot be removed f calbraton s unknown. Ths generates an error n the fnal results (more detals about the nfluence of mage dstorton are presented n Secton 3.5). The crcle represents the poston of each target for the frst epoch, whle the length of each lne gves D movements (a known scale factor s needed). Fgure 6 shows a detal. In ths case the postons of some targets measured at dfferent epochs are shown, n whch the frst mage s used for ths vsualzaton. Target coordnates can be measured nteractvely and ths knd of vsualzaton offers a global vsualzaton about the deformaton feld.

13 Sensors 1, Fgure 5. Some rectfed mages of the sequence and the magntude of the dsplacements. Fgure 6. Target dsplacements projected onto the ntal rectfed mage.

14 Sensors 1, Automated Elaboraton of Target-Less Images Targets are very useful to montor deformatons emergng n loadng tests: they can be easly measured wth a hgh precson and ther applcaton onto the body s smple and cheap. However, n some cases targets cannot be permanently nstalled or can be lost durng the test. To overcome ths drawback a synthetc texture can be generated (e.g., by pantng the object) but we developed a new soluton capable of workng wth target-less mages. It uses the natural texture of the object after a prelmnary mage enhancement. Interest operators can be used to detect a suffcent number of features n the frst mage of the sequence. Then, these features are tracked wth the proposed methodology based on cross-correlaton and LSM along the sequence. Before the begnnng of the test t s hghly recommended to process some mages. Ths operaton s really useful to verfy the qualty of the mages and the possblty to use the natural texture (). In the case of a falure wth the natural features, a procedure based on synthetc corners () can be used. The applcaton of targets onto the object remans the last choce when the prevous method cannot be employed. Several features can be detected n an mage (e.g., corners, edges, regons ) and several operators are avalable (probably too many to be lsted here). For a more exhaustve revew the reader s referred to [3-3]. The method used n the mplemented tool s the FAST (Features from Accelerated Segment Test) operator [33], whch s a corner detector for hgh speed processng, today employed n vdeo analyss and trackng. The functonng of FAST s based on the analyss of a crcle of 16 pxels around a generc corner p. A pxel s a corner f n contguous pxels are all brghter than the ntensty I p of the canddate pxel plus a threshold t, or all darker than I p t. The choce of ths operator s supported by the mpressve number of corners that can be extracted from an mage. However, corners are extracted only for the frst mage of the sequence, whle for the next ones a trackng va cross-correlaton and LSM s used. In some cases mages mght present a bad texture and a lmted number of corners could be extracted. In addton, the dstrbuton of ponts could be nhomogeneous. To solve ths problem a procedure based on a prelmnary mage enhancement can be used. Many methods are today avalable and generally work by consderng global parameters: most software for mage enhancement have automatc functons capable of modfyng the contrast of the mage, but the same level s used for the whole mage. If a homogenous dstrbuton of all ponts s needed ths can lead to a poor soluton. Ths s the reason why we prefer to optmze the contrast locally. Walls [34] proposed an ad hoc mage flter whch splts the mage nto small rectangular blocks. These blocks are progressvely analyzed by consderng ther local statstcs. The Walls flter has the form: where: and: I f ( x, y) r Io( x, y) r1 (18) st r1 csocso (19) c

15 Sensors 1, r ) bmt ( 1 b r1 m o () I f and I o are the fltered and orgnal mages, r and r 1 the addtve and multplcatve parameters, m o and s o the mean and standard devaton of orgnal mages, m t and s t the target mean and standard devaton for the fltered mages, c the contrast expanson constant and b the brghtness forcng constant. Bascally, the user has to select the block sze: a small block (e.g., 7 7 pxels) results n a strong enhancement, whle a large block (e.g., pxels) generates a loss of detal. For each sngle block m o and s o are estmated and the resultng values are assgned to the central pxel of each block, whle for other pxels these values are estmated wth a blnear nterpolaton. The target mean and standard devaton m t and s t are manually selected. Normally, for an 8-bt mage a good choce of the target mean s 17, whle the target standard devaton value can be 5. Good values for the constant expanson constant c are n the range [.7, 1], whle for the brghtness forcng constant b the suggested range s [.5, 1]. An optmal combnaton of all these parameters can be determned wth few tres n whch a vsual check of the fltered mage s suffcent. Moreover, t s possble to extract FAST corners and check ther number and dstrbuton. Fgure 7. Results n the case of markerless mage sequences: (a) orgnal mage and (b) extracted corners, (c) fltered mage and (d) extracted corners, (e) corner reducton accordng to a quas regular grd.

16 Sensors 1, Fgure 7 shows the orgnal mage of a small beam (a) and the detected corners wth the FAST operator (b). As can be seen, few ponts can be measured. The fltered mage (c) provdes more corners (d) wth a better dstrbuton. At the end of the process ponts can be mapped (d) onto a quas regular grd (n ths case each cell s 5 5 pxels roughly). The dynamc analyss s carred out by flterng all mages and trackng the orgnal FAST corners wth cross-correlaton and LSM along the mage sequence, whch must be fltered wth the same parameters. It s also recommended to use stable llumnaton condtons durng the analyss (e.g., external lght sources lke lamps), a very hgh acquston frequency accordng to the duraton of the test (to lmt the dfferences between consecutve mages) and small blocks (e.g., 9 9 pxels) for the flterng process (to reduce the effect of local deformatons durng the test). Moreover, ths procedure should be used when lmted deformatons are expected. Wth these expermental condtons we verfed that only a lmted number of ponts s lost durng the sequence analyss Influence of Camera Calbraton The mathematcal analyss proposed n Secton 3.3 demonstrated that no nformaton about the camera used s requred when the relaton between mage and object spaces s a planar homography. Thus, mage coordnates and few reference object ponts are adequate to complete the elaboraton. Camera calbraton s ntended as the process to estmate the ntrnsc parameters of the camera, comprehendng the prncpal dstance, prncpal pont and dstorton coeffcents. A good calbraton s an essental prerequste for precse and relable measurements from mages, and s wdely adopted n several surveys where hgh accuraces must be acheved. Several software use an 8-terms model derved from the orgnal formulaton for mage dstorton proposed by Brown [35]: x x*( k r 1 y y *( k r 1 k k r r k r ) p ( r 6 1 k r ) p ( r x* ) p x* y * y * ) p x* y * 1 (1) where Δx and Δy are the correctons for a generc mage pont wth coordnates (x, y), x* = x x p and y* = y y p are the mage coordnates referred to the prncpal pont, r = x* y* s the squared radal dstance. The coeffcents k 1, k, k 3 model the radal dstorton. In partcular, the coeffcent k 1 s generally suffcent durng most surveys, but when a hgh accuracy s needed, the coeffcents k and k 3 have to be used as well. Tangental dstorton, that s due to a msalgnment of the camera lenses along the optcal axs, can be modeled wth p 1 and p. The magntude of tangental dstorton s lmted f compared to radal dstorton, especally wth wde-angle lenses. Dgtal cameras should be calbrated perodcally, because several ssues about the stablty of the sensor could arse. A standard camera calbraton procedure can be performed by usng known ponts (and few mages), or wthout any external nformaton and specal coded targets. The former (termed as feld calbraton) needs external 3D nformaton provded through a framework wth several targets, whose 3D coordnates have been prevously measured (e.g., wth a total staton). The latter (self-calbraton) s based on a free-net adjustment [36] n whch ponts wth known 3D coordnates are not necessary. The calbraton framework needs a set of targets, whch are measured n the mages. Furthermore, some addtonal mathematcal constrants and a block composed of several mages wth a

17 Sensors 1, sutable network geometry are needed. For a general revew about calbraton methods the reader s referred to [37]. Equatons 1 allow one to model mage dstorton n order to correct each sngle measured mage pont. In addton, dstorton coeffcents can be used to derve dstorton-free mages, although ths operaton needs a longer elaboraton tme. If calbraton s gven, the correcton of mage coordnates should be always carred out n order to mprove the precson of the fnal result. However, mage dstorton can be consdered locally constant, thus when dfferent epochs are analyzed ts contrbute can be assumed as a systematc error and removed by usng relatve dfferences. Anyway, ths assumpton s vald for small pont dsplacements, therefore a camera should be always calbrated especally wth consumer cameras and wde-angle lenses. 4. 3D Image-based Deformaton Measurements 4.1. Combnng Multple Images for 3D Analyss When 3D measurements are necessary at least two mages for each epoch are needed. Images must be taken at the same tme, thus all cameras must be synchronzed. Several mages can be used to mprove the precson of the object coordnates, however a more expensve nstrumentaton becomes necessary. Fraser [38] proposed the followng formula to estmate the theoretcal precson of a 3D mage-based survey: qs xy XYZ () k where q s an emprcal factor (between.4 and accordng to the number of mages and ther spatal dstrbuton), S s the scale number (camera-object dstance dvded by the focal length), σ xy s the precson of mage coordnates and k s the number of mages. Precson can be enhanced by ncreasng the number of cameras (.e. more observatons for the same 3D pont), though ths mprovement s proportonal to the square root of the number of mages. The mathematcal model for mage orentaton s based on collnearty equatons [35]. An mage can be consdered as a central projecton n space, n whch the relatonshp between an mage pont (x j, y j ) and the correspondng object pont (X j, Y j, Z j ) can be wrtten wth a 7-parameters transformaton: x y j j x y p p c x y j j X j X j R Yj Y j (3) Z j Z j for each mage and pont j. In Equaton 3 R s a rotaton matrx, (X, Y, Z ) are the coordnates of the perspectve centre, c s the prncpal dstance, (x p, y p ) are the locatons of the prncpal pont n the mage and (Δx j, Δy j ) are the correcton terms for mage dstorton. Equaton 3 s non-lnear and a rgorous soluton requres ther lnearzaton (thus the knowledge of good approxmate values) and an teratve approach to estmate the unknowns (.e., camera orentaton parameters and 3D object coordnates). A rgorous bundle soluton, coupled wth the estmaton of the unknown parameters based on the Gauss-Markov model of the Least Squares (LS), provdes an effcent, precse and relable soluton n a functonal and stochastc sense [39]. The unknown parameters are estmated usng proper

18 Sensors 1, coeffcents to weght the observatons wth dfferent precsons. Ther theoretcal precson can be evaluated through the estmated covarance matrx, whle the posteror varance of unt weght (σ ) gves the fnal qualty of the adjustment. The functonal model of the system of Equaton 3 s solved wth the LS soluton: T 1 T x ( A WA) A Wl (4) where A s the desgn matrx, contanng the partal dervatves of Equatons 3 wth respect to the unknowns and evaluated at the approxmated values. W s a weght matrx, x s the unknown vector and l s the observaton vector. The resduals v of the observatons and σ can be estmated as: v Ax l (5) v T Wv (6) r where r s the redundancy (.e., the dfference between the number of observatons and unknowns). The precson of the estmated unknowns can be retreved from the covarance matrx: T 1 ( A WA ) K xx (7) The dagonal elements of the K xx matrx are the varances of each sngle unknown, whle the other elements represent the covarances between the unknowns. To nvert the sem-defnte postve matrx A T WA n Equaton 4 an external datum must establshed. Ths operaton can be performed by fxng seven parameters (three translatons, three rotatons and a scale factor) and can be carred out n several ways. In our mplementaton we mplemented two strateges: 1. the use of an orentaton frame (.e., a support wth known ponts), whch must be placed on the body at the begnnng (or at the end) of the test;. a free-net adjustment based on nner constrants [4], whch requres at least the scale of the project usng an element wth a known length (e.g., a calbrated bar). After processng the mages of the frst epoch wth the developed methodology, f cameras are placed on stable supports exteror orentaton parameters can be consdered constant. Then, the dynamc analyss s based on the measurement of the mage coordnates by trackng the ponts along the mage sequences wth cross-correlaton and LSM. The computaton of object coordnates s performed by usng the fxed orentaton parameters. An mportant pont s related to occlusons. However, durng ths knd of analyss the deformaton emergng s lmted wth respect to the sze of the object. Therefore a good ntal setup of the cameras around the object avods the creaton of occlusons durng the test. 4.. Measurement of Image Coordnates In the case of D dynamc measurements an mage pont must be tracked along the mage sequence. When multple vews must be analyzed, t s necessary to determne the same pont among the mages captured at the same epoch, then the pont can be tracked along the sequence. The determnaton of the

19 Sensors 1, mage correspondences can be carred out by usng targets or wth the texture of the object. Targets can be automatcally detected for all mages, but t s often necessary to (manually) select homologous ponts for the mages of the frst epoch. However, an opportune codng can be added to each target to automate the whole process. Fgure 8 shows the typcal case of a target-based survey. Here, two synchronzed Nkon D7 cameras wth mm Sgma lenses were employed. Several targets placed n partcular postons were tracked after ther sem-automatc matchng for the frst epoch. The method used durng the trackng phase s based on cross-correlaton and LSM. More detals about ths test are descrbed n the next secton. Fgure 8. A target-based survey wth two synchronzed cameras. However, durng some analyss targets cannot be fxed, thus we mplemented a soluton based on the texture of the object and projectve geometry. Ths new method s based on detectors and descrptors capable of determnng te ponts among the mages. In our mplementaton we use two operators able to extract and match these mage correspondences: SIFT (Scale Invarant Feature Transform) [41] and SURF (Speeded Up Robust Features) [4]. They are nvarant to changes n rotaton (around the optcal axs of the camera) and scale, and are robust to affne deformatons and changes n llumnaton. The method s based on the extracton of the features durng the frst epoch (usng the detector) and the dentfcaton of the correspondences among multple mages (usng the descrptor). The man advantage gven by these operators s the possblty to match te ponts by analyzng the descrptor, whch s a vector that contans nformaton about the local characterstcs of the extracted features: te ponts are matched by usng the 18-element vector assocated to each feature wthout any prelmnary nformaton. The L norm of the dfferences (Eucldean dstance) s employed. More detals about ths procedure can be found n [43]. The choce of the matcher (SIFT or SURF) depends on the characterstcs of the object. Generally SIFT fnds more ponts than SURF, but t s computatonally more expensve. On the other hand, f an object has a good texture SURF provdes a good number of mage ponts. At the end of the matchng phase several outlers can be found, especally n the case of repettve patterns. We remove all these wrong correspondences wth the robust estmaton of the fundamental matrx F [44], whch s a 3 3 matrx of rank that encapsulates the geometry of an uncalbrated stereo par. If the scene s planar, t s more convenent to estmate the essental matrx E [45], because

20 Sensors 1, a D scene s a crtcal case for the F-matrx. We use the Least Medan of Squares (LMedS) method [46] to estmate F (or E) wth 7 mage ponts [47]. Gven a set of mage correspondences x = (x, y, 1) T (x, y, 1) T = x between two mages pcturng the same object from dfferent camera statons, the condton x T F x = must be satsfed. Ths condton can be easly demonstrated by consderng that the F-matrx represents a connecton between a pont n the frst mage and the eppolar lne n the second one: l = Fx,n whch ponts and lnes are expressed by homogeneous vectors. Indeed, the dot product between a pont n the second mage and the eppolar lnes of the correspondng pont n the frst one must be zero (x T l = ) because the pont les on the lne. In ths work robust technques play a fundamental role. They allow an effcent detecton of all msmatches and are mandatory n the case of fully automated technques. Normally, these procedures are based on the selecton of mnmal dataset and the followng estmaton of several F-matrces wth the presented methodology. However, we are not drectly nterested n the value of the computed F, but we want to detect wrong correspondences. In ths case the F-matrx s extremely useful, because all msmatches gven by the comparson between the detectors can be removed by analyzng ther mage coordnates, wthout any consderaton about the 3D geometry of the object. A method based on seven correspondng ponts represent the mnmal case for the estmaton of F. In fact, the F-matrx has a scale ambguty that coupled wth the sngular constrant det(f) = reduces the number of ndependent elements to seven. A soluton can be estmated usng the followng system: f f x ' x x ' y x ' y ' x y ' y y ' x y 1... where f 1,, f 9 are the nne elements of F. The soluton s a D space of the form αf 1 + (1 α)f =, whch coupled wth the determnant constrant gves det αf 1 + (1 α)f =. Ths last equaton s a cubc polynomal equaton n α that can be easly solved. The LMedS technque evaluates each soluton wth the medan symmetrc eppolar dstance to the data [48]. The soluton whch mnmzes the medan s chosen. To estmate F, subsets of seven correspondences are randomly extracted from the orgnal dataset. The mnmum number of trals m S to obtan an error-free subsample wth a gven probably P and an expected fracton of outlers ε s: m s 1 9 (8) log(1 P) p (9) log(1 (1 ) ) (p s the number of correspondences,.e., the parameters to estmate 7 n ths case). For any subsample k of mage coordnates a fundamental matrx s estmated, thus the medan of the squared resduals s calculated by usng the whole dataset of mage coordnates and the dstances between ther eppolar lnes: μ k = medan [ d (x, F k x ) + d (x, F k T x ) ] (3) The method does not need a prelmnary threshold to classfy a pont as nler (or outler). A robust estmaton of the standard devaton can be derved from the data wth the relaton:

21 Sensors 1, c1 n p k (31) where μ k s the mnmal medan and c = Then, a weght w based on σ s determned for each correspondence and s used to detect outlers (w = ): 1 r w (.5 ) otherwse (3) where r = [d (x, F x ) + d (x, F T x )] 1/. After all these steps outlers can be removed and a fnal LSM refnng s carred out to mprove the precson of mage coordnates. Fgure 9 shows the applcaton of the procedure. Two synchronzed Nkon D8s wth mm lenses were placed n front of a mcrometrc sledge. Two peces of rocks, whch smulate a real constructon element, were leant on the plates of the sledge and several shfts were gven n order to compare the mage measurements wth the mechancal ones. The results of the matchng wth the descrptors are shown n Fgure 9a, n whch several outlers can be seen. After the robust estmaton of the fundamental matrx, all these msmatches were correctly removed (Fgure 9b) and can be refned va LSM by fxng each pont n the frst mage and searchng the homologous n the second one. Lastly, a free-net bundle adjustment was carred out and the scale of the reconstructon was fxed wth a small calbrated bar. Fnally, a trackng process based on cross-correlaton and LSM s carred out along the mage sequence and 3D ponts are estmated wth a smple ntersecton by usng the computed orentaton parameters. Fgure 9. Matchng results durng a markerless 3D survey wth two cameras: (a) pont matched wth the descrptors and (b) ponts after the robust estmaton of the fundamental matrx.