Detection of Outliers in the Adjustment of Accurate Geodetic Measurements

Transcription

1 105 Detecton of Outlers n the Adjustment of Accurate Geodetc Measurements asák, P. and Štroner, M. Department of Specal Geodesy, Faculty of Cvl Engneerng, CU n Prague, hákurova 7, Prague, Czech Republc, Web ste: el.: , E-mal: pavel.trasak@fsv.cvut.cz, martn.stroner@fsv.cutv.cz Abstract he paper deals wth the possbltes of automatc detecton of outlers n processng (adjustment) of accurate terrestral geodetc measurements. hese measurements are obtaned n the determnaton of hghly accurate terrestral geodetc networks, n whch have been repeatedly measured lengths, horzontal drectons and zenth angles and n whch s known a large number of redundant measurements. For automatc detecton of outlers of measurements authors use the robust statstcal methods, namely robust estmates based on the maxmum lkelhood method, so-called M-estmates. hs paper descrbes orgnally desgned experment based on a model of geodetc network wth deal desgned terrestral geodetc measurements. Measurements of ths deal model (measurements fully correspond wth the normal probablty dstrbuton) are frst processed by least square method. In the next phase of the experment the deal measurements are gradually dsturbed by contamnaton of outlers (outlers of lengths, horzontal drectons and zenth angles) and for ther followng processng (adjustment by least square method) are appled selected ndvdual robust M-estmates. hese selected M-estmates are assessed accordng to ther ablty to detect outlers,.e. accordng to ther usablty to automatc detecton of outlers n adjustment of terrestral geodetc measurements by least square method. he result of the present paper s to propose a procedure for processng (adjustment) of geodetc measurements of hgh accurate engneerng-geodetc networks when are exposed by outlers. Key words: outlers, adjustment of geodetc measurements, geodetc networks 1 INRODUCION he prncpal objectve of the paper s the descrpton and testng of a detecton technque of outlers n a set of values of measured geodetc varables. he proposed technque s appled for the adjustment of hghly accurate engneerng geodetc networks where large numbers of repeatedly measured values of geodetc varables are presumed,.e. large numbers of redundant measurements are consdered. S 3 Poster Presentaton INGEO th Internatonal Conference on Engneerng Surveyng Brjun, Croata, September 22-24, 2011

2 106 INGEO EXPERIMEN PROPOSAL In order to detect outlers n a set of hghly accurate engneerng geodetc measurements the authors have appled a technque whose effcency s assessed usng the experment descrbed below. 2.1 DEECION ECHNIQUE OF OULIERS he proposal s based on the classc adjustment of geodetc measurements usng the Method of Least Squares beng subdvded nto two basc steps. 1. Detecton of the robust estmaton of measured varables. In the frst phase of the geodetc data processng, the Robust M-Estmator (Huber, 1981) s used to make the estmaton of measured varables. he robust M-estmator applcaton s based on the modfcaton of a commonly used technque for the adjustment of geodetc measurements usng the Method of Least Squares (MLS). Unlke MLS the appled Robust M-Estmator s much less senstve to the fulflment of the assumpton of the normalty of processed data beng, therefore, to a certan extent, resstant to the nfluence of outlers. he resultng robust estmaton s not sgnfcantly affected by outlers. 2. Elmnaton of outlers of measured varables. Outlers are removed from the data set usng the selected rejecton rule. he detecton of outlyng values s performed on the bass of the assessment of ther dstance from the computed robust estmaton. 2.2 DESCRIPION OF HE PROPOSED EXPERIMEN he usablty of the proposed technque for the detecton of outlers n a set of geodetc measurements s assessed on the bass of the evaluaton of expermental data. he proposed experment may be splt nto the basc phases below. 1. Creaton of a model of geodetc measurements wth accurately defned parameters. 2. Artfcal ntroducton of outlers nto the model set of geodetc measurements. 3. Detecton of outlers usng the proposed technque. 4. Assessment of the effcency of the selected technque by comparng the numbers of ntroduced and successvely detected outlers. 3 MODEL OF GEODEIC MEASUREMEN he descrbed experment s based on the applcaton of proposed methods and technques onto accurately desgned artfcally modelled geodetc data. he reason for the modellng of geodetc data s the necessty of perfect knowledge of a pror accuraces of generated geodetc measurements whch are necessary for the objectve assessment of the results of the outputs acheved. 3.1 GEODEIC NEWORK he model geodetc network s desgned as a standard spatal medum-sze surveyng network shaped lke an rregular pentagon wth a sx standponts. Whle creatng the model,

3 asák, P. et al.: Detecton of Outlers n the Adjustment of 107 only the rough shape of the network was set and the detaled spatal poston of ndvdual ponts was randomly selected. he maxmum horzontal dstance between the ponts n the network s m, and the maxmum heght dfference equals 6.728m. A detaled dstrbuton of ndvdual ponts n the network (together wth ther spatal rectangular coordnates) s shown n Fgure 1. Fgure 1 Layout of the geodetc network 3.2 GEODEIC MEASUREMEN he model set of geodetc measurements smulates the classc output of a hghly accurate terrestral geodetc measurement obtaned wth the use of the total staton and a set of reflectng prsms. he set contans a large number of repeatedly dentfed horzontal drectons, zenth angles and slope dstances measured between ponts of the network. All values of the above-measured varables wthn the entre set have been arranged nto ndvdual sets of horzontal drectons, zenth angles z and slope dstances d measured n one set (.e. n two faces of the telescope). he set of geodetc measurements was modelled n the followng steps: 1. Determnaton of the number of values of measured varables n a model network. 6 sets (one set n each pont) are consdered wthn the model network, whch represents the total of 85 values of geodetc varables measured n one set (30 horzontal drectons, 30 zenth angles, 25 slope dstances), producng the total of 304% of redundant measurements. 2. Modellng of the set of measurements for each face of the telescope. Usng the pseudorandom number generator (asák et al., 2010) a set (random sample) of values of a standard normal dstrbuton N(0,1) was generated, whch was subsequently transformed nto a set wth a non-standard normal dstrbuton (X 0, 1 2 ) where X 0 s the true value of the measured varable ( 0, z 0, d 0 computed from modelled coordnates of the ponts n the network) and 1 the standard devaton of the measured varable n one face of the telescope ( 1, 1z, 1d measured n one face of the telescope). he standard devatons of measured varables correspond to the accuracy of total statons n a hgher accuracy class, such as (rmble, 2011) (numercal values of standard devatons are lsted n ab. 1).

4 108 INGEO Processng of measurements wthn one set (2 faces of the telescope). he fnal phase of the model data set creaton ncluded the computaton of the values of measured varables determned n two faces of the telescope: a) Averagng of 2 respectve values of horzontal drectons dfferng by 200gon. b) Correcton of the zenth angle elmnatng the ndex error. c) Averagng of 2 respectve values of slope dstances. Correspondng standard devatons were assgned to the resultng values of measured varables (see ab. 1). he obtaned values were used for subsequent processng,.e. for the adjustment of geodetc network measurements. ab. 1 Standard devatons of measured varables horzontal drecton 1 face of telescope mgon 1 set mgon (2 faces of telescope) 2 zenth angle z 1z z z mgon mgon slope dstance d 1 d 2. 0mm d d mm 4 GEODEIC NEWORK ADJUSMEN USING MLS he method of the free spatal geodetc network adjustment was used for the adjustment of the model of geodetc measurements. In ths case, the geodetc network s not frmly bound to any pont of the network. he adjustment (estmaton) of measured varables results n the adjustment (estmaton) of the coordnates of all ponts n the network. As the geodetc network s not frmly bound to any pont, ts general localzaton n space must be ensured. For ths reason, the soluton of the free network problem apples the method of the adjustment of observaton equatons wth condton equatons Böhm (1990) whch wll ensure ths localzaton. Intermedate varables consdered n the soluton of the model network were drectly measured varables whch, therefore, allow a drect expresson of the relaton l f (x), (1) where l s the vector of adjusted measured varables (the vector of estmatons of measured varables l) and x s the vector of adjusted unknown varables (the vector of estmatons of unknown varables x). Furthermore, we may state that l v f ( x 0 dx), (2) where v s the vector of resduals of measured varables, x 0 s the vector of approxmate values of unknown varables and dx s the vector of dfferences to approxmate values of unknown varables. Followng the lnearzaton of the above expresson t holds true that v Adx l, (3) l f x ) l, (4) ( 0 where l s the vector of reduced observatons and A s the matrx of lnearzed expressons among measured and unknown varables (the matrx of partal dervatons of functons of measured varables accordng to ndvdual unknowns). By expressng the necessary condton of the network localzaton for unknown varables as ( x) O, (5)

5 asák, P. et al.: Detecton of Outlers n the Adjustment of 109 and ts lnearzaton Bdx b O, (6) b ( x 0 ), (7) and, further, by ntroducng the condton of the Method of Least Squares v Pv mn, (8) the Method of Lagrange Multplers (Böhm, 1990) s used for the defnton of a system of normal equatons and the formula for the calculaton of the sought vector of dfferences to approxmate values of unknown varables dx A PA k B B O 1 A Pl, (9) b where k s the auxlary vector of Lagrange multplers (correlate), B s the matrx of lnearzed condtons of unknown varables (the matrx of partal dervatons of ndvdual condtons accordng to ndvdual unknowns), b stands for the vector of condtons of unknown varables expressed by means of ther approxmate values x 0, O s the zero matrx and P the weght coeffcent matrx of measured varables l. In a model geodetc network contanng measured horzontal drectons, zenth angles z and slope dstances d, the vector of measured varables l(m,1) takes the form l ( 1,, m, z,, zm, d,, dm ) , (10) 3 where m 1, m 2, m 3 are the numbers of measured horzontal drectons, zenth angles and slope dstances and m = m 1 + m 2 + m 3 s the total number of measured varables. he selected unknown varables are spatal rectangular coordnates of all ponts of the network (X, Y, Z) and also orentaton shfts of ndvdual sets of horzontal drectons at ndvdual standponts o. Hence, the vector of unknown varables x(n,1) takes the form x ( X 1, Y1, Z1,, X n, Yn, Z n, o,, on ) , (11) 3 where n 1 s the number of ponts n the network, n 2 s the number of orentaton shfts and n = n 1 + n 2 s the total number of unknown varables. o be able to plot the vector of measured varables l and the vector of unknown varables x, ndvdual elements of the matrx of lnearzed expressons of measured and unknown varables wth a total magntude A(m,n) are defned as partal dervatons f ( x) A, j, (12) x xx0 where f(x) s the functon of unknown varables x expressng the measured varable l. In order to localze the geodetc network n space, Helmert s transformaton condton (Koch, 1999) was selected for the model network at all ponts of the network,.e. the squares of coordnate dfferences of approxmate and adjusted ponts of the network were mnmzed dx dx mn. (13) Provded all three types of varables (horzontal drectons, zenth angles and slope dstances) are measured n a spatal network, t holds true for ths condton that,1 o b 4, (14) B, n B (4,3) Bn (4,3) O(4, ) 4 1 n 1 2, (15) where the submatrx

6 110 INGEO 2011 Y0 X B (16) he weght coeffcent matrx of measured varables s composed as the dagonal matrx P dag( p 1,, p ), (17) m where the weght coeffcent of ndvdual measured varables s expressed by the relaton p, (18) where 0 s the a pror unt standard devaton and stands for the standard devaton of the measured varable. In the case of the model network, the a pror unt standard devaton selected was 0 = 1, whle the standard devatons of measured varables were defned by the modellng parameters (see ab. 1). 5 ROBUS SAISICAL MEHODS As already stated, ths paper s focused on the assessment of potental detecton of outlers of measured varables by means of robust statstcal estmatons, usng one group of robust estmatons n partcular, so-called M-estmators. As the paper solely deals wth the assessment of potental applcatons of such estmatons, deeper theoretcal nsght s not ncluded n t, but only a bref descrpton of the ntroducton of the robust M-estmator nto the adjustment of a free geodetc network usng MLS. A detaled descrpton of robust estmatons appled here s n (asák et al., 2011). he prncple of the robust M-estmator applcaton s based on a gradual teratve adjustment of geodetc measurements usng the Method of Least Squares under the condton of a gradual change n the weght coeffcent of ndvdual measurements n relaton to the development of the magntude of ther normed resduals determned by the adjustment. In ths way, outlers of measured varables are gradually elmnated. In each teraton step, therefore, the robust weght coeffcent of each measured varable s computed w f (vˆ), (19) and the matrx of robust weght coeffcents s defned W dag( w, w2, w ). (20) 1 m In the zero teraton step, the robust weght coeffcents of all measurements are set as w (0) = 1 (W = E(m,m)), the weght coeffcent s not ntroduced and the estmaton of measured varables s computed usng the non-robust Method of Least Squares ( 1 0) (0) (0) dx A W PA B A W Pl A PA k B O b B Normed resduals of measured varables are computed v Adx p 0 B O 1 A Pl. (21) b l, (22) vˆ v, (23) and the computatonal model s adjusted

7 asák, P. et al.: Detecton of Outlers n the Adjustment of 111 x x dx. (24) 0 In the next step, the computatonal model s made more accurate x ( j1) 0 x A, B, l. (25) New robust weght coeffcents are computed usng normed resduals of measured varables w ( j1) f ( ) vˆ j, (26) ( j1) ( j1) ( j1) W dag( w,, w ). (27) 1 m and a new estmaton of measured varables s defned 1 ( j1) ( j1) ( j1) dx A W PA B A W Pl. (28) k B O b he convergence of ths teratve computaton s proved n (Huber, 1981). 5.1 USED M-ESIMAORS he totals of 12 dfferent formulae for the robust weght coeffcent computaton were used n the soluton of the descrbed experment (19). he lst of M-estmators used s dsplayed n ab. 2. he descrpton of all used estmatons together wth the formulae for ther weght functons are n (asák et al., 2011). ab. 2 Lst of M-estmators used Cauchy dstrbuton 1 Huber estmator 5 estmator Modfed Huber ukey s bweght 2 6 estmator estmator Geman McClure 3 Hampel estmator 7 estmator 4 alwar estmator 8 Andrews estmator 12 9 Welsch estmator 10 Far estmator 11 L1 standard Hybrd L1/L2 standard 6 DEECION OF OULIERS IN MEASUREMENS he proposed prncple of the detecton of outlers n a set of measured varables s based on the assessment of the magntude of resduals of ndvdual measurements. hese resduals are obtaned from the results of the adjustment of geodetc measurements applyng the robust M-estmator. By usng the robust M-estmator n the adjustment of geodetc networks the effect of measured outlyng values s reduced thus obtanng an estmaton of measured varables ndependent of outlers. Assumng that the obtaned estmaton of the measured varable X approaches ts true value X 0 and that the set of values (random sample) of the measured varable comes from the normal probablty dstrbuton N(X 0, 1 2 ), the technque below may be used for the detecton of outlyng values. Followng suffcent stablzaton of the teratve computaton of the adjustment of geodetc measurements, by fulfllng the condton

8 112 INGEO 2011 ( ) ( j1) max( w j w ), (29) where s the maxmum tolerated change n the robust weght (ths lmt was set as = n the experment), the resduals of measured values v are computed n accordance wth (3) and ther lmt values,.e. the lmt resduals of measured values, are further detected as v, (30) M u p v where u p s the standard normal probablty dstrbuton value set for the sgnfcance level (u p = 1.96 for = 0.05 was selected for the experment) and v s the standard devaton of the resdual of the measured value l, whch s defned by the formula, (31) v 0 Q v, v where Q v,v s the dagonal element of the covarance matrx of resduals of measured varables Q 1 1 v, v P A( A W PA) A. (32) A comparson s subsequently made and provded t holds true that v v M, (33) the value of the measured varable v s declared as outlyng and s elmnated from the set of measurements. After the elmnaton of all outlers the reduced set of measured varables should fulfl the condton of normalty of measured data, and the non-robust MLS may then be used for the computaton of the best objectve estmaton of measured varables. 7 ESING HE MEHOD OF DEECION OF OULIERS IN MEASUREMENS he testng of the proposed technque of the detecton of outlers of measured varables s based on repettve processng (adjustments) of an artfcally generated model set of geodetc measurements wth a gradual ntroducton of dfferent quanttes of dfferently outlyng values of measured varable (, z, d). he result of the testng s the determnaton of the effcency of the detecton of outlers by all methods presented here (see ab. 2) n relaton to specfc confguratons of the model set (.e. the number and magntude of outlers present n the set). 7.1 INRODUCION OF OULIERS Outlyng values were ntroduced nto the model set n a completely random way regardless of the type of measurement (, z, d). o make the nterpretaton of the results smpler, the set was always contamnated wth numbers of outlers of the same magntude. he magntude of outlers was defned by the coeffcent h of the standard devaton of the measured varable j l j, l h j, j, z, d, (34) he used coeffcents of standard devatons h are lsted n ab. 3. o smplfy the nterpretaton of the results, the outlers are subdvded nto 3 groups by magntude. he quantty of outlers s expressed relatvely n relaton to the total number of measurements n the model set ( ab. 4). he used quanttes are splt nto two nterpretaton groups.

9 asák, P. et al.: Detecton of Outlers n the Adjustment of 113 ab. 3 Magntude of ntroduced outlers coeffcents of standard devatons of measured varables Outlyng values More extreme outlyng Gross errors of medum sze values 2 2,5 3 3, ab. 4 Quanttes of ntroduced outlers Smaller numbers of outlers [%] Larger numbers of outlers [%] RESULS OF HE EXPERIMEN As already mentoned above, the respectve experment results n the determnaton of the effcency of the detecton of outlers. hs effcency s descrbed by two values: the number of correctly detected ntroduced outlers a (the number of values of measured varables whch are correctly consdered as outlyng by the method; expressed as the percentage of the number of outlers ntroduced nto the set) and the number of wrongly detected non-ntroduced outlers b (the number of values of measured varables whch are ncorrectly consdered as outlyng by the method; expressed as the percentage of the total number of values n the set). For the reason of reducng the large number of outputs, the acheved effcency rates of ndvdual methods were arranged nto nterpretaton domans (see ab. 5). he acheved effcency rate was averaged wthn ndvdual domans and each doman s, therefore, represented by only two values: the average number of correctly and wrongly detected outlers n the set, whch are dsplayed n ab. 6 (ndvdual types of M-estmators are numbered accordng to ab. 2). Effcency rates (a, b) are presented only for Huber s M-estmator technque. For the other technques, dfferences ( a, b ) related to Huber s M- estmator are descrbed. Fgure 2 further hypsometrcally dsplays the effcency of Huber s M-estmator (wthout the subdvson of the results nto nterpretaton groups). o ncrease the relablty of the acheved results the whole experment was repeated 10 tmes and the presented values represent the average of all repettons. ab. 5 Interpretaton domans of the magntude and quantty of ntroduced outlers Name of nterpretaton doman Magntude Quantty [h] [%] A Doman of a smaller number of medum-sze outlers B Doman of a larger number of medum-sze outlers C Doman of a smaller number of more extreme outlers D Doman of a larger number of more extreme outlers E Doman of a smaller number of gross errors F Doman of a larger number of gross errors

10 114 INGEO 2011 ab. 6 Numbers of correctly ([%], a [%]) and wrongly ([%], b [%]) detected outlers a b a b a b a b a b a b A B C D E F a b a b a b a b a b a b A B C D E F

11 asák, P. et al.: Detecton of Outlers n the Adjustment of 115 Fgure 2 Huber s M-estmator - numbers of correctly and wrongly detected outlers 7.3 ASSESSMEN OF ACHIEVED RESULS he presented results mply that the effcency of ndvdual robust M-estmators s comparable. he values n felds E and F are consdered as completely tentatve and not much weght should be attrbuted to them n the total assessment. In these cases, the set was contamnated wth gross errors whch can easly be removed n advance durng the processng of data from accurate geodetc networks. he best effcency results have been acheved by usng Huber s M-estmator (the drawbacks n the effcency of the other estmatons of the robust estmaton type are lsted n ab. 6). Huber s M-estmator technque s paradoxcally the oldest estmaton technque, and the other robust estmaton technques were desgned later wth the am of mprovng ths estmaton technque. he effcency of the correct dentfcaton of outlers n a set of geodetc measurements (a) s drectly proportonal to the magntude of the error of outlyng measurements (expressed by the coeffcent of the standard devaton h n the experment). ogether wth the growng magntude of the error of outlers the growng amount of detecton of wrongly dentfed nonoutlers s negatvely manfested (b). As compared to the effect of the growng magntude of the error of outlers, the effect of the growng number of these errors s much less sgnfcant. A detaled effcency pattern (correct and wrong detecton of ntroduced outlers) for Huber s M-estmator technque s dsplayed n Fgure 2. 8 CONCLUSION he experment resulted n the determnaton of the usablty of robust M-estmator technques for the detecton of outlers of geodetc varables measured n hghly accurate engneerng geodetc networks. he applcaton of robust estmatons for the detecton of outlers s a hghly effcent method whose feasblty, however, s condtonal upon numerous factors and nput condtons. o acheve adequate results, a suffcent number of redundant measurements must be provded (correspondng to the number of redundant measurements appled n a commonly measured hghly accurate engneerng geodetc network), and the absence of extreme gross measurement errors must be ensured (whch may be elmnated by checkng the data before the adjustment of the measurement). Unlke the commonly used technques of the detecton of outlers whose prncple s based on repettve rejectons of ndvdual measurements wth large resduals aganst the MLS estmaton (thus resemblng the tral and error method), the applcaton of robust estmatons s fully automatc and the decson on the rejecton of all measured outlers may be made all at once and not gradually by assessng ndvdual measurements. he weak pont of the descrbed technque s ts dependence on the classc adjustment technque of geodetc measurements (MLS adjustment), whch n some cases, wth the presence of extreme gross errors, tends to be computatonally unstable, and once the computaton of the estmaton of measured varables by means of MLS has faled, the whole technque of the detecton of measured outlers fals, too. Acknowledgement

12 116 INGEO 2011 he artcle was wrtten wth support from the nternal grant SGS10/153/OHK1/2/11 Complex software processng of measurements n engneerng surveyng REFERENCES HUBER, P. J.: Robust Statstcs. New York, John Wley and Sons, ASÁK, P. - ŠRONER, M.: estng of Generators of Pseudorandom Numbers from the Normal Dstrbuton for Use n Smulaton of Geodetc Measurements - Part 1 (n Czech). Stavební obzor. 2010, vol. 19, no. 2, pp ISSN RIMBLE: Corporate lterature for rmble S6 nstrument (n Czech). t_d=4&category_d=15&opton=com_vrtuemart&itemd= BÖHM, J. RADOUCH, V. HAMPACHER, M.: heory of Errors and Adjustment Calculus (n Czech). 2nd ed., Praha, Geodetcký a kartografcký podnk ISBN KOCH, K. R.: Parameter Estmaton and Hypothess estng n Lnear Models. Berln Hedelberg New York, Sprnger Verlag, 1999, ISBN ASÁK, P. - ŠRONER, M.: Robust Adjustment Methods (n Czech). Geodetcký a kartografcký obzor. 2011, vol. 57, no. 7, ISSN