Define - starting approximation for the parameters (p) - observational data (o) - solution criterion (e.g. number of iterations)

Transcription

1 Global Iterative Solution Distributed proessing of the attitude updating L. Lindegren (21 May 2001) SAG LL 37 Abstrat. The attitude updating algorithm given in GAIA LL 24 (v. 2) is modified to allow distributed proessing in whih eah proessor takes are of a subset of the soures aording to the sky partitioning. A detailed (but very simplified) numerial example is provided. The same priniple an be used to distribute the alibration and global proessing. 1 Introdution In GAIA LL 34 (v. 2), detailed algorithms in the form of fortran subroutines were provided for a number of proesses in the Global Iterative Solution (GIS) to be implemented in the on-going GAIA Data Aess and Analysis Study (GDAAS). It is realled that the GIS basially onsists of the following proesses to be exeuted ylially until onvergene: ffl soure proessing one soure (astronomial objet) at a time; ffl attitude proessing one attitude interval (some 12 hr) at a time; ffl alibration one CCD at a time; ffl global proessing needs to onsider the whole mission. It is envisaged that the GIS will be exeuted in a distributed enviroment with several proesses running in parallel. In order to minimize large-sale data shuffling it seems natural to assoiate a speifi area on the sky (following the HTM division) with eah loal proessor. This makes it very straightforward to distribute the soure proessing, sine that only requires aess to a very small area of the sky at a time. It is less obvious if and how the other proesses an be distributed without a dramati inrease in overhead. In this doument the attitude proessing is desribed in some detail, inluding a `worked example' using the subroutines supplied with GAIA LL 34 (v. 2). All the proesses mentioned above are instanes of non-linear least-squares estimation, implemented (as proposed for the GIS) by the lassial method using normal equations. The general idea for distributed proessing using normal equations is disussed in Set. 2 and then applied to the attitude proessing as an example involving fairly many unknowns (Set. 3). The same method an be used for the alibration and global proessing; here the normal equations only involve one or a few unknowns, so the situation is muh simpler than for the attitude proessing. 1

2 2 Least-squares by distributed proessing All the proesses desribed above solve a non-linear least-squares problem by the use of normal equations. The non-linearity arises primarily from the need to take are of outliers by means of downweighting. Let p be the parameter vetor to be estimated and m = dim(p) the number of unknowns (parameters). For given observations o (with dim(o) fl m) the quantity to minimize is Q(p) = X k ρ ok f(t k jp) ff k where f is the observation model, ρ the loss funtion, ff the standard error assigned to eah observation, and k the observation index. For ordinary least-squares the loss funtion is simply ρ(z) = z 2, but in order to aommodate outliers this should be modified by a fator whih redues the weight of observations with large jzj. Tentatively, the Huber metri has been proposed, (1) ρ(z) =z 2 ( 1 if jzj» (2jzj )z 2 otherwise (2) (note the typo in Eq. 2 of GAIA LL 34.2). Problem (1) an be solved iteratively from an initial approximation p by means of the equations: A ij = X k w k ff 2 j ; i; j =1:::m; (3) X w k b i = ff 2 (o k f k ) ; i =1:::m; (4) k i x = A 1 b ; (5) p ψ p + x : (6) This is iterated until jxj is negligible. Here w k is the weight redution fator alulated for eah observation based on the size of the normalized residual z k (o k f k )=ff k. 1 The linear system of equations Ax = b is alled the normal equations (A 1 is also an estimate of the ovariane of the resulting p). Sine A is symmetri of size m m and b is of size m 1 the total number of reals to be stored for the normal equations is (at most) m(m + 3)=2. (For the attitude updating A is band-diagonal whih means that the atual storage is muh smaller and inreases only linearly with m.) Figure 1 is a shemati illustration of the steps neessary to perform the non-linear leastsquares estimation aording to this method. In addition to the normal equations, some additional statistis are usually alulated in step 2, here exemplified by the total number of observations n and the mean weight redution fator w. 1 Atually, thedownweighting is not done quite as intended in the attahed subroutines, i.e. what is done does not orrespond to the Huber metri. This is of little onsequene at the present stage, but eventually the treatment ofoutliers should be onsidered more arefully, whih probably will lead to a modifiation at least of the funtion whuber. 2

3 Step 0: Initialize leastsquares problem Define - starting approximation for the parameters (p) - observational data (o) - solution riterion (e.g. number of iterations) Step 1: Initialize normal equations A := 0 b := 0 n := 0 wsum := 0 (et.) Step 2: Aumulate normal equations A := A + (new data) b := b + (new data) n := n + 1 wsum := wsum + (new data) (et.) loop over observations Iterate a few times, or until x is negligible Step 3: Solve n.e. and update parameters x := A -1 b p := p + x w := wsum/n (et.) After onvergene, output p, n, w and any other relevant statisti Figure 1. Flow diagram for a typial non-linear least-squares problem suh as the attitude updating. An important onsideration is that the aumulated arrays (A, b and various statistis) are relatively small and of onstant sizeforagiven problem. For instane, A and b together need at most m(m +3)=2 reals, if m is the dimension of p. For the attitude updating A is a sparse (band-diagonal) matrix whih an be stored muh more ompatly. The algorithm attupd takes full advantage of this property. The onveniene of the normal-equations approah derives partly from the fats (a) that their size is typially muh less than what is needed to store the observations thus it is more onvenient to send normal equations around than observations, and (b) that the sums in Eq. (2) an be aumulated independently for disjoint sets of observations (k), and then o-added before solving the normals and updating the parameter estimate. If the observational data are stored loally on N different nodes (eah with its own proessor or proessors), then an obvious modifiation of this sheme is as shown in Figure 2. 3 A worked example A numerial example demonstrates how the attitude updating subroutine attupd from GAIA LL 34.2 was modified to allow distributed proessing. 3

4 Step 0: Initialize leastsquares problem Define - starting approximation for the parameters (p) - observational data (o) - solution riterion (e.g. number of iterations) Distribute these to N proessors Pro 1 Pro 2 Pro N Step 1: Initialize loal normal equations A := 0 b := 0 n := 0 (et.) A := 0 b := 0 n := 0 (et.)... A := 0 b := 0 n := 0 (et.) Step 2: Aumulate loal normal equations A := A + (new data) b := b + (new data) n := n + 1 (et.) A := A + (new data) b := b + (new data) n := n + 1 (et.)... A := A + (new data) b := b + (new data) n := n + 1 (et.) loop over observations (N disjoint sets) Collet normal equations from loal proessors A := A 1 + A A N b := b 1 + b b N n := n 1 + n n N (et) Step 3: Solve n.e. and update parameters x := A -1 b p := p + x (et.) Iterate a few times, or until x is negligible After onvergene, output p, n, w and any other relevant statisti Figure 2. Flow diagram for the same non-linear least-squares problem as in Fig. 1, but with the initialization and aumulation of normal equations distributed on N loal proessors. Eah proessor takes are of a disjoint set of observations, e.g. orresponding to a ertain area on the sky. The data flow between proessors is minimal beause the dimensions of p, A, b, et are small ompared with the observations, whih are already stored loally. 3.1 The soure atalogue In order to avoid having to generate and store a huge soure atalogue, a seeded soure approah is used. This means that the soure identifiation number (an integer in the range from 1 to ' )isusedtoseedarandomnumber generator whih then provides the soure parameters aording to a given distribution model. The model used here is extremely simple: the position (ff; ffi) is drawn from a uniform distribution over the sky while the remaining four astrometri parameters (parallax, proper motion and radial veloity) are set to zero. The subroutine getstar returns the six astrometri parameters for a given soure number. Note that getstar uses a separate, very primitive random 4

5 number generator alled ran Observational data for input Simple geometri observations in the form of angular oordinates ( ; ) were generated for a time interval of one spin period (3 hr). The text file obs.prn is a list of 4253 simulated observations for the time interval 0:01 <t<0:135 days. The nominal sanning law was used as `true' attitude. Both Astro fields were used, eah of size 0:66 0:66 deg, and a basi angle of 106 deg. The path of eah soure aross the several CCDs was not simulated; instead seleted observations were taken at random time intervals. The total number of observations per time interval is muh smaller that for the real mission, but suffiient to fix the attitude for the hosen knot frequeny. The observational standard errors were taken to be onstant, viz. 0.1 mas along-san and 2 mas aross-san per observation. For eah observation, obs.prn gives the time (days), the soure number, the field used (1 or 2), the observed along-san oordinate (rad), the standard error ff (rad), the observed aross-san oordinate (rad), and the standard error ff (rad). The first five lines of obs.prn are shown by the following table: t ID Field ff ff Starting approximation The starting approximation for the attitude spline is given by the file q initial.prn, whih ontains one line per knot. The olumns are: time (days), S 1n, S 2n, S 3n, and S 4n, where S mn are the basi spline oeffiients defined in Set. 5.3 of GAIA LL Note that the first olumn ontains the adopted knot sequene, inluding three points on either side of the time interval over whih the attitude is defined. t S 1n S 2n S 3n S 4n Fitting the attitude spline (non-distributed proessing) The following main program implements the method in Figure 1 by suessive alls to the subroutine attupd (supplied with GAIA LL 34.2). 2 This routine should be kept separate from the more sophistiated routines used to generate observation noise and similar things where a good degree of randomness is important. never be reinitialized (given a new seed) within a simulation. These latter routines should 5

6 PROGRAM fit_attitude Fit an attitude spline to the observations in file_obs. An initial approximation of the attitude (as well as the knot sequene) is defined in file_ini. The final attitude estimate (after niter least-squares iterations) is given in file_out. N.B.: the parameter N must hosen suh that N+4 does not exeed the number of rows in file_ini; moreover, file_obs must ontain suffiient observations overing the time interval tau(2:n-1), where tau(-1:n+2) is the knot sequene defined by the first olumn of file_ini. L Lindegren (Lund Observatory) 19 May 2001 IMPLICIT REAL*8 (a-h,o-z) INCLUDE 'onst_math.h' PARAMETER (N = 273, NB = 4*N, LDA = 16) REAL*8 tau(-1:n+2), bs0(4,n), bs1(4,n), wa(lda,nb), wb(nb), + w(nb), wd(nb), glb(1), apar(6), + wr(2), fobs(2), ferr(2) CHARACTER file_obs*80, file_ini*80, file_out*80 lu0 = 10 lu1 = 11 file_obs = 'obs.prn' file_ini = 'q_initial.prn' file_out = 'qdata1.prn' glb(1) = 1d STEP Define knot sequene (tau), initial spline approximation (bs0) and number of iterations (niter): OPEN(lu0,file=file_ini) DO i = -1, N+2 IF (i.ge. 1.AND. i.le. N) THEN READ(lu0,*) tau(i), (bs0(j,i),j=1,4) ELSE READ(lu0,*) tau(i) ENDIF ENDDO CLOSE(lu0) tbeg = tau(2) tend = tau(n-1) niter = 5 This is the main iteration loop for the non-linear least-squares -- in eah loop, bs0 is improved to bs1: DO iter = 1, niter STEP Initialize normal equations et: mode = 0 CALL attupd(n, tau, bs0, mode, glb, apar, tobs, fobs, ferr, + wa, wb, w, wd, bs1, ierr, wr) Loop over the observations, aepting those in [tbeg,tend]: OPEN(lu0,file=file_obs) nobs = CONTINUE READ(lu0,*,END=200) tobs, id, ia, (fobs(i),ferr(i),i=1,2) 6

7 IF (tobs.lt. tbeg) GOTO 100 IF (tobs.gt. tend) GOTO 200 CALL getstar(id, apar) STEP Update normal equations: mode = 2 CALL attupd(n, tau, bs0, mode, glb, apar, tobs, fobs, ferr, + wa, wb, w, wd, bs1, ierr, wr) IF (ierr.ne. 0) print *, 'ierr,tobs=',ierr,tobs nobs = nobs + 1 GOTO 100 No more observations (or tobs exeeds tend): 200 CONTINUE CLOSE(lu0) STEP Solve normal equations and update spline oeffiients: mode = 3 CALL attupd(n, tau, bs0, mode, glb, apar, tobs, fobs, ferr, + wa, wb, w, wd, bs1, ierr, wr) WRITE(*,'(/x,a,i10)') 'iteration = ', iter WRITE(*,'(x,a,i10)') 'attupd ierr = ', ierr WRITE(*,'(x,a,2i10)') 'nobs = ', nobs, nobs WRITE(*,'(x,a,2f10.6)') 'mean weight redution = ', wr Calulate the RMS update and (in preparation for the next iteration) opy bs1 to bs0: rms = 0d0 DO i = 1, N DO j = 1, 4 rms = rms + (bs1(j,i) - bs0(j,i))**2 bs0(j,i) = bs1(j,i) ENDDO ENDDO rms = dsqrt(rms/dble(n*4)) WRITE(*,'(x,a,g12.6)') 'RMS update (uas) = ', rms/uas ENDDO OUTPUT Output updated spline data to file_out: OPEN(lu1,file=file_out) DO i = -1, N+2 IF (i.ge. 1.AND. i.le. N) THEN WRITE(lu1,'(f15.9,4f20.16)') tau(i), (bs1(j,i),j=1,4) ELSE WRITE(lu1,'(f15.9,4f20.16)') tau(i), (0d0,j=1,4) ENDIF ENDDO CLOSE(11) WRITE(*,'(/x,a,a)') 'spline data written to: ', file_out(1:20) END Exeuting fit attitude generates the following standard output (out fit attitude), where the onvergene of the five iterations an be seen. Between the last two iterations the RMS update of the spline oeffiients S mn is below 0:02 μas. 7

8 iteration = 1 attupd ierr = 0 nobs = mean weight redution = RMS update (uas) = iteration = 2 attupd ierr = 0 nobs = mean weight redution = RMS update (uas) = iteration = 3 attupd ierr = 0 nobs = mean weight redution = RMS update (uas) = iteration = 4 attupd ierr = 0 nobs = mean weight redution = RMS update (uas) = iteration = 5 attupd ierr = 0 nobs = mean weight redution = RMS update (uas) = E-01 spline data written to: qdata1.prn The first five lines of the output data file qdata1.prn are shown below. The meaning of the olumns is the same as for the starting approximation approximation. t S 1n S 2n S 3n S 4n Fitting the attitude spline (distributed proessing) In attupd the same subroutine is used for steps 1, 2 and 3, as ontrolled by the argument mode. In order to implement the sheme in Figure 2, the routine is broken up into three separate routines alled attupd ini (initialize), attupd a (aumulate) and attupd fin (finalize). These are ontained in the file attupd dis.f. It would be possible to run these routines from a single main program (as for attupd), but in order to bring out more learly the modularity and the interation between the different modules, they are here atually run by separate main programs alled att ini, att a and att fin. Standard input is used to ontrol e.g. the proessor number and data are exhanged between them by means of temporary binary files (*.uf). This struture in priniple allows the attitude fitting to be run on a luster without speial parallellization ode, although it would not be very effiient due to the use of disk files for data exhange. 8

9 Here the NPROC = 4 proesses are atually run in sequene rather than in parallel, using the following shell sript (att bat): #!/bin/sh # unix bath program to illustrate distributed proessing # of attitude determination. # NITER iterations are made, eah onsisting of three steps: # 1. initialize data (att_ini) - only needed for first iteration # 2. aumulate normals (att_a) # 3. finalize and solve normals (att_fin) # After the last iteration, results are written to a text file. # The seond step is divided into NPROC separate proesses, # here run sequentially, but they ould also run in parallel. # Step 3 has to wait until all NPROC proesses are ready. # define number of iterations and number of distributed proesses: set NITER = 5 set NPROC = 4 # Step 1 (initialize): eho $NPROC att_ini # this is the iteration loop: set ITER = 1 while ( $ITER <= $NITER ) # Step 2 (here written expliitly for IPROC = 1, NPROC): eho 1 att_a eho 2 att_a eho 3 att_a eho 4 att_a # Step 3: eho $ITER $NPROC ITER++ end # onvert final result (qdata.uf) to text file (qdata.prn) # for easier inspetion: print_qdata # leanup: rm work*.uf obs_*.uf qdata.uf Exeuting att bat generates standard output (out att bat) whih is idential to the output from fit attitude exept for the name of the output data file. The output data file (qdata.prn) is also virtually idential to the previous one (qdata1.prn). This merely shows that the distribution of steps 1 and 2 is matematially (if not quite numerially) equivalent to the original algorithm. If there is some slight numerial differene between qdata.prn and qdata1.prn it will be aused by rounding errors in the aumulation of the normal equations. 9

10 4 Files distributed with this doument The files available with this doument are as follows: --- program units for non-distributed proessing (Set. 3.4): fit_attitude.f out_fit_attitude qdata1.prn --- program units for distributed proessing (Set. 3.5): att_a.f main program to aumulate normals att_bat bath program to run att_ini, att_a, att_fin att_fin.f main program to finalize attitude updating att_ini.f main program to initialize attitude updating attupd_dis.f subroutines for attitude updating out_att_bat standard output from att_bat print_qdata.f onvert binary output data to printable qdata.prn output data file --- general routines: onst_astr.h inlude file for astronomial onstants onst_math.h inlude file for mathematial onstants getstar.f return soure parameters (Set. 3.1) LL34.f subroutine pakage (see below) makefile makefile for ompiling all programs obs.prn input observational data (Set. 3.2) q_initial.prn initial spline approximation (Set. 3.3) Remark: the subroutine pakage LL34.f (v. 2.1) used here is in priniple the same as distributed with GAIA LL 34.2, exept that some unneessary print statements used for debugging and then left there by mistake have been removed. 10