represent = as a finite deimal" either in base 0 or in base. We an imagine that the omputer first omputes the mathematial = then rounds the result to

Transcription

1 Sientifi Computing Chapter I Computer Arithmeti Jonathan Goodman Courant Institute of Mathemaial Sienes Last revised January, 00 Introdution One of the many soures of error in sientifi omputing is inexat omputer arithmeti, whih is alled roundoff error". Roundoff error is inevitable but its onsequenes vary. Some omputations yield nearly exat results, while others give answers that are ompletely wrong. This is true even for problems that involve no other approximations. For example, the gaussian elimination algorithm for solving systems of linear equations would give the exat solution if all the omputations were performed exatly. When these omputations are done in finite preision floating point arithmeti, we might or might not get something lose to the right answer. Suppose your omputed answer is ompletely wrong when a omputation without roundoff would be exatly right. There are three possible situations. Distinguishing these is one of the most important lessions of the ourse. The first guess would be that your program has a bug. You an test for this by finding an instane of your problem that is small enough and easy enough for roundoff to be a less likely exuse. Assuming the oding is orret, the problem may be so ill onditioned" that no algorithm ould reasonably be expeted to work. The last possibility is that there is an aurate method for getting the answer in floating point, just not yours. In this ase, your method is alled unstable". You an analyse an unstable algorithm and try to find a stable variant. However, if your problem is ill onditioned, you took a wrong path long before the solution algorithm design stage. A large lass of problems ranging from sophistiated inverse problems" to simple polynomial interpolation are ill onditioned. Roundoff and anellation If x is a floating point variable (type float or double in C) and we say, for example, x=/, then x will not get the value exatly =. It is impossible to

2 represent = as a finite deimal" either in base 0 or in base. We an imagine that the omputer first omputes the mathematial = then rounds the result to a nearby number with a finite omputer representation. Most arithmeti operations involving floating point numbers require roundoff. As a result, many properties of arithmeti, suh as assoiativity of addition ((x + y)+ z = x +(y + z)), and distributivity (x (y + z) =x y + x z), do not hold exatly when done in floating point. We must be areful not to write odes that depend on exat arithmeti to work orretly. For example, suppose we want to divide the interval [0;T] into n equal parts. The following ode will probably be an infinite loop: float deltat, t; deltat = T/n; for ( t = deltat; t!= T; t+= deltat )...} If we replae the ontinuation riterion t!= T by t <= T, we will get either n or n + trips through the loop, depending on the sign of the roundoff error. To get the orret result, use exat integer arithmeti suh as this: float deltat, t; int i; deltat = T/n; t = (float) 0; for ( i = 0; i < n; i++) t+=deltat;...} In disussing the aumulation of roundoff errors, we suppose x is a real number that would result from a sequene of omputations, and ~x is what the omputer produes in floating point arithmeti. To start, we onsider the ase where ~x is simply the result of rounding" x, that is, finding the omputer floating point number losest to x. This rounding error, x ~x, is generally proportional to x. The small onstant of proportionality haraterizes the auray of the arithmeti. For example, if we use deimal arithmeti and keep 4 signifiant deimals, then ß = : would round to ~ß = :4. The rounding error is about The mass of a proton, in grams, is m ß : This number would round to ~m = :67 0 4, for a roundoff error of ~m m ß 0 8. It might seem that the rounding error for m is muh less than for ß. However, in relative terms they are about the same: fi fi fi fi fi ß ~ß fi fi m fi fi fi ß fi ο ~m fi fi fi fi m fi ο 0 4 ; where we write a ο b to mean that a and b have the same order of magnitude. It is onvenient to represent the above rounding error by writing ~x =(+ffl)x : () It might be more orret to write m =: grams. In the omputer we need to give a pure number. It is up to the programmer to remember that m is the number of grams.

3 We say that ffl mah is the mahine preision" if the rounding error in () satisfies jfflj»ffl mah ; () as long as x is in the range of normalized numbers" where floating point arithmeti usually takes plae. Atually, the mahine preision", ffl mah, is no longer a funtion of the mahine sine most mahines use IEEE arithmeti. It does depend on whether you use single or double preision. Sine the relative distane between omputer floating point numbers is more or less uniform aross the normal range, we mayaswell take x =. Thus, is possible to haraterize ffl mah as the largest ffl so that +ffl rounds to. This is often taken as the definition of ffl mah. Our model of omputer arithmeti is that all error omes from rounding error. The result of any arithmeti operation should be the exat result, orretly rounded". Suppose, for example, some plae in a omputation we have z = x + y. At that point, we have ~x and ~y. First we perform the mathematial addition to get the exat z =~x +~y. Then we round, finding the losest floating point number to z : ~z =(+ffl r )z, where ffl r satisfies () beause it orresponds to a single rounding operation. Rounding error an aumulate in a omputation. If we have ~x =(+ffl x )x and ~y =(+ffl y )y, then ~z =(+ffl z )z, where ffl z ß ffl r + xffl x + yffl y z : () In this way, roundoff error an aumulate during a long omputation and lead to errors muh larger than roundoff. If there are many steps in the omputation, then we mayhave jffl z j >> ffl mah, and it is all but impossible that jffl z j << ffl mah. Even the aumulation of roundoff error would be usually harmless but for anellation", the inrease in error that omes from subtrating two positive (or negative) numbers. A world with no negative numbers or subtration would have more aurate omputer arithmeti than ours. Subtration magnifies error beause we measure error in relative terms rather than absolute terms. If x and y are positive, then () shows that jffl z j»ffl mah + jffl x j + jffl y j. Error aumulates but it not amplified. On the other hand, if x and y have opposite signs but nearly equal magnitudes, then we mayhave jzj >> jxj + jyj and we an have jffl z j >> ffl mah + jffl x j + jffl y j ; beause the denominator in () is muh smaller than x or y. The error has not grown muh in absolute terms: jz ~zj»ffl mah jzj + jx ~xj + jx ~xj : The error has grown in relative terms beause the answer has shrunk. This is beause of the auray of omputer arithmeti. It would take avery long omputation build up enough error to overwhelm the answer.

4 Error amplifiation through anellation does not have to happen all at one. It an build up through a long omputation with many subtrations. For example, start with the geometri series We denote the tails of this series d k = =+ + k = k + + : k+ + : Clearly d k+ = d k. k Suppose we start with d 0 = and subtrat the right hand terms kone by one, giving = ; 4 = ; 8 9 = ; et. If we do this on the omputer, we quikly ome to a number that is 00% wrong even though the two terms on the right side of the subtrations differ by a fator of always. The C program below illustrates these points. #inlude <stdio.h> #define MAXN 00 #define PRINTFREQUENCY 0 int main() int n; double twoby, x, dapprox, dbetter, diff, rdiff; twoby = ( (double) )/( (double) ); dapprox = (double) ; dbetter = (double) ; x = (double) ; for( n=0; n < MAXN; n++) dapprox -= x; dbetter *= twoby; x *= twoby; diff = dbetter - dapprox; rdiff = diff/dbetter; if( n % PRINTFREQUENCY == 0 ) printf("n = %d, d = %0.e,", n, dbetter); printf(" abs diff = %.4e,", diff); 4

5 printf(" rel diff = %0.e n", rdiff); } } return 0; } The output is n = 0, d =.000e+00, abs diff = e+00, rel diff = 0.000e+00 n = 0, d =.468e-0, abs diff = e-6, rel diff = -9.40e-5 n = 0, d = 6.05e-04, abs diff = e-6, rel diff = -5.50e- n = 0, d =.04e-05, abs diff = e-6, rel diff = -.75e- n = 40, d =.809e-07, abs diff = e-6, rel diff = -.8e-09 n = 50, d =.7e-09, abs diff = e-6, rel diff = -.056e-07 n = 60, d = 5.49e-, abs diff = e-6, rel diff = e-06 n = 70, d = 9.4e-, abs diff = e-6, rel diff = -.50e-04 n = 80, d =.66e-4, abs diff = e-6, rel diff = -.04e-0 n = 90, d =.87e-6, abs diff = e-6, rel diff = -.67e+00 The output illustrates this point. The absolute error is nearly onstant after the first ten steps but the relative error inreases exponentially as the answer shrinks. Stability and onditioning The onditioning of a problem refers to how sensitive the answer is to small hanges in the data. For this general disussion, we write A(x) for some omputed quantity, the answer", that depends on data, x. The sensitivity of A with respet to x This means that if x is a small hange in x, and is the resulting hange in A, then x : As explained in the previous setion, it is often more useful to alulate relative hanges. The ratio between the relative hange in A and the relative hange in x is», the ondition number: A From this, a formula for the ondition number is A ß» x x : (4)» A : (5) The ondition number is a dimensionless measure of sensitivity. It is the same, for example, when A is the number of seonds or the number of hours. The ondition number limits the auray whith shih A an be alulated in floating point arithmeti. The first thing we do in any alulation is to round 5

6 the data, x, to the nearest floating point number, ausing an error on the order of the mahine preision fi fi fifi x fi fi fi x fi < ο ffl mah : Therefore, the error in A will be at least fi fi fi A fi fi fi fi A fi > ο»ffl mah : We will enounter problems in later hapters whose ondition numbers are so large that A is essentially unomputable even in double preision arithmeti. Here is a simple example of an ill onditioned problem. Suppose x =(x ;x ) t is a two dimensional vetor (the t supersript means transpose so that x is a olumn vetor). Let P = ψ 4 Consider the iteration proess x k+ = Px k, starting with some x 0. The problem is to use the two numbers x k to reonstrut the two numbers x 0. We an understand what happens here using the eigenvalues and eigenvetors of P. It is easy to hek that P = 4! and P This gives the eigenvalues ( =, = 4 x = y + y then, (letting y k orrespond to the iterate x k ) so : = 4 y k+ = y k and y k+ = 4 yk : y k = y 0 and y k = and the eigenvetors. If we write ; k 4 y 0 : This means that the iteration proess rapidly removes the information about y while preserving y. If we perturb x k by roundoff error in eah omponent, that will perturb two omponents of y k by about the same amount. However, the y perturbation will be amplified by a fator of ( 4) k. For k = 5 in single preision or k = 50 in double preision, this will lead to omplete loss of auray. The underlying reason it is impossible to reonstrut y 0 from x 50 is that the information is not there. The iteration proess has essentially removed all trae of y. : 6

7 Even if A is well ondtioned", i.e.» is not too large, we may still get the wrong answer in a omputation, if we use an unstable" algorithm. The stability of a omputational algorithm is a measure of the relative error amplifiation beyond the minimum (4) implied by the ondition number. For example, in the program above, the number d is omputed in two ways, by repeated subtration (the variable dapprox) and through repeated multipliation by = (the variable dbetter). It is lear that dbetter is orret to within a fator of 00 of ffl mah. On the other hand, the relative error in dapprox grows exponentially beause d dereases exponentially. Note that the absolute differene between d and dapprox essentially stops hanging after the first few steps. In this ase, repeated subtration is an unstable way to ompute d. 4 The IEEE floating point standard The IEEE floating point standard is a set of rules issued by the IEEE (Institute of Eletrial and Eletronis Engineers) on omputer representation and proessing of floating point numbers. Today, most omputers laim to be IEEE ompliant but many ut orners in what (they onsider to be) minor details. The standard is urrently being enlarged to speify some details left open in the original standard, mostly on how programmers interat with flags and traps. The standard has four main goals: () To make floating point arithmeti as aurate as possible. () To produe sensible outomes in exeptional situations. () To standardize floating point operations aross omputers. (4) To give the programmer ontrol over exeption handling. The IEEE standard speifies exatly what floating point numbers are and how they are to be represented in hardware. The most basi unit of information that a omputer stores is a bit, a variable whose value may be either 0 or. Bits are organized into groups of 8 alled bytes. The most ommon unit of omputer number information is the 4 byte ( bit) word. For higher auray this is doubled to 8 bytes or 64 bits. There are possible values for a bit, there are 8 = 56 possible values for a byte, and there are different 4byte words, about 4: billion. A typial omputer should take less than an hour to list all 4: billion bit words. There are two omputer data types that represent numbers. A fixed point number has type int in C and type integer in FORTRAN. A floating point number has type float in C and real in FORTRAN. In most C ompilers, a float by by default has 8 bytes instead of 4. Integer arithmeti is very simple. There are ß bit integers filling the range from about 0 9 to 0 9. Addition, subtration, and multipliation are done exatly whenever the answer is within this range. Most omputers will do something unpreditable when the answer is out of range 7

8 (overflow). For sientifi omputing, fixed point (integer) arithmeti has two drawbaks. One is that there is no representation for numbers that are not integers. Equally important is the small range of values. The number of dollars in the US national debt, several trillion (0 ), annot be represented as a bit integer but is easy to approximate in bit floating point. A floating point (or real") number is a omputer version of the exponential (or sientifi") notation used on alulator displays. Consider the example expression: :49E 5 whih isone way a alulator ould display the number : This expression onsists of a sign bit, s =, a mantissa, m = 49 and an exponent, e = 5. The expression s:mee orresponds to the number s :m 0 e. The IEEE format replaes the base 0 with base, and makes a few other hanges. When a bit word is interpreted as a floating point number, the first bit is the sign bit, s = ±. The next 8 bits form the exponent", e, and the remaining bits determine the fration", f. There are two possible signs, 56 possible exponents (ranging from 0 to 55), and ß 8:4 million possible frations. Normally a floating point number has the value x = ± e 7 (:f) ; where f is base and the notation (:f) means that the expression :f is interpreted in base. Note that the mantissa is :f rather than just the frational part, f. In base anynumber (exept 0) an be normalized so that the mantissa has the form :f. There is no need to store the :" expliitly. For example, the number :75 0 = 57 an be written 75 = = = (+(:0) +(:000) +(:0000) ) = (:00) : Thus, the representation of this number would have sign s = +, exponent e 7 = so that e = 8 = (00000), and fration f = ( ). The entire bit string orresponding to :75 0 then is: z} s z } e z } f : The exeptional ases e = 0 (whih would orrespond to 7 ) and e = 55 (whih would orrespond to 8 ) have omplex and arefully engineered interpretations that make the IEEE standard distintive. If e = 0, the value is x = ± 6 0:f : This feature is alled gradual underflow". ( Underflow" is the situation in whih the result of an operation is not zero but is loser to zero than any 8

9 floating point number.) The orresponding numbers are alled denormalized". Gradual underflow has the onsequene that two floating point numbers are equal, x = y, if and only if subtrating one from the other gives exatly zero. The use of denormalized (or subnormal") numbers makes sense when you onsider the spaing between floating point numbers. If we exlude denormalized numbers then the smallest positive floating pont number (in single preision) is a = 6 (orresponding to e = and f =00 00 ( zeros)) but the next positive floating point number larger than a is b, whih also has e = but now has f =00 0 ( zeros and a ). Beause of the impliit leading in :f, the distane between b and a is times smaller than the distane between a and zero. That is, without gradual underflow, there is a large and unneessary gap between 0 and the nearest nonzero floating point number. The other extreme ase, e = 55, has two subases, inf (for infinity) if f =0 and NaN (for Not a Number) if f 6= 0. Both C and FORTRAN print inf" and NaN" when you printoutavariable in floating point format that has one of these values. The omputer produes inf if the result of an operation is larger than the largest floating point number, in ases suh asx*x*x*x when x =5: 0 5, or exp(x) when x = 04, or /x if x = ±0. (Atually = +0 = +inf and = 0 = -inf). Other invalid operations suh as sqrt(-.), log(-4.), and inf/inf, produe NaN. It is planned that f will ontain information about how or where in the program the NaN was reated but this is not standardized yet. It might be worthwhile to look in the hardware or arithmeti manual of the omputer you are using. Exatly what happens when a floating point exeption, the generation of an inf or a NaN, ours is supposed to be under software ontrol. There is supposed to be a flag (an internal omputer status bit with values 0 or ) that the program an set. If the flag is on, the exeption will ause a program interrupt (the program will halt there and print an error message), otherwise, the omputer will just give the inf or NaN as the result of the operation and ontinue. The default in most situations is the worst of both ations. The omputer generates an interrupt, whih has enormous omputational overhead (see hapter??), then it puts in the inf or NaN without stopping to print an error message. In pratie, it may be hard for the programmer to figure out how to set the arithmeti flags and other arithmeti defaults. The floating point standard speifies that all arithmeti operations should be a aurate as possible within the onstraints of finite preision arithmeti. The result of arithmeti operations is to be the exat result orretly rounded". This means that you an get the omputed result in two steps. First interpret the operand(s) as mathematial real number(s) and perform the mathematial operation exatly. This usually gives a result that annot be represented exatly in IEEE format. The seond step is to round" this mathematial answer, that is, replae it with the IEEE number losest to it. Ties (two IEEE numbers the same distane from the mathematial answer) are broken in some way (e.g. In keeping with its mission to maximize inompatibility, Mirosoft has hanged the names inf and NaN to something similar but different. 9

10 round to nearest even). Any operation involving a NaN produes another NaN. Operations with inf are ommon sense: inf + finite = inf, inf=inf = NaN, finite=inf =0:, inf inf = NaN. From the above, it is lear that the auray of floating point operations is determined by the size of rounding error. This rounding error is determined by the distane between neighboring floating point numbers. Exept for denormalized numbers, neighboring floating numbers differ by one bit in the last bit of the fration, f. This is one part in ß 0 7 in single preision. Note that this is relative error, rather than absolute error. If the result is on the order of 0 then the roundoff error will be on the order of 0 5. This is sometimes expressed by saying that z = x + y produes (x + y)( + ffl) where jfflj»ffl mah, and ffl mah, the mahine epsilon" is on the order of 0 7 in single preision. Double preision IEEE arithmeti uses 8 bytes (64 bits) rather than 4 bytes. There is one sign bit, exponent bits, and 5 fration bits. Therefore the double preision floating point preision is determined by ffl mah = 5 ß That is, double preision arithmeti gives roughly 5 deimal digits of auray instead of 7 for single preision. There are possible exponents in double preision, ranging from 0 to 0. The largest double preision number is of the order of 0 ß Not only is double preision arithmeti more aurate than single preision, but the range of numbers is far greater. Many features of IEEE arithmeti are illustrated in the program below. The reader would do well do do some of this kind of experimentation for herself or himself. The first setion illustrates various how inf and NaN work. Note that e 04 = inf in single preision but not in double preision beause the range of values is larger in double preision. The next setion shows that IEEE addition is ommutative. This is a onsequene of the exat answer orretly rounded" proedure. The exat answer is ommutative so either way the omputer have the same number to round. This ommutativity does not apply to triples of numbers beause the omputer only adds two numbers at a time. The ompiler turns the expression x + y + z into (x + y) + z. It adds x to y, rounds the answer and adds the result to z. The expression z + x + y auses z + x to be rounded and added to y, whih gives a (slightly) different result. Then omes an illustration of type onversion. Doing the integer division i/j gives = whih is rounded to the integer value, 0, and then onverted to floating point format. When y is omputed, the onversion to floating point format is done before the division. Finally there is another example in whih two variables might be expeted to be equal but aren't beause of inexat arithmeti. It is almost always wrong to ask whether two floating point numbers are equal. Last is a serendipitous example of getting exatly the right answer by aident. Don't ount on this! 0

11 A program that explores IEEE arithmeti Some double preision (8 byte or 64 bit) variables double preision xd, yd C programmers note that other variables are delared automatially and have default types. Take an exponential that is out of range x = 04. y = exp(x) write(6,0) x,y 0 format(' The exponential of ',e.4,' is ', e.4) Divide a normal number by infinity z = x/y write(6,40) x, y, z 40 format(' ',e.4,' divided by ',e.4,' gives ',e.4) Divide infinity by infinity w = y z = w/y write(6,60) w, y, z 60 format(' ',e.4,' divided by ',e.4,' gives ',e.4) Take the square root of a negative number z = sqrt( -. ) write(6,80) z 80 format(' The square root of -. is ',e.4) Add NaN to a normal number w = x + z write(6,00) x, z, w 00 format(' ',e.4,' plus ', e.4,' gives ',e.4) Take the same exponential in double preision xd = 04.d0

12 yd = exp( xd ) write(6,0) xd, yd 0 format(' The double preision exponential of ',e.4,' is ',e.4) Trunate a very large double preision number y = yd write(6,40) yd, y 40 format(' The single preision version of ',e.4,' is ',e.4) Explore the arithmeti, does addition ommute? x = sin(.) y = 0.*sin(.) z = 00.*sin(.) s = x + y s = y + x s = x + z s4 = z + x if ( ( s.eq. s ).and. ( s.eq. s4 ) ) then write (6,60) x, y, z 60 format(' Adding pairs of ',e.4,', ',e.4,', and ',e.4,. ' ommutes') else write (6,80) x, y, z 80 format(' Adding pairs of ',e.4,', ',e.4,', and ',e.4,. ' does not ommute') endif s = x + y + z s = y + z + x s = z + x + y if ( ( s.eq. s ).and. ( s.eq. s ) ) then write (6,00) x, y, z 00 format(' Adding triples of ',e.4,', ',e.4,', and ',e.4,. ' ommutes') else write (6,0) x, y, z 0 format(' Adding triples of ',e.4,', ',e.4,', and ',e.4,. ' does not ommute') endif The result of an operation depends on the type of the operand i = j =

13 x = i/j y = real(i)/real(j) write(6,40) i,j,x,y 40 format(' Dividing ',i4,' by ',i4,' gives ',e.4,. ' without onversion',/,. ' and ',e.4,. ' with onversion') Don't expet other arithmeti identities to be exatly true. x = z / y = x + x + x + x + x + x + x + x + x + x + x x = z / 5. w = 5*x if ( y.eq. z ) then write(6,60) z 60 format(' Divided ',e.5,' by and got it bak ',. 'by addition!!!') else write(6,80) z 80 format(' Divided ',e.5,' by and did not get it bak ',. 'by addition') endif if ( w.eq. z ) then write(6,00) z 00 format(' Divided ',e.5,' by 5 and got it bak ',. 'by multipliation!!!') else write(6,0) z 0 format(' Divided ',e.5,' by 5 and did not get it bak ',. 'by multipliation') endif stop end This program produes the output The exponential of 0.040E+0 is Inf 0.040E+0 divided by Inf gives E+00 Inf divided by Inf gives NaN The square root of -. is NaN 0.040E+0 plus NaN gives NaN The double preision exponential of 0.040E+0 is 0.945E+89 The single preision version of 0.945E+89 is Inf Adding pairs of 0.845E+00, 0.909E+0, and 0.4E+0 ommutes

14 Adding triples of 0.845E+00, 0.909E+0, and 0.4E+0 does not ommute Dividing by gives E+00 without onversion and E+00 with onversion Divided 0.4E+0 by and did not get it bak by addition Divided 0.4E+0 by 5 and got it bak by multipliation!!! Note: the following IEEE floating-point arithmeti exeptions ourred and were never leared; see ieee_flags(m): Inexat; Overflow; Invalid Operand; Note: IEEE Infinities were written to ASCII strings or output files; see eonvert(). Note: IEEE NaNs were written to ASCII strings or output files; see eonvert(). Sun's implementation of IEEE arithmeti is disussed in the Numerial Computation Guide. 4