1 date: December 12, 1997 le: walters9. Automatic Dierentiation. form of an algorithm (e.g. computer program) so

Transcription

1 1 date: December 12, 1997 le: walters9 AUTOMATIC DIFFERENTIATION: POINT AND INTERVAL TAYLOR OPERATORS Abstract. Frequently of use in optimization problems, automatic dierentiation may be used to generate Taylor coecients. Specialized software tools generate Taylor series approximations, one term at a time, more eciently than the general AD software used to compute (partial) derivatives. Through the use of operator overloading, these tools provide a relatively easy-to-use interface that minimizes the complications of working with both point and interval operations. Introduction. First, we briey survey the tools of automatic dierentiation and operator overloading used to compute point- and interval-valued Taylor coef- cients. We assume that f is an analytic function f : R! R. Automatic dierentiation (AD or computational dierentiation) is the process of computing the derivatives of a function f at a point t = t 0 by applying rules of calculus for dierentiation [10, 11, 18, 19]. One way to implement AD uses overloaded operators. Operator Overloading. An overloaded (or generic) operator invokes a procedure corresponding to the types of its operands. Most programming languages implement this technique for arithmetic operations. The sums of two oating point numbers, two integers, or one oating point number and one integer are computed using three dierent procedures for addition. Fortran 77 or C denies the programmer the ability to replace or modify the various routines used implicitly for integer, oating point, or mixed-operand arithmetic, but Fortran 90, C++, and Ada support operator overloading for user-dened types. Once we have de- ned an overloaded operator for each rule of differentiation, AD software performs those operations on program code for f, as shown below. The operators either propagate derivative values or construct a code list for their computation. We give prototypical examples of operators overloaded to propagate Taylor coecients below. Automatic Dierentiation. The AD process requires that we have f in the form of an algorithm (e.g. computer program) so that we can easily separate and order its operations. For example, given f(t) = e t =(2 + t), we can express f as an algorithm in Fortran 90 or in C++ (using an assumed AD module or class): program Example1 use AD_Module type(ad_independent) :: t type(ad_dependent) f = exp(t) / (2 + t) end program Example1 #include "AD_class.h" void main (void) { AD_Independent t(0); AD_Dependent f; = AD_Independent(0) f = exp (t) / (2 + t); :: f Figure 1. Fortran & C++ calls to AD operators. In this section, we use AD to compute rst derivatives. In the next section, we extend to point- and interval-valued Taylor series. To understand the AD process, we parse the program above into a sequence of unary and binary operations, called a code list, computational graph, or \tape" [10]: x 0 = t 0 x 2 = 2 + x 0 x 1 = exp(x 0 ) x 3 = x 1 =x 2 Dierentiation is a simple mechanical process for propagating derivative values. Let t = t 0 represent the value of the independent variable with respect to which we dierentiate. We know how to take the derivative of a variable, a constant, and unary and binary operations (i.e. +,?,, =, sin, cos, exp, etc.). Then AD software annotates the code list: x 0 = t 0 ; rx 0 = 1 x 1 = exp(x 0 ); rx 1 = exp(x 0 ) rx 0 x 2 = 2 + x 0 ; rx 2 = 0 + rx 0 x 3 = x 1 =x 2 ; rx 3 = (rx 1? rx 2 x 3 )=x 2

2 le: walters9 date: December 12, AD propagates values of derivatives, not expressions as symbolic dierentiation does. AD values are exact (up to round-o), not approximations of unknown quality as nite dierences. For more information regarding AD and its applications, see [2, 9, 10, 11, 18, 19], or the bibliography [22]. AD software can use overloaded operators in two dierent ways. Operators can propagate both the value x i and its derivative rx i, as suggested by the annotated code list above. This approach is easy to understand and to program. We give prototypical Taylor operators of this avor below. The second approach has the operators construct and store the code list. Various optimizations and parallel scheduling [1, 4, 13] may be applied to the code list. Then the code list is interpreted to propagate derivative values. This is the approach of AD tools such as ADOL{C [12], ADOL{F [21], AD01 [17], or INTOPT 90 [14]. The second approach is much more exible, allowing the code list to be traversed in either the forward or reverse modes of AD (see [10]) or with various arithmetics (e.g. point- or intervalvalued series). AD may be applied to functions of more than one variable, in which partial derivatives with respect to each are computed in turn, and to vector functions, in which the component functions are dierentiated in succession. In addition, we can compute higher order derivative values. One application of AD involving higher order derivatives of f is the computation of Taylor (series) coecients to which we turn in the next section. Source code transformation is a third approach to AD software used by ATOMFT [5] for Taylor series and by ADIFOR [3], PADRE2 [15], or Odyssee [20] for partial derivatives. Such tools accept the algorithm for f as data, rather than for execution, and produce code for computing the desired derivatives. The resulting code often executes more rapidly than code using overloaded operators. Taylor Coecients. We dene the Taylor coecients of the analytic function f at the point t = t 0 : (fjt 0 ) i := 1 d i f(t 0 ) ; i! dt i for i = 0; 1; 2; 3; : : :, and let F := ((fjt 0 ) i ) denote the vector of Taylor coecients. Then Taylor's Theorem says that there exists some point (usually not practically obtainable) between t and t 0 such that f(t) = px i=0 (fjt 0 ) i (t? t 0 ) i (1) + 1 d p+1 f() (t? t (p + 1)! dt p+1 0 ) p+1 : Computation of Taylor coecients requires dierentiation of f. We generate Taylor coef- cients automatically using recursion formulas for unary and binary operations. For example, the recurrences we need for our example f(t) = e t =(2 + t) are x(t) = exp u(t) =) x 0 = xu 0 (x) 0 = exp(u) 0 (x) i = Pi?1 (x) j=0 j (u) i?j (i? j)=i x(t) = u(t) + v(t) (x) i = (u) i + (v) i x(t) = u(t)=v(t) =) xv = u (x) i = (u) i? Pi?1 j=0 (x) j (v) i?j =(v) 0 : The recursion relations are described in more detail in [18]. Except for + and?, each recurrence follows from Leibnitz' rule for the Taylor coecients of a product. The relations can be viewed as a lower triangular system. The recurrence represents a solution by forward substitution, but there are sometimes accuracy or stability advantages in an iterative solution to the lower triangular system. The recurrences for each operation can be evaluated in oatingpoint, complex, interval, or other appropriate arithmetic.

3 3 date: December 12, 1997 le: walters9 To compute the formal series for f(t) = e t =(2 + t) expanded at t = 0, X 0 := (t 0 ; 1; 0; : : :) = (0; 1; 0; : : :) X 1 := exp X 0 = (1; 1; 1=2!; 1=3!; : : :) X 2 := 2 + X 0 = (2; 1; 0; : : :) X 3 := X 2 =X 3 = (1=2; 1=4; 1=8; 1=48; : : :) (2) Point and Interval Taylor Operators. As foreshadowed by this example, we dene an abstract data type for Taylor series and use operator overloading to dene actions on objects of that type using previously dened oatingpoint and interval operations. Design of Operators. In this section, we give prototypical operators for the direct propagation of Taylor coecients such as might be called from code similar to that shown in Figure 1. Direct propagation of values works by translating each operation into a call to the appropriate AD routine at compile time. Thus, simply compiling the source code for f and linking it with the overloaded operator routines creates a program that computes the Taylor coecients of f at t = t 0. For illustration, we provide only a stripped-down prototype with operators required for the example f(t) = e t =(2+t). We surpress issues of references and the like that are essential to the design of a useful class. See [6] for a description of a set of interval Taylor operators in Ada. class Taylor { private: // Or make a template: const int Max_Length = 20; Value_type coef[max_length]; public: Taylor ( Value_type t_0 ) { // Constructor for Independents coef[0] = t_0; coef[1] = 1; for (int i = 2; i < Max_Length; i++) { coef[i] = 0; Taylor ( void ) { // Constructor for Dependents for (int i = 0; i < Max_Length; i++) { coef[i] = 0; Taylor ( Taylor &U ) { // Copy Constructor for (int i = 0; i < Max_Length; i++) {coef[i] = Value_type(U.coef[i]); friend Taylor operator + (int u, Taylor V) { V.coef[0] += u; return V; friend Taylor operator / Taylor X; (Taylor U, Taylor V) { for (int i = 0; i < Max_Length; i++){ Value_type sum = U.coef[i]; for (int j = 0; j < i; j++) {sum -= X.coef[j] * V.coef[i-j]; X.coef[i] = sum / V.coef[0]; return X; friend Taylor exp (Taylor U) { /* Similar to divide */ Value_type getcoef (int i) { return coef[i]; ; // end class Taylor If instead, operators for AD type record a code list, then an interpreter reads each node from the code list and calls the appropriate operator from class Taylor: Taylor Operand[MemSize]; for (int i = 0; i < CodeSize; i++) { Node = getnextoperation (); switch (Node.OpCode) { case PLUS : Operand[Node.Result]... = Operand[Node.Left] + Operand[Node.Right]; break; case EXP : Operand[Node.Result]... = exp ( Operand[Node.Left] ); break; Use of Interval Operators. We have mentioned the possibility of working with interval values but not the signicance of doing so. From Equation (1) for an interval t,

4 le: walters9 date: December 12, and for all t 2 t, f(t) 2 px i=0 (fjt 0 ) i (t? t 0 ) i (3) + 1 d p+1 f(t) (t? t (p + 1)! dt p+1 0 ) p+1 : f In a computer implementation, the summation is done in interval arithmetic to ensure enclosure. The series Taylor coecients (fjt0) i are narrow intervals whose width comes only from outward rounding. The remainder term is the Taylor coecient (fjt) i, where the recurrence relations are evaluated in interval arithmetic. The series (3) can be used to bound the range of f (see [7]), for validated quadrature [8], or for rigorous solution of ODEs [16]. For the example f(t) = e t =(2 + t), we repeat the sequence of computations of Equation (2) for the interval t0 = [0; 0] and for t = [?1; 1]: ((fj[0]) i ) = (1=2; 1=4; 1=8; 1=48; : : :) ((fj[?1; 1]) i ) = ([0:12; 2:72]; [?2:59; 2:68]; [?2:64; 4:04]; : : :) Assembling these according to Equation (3) yields enclosures for all t 2 [?1; 1]: f(t) 2 (fj[?1; 1]) 0 = [0:12; 2:72] 2 (fj[0]) 0 + (fj[?1; 1]) 1 (t? 0) = 1 + [?2:59; 2:68]t 2 2 (fj[0]) 0 + (fj[0]) 1 (t? 0). +(fj[?1; 1]) 2 (t? 0) 2 = t + [?2:64; 4:04]t2 4 To demonstrate the true power of this approximation technique, we plot the corresponding 5, 10, and 20 term enclosures in Figure t Figure 2 { Taylor series enclosures of f One-at-a-time Coecient Generation. The Taylor operators described the preceding section accept vectors of p Taylor coecients for operands u and v and return Taylor coecients for result x with complexity O(p 2 ). However, for applications such as ODEs or orderadaptive quadrature, the entire operand series is not known, and we need to compute terms one at a time [6]. For example, for the DE u 0 = f(t; u) = exp(u)=(2 + t); u(0) = 1; initial condition u(0) = 1 =) (uj0) 0 = 1; DE u 0 = exp(u)=(2 + t) =) (uj0) 1 = exp(1)=(2 + 0) = e=2; u 00 = u0 exp(u) 2+t? exp(u) (2+t) 2 =) (uj0) 2 = (e=2) exp(1)=(2 + 0)? e=4 = e(e? 1)=4 etc. Successive terms can be computed by interpreting the code list for f(t; u) repeatedly for series of increasing length for u. Each iteration of the automatic generation process yields an additional Taylor coecient. Unfortunately, a simple implementation of Taylor operators has complexity O(p 3 ) because already known coecients of u 0 are recomputed. However, since the order of operations is the same in each iteration, we can increase the eciency of the computations by storing intermediate results [6]. Each overloaded operator routine calls a memory allocation procedure that refers it to the next space in an array. If that space is empty, we store Taylor

5 5 date: December 12, 1997 le: walters9 coecient values for that variable. Otherwise, the space must contain the previously computed Taylor coecients of that variable, which we can then use to more quickly compute the next coef- cient in the set. With clever book-keeping, we compute p oating-point or interval-valued Taylor coecients one at a time in O(p 2 ) time. Trade-os. We may strive for three goals when writing software for point and interval Taylor operations: storage space eciency, time eciency, and ease of use. These three factors are often at odds with each other. Carefully implemented operator overloading provides an easy to use interface and provides reasonable time and space eciency. We may achieve greater time and space eciency by using source code transformation. In conclusion, automatic dierentiation through Taylor operators shows merit as a technique for computing guaranteed interval enclosures about a function f. Further eorts to rene this technique may provide us with a tool that handles multi-variate functions, and runs signicantly faster thanks to parallelization and improved optimization techniques. References [1] Benary, Jochen: `Parallelism in the Reverse Mode', Computational Dierentiation: Techniques, Applications, and Tools, in [2], pp. 137{147. [2] Berz, Martin, Bischof, Christian, Corliss, George, and Griewank, Andreas (eds.): Computational Dierentiation: Techniques, Applications, and Tools, SIAM, Philadelphia, Penn., [3] Bischof, Christian H., Carle, Alan, Khademi, Peyvand M., Mauer, Andrew, and Hovland, Paul: ADIFOR 2.0 User's Guide, Technical Memorandum ANL/MCS{TM{192, Mathematics and Computer Science Division, Argonne National Laboratory, [4] Bischof, Christian H., and Haghighat, Mohammad R.: `Hierarchical Approaches to Automatic Dierentiation', Computational Dierentiation: Techniques, Applications, and Tools, in [2], pp. 83{94. [5] Chang, Y. F., and Corliss, George F.: `ATOMFT: Solving ODEs and DAEs Using Taylor Series', Computers and Mathematics with Applications 28 (1994), 209{233. [6] Corliss, George F.: `Overloading Point and Interval Taylor Operators', Automatic Dierentiation of Algorithms: Theory, Implementation, and Application, in [11], pp. 139{146. [7] Corliss, George F.: `Automatic Dierentiation: Computing the Range of Derivatives', this volume. [8] Corliss, George F., and Rall, Louis B.: `Adaptive, Self-Validating Quadrature', SIAM J. Sci. Stat. Comput. 8, no. 5 (1987), 831{847. [9] Griewank, Andreas: `On Automatic Dierentiation', Mathematical Programming: Recent Developments and Applications, in Masao Iri and Kunio Tanabe (eds.). Kluwer Academic Publishers, Dordrecht, 1989, pp. 83{108. [10] Griewank, Andreas: `The Chain Rule Revisited in Scientic Computing', SIAM News 24 (May & July 1991), no. 3, p. 20 & no. 4, p. 8. [11] Griewank, Andreas, and Corliss, George F. (eds.): Automatic Dierentiation of Algorithms: Theory, Implementation, and Application, SIAM, Philadelphia, Penn., [12] Griewank, Andreas, Juedes, David, and Utke, Jean: `ADOL{C, A Package for the Automatic Dierentiation of Algorithms Written in C/C++', ACM Trans. Math. Software 22, no. 2 (1996), 131{ 167. [13] Griewank, Andreas, and Reese, Shawn: `On the Calculation of Jacobian Matrices by the Markowitz Rule', Automatic Dierentiation of Algorithms: Theory, Implementation, and Application, in [11], pp. 126{135. [14] Kearfott, R. Baker: `A Fortran 90 Environment for Research and Prototyping of Enclosure Algorithms for Nonlinear Equations and Global Optimization', ACM Trans. Math. Software 21, no. 1 (1995), 63{78. [15] Kubota, Koichi: `PADRE2 - Fortran Precompiler for Automatic Dierentiation and Estimates of Rounding Error', Computational Dierentiation: Techniques, Applications, and Tools, in [2], pp. 367{ 374. [16] Lohner, Rudolf J.: `Enclosing the solutions of ordinary initial and boundary value problems', Computer Arithmetic: Scientic Computation and Programming Languages, in Edgar W. Kaucher, Ulrich W. Kulisch, and Christian Ullrich (eds.). Wiley-Teubner Series in Computer Science, Stuttgart, 1987, pp. 255{286. [17] Pryce, John D., and Reid, John K.: AD01: a Fortran 90 Code for Automatic Dierentiation, Technical Report RAL-TR-97xxx, Rutherford-Appleton Laboratory, [18] Rall, Louis B.: Automatic Dierentiation: Techniques and Applications, Vol. 120 of Lecture Notes in Computer Science, Springer-Verlag, Berlin, 1981.

6 le: walters9 date: December 12, [19] Rall, Louis B., and Corliss, George F.: `An Introduction to Automatic Dierentiation', Computational Dierentiation: Techniques, Applications, and Tools, in [2], pp. 1{17. [20] Rostaing, Nicole, Dalmas, Stephane, and Galligo, Andre: `Automatic Dierentiation in Odyssee', Tellus 45A (1993), 558{568. [21] Shiriaev, Dmitri: `ADOL{F Automatic Dierentiation of Fortran Codes', Computational Dierentiation: Techniques, Applications, and Tools, in [2], pp. 375{384. [22] Yang, Wenhong, and Corliss, George: `Bibliography of Computational Dierentiation', Computational Dierentiation: Techniques, Applications, and Tools, in [2], pp. 393{418. James B. Walters and George F. Corliss Marquette University, Milwaukee, Wisc. USA address: [walters georgec]@mscs.mu.edu Key words and phrases: automatic dierentiation, code list, interval arithmetic, overloaded operator, Taylor series.