operator overloading algorithmic differentiation the class DifferentialNumber operator overloading

Transcription

1 operator overloading 1 Computing with Differential Numbers algorithmic differentiation the class DifferentialNumber operator overloading 2 Computing with Double Doubles the class DoubleDouble defining +, -, *, and / expression evaluation 3 Wrapping C Code with SWIG swig = simplified wrapper and interface generator MCS 507 Lecture 26 Mathematical, Statistical and Scientific Software Jan Verschelde, 25 October 2013 Scientific Software (MCS 507 L-26) operator overloading 25 October / 43

2 operator overloading 1 Computing with Differential Numbers algorithmic differentiation the class DifferentialNumber operator overloading 2 Computing with Double Doubles the class DoubleDouble defining +, -, *, and / expression evaluation 3 Wrapping C Code with SWIG swig = simplified wrapper and interface generator Scientific Software (MCS 507 L-26) operator overloading 25 October / 43

3 computing with differential numbers A differential number is a tuple df = (f, f ) of two doubles where f is the evaluated derivative of f. The operator overloading encodes the elementary derivative rules for addition, subtraction, multiplication, and division of two real numbers. As we evaluate any expression written in these four elementary operations, we also compute its derivative. With a Python script we illustrate the forward mode of algorithmic differentiation. Reference: section (c) on computing with differential numbers of Introduction to Numerical Analysis, pages 7 to 10, by Arnold Neuimaier, Cambridge Scientific Software (MCS 507 L-26) operator overloading 25 October / 43

4 encoding differentiation rules Let df = (f, f ) and dg = (g, g ), where f and g are dependent on one single variable x. The four basic elementary operations are +,, and /, defined as follows: df + dg = (f, f ) + (g, g ) = (f + g, f + g ) df dg = (f, f ) (g, g ) = (f g, f g ) df dg = (f, f ) (g, g ) = (f g, f g + f g ) df/dg = (f, f )/(g, g ) = (f/g,(f (f/g) g )/g) Constants and the independent variable as differential numbers: Any constant c (independent of x) is represented as (c, 0). The independent variable x is represented as (x, 1). Scientific Software (MCS 507 L-26) operator overloading 25 October / 43

5 evaluating derivatives without explicit formulas Consider f(x) = (x 1)(x + 3) x + 2 (f, f ) ± (g, g ) = (f ± g, f ± g ), (f, f ) (g, g ) = (f g, f g + f g ), (f, f )/(g, g ) = (f/g,(f (f/g) g )/g). at x 0 = 3. Recall the rules: We substitute the differential number (3, 1) in f : (f(3), f (3)) = ((3, 1) 1) ((3, 1) + 3) f((3, 1)) = (3, 1) + 2 (2, 1) (6, 1) = (5, 1) (12, 8) = (5, 1) = (2.4,( )/5) = (2.4, 1.12) Scientific Software (MCS 507 L-26) operator overloading 25 October / 43

6 operator overloading 1 Computing with Differential Numbers algorithmic differentiation the class DifferentialNumber operator overloading 2 Computing with Double Doubles the class DoubleDouble defining +, -, *, and / expression evaluation 3 Wrapping C Code with SWIG swig = simplified wrapper and interface generator Scientific Software (MCS 507 L-26) operator overloading 25 October / 43

7 the data attribute of the class class DifferentialNumber(object): A differential number is a tuple df = (f, f ) of two doubles where f is the evaluated derivative of f. def init (self, xv, xp=1): Initializes a differential number to the tuple (xv, xp). self.val = float(xv) self.der = float(xp) By default, DifferentialNumber(x) for any x is (x, 1). Note the explicit conversion to type float. Scientific Software (MCS 507 L-26) operator overloading 25 October / 43

8 the string respresentation A differential number (f, f ) is represented as a tuple. def str (self): Returns the tuple representation of the differential number. return str((self.val, self.der)) Scientific Software (MCS 507 L-26) operator overloading 25 October / 43

9 operator overloading 1 Computing with Differential Numbers algorithmic differentiation the class DifferentialNumber operator overloading 2 Computing with Double Doubles the class DoubleDouble defining +, -, *, and / expression evaluation 3 Wrapping C Code with SWIG swig = simplified wrapper and interface generator Scientific Software (MCS 507 L-26) operator overloading 25 October / 43

10 adding differential numbers The rule df + dg = (f, f ) + (g, g ) = (f + g, f + g ) is implemented by overriding the builtin operator + defined by add. The self refers to the first operand, while the second operand is given by the other argument. def add (self, other): Defines the addition of two differential numbers. return DifferentialNumber(self.val + other.val, \ self.der + other.der) Scientific Software (MCS 507 L-26) operator overloading 25 October / 43

11 dealing with cases like x def add (self, other): Defines the addition of two differential numbers. if isinstance(other, float): return DifferentialNumber \ (self.val + other, self.der) else: return DifferentialNumber \ (self.val + other.val, \ self.der + other.der) For x we must do DifferentialNumber(2.0, 0.0) + x. Scientific Software (MCS 507 L-26) operator overloading 25 October / 43

12 subtraction of two differential numbers The operator is defined by the sub () method: def sub (self, other): Defines the subtraction of two differential numbers. if isinstance(other, float): return DifferentialNumber \ (self.val - other, self.der) else: return DifferentialNumber \ (self.val - other.val, \ self.der - other.der) This implements the rule df dg = (f, f ) (g, g ) = (f g, f g ) and also deals with the case g = (c, 0). Scientific Software (MCS 507 L-26) operator overloading 25 October / 43

13 multiplying two differential numbers The rule (f, f ) (g, g ) = (f g, f g + f g ) is implemented by overriding the method mul (): def mul (self, other): Defines the product of two differential numbers. return DifferentialNumber \ (self.val*other.val, \ self.der*other.val + \ self.val*other.der) Scientific Software (MCS 507 L-26) operator overloading 25 October / 43

14 multiplication with a constant For a constant c: (f, f ) (c, 0) = (f c, f c). def mul (self, other): Defines the product of two differential numbers. if isinstance(other, float): return DifferentialNumber \ (self.val*other, self.der*other) else: return DifferentialNumber (self.val*other.val, \ self.der*other.val + \ self.val*other.der) Scientific Software (MCS 507 L-26) operator overloading 25 October / 43

15 dividing two differential numbers We override the div () method to define division. Coding the rule (f, f )/(g, g ) = (f/g,(f (f/g) g )/g): def div (self, other): Defines the division of two differential numbers. val = self.val/other.val return DifferentialNumber \ (val, (self.der \ - val*other.der)/other.val) Scientific Software (MCS 507 L-26) operator overloading 25 October / 43

16 dividing by a constant For a constant c: (f, f )/(c, 0) = (f/c, f /c). def div (self, other): Defines the division of two differential numbers. if isinstance(other, float): return DifferentialNumber \ (self.val/other, self.der/other) else: val = self.val/other.val return DifferentialNumber \ (val, (self.der \ - val*other.der)/other.val) Scientific Software (MCS 507 L-26) operator overloading 25 October / 43

17 the main test program def main(): Test on the expression ((x-1)*(x+3))/(x+2). dfx = DifferentialNumber(3, 1) fun = lambda x: ((x-1.0)*(x+3.0))/(x+2.0) print ((x-1)*(x+3))/(x+2) at, dfx, : dfy = fun(dfx) print dfy if name == " main ": main() Scientific Software (MCS 507 L-26) operator overloading 25 October / 43

18 running the test $ python differential_numbers.py ((x-1)*(x+3))/(x+2) at (3.0, 1.0) : (2.4, ) $ Scientific Software (MCS 507 L-26) operator overloading 25 October / 43

19 operator overloading 1 Computing with Differential Numbers algorithmic differentiation the class DifferentialNumber operator overloading 2 Computing with Double Doubles the class DoubleDouble defining +, -, *, and / expression evaluation 3 Wrapping C Code with SWIG swig = simplified wrapper and interface generator Scientific Software (MCS 507 L-26) operator overloading 25 October / 43

20 double double arithmetic in Python Recall what we did in the second half of lecture 25: 1 We looked at a C program to use the double double arithmetic of the QD library to apply Newton s method to approximate 2. 2 We defined the module doubledouble that exports the basic functionality of the C interface of the QD library to perform double double arithmetic in Python. In Python, a double double was seen as a tuple of two doubles. The arithmetic was performed by methods that took 4 arguments: the high and low parts of both operators. Scientific Software (MCS 507 L-26) operator overloading 25 October / 43

21 operator overloading To add two double doubles x and y, instead of z = doubledouble.add(x[0],x[1],y[0],y[1]) we would like to write z = x + y. We will define the class DoubleDouble() and export the method add. Defining the method add allows to use the + operator on any two elements of the class DoubleDouble. The + is a binary operator: 1 the self is the first operand of +, 2 other is the second operand of +. The definition starts with def add (self,other):. Subtraction, multiplication, and division are defined respectively by sub, mul, and div. Scientific Software (MCS 507 L-26) operator overloading 25 October / 43

22 data attributes of the class DoubleDouble Importing the module doubledouble that wraps some of the basic functionality of the QD library: import doubledouble class DoubleDouble(): Wraps some of the functionality of the QD library to perform double double arithmetic. def init (self, hi=0.0, lo=0.0): A double double consists of a high and a low part. self.high = hi self.low = lo Scientific Software (MCS 507 L-26) operator overloading 25 October / 43

23 string representations and deep copy def str (self): Returns the string representation of a double double. return doubledouble.str(self.high, self.low) def repr (self): Returns the representation of a double double. return self. str () def copy(self): Returns a copy of the double double. return DoubleDouble(self.high, self.low) Scientific Software (MCS 507 L-26) operator overloading 25 October / 43

24 operator overloading 1 Computing with Differential Numbers algorithmic differentiation the class DifferentialNumber operator overloading 2 Computing with Double Doubles the class DoubleDouble defining +, -, *, and / expression evaluation 3 Wrapping C Code with SWIG swig = simplified wrapper and interface generator Scientific Software (MCS 507 L-26) operator overloading 25 October / 43

25 addition and subtraction def add (self, other): Returns the sum of two double doubles. z = doubledouble.add(self.high, self.low, \ other.high, other.low) return DoubleDouble(z[0], z[1]) def sub (self, other): Returns the difference of self and the other. z = doubledouble.sub(self.high, self.low, \ other.high, other.low) return DoubleDouble(z[0], z[1]) Scientific Software (MCS 507 L-26) operator overloading 25 October / 43

26 multiplication and division def mul (self, other): Returns the product of two double doubles. z = doubledouble.mul(self.high, self.low, \ other.high, other.low) return DoubleDouble(z[0], z[1]) def div (self, other): Returns the division of self by the other. z = doubledouble.div(self.high, self.low, \ other.high, other.low) return DoubleDouble(z[0], z[1]) Scientific Software (MCS 507 L-26) operator overloading 25 October / 43

27 computing powers To compute x**5, we overload the pow method: def pow (self,n): Returns self to the power n. z = self.copy() for i in range(1,n): z = z*self return z Note: it is more efficient to wrap c_dd_npwr(). Scientific Software (MCS 507 L-26) operator overloading 25 October / 43

28 a basic test on the arithmetic def test(): Basic test on arithmetical operations. hi = lo = e-17 x = DoubleDouble(hi, lo) # defines the sqrt(2) print x = sqrt(2) =, str(x) y = x**2 print x*x =, y z = y/x print x*x/x =, z u = x+x print x+x =, u two = DoubleDouble(2.0) print 2*x =, two*x v = u-x print x+x-x =, v Scientific Software (MCS 507 L-26) operator overloading 25 October / 43

29 running the test Adding to the end of double_double.py the line if name ==" main ": test() so we can run $ python double_double.py x = sqrt(2) = x*x = x*x/x = x+x = *x = x+x-x = Scientific Software (MCS 507 L-26) operator overloading 25 October / 43

30 Newton s method for 2 def newton4sqrt2(): Applies Newton s method to approximate sqrt(2). x = DoubleDouble(2.0) print step 0 :, x for i in range(1,9): z = x**2 z = z + DoubleDouble(2.0) z = z/x z = DoubleDouble(0.5)*z x = z.copy() print step, i, :, x Scientific Software (MCS 507 L-26) operator overloading 25 October / 43

31 operator overloading 1 Computing with Differential Numbers algorithmic differentiation the class DifferentialNumber operator overloading 2 Computing with Double Doubles the class DoubleDouble defining +, -, *, and / expression evaluation 3 Wrapping C Code with SWIG swig = simplified wrapper and interface generator Scientific Software (MCS 507 L-26) operator overloading 25 October / 43

32 a motivating example for interval arithmetic Problem: Evaluate f(x, y) = ( x 2 )y 6 + x 2 (11x 2 y 2 121y 4 2) + 5.5y 8 + x/(2y) at (77617, 33096). An example of Stefano Taschini: Interval Arithmetic: Python Implementation and Applications. In the Proceedings of the 7th Python in Science Conference (SciPy 2008). Let us evaluate this expression with double doubles... Scientific Software (MCS 507 L-26) operator overloading 25 October / 43

33 evaluating an expression print checking the motivating interval arithmetic example f = lambda x,y: (DoubleDouble(333.75) - x**2)*y**6 \ + x**2*(doubledouble(11)*x**2*y**2 \ - DoubleDouble(121)*y**4 - DoubleDouble(2)) \ + DoubleDouble(5.5)*y**8 + x/(doubledouble(2)*y); a = 77617; b = z = f(doubledouble(a),doubledouble(b)) # to check the answer with SymPy : import sympy as sp x,y = sp.var( x,y ) g = (sp.rational(33375)/100 - x**2)*y**6 \ + x**2*(11*x**2*y**2-121*y**4-2) \ + sp.rational(55)/10*y**8 \ + sp.rational(1)*x/(2*y); print evaluating, g, at, (a,b) e = sp.subs(g,(x,y),(a,b)).doit() Scientific Software (MCS 507 L-26) operator overloading 25 October / 43

34 script continued and running the code e15 = e.evalf(15) print numerical value :, z print exact value :, e, ~, e15 print error :, abs(e15 - z.high) We run the script as follows: $ python double_double_eval.py checking the motivating interval arithmetic example evaluating x**2*(11*x**2*y**2-121*y**4-2) + x/(2*y) + 1 numerical value : e+00 exact value : /66192 ~ error : $ Exercise: wrap quad doubles for an accurate evaluation. Scientific Software (MCS 507 L-26) operator overloading 25 October / 43

35 operator overloading 1 Computing with Differential Numbers algorithmic differentiation the class DifferentialNumber operator overloading 2 Computing with Double Doubles the class DoubleDouble defining +, -, *, and / expression evaluation 3 Wrapping C Code with SWIG swig = simplified wrapper and interface generator Scientific Software (MCS 507 L-26) operator overloading 25 October / 43

36 Simplified Wrapper and Interface Generator Available at SWIG is a software development tool that connects programs written in C and C++ with high-level programming languages. SWIG is used with different types of target languages including scripting languages such as Perl, PHP, Python, Tcl and Ruby. SWIG is most commonly used to create high-level interpreted or compiled programming environments, user interfaces, and as a tool for testing and prototyping C/C++ software. SWIG is typically used to parse C/C++ interfaces and generate the glue code required for the above target languages to call into the C/C++ code. SWIG is free software and the code that SWIG generates is compatible with both commercial and non-commercial projects. We will work out the example of the tutorial. Scientific Software (MCS 507 L-26) operator overloading 25 October / 43

37 example.c from #include <time.h> double My_variable = 3.0; int fact(int n) { if (n <= 1) return 1; else return n*fact(n-1); } int my_mod(int x, int y) { return (x%y); } char *get_time() { time_t ltime; time(&ltime); return ctime(&ltime); } Scientific Software (MCS 507 L-26) operator overloading 25 October / 43

38 example.i from %module example %{ /* put header files here or function declarations */ extern double My_variable; extern int fact(int n); extern int my_mod(int x, int y); extern char *get_time(); %} extern double My_variable; extern int fact(int n); extern int my_mod(int x, int y); extern char *get_time(); Scientific Software (MCS 507 L-26) operator overloading 25 October / 43

39 the makefile The makefile defines the compilation instructions, on RHEL: PYTHON=/usr/include/python2.6 example: swig -python example.i gcc -c -fpic example.c example_wrap.c -I$(PYTHON) gcc -shared example.o example_wrap.o \ -o _example.so Executing: $ make example swig -python example.i gcc -c -fpic example.c example_wrap.c -I/usr/include/python gcc -shared example.o example_wrap.o -o _example.so $ Scientific Software (MCS 507 L-26) operator overloading 25 October / 43

40 input and output files Two input files: 1 example.c with C code, 2 example.i is the interface file. To compile successfully, we must know 1 the location of the Python header files, 2 proper linking options, e.g.: -framework Python on Mac. Four output files: 1 example_wrap.c generated C code, 2 example_wrap.o compiled object file, 3 _example.so shared object file, and 4 example.py Python code for wrapper. Scientific Software (MCS 507 L-26) operator overloading 25 October / 43

41 Summary + Exercises Operator overloading makes double double arithmetic natural. Another benefit of operator overloading is to add properties of the computed results, e.g.: derivatives, roundoff errors, operation counts. 1 Extend the class DifferentialNumber so it works for expressions in two variables and a differential number df is (f, f x, f y ), where f x is the derivative with respect to the first variable and f y is the derivative with respect to the second variable. 2 Extend the class DifferentialNumber to compute also the second derivative and a differential number df is (f, f, f ). 3 Wrap the basic operations for quad double arithmetic of the QD library and make the class QuadDouble. Use the class to evaluate the expression of the motivating example for interval arithmetic. Scientific Software (MCS 507 L-26) operator overloading 25 October / 43

42 some more exercises 4 We estimate the roundoff error of an arithmetical operation, for {+,,,/} and floating-point numbers x and y as x y = (x y)(1 + ǫ), where is the floating-point version of and 0 ǫ < ǫ mach, for the machine precision ǫ mach. The roundoff error is bounded by x y ǫ mach. For accumulated roundoff: x 0 x 1 x }{{ n (x } 0 x 1 x n )(1 + ǫ 1 + ǫ ǫ n + ), n operations Truncating higher-order terms, the accumulated roundoff on n operations is estimated by x 0 x 1 x n nǫ mach. Define a class NumericalFloat, overriding +,,, and /, to also compute the upper bound on the roundoff for each operation. The accumulated roundoff is stored as an extra data attribute with each float. Scientific Software (MCS 507 L-26) operator overloading 25 October / 43

43 and some more exercises 4 Using inheritance extend the class DoubleDouble with an extra field count to count the number of arithmetical operations performed to compute a double double. For example, if two constants have counts m and n, their sum has count m + n. 5 Use SWIG to wrap the dd_sqrt() function of lecture 25. Project Two due Friday 1 November at 9AM. Scientific Software (MCS 507 L-26) operator overloading 25 October / 43