Rigorous Estimation of Floating- point Round- off Errors with Symbolic Taylor Expansions

Transcription

1 Rigorous Estimation of Floating- point Round- off Errors with Symbolic Taylor Expansions Alexey Solovyev, Charles Jacobsen Zvonimir Rakamaric, and Ganesh Gopalakrishnan School of Computing, University of Utah, Salt Lake City, Utah, USA

2 Floating-point (FP) numbers Used for a very long time Zuse Z1 (~1938), and even before Standardized, esp since 1985 (IEEE) Standardization is mainly for portable behavior Portable confusion too J Does NOT make FP inherently simpler to reason about FP behavior remains highly non-intuitive, non-compositional, 2

3 Floating-point (FP) numbers Used for a very long time Zuse Z1 (~1938), and even before Standardized, esp since 1985 (IEEE) Standardization is mainly for portable behavior Portable confusion too J Does NOT make FP inherently simpler to reason about FP behavior remains highly non-intuitive, non-compositional, 3

4 Floating-point (FP) numbers Used for a very long time Zuse Z1 (~1938), and even before Standardized, esp since 1985 (IEEE) Standardization is mainly for portable behavior Portable confusion too J Does NOT make FP inherently simpler to reason about FP behavior remains highly non-intuitive, non-compositional, 4

5 Example: Let s do reduction using OpenMP to realize dot-product 5

6 Example: Let s do reduction using OpenMP to realize dot-product 6

7 Example: Let s do reduction using OpenMP to realize dot-product 7

8 Example: Let s do reduction using OpenMP to realize dot-product 8

9 Example: Let s do reduction using OpenMP to realize dot-product 10 tests agree;; let s ship it! 9

10 Example: Let s do reduction using OpenMP to realize dot-product Wait, what s going on?! (3 tests bombed) 10

11 Example: Let s do reduction using OpenMP to realize dot-product (whom to blame?) 11

12 Example: Let s do reduction using OpenMP to realize dot-product + + Aha! + c a + a b b c 12

13 Kahan on Floating-point (Quote obtained from PhD dissertation of Eva Darulova) 13

14 FP behavior can be very non-intuitive! Often, we have a+(b+c) = (a+b)+c 14

15 FP harbors many surprises x is more accurate than log(e x ) but (e x 1) / x can be less accurate than (e x 1) / log(e x ) 15

16 FP harbors many surprises x is more accurate than log(e x ) but (e x 1) / x can be less accurate than (e x 1) / log(e x ) Don t blame the IEEE standard for this J 16

17 Floating-point numbers are ubiquitous! Financial software Embedded controllers Computer graphics Scientific simulation software Air-traffic trajectory definition Machining of prosthetic devices (e.g. hip replacement) Imprecise floating-point results can be really bad!! acement-surgery/multimedia/vid

18 Examples of recent FP errors Simulation in the Large Hadron Collider Need to track charged particles with exquisite precision 10 microns over 10 meters Round-off resulted in missed / mis-identified collisions (cf. Bailey and Borwein) Intel issues specification update for trig library Originally guaranteed to have a one ULP error Measured error was 164-billion ULPs 37 bits of the mantissa were wrong 18

19 A recent rounding-related bug 19

20 A recent rounding-related bug 20

21 Floating-point error analysis Important for FP library designers Even small errors can send simulations astray Important for FP hardware designers Remember the Pentium FDIV bug 21

22 Why only estimate FP error? Answer: Floating-point numbers are unevenly distributed IEEE Std. requires half ULP error for standard operators But, half ULP can be anywhere from to From Goualard,..Midpoint of an interval.. half ULP is this much here..and this much here 22

23 FP : Elephant in a bottle J 23

24 Of course, we can plot and see with a few lines of Racket code (N. Toronto) 24

25 Error Plot for Multiplication Absolute Error for Multiplication Y values X values 25

26 Error Plot for Addition Absolute Error for Addition Y values X values 26

27 Testing is inconclusive Low coverage Missed corner cases No rigorous guarantees 27

28 Formal verification methods are essential for obtaining rigorous floating-point error bounds! 28

29 Why tighter round-off error bounding? Avoids un-necessary verification failure Helps detect important corner-cases Avoids over-compensation With tighter FP error prediction, arithmetic library designers don t need to execute un-necessary compensation steps (Courtesy, conversation with John Harrison, Intel) 29

30 Problems with today s tools Often give pessimistic bounds Time-out Do not handle non-linear operators well Do not handle transcendental functions Exp, Log, Sin, Cos, Some tools support polynomial approximations;; we analyze them without such approximations 30

31 Our contribution: Symbolic Taylor Forms Handles non-linear and transcendental functions Tight error upper bounds Rigorous: Emits a HOL-Lite proof certificate Verification of the certificate guarantees estimate Tool + extended paper : Name of the tool : FPTaylor 31

32 Flow of FPTaylor Given FP Expression and Bounds Obtain Symbolic Taylor Form Obtain Error Function Maximize the Error Function Generate Certificate in HOL-Lite 32

33 Some basics of floating-point s m 22 m 0 e 7 e 0 We consider single-precision for most of our illustrations 33

34 Some basics of floating-point s m 22 m 0 e 7 e 0 Normal numbers : +/ to

35 Some basics of floating-point s m 22 m 0 e 7 e 0 Normal numbers : +/ to Subnormal numbers : +/

36 Some basics of floating-point s m 22 m 0 e 7 e 0 Normal numbers : +/ to Subnormal numbers : +/ While tiny, subnormals do matter! 36

37 FPTaylor is based on the standard IEEE Rounding Model Consider op(x, y) where x and y are floating-point values, and op is a function from floats to reals 37

38 FPTaylor is based on the standard IEEE Rounding Model Consider op(x, y) where x and y are floating-point values, and op is a function from floats to reals IEEE round-off errors are bounded by op(x, y) e op + d op 38

39 FPTaylor is based on the standard IEEE Rounding Model Consider op(x, y) where x and y are floating-point values, and op is a function from floats to reals IEEE round-off errors are bounded by op(x, y) e op + d op For normal values 39

40 FPTaylor is based on the standard IEEE Rounding Model Consider op(x, y) where x and y are floating-point values, and op is a function from floats to reals IEEE round-off errors are bounded by op(x, y) e op + d op For normal values For subnormal values 40

41 FPTaylor is based on the standard IEEE Rounding Model Consider op(x, y) where x and y are floating-point values, and op is a function from floats to reals IEEE round-off errors are bounded by op(x, y) e op + d op For normal values For subnormal values Only one of e op or d op is non-zero 41

42 FPTaylor is based on the standard IEEE Rounding Model Consider op(x, y) where x and y are floating-point values, and op is a function from floats to reals IEEE round-off errors are bounded by op(x, y) e op + d op For normal values For subnormal values Only one of e op or d op is non-zero e op 2 24 d op

43 Example of round-off error modeling (ignoring subnormals) Consider x/(x + y) 43

44 Example of round-off error modeling (ignoring subnormals) Consider x/(x + y) Round-off errors are incurred at both operations 44

45 Example of round-off error modeling (ignoring subnormals) Consider x/(x + y) Round-off errors are incurred at both operations (x + y) Error in bounded by (x + y) e+ 45

46 Example of round-off error modeling (ignoring subnormals on this slide) Consider x/(x + y) Round-off errors are incurred at both operations Error in Error in (x + y) x/(x + y) bounded by bounded by (x + y) e + (x/(x + y)) e / 46

47 Modeling max. round-off estimation Given the task of analyzing x/(x + y) 47

48 Modeling max. round-off estimation Given the task of analyzing x/(x + y) Model it as f(x, y) =x/(x + y) 48

49 Modeling max. round-off estimation Given the task of analyzing x/(x + y) Model it as f(x, y) =x/(x + y) Capturing round-off : f(x, y, e 1,e 2,d 1,d 2 )= (x/((x + y).(1 + e 1 )+d 1 )).(1 + e 2 )+d 2 49

50 Modeling max. round-off estimation Given the task of analyzing x/(x + y) Model it as f(x, y) =x/(x + y) Capturing round-off : f(x, y, e 1,e 2,d 1,d 2 )= (x/((x + y).(1 + e 1 )+d 1 )).(1 + e 2 )+d 2 Note that: f(x, y, 0, 0, 0, 0) = f(x, y) 50

51 Modeling max. round-off estimation Max round-off error in x/(x + y) 51

52 Modeling max. round-off estimation Max round-off error in x/(x + y) can be written as 52

53 Modeling max. round-off estimation Max round-off error in x/(x + y) can be written as Max over x, y, e 1,e 2,d 1,d 2 : f(x, y, e 1,e 2,d 1,d 2 ) f(x, y) 53

54 Unfortunately, this makes an already exponential problem worse Max over x, y, e 1,e 2,d 1,d 2 : f(x, y, e 1,e 2,d 1,d 2 ) f(x, y) We have to find the max of this function over Input variables Two additional variables per operator Curse of Dimensionality 54

55 Illustration of Curse of Dimensionality To analyze this example 55

56 Illustration of Curse of Dimensionality Optimization needs to run over 60 variables!! 56

57 Illustration of Curse of Dimensionality Optimization needs to run over 60 variables!! Two problem variables (x1 and x2) 58 noise variables (two per operator) 57

58 Illustration of Curse of Dimensionality Optimization needs to run over 60 variables!! Two problem variables (x1 and x2) 58 noise variables (two per operator) 58

59 Another problem with existing tools: Exaggerated error bounds 59

60 Another problem with existing tools: Exaggerated error bounds Consider something as simple as this: 60

61 Another problem with existing tools: Exaggerated error bounds Consider something as simple as this: 61

62 Another problem with existing tools: Exaggerated error bounds Consider something as simple as this: 62

63 Reason for problem: no correlation of variable values across sub-expressions 63

64 Reason for problem: no correlation of variable values across sub-expressions Say, we use interval arithmetic 64

65 Reason for problem: no correlation of variable values across sub-expressions Pick 999 for denominator x 65

66 Reason for problem: no correlation of variable values across sub-expressions Now switch things around (i.e. no correlation!!) 66

67 Symbolic Taylor Forms helps with both these problems Helps reduce dimensionality Helps reflect magnitudes of sub-expressions and also correlation of their values 67

68 What are Symbolic Taylor Forms? Basic idea : Treat each e and d as noise (error) variables Now expand (Taylor s theorem) Coefficients are Symbolic Coefficients weigh the noise correctly And are correlated Idealized Function First-order Error Estimated Second Order Error 68

69 Illustration of Symbolic Taylor Forms Example : 69

70 Illustration of Symbolic Taylor Forms Example : 70

71 Illustration of Symbolic Taylor Forms Example : 71

72 Illustration of Symbolic Taylor Forms Example : This is in Symbolic Taylor Form, i.e. FPTaylor generates this form very fast, using canned rules that are rigorous 72

73 Illustration of Symbolic Taylor Forms Example : Pristine (original) expression 73

74 Illustration of Symbolic Taylor Forms Example : Pristine (original) expression First Taylor (partial) derivatives w.r.t. e 1 and e 2 74

75 Illustration of Symbolic Taylor Forms Example : Pristine (original) expression First Taylor (partial) derivatives w.r.t. e 1 and e 2 All second (partial) derivatives w.r.t. e1 and e2 are here (plus the sub-normal error terms). 75

76 Illustration of Symbolic Taylor Forms Example : Round-off Error Estimate 76

77 Illustration of Symbolic Taylor Forms Example : Round-off Error Estimate Our global optimizer finds the max. of this function 77

78 FPTaylor Approach on jetengine After obtaining the Taylor forms We optimize just over TWO variables!! Not 60 variables!! 78

79 This is the payoff moment : Obtaining these rigorously verified error plots 79

80 FPTaylor result on jetengine Simpler optimization 80

81 FPTaylor result on jetengine Simpler optimization 81

82 Cool novelty of FPTaylor Based on rigorous global optimization Results are formally verified (so, fear not!) Can be parallelized! We are doing this now In a new tool called Gelpia FPTaylor release has a 100-line Ocaml global optimizer Still we are able to beat most existing tools, while offering better precision estimates 82

83 Symbolic Taylor Forms handle all functions (even Transcendental) the same Example : 83

84 Symbolic Taylor Forms handle all functions (even Transcendental) the same Example : This whole derivation works even if + is replaced by exp and / by log (say) 84

85 Symbolic Taylor Forms handle all functions (even Transcendental) the same Example : This whole derivation works even if + is replaced by exp and / by log (say) Symbolic Taylor Form derivation passes all the operators unopened to the Global Optimizer Global Optimizers can deal with many dirty functions (e.g. sin, exp, log ) rigorously 85

86 FPTaylor Results 86

87 FPTaylor Results 87

88 Non-monotonicity of accuracy 88

89 Non-monotonicity of accuracy 89

90 FPTaylor Results 90

91 Concluding Remarks We propose a new method for rigorous floating-point round-off error estimation Our method is embodied in new tool FPTaylor FPTaylor performs well (despite prototype global optimizer) and returns tighter bounds Bounds are within 2x of true errors Computed by running against MPFR 91

92 Future Work Faster global optimizer More benchmarks analyzed using above Characterization of libraries Potential uses : In reasoning tools (e.g., alongside Gappa) Say, in Frama-C or SMACK Ideal for designers of new FP libraries 92

93 Related Efforts Interval arithmetic Affine arithmetic SMT-based analysis Testing 93

94 Big Picture Floating-point has been the (nearly) centuryold approximate computing method People are going to skimp on precision even more Why move around all those dead bits Your software chain may throw surprises at any stage users beware! Invest in rigorous floating-point methods all along! 94

95 Extras 95

96 Example of Problem with Existing Tools 96

97 Example of Problem with Existing Tools 97

98 Example of Problem with Existing Tools But then change the settings for x in an un-correlated manner when calculating the overall error! (See the paper for details.) 98

99 Possible first principles Approach Start with the high level problem statement 99

100 Possible first principles Approach Start with the high level problem statement 100

101 Possible first principles Approach We can model the error in each operator as follows: Rounding Idealized (real number) operator 101

102 Possible first principles Approach We can model the error in each operator as follows: Rounding Idealized (real number) operator Mathematically proven identity for rounding 102

103 Possible first principles Approach Thus, lift the function to be analyzed as follows where given function additional parameters 103

104 Problem with such a direct approach Yikes! This max problem now has two extra variables per operator!! given function additional parameters 104

105 Direct Approach Results in Complex Global Optimization Problem Curse of dimensionality! With higher dimensionality, the exponential problem becomes IMPOSSIBLY HARD Key feature of our approach We drastically reduce dimensionality through the construction of Symbolic Taylor Forms Symbolic Taylor Forms serve two roles: By obtaining them, we reduce dimensionality They also help measure error proportional to magnitude of the FP expressions FPTaylor produces the maximum Absolute Error and Relative Error estimates on chosen input intervals 105

106 We avoid Curse of Dimensionality thru Symbolic Taylor Forms Basic idea : Treat each e and d as noise variables Expand given expression using Taylor s theorem Coefficients end up being Symbolic Self-adjust (to reflect magnitudes) As well as Correlate!! Idealized Function First-order Error Estimated Second Order Error 106