Efficient Search for Inputs Causing High Floating-point Errors

Transcription

1 Efficient Search for Inputs Causing High Floating-point Errors Wei-Fan Chiang, Ganesh Gopalakrishnan, Zvonimir Rakamarić, and Alexey Solovyev University of Utah Presented by Yuting Chen February 22, 2015

2 BACKGROUND: WHY DO WE HAVE FLOATING-POINT ERRORS Slides are from Computer Arithmetic: Algorithms and Hardware Designs by Behrooz Parhami

3 The IEEE Floating-Point Standard Short (32-bit) format IEEE Standard (supersedes IEEE ) 8 bits, bias = 127, 126 to 127 Sign Exponent 23 bits for fractional part (plus hidden 1 in integer part) Significand Also includes half- & quad-word binary, plus some decimal formats 11 bits, bias = 1023, 1022 to bits for fractional part (plus hidden 1 in integer part) Long (64-bit) format

4 Floating-Point Number Distribution Negative numbers Positive numbers max FLP min min + FLP max Sparser Denser Denser Sparser Overflow region Underflow regions Overflow region Midway example Underflow example Typical example Overflow example

5 Sources of Computational Errors FLP approximates exact computation with real numbers Two sources of errors to understand and counteract: Representation errors e.g., no machine representation for 1/3, 2, or p Arithmetic errors e.g., ( ) 2 = not representable in IEEE 754 short format

6 Example Showing Representation and Arithmetic Errors Compute 1/99 1/100, using a decimal floating-point format with 4-digit significand in [1, 10) and single-digit signed exponent Precise result = 1/ (error 10 8 or 0.01%) Chopped to 3 decimals x = 1/ Error 10 6 or 0.01% y = 1/100 = Error = 0 z = x fp y = = Error 10 6 or 1%

7 Forward Error Analysis Consider the computation y = ax + b and its floating-point version y fp = (a fp fp x fp ) + fp b fp = (1 + h)y Can we establish any useful bound on the magnitude of the relative error h, given the relative errors in the input operands a fp, b fp, x fp? The answer is no Forward error analysis = Finding out how far y fp can be from ax + b, or at least from a fp x fp + b fp, in the worst case

8 Backward Error Analysis Backward error analysis replaces the original question How much does y fp = a fp fp x fp + b fp deviate from y? with another question: What input changes produce the same deviation? In other words, if the exact identity y fp = a alt x alt + b alt holds for alternate parameter values a alt, b alt, and x alt, we ask how far a alt, b alt, x alt can be from a fp, x fp, x fp Thus, computation errors are converted or compared to additional input errors

9 Efficient Search for Inputs Causing High Floating-point Errors Wei-Fan Chiang, Ganesh Gopalakrishnan, Zvonimir Rakamarić, and Alexey Solovyev School of Computing, University of Utah, Salt Lake City, UT Supported in part by NSF grants ACI , CCF , CCF and CCF

10 Floating-point Computations in Sequential and Parallel Software Important applications such as weather prediction are accuracy-critical Everyday applications (e.g. cell-phone apps) run at lower FP precision Challenge : Knowing whether they give imprecise results for any input Photo courtesy to drroyspencer.com, aptito.com/blog, and itunes.apple.com. 1

11 Dangers of Inadequate or Inconsistent Precision Patriot Missile Failure in Miscalculated distance due to floating-point error. Inconsistent FP Calculations [Meng et al, XSEDE 13] P = C = Compute: floor( P / C ) Xeon Expecting 161 msgs Sent 162 msgs Xeon Phi P / C = floor( P / C ) = 161 P / C = 162 floor( P / C ) = 162 2

12 Problem Addressed How to tell which inputs maximize error? This is important for many reasons: Characterize libraries precisely Support tuning precision Help decide where error-compensation is productive Relative Error Feasible Inputs 3

13 Difficulties Large code-sizes Presence of non-linear operators Presence of data-dependent conditionals Concurrency (schedules may affect results) Relative Error Feasible Inputs 4

14 Main Contribution A practical technique for reliable precision estimation for sequential and parallel programs. Search based input generation. Handles diverse operations. Improves scalability. Usage scenarios: Precision bottleneck detection. Auto-tuning. 5

15 Previous Work Over-approximation based (false alarms likely) : Interval arithmetic: Examples x in [-1, 2] and y in [2, 5]. Then (x * y) returned as [-5, 10]. x in [-1, 1]. Then (x x) returned as [-2, 2] (must be 0) Affine arithmetic: Basic idea SMT Each number is represented by a polynomial. Linear approximation of non-linear operation. Encodes error bound described in IEEE-754 standard. Under-approximation based (no false alarms): Random testing. 6

16 Illustration of Interval Arithmetic 1. float x0, x1, x2 in [1.0, 2.0] 2. float p0 = (x0 + x1) x2 3. float p1 = (x1 + x2) x0 4. float p2 = (x2 + x0) x1 5. float sum = (p0 + p1) + p2 6. Error? sum // (x0 + x1) + x2 Value of sum Error on sum Exact Interval Arithmetic (Gappa) Affine Arithmetic (SmartFloat) SMT based [3.0, 6.0] [0.0, 9.0] [3.0, 6.0] [3.0, 6.0]? Infinite e e-15 7

17 Illustration of Affine Arithmetic / SMT 1. float xi in [1.0, 3.0] // 0 i 7 2. float sum = summation of xi 3. Consider xi in [1.0, 2.0] 4. Error? sum Value of sum Error on sum Exact Interval Arithmetic (Gappa) Affine Arithmetic (SmartFloat) SMT based [8.0, 16.0] [8.0, 16.0] N/A [8.0, 16.0]? e-16 N/A Timeout 8

18 Over-approximation: Previous Work Poor scalability Overly pessimistic results Limited support for non-linear operation Limited support for conditionals Interval Arithmetic Affine Arithmetic SMT based Our overall approach: Under-approximation based Naïve Random Testing produces VERY LOOSE lower bounds Our focus : How to produce tight lower-bounds? 9

19 Why do we base our approach on Guided Random Testing? Seems to be the only approach that can handle Large Programs Non-linear operators Data dependent conditionals No closed form solutions are possible At present, designers have no tools that can analyze programs with these features Ours is the first practical tool in this area 10

20 Precision Measurement by Random Testing Low Precision Program Low Precision Result configuration X0 X1 X2 Error Calculation* High Precision Program High Precision Result * Error = Relative Error (See paper for details) 11

21 Search Based Random Testing Our Contribution : Random Testing with Good Guidance Heuristics can Outperform Naïve Random We propose Binary Guided Random Testing 0 Pure Random Search Based Random Real Max. Error? Over-approximation 12

22 Search Based Random Testing Randomly sample inputs around sour-spots! A sour-spot causes highly imprecise program output. Definition of Configuration: An assignment from input variables to their probing intervals. Configuration: X X X2 Program Result 13

23 Search Based Random Testing Randomly sample inputs around sour-spots! A sour-spot causes highly imprecise program output. Definition of Configuration: An assignment from input variables to their probing intervals. Configuration: X X X2 X0 = 0.5 X1 = 1.5 X2 = 3.0 Program Imprecise Result 14

24 Search Based Random Testing Randomly sample inputs around sour-spots! A sour-spot causes highly imprecise program output. Definition of Configuration: An assignment from input variables to their probing intervals. New Configuration: X X X2 Program Result for New Config. 15

25 Importance of Selecting Good Configurations Good Conf. x0 x1 Original Conf. x0 x1 Bad Conf. x0 x1 Number of Samples 16

26 Binary Guided Random Testing: Search and Test Around Sour-spots Key Observations: Sour spots can be improved with more probing Configurations can be ranked without too much probing The optimization problem: Find a configuration that contains inputs causing high floating-point errors. We propose Binary Guided Random Testing (BGRT). We compared BGRT against other search methods, obtaining encouraging results 17

27 Original Conf. Init High-level View of BGRT Derive Configuration to Generate Candidates sub-... subconf. 1 conf. n Candidates Program 18

28 Original Conf. Init High-level View of BGRT Derive Configuration to Generate Candidates sub-... subconf. 1 conf. n Candidates Choose the BEST Sub-conf. Evaluate Program 19

29 Original Conf. Init High-level View of BGRT Derive Configuration to Generate Candidates sub-... subconf. 1 conf. n Candidates For each sub-conf., sample few inputs. Also Record the detected highest error. Choose the BEST Sub-conf. Evaluate Program 20

30 High-level View of BGRT Original Conf. Init Derive Configuration to Generate Candidates sub-conf. k The BEST among candidates sub-... subconf. 1 conf. n Candidates Choose the BEST Sub-conf. Evaluate Program For each sub-conf., sample few inputs. 21 Also Record the detected highest error.

31 High-level View of BGRT Original Conf. Init Derive Configuration to Generate Candidates Restart? sub-conf. k The BEST among candidates sub-... subconf. 1 conf. n Candidates Choose the BEST Sub-conf. Evaluate Program For each sub-conf., sample few inputs. 22 Also Record the detected highest error.

32 High-level View of BGRT Original Conf. Init Derive Configuration to Generate Candidates Restart? OR sub-conf. k The BEST among candidates sub-... subconf. 1 conf. n Candidates Choose the BEST Sub-conf. Evaluate Program For each sub-conf., sample few inputs. 23 Also Record the detected highest error.

33 X1 A Closer View of BGRT X0 X1 X2 X0 X2 Partition the variables (with their ranges). 24

34 A Closer View of BGRT X0 X1 X2 X1 X0 X2 X1 X1 Shrink variables ranges. X0 X2 X0 X2 Each partition generates its upper and lower subpartitions. 25

35 A Closer View of BGRT X0 X1 X2 X1 X0 X2 X1 X1 X0 X2 X0 X2 X0 X1 X2 X0 X1 X2 X0 X1 X2 X0 X1 X2 26

36 A Closer View of BGRT X0 X1 X2 Candidates These candidates are evaluated using random sampled inputs. X0 X1 X2 X0 X1 X2 X0 X1 X2 X0 X1 X2 27

37 Other Search Strategies We Investigated Iterated Local Search (ILS) Particle Swarm Optimization (PSO) Our results suggest BGRT as the better search strategy for precision measurement. Focuses the search near sour-spots. Website for additional documents: 28

38 Experimental Results Comparison among search strategies Unguided Random Testing (URT), BGRT, ILS, and PSO Benchmarks Various reduction-tree shapes Direct Quadrature Method of Moments (DQMOM) GPU primitives 29

39 Evaluation of BGRT (Reductions) Imbalanced reduction (IBR) Balanced reduction (BR) Compensated imbalanced reduction (IBRK) Over-approximation techniques cannot report that the compensated reduction is the most precise. Balanced Reduction ((v0 + v1) + (v2 + v3)) Imbalanced Reduction (((v0 + v1) + v2) + v3) (v0 + v1) (v2 + v3) ((v0 + v1) + v2) v3 v0 v1 v2 v3 (v0 + v1) v2 v0 v1 30

40 Evaluation of BGRT (Reductions) 2048 input variables Exp1 and Exp2 share all the same experiment settings except the seed for random number generation. 31

41 A Real-world Sequential Benchmark Direct Quadrature Method of Moments (DQMOM): A sequential core function of a combustion simulation component of Uintah parallel computational framework. (960 variables) 32

42 Evaluation of BGRT (GPU Primitives) Fast Fourier Transform (FFT) from Parboil LU decomposition from MAGMA library QR decomposition from MAGMA library Matrix multiplication (MM) from MAGMA library 33

43 Evaluation of BGRT (GPU Primitives) Input size: FFT: LU, QR: MM:

44 Challenges and Future Work Improvements: Coverage Scalability Search strategy improvement Applications: Combine with auto-tuning Combine with precision bottleneck detection Algorithm comparison 35

45 Conclusions Guided random testing can detect higher errors than pure random testing. Guided random testing overcomes some drawbacks of previous approaches: Improves scalability Handles diverse (e.g. non-linear) operations Supports precision bottleneck detection and auto-tuning Our project website 36

46 A Comparison Among BGRT, Genetic Algorithm, and Delta Debugging BGRT v.s. Genetic algorithm BGRT doesn t have mutation. BGRT only selects one of the best among current candidates to generate next candidates. BGRT v.s. Delta debugging BGRT could restart the search from the initial conf. Each conf. represents a set of inputs instead of a single input. 38