GPU & Computer Arithmetics

Transcription

1 GPU & Computer Arithmetics David Defour University of Perpignan

2 Key multicore challenges Performance challenge How to scale from 1 to 1000 cores The number of cores is the new MegaHertz Power efficiency challenge Performance per watt is the new metric Programming challenge How to program efficiently an increasing numbers of cores Precision challenge What is the meaning of a result produced after millions of operations

3 Reliability Data corruption from cosmic rays bit errors / GB / month ~bianca/papers/ sigmetrics09.pdf ECC RAM up to the register level on FERMI Fault tolerant algorithm; Checkpoint

4 «Computer lets you make more mistakes faster than any invention in human history- with the possible exceptions of handguns and tequila» Mitch Ratcliffe

5 Results meaningless? IEEE standard FMA, Binary16, Binary32, Binary64 Accumulation of rounding error WCEA of an accumulator on a Cypress architecture (2.5 TFLOPS) All bits of accuracy lost after Binary64: 2 54 op. or 2h Binary32: 2 25 op. or 13 μs

6 Menu Appetizer: A few words about FP standards Starter: Numerical accuracy & related problems Main course: What is inside a GPU Desert: Interval arithmetics

7 Background Computer arithmetics Reliability, accuracy, efficiency Existence of a standard for usual floating-point operations (+, -,, /, ) Advantages Preservation of mathematical properties Interval arithmetic Possibility to build proof Programs with deterministic behavior (portability) The same program gives the same results independently of the configuration (processor + compiler + system)

8 IEEE-754 standard Defines: Representation format Interchange format Rounding algorithms Operations behavior Exception handling & Recommendations ( elementary functions, expression evaluation,...) History: 1985: Radix-2 FP arithmetic 1987: Radix-Independent FP arithmetic 2008: Latest

9 Formats Representation format (-1) s m 2 e Sign s Name Common name Base Digits E min E max binary16 Half precision binary32 Single precision binary64 Double precision Mantissa m binary128 Quadruple precision decimal Exponent e decimal decimal Format: s e m = 32 bits Binary32 (-1) s 1.m 2 e Nb max: 3, Nb min : 1, s e m = 64 bits Binary64 (-1) s 1.m 2 e Nb max: 1, Nb min : 2,

10 Special Values Denormal numbers Split the interval between 0 and the smallest normalized number Extreme values Denormalized number Normalized number +/- to handle overflow +/- 0 Error handling Nan (Not A Number)

11 Correct rounding Rounding The result of a floating-point operation might not be exactly representable example: 1/3= It has to be rounded to the nearest toward + Representable numbers 0 + toward - toward -inf

12 Multiply- Accumulate R = A + (B. C) 2 possibles implementations: 2 rounding = MAD (common in DSP) B.C B C 1 rounding = FMA or FMAC + A Speed-up and improve the accuracy of Dot product R (MAD) R (FMA) Matrix multiplication Polynomial evaluation Newton s method

13 Menu Appetizer: A few words about FP standards Starter: Numerical accuracy & related problems Main course: What is inside a GPU Desert: Interval arithmetics

14 Surprises with FP Operations are commutative but not associative/distributive anymore A+B = B+A A+(B+C) (A+B)+C A=1; B= ; C=10-26 ; (A+B).C (A.C) + (B.C) A=1; B= ; C=10-26 ; Can t represent exactly 0.1 in binary32/ Testing for equality is problematic Limited exponent range

15 Limited exponent range Solution 1: Mathematical library (MPFR, SCSLIB,...) Solution 2: Scaling Introduce tests to determine if scaling is necessary Solution 3: Logarithmic Number System The value x is represented with log 2 (x) log 2 (a.b)= log 2 (a)+log 2 (b) log 2 (a+b) =?

16 Cancellation Symptom: Subtraction of nearly equal operands A = ; B = ; C = ; A.B = ; (A.B)-C = e-4; correct result = e-4; Consequences: Most of the a accuracy is lost What can we do: Be aware and careful

17 Menu Appetizer: A few words about FP standards Starter: Numerical accuracy & related problems Main course: What is inside a GPU Desert: Interval arithmetics

18

19

20

21

22

23

24

25

26

27

28 The big steps 2006 : G70, ATI R500 Permissive Floating-Point arithmetic Truncated multiplier Adder with 2 guard bits and no sticky bit int (14.0/7.0)=1 2007: DirectX10, G80, R600, GMA X3100 GPGPU API Rounding to the nearest (no denormal numbers) ALU 2008: RV670, RV770, GT200 FP64 Standardization for GPGPU API (OpenCL, Direct3D 11) S. Collange, M. Daumas, D. Defour. État de l'intégration de la virgule flottante dans les processeurs graphiques. RSTI TSI 27/2008, p

29 FP OXO Format FMA UMA Rounding Denormal Inf, NaN Flags Exceptions Intel X86 80 bits 4 Dynamics Microcode IBM PowerPC 64 bits 4 Dyn. Microcode Intel IA bits 4 Dyn. + Stat. Microcode IBM Cell SPU 32 bits RZ 64 bits 4 Dyn. Output NVIDIA GT bits 2 Static 64 bits 4 Static NVIDIA GF bits 4 Static 64 bits 4 Static AMD RV bits RN 64 bits RN AMD Evergreen 32 bits 4 Dyn.? 64 bits 4 Dyn.? OpenCL 32 bits Opt N/A RN Opt 64 bits 4 Static Direct3D bits RN 64 bits RN Source: S. Collange

30 Elementary functions evaluations In hardware: Since 1999 (DirectX 7.0) 1/x, 1/sqrt(x), log2, 2 x, sin, cos accuracy : 10 bits -> ~24 bits Based on tabulated values and interpolators In software Since GPGPU API More accurate (for big arguments, better relative error)

31 Menu Appetizer: A few words about FP standards Starter: Numerical accuracy & related problems Main course: What is inside a GPU Desert: Interval arithmetics

32 Interval arithmetics Take into account rounding errors Provides bounded and reliable results Represents value as a range of possibilities Example of applications Ray tracing of implicit surfaces Search of roots using Newton method Polygonal Approximation Ray tracing, not guaranteed Ray tracing, guaranteed S. Collange, J. Flórez, D. Defour. A GPU interval library based on Boost.Interval. 8th Conference on Real Numbers and Computers (RNC8) 2008.

33 Interval arithmetic Redefinition of basic operators +, -, x, /, on intervals [1, 5] + [-2, 3] = [-1, 8] May return a wider interval due to FP rounding errors Loss of variable dependency Lead to an unwanted expansion of the resulting intervals example: x 2 +x with x in [-1, 1] with IA : [ 1,1] 2 + [ 1,1] = [0,1] + [ 1,1] = [ 1,2] in real : [-0.25, 2]

34 implementation on GPU IA on CPU Switch cases depending on sign Changes in rounding mode IA on GPU Should avoid branches G80: rounding to the nearest, toward 0, direct rounding of IA emulated with rounded toward ulp GT200: 4 rounding modes for FP64 FERMI : 4 rounding modes for FP32 & FP64 (free changes of rounding mode)

35 BOOST-IA on GPU Cg Version with GPU-specific routines (outdated) CUDA version in C++ (adapted from Boost) Classes, Templates, Overload Officially supported since CUDA 3.0

36 results on synthetic benchmark a b + c Performance (Giter/s) Peak perf/ perf (ops/iter) Core i7 default (1 core) Core i7 unprote cted Tesla C1060 GTX 480 Core i7 default (1 core) Core i7 unprote cted Tesla C1060 GTX 480 FP FP

37 Some readings Numerical recipes in C W. H. Press, S. A. Teukolsky, W. T. Vetterling, B. P. Flannery 1992 Cambridge University Press ISBN : Accuracy and Stability of Numerical Algorithms Nicholas J. Higham 2002 SIAM ISBN :