Floats are not Reals. BIL 220 Introduc6on to Systems Programming Spring Instructors: Aykut & Erkut Erdem

Transcription

1 Floats are not Reals BIL 220 Introduc6on to Systems Programming Spring 2012 Instructors: Aykut & Erkut Erdem Acknowledgement: The course slides are adapted from the slides prepared by R.E. Bryant, D.R. O Hallaron, G. Kesden and Markus Püschel of Carnegie- Mellon Univ.

2 Today: Floa8ng Point Background: Frac6onal binary numbers IEEE floa6ng point standard: Defini6on Example and proper6es Rounding, addi6on, mul6plica6on Floa6ng point in C Summary 2

3 Frac8onal binary numbers What is ? 3

4 Frac8onal Binary Numbers 2 i 2 i bi bi- 1 b2 b1 b0 b- 1 b- 2 b- 3 b- j 1/2 1/4 1/8 Representa6on 2 - j Bits to right of binary point represent frac6onal powers of 2 Represents ra6onal number: 4

5 Frac8onal Binary Numbers: Examples Value Representa6on 5 3/ / / Observa6ons Divide by 2 by shiding right Mul6ply by 2 by shiding led Numbers of form are just below 1.0 1/2 + 1/4 + 1/ /2 i Use nota6on 1.0 ε 5

6 Representable Numbers Limita6on Can only exactly represent numbers of the form x/2 k Other ra6onal numbers have repea6ng bit representa6ons Value Representa6on 1/ [01] 2 1/ [0011] 2 1/ [0011] 2 6

7 Today: Floa8ng Point Background: Frac6onal binary numbers IEEE floa6ng point standard: Defini6on Example and proper6es Rounding, addi6on, mul6plica6on Floa6ng point in C Summary 7

8 IEEE Floa8ng Point IEEE Standard 754 Established in 1985 as uniform standard for floa6ng point arithme6c Before that, many idiosyncra6c formats Supported by all major CPUs Driven by numerical concerns Nice standards for rounding, overflow, underflow Hard to make fast in hardware Numerical analysts predominated over hardware designers in defining standard 8

9 Floa8ng Point Representa8on Numerical Form: ( 1) s M 2 E Sign bit s determines whether number is nega6ve or posi6ve Significand M normally a frac6onal value in range [1.0,2.0). Exponent E weights value by power of two Encoding MSB s is sign bit s exp field encodes E (but is not equal to E) frac field encodes M (but is not equal to M) s exp frac 9

10 Precisions Single precision: 32 bits s exp frac 1 8-bits 23-bits Double precision: 64 bits s exp frac 1 11-bits 52-bits Extended precision: 80 bits (Intel only) s exp frac 1 15-bits 63 or 64-bits 10

11 Normalized Values Condi6on: exp and exp Exponent coded as biased value: E = Exp Bias Exp: unsigned value exp Bias = 2 k- 1-1, where k is number of exponent bits Single precision: 127 (Exp: 1 254, E: ) Double precision: 1023 (Exp: , E: ) Significand coded with implied leading 1: M = 1.xxx x2 xxx x: bits of frac Minimum when (M = 1.0) Maximum when (M = 2.0 ε) Get extra leading bit for free 11

12 Normalized Encoding Example Value: Float F = ; = = x 2 13 Significand M = frac= Exponent E = 13 Bias = 127 Exp = 140 = Result: s exp frac 12

13 Denormalized Values Condi6on: exp = Exponent value: E = Bias + 1 (instead of E = 0 Bias) Significand coded with implied leading 0: M = 0.xxx x2 xxx x: bits of frac Cases exp = 000 0, frac = Represents zero value Note dis6nct values: +0 and 0 (why?) exp = 000 0, frac Numbers very close to 0.0 Lose precision as get smaller Equispaced 13

14 Special Values Condi6on: exp = Case: exp = 111 1, frac = Represents value (infinity) Opera6on that overflows Both posi6ve and nega6ve E.g., 1.0/0.0 = 1.0/ 0.0 = +, 1.0/ 0.0 = Case: exp = 111 1, frac Not- a- Number (NaN) Represents case when no numeric value can be determined E.g., sqrt( 1),, 0 14

15 Visualiza8on: Floa8ng Point Encodings Normalized Denorm +Denorm +Normalized + NaN NaN 15

16 Today: Floa8ng Point Background: Frac6onal binary numbers IEEE floa6ng point standard: Defini6on Example and proper6es Rounding, addi6on, mul6plica6on Floa6ng point in C Summary 16

17 Tiny Floa8ng Point Example s exp frac 1 4-bits 3-bits 8- bit Floa6ng Point Representa6on the sign bit is in the most significant bit the next four bits are the exponent, with a bias of 7 the last three bits are the frac Same general form as IEEE Format normalized, denormalized representa6on of 0, NaN, infinity 17

18 Crea8ng Floa8ng Point Number Steps Normalize to have leading 1 Round to fit within frac6on Postnormalize to deal with effects of rounding s exp frac 1 4-bits 3-bits Case Study Convert 8- bit unsigned numbers to 6ny floa6ng point format Example Numbers

19 Normalize Requirement Set binary point so that numbers of form 1.xxxxx Adjust all to have leading one Decrement exponent as shid led Value Binary Frac6on Exponent s exp frac 1 4-bits 3-bits 19

20 Dynamic Range (Posi8ve Only) s exp frac E Value Denormalized numbers Normalized numbers /8*1/64 = 1/ /8*1/64 = 2/ /8*1/64 = 6/ /8*1/64 = 7/ /8*1/64 = 8/ /8*1/64 = 9/ /8*1/2 = 14/ /8*1/2 = 15/ /8*1 = /8*1 = 9/ /8*1 = 10/ /8*128 = /8*128 = n/a inf closest to zero largest denorm smallest norm closest to 1 below closest to 1 above largest norm 20

21 Distribu8on of Values 6- bit IEEE- like format e = 3 exponent bits f = 2 frac6on bits Bias is = 3 s exp frac 1 3-bits 2-bits No6ce how the distribu6on gets denser toward zero. 8 values Denormalized Normalized Infinity 21

22 Distribu8on of Values (close- up view) 6- bit IEEE- like format e = 3 exponent bits f = 2 frac6on bits Bias is 3 s exp frac 1 3-bits 2-bits Denormalized Normalized Infinity 22

23 Interes8ng Numbers {single,double} Descrip.on exp frac Numeric Value Zero Smallest Pos. Denorm {23,52} x 2 {126,1022} Single 1.4 x Double 4.9 x Largest Denormalized (1.0 ε) x 2 {126,1022} Single 1.18 x Double 2.2 x Smallest Pos. Normalized x 2 {126,1022} Just larger than largest denormalized One Largest Normalized (2.0 ε) x 2 {127,1023} Single 3.4 x Double 1.8 x

24 Special Proper8es of Encoding FP Zero Same as Integer Zero All bits = 0 Can (Almost) Use Unsigned Integer Comparison Must first compare sign bits Must consider 0 = 0 NaNs problema6c Will be greater than any other values What should comparison yield? Otherwise OK Denorm vs. normalized Normalized vs. infinity 24

25 Today: Floa8ng Point Background: Frac6onal binary numbers IEEE floa6ng point standard: Defini6on Example and proper6es Rounding, addi6on, mul6plica6on Floa6ng point in C Summary 25

26 Floa8ng Point Opera8ons: Basic Idea x +f y = Round(x + y) x f y = Round(x y) Basic idea First compute exact result Make it fit into desired precision Possibly overflow if exponent too large Possibly round to fit into frac 26

27 Rounding Rounding Modes (illustrate with $ rounding) $1.40 $1.60 $1.50 $2.50 $1.50 Towards zero $1 $1 $1 $2 $1 Round down ( ) $1 $1 $1 $2 $2 Round up (+ ) $2 $2 $2 $3 $1 Nearest Even (default) $1 $2 $2 $2 $2 What are the advantages of the modes? 27

28 Closer Look at Round- To- Even Default Rounding Mode Hard to get any other kind without dropping into assembly All others are sta6s6cally biased Sum of set of posi6ve numbers will consistently be over- or under- es6mated Applying to Other Decimal Places / Bit Posi6ons When exactly halfway between two possible values Round so that least significant digit is even E.g., round to nearest hundredth (Less than half way) (Greater than half way) (Half way round up) (Half way round down) 28

29 Rounding Binary Numbers Binary Frac6onal Numbers Even when least significant bit is 0 Half way when bits to right of rounding posi6on = Examples Round to nearest 1/4 (2 bits right of binary point) Value Binary Rounded Ac6on Rounded Value 2 3/ (<1/2 down) 2 2 3/ (>1/2 up) 2 1/4 2 7/ ( 1/2 up) 3 2 5/ ( 1/2 down) 2 1/2 29

30 Rounding 1.BBGRXXX Guard bit: LSB of result Round bit: 1 st bit removed S6cky bit: OR of remaining bits Round up condi6ons Round = 1, S6cky = 1 > 0.5 Guard = 1, Round = 1, S6cky = 0 Round to even Value Frac6on GRS Incr? Rounded N N N Y Y Y

31 Postnormalize Issue Rounding may have caused overflow Handle by shiding right once & incremen6ng exponent Value Rounded Exp Adjusted Result /

32 FP Mul8plica8on ( 1) s1 M1 2 E1 x ( 1) s2 M2 2 E2 Exact Result: ( 1) s M 2 E Sign s: s1 ^ s2 Significand M: M1 x M2 Exponent E: E1 + E2 Fixing If M 2, shid M right, increment E If E out of range, overflow Round M to fit frac precision Implementa6on Biggest chore is mul6plying significands 32

33 Floa8ng Point Addi8on ( 1) s1 M1 2 E1 + (- 1) s2 M2 2 E2 Assume E1 > E2 Exact Result: ( 1) s M 2 E Sign s, significand M: Result of signed align & add Exponent E: E1 + ( 1) s1 M1 ( 1) s M E1 E2 ( 1) s2 M2 Fixing If M 2, shid M right, increment E if M < 1, shid M led k posi6ons, decrement E by k Overflow if E out of range Round M to fit frac precision 33

34 Mathema8cal Proper8es of FP Add Compare to those of Abelian Group Closed under addi6on But may generate infinity or NaN Commuta6ve NOT Associa6ve Overflow and inexactness of rounding 0 is addi6ve iden6ty Every element has addi6ve inverse Except for infini6es & NaNs Monotonicity a b a+c b+c? Except for infini6es & NaNs Yes Yes No Yes Almost Almost 34

35 Mathema8cal Proper8es of FP Mult Compare to Commuta6ve Ring Closed under mul6plica6on But may generate infinity or NaN Mul6plica6on Commuta6ve Mul6plica6on is NOT Associa6ve Possibility of overflow, inexactness of rounding 1 is mul6plica6ve iden6ty Mul6plica6on does not distribute over addi6on Possibility of overflow, inexactness of rounding Yes Yes No Yes No Monotonicity a b & c 0 a * c b *c? Except for infini6es & NaNs Almost 35

36 Today: Floa8ng Point Background: Frac6onal binary numbers IEEE floa6ng point standard: Defini6on Example and proper6es Rounding, addi6on, mul6plica6on Floa6ng point in C Summary 36

37 Floa8ng Point in C C Guarantees Two Levels float single precision double double precision Conversions/Cas6ng Cas6ng between int, float, and double changes bit representa6on double/float int Truncates frac6onal part Like rounding toward zero Not defined when out of range or NaN: Generally sets to TMin int double Exact conversion, as long as int has 53 bit word size int float Will round according to rounding mode 37

38 Ariane 5: A Bug and A Crash On June 4, 1996, Ariane 5 rocket self destructed just ader 37 seconds ader lidoff Cost: $500 million Cause: An overflow in the conversion from a 64 bit floa6ng point number to a 16 bit signed integer A design flaw: 5 6mes faster than Ariane 4 Reused same sodware specifica6ons from Ariane 4 Ariane 4 assumes horizontal velocity would never overflow a 16- bit number Fourmy/REA/SABA/Corbis 38

39 Floa8ng Point Puzzles For each of the following C expressions, either: Argue that it is true for all argument values Explain why not true x == (int)(float) x int x = ; float f = ; double d = ; Assume neither d nor f is NaN x == (int)(double) x f == (float)(double) f d == (float) d f == -(-f); 2/3 == 2/3.0 d < 0.0 ((d*2) < 0.0) True (OF? d > f -f > -d d * d >= 0.0 (d+f)-d == f False True True False True False True True (OF? False 39

40 Today: Floa8ng Point Background: Frac6onal binary numbers IEEE floa6ng point standard: Defini6on Example and proper6es Rounding, addi6on, mul6plica6on Floa6ng point in C Summary 40

41 Summary IEEE Floa6ng Point has clear mathema6cal proper6es Represents numbers of form M x 2 E One can reason about opera6ons independent of implementa6on As if computed with perfect precision and then rounded Not the same as real arithme6c Violates associa6vity/distribu6vity Makes life difficult for compilers & serious numerical applica6ons programmers 41

42 Reading Assignment What Every Computer Scientist Should Know About Floating-Point Arithmetic DAVID GOLDBERG Xerox Palo Alto Research Center, 3333 Coyote Hill Road, Palo Alto, CalLfornLa Floating-point arithmetic is considered an esotoric subject by many people. This is rather surprising, because floating-point is ubiquitous in computer systems: Almost every language has a floating-point datatype; computers from PCs to supercomputers have floating-point accelerators; most compilers will be called upon to compile floating-point algorithms from time to time; and virtually every operating system must respond to floating-point exceptions such as overflow This paper presents a tutorial on the aspects of floating-point that have a direct impact on designers of computer systems. It begins with background on floating-point representation and rounding error, continues with a discussion of the IEEE floating-point standard, and concludes with examples of how computer system builders can better support floating point, ACM Compu6ng Surveys, Vol. 23, No 1, March