Mastering Multiprecision Arithmetic

Transcription

1 Mastering Multiprecision Arithmetic Norman Ramsey April 2018 Introduction Uncivilized programming languages provide crappy integer arithmetic. You better hope your results fit in a machine word if they don t, your program might silently produce wrong answers. You can do this easily enough using Impcore or µscheme. Civilized languages provide integer arithmetic that works on any integer up to the size you can fit in RAM. By this standard, Standard ML is not quite civilized: arbitrary-precision arithmetic is in the specification as module IntInf, but implementations don t have to provide it. Full Scheme, on the other hand, is supercivilized if you re interested in arithmetic, it s worth studying. Arithmetic plays a prominent role in the last two projects, for these reasons: Civilized arithmetic is a topic that every educated computer scientist should know about if only so that you can insist on it for your projects. Don t tolerate uncivilized arithmetic! Arithmetic brilliantly illustrates the relative strengths and weaknesses of the two major ways of abstracting over data: abstract data types versus objects. Basics A natural number is represented by the sum 0 i<n x i b i. b is called the base of natural numbers. Each x i is called a digit. It satisfies the invariant 0 x i < b. Hardware integer arithmetic universally uses base b = 2. But software arithmetic, which you will implement, typically uses a base b = 2 k, where k is chosen as large as possible, subject to the constraint that (b 1) 2 can be computed without overflow. Be cautious! Moscow ML provides only 31 bits of integer precision. To help you get started, it is reasonable to begin with val base = 10 Once your code is working, you ll want to replace 10 with a larger number. Performance matters. Larger b gives better performance. Choice of representation A natural number should be represented by a sequence of digits. But this still leaves you with interesting design choices: Do you prefer an array or a list? If you prefer an array, do you want a mutable array (type 'a array) or an immutable array (type 'a vector)? If you prefer a list, do you prefer to store the digits of a number in big-endian order (most significant digit first) or little-endian order (least significant digit first)? (If you prefer an array, the only order that makes sense is to store the least significant digit x 0 in slot 0, and in general, to store digit x i in slot i. Anything else would be confusing.) If you prefer an array, how will be you be sure it contains enough digits to hold the results? If you prefer a list, how will you make sure you produce a list with the right number of digits? Some implementations of arithmetic operations, especially multiplication, can leave leading zeroes in a representation. The number of leading zeroes must be controlled so that it doesn t grow without bound. Do you prefer a representation invariant that eliminates leading zeroes? If not, how will you track the number of leading zeroes in each representation? Will you store it in the representation itself, or will you compute it dynamically? The beauty of data abstraction is that the interface isolates these decisions, so you can easily change them. Representing individual digits You can represent a digit by a value of type int. But I recommend against it: a value of type int is too easily confused with an array index. Instead, I recommend that you define datatype digit = D of int and use digit list, digit array, or digit vector in your representation. You will have to do a little extra pattern matching, but in order to avoid confusing an index with the digit at that index, the extra pattern matching is well worth it. 1

2 Mutable representation? Really? provided p is not too large as above *) If you scrutinize the types and contracts of the functions in the NATURAL interface, you ll see that the nat abstraction is immutable: nothing in the interface allows a client to change the value of a natural number. So why would it be OK to choose a mutable representation, like a digit array? Simple: you re allowed to pick any representation you like, because the details are sealed behind the interface. As long as you don t make a silly mistake like allow x /+/ y to mutate x or y, no client program can tell that your representation is mutable. In industrial implementations of multiprecision numbers, mutable representations are quite popular: they minimize allocation and they use memory efficiently. It s fine to pick a mutable representation, as long as no client code can tell the difference. Tackling arithmetic in Standard ML If you study the NATURAL signature, you should observe that the first step in creating any natural number is always to call of_int, and the only way to get information out of a natural number is to call decimals. But you will have an easier time if you start with two auxiliary functions, which are closely related. If your representation is immutable, here are their specifications: val addint : nat * int -> nat (* addint (x, p) returns a natural number that represents x + p, provided the following condition holds: *) for every digit x_i in x, (p + x_i) can be computed without overflow val shiftadd : int * int * nat -> nat (* shiftadd (p, n, x) = p * power (base, n) + x, provided the same condition above holds *) These two functions are related by this algebraic law: shiftadd (p, 0, x) = addint (x, p) This law can be used in either direction: you can use shiftadd to implement addint, or you can use addint to implement shiftadd. Which direction is easier depends on your choice of representation. If your representation is mutable, the types and contracts will be slightly different: val addint : nat * int -> unit (* addint (x, p) overwrites x with x + p, provided p is not too large as above *) Implementing function of_int Once you have addint, you can use it to implement of_int. Depending on the mutability of your chosen representation, the details vary. Here s the immutable version: fun of_int n = (* immutable representation *) if n < 0 then raise Negative else let val zero =... something suitable... in addint (zero, n) end The mutable version is the same, except you call addint for side effect and then return zero. Implementing short division using arrays In short division, a multi-digit number is divided by a single digit. The algorithm is one you learned in primary school. The algorithm is discussed in an excerpt from the Spring 2017 edition of Programming Languages: Build, Prove, and Compare. (The excerpt is appended to this handout.) Also, a full implementation is presented in David R. Hanson s C Interfaces and Implementations, on pages 311 and 312 (the XP_quotient function). 1 What Ramsey calls a remainder r i is called carry by Hanson. Short division loops down from most significant digit to least significant digit. If you re dividing x by d, at each step i we have q i = (r in b + x i ) div d r out = (r in b + x i ) mod d This algorithm works for any d in the range 0 < d b. If you are using an array representation, you can write the loop using higher-order function Array.foldri. Try help "Array"; in Moscow ML. The accumulating parameter holds the remainder r in. The final remainder out is returned with the quotient. Implementing short division inductively If you are using a little-endian list representation, short division can work with the same inductive structure we use for addition, subtraction, and multiplication. Here s a mathematical sketch. We are given a natural number n and a one-digit divisor v. We want to find quotient q (a natural number) and remainder r (a machine integer in the range 0 r < v) such that val shiftadd : int * int * nat -> unit (* shiftadd (p, n, x) overwrites x with p * power (base, n) + x, n = q v + r. 1 If you took COMP 40 at Tufts, you may have this book. If not, it is part of the university s Safari subscription, so your access to it is already paid for. 2

3 We calculate quotient and remainder inductively, using the proof system for natural numbers base b. In the base case, n = 0, q = 0, and r = 0. In the inductive step, n = m b + d, where b is the base of natural numbers, and d is a digit in the range 0 d < b. To solve this case, we have to solve two smaller division problems: Find q and r such that m = q v+r and 0 r < v. This problem can be solved by a recursive call to the short divider. Find q 0 and r such that r b + d = q 0 v + r and 0 r < v. Provided (b 1) (v 1) can be represented as a machine integer, this problem can be solved by using div and mod on machine integers. We then combine the solutions to get q = q b + q 0, which is a natural number. We can prove this solution correct using an equational proof equating q v + r to m b + d, in five steps. This information can be expressed neatly using algebraic laws: 0 div v = 0 0 mod v = 0 (m b + d) div v = (m div v) b + ((m mod v) b + d) div v (m b + d) mod v = ((m mod v) b + d) mod v These laws must not be used directly for implementation. Why not? Because in a direct implementation, the inductive case calls both div and mod recursively. That s an exponential number of calls. For implementation, you need to compute both m mod v and m div b in a single call. Using short division for base conversion To convert a number from a large base to a smaller base, you continually short-divide by the small base until you run out of digits. Each short division produces a remainder, which is a digit of the result. The least-significant digit is produced first. Here s an example using repeated division by 2, converting decimal 11 to binary: 11 2 = 5 remainder = 2 remainder = 1 remainder = 0 remainder 1 Reading off the digits in order says that decimal 11 is binary There is a similar problem on the homework, where you complete this table: 13 2 = q 0 remainder r 0 q 0 2 = q 1 remainder r 1 q 1 2 = q 2 remainder r 2 q 2 2 = q 3 remainder r 3 The converted numeral is the sequence of digits r 3 r 2 r 1 r 0 (in traditional big-endian order). Implementing decimals When you have a large base, short division is the way you implement decimals. You have to get this right or your code won t pass any test cases. Here is part of a definition of decimals, using a mutable representation: fun reverse_decimals x = if... x is zero... then [] else let val { quotient = q, remainder = r } = divdestroy (x, 10) in r :: reverse_decimals q end Function divdestroy divides x by 10 and overwrites x. Code for an immutable representation is about the same. Implementing addition and subtraction If you are using an array to represent a sum or product, you will need to provide it with enough digits in advance: For addition, you may need as many digits as are in the larger addend, plus one more. For subtraction, you may need as many digits as are in the minuend (left-hand operand). Subtraction may leave you with a representation that contains leading zeroes. If leading zeroes violate your representation invariant, you ll have to do something about them. Both addition and subtraction work from the least significant digit to the most significant digit. Array users can use Array.foldli or Vector.foldli. The borrowing needed to subtract is slightly more fiddly than the carrying needed to add. I found it useful to define a helper function that subtracts a borrow bit from a single natural number. 3

4 Addition For addition, the accumulating parameter is a carry in. The key step is to produce a sum of digits from the two addends, plus the carry in. Part of that some becomes a digit of the result, and part becomes the carry out. Code might look something like this: let val s = x_i + y_i + carry_in val carry_out = s div base val sum_i = s mod base in... do something with sum_i and carry_out... end What happens in the... depends on your representation, but sum_i becomes a digit in the output, and carry_out becomes the carry_in on the next cycle. The final carry out should be zero. If not, you didn t allocate enough digits, and there is a bug in your code. Subtraction Subtraction uses the same looping structure as addition, but the computation is a little more complicated. If the difference goes negative, we have to borrow a from the next most significant digit (the borrow_out ). let val initial_diff = x_i - y_i - borrow_in val (borrow_out, diff_i) = if initial_diff < 0 then (1, initial_diff + base) else (0, initial_diff) in... do something with diff_i and borrow_out... end If your final borrow_out is nonzero, the result of subtraction is negative, and you need to raise the Negative exception. Implementing comparison The final operation you meet to implement compares two natural numbers and returns EQUAL, LESS, or GREATER. You can approach the problem the direct way or the indirect way. The direct approach looks at the digits. If x = x b + x 0 and y = y b + y 0 then you first compare the more significant digits x and y. If they are EQUAL, return Int.compare (x 0, y 0 ). Otherwise, x and y compare the same as x and y. In the indirect approach you implement < by using /-/ and seeing if it raises Negative. Then you use < to implement compare. The problem with the direct approach is that depending on your representation, comparing a long number with a short number could be tricky especially if your representation permits leading zeroes. The problem with the indirect approach is that it allocates at least one natural number, then throws it away. 2 Implementing multiplication Done carefully, multiplication is the easiest operation to implement. The equation for a product is x y = 0 i<n 0 j<m (x i y j ) b i+j. The number of digits needed to hold the double sum is potentially the total number of digits in x and y together: m + n. And the algorithm couldn t be simpler: it s a doubly nested loop in which each iteration adds the partial product (x i y j ) b i+j to a running total. Call shiftadd(x i y j, i + j, total). I recommend using Array.foldli or Vector.foldli on an array representation, and foldli or foldri on a list representation. (You will have to define foldli or foldri on lists.) The accumulating parameter passed to the fold is the running total or if you are using a mutable representation, the best accumulating parameter is the empty tuple (value () of type unit). 2 While I find the indirect approach intellectually unsatisfying, I can t rebut the argument that says "the garbage collector is good at reclaiming short-lived objects; shut up and let it do its job. 4

5 604 CHAPTER 8. CLU, ABSTRACT DATA TYPES, AND MODULES Sidebar: Notation: Multiplication, visible and invisible Mathematicians and physicists often multiply quantities simply by placing one next to another; for example, in the famous equation E = mc 2, m and c 2 are multiplied. But in a textbook on programming languages, this notational convention will not do. First, it is better for multiplication to be visible than to be invisible. And second, when one name is placed next to another, it usually means function application at least that s what it means in ML, Haskell, and the lambda calculus. Among the conventional infix operators, * is more suited to code than to mathematics, and the symbol is better reserved to denote a Cartesian product in a type system. In this book, on the rare occasions when we need to multiply numbers, I write an infix, so Einstein s famous equation would be written E = m c 2. Completing the implementation of int-set and measuring the effect of the optimized union operation is the subject of Exercise 7 on page Arbitrary-precision integer arithmetic Values of type int aren t true integers; they are machine integers. For much of the history of computing, a typical machine integer has been represented by a half word or full word of the hardware of the day. In the 21st century, that usually means 32 or 64 bits. Using 64 or even 32 bits is good enough for most purposes, but some computations need more precision; examples include some cryptographic computations as well as exact rational arithmetic. In many languages, arbitrary-precision integers are available as an abstraction, and in highly civilized languages like Scheme, Smalltalk, and Python, arbitrary precision is the default. You can implement this abstraction. In this chapter, I encourage you to implement it using abstract data types, and in Chapter 11, I encourage you to implement it again using objects. The similarities and differences illuminate what abstract data types are good at and what objects are good at. An implementation of arbitrary-precision arithmetic begins with natural numbers the nonnegative integers. And the standard representation is a sequence of digits indexed from zero. A natural number X is represented by a sequence of digits x 0,...,x n. The representation s meaning depends on the choice of a base, written b. In everyday life, we use base b = 10, and we write the most significant digit x n on the left. In hardware, our computers famously use base b = 2; the very word bit is a contraction of binary digit. Whatever b is, the digits x 0,...,x n stand for the number X = n x i b i. i=0 Moreover, the representation satisfies the invariant that for each i, 0 x i < b. Algorithms for addition, subtraction, multiplication, and division are independent of b. For COMP 105, Tufts University, Spring 2017 only --- do no

6 8.9. INSPECTING MULTIPLE REPRESENTATIONS 605 Addition and subtraction When we want to add an n-digit number X to an m-digit number Y, we compute ( n ( m ) X +Y = x i b )+ i y j b j = = i=0 max(m,n) i=0 max(m,n) i=0 j=0 x i b i +y i b i (x i +y i ) b i Unfortunately, the sum of two digits is not necessarily a digit: we might have x i +y i > b. For example, if b = 10, the sum 7+8 is not a digit. But we can express the sum if we use an additional carry bit: = There may be a carry in as well; here is an example in the style of elementary school: The superscript 1 over the 7 is the carry bit from adding 3 and 9. In the general case, we add X and Y as follows: at every nonnegative index i we add input digits x i and y i to the carry in c i, and we produce output digit z i and carry out c i+1, which satisfy the equation x i +y i +c i = z i +c i+1 b. Because for any nonnegative z, z = z mod b+(zdivb) b, the equation is satisfied by z i = (x i +y i +c i ) mod b c i+1 = (x i +y i +c i )divb The initial carry in c 0 is zero. In the specific case shown above, z 0 +c 1 b = x 0 +y 0 +c 0, where x 0 = 3,y 0 = 9,c 0 = 0,z 0 = 2,c 1 = 1 z 1 +c 2 b = x 1 +y 1 +c 1, where x 1 = 7,y 1 = 8,c 1 = 1,z 1 = 6,c 2 = 1 z 2 +c 3 b = x 2 +y 2 +c 2, where x 2 = 0,y 2 = 0,c 2 = 1,z 2 = 1,c 2 = 0 In Exercises 39 and 40, you implement the general case in µclu. Once all the input digits and carries are zero, you re done. Subtraction is like addition, except the carry bit is called a borrow, and digit z is computed from the difference x i y i c i. If this difference is negative, you must borrow b from a more significant digit, exploiting the identity z i+1 b i+1 +z i b i = (z i+1 1) b i+1 +(z i +b) b i. For COMP 105, Tufts University, Spring 2017 only --- do no

7 606 CHAPTER 8. CLU, ABSTRACT DATA TYPES, AND MODULES Multiplication To compute a product, we compute ( ) ( ) X Y = x i b i y j b j = i i (x i y j ) b i+j j Again, to satisfy the representation invariant, the partial product (x i y j ) b i+j has to be split into two digits ((x i y j ) mod b) b i+j + ((x i y j )divb) b i+j+1. Then all the partial products are added. Here s an example: j Short division Longdivision, whereyoudivideonebignumberbyanother, isbeyondthe scopeofthisbook. Consult Hanson (1996) or Brinch Hansen (1994). But short division, where you divide a big number by a digit, is within the scope of the book, and it is useful for implementing print. If X is a large natural number and d is a digit, we can compute quotient Q, which is large, and remainder r, which is small. The algorithm uses these variables: b Base of multiprecision arithmetic X Dividend d Divisor d < b Q Quotient r Remainder, always 0 r < d x i Digit i of dividend, 0 x i < b q i Digit i of quotient, 0 q i < b r i Remainder in at digit i, 0 r i < d Remainder out at digit i, 0 r i < d r i We are given b, d, and all the x i s, and we want to find r, all the q i s, and on the way, all the r i s and r i s. Here are the equations: X = i x i b i Q = i q i b i q i = (r i b+x i)divd r = r 0 r i = (r i b+x i ) mod d r i 1 = r i r n = 0 For COMP 105, Tufts University, Spring 2017 only --- do no

8 8.9. INSPECTING MULTIPLE REPRESENTATIONS 607 The basic law of short division, justified by the definitions of q i and r i, is q i d+r i = b r i +x i. Theq i sandr i sarecomputedfromthemostsignificanttotheleastsignificant. Forexample, 738 divided by 4 is 184, with remainder (remainder 2) The answer is computed by a loop that iterates three times: once to compute each pair of q i and r i. The steps of the computation may be summarized as follows: 1. X = d = 4 3. n = 2 4. r 2 = 0 5. x 2 = 7 6. q 2 = (0 10+7)div4 = 1 7. r 2 = (0 10+7) mod 4 = 3 8. x 1 = 3 9. q 1 = (3 10+3)div4 = r 1 = (3 10+3) mod 4 = x 0 = q 0 = (1 10+8)div4 = r 0 = (1 10+8) mod 4 = r = r 0 = 2 Implementing arithmetic using abstract data types When we re implementing numbers, from complex numbers to fractions to arbitraryprecision integers, the representation is hidden. Client code that uses natural numbers doesn t know that the representation is a sequence of digits interpreted with respect to a fixed base b. But when we use abstract data types to implement the abstraction, as in CLU, the implementation code knows the representation of every argument of abstract type, so adding, subtracting, and multiplying pairs of numbers is easy: the static type guarantees the representation we expect. Using abstract data types to implement arithmetic also has a cost: the static type system prevents numbers of different types from interoperating. For example, there is no way to subtract a machine integer from an arbitrary-precision natural number you must first coerce the machine integer to a natural number. This limitation makes such mixed arithmetic much less convenient than it is in an object-oriented language like Smalltalk (Chapter 11, Exercises 32 to 34) Priority queues optimized for merging As a final example of code that benefits from inspecting multiple representations in a single operation, we return to priority queues. The priority queue in Section on page 590 uses a representation that is often taught to undergraduate students: a complete binary tree represented by an array. This representation offers insertion and removal in O(log N) time, but it does not offer a merge operation; the only way to combine two heaps is to remove all the elements from one and add them to the other just like the naïve version of set union above. The underlying arrays aren t sorted, so there s no easy, efficient way to merge them. 8 In many languages, including C and C++, the compiler inserts coercions automatically between, say, integer types and floating-point types. But this privilege cannot be extended to user-defined types like arbitrary-precision integers. For COMP 105, Tufts University, Spring 2017 only --- do no