Applications of Sorting

Transcription

1 Applications of Sorting The key to understanding sorting is seeing how it can be used to solve many important programming tasks: Uniqueness Testing How can we test if the elements of a given collection of items S are all distinct? Deleting Duplicates How can we remove all but one copy of any repeated elements in S? Median/Selection How can we find the kth largest item in set S? Frequency Counting Which is the most frequently occurring element in S, i.e., the mode?

2 Reconstructing the Original Order How can we restore the original arrangement of a set of items after we permute them for some application? Set Intersection/Union How can we intersect or union the elements of two containers? Finding a Target Pair How can we test whether there are two integers x,y S such that x + y = z for some target z? Efficient Searching How can we efficiently test whether element s is in set S?

3 Basic Sorting Algorithms What are the time complexities and number of data moves for each algorithm? Selection Sort This algorithm splits the input array into sorted and unsorted parts, and with each iteration finds the smallest element remaining in the unsorted region and moves it to the end of the sorted region. Heapsort Selection sort with the right data structure! Bubble sort Percolate elements forward through adjacent element swaps. No element is guaranteed to be in the proper position until the end.

4 Insertion Sort This algorithm also maintains sorted and unsorted regions of the array. In each iteration, the next unsorted element moves up to its appropriate position in the sorted region. Quicksort This algorithm reduces the job of sorting one big array into the job of sorting two smaller arrays by performing a partition step. The partition separates the array into those elements that are less than the pivot/divider element, and those which are strictly greater than this pivot/divider element.

5 Multicriteria Sorting How can we break ties in sorting using multiple criteria, like sorting first on last name breaking ties on first name? One way is to use a complicated comparison function that combines all the keys to break ties: int suitor_compare(suitor *a, suitor *b) { int result; /* result of comparison */ if (a->height < b->height) return(-1); if (a->height > b->height) return(1); if (a->weight < b->weight) return(-1); if (a->weight > b->weight) return(1); if ((result=strcmp(a->last,b->last))!= 0) return result; return(strcmp(a->first,b->first));

6 Stable Sorting Another way is to use repeated passes through a stable sorting function, one which is guaranteed to leave identical keys in the same relative order they were in before the sorting. If so, we could sort by first name, and then do stable sort by last name and know the final results are sorted by both major and minor keys. STL provides a stable sorting routine, where keys of equal value are guaranteed to remain in the same relative order. void stable_sort(randomaccessiterator bg, RandomAccessIterator end) void stable_sort(randomaccessiterator bg, RandomAccessIterator end, BinaryPredicate op)

7 Sorting Library Functions in C The stdlib.h contains library functions for sorting and searching. For sorting, there is the function qsort: #include <stdlib.h> void qsort(void *base, size_t nel, size_t width, int (*compare) (const void *, const void *)); It sorts the first nel elements of an array (pointed to by base), where each element is width-bytes long. Thus we can sort arrays of 1-byte characters, 4-byte integers, or 100- byte records, all by changing the value of width. Binary search is also included: bsearch(key, (char *) a, cnt, sizeof(int), intcompare);

8 Comparison Functions The ultimate desired order is determined by the function compare. It takes as arguments pointers to two width-byte elements, and returns a negative number if the first belongs before the second in sorted order, a positive number if the second belongs before the first, or zero if they are the same.

9 Sorting and Searching in C++ The C++ Standard Template Library (STL) includes methods for sorting, searching, and more. Serious C++ users should get familiar with STL. To sort with STL, we can either use the default comparison (e.g., ) function defined for the class, or override it with a special-purpose comparison function op: void sort(randomaccessiterator bg, RandomAccessIterator end) void sort(randomaccessiterator bg, RandomAccessIterator end, BinaryPredicate op)

10 STL Sorting Applications Other STL functions implement some of the applications of sorting, including, nth element Return the nth largest item in the container. set union, set intersection, set difference Construct the union, intersection, and set difference of two containers. predicate unique Remove all consecutive duplicates.

11 Sorting and Searching in Java The java.util.arrays class contains various methods for sorting and searching. In particular, static void sort(object[] a) static void sort(object[] a, Comparator c) sorts the specified array of objects into ascending order using either the natural ordering of its elements or a specific comparator c. Stable sorts are also available. Methods for searching a sorted array for a specified object using either the natural comparison function or a new comparator c are also provided: binarysearch(object[] a, Object key) binarysearch(object[] a, Object key, Comparator c)

12 (Vito s Family) Find the most central location on a street, to minimize total distance to certain neighbors. Is the right version of average mean, median, or something else?

13 (Bridge) How should n people of variable speeds cross a two-man bridge at night when there is only one flashlight? Does sorting the people by speed help determine who should be paired up?

14 (Shoemaker s Problem) How should a shoemaker order the jobs he does so as to minimize total penalty costs for lateness, when different jobs have different penalties? Does it help to sort jobs by their length, or by their penalty, or both?

15 (CDVII) Compute the total amount of tolls people owe in a world where toll costs vary with time. How do we pair up entry and exit records?

16 How Long is Long? Today s PCs are typically 32-bit machines, so standard integer data types supports integers roughly in the range ±2 31 = ± 2,147,483,648. Thus we can safely count up to a billion or so with standard integers on conventional machines. We can get an extra bit by using unsigned integers. Most programming languages support long or even long long integer types, which define 64-bit or occasionally 128- bit integers = 9,223,372,036,854,775,808, so we are talking very large numbers!

17 Floating Point Precision The magnitude of numbers which can be represented as floats is astonishingly large, particularly with doubleprecision. This magnitude comes by representing the number in scientific notation, as a 2 c. Since a and c are both restricted to a given number of bits, there is still only a limited precision. Thus don t be fooled into thinking that floats give you the ability to count to very high numbers. Use integers and longs for such purposes.

18 Representing Enormous-Precision Integers Representing truly enormous integers requires stringing digits together. Two possible representations are Arrays of Digits The easiest representation for long integers is as an array of digits, where the initial element of the array represents the least significant digit. Maintaining a counter with the length of the number in digits can aid efficiency by minimizing operations which don t affect the outcome.

19 Linked Lists of Digits Dynamic structures are necessary if we are really going to do arbitrary precision arithmetic, i.e., if there is no upper bound on the length of the numbers. Note, however, that 100,000-digit integers are pretty long yet can be represented using arrays of only 100,000 bytes each. What dynamic memory really provides is the freedom to use space where you need it. If you wanted to create a large array of high-precision integers, most of which were small, then you would be better off with a list-of-digits representation.

20 Bignum Data Type Our bignum data type is represented as follows: #define MAXDIGITS 100 /* maximum length bignum */ #define PLUS 1 /* positive sign bit */ #define MINUS -1 /* negative sign bit */ typedef struct { char digits[maxdigits]; /* represent the number */ int signbit; /* PLUS or MINUS */ int lastdigit; /* index of high-order digit */ bignum;

21 Addition Adding two integers is done from right to left, with any overflow rippling to the next field as a carry. Allowing negative numbers turns addition into subtraction. add_bignum(bignum *a, bignum *b, bignum *c) { int carry; /* carry digit */ int i; /* counter */ initialize_bignum(c); if (a->signbit == b->signbit) c->signbit = a->signbit; else { if (a->signbit == MINUS) { a->signbit = PLUS; subtract_bignum(b,a,c); a->signbit = MINUS; else { b->signbit = PLUS; subtract_bignum(a,b,c); b->signbit = MINUS; return; c->lastdigit = max(a->lastdigit,b->lastdigit)+1; carry = 0; for (i=0; i<=(c->lastdigit); i++) { c->digits[i] = (char) (carry+a->digits[i]+b->digits[i]) % 10; carry = (carry + a->digits[i] + b->digits[i]) / 10; zero_justify(c);

22 Signbits Manipulating the signbit is a non-trivial headache. We reduced certain cases to subtraction by negating the numbers and/or permuting the order of the operators, but took care to replace the signs first. The actual addition is quite simple, and made simpler by initializing all the high-order digits to 0 and treating the final carry over as a special case of digit addition.

23 Zero Justif cation The zero justify operation adjusts lastdigit to avoid leading zeros. It is harmless to call after every operation, particularly as it corrects for 0: zero_justify(bignum *n) { while ((n->lastdigit > 0) && (n->digits[ n->lastdigit ]==0)) n->lastdigit --; if ((n->lastdigit == 0) && (n->digits[0] == 0)) n->signbit = PLUS; /* hack to avoid -0 */

24 Subtraction Subtraction is trickier than addition because it requires borrowing. To ensure that borrowing terminates, it is best to make sure that the larger-magnitude number is on top. subtract_bignum(bignum *a, bignum *b, bignum *c) { int borrow; /* anything borrowed? */ int v; /* placeholder digit */ int i; /* counter */ if ((a->signbit == MINUS) (b->signbit == MINUS)) { b->signbit = -1 * b->signbit; add_bignum(a,b,c); b->signbit = -1 * b->signbit; return; if (compare_bignum(a,b) == PLUS) { subtract_bignum(b,a,c); c->signbit = MINUS; return; c->lastdigit = max(a->lastdigit,b->lastdigit); borrow = 0; for (i=0; i<=(c->lastdigit); i++) { v = (a->digits[i] - borrow - b->digits[i]); if (a->digits[i] > 0) borrow = 0; if (v < 0) { v = v + 10; borrow = 1; c->digits[i] = (char) v % 10; zero_justify(c);

25 Comparison Deciding which of two numbers is larger requires a comparison operation. Comparison proceeds from highest-order digit to the right, starting with the sign bit: compare_bignum(bignum *a, bignum *b) { int i; /* counter */ if ((a->signbit==minus) && (b->signbit==plus)) return(plus); if ((a->signbit==plus) && (b->signbit==minus)) return(minus); if (b->lastdigit > a->lastdigit) return (PLUS * a->signbit); if (a->lastdigit > b->lastdigit) return (MINUS * a->signbit); for (i = a->lastdigit; i>=0; i--) { if (a->digits[i] > b->digits[i]) return(minus * a->signbit); if (b->digits[i] > a->digits[i]) return(plus * a->signbit); return(0);

26 Multiplication Multiplication seems like a more advanced operation than addition or subtraction. A people as sophisticated as the Romans had a difficult time multiplying, even though their numbers look impressive on building cornerstones and Super Bowls. The Roman s problem was that they did not use a radix (or base) number system. Certainly multiplication can be viewed as repeated addition and thus solved in that manner, but it will be hopelessly slow. Squaring 999,999 by repeated addition requires on the order of a million operations, but is easily doable by hand using the row-by-row method we learned in school.

27 Schoolhouse Multiplication multiply_bignum(bignum *a, bignum *b, bignum *c) { bignum row; /* represent shifted row */ bignum tmp; /* placeholder bignum */ int i,j; /* counters */ initialize_bignum(c); row = *a; for (i=0; i<=b->lastdigit; i++) { for (j=1; j<=b->digits[i]; j++) { add_bignum(c,&row,&tmp); *c = tmp; digit_shift(&row,1); c->signbit = a->signbit * b->signbit; zero_justify(c);

28 Each operation involves shifting the first number one more place to the right and then adding the shifted first number d times to the total, where d is the appropriate digit of the second number. We might have gotten fancier than using repeated addition, but since the loop cannot spin more than nine times per digit, any possible time savings will be relatively small.

29 Digit Shift Shifting a radix-number one place to the right is equivalent to multiplying it by the base of the radix, or 10 for decimal numbers: digit_shift(bignum *n, int d) /* multiply n by 10ˆd */ { int i; /* counter */ if ((n->lastdigit == 0) && (n->digits[0] == 0)) return; for (i=n->lastdigit; i>=0; i--) n->digits[i+d] = n->digits[i]; for (i=0; i<d; i++) n->digits[i] = 0; n->lastdigit = n->lastdigit + d;

30 Exponentiation Exponentiation is repeated multiplication, and hence subject to the same performance problems as repeated addition on large numbers. The trick is to observe that a n = a n 2 a n 2 a nmod2 so it can be done using only a logarithmic number of multiplications.

31 Division Although long division is an operation feared by schoolchildren and computer architects, it too can be handled with a simpler core loop than might be imagined. Division by repeated subtraction is again far too slow to work with large numbers, but the basic repeated loop of shifting the remainder to the left, including the next digit, and subtracting off instances of the divisor is far easier to program than guessing each quotient digit as we were taught in school:

32 Division Code divide_bignum(bignum *a, bignum *b, bignum *c) { bignum row; /* represent shifted row */ bignum tmp; /* placeholder bignum */ int asign, bsign; /* temporary signs */ int i,j; /* counters */ initialize_bignum(c); c->signbit = a->signbit * b->signbit; asign = a->signbit; bsign = b->signbit; a->signbit = PLUS; b->signbit = PLUS; initialize_bignum(&row); initialize_bignum(&tmp); c->lastdigit = a->lastdigit;

33 for (i=a->lastdigit; i>=0; i--) { digit_shift(&row,1); row.digits[0] = a->digits[i]; c->digits[i] = 0; while (compare_bignum(&row,b)!= PLUS) { c->digits[i] ++; subtract_bignum(&row,b,&tmp); row = tmp; zero_justify(c); a->signbit = asign; b->signbit = bsign;

34 Computing the Remainder This routine performs integer division and throws away the remainder. If you want to compute the remainder of a b, you can always do a b(a b).

35 Numerical Bases and Conversion The digit representation of a given radix-number is a function of which numerical base is used. Particularly interesting numerical bases include: Binary Base-2 numbers are made up of the digits 0 and 1. They provide the integer representation used within computers, because these digits map naturally to on/off or high/low states. Octal Base-8 numbers are useful as a shorthand to make it easier to read binary numbers, since the bits can be read off from the right in groups of three. Thus = = Why do programmers think Christmas is Halloween? Because 31 Oct = 25 Dec!

36 Decimal We use base-10 numbers because we learned to count on our ten fingers. Hexadecimal Base-16 numbers are an even easier shorthand to represent binary numbers, once you get over the fact that the digits representing 10 through 15 are A to F. Alphanumeric Occasionally, one sees even higher numerical bases. Base-36 numbers are the highest you can represent using the 10 numerical digits with the 26 letters of the alphabet. Any integer can be represented in base-x provided you can display X different symbols.

37 Base Conversion There are two distinct algorithms you can use to convert basea number x to a base-b number y Left to Right Here, we find the most-significant digit of y first. It is the integer d l such that (d l + 1)b k > x d l b k where 1 d l b 1. In principle, this can be found by trial and error, although you must to be able to compare the magnitude of numbers in different bases. This is analogous to the long-division algorithm described above.

38 Right to Left Here, we find the least-significant digit of y first. This is the remainder of x divided by b. Remainders are exactly what is computed when doing modular arithmetic. The cute thing is that we can compute the remainder of x on a digit-by-digit basis, making it easy to work with large integers. Right-to-left translation is similar to how we translated conventional integers to our bignum presentation. Taking the long integer mod 10 (using the % operator) enables us to peel off the low-order digit.

39 (Reverse and Add) Does repeatedly adding a number to its digit-reversal eventually end on a palindrome?

40 (The Archeologists Dilemma) What is the smallest power of 2 begining with the given digit sequence?

41 (Ones) How many digits is the smallest multiple of n such that the resulting digit sequence is all 1s? Why is this possible for every non-multiple of 2 and 5?

42 (A multiplication game) What is the right strategy for a two-person digit multiplication game? Is it recursive/minimax?

43 Learning to Count Combinatorics problems are notorious for their reliance on cleverness and insight. Once you look at the problem in the right way, the answer suddenly becomes obvious. Basic counting techniques include: Product Rule The product rule states that if there are A possibilities from set A and B possibilities from set B, then there are A B ways to combine one from A and one from B. Sum Rule The sum rule states that if there are A possibilities from set A and B possibilities from set B, then there are A + B ways for either A or B to occur assuming the elements of A and B are distinct.

44 Inclusion-Exclusion Formula The sum rule is a special case of a more general formula when the two sets can overlap, namely, A B = A + B A B The reason this works is that summing the sets double counts certain possibilities, namely, those occurring in both sets. Double counting is a slippery aspect of combinatorics, which can make it difficult to solve problems via inclusionexclusion.

45 Combinatorial Objects A bijection is a one-to-one mapping between the elements of one set and the elements of another. Counting the size of one of the sets automatically gives you the size of the other set. Exploiting bijections requires us to have a repertoire of sets which we know how to count, so we can map other objects to them. It is useful to have a feeling for how fast the number of objects grows, to know when exhaustive search breaks down as a possible technique:

46 Permutations A permutation is an arrangement of n items, where every item appears exactly once. There are n! = n i=1 i different permutations. The 3! = 6 permutations of three items are 123, 132, 213, 231, 312, and 321. For n = 10, n! = 3, 628, 800.

47 Subsets A subset is a selection of elements from n possible items. There are 2 n distinct subsets of n things. Thus there are 2 3 = 8 subsets of three items, namely, 1, 2, 3, 12, 13, 23, 123, and the empty set: never forget the empty set. For n = 20, 2 n = 1, 048, 576.

48 Strings A string is a sequence of items which are drawn with repetition. There are m n distinct sequences of n items drawn from m items. There are 27 length-3 strings on 123. The number of binary strings of length n is identical to the number of subsets of n items (why?)

49 Recurrence Relations Recurrence relations make it easy to count a variety of recursively defined structures. Recursively defined structures include trees, lists, well-formed formulae, and divide-andconquer algorithms so they lurk wherever computer scientists do. A recurrence relation is an equation which is defined in terms of itself, such as the Fibonacci numbers: F n = F n 1 + F n 2

50 Why Recurrence Relations? They are useful because many natural functions are easily expressed as recurrences: Polynomials: Exponentials: a n = a n 1 + 1, a 1 = 1 a n = n a n = 2a n 1,a 1 = 2 a n = 2 n Weird but interesting functions otherwise hard to represent: a n = na n 1, a 1 = 1 a n = n! It is often easy to find a recurrence as the answer to a counting problem.

51 Closed Form Solutions of Recurrences Solving the recurrence to get a nice closed form can be somewhat of an art, but as we shall see, computer programs can easily evaluate the value of a given recurrence even without the existence of a nice closed form.

52 Binomial Coeff cients The most important class of counting numbers are the binomial coefficients, where ( n k) counts the number of ways to choose k things out of n possibilities. What do they count? Committees How many ways are there to form a k- member committee from n people? By definition, ( n k) is the answer. Paths Across a Grid How many ways are there to travel from the upper-left corner of an n m grid to the lowerright corner by walking only down and to the right? Every path must consist of n + m steps, n downward and m to the right. Every path with a different set of downward moves is distinct, so there are ( n + m n ) such sets/paths.

53 Coefficients of (a + b) n Observe that (a + b) 3 = 1a 3 + 3a 2 b + 3ab 2 + 1b 3 What is the coefficient of the term a k b n k? Clearly ( n k), because it counts the number of ways we can choose the k a-terms out of n possibilities.

54 Computing Binomial Coeff cients Since ( n k) = n!/((n k)!k!), in principle you can compute them straight from factorials. However, intermediate calculations can easily cause arithmetic overflow even when the final coefficient fits comfortably within an integer. A more stable way to compute binomial coefficients is using the recurrence relation implicit in the construction of Pascal s triangle, namely, that ( n k) = ( n 1 k 1) + ( n 1 k )

55 Why Pascal? Consider whether the nth element appears in one of the ( n k) subsets of k elements. If so, we can complete the subset by picking k 1 other items from the other n 1. If not, we must pick all k items from the remaining n 1. There is no overlap between these cases, and all possibilities are included, so the sum counts all k-subsets. No recurrence is complete without basis cases. How many ways are there to choose 0 things from a set? Exactly one, the empty set. The right term of the sum drives us up to ( k k). How many ways are there to choose k things from a k-element set? Exactly one, the complete set.

56 Implementation The following implementation of this recurrence is a good example of programming techniques used in dynamic programming, namely storing a table of results so we can evaluate terms of the recurrence by looking up the appropriate values. #define MAXN 100 /* largest n or m */ long binomial_coefficient(n,m) int n,m; /* computer n choose m */ { int i,j; /* counters */ long bc[maxn][maxn]; /* table of binomial coefficients */ for (i=0; i<=n; i++) bc[i][0] = 1; for (j=0; j<=n; j++) bc[j][j] = 1; for (i=1; i<=n; i++) for (j=1; j<i; j++) bc[i][j] = bc[i-1][j-1] + bc[i-1][j]; return( bc[n][m] );

57 Other Counting Sequences Several other counting sequences which repeatedly emerge in applications, and which are easily computed using recurrence relations. Fibonacci numbers Defined by the recurrence F n = F n 1 + F n 2 and the initial values F 0 = 0 and F 1 = 1, they emerge repeatedly because this is perhaps the simplest interesting recurrence relation. The first several values are 0, 1, 1, 2, 3, 5, 8, 13, 21, 34, 55,...

58 Catalan Numbers The recurrence and associated closed form C n = n 1 C k C n 1 k = 1 n) k=0 n + 1 (2n defines the Catalan numbers, which occur in a surprising number of problems in combinatorics. The first several terms are 2, 5, 14, 42, 132, 429, 1430,... when C 0 = 1. How many ways are there to build a balanced formula from n sets of left and right parentheses? For example, there are five ways to do it for n = 3: ((())), ()(()), (())(), (()()), and ()()(). The leftmost parenthesis l matches some right parenthesis r, which must partition the formula into two balanced pieces, leading to the recurrence.

59 The exact same reasoning arises in counting the number of triangulations of a convex polygon, counting the number of rooted binary trees on n + 1 leaves, and counting the number of paths across a lattice which do not rise above the main diagonal.

60 Eulerian Numbers The Eulerian numbers n k count the number of permutations of length n with exactly k ascending sequences or runs. A recurrence can be formulated by considering each permutation p of 1,...,n 1. There are n places to insert element n, and each either splits an existing run of p or occurs immediately after the last element of an existing run, thus preserving the run count. Thus n k = k n 1 k + (n k + 1) n 1 k 1.

61 Integer Partitions An integer partition of n is an unordered set of positive integers which add up to n. For example, there are seven partitions of 5, namely, (5), (4, 1), (3, 2), (3, 1, 1), (2, 2, 1), (2, 1, 1, 1), and (1, 1, 1, 1, 1). The easiest way to count them is to define a function f(n,k) giving the number of integer partitions of n with largest part at most k. In any acceptable partition the largest part either does or does not reach with limit, so f(n,k) = f(n k,k) + f(n, k 1). The basis cases are f(1, 1) = 1 and f(n,k) = 0 whenever k > n.

62 (Counting) How many ways can we express n as the sum of 2s, 3s, and two types of 1?

63 (Expressions) How many ways can we build a well-formed formula from n parentheses with nesting depth d?

64 (The Priest Mathematician) What is the best way to solve a 4-peg Tower of Hanoi puzzle?

65 (Self-describing Sequence) What is the ith value of the sequence containing f(k) occurrences of k for each k, i.e. 1, 2, 2, 3, 3, 4,4, 4, 5, 5, 5, 6...? What if i is too large to construct the entire sequence in memory?

66 Number Theory and Divisibility G-d created the integers. All else is the work of man. Kronecker. Number theory is the study of the properties of the integers, specifically integer divisibility. It is a fascinating area is beautiful proofs and surprising results. We say b divides a (denoted b a) if a = bk for some integer k. Equivalently, we say that b is a divisor of a or a is a multiple of b if b a. As a consequence of this definition, the smallest natural divisor of every non-zero integer is 1. In general there is no integer k such that a = 0 k.

67 Prime Numbers A prime number is an integer p > 1 which is only divisible by 1 and itself. Said another way, if p is a prime number, then p = a b for integers a b implies that a = 1 and b = p. The fundamental theorem of arithmetic. states is that every integer can be expressed in only one way as the product of primes. 105 is uniquely expressed as 3 5 7, and 32 is uniquely expressed as This unique set of numbers multiplying to n is called the prime factorization of n. Any number which is not prime is said to be composite.

68 Primality Testing and Factorization Not only are there an infinite number of primes (see Euclid s proof), but there are a lot of them. There are roughly x/ lnx primes less than or equal to x, i.e. roughly one out of every ln x numbers is prime. The easiest way to find the prime factorization of an integer n is repeated division. Note the that smallest prime factor is must at most n unless n is prime.

69 Constructing all Divisors Every divisor is the product of some subset of these prime factors. Such subsets can be constructed using backtracking techniques, but we must be careful about duplicate prime factors. For example, the prime factorization of 12 has three terms (2, 2, and 3) but 12 has only 6 divisors (1, 2, 3, 4, 6, 12).

70 Greatest Common Divisor The greatest common divisor, or gcd, the largest divisor shared by a given pair of integers. The reduced form of the fraction 24/36 comes after we divide both the numerator and denominator by gcd(x,y), in this case 12. Euclid s GCD algorithm rests on two observations. First, If b a, then gcd(a,b) = b. This should be pretty clear. If b divides a, then a = bk for some integer k, and thus gcd(bk,b) = b. Second, If a = bt + r for integers t and r, then gcd(a,b) = gcd(b,r).

71 Why? By definition, gcd(a,b) = gcd(bt +r, b). Any common divisor of a and b must rest totally with r, because bt clearly must be divisible by any divisor of b. It can also find integers x and y such that a x + b y = gcd(a,b) which will prove quite useful in solving linear congruences.

72 Implementation /* Find the gcd(p,q) and x,y such that p*x + q*y = gcd(p,q) */ long gcd(long p, long q, long *x, long *y) { long x1,y1; /* previous coefficients */ long g; /* value of gcd(p,q) */ if (q > p) return(gcd(q,p,y,x)); if (q == 0) { *x = 1; *y = 0; return(p); g = gcd(q, p%q, &x1, &y1); *x = y1; *y = (x1 - floor(p/q)*y1); return(g); The least common multiple (lcm), the smallest integer which is divided by both of a given pair of integers. For example, the least common multiple of 24 and 36 is 72. To compute it, observe that lcm(x,y) = xy/gcd(x,y).

73 Modular Arithmetic Sometimes computing the remainder is more important than a quotient. What day of the week will your birthday fall on next year? All you need to know is the remainder of the number of days between now and then (either 365 or 366) when dividing by the 7 days of the week. Thus it will fall on this year s day plus one (365mod7) or two (366mod7) days, depending upon whether it is affected by a leap year. The key to such efficient computations is modular arithmetic. The number we are dividing by is called the modulus, and the remainder left over is called the residue.

74 What is (x + y)modn? We can simplify this to ((xmodn) + (ymodn))modn to avoid adding big numbers. Subtraction s just a special case of addition. (12mod100) (53mod100) = 41mod100 = 59mod100 Notice how we can convert a negative number mod n to a positive number by adding a multiple of n to it.

75 Exponentiation Since multiplication is just repeated addition, xymod n = (xmod n)(ymod n)mod n Since exponentiation is just repeated multiplication, x y modn = (xmodn) y modn Since exponentiation is the quickest way to produce really large integers, this is where modular arithmetic really proves its worth. Division proves considerably more complicated to deal with...

76 What is the Last Digit? What is the last digit of 2 100? We can do this computation by hand. What we really want to know is what mod10 is. By doing repeated squaring, and taking the remainder mod10 at each step we make progress very quickly: 2 3 mod10 = mod10 = 8 8mod mod10 = 4 4mod mod10 = 6 6mod mod10 = 6 6mod mod10 = 6 6mod mod10 = mod10 6

77 Congruences Congruences are an alternate notation for representing modular arithmetic. We say that a b(modm) if m (a b). By definition, if amodm is b, then a b(mod m). It gets us thinking about the set of integers with a given remainder n, and gives us equations for representing them. What integers x satisfy the congruence x 3(mod 9)? The set of solutions is all integers of the form 9y + 3, where y is any integer. What about 2x 3(mod 9) and 2x 3(mod 4)? Trial and error should convince you that exactly the integers of the form 9y + 6 satisfy the first example, while the second has no solutions at all.

78 Operations on Congruences Congruences support addition, subtraction, and multiplication, as well as a limited form of division provided they share the same modulus: Addition and Subtraction Suppose a b(mod n) and c d(mod n). Then a + c b + d(mod n). If 4x 7(mod 9) and 3x 3(mod 9), then 4x 3x 7 3(mod 9) x 4(mod 9) Multiplication General multiplication holds, i.e., a b(mod n) and c d(mod n) implies ac bd(mod n).

79 Division We cannot cavalierly cancel common factors from congruences. Note that (mod 3), but clearly 2 1(mod 3). Note that division can be defined as multiplication by an inverse, so a/b is equivalent to ab 1. But this inverse does not always exist try to find a solution to 2x 1(mod 4).

80 Simplifying and Solving Congruences We can simplify a congruence ad bd(mod dn) to a b(mod n), so we can divide all three terms by a mutually common factor if one exists. Thus (mod 140) implies that 17 3(mod 14). A linear congruence is an equation of the form ax b(mod n). Solving this equation means identifying which values of x satisfy it. Not all such equations have solutions. ax 1(mod n) has a solution if and only if the modulus and the multiplier are relatively prime, i.e., gcd(a, n) = 1. We may use Euclid s algorithm to find this inverse through the solution to a x + n y = gcd(a,n) = 1.

81 In general, there are three cases, depending on the relationship between a, b, and n: gcd(a,b,n) > 1 Then we can divide all three terms by this divisor to get an equivalent congruence. This gives us a single solution mod the new base, or equivalently gcd(a, b, n) solutions (mod n). gcd(a,n) does not divide b The congruence can have no solution. gcd(a,n) = 1 Then there is one solution (mod n). Further, x = a 1 b works, since aa 1 b b(mod n). This inverse exists and can be found using Euclid s algorithm.

82 More Advanced Tools The Chinese remainder theorem gives us a tool for working with systems of congruences over different moduli. Suppose there is exists an integer x such that x a 1 (mod m 1 ) and x a 2 (mod m 2 ). Then x is uniquely determined (mod m 1 m 2 ) if m 1 and m 2 are relatively prime.

83 Diophantine Equations Diophantine equations are formulae in which the variables are restricted to integers. For example, Fermat s last theorem concerned answers to the equation a n + b n = c n. The most important class are linear Diophantine equations of the form ax ny = b, where x and y are the integer variables and a, b, and n are integer constants. These are readily shown to be equivalent to the solving the congruence ax b(mod n) and hence can be solved using the techniques above.

84 (Carmichael Numbers) Which composite integers n always satisfy the equation a n modn = a Does this require large integer arithmetic?

85 (Factovisors) Does a divide n!? Does this require large integer arithmetic?

86 (Marbles) What is the cheapest way to perfectly store n among two types of boxes?

87 (Repackaging) Cups of three sizes are sold in certain predefined sets with a given number of each type of cup per package. Can we buy packages so we end up with exactly the same number of all three types of cups?

88 Backtracking Backtracking is a systematic method to iterate through all the possible configurations of a search space. It is a general algorithm/technique which must be customized for each individual application. In the general case, we will model our solution as a vector a = (a 1,a 2,...,a n ), where each element a i is selected from a finite ordered set S i. Such a vector might represent an arrangement where a i contains the ith element of the permutation. Or the vector might represent a given subset S, where a i is true if and only if the ith element of the universe is in S.

89 At each step in the backtracking algorithm, we start from a given partial solution, say, a = (a 1, a 2,...,a k ), and try to extend it by adding another element at the end. After extending it, we must test whether what we have so far is a solution. If not, we must then check whether the partial solution is still potentially extendible to some complete solution. If so, recur and continue. If not, we delete the last element from a and try another possibility for that position, if one exists.

90 Implementation We include a global finished flag to allow for premature termination, which could be set in any application-specific routine. bool finished = FALSE; /* found all solutions yet? */ backtrack(int a[], int k, data input) { int c[maxcandidates]; /* candidates for next position */ int ncandidates; /* next position candidate count */ int i; /* counter */ if (is_a_solution(a,k,input)) process_solution(a,k,input); else { k = k+1; construct_candidates(a,k,input,c,&ncandidates); for (i=0; i<ncandidates; i++) { a[k] = c[i]; backtrack(a,k,input); if (finished) return; /* terminate early */

91 Application-Specif c Routines The application-specific parts of this algorithm consists of three subroutines: is a solution(a,k,input) This Boolean function tests whether the first k elements of vector a are a complete solution for the given problem. The last argument, input, allows us to pass general information into the routine. construct candidates(a,k,input,c,ncandida This routine fills an array c with the complete set of possible candidates for the kth position of a, given the contents of the first k 1 positions. The

92 number of candidates returned in this array is denoted by ncandidates. process solution(a,k) This routine prints, counts, or somehow processes a complete solution once it is constructed. Backtracking ensures correctness by enumerating all possibilities. It ensures efficiency by never visiting a state more than once. Because a new candidates array c is allocated with each recursive procedure call, the subsets of not-yet-considered extension candidates at each position will not interfere with each other.

93 Constructing All Subsets We can construct the 2 n subsets of n items by iterating through all possible 2 n length-n vectors of true or false, letting the ith element denote whether item i is or is not in the subset. Using the notation of the general backtrack algorithm, S k = (true, f alse), and a is a solution whenever k n.

94 is_a_solution(int a[], int k, int n) { return (k == n); /* is k == n? */ construct_candidates(int a[], int k, int n, int c[], int *ncandidates) { c[0] = TRUE; c[1] = FALSE; *ncandidates = 2; process_solution(int a[], int k) { int i; /* counter */ printf("{"); for (i=1; i<=k; i++) if (a[i] == TRUE) printf(" %d",i); printf(" \n"); Finally, we must instantiate the call to backtrack with the right arguments. generate_subsets(int n) { int a[nmax]; /* solution vector */ backtrack(a,0,n);

95 Constructing All Permutations To avoid repeating permutation elements, we must ensure that the ith element is distinct from all elements before it. To use the notation of the general backtrack algorithm, S k = {1,...,n a, and a is a solution whenever k = n: construct_candidates(int a[], int k, int n, int c[], int *ncandidates) { int i; /* counter */ bool in_perm[nmax]; /* who is in the permutation? */ for (i=1; i<nmax; i++) in_perm[i] = FALSE; for (i=0; i<k; i++) in_perm[ a[i] ] = TRUE; *ncandidates = 0; for (i=1; i<=n; i++) if (in_perm[i] == FALSE) { c[ *ncandidates] = i; *ncandidates = *ncandidates + 1;

96 Completing the job of generating permutations requires specifying process solution and is a solution, as well as setting the appropriate arguments to backtrack. All are essentially the same as for subsets: process_solution(int a[], int k) { int i; /* counter */ for (i=1; i<=k; i++) printf(" %d",a[i]); printf("\n"); is_a_solution(int a[], int k, int n) { return (k == n); generate_permutations(int n) { int a[nmax]; /* solution vector */ backtrack(a,0,n);

97 The Eight-Queens Problem The eight queens problem is a classical puzzle of positioning eight queens on an 8 8 chessboard such that no two queens threaten each other. Implementing a backtrack search requires us to think carefully about the most concise, efficient way to represent our solutions as a vector. What is a reasonable representation for an n-queens solution, and how big must it be? To make a backtracking program efficient enough to solve interesting problems, we must prune the search space by terminating every search path the instant it becomes clear it cannot lead to a solution.

98 Since no two queens can occupy the same column, we know that the n columns of a complete solution must form a permutation of n. By avoiding repetitive elements, we reduce our search space to just 8! = 40,320 clearly short work for any reasonably fast machine. The critical routine is the candidate constructor. We repeatedly check whether the kth square on the given row is threatened by any previously positioned queen. If so, we move on, but if not we include it as a possible candidate:

99 Implmementation construct_candidates(int a[], int k, int n, int c[], int *ncandidates) { int i,j; /* counters */ bool legal_move; /* might the move be legal? */ *ncandidates = 0; for (i=1; i<=n; i++) { legal_move = TRUE; for (j=1; j<k; j++) { if (abs((k)-j) == abs(i-a[j])) /* diagonal threat */ legal_move = FALSE; if (i == a[j]) /* column threat */ legal_move = FALSE; if (legal_move == TRUE) { c[*ncandidates] = i; *ncandidates = *ncandidates + 1;

100 The remaining routines are simple, particularly since we are only interested in counting the solutions, not displaying them: process_solution(int a[], int k) { int i; /* counter */ solution_count ++; is_a_solution(int a[], int k, int n) { return (k == n);

101 Finding the Queens The main program is as follows: nqueens(int n) { int a[nmax]; /* solution vector */ solution_count = 0; backtrack(a,0,n); printf("n=%d solution_count=%d\n",n,solution_count); and yields the following answers: n=1 solution_count=1 n=2 solution_count=0 n=3 solution_count=0 n=4 solution_count=2 n=5 solution_count=10 n=6 solution_count=4 n=7 solution_count=40 n=8 solution_count=92 n=9 solution_count=352 n=10 solution_count=724 n=11 solution_count=2680 n=12 solution_count=14200 n=13 solution_count=73712 n=14 solution_count=365596

102 (Little Bishops) How many ways can we put down k mutally non-attacking bishops on an n n board? Can we count the bishops without explicitly constructing every configuration?

103 (15-Puzzle Problem) Can you find a minimum- or near-minimum length path to solve the 15-puzzle? How do we prune quickly, and how do we eliminate duplicates?

104 (Garden of Eden) Given a cellular automata state t and a transition rule, does there exist a previous state s such that s goes to t?

105 (Colour Hash) Does there exist a sequence of moves to reorder the pieces of this puzzle? What is the right representation of the puzzle?

106 Graphs Graphs are one of the unifying themes of computer science. A graph G = (V, E) is defined by a set of vertices V, and a set of edges E consisting of ordered or unordered pairs of vertices from V. In modeling a road network, the vertices may represent the cities or junctions, certain pairs of which are connected by roads/edges. In analyzing human interactions, the vertices typically represent people, with edges connecting pairs of related souls.

107 Flavors of Graphs The first step in any graph problem is determining which flavor of graph you are dealing with: Undirected vs. Directed A graph G = (V, E) is undirected if edge (x,y) E implies that (y, x) is also in E. Weighted vs. Unweighted In weighted graphs, each edge (or vertex) of G is assigned a numerical value, or weight.

108 Cyclic vs. Acyclic An acyclic graph does not contain any cycles. Trees are connected acyclic undirected graphs. Directed acyclic graphs are called DAGs. Implicit vs. Explicit Many graphs are not explicitly constructed and then traversed, but built as we use them. A good example is in backtrack search.

109 Data Structures for Graphs There are several possible ways to represent graphs. graph G = (V, E) contains n vertices and m edges. Say Adjacency Matrix We can represent G using an n n matrix M, where element M[i,j] is, say, 1, if (i,j) is an edge of G, and 0 if it isn t. It may use excessive space for graphs with many vertices and relatively few edges, however. Adjacency Lists in Lists We can more efficiently represent sparse graphs by using linked lists to store the neighbors adjacent to each vertex. Adjacency lists require pointers but are not frightening.

110 Adjacency Lists in Matrices Adjacency lists can also embedded in matrices, thus eliminating the need for pointers. We can represent a list in an array counting how many elements there are, and packing them into the first elements of the array. Now we can visit successive the elements from the first to last by incrementing an index in a loop instead of cruising through pointers. This data structure looks like it combines the worst properties of adjacency matrices (large space) with the worst properties of adjacency lists (the need to search for edges). However, it is simple to program and understand and what I will use in my examples.

111 List in Array Representation For each graph, we keep count of the number of vertices, and assign each vertex a unique number from 1 to nvertices. The edges go in an MAXV MAXDEGREE array, so each vertex can be adjacent to MAXDEGREE others. Defining MAXDEGREE to be MAXV can be wasteful of space for lowdegree graphs: #define MAXV 100 /* maximum number of vertices */ #define MAXDEGREE 50 /* maximum vertex outdegree */ typedef struct { int edges[maxv+1][maxdegree]; /* adjacency info */ int degree[maxv+1]; /* outdegree of each vertex */ int nvertices; /* number of vertices in graph */ int nedges; /* number of edges in graph */ graph;

112 Directed and Undirected Edges We represent a directed edge (x, y) by the integer y in x s adjacency list, which is located in the subarray graph->edges[x]. The degree field counts the number of meaningful entries for the given vertex. An undirected edge (x, y) appears twice in any adjacencybased graph structure, once as y in x s list, and once as x in y s list.

113 Reading a Graph A typical graph format consists of an initial line featuring the number of vertices and edges in the graph, followed by a listing of the edges at one vertex pair per line. read_graph(graph *g, bool directed) { int i; /* counter */ int m; /* number of edges */ int x, y; /* vertices in edge (x,y) */ initialize_graph(g); scanf("%d %d",&(g->nvertices),&m); for (i=1; i<=m; i++) { scanf("%d %d",&x,&y); insert_edge(g,x,y,directed); initialize_graph(graph *g) { int i; /* counter */ g -> nvertices = 0; g -> nedges = 0; for (i=1; i<=maxv; i++) g->degree[i] = 0;

114 Inserting an Edge The critical routine is insert edge. We parameterize it with a Boolean flag directed to identify whether we need to insert two copies of each edge or only one. Note the use of recursion to solve the problem: insert_edge(graph *g, int x, int y, bool directed) { if (g->degree[x] > MAXDEGREE) printf("warning: insertion(%d,%d) exceeds max degree\n",x,y); g->edges[x][g->degree[x]] = y; g->degree[x] ++; if (directed == FALSE) insert_edge(g,y,x,true); else g->nedges ++;

115 Breadth-First Traversal The basic operation in most graph algorithms is completely and systematically traversing the graph. We want to visit every vertex and every edge exactly once in some welldefined order. Breadth-first search is appropriate if we are interested in shortest paths on unweighted graphs.

116 Discovered vs. Processed Vertices Our implementation bfs uses two Boolean arrays to maintain our knowledge about each vertex in the graph. A vertex is discovered the first time we visit it. A vertex is considered processed after we have traversed all outgoing edges from it. Thus each vertex passes from undiscovered to discovered to processed over the search. Once a vertex is discovered, it is placed on a queue. Since we process these vertices in first-in, first-out order, the oldest vertices are expanded first, which are exactly those closest to the root:

117 Initializing Search bool processed[maxv]; /* which vertices have been processed */ bool discovered[maxv]; /* which vertices have been found */ int parent[maxv]; /* discovery relation */ initialize_search(graph *g) { int i; /* counter */ for (i=1; i<=g->nvertices; i++) { processed[i] = discovered[i] = FALSE; parent[i] = -1;

118 BFS Implementation bfs(graph *g, int start) { queue q; /* queue of vertices to visit */ int v; /* current vertex */ int i; /* counter */ init_queue(&q); enqueue(&q,start); discovered[start] = TRUE; while (empty(&q) == FALSE) { v = dequeue(&q); process_vertex(v); processed[v] = TRUE; for (i=0; i<g->degree[v]; i++) if (valid_edge(g->edges[v][i]) == TRUE) { if (discovered[g->edges[v][i]] == FALSE) { enqueue(&q,g->edges[v][i]); discovered[g->edges[v][i]] = TRUE; parent[g->edges[v][i]] = v; if (processed[g->edges[v][i]] == FALSE) process_edge(v,g->edges[v][i]);

119 Exploiting Traversal The exact behavior of bfs depends upon the functions process vertex() and process edge(). Through these functions, we can easily customize what the traversal does as it makes one official visit to each edge and each vertex. By setting the functions to process_vertex(int v) { printf("processed vertex %d\n",v); process_edge(int x, int y) { printf("processed edge (%d,%d)\n",x,y); we print each vertex and edge exactly once.

120 Finding Paths The vertex which discovered vertex i is defined as parent[i]. The parent relation defines a tree of discovery with the initial search node as the root of the tree. The unique BFS tree path from the root to any node x V uses the smallest number of edges (or equivalently, intermediate nodes) possible on any root-to-x path in the graph. We can reconstruct this path by following the chain of ancestors backwards from x to the root. We cannot find the path from the root to x, since that does not follow the direction of the parent pointers.

121 Since this is the reverse of how we normally want the path, we can either (1) store it and then explicitly reverse it using a stack, or (2) let recursion reverse it for us, as in the following slick routine: find_path(int start, int end, int parents[]) { if ((start == end) (end == -1)) printf("\n%d",start); else { find_path(start,parents[end],parents); printf(" %d",end);

122 Depth-First Search Depth-first search uses essentially the same idea as backtracking. Both involve exhaustively searching all possibilities, and backing up as soon as there is no unexplored possibility for further advancement. Both are best understood as recursive algorithms. Depth-first search can be thought of as breadth-first search with a stack instead of a queue. The beauty of implementing dfs recursively is that recursion eliminates the need to keep an explicit stack: