Goal of the course: The goal is to learn to design and analyze an algorithm. More specifically, you will learn:

Transcription

1 CS341 Algorithms 1. Introduction Goal of the course: The goal is to learn to design and analyze an algorithm. More specifically, you will learn: Well-known algorithms; Skills to analyze the correctness and complexities of an algorithm; Skills to adapt an existing algorithm to solve new problems; Skills to design new algorithms; In prerequisite courses, you have already learned some algorithms, such as sorting, string matching. Also data structures are associated with algorithms. This course is a more systematic study of algorithm analysis and design techniques. We will make use of data structures. However, the focus is not about data structures. Problems and Algorithms Roughly speaking, an algorithm is a defined and finite procedure that solves a problem: the algorithm takes any input of the problem, and produces the desired output. Some notes about a problem, A problem defines the format of input and desired property of the output. For the purpose of this course, input size is not bounded by any constants. A problem does not specify which algorithm to use. There is an important distinction between a problem and an instance of a problem. For example, multiplication of two decimal integers of m and n digits is a problem. Its input are two numbers a and b, and output is the product of the two numbers c = a b. However, an instance is a specific input. For example, is an instance of the integer multiplication problem. Algorithms must: have a finite description (only finitely many lines of code) finish in finite time (some exceptions to this rule, such as servers or operating systems). be deterministic (although randomized algorithms are also studied these days) work on all instances of the problem. There may be different algorithms that compute the same thing. E.g., ways to compute multiplication: Algorithm 1: The standard algorithm (learned in elementary school) for long multiplication. (What s the time complexity when inputs are two m and n-digits numbers?) Algorithm 2: Add a by b times. (What s the time complexity now?) In the first algorithm, the most time-consuming part is the multiplication of each digit of a with each digit of b. This takes O(mn). For the second algorithm, we need to repeat b times of the additions of m digits. So the time complexity is about O(m b). Note that b = Θ(10 n ). Clearly, the first algorithm is way better.

2 This seems to be a trivial comparison and you may think, Nobody will use the second algorithm even if we do not study this course. But later in this course we will study a more efficient integer multiplication algorithm that runs in O(n ) for two n-digit integers. This demonstrates that it is useful to separate the problem definition from the algorithm design. In particular, do not describe a problem s output as the output of a specific algorithm. Such separation allows the maximum freedom to discover new algorithms that can carry out the same computation more efficiently. Model of Computation In comparing the two algorithms for multiplication, we made a few important and intuitive assumptions in order to make the comparison possible. Unit cost. We assumed arithmetic operation on a decimal digit takes a unit time. Asymptotic comparison. We care about the complexity with large input size, and neglected the small size inputs (for example, b = 1). Also because of this, we only focused on the most costly steps of the algorithms. Worst case. We do not care if an algorithm may run faster on any specific inputs: e.g. multiplication by We want a guaranteed performance on all instances. To formalize these intuitions, we make the following assumptions on our model of computation. The basic unit of data the computer deals with is a word, which consists of several (many) bits. The input size of an instance of a problem is counted in the number of words. Each basic operation on a word takes a unit of time. The basic operations include: Random access (Indexing into an array and read and write) Allocating one word of storage Arithmetic operations (+, -, *, /, %) Comparison Logical operations (and, or, xor, not, etc.) Bit operation (shift a word by a number of bits, bit-wise logical operations) The model with the above assumptions is called the word-ram model. It roughly reflects the computers we use today, but is much simpler. Such simplification is needed to make algorithm analysis possible. It allows us to focus on the essence of the algorithm s complexity (the asymptotic growth of running time), without worrying much of the implementation details (such as whether the processor computes multiplications faster than additions, or whether the data is in the main memory or the L1 cache of the processor). In the word-ram model, the word size deserves some special discussion. One needs to be very careful about the word size to make the model realistic. For almost all problems/algorithms concerned in this course, we assume an integer value can be held within a single word. So the word size cannot be bounded by a constant. However, this creates an issue for the integer multiplication problem. If we allow the word size to hold the entire input integer, then the computation will require only a single basic

3 operation and constant time. Therefore, in the integer multiplication problem, we require to deal with extremely large numbers that may occupy n words. Exercise: If you are allowed to put an arbitrary large integer in a single word, can you sort n integers in O(n) time? (Hint: concatenate all integers into a big word.) A way to avoid the vagueness of word size is to study the bit complexity, where each word can only hold a constant number of bits (or 1 bit). While this definition removes the ambiguity, it creates a great deal of additional work in analyzing the time complexity. The benefits is very limited. Thus, we use word- RAM model instead, with the convention that we do not play the trick of concatenating more than constant number words into one word. Bit complexity is studied in certain areas of algorithms, but not in this course. But does it matter if we put constant number of words into one? It doesn t, because we only care about the asymptotic complexity, in another word, big O complexity. Having a constant factor does not affect the big O complexity (particularly when the time complexity is a polynomial of the input size). Example: Set overlap. You have two array of integers. Find if there is an element shared by the two arrays. Algorithm 1. Input: Array A and B of size n. 1. for i from 1 to n 2. for j from 1 to n 3. if (A[i]==B[j]) return yes. 4. return no. Correctness of the algorithm: If the algorithm returns yes, then a shared element is found. If the algorithm returns no, then every pair of elements between A and B have been compared and no duplication is found. Therefore, the two array do not share any element. Thus, the algorithm is correct. Time complexity: The two nested for loops will repeat at most n 2 times and line 3 takes O(1) for each loop. Therefore, the time complexity is O(n 2 ). Remark: The index of an array starts from 1 in most pseudocode in this course. This is recommended. Guideline: In this course, we require you to present your algorithm in a pseudocode format whenever possible. Also, briefly argue the correctness and analyze the time complexity for each algorithm. The level of details depends on the situation. On one hand, try always say a few words no matter how simple the algorithm/problem is. On the other hand, when the pseudocode, proof, or analysis takes too long (e.g. over a page) to write, try to describe them in a higher level so that the total length is not overly long. Whenever necessary, do not hesitate to introduce subroutines in the algorithm and lemmas in the proof to make it easier for the readers to read. When writing, keep in mind that you re doing a one-way written communication with your reader and you have no chance (or it is very expensive) to answer the readers questions during their reading. A good suggestion is that you need to write to convince a skeptical TA. Now more efficient algorithms for the duplication detection problem:

4 (Solutions at end of this note. Do not read solutions before you spend time on it.) Guideline: I will publish the course notes before the class so you can print and add hand-written notes on it. However, it is recommended that you do not read the notes before class participate to the thinking an discussion when I give you time in class to invent an algorithm do review the notes again after the class. This practice allows you the maximum opportunity to practice your skill to develop new algorithms rather than just learning existing algorithms by reading. Time complexity analysis can be challenging: So far, the time complexity analysis has been straightforward. But it is not always like that. For example, here is a very simple algorithm whose analysis is still incomplete: PIERCE(a, b) /* a, b are integers with a > b > 0 */ while (b 0) do b = a mod b; On input integers a, b with a > b, how many "mod" steps does PIERCE perform? It is known that this number O(a 1/3 ). It is also known that this number is Ω(log a) infinitely often. But the gap between the upper and lower bounds are huge. Exercise: you will get 100 if you can substantially these bounds. Course Contents Algorithm analysis: correctness & time complexity Algorithm design techniques: Reduction Recursion Divide-and-conquer Dynamic programming Greedy Exhaustive search (briefly discussed in this course) Graph algorithms: A lot of problems can be reduced to graph problems and call standard graph algorithms. Intractability: Not every problem has an efficient algorithm. Undecidability: Not every problem has an algorithm.

5 Answers to the in class exercise. (Do not read before you have tried to solve by yourself.) Algorithm 2: Sort array A. Then for each element in B binary search in A. O(n log n) Algorithm 3: Put A in a hash table. O(n) average complexity. Algorithm 4: When the integers in the array are all small positive numbers. One can use an array instead of a hash table. O(n) worst-case complexity. In-depth exercise: Can you modify Algorithm 4 so you do not need even to initialize the array?