CS420/520 Algorithm Analysis Spring 2010 Lecture 01

Transcription

1 CS420/520 Algorithm Analysis Spring 2010 Lecture 01 I. Review the syllabus A. How to reach me this semester B. course description from catalog C. prerequisites 1. C++ 2. Java D. Computer Requirement 1. JDK 2. GNU C/C++ E. instructional objectives and relationship to exams this semester F. text and related resource texts G. Internet resources reserve holdings for this class H. homework assignments 1. reading 2. homework 3. programming a) languages b) style & documentation (1) RCS/CVS keyword substitution (2) C++ header located on WWW page (3) Javadoc template located on WWW page (4) Rsm cef H O rsm.html *.java (5) Javadoc *.java (6) Print in such a way that lines do not wrap c) test cases 4. grading policies I. Attendance J. Academic standards K. general schedule II. Chapter 1 The Role of Algorithms in Computing A. 1.1 Algorithms 1. informal algorithm definition - any well-defined computational procedure that takes from some value, or set of values, as input and produces some value, or set of values, as output. 2. view algorithm as a tool for solving a well-specified computational problem a) statement of the problem specifies in general terms the desired input/output relationship b) the algorithm describes a computational procedure for achieving that input/output relationship 3. what is an algorithm? a) a step-by-step procedure for solving a specific problem in a finite amount of time b) a finite set of rules which gives a sequence of operations for solving a specific type of problem [Knuth v1 pg 4] (1) finiteness - an algorithm must terminate after a finite number of steps Spring

2 (a) a procedure which lacks finiteness is a computational method (2) definiteness - each step of an algorithm must be precisely defined; the actions to be carried out must be rigorously and unambiguously specified for each case (a) an expression of a computational method in a computer language is called a program (3) input - an algorithm has zero or more inputs (quantities which are given initially before the algorithm begins) (4) output - an algorithm has one or more outputs (quantities which have a specified relation to the inputs) (5) effectiveness - all operations to be performed in the algorithm must be sufficiently basic that they can in principle be done exactly and in a finite length of time by a person using pencil and paper 4. DIFFERENCE BETWEEN AN ALGORITHM AND A COMPUTATIONAL PROCEDURE 5. sorting problem (defined page 5, CLRS text) a) the specific input sequence specified is an instance of the sorting problem b) an instance of a problem consists of all the inputs (satisfying any constraints imposed in the problem statement) needed to compute a solution to the problem 6. an algorithm is said to be correct if, for every input instance, it halts with the correct output. 7. Tonight we are considering only deterministic programs later lectures will address nondeterministic algorithms. 8. Bertrand Meyer, CHAPTER 9, AXIOMATIC SEMANTICS, Introduction to the Theory of Programming Languages. a) assertions (1) a property of the program's objects which may or may not be satisfied by a state of the program during execution (2) (will be expressed as Boolean expressions in concrete syntax for now) b) preconditions and postconditions (programming by contract) (1) preconditions (P) - assumed to be satisfied before the fragment is executed (a) (b) should be as broad as possible it is the responsibility of the environment to invoke the program or program fragment only for cases that fall within the precondition (2) postconditions (Q)- guaranteed to be satisfied after the fragment has been executed (3) a program or program fragment will be said to be correct with respect to a certain precondition P and a certain postcondition Q IFF when executed Spring

3 in a state in which P is satisfied it yields a state in which Q is satisfied (4) precondition obligates the environment, postcondition obligates the program (5) programming by contract (6) defined this way program correctness is only a relative concept...but programs must terminate c) definitions of total and partial correctness (1) definition (Total Correctness): A program fragment a is totally correct for P and Q IFF the following holds: whenever a is executed in any state in which P is satisfied, the execution terminates and the resulting state satisfies Q. (a) achieves the postcondition (b) and terminates (2) definition (Partial Correctness): A program fragment a is partially correct for P and Q IFF the following holds: whenever a is executed in any state in which P is satisfied and this execution terminates, the resulting state satisfies Q. (a) achieves the postcondition (b) if it terminates 9. What kinds of problems are solved by algorithms? a) Types of Problems (1) Decision Problem - decide whether the input satisfies property Y (2) Adaptive Problem - maintain property Y over time (3) Search Problem - find an X in the input that satisfies property Y (4) Structuring Problem - transform input to satisfy property Y (5) Construction Problem - build an X to satisfy property Y (6) Optimization Problem - find the best X that satisfies property Y b) problem categories (1) conceptually hard problems - don't have an algorithm to solve the problem because don't understand it well enough (2) analytically hard problems - have an algorithm but don't know how to analyze how long it will take to solve every problem instance (3) computationally hard problems - analyzed and analyzed it, but only small problems can be solved efficiently (a) know? (b) Suspect? (4) Computationally unsolvable - no efficient algorithm exists c) Examples in CLRS Spring

4 (1) Sorting (2) Biological problems, such as gene identification (3) Information retrieval (4) Privacy and security (5) Allocation of scarce resources (6) Shortest route from one point to one other point/all other points (7) Longest common string subsequence (8) Topological sorting (9) Convex hull d) Some general characteristics (1) Finding a solution we do want from among many candidate solutions (2) The problems solved in the text are often practical problems 10. Data structures a) A data structure is a way to store and organize data in order to facilitate access and modification 11. Why study computational complexity? a) analyzing computational methods helps us to focus on what must be determined and the most efficient way to determine it b) "A detailed consideration of computational complexity serves as a kind of "Occam's razor": the most efficient algorithms are generally those that compute exactly the information relevant to the problem situation. Thus the development of an especially efficient algorithm often gives us added insight into the problem we are considering, an the resulting algorithm is not only efficient but simple and elegant. Such algorithms are the kind we are after." [Tarjan] c) (showed up in the movie "Contact") d) William of Ockham (1284?-1347?) (1) believed the primary form of knowledge came from experience gained through the senses (2) based scientific knowledge on such experience and on self-evident truths (3) and on logical propositions resulting from these two sources (4) stressed the Aristotelian principle that "entities must not be multiplied beyond what is necessary" (5) What can be done with fewer assumptions is done in vain with more. (Occam's Razor sometime in the 1320s) (6) the simplest theory that fits the facts of a problem is the one that should be selected (7) he was persecuted in Pope John XXII's purge of spiritualists in the Franciscan order and other orders in 1324 e) The importance of hard problems (1) Family of NP-complete problems (a) It is unknown whether or not efficient Spring

5 algorithms exist for NP-complete problems (b) If an efficient algorithm for one is found, all others in the family have efficient algorithms (c) Problems in the family are similar to problems for which there are efficient algorithms (d) Importance of solutions by approximation techniques 12. two example problems a) Given a set of n distinct positive integers x 1, x 2,..., x n (where n is an even number) (1) partition the set into two subsets each of cardinality n/2 such that the difference between the sums of the two subsets is maximized (2) partition the set into two subsets such that the difference between the sums of the two subsets is minimized b) Which problem looks harder? B. 1.2 Algorithms as technology 1. algorithms, like computer hardware, are a technology 2. we need to consider algorithms that are space and time efficient 3. Efficiency a) Sorting algorithm examples (1) Insertion sort takes c 1 n 2 time to sort n items (2) Merge sort takes c 2 nlgn time to sort n items (3) There is some partition of the integers such that below that partition insertion sort is faster than merge sort, and above which merge sort is faster than insertion sort Spring

6 Run Time Sort Comparisons (c1=12, c2=100) Number of items sorted insertion sort merge sort III. b) a) algorithm efficiency is more important than computer speed when the size of the input becomes large 2. Algorithms and other technologies a) Total system performance depends on choosing efficient algorithms as much as on choosing fast hardware b) Importance of algorithms (1) Used in routing and network control (2) Used in design of hardware (computational devices as well as interface devices) (3) Problems to be solved are larger than ever before IV. Chapter 2 Getting Started A. 2.1 Insertion Sort 1. (SEE KNUTH VOL 3 page 382 ff for history of sorting and developments associated with these historical events) 2. Insertion Sort a) efficient for sorting a small number of elements (1) works much like sorting cards held in a hand of play (2) empty left hand (assumes you are right handed) and cards face down on the table (3) choose a card from the table and insert it in the correct position in the left hand (a) compare to each card already in the hand from right to left b) specification, page 16 Spring

7 (1) input (2) output c) pseudocode intro (1) employs whatever expressive method is most clear and conside to specify a given algorithm (2) pseudocode not typically concerned with issues of software engineering (a) data abstractions, (b) modularity, (c) error handling d) INSERTION-SORT pseudocode, page 18 (1) sorts in place: the numbers are rearranged within the input array A (2) BE AWARE THAT THIS TEXT USES ARRAYS WITH MIN INDEX = 1 (3) note that parameters are passed by value, except for arrays which are passed as pointers (Pascaltype code conventions) (4) note use of short-circuit evaluation on line 5 e) Loop invariants and the correctness of insertion sort (From Chapter 4 of Discrete Mathematics with Applications, Susanna S. Epp, 1990, Wadsworth) Spring

8 (1) [pre-condition for loop] while (G) [Statements in loop body. None contain branching statements that leap outside the loop] end while [post-condition for loop] (2) A loop is defined to be correct with respect to its pre-and post-conditions if, and only if, whenever the algorithm variables satisfy the precondition for the loop and the loop is executed, then the algorithm variables satisfy the postcondition for the loop. (3) G is the guard, which restricts entry to the loop (4) Loop Invariant Theorem (a) Let a while loop with guard G be given, together with pre- and post-conditions that are predicates in the algorithm variables. Also let a predicate I(n), called the loop invariant, be given. If the following four properties are true, then the loop is correct with respect to its pre- and post-conditions (i) Basis Property: The pre-condition for the loop implies that I(0) is true before the first iteration of the loop. (ii) Inductive Property: If the G and the loop invariant I(k) are both true for an integer k 0 before an iteration of the loop, then I(k + 1) is true after iteration of the loop. (iii) Eventual Falsity of Guard: After a finite number of iterations of the loop, the guard G becomes false. (iv) Correctness of the Post-Condition: If for some nonnegative integer N, G is false and I(N) is true, then the values of the algorithm variables will be specified in the post-condition of the loop f) Principle of Strong Mathematical Induction (Epps page 249) (1) Let P(n) be a predicate that is defined for integers n, and let a and b be fixed integers with a b. Suppose the following two statements are true: (a) (b) P(a), P(a+1), & P(b) are all true [BASIS STEP] For any integer k > b, if P(I) is true for all integers I with a I < k, then P(k) is true [INDUCTIVE STEP] (2) Then the statement (a) For all integers n a, P(n) is true (3) (The supposition the P(I) is true for all integers I Spring

9 with a I < k is called the inductive hypothesis) (4) g) At the beginning of each iteration of the outer for loop, which is indexed by j, the subarray of A[1 j-1] constitute the sorted hand and the elements A[j+1...n] correspond to the cards on the table. (1) Properties of A[1 j-1] as a loop invariant (a) At the start of each iteration of the for loop in lines 1-8, the subarray A[1 j-1] consists of the elements originally in A[1 j-1] but in sorted order (2) Must show three things (a) Initialization: it is true prior to the first iteration of the loop (b) Maintenance: if it is true before an iteration of the loop, it remains true before the next iteration (c) Termination: when the loop terminates, the invariant gives us a useful property that helps show the algorithm is correct (3) Walkthrough analysis page Pseudocode Conventions a) Note also return is allowed for functions (e.g., page 128) b) Parameters passed by value B. 2.2 Analyzing Algorithms 1. analyzing an algorithm has come to mean predicting the resources that the algorithm requires 2. need a model of the implementation technology that will be used, including a model for the resources of technology and their costs 3. this text uses a random-access machine model of computation 4. Computer Models (architecture)(tarjan Notes) a) Turing Machines (1) See PowerPoint slides for Lecture 01 (2) Byte magazine article Nov 87 page 345 Malitz "The Turing Machine", Isaac Mailitz, BYTE, November, 1987, pp (3) a finite state control (4) two-way infinite memory tape divided into squares, each holds a finite number of symbols (5) R/W head (6) in one step can (a) read a square (b) write to a square (c) move to an adjacent square left or right (d) change state of the control b) random-access machine (1) finite program (2) finite collection of registers, each stores a single Spring

10 integer or real number (3) memory (array of n words with unique addresses 1 through n holding single integer or real numbers) (4) in one step can (a) perform a single arithmetic operation on specified register contents (b) fetch a word into a register using an (c) address in a register store the contents of a register in a word whose address is in a register c) pointer machines (1) memory is an extendable collection of nodes divided into a fixed number of fields (2) field holds a number of a pointer to a node (3) to read/write a field in a node the machine must have in a register a pointer to the node (4) operations on register contents, fetching/storing, creating and destroying nodes take constant time (5) cannot do address arithmetic (6) good for lower bounds on list-processing algorithms d) common machine attributes (1) all are deterministic and sequential e) C op = ln(number of operands) - logarithmic cost measure f) Tarjan assumptions (1) all algorithms implementable in random-access or pointer machines (2) integers of size at most nc for some small constant c (3) comparison, addition, and multiplication of input values allowed on real numbers (4) no clever encoding to hide parallel computations g) RAM model contains instructions commonly found in real computers (1) Arithmetic (2) Data movement (3) Control (4) (each instruction takes a constant amount of time) h) Date in the RAM model (1) Integer or floating point (2) Limit on size of each word (a) Integers represented by clgn, where c 1 i) What isn t included (1) Compound arithmetical operations (exponentiation) (2) Shift left (equivalent to multiplication by 2) (3) Do not attempt to model the memory hierarchy of real machines 5. Analysis of insertion sort a) input size depends on the problem being studied (total Spring

11 number of items in the input, total number of bits needed to represent the input in binary form) b) running time is the number of primitive operations or steps executed (1) make the notion as machine independent as possible (2) a constant amount of time is required to execute each line of pseudocode, i.e. each time the line is executed it takes the same amount of time (c i ) c) time for different pseudocode lines may be different, though d) if the data are already in sorted order, the running time is linear function of n with constants a and b depending on statement costs e) if the data are in reverse order, the running time is a quadratic function of n with constants a, b, and c depending on statement costs f) walkthrough pages of text 6. Worst-case and average-case analysis a) text usually concentrates on finding the worst-case running time (1) its an upper bound on the running time for any input (2) worst-case can occur fairly often for some algorithms (3) average-case is roughly as bad as the worst-case 7. order of growth a) we'll consider only the leading term of a formula, since the lower-order terms are relatively insignificant, in comparison, for large n C. 2.3 Designing Algorithms 1. incremental approach divide-and-conquer (running times are often easily determined using techniques described in chapter 4) a) The divide-and-conquer approach (using a recursive approach) b) three steps at each level of the recursion (1) DIVIDE the problem into a number of subproblems (2) CONQUER the subproblems by solving them recursively or as a recursion base case (3) COMBINE the solutions to the subproblems into the solution for the original problem c) merge sort (page 31) (1) recursive base case is when a subsequence has length 1 (2) need MERGE(A,p,q,r) to combine two sorted sequences (a) A is an array (b) p, q, and r are indices numbering the elements of the array such that p q < r. (c) merges A[p..q] and A[q+1..r] replacing Spring

12 A[p..r] (d) MERGE is a linear time function Θ(n), where n = r p +1 is the number of elements being merged (i) Choose smaller of two sets, place in merged set (ii) Repeat until both sets exhausted or one set remains (in which case all move to the merged set) (e) Use of a sentinel value (3) Loop invariant analysis of for loop lines (4) Runtime analysis of MERGE pages (5) MERGE-SORT(A,p,r) page 34 (a) (consult handwritten notes on tree here..) Analyzing divide-and-conquer algorithms a) Introduction (1) use recurrence equation or recurrence to describe the running time of an algorithm with a recursive call (2) a subproblems, each of which is 1/b the size of the original data (3) see page 35 b) Analysis of MERGE-SORT (1) Determining the runtime of the divide/conquer/combine steps (2) (Log functions grows more slowly than any linear function) (3) analysis of T(n) using recursion trees, pages (a) (b) add across the levels of the tree in general, level I below the top has 2 i nodes, each contributing a cost of c(n/2 i ), so that the ith level below the top has cost 2 i c(n/2 i ), = cn (4) The number of levels in the recursion tree is lgn + 1 (a) Base occurs when n = 1 and lg1 = 0 (5) Total cost cn(lgn + 1) = cnlgn + cn V. Let us review how we'll view computational complexity A. to solve a problem means to solve all possible instances of a problem B. we are interested in worst case behavior of algorithms C. we are interested in large input problems D. we will analyze time (space) using asymptotic behavior E. we can choose how we represent problem size F. all reasonable such choices will not change our conditions G. analysis is independent of computer model selected Spring