Lecture 2: Algorithms, Complexity and Data Structures. (Reading: AM&O Chapter 3)

Size: px

Start display at page:

Download "Lecture 2: Algorithms, Complexity and Data Structures. (Reading: AM&O Chapter 3)"

Sibyl Cain
6 years ago
Views:

1 Lecture 2: Algorithms, Complexity and Data Structures (Reading: AM&O Chapter 3)

2 Algorithms problem: class of situations for which a computergenerated solution is needed. instance: specification of a particular example of the problem. input: prearranged format under which an instance is described to the computer. length of input: number of bits needed to describe this instance. output: specified answer corresponding to a given instance. algorithm: a mathematically precise description suitable for coding as a computer program of how to produce the (correct) solution for problem on any specified instance.

3 Example: Sorting Problem Instance: set S = {a 1,..., a n } of rational numbers. Output: ordered list L = [a i1 a i2... a ik ] with a i1 a i2... a in of the elements of S. Algorithm I procedure L = EXTEND(L 0,S 0 ) Input: partial ordered list L 0 = [a i1 a i2... a ik ] of length k with a i1 a i2... a ik, remaining set S 0 of elements left to sort. Output: complete ordered list L obtained by adding the elements of S 0 to the end of L 0. Returns nosort if this cannot be done. Procedure: if S 0 = then output L 0 and RETURN else L = nosort for each a j S 0 if L0 = or a j a ik Initial Call: EXTEND(,S) then L = EXTEND([L 0 a j ],S 0 \{a j }) next in order a i1 a i2... a ik {a ik+1,..., a j,..., a in }

4 Algorithm II procedure L =EXTEND FROM BOTTOM(L 0,S 0 ) Input: partial ordered list L 0 = [a i1 a i2... a ik ] of the smallest k elements of S, remaining set S 0 of elements left to sort. Output: complete ordered list L starting with L 0 Procedure: if S 0 = then output L 0 and RETURN else begin end let a j be the minimum element in S 0 EXTEND FROM BOTTOM([L 0 a j ],S 0 \{a j }) Initial Call: EXTEND FROM BOTTOM(,S) smallest a i1 a i2... a ik {a ik+1,..., a j,..., a in }

5 Algorithm III Assume that the input set S has exactly 2 k elements (by padding with very large elements and stripping them off at the end) procedure MERGE(L 1,L 2 ) Input: partial ordered lists L 1 = [a i1... a ik ] and L 2 = [a j1... a jk ], each of size k Output: ordered list L 3 of size 2k made up of the union of the elements in L 1 and L 2 Procedure: L 3 = while either L 1 or L 2 is nonempty begin end return L 3 procedure SORT(S 0,i) let a min be the smallest of the first elements of L 1 or L 2 (if there is one) set L 3 = L 3 a min, and remove a min from whichever list it is in Input: set S 0 of size 2 i Output: ordered list L of elements of S 0 Procedure: if i = 0 then return the list of the single element of S 0 else begin end Initial Call: SORT(S,k) partition S 0 arbitrarily into two sets S 1 and S 2 of size 2 i 1 return MERGE(SORT(S 1,i 1),SORT(S 2,i 1))

6 a ik1 a ik2... a ik,n 1 a ikn a ik 1,1... a ik 1,n/2 a ik 1,n/ a ik 1,n a i31 a i32 a i33 a i34 ai3,n 3 a i3,n 2 a i3,n 1 a i3,n a i21 a i22 a i23 a i24 ai2,n 3 a i2,n 2 a i2,n 1 a i2,n a 1 a 2 a 3 a 4 a n 3 a n 2 a n 1 a n The Picture

7 Examples Suppose we want to sort the sequence 10, 2, 6, 4 Algorithm I: Sequence of calls is EXTEND(,{10, 2, 6, 4}), EXTEND([10],{2, 6, 4}), EXTEND([2],{10, 6, 4}), EXTEND([2 10],{6, 4}), EXTEND([2 6],{10, 4}), EXTEND([2 6 10],{4}), EXTEND([2 4],{10, 6}), EXTEND([2 4 10],{6}), EXTEND([2 4 6],{10}), EXTEND([ ], ) Algorithm II: Sequence of calls is EXTEND FROM BOTTOM(,{10, 2, 6, 4}) (find min{10, 2, 6, 4}) EXTEND FROM BOTTOM([2],{10, 6, 4}) (find min{10, 6, 4}) EXTEND FROM BOTTOM([2 4],{10, 6}) (find min{10, 6}) EXTEND FROM BOTTOM([2 4 6],{10}) (find min{10}) EXTEND FROM BOTTOM([ ], ) Algorithm III: Size of initial set is 4 = 2 2. Sequence of calls is SORT({10, 2, 6, 4},2) MERGE(SORT({10, 2},1)), MERGE(SORT({6, 4},1)), MERGE(SORT({10},0),SORT({2},0)), MERGE(SORT({6},0),SORT({4},0)), MERGE({10}, {2}),MERGE({6}, {4}), MERGE({2, 10}, {4, 6})

8 Complexity of Algorithms What is the running time of these algorithms? In particular, 1. How should we measure the running time? 2. How do we account for different running times with respect to different inputs? with respect to different machines? 3. What should we count as a basic computational step?

9 Worst-Case Complexity Basic operations arithmetical operation (multiplication/division vs. addition/subtraction) comparisions assignments retrieval (RAM model) logical operations The worst-case complexity of an algorithm will 1. treat all basic operations as taking one step (regardless of the size of any numbers involved) 2. ignore running time constants 3. measure everything in terms of worst case (most pessimistic) performance 4. state complexity as a function of the relevant problem parameters

10 Orders of Complexity Let f(n) and g(n) be two functions mapping positive integers (representing problem size) into nonnegative integers (representing running time). We say that f(n) = O(g(n)) (f(n) is order g(n)) if there is a constant C such that, for all n sufficiently large, f(n) Cg(n) Example: 3n n log n = O(n 2 ) Let T (n) be a function which, for each n, represents the maximum number of computational steps necessary to run the algorithm on any input of size n. The complexity of an algorithm is denoted by O(g(n)), where g(n) is a simple function such that T (n) = O(g(n)).

11 constant time algorithm: algorithm with complexity O(1) linear time algorithm: algorithm with complexity O(n) (best possible). polynomial time algorithm: algorithm with complexity O(n k ) for some fixed k lower bounds on complexity: We use T (n) = Ω(g(n)) to represent the fact that for every N there exists some input of size n N for which the algorithm will take at least T (n) Cg(n) steps, for some fixed constant C.

12 Complexity of Sorting Algorithms Input size: represented by n Algorithm I: If numbers come in almost sorted, then Algorithm I takes almost linear time. However, if the numbers come in decreasing order (worst case), then Algorithm I will consider every ordered sublist at some point in the algorithm. Therefore Algorithm I has complexity Ω(2 n ) (not polynomial). Algorithm II: Let T (n) represent the running time of EXTEND FROM BOTTOM. First note that finding the min takes O(n) time. Then looking at the computations and recursive call, we get T (n) = Cn + T (n 1) = Cn + C(n 1) C + C = Cn(n 1) = O(n 2 )

13 Algorithm III: Note that MERGE requires O(n) comparisons, where n is the size of the union of the two lists. Then letting T (2 i ) represent the running time of SORT, and looking at the computations and recursive call, we get T (2 i ) = C2 i + 2T (2 i 1 ) = C2 i + C2 i C2 i }{{} i times = Ci2 i = O(n log n) Problem complexity: Complexity of the best possible algorithm for solving a problem. Upper bounds on problem complexity are easy just produce an algorithm. Lower bounds on problems complexity are generally very difficult.

14 A comparison of running times At 10 6 operations/sec: Input size complexity n.01 sec..02 sec..03 sec..04 sec..05 sec..1 sec. 100n log n.03 sec..08 sec..14 sec..21 sec..28 sec. 1.2 sec. 100n 2.01 sec..04 sec..09 sec..16 sec..25 sec. 10 sec. n 5.1 sec. 3.2 sec sec. 1.7 min. 5.2 min. 2.8 hr. 2 n.001 sec. 1 sec. 18 min. 13 da. 36 yr cen. n! 3.6 sec. 771 cen cen. Improving computing time to 10 9 operations/sec: complexity 1000n 1000n log n 100n 2 n 5 2 n n! improvement in problem size per 1 hour of computing more 3 more Moral: The only good algorithm is a polynomial algorithm.

15 A Lower Bound on Sorting Assume that you may only sort by actually comparing the input values pairwise (no fancy use of buckets, digits, etc.). Assume also that the input numbers are all distinct. Then number of possible inputs that need to be distinguished: n! number of possible outcomes of one comparison: 2 number of possible outcomes of r comparisons: 2 r Therefore, the number of comparisons r necessary to obtain at least n! outcomes must satisfy or 2 r n! r log(n!) = Ω(n log n) It follows that Algorithm III has the best possible complexity of a sorting algorithm (that sorts by comparison!)

16 A Better Bound by Use of Data Structures Suppose the input has a special form where the numbers come from a fixed set, say {1, 2,..., C}. We can construct a sorting algorithm having data-dependent complexity of O(n + C) by using buckets. Then the algorithm is Algorithm IV Set up buckets B 1,..., B C, where bucket B i is to contain all elements having value i for i = 1,..., n Put a i into bucket B ai Output the contents of buckets B 1,..., B C in order of increasing index. This is a linear time sorting algorithm for any input set having a fixed number of values.

17 Other Basic Data Structures Stacks and queues are simply lists of objects, arranged so that objects in the list can be added or removed easily. stack/queue operations: push: put a new element on the list pop: remove an element from the list data structure: array with moving begin-end-ofstack pointers, or linked list queue removal order: FIFO objects are removed in the same order they are placed on the stack. stack removal order: LIFO objects are removed in the opposite order they are placed on the stack. complexity of push/pop operations: O(1)

18 Data Structures for Describing Networks standard input structure: n = V, m = A, and a list tail(e), head(e), e = 1,..., m of arcs (assuming V = {1, 2,..., n}. node-arc incidence matrix: n m matrix N = (a ve ) whose entries are: directed case: a ve = undirected case: a ve = forward star structure: +1 v = tail(e) 1 v = head(e) 0 v not incident to e { 1 v incident to e 0 v not incident to e A(v) = set of arcs whose tail is v G.node(v).first = first arc in A(v) (0 if none) G.arc(e).next = arc after e in A(v) (0 if end of list) Order of arcs in the forward star lists (usually) arbitrary.

19 Examples of Data Structures a 1 2 a 5 a 1 2 a 4 1 a a a 3 4 a 2 3 a 6 a 2 3 a 5 Directed Network Adjacency matrices ( Undirected Network ) ( ) Forward star stucture (directed graph): v G.node(v).first a G.arc(e).next

Introduction to Data Structure

Introduction to Data Structure CONTENTS 1.1 Basic Terminology 1. Elementary data structure organization 2. Classification of data structure 1.2 Operations on data structures 1.3 Different Approaches to