Extended Introduction to Computer Science CS1001.py

Transcription

1 Extended Introduction to Computer Science CS00.py Lecture 6: Basic algorithms: Search, Merge and Sort; Time Complexity Instructors: Benny Chor, Amir Rubinstein Teaching Assistants: Michal Kleinbort, Yael Baran School of Computer Science Tel-Aviv University Spring Semester 05

2 Lecture 5: Highlights Integer exponentiation Naïve method vs. iterated squaring Search sequential vs. binary

3 Lecture 6: Plan Basic algorithms: Binary search reminder and overview Sort Merge Complexity of algorithms The O( ) notation Worst / best case analysis Tractable and intractable problems 3

4 Binary search reminder and overview 4

5 5

6 6

7 7

8 8

9 9 Sort

10 The sorting problem The computational problem: Input: a set of elements Output: a sequence of the same elements, ordered by size Note that a computational problem is described in abstract terms, and is merely a desired relation between legal inputs and their outputs Technically, we will represent a sequence as a python list Possible algorithms? We will see at least 3 in this course, one today These solutions employ different strategies, which has consequences in terms of efficiency, as we will see 0

11 The algorithm: Selection sort Selection-Sort(lst of size n) for i=0 to n-: find the minimum of lst[i:] swap it with the element at index i Implementation in code: def selection_sort(lst): ''' sort lst using selection method ''' n = len(lst) for i in range(n): m_index = i #index of minimum for j in range(i+,n): if lst[m_index] > lst[j]: m_index = j swap(lst, i, m_index) return lst def swap(lst, i, j): tmp = lst[i] lst[i] = lst[j] lst[j] = tmp

12 Selection sort (another version) Another, more Pythonic implementation: def selection_sort(lst): n = len(lst) for i in range(n): m = min(lst[i:n]) m_index = lst.index(m) #find the index of the minimum lst[i], lst[m_index] = lst[m_index], lst[i] return lst Is this version more efficient? We leave this for you to test.

13 Selection Sort - efficiency Let us first measure actual running times. import time import random for n in [000,000,4000]: lst = list(range(n)) random.shuffle(lst) #balagan t0 = time.clock() #start stopper selection_sort(lst) t= time.clock() #end stopper print("n=", n, t-t0) n= n= n= >>> How does running time seem to change with input size? 3

14 Selection Sort - analysis As a measure for efficiency, we will look at the number of iterations. An underlying assumption: the time needed for simple operations (arithmetic ones, comparisons, reading / writing to a variable etc.) is bounded by some constant. Therefore each iteration takes a constant amount of time This assumption holds only for variables taking up to a fixed number of computer words. recall integer exponentiation as a scenario in which this assumption does not hold. We will encounter soon more such scenarios (e.g. in cryptography). So, how many iterations are needed, as a function of the input size? Input size in this case is the list s length, denoted n Does the answer depend on the content of the list, or on its length only? Answers: on board 4

15 5 Merge

16 The computational problem: Merge Input: two sorted sequences of elements Output: one sorted sequence containing all elements in both sequences Possible algorithms? 6

17 Merge - possible algorithms Simply concatenate both lists and sort them all def merge_by_sort(a,b): """ merging two lists """ return selection_sort(a+b) However this solution does not take advantage of the input lists being sorted already. 3 running indices, for input lists (A, B) and the output (C). At each iteration, select the minimal element from A or B and copy it to C. What happens when one of the lists is completed? 7

18 Example A B 6 6 C 8

19 A B 6 6 C 9

20 A B 6 6 C 0

21 A B 6 6 C 5

22 A B 6 6 C 5 6

23 A B 6 6 C

24 A B 6 6 C

25 A B 6 6 C

26 A B 6 6 C

27 A B 6 6 C

28 A B 6 6 C

29 A B 6 6? C

30 A B 6 6? C

31 A ? B 6 6? C

32 def merge(a, B): ''' Merge list A of size n and list B of size m A and B must be sorted! ''' n = len(a) m = len(b) C = Code a=0; b=0; c=0 while a<n and b<m: #more element in both A and B if A[a] < B[b]: C[c] a = a+ else: C[c] = B[b] c = c+ if else: :#A was completed while : C[c] = B[b] b = b+ c = c+ #B was completed : return 3

33 def merge(a, B): ''' Merge list A of size n and list B of size m A and B must be sorted! ''' n = len(a) m = len(b) C = [0 for i in range(n + m)] Code a=0; b=0; c=0 while a<n and b<m: #more element in both A and B if A[a] < B[b]: C[c] = A[a] a+= else: C[c] = B[b] b+= c+= if a==n: #A was completed while b<m: C[c] = B[b] b+= c+= else: #B was completed while a<n: C[c] = A[a] a+= c+= return C C[c:] = A[a:]+B[b:] #append remaining elements one of the lists A or B is non-empty 33

34 Merge - analysis Again, we will look at the number of iterations. So, how many iterations are needed, as a function of the input size? Denote: A = n, B =m Does the answer depend on the content of the lists, or on their length only? Compare to merge_by_sort we saw earlier: def merge_by_sort(a,b): ''' Merge list A of size n and list B of size m return selection_sort(a+b) 34

35 Merge running times for merge_func in [merge_by_sort, merge]: print(merge_func. name ) for n in [000,000,4000]: lst = [random.choice(range(0000)) for i in range(n)] lst = sorted(lst) lst = [random.choice(range(0000)) for i in range(n)] lst = sorted(lst) t0=time.clock() merge_func(lst,lst) t=time.clock() print("n=", n, t-t0) merge_by_sort n= n= n= merge n= n= n= Consistent with the theoretical analysis: -merge_by_sort is quadratic - merge is linear (note we chose n=m for simplicity) 35

36 Merge_by_python_sort merge_by_sort n= n= n= merge_by_python_sort n= n= n= merge n= n= n= def merge_by_python_sort(a,b): return sorted(a+b) So, python s sorted does a farely nice job! However we will not get into the interiors of sorted (at least not now). 36

37 37 Time Complexity: A Crash Intro

38 Time Complexity: A Crash Intro A problem is a general computational question: description of parameters (input) description of solution (output) An algorithm is a step-by-step procedure, a recipe can be represented e.g. as a computer program (but also in natural languages, diagrams, animations, etc.) an abstract notion Efficient algorithms are usually preferred fastest time complexity most economical in terms of memory space complexity Time complexity analysis: measured in terms of operations, not actual timings We want to say something about the algorithm, not a specific machine/execution/programming language implementation expressed as a function of problem size We will be interested in how the number of operations changes with input size 38

39 Growth rate We will be interested in how the number of operations changes with input size. In most cases, we will not care about the exact function, but in its order, or growth rate. Sometimes we will only be interested/able to give an upper bound for this growth rate. We will, however, strive to make this upper bound as tight (=low) as we can. In this course, we will almost always be able to give tight upper bounds. We need some formal definition for growth rate upper bound. 39

40 Big O Notation We say that a function f(n) is O(g(n)) if there is a constant c such that for large enough n, f(n) c g(n) We denote this as f(n) = O(g(n)) For example: 5n log (n) = O(n log(n)) log (n) = O(n) 000 n log (n) = O(n ) n/00 O(n 00 ) [where did the log base disappear?] [not the tightest possible bound] [not the tightest possible bound] 40

41 Big O Notation - example Consider the two functions g(n) = 0 n log n +, and f(n) = n ( + sin(n)/3) +. It is not hard to verify that g(n) = O(f(n))). Yet, for small values of n, g(n) > f(n), as can be seen in the following plot. 4

42 Big O Notation example (cont) But for large enough n, g(n) < f(n), as can be seen in the next plot. 4 But remember that for big O, g(n) may be larger than f(n), as long as there is a constant c such that g(n) < c f(n)

43 Complexity Hierarchy constant O() logarithmic Poly-logarithmic O(logn) O(log n) linear quadratic O(n) O(n ) O(nlogn) exponential O( n ) O(3 n ) Will meet this guy again later in the course 43

44 Big O Notation more examples On the board. 44

45 Basic algorithms time complexity summary Algorithm Search - sequential Binary search Time complexity Best case scenario O() O() Worst case scenario O(n) O(logn) Selection sort Merge O(n ) O(n+m) O(n ) O(n+m) 45

46 Big O Notation: A Discussion how can an algorithm with O(logn) time complexity look like? Halves (or divides by some other constant) the problem size example: binary search how can an algorithm with O(n) time complexity look like A constant number of operation for each element example: the palindrome problem, merge ( input sizes) how can an algorithm with O(n ) time complexity look like? Have to look at all pairs of elements example: selection sort how can an algorithm with O( n ) time complexity look like? Need to look at all possible solutions example: integer exponentiation (exponential in the number of bits) soon: towers of Hanoi, trial division, gcd We deal with them, point they are fine for small size inputs, AND try to improve upon them (not always successfully). 46

47 Tractability - Basic Distinction: How would execution time for a very fast, modern processor (0 0 ops per second, say) vary for a task with the following time complexities and n = input sizes? E-09.0E E E E E-09 n seconds seconds seconds seconds seconds seconds n.0e E E-08.6E-07.5E E-07 seconds seconds seconds seconds seconds seconds n 3.0E E-07.7E E-06.3E-05.E-05 seconds seconds seconds seconds seconds seconds n 5.0E seconds seconds seconds seconds seconds seconds n.0e-07.05e seconds seconds seconds minutes days years 3 n 5.9E E+09 seconds seconds hours years centuries centuries Modified from Garey and Johnson's classical book 47 Polynomial time = tractable. Exponential time = intractable.

48 Time Complexity - What is tractable in Practice? A polynomial-time algorithm is good. An exponential-time algorithm is bad. n 00 is polynomial, hence good. n/00 is exponential, hence bad. Yet for input of size n = 4000, the n 00 time algorithm takes more than 0 35 centuries on the above mentioned machine, while the n/00 algorithm runs in just under two minutes. 48

49 Time Complexity - Advice Trust, but check! Don't just mumble "polynomial-time algorithms are good", "exponential-time algorithms are bad" because the lecturer told you so. Asymptotic run time and the O notation are important, and in most cases help clarify and simplify the analysis. But when faced with a concrete task on a specific problem size, you may be far away from "the asymptotic". In addition, constants hidden in the O notation may have unexpected impact on actual running time. 49

50 Time Complexity Advice (cont.) We will employ both asymptotic analysis and direct measurements of the actual running time. For direct measurements, we will use either the time package and the time.clock() function. Or the timeit package and the timeit.timeit() function. Both have some deficiencies, yet are highly useful for our needs. 50