Data Structures and Algorithms Course Introduction. Google Camp Fall 2016 Yanqiao ZHU

Transcription

1 Data Structures and Algorithms Course Introduction Google Camp Fall 2016 Yanqiao ZHU

2 What are Algorithms? An algorithm is a self-contained step-by-step set of operations to be performed. Algorithms perform calculation, data processing, and/or automated reasoning tasks. An algorithm is an effective method that can be expressed within a finite amount of space and time and in a well-defined formal language for calculating a function. DATA STRUCTURES AND ALGORITHMS GOOGLE CAMP FALL 2016 YANQIAO ZHU 9/22/2016 2

3 What are Algorithms? (cont.) Starting from an initial state and initial input (perhaps empty), the instructions describe a computation that, when executed, proceeds through a finite number of well-defined successive states, eventually producing output and terminating at a final ending state. The transition from one state to the next is not necessarily deterministic; some algorithms, known as randomized algorithms, incorporate random input. DATA STRUCTURES AND ALGORITHMS GOOGLE CAMP FALL 2016 YANQIAO ZHU 9/22/2016 3

4 What are Data Structures? A data structure is a particular way of organizing data in a computer so that it can be used efficiently. Data structures can implement one or more particular abstract data types (ADT), which specify the operations that can be performed on a data structure and the computational complexity of those operations. In comparison, a data structure is a concrete implementation of the specification provided by an ADT. DATA STRUCTURES AND ALGORITHMS GOOGLE CAMP FALL 2016 YANQIAO ZHU 9/22/2016 4

5 Importance of Algorithms and Data Structures Computing Curricula 2005: The Overview Report CC 2005 provides undergraduate curriculum guidelines for five defined sub-disciplines of computing: 1. Computer Science (CS) 2. Computer Engineering (CE) 3. Information Systems (IS) 4. Information Technology (IT) 5. Software Engineering (SE) DATA STRUCTURES AND ALGORITHMS GOOGLE CAMP FALL 2016 YANQIAO ZHU 9/22/2016 5

6 Importance of Algorithms and Data Structures (cont.) Table: Comparative weight of computing topics across the five kinds of degree programs DATA STRUCTURES AND ALGORITHMS GOOGLE CAMP FALL 2016 YANQIAO ZHU 9/22/2016 6

7 Common Requirements of Computing Degrees A foundation in the concepts and skills of computer programming. The foundation has five layers: a) an intellectual understanding of, and an appreciation for, the central role of algorithms and data structures; b) an understanding of computer hardware from a software perspective, for example, use of the processor, memory, disk drives, display, etc. c) fundamental programming skills to permit the implementation of algorithms and data structures in software; d) skills that are required to design and implement larger structural units that utilize algorithms and data structures and the interfaces through which these units communicate; e) software engineering principles and technologies to ensure that software implementations are robust, reliable, and appropriate for their intended audience. DATA STRUCTURES AND ALGORITHMS GOOGLE CAMP FALL 2016 YANQIAO ZHU 9/22/2016 7

8 Course Overview Fundamentals Models of Computation Asymptotic Notations Solving Recurrences Divide-and-Conquer Case Studies Introduction to Multi-Thread Algorithms and their Analysis Sorting Quicksort Priority Queue, Heap and Heapsort Linear-Time Sorting Coding and Trees Concepts Huffman Coding Binary Search Trees AVL Trees Red-Black Trees Skip Lists DATA STRUCTURES AND ALGORITHMS GOOGLE CAMP FALL 2016 YANQIAO ZHU 9/22/2016 8

9 Course Overview (cont.) Hashing Hashing with Chaining Open Addressing Amortized Analysis Graphs Breadth-First Search Depth-First Search Priority-First Search Minimal Spanning Trees Shortest Paths Dijkstra Bellman Ford Floyd-Warshall DATA STRUCTURES AND ALGORITHMS GOOGLE CAMP FALL 2016 YANQIAO ZHU 9/22/2016 9

10 Course Overview (cont.) String Matching Knuth-Morris-Pratt Algorithm Boyer-Moore Algorithm Karp-Rabin Algorithm Dynamic Programming Case Studies: Fibonacci, Test Justification, Edit Distance and Knapsack Polynomial Time and Pseudo- Polynomial Time 2 Kinds of Guessing Computational Complexity P, EXP and R NP Reductions DATA STRUCTURES AND ALGORITHMS GOOGLE CAMP FALL 2016 YANQIAO ZHU 9/22/

11 Data Structures and Algorithms Asymptotic Notation Google Camp Fall 2016 Yanqiao ZHU

12 Measuring the Cost of an Algorithm Time consumption: dependent on multiple factors Input scale Machine performance Runtime and environment factors Compiler optimization A key factor: the scale of the problem instance Generally, the time cost increases with the scale of the problem instance 9/22/2016 Data Structures and Algorithms Google Camp Fall 2016 Yanqiao ZHU 2

13 Measuring the Cost of an Algorithm (cont.) For measurement, we take a function denoting the change of a program s execution time against the input scale, which is defined as the time complexity of an algorithm Specifically, with a certain algorithm dealing with a certain problem, T(n) denotes the needed time However, the scale is not always proportional to execution times: Sorting n numbers: sometimes all elements need to be swapped, sometimes none. From all inputs with the size of n, T(n) is chosen to be the longest of the execution time and T(n) denotes the time complexity of the algorithm 9/22/2016 Data Structures and Algorithms Google Camp Fall 2016 Yanqiao ZHU 3

14 Asymptotic Complexity For a problem with two different algorithms A and B, we could evaluate the efficiency through comparing the time complexity T A (n) and T B (n). For some certain problems, one algorithm may be more suitable for smallscale inputs while another for larger-scale ones. Luckily, when evaluating the performance of an algorithm, the differences dealing with small-scale problems are usually ignored while that dealing with large-scale problems are focused on. Asymptotic analysis: taking a long-term perspective and emphasis on strategies and methods of the overall trends and growth rates of the time consumption. 9/22/2016 Data Structures and Algorithms Google Camp Fall 2016 Yanqiao ZHU 4

15 Big-O Notation Express the upper boundary of T(n) f(n) = O(g(n)) means c > 0, n 0 > 0, n n 0, 0 f(n) c g(n) Properties: For any constants c > 0, O(f(n)) = O(c f(n)) For any constants a > b > 0, O(n a + n b ) = O(n a ) 9/22/2016 Data Structures and Algorithms Google Camp Fall 2016 Yanqiao ZHU 5

16 Random Access Machine (RAM) Random Access Memory (RAM) modeled by a big array Θ(1) registers (each 1 word) In Θ(1) time, can load r i into register r j compute (+,,, /, &,, ˆ) on registers store register r j into r i What s a word? w lg(memory size) bits assume basic objects (e.g., int) fit in word unit 4 in the course deals with big numbers realistic and powerful implement abstractions 9/22/2016 Data Structures and Algorithms Google Camp Fall 2016 Yanqiao ZHU 6

17 Pointer Machine An atomistic abstract computational machine model akin to the Random access machine. From its read-only tape (or equivalent) a pointer machine receives input bounded symbol-sequences ( words ) made of at least two symbols e.g. { 0, 1 } -- and it writes output symbol-sequences on an output write-only tape (or equivalent). To transform a symbol-sequence (input word) to an output symbol-sequence the machine is equipped with a program a finite-state machine (memory and list of instructions). Via its state machine the program reads the input symbols, operates on its storage structure a collection of nodes (registers) interconnected by edges (pointers labelled with the symbols e.g. { 0, 1 }), and writes symbols on the output tape. 9/22/2016 Data Structures and Algorithms Google Camp Fall 2016 Yanqiao ZHU 7

18 Pointer Machine (cont.) Cannot do arithmetic. Computation proceeds only by reading input symbols, modifying and doing various tests on its storage structure the pattern of nodes and pointers, and outputting symbols based on the tests. 9/22/2016 Data Structures and Algorithms Google Camp Fall 2016 Yanqiao ZHU 8

19 Basic Operations Instruction statements: Arithmetic operations Comparisons Branches Calling and returning from sub-functions etc. T(n) denotes the total number of basic operations to perform in an algorithm. T(n) is determined by the number of: Statements to execute Basic operations included to perform 9/22/2016 Data Structures and Algorithms Google Camp Fall 2016 Yanqiao ZHU 9

20 Constant O(1) OrdinaryElement(S, n) // select an ordinary element which is neither the maximum nor the minimum of n different numbers 1. select x, y, z S at random // x, y, z are different from each other 2. compare and sort them // assume after sorting a < b < c 3. print b 9/22/2016 Data Structures and Algorithms Google Camp Fall 2016 Yanqiao ZHU 10

21 Constant O(1) (cont.) Since set S is limited, there are exact one maximum and one minimum element in S. Hence, no matter how large S is, there will be at least one ordinary element in any three elements. Step 1 needs O(3) time to take three different elements from S. (Take x = S[0], y = S[1], z = S[2]) Then, to determine the relationship among x, y and z, at most three comparisons have to be done. Thus step 2 takes O(3) time. Finally, printing the ordinary element takes O(1) time. In summary, T(n) = O(3) + O(3) + O(1) = O(7) = O(1). 9/22/2016 Data Structures and Algorithms Google Camp Fall 2016 Yanqiao ZHU 11

22 Constant O(1) (cont.) Algorithms whose T(n) O(1), are called constant-time algorithm. These algorithms are ideal. Plus, there aren t better algorithms, because it s impossible reaping without sowing. Generally, algorithms containing one or a constant number of basic operations are constant-time algorithm. 9/22/2016 Data Structures and Algorithms Google Camp Fall 2016 Yanqiao ZHU 12

23 c 2 g(n) f(n) Asymptotic Notations c 1 g(n) Big-O notation: expressing upper boundary Ex: 2n 2 = O(n 3 ) n 0 f(n) = Θ(g(n)) (a) n Big-Ω notation: expressing lower boundary Ex: n = Ω(lgn) c g(n) f(n) Big-Θ notation: expressing asymptotically tight boundary Θ(g(n)) = O(g(n)) Ω(g(n)) define Θ(g(n)) = {f(n): c 1 > 0, c 2 >0, n 0 > 0, n n 0, c 1 g(n) f(n) c 2 g(n)} n 0 f(n) = O(g(n)) (b) f(n) n c g(n) n 0 n 9/22/2016 Data Structures and Algorithms Google Camp Fall 2016 Yanqiao ZHU f(n) = Ω(g(n)) 13 (c)

24 Strict Notations: Little-o and Little-ω f(n) = o(g(n)) means c > 0, n 0 > 0, n n 0, 0 f(n) < c g(n) in o-notation, the function f(n) becomes insignificant relative to g(n) as n approaches infinity Ex: 2n 2 = o(n 3 ) f(n) = ω(g(n)) means c > 0, n 0 > 0, n n 0, 0 c g(n) < f(n) Ex: 0.5n 2 = Θ(n 2 ), 0.5n 2 o(n 2 ) 9/22/2016 Data Structures and Algorithms Google Camp Fall 2016 Yanqiao ZHU 14

25 Space Complexity Asymptotic notations are suitable for measuring space complexity. Unless otherwise stated, space complexity usually excludes raw inputs but includes space of: Dumping Transiting Indexing Mapping Buffering etc. Algorithms need constant-scale additional space are called in-place algorithms. 9/22/2016 Data Structures and Algorithms Google Camp Fall 2016 Yanqiao ZHU 15

26 Worst, Best and Average Case Big-O notation: a conservative estimate Any case: Actual time consumption f(n) Worst case: Actual time consumption = f(n) A conservative estimate does not exclude the best case or a better case. Average case: the expectation of time complexity. 9/22/2016 Data Structures and Algorithms Google Camp Fall 2016 Yanqiao ZHU 16

27 Data Structures and Algorithms Linear-Time Sorting Google Camp Fall 2016 Yanqiao ZHU

28 Lower Bounds The following will be claimed in the comparison model: searching among n preprocessed items requires Ω(lgn) time binary search, AVL tree search optimal sorting n items requires Ω(nlgn) merge sort, heap sort, AVL sort optimal 9/22/2016 Data Structures and Algorithms Google Camp Fall 2016 Yanqiao ZHU 2

29 Comparison Model of Computation Input items are black boxes (ADTs) Only support comparisons (<; >; ; ; =) Time cost is the number of comparisons 9/22/2016 Data Structures and Algorithms Google Camp Fall 2016 Yanqiao ZHU 3

30 Decision Trees Any comparison algorithm can be viewed/specified as a tree of all possible comparison outcomes & resulting output, for a particular n: Decision Trees Algorithms Internal node Leaf Root-to-leaf path Path length (depth) Height of Tree Binary decision (comparisons) Found answer Algorithm execution Running time Worst-case running time 9/22/2016 Data Structures and Algorithms Google Camp Fall 2016 Yanqiao ZHU 4

31 Decision Trees (cont.) Example: binary search for n = 3: Index A[0] A[1] A[2] Value a b c No A[1] < x Yes A[0] < x A[2] < x No Yes No Yes x A[0] A[0]<x A[1] A[1]<x A[2] A[2] < x 9/22/2016 Data Structures and Algorithms Google Camp Fall 2016 Yanqiao ZHU 5

32 Search Lower Bound Suppose we have n preprocessed items. Finding a given item among all items in comparison model requires Ω(lgn) comparisons in worst case. Proof: Decision tree is binary and the number of leaves number of possible answers n (at least 1 per A[i]) height lgθ(n) = lgn±θ(1) 9/22/2016 Data Structures and Algorithms Google Camp Fall 2016 Yanqiao ZHU 6

33 Sorting Lower Bound Leaf species answer as permutation: All n! are possible answers Proof: number of leaves n! height lg(n!) = lg(1 2 (n 1) n) = lg1 + lg2 + + lg(n 1) + lgn n/2 lgn n/2 = Ω(nlgn) 9/22/2016 Data Structures and Algorithms Google Camp Fall 2016 Yanqiao ZHU 7

34 Linear-Time Sorting If n keys are integers (fitting in a word) {0, 1,, k 1}, we can do more than compare them. If k = n O(1), can sort in O(n) time. One conjecture is that for all k, can sort in linear time (not solved yet) Best algorithm: O((lglgn) 1/2 ) 9/22/2016 Data Structures and Algorithms Google Camp Fall 2016 Yanqiao ZHU 8

35 Counting Sort L = array of k empty lists linked list or Python lists for j in range n: L[key(A[j])].append(A[j]) O(1) random access using integer key output = [] for i in range k: output.extend(l[i]) O(k) O(n) O(Σ(1 + L[i] )) = O(k + n) Time and space complexity: Θ(n + k) Intuition: Count key occurrences using RAM output (count) copies of each key in order, but item is more than just a key 9/22/2016 Data Structures and Algorithms Google Camp Fall 2016 Yanqiao ZHU 9

36 Radix Sort Imagine each integer in base b d = log b k digits {0, 1,, b 1} Sort (all n items) by least significant digit [can extract in O(1) time] Sort by most significant digit [can extract in O(1) time] 9/22/2016 Data Structures and Algorithms Google Camp Fall 2016 Yanqiao ZHU 10

37 Radix Sort (cont.) Analysis: use counting sort for digit sort: Θ(n + b) per digit Θ[(n + b)d] = Θ[(n + b) log b k] total time minimized when b = n Θ(nlog n k) = O(nc) if k n c sort sorted sorted sorted 9/22/2016 Data Structures and Algorithms Google Camp Fall 2016 Yanqiao ZHU 11

38 Google Camp Fall 2016 Yanqiao ZHU

39 Cryptographic Hashing A cryptographic hash function is a deterministic procedure that takes an arbitrary block of data and returns a fixed-size bit string, the (cryptographic) hash value, such that an accidental or intentional change to the data will change the hash value. The data to be encoded is often called the message, and the hash value is sometimes called the message digest or simply digest. The ideal cryptographic hash function has the properties listed below. d is the number of bits in the output of the hash function. You can think of m as being 2 d. d is typically 160 or more. These hash functions can be used to index hash tables, but they are typically used in computer security applications. 9/22/2016 Data Structures and Algorithms Google Camp Fall 2016 Yanqiao ZHU 2

40 Desirable Properties One-Way (OW): Infeasible, given y R {0, 1} d to find any x s.t. h(x) = y. This means that if you choose a random d-bit vector, it is hard to find an input to the hash that produces that vector. This involves inverting the hash function. Collision-resistance (CR): Infeasible to find x, x, s.t. x x and h(x) = h(x ). This is a collision: two input values have the same hash. Target collision-resistance (TCR): Infeasible given x to find x = x s.t. h(x) = h(x ) TCR is weaker than CR. If a hash function satisfies CR, it automatically satisfies TCR. There is no implication relationship between OW and CR/TCR. 9/22/2016 Data Structures and Algorithms Google Camp Fall 2016 Yanqiao ZHU 3

41 Applications Password storage: Store h(pw), not PW on computer. When user inputs PW, compute h(pw ) and compare against h(pw). The property required of the hash function is OW. The adversary does not know PW or PW so TCR or CR is not really required. Of course, if many, many passwords have the same hash, it is a problem, but a small number of collisions doesn t really affect security. 9/22/2016 Data Structures and Algorithms Google Camp Fall 2016 Yanqiao ZHU 4

42 Applications (cont.) File modification detector: For each file F, store h(f) securely. Check if F is modified by recomputing h(f). The property that is required is TCR, since the adversary wins if he/she is able to modify F without changing h(f). 9/22/2016 Data Structures and Algorithms Google Camp Fall 2016 Yanqiao ZHU 5

43 Applications (cont.) Digital signatures: In public-key cryptography, Alice has a public key PK A and a private key SK A. Alice can sign a message M using her private key to produce σ = sign(sk A, M). Anyone who knows Alice s public key PK A and verify Alice s signature by checking that verify(m, σ, PK A ) is true. The adversary wants to forge a signature that verifies. For large M it is easier to sign h(m) rather than M, i.e., σ = sign(sk A, h(m)). The property that we require is CR. We don t want an adversary to ask Alice to sign x and then claim that she signed x, where h(x) = h(x ). 9/22/2016 Data Structures and Algorithms Google Camp Fall 2016 Yanqiao ZHU 6

44 Implementations There have been many proposals for hash functions which are OW, CR and TCR. Some of these have been broken. MD-5, for example, has been shown to not be CR. There is a competition underway to determine SHA-3, which would be a Secure Hash Algorithm certified by NIST. Cryptographic hash functions are significantly more complex than those used in hash tables. You can think of a cryptographic hash as running a regular hash function many, many times with pseudo-random permutations interspersed. 9/22/2016 Data Structures and Algorithms Google Camp Fall 2016 Yanqiao ZHU 7

45 Data Structures and Algorithms String Matching: KMP Google Camp Fall 2016 Yanqiao ZHU

46 Basic Requirements Given two strings s and t, does s occur as a substring of t? (and if so, where and how many times?) e.g. s = and t = your entire INBOX (grep on UNIX) 9/22/2016 Data Structures and Algorithms Google Camp Fall 2016 Yanqiao ZHU 2

47 Simple Algorithm 1. any(s == t[i : i + len(s)]) 2. for i in range(len(t) len(s))) Analysis: O( s ) time for each substring comparison O( s ( t s )) time = O( s t ) potentially quadratic t s s 9/22/2016 Data Structures and Algorithms Google Camp Fall 2016 Yanqiao ZHU 3

48 Knuth-Morris-Pratt Algorithm In the Brute-Force algorithm, if a mismatch occurs at pattern s[j] (j > 1), it only slides s to right by 1 step. It throws away one piece of information that we ve already known. Let u be the current shift value. Since it is a mismatch at s[j], we know s[u u + j 1] = s[1.. j 1] How can we make use of this information to make the next shift? In general, s should slide by u > u such that s[1.. k] = t[u u + k]. We then compare s[k + 1] with t[u + k + 1]. u + 1 u + j 1 t s u 1 j 9/22/2016 Data Structures and Algorithms Google Camp Fall 2016 Yanqiao ZHU 4

49 Knuth-Morris-Pratt Algorithm (cont.) When we slide s to right, it should be a place where s could possibly occur in t t b a c b a b a b a a b c b a b s u a b a b a c a 1 q 1 q a b a b a s[1.. q] t b a c b a b a b a a b c b a b a b a s[1.. k] is a suffix of s[1.. q] 1 k s u a b a b a c a 1 k 9/22/2016 Data Structures and Algorithms Google Camp Fall 2016 Yanqiao ZHU 5

50 The next Function Given s[1.. q] match text characters t[u u + q], what is the least shift u > u such that s[1.. k] = t[u u + k], where u + k = u + q? In practice, the shift u can be precomputed by comparing s against itself. Observe that t[u u + k] is a known text, and it is a suffix of s[1.. q]. To find the least shift u > u, it is the same as finding the largest k < q, s.t., s[1.. k] is a suffix of s[1.. q]. 9/22/2016 Data Structures and Algorithms Google Camp Fall 2016 Yanqiao ZHU 6

51 Computing next Function Given next[1], next[2],, next[q], how can we compute next[q + 1]? 1. If s[q + 1] == s[next[q] + 1], then next[q + 1] = next[q] + 1. s s 1 q next[q] + 1 q 1 next[q] 9/22/2016 Data Structures and Algorithms Google Camp Fall 2016 Yanqiao ZHU 7

52 Computing next Function (cont.) 2. If s[q + 1]!= s[next[q] + 1], then do what? s should slide to a place such that the prefix of [1.. next[q]] occurs as a suffix of [q next[q + 1].. q]; this information is stored in next[next[q]]. s s 1 q next[q] + 1 q 1 next[q] observe that s[1.. next[q]] = s[q next[q] q] s s 1 q next[q] + 1 q 1 next[next[q]] 9/22/2016 Data Structures and Algorithms Google Camp Fall 2016 Yanqiao ZHU 8

53 Computing next Function (cont.) 3. We first set next[1] = 0, then compute next[q] with q = 2, 3,, m, one by one in m 1 iterations. 9/22/2016 Data Structures and Algorithms Google Camp Fall 2016 Yanqiao ZHU 9

54 Computing next Function (cont.) Compute_Next(s) 1. m = s.length 2. next[1] = 0 // initialization 3. k = 0 // number of characters matched so far 4. for q = 2 to m do 5. while k > 0 and s[k + 1]!= s[q] do 6. k = next[k] 7. if s[k + 1] = s[q] then k = k next[q] = k 9. return next 9/22/2016 Data Structures and Algorithms Google Camp Fall 2016 Yanqiao ZHU 10

55 The KMP Algorithm Given s[1.. m], let next be a function {1, 2,, m} {0, 1,, m 1} such that next(q) = max{k : k < q and s[1.. k] is a suffix of s[1.. q]} Given next(q) for all 1 q m, we can use the KMP algorithm. 9/22/2016 Data Structures and Algorithms Google Camp Fall 2016 Yanqiao ZHU 11

56 The KMP Algorithm (cont.) KMP_StringMatcher(t, s) 1. n = t.length 2. m = s.length 3. next = Compute_Next(p) 4. q = 0 // number of characters matched 5. for i = 1 to n do // scan t from left to right 6. while q > 0 and s[q + 1]!= t[i] do 7. q = next[q] // next character does not match 8. if s[q + 1] = t[i] then 9. q = q + 1 // next character matches 10. if q = m then // is all of s matched? 11. print Pattern occurs with shift, i m 12. q = next[q] // look for the next match 9/22/2016 Data Structures and Algorithms Google Camp Fall 2016 Yanqiao ZHU 12

57 The KMP Algorithm (cont.) q s[q] a b a b a b c next(q) t i = 3, q = 0 a b a b a k s a b a b a b c 1 5 q + 1 s a b a b a b c 1 3 q + 1 s a b a b a 1 q + 1 s a b a b q + 1 i = 8, q = 5; s[q + 1]!= t[i] (enters the while loop) q assigns to next[5] (=3) i = 8, q = 3; s[q + 1]!= t[i] (in the while loop) q assigns to next[3] (=1) i = 8, q = 1; s[q + 1]!= t[i] (in the while loop) q assigns to next[1] (=0) i = 8, q = 0 (exits the while loop) 9/22/2016 Data Structures and Algorithms Google Camp Fall 2016 Yanqiao ZHU 13

58 Running Time The Compute_Next function: The for loop from line 4 to line 8 runs m times. Line 1 to line 3 take constant time. Hence the running time of compute next function is Θ(m). The KMP_StringMatcher function: The for loop beginning in line 5 runs n times, i.e., as long as the length of the string t. Since line 1 to line 4 take constant time, the running time is dominated by this for loop. Thus running time of matching function is Θ(n). 9/22/2016 Data Structures and Algorithms Google Camp Fall 2016 Yanqiao ZHU 14