T(n) = expected time of algorithm over all inputs of size n. half the elements in A[1.. j 1] are less than A[ j ], and half the elements are greater.

Transcription

1 Algorithms Design and Analysis Definitions: An algorithm: It is any well-defined computational procedure that takes some value, or set of values, as input and produces some value, or set of values, as output. It is thus a sequence of computational steps that transform the input into the output. Analysis: predict the cost of an algorithm in terms of resources and performance Design: design algorithms which minimize the cost An instance of a problem: consists of the input (satisfying whatever constraints are imposed in the problem statement) needed to compute a solution to the problem. A data structure is a way to store and organize data in order to facilitate access and modifications. The keys: The numbers that we wish to sort. Insertion sort: is an efficient algorithm for sorting a small number of elements. Asymptotic notation: is used to classify algorithms by how they respond to changes in input size. Notes: An algorithm is said to be correct if, for every input instance, it halts with the correct output. What kinds of problems are solved by algorithms? The Human Genome Project - has the goals of identifying all the genes in human DNA, determining the sequences of the chemical base pairs that make up human DNA - storing this information in databases, and developing tools for data analysis.. The Internet enables people all around the world to quickly access and retrieve large amounts of information. (to manage and manipulate this large volume of data) Electronic commerce. In manufacturing and other commercial settings. Hard problems: NP-complete problems (decision problem) is unknown whether or not efficient algorithms exist for NPcomplete problems. Algorithms as a technology: memory may be cheap, but it is not free. Computing time is therefore a bounded resource, and so is space in memory. These resources should be used wisely, and algorithms that are efficient in terms of time or space will help you do so. Which computer/ algorithm is faster? The example of two computers and two algorithms fast/slow. Analyzing algorithms: predicting the resources that the algorithm requires. random-access machine (RAM)- Instructions are (executed one after another, with no concurrent operations). The data types in the RAM model are integer and floating point and limit on the size of each word of data. Is exponentiation a constant time instruction? shift left instruction. Running time: The running time depends on the input: an already sorted sequence is easier to sort. Major Simplifying Convention: Parameterize the running time by the size of the input, since short sequences are easier to sort than long ones. T A (n) = time of A on length n inputs. Generally, seek upper bounds on the running time, to have a guarantee of performance. Kinds of analyses: Worst-case: (usually) Average-case: (sometimes) Best-case: T(n) = maximum time of algorithm on any input of size n. (upper bound on the running time) T(n) = expected time of algorithm over all inputs of size n. T(n) = minimum time of algorithm ((fastest time to complete, with optimal inputs chosen) (lower bound on the running time)) in which the input array was reverse sorted Naseebah Malaibari - CS page (1) half the elements in A[1.. j 1] are less than A[ j ], and half the elements are greater. in which the input array was already sorted

2 Insertion sort: The time taken by the INSERTION-SORT procedure depends on the input. INSERTION SORT can take different amounts of time: - to sort two input sequences of the same size depending on how nearly sorted they already are. - to sort thousand numbers or three numbers. The running time of an algorithm on a particular input is the number of primitive operations or steps executed. Algorithm: Worst case t j = j, O(n 2 ) Best case Average case t j = 1 O(n) we check half of the subarray A[1.. j 1], so t j = j/2. The divide-and-conquer approach: Divide the n-element sequence to be sorted into two subsequences of n/2 elements each. The three steps of divide-and-conquer paradigm at each level of the recursion: Divide the problem into a number of subproblems. Conquer the subproblems by solving them recursively. If the subproblem sizes are small enough, however, just solve the subproblems in a straightforward manner. Combine the solutions to the subproblems into the solution for the original problem. Naseebah Malaibari - CS page (2)

3 Merge sort: is closely follows the divide-and-conquer paradigm. Divide: Divide the n-element sequence to be sorted into two subsequences of n/2 elements each. Conquer: Sort the two subsequences recursively using merge sort. Combine: Merge the two sorted subsequences to produce the sorted answer. The recursion bottoms out when the sequence to be sorted has length 1, in which case there is no work to be done, since every sequence of length 1 is already in sorted order. T(n) Recurrence Equation Q(1) 2T(n/2) Q(n) Time = (n) to merge a total of n elements (linear time). Divide: The divide step just computes the middle of the subarray, which takes constant time. D(n) = (1). Conquer: We recursively solve two subproblems, each of size n/2, which contributes 2T (n/2) to the running time. Combine: We have already noted that the MERGE procedure on an n-element subarray takes time (n), so C(n) = (n). Solve T(n) = 2T(n/2) + cn, where c > 0 is constant: high of tree = lg n. No. of levels = lg n + 1. cost for merge sort = cn (lgn+1) = cn lgn +cn merge sort running time = (n lg n) (n lg n) grows more slowly than (n2). Therefore, merge sort asymptotically beats insertion sort in the worst case. In practice, merge sort beats insertion sort for n > 30 or so. Naseebah Malaibari - CS page (3)

4 Asymptotic notation: Notation characterizes functions according to their growth rates: different functions with the same growth rate may be represented using the same notation. Naseebah Malaibari - CS page (4)

5 Recurrences: is an equation or inequality that describes a function in terms of its value on smaller inputs. Recurrence Methods: 1. Substitution Method. 2. Recursion-tree method. 3. Master method. The substitution method: The substitution method for solving recurrences طريقة احلل: entails two steps: 1( نضع guess أو هو يعطينا هو يف السؤال. 1. Guess the form of the solution. 2( نفرض أهنا صحيحة عند املشكلة الصغرية: 2. Use mathematical induction to find the constants and show that the solution works. نضع التعريف. The substitution method can be used to establish نعوض عنكل n يف التعريف ابملشكلة الصغرية. either upper or lower bounds on a recurrence. 3( نعوض عن الفرضية يف املعادلة األصلية. T(n) نوجد C حبيث : املشكلة الكبرية The recursion-tree method: A recursion tree is best used to generate a good guess, which is then verified by the substitution method. The master method: The recurrence (4.5) describes the running time of an algorithm that divides a problem of size n into a subproblems, each of size n/b, where a and b are positive constants. The a subproblems are solved recursively, each in time T (n/b). Naseebah Malaibari - CS page (5)

6 The cost of dividing the problem and combining the results of the subproblems is described by the function f (n). Priority Queue: A data structure implementing a set S of elements, each associated with a key, supporting the following operations: Operation Explanation insert(s, x) insert element x into set S max(s) return element of S with largest key extract_max(s) return element of S with largest key and remove it from S increase_key(s, x, k) increase the value of element x s key to new value k Heap sort: Like merge sort Like insertion sort Heap sort s running time is O(n lg n) heap sort sorts in place: only a constant number of array elements are stored outside the input array at any time. Heap: The (binary) heap data structure is an array object that can be viewed as a nearly complete binary tree. *Each node of the tree corresponds to an element of the array that stores the value in the node. A heap is an almost-complete binary tree A complete binary tree: is a binary tree in which every level, except possibly the last, is completely filled, and all nodes are as far left as possible. An array A that represents a heap is an object with two attributes: length[a], which is the number of elements in the array, and heap-size[a], the number of elements in the heap stored within array A. That is, although A[1.. length[a]] may contain valid numbers, no element past A[heap-size[A]], where heap-size[a] length[a], is an element of the heap. Height of node = # of edges on a longest simple path from the node down to a leaf. Height of heap = height of root = (lg n). A heap can be stored as an array A. Root of tree is A[1]. Parent of A[i ] = A. Left child of A[i ] = A[2i ]. Right child of A[i ] = A[2i + 1]. Notes: The basic operations on heaps run in time at most proportional to the height of the tree and thus take O(lg n) time. Naseebah Malaibari - CS page (6)

7 Kinds of binary heaps: max-heaps min-heaps (largest element at root), max-heap property: (smallest element at root), min-heap property: for for all nodes i, excluding the root, A[PARENT(i )] all nodes i, A[i ]. excluding the root, A[PARENT(i )] A[i ]. The value of a node is at most the value of its parent. The sub-tree rooted at a node contains values no larger than that contained at the node itself. In both kinds, the values in the nodes satisfy a heap property, the specifics of which depend on the kind of heap. Number of Nodes: Minimum Number Of Nodes Maximum Number Of Nodes At least one node at each level all possible nodes at first h level are represent HEAP OPERATIONS: The MAX-HEAPIFY procedure, which runs in O(lg n) time, is the key to maintaining the maxheap property. The BUILD-MAX-HEAP procedure, which runs in linear time, produces a max_heap from an unordered input array. The HEAPSORT procedure, which runs in O(n lg n) time, sorts an array in place. BUILD-MAX-HEAP(A) takes Θ(n), MAX-HEAPIFY(A,1) takes Θ(lgn) and repeated n 1 times. Operation Explanation max_heapify correct a single violation of the heap property in a sub-tree at its root build_max_heap produce a max-heap from an unordered array The children s sub-trees each have size at most 2n/3 The solution to this recurrence, by case 2 of the master theorem (Theorem 4.1), is T (n) = O(lg n). Naseebah Malaibari - CS page (7)

8 Build_Max_Heap(A) Analysis: 1. Construct the heap in linear time. In particular, the BuildHeap procedure actually runs in Θ(n) time. 2. Execute the loop and perform a logarithmic time operation n 1 times. 3. Other operations take constant time. Hence, your running time is Naseebah Malaibari - CS page (8)

9 There are two factors at work: the time it takes to create a heap by adding each element and the time it takes to remove all of the elements from a heap A naive implementation requires additional space, but it is possible to do a heap sort in place. Heap sort has guaranteed O(n log(n)) performance, though the constant factor is typically a bit higher than for other algorithms such as quicksort. Heap sort is not a stable sort. The running time of BUILD-MAX-HEAP = O(n): Recursion-tree method: Description of quicksort: The basic idea behind quicksort is: partition; sort one half; sort the other half. Quicksort is a sorting algorithm whose worst-case running time is O (n 2 ) on an input array of n numbers. Quicksort, like merge sort, is based on the divide-and-conquer paradigm. Steps - divide-and-conquer process for sorting a typical sub-array A[p.. r]: Divide: Partition (rearrange) the array A[p.. r] into two (possibly empty) subarrays A[p.. q 1] (element is less than or equal to A[q])and A[q +1.. r], Compute the index q as part of this procedure. Naseebah Malaibari - CS page (9)

10 Conquer: Sort the two subarrays A[p.. q 1] and A[q +1.. r] by recursive calls to quicksort. Combine: Since the subarrays are sorted in place, no work is needed to combine them: the entire array A[p.. r] is now sorted. The steps are: Pick an element, called a pivot, from the list. Reorder the list so that all elements with values less than the pivot come before the pivot, while all elements with values greater than the pivot come after it. After this partitioning, the pivot is in its final position. This is called the partition operation. Recursively sort the sub-list of lesser elements and the sub-list of greater elements. Performance of quicksort: The running time of quicksort depends on whether the partitioning is balanced or unbalanced, and this in turn depends on which elements are used for partitioning. If the partitioning is balanced, the algorithm runs asymptotically as fast as merge sort. If the partitioning is unbalanced, however, it can run asymptotically as slowly as insertion sort. Notes: In the most unbalanced case, each time we perform a partition we divide the list into two sub lists of size 0 and n 1. In the most balanced case, each time we perform a partition we divide the list into two nearly equal pieces. In fact, it's not necessary to be perfectly balanced; even if each pivot splits the elements with 75% on one side and 25% on the other side (or any other fixed fraction). Naseebah Malaibari - CS page (10)

11 Worst Case Best Case The two partitions will always be about the same size as each other, and Quicksort will take O(n lg n) execution time. Average Case Naseebah Malaibari - CS page (11)

12 Analysis of quicksort: What is the proof of O (n 2 )? Naseebah Malaibari - CS page (12)

13 What is the proof of O (n lg n)? Why is quicksort better than other sorting algorithms in practice? (Efficient Sorting Algorithm) It is one of the most common sorting algorithms for sequential computers because of its simplicity, low overhead, and optimal average complexity. Quicksort is O(n lg n) on average and O(n 2 ) in the worst case. At the same time, other sorting algorithms are O(n lg n) in the worst case (like mergesort and heapsort). A very good partition splits an array up into two equal sized arrays. A bad partition, on other hand, splits an array up into two arrays of very different sizes. The worst partition puts only one element in one array and all other elements in the other array. Naseebah Malaibari - CS page (13)

14 Quicksort Choosing a pivot : Choosing a random pivot minimizes the chance that you will encounter worst-case O(n 2 ) performance (Choosing first or last would cause worst-case performance for nearlysorted or nearly-reverse-sorted data). Choosing the middle element would also be acceptable in the majority of cases SCIENTIFIC ADVICE: Never ever choose a fixed pivot - this can be attacked to exploit your algorithm's worst case O(n 2 ) runtime. Suppose you choose the first element as your partition. If someone feeds an array to your algorithm that is in decreasing order, your first pivot will be the biggest, so everything else in the array will move to the left of it. Then when you recursive, the first element will be the biggest again, so once more you put everything to the left of it, and so on. A better technique is the median-of-3 method, where you pick three elements at random, and choose the middle. You know that the element that you choose won't be the first or the last, but also, by the central limit theorem, the distribution of the middle element will be normal, which means that you will tend towards the middle (and hence, n lg n time). Search trees: Search trees are data structures that support many dynamic-set operations, including SEARCH, MINIMUM, MAXIMUM, PREDECESSOR, SUCCESSOR, INSERT, and DELETE. Basic operations on a binary search tree take time proportional to the height of the tree. o For a complete binary tree with n nodes, such operations run in O(lg n) = height worst-case time. o If the tree is a linear chain linear time algorithm of n nodes, however, the same operations take O (n) worst-case time. o The height of the Binary Search Tree equals the number of links from the root node to the deepest node. What is a binary tree? Property 1: each node can have up to two successor nodes (children). The predecessor node of a node is called its parent. The "beginning" node is called the root (no parent). A node without children is called a leaf. Property 2: a unique path exists from the root to every other node. a tree can be represented by a linked data structure in which each node is an object. key field: In addition to a key field and satellite data, each node contains fields left, right, and p that point to the nodes corresponding to its left child, its right child, and its parent, respectively. If a child or the parent is missing, the appropriate field contains the value NIL. The root node is the only node in the tree whose parent field is NIL. The keys in a binary search tree are always stored in such a way as to satisfy the binary-search-tree property: Naseebah Malaibari - CS page (14)

15 The binary-search-tree property allows us to print out all the keys in a binary search tree in sorted order by a simple recursive algorithm: Inorder tree walk preorder tree walk postorder tree walk Print the root of a subtree between the values in its left subtree and the right subtree. prints the root before the values in either subtree prints the root after the values in its subtrees It takes O(n) time to walk a tree of n nodes Naseebah Malaibari - CS page (15)

16 Querying a binary search tree: Search the BST x for a value k: To search for the key 13 in the tree, we follow the path from the root. Note: Search operation takes time O(h) proportional to the height of the tree, where h is the height of a BST. Minimum and maximum: Minimum Maximum Note: max and min operations take time O(h) proportional to the height of the tree, where h is the height of a BST Successor and predecessor: in binary search tree (t): Successor Predecessor Naseebah Malaibari - CS page (16)

17 - If the right subtree of node x is nonempty, then the successor of x is just the leftmost node in the right subtree - If the right subtree of node x is empty and x has a successor y, then y is the lowest ancestor of x whose left child is also an ancestor of x. Insertion and deletion: take time proportional to the height of the tree O(h) Deletion: Deleting a node z from a binary search tree. Which node is actually removed depends on how many children z has; this node is shown lightly shaded. (a) If z has no children, we just remove it. (b) If z has only one child, we splice out z. (c) If z has two children, we splice out its successor y, which has at most one child, and then replace z s key and satellite data with y s key and satellite data. Naseebah Malaibari - CS page (17)

18 Conclusion: Binary search trees come in many shapes. The shape of the tree determines the efficiency of its operations. Logarithmic time is generally much faster than linear time. For example, for n = 1,000,000: log2 n = 20. Using binary search trees to represent sets: insert, search, remove, O(h) = O(log2 n) Using linear data structures to represent sets: insert, search, remove, O(n) Balanced BTS: Easier insertion/deletion Running time = O(log2 n) The important techniques for the design and analysis of efficient algorithms: dynamic programming. greedy algorithms. Earlier parts have presented other widely applicable techniques: divide-and-conquer. randomization. the solution of recurrences. Dynamic programming: is an optimization approach that transforms a complex problem into a sequence of simpler problems. It is used in many areas such as Computer science theory, graphics, AI, systems,.etc. Typically applies to optimization problems in which a set of choices must be made in order to arrive at an optimal solution. Each solution has a value, and we wish to find a solution with the optimal (minimum or maximum) value. We call such a solution an optimal solution to the problem, as opposed to the optimal solution, since there may be several solutions that achieve the optimal value. It is effective when a given subproblem may arise from more than one partial set of choices; the key technique is to store the solution to each such subproblem in case it should reappear. It likes the divide-and-conquer method, solves problems by combining the solutions to subproblems. Divide and- conquer algorithms partition the problem into independent subproblems, solve the subproblems recursively, and then combine their solutions to solve the original problem. Dynamic programming is applicable when the subproblems are not independent, that is, when subproblems share subsubproblems. A dynamic-programming algorithm solves every subsubproblem just once and then saves its answer in a table, thereby avoiding the work of recomputing the answer every time the subsubproblem is encountered. The development of a dynamic-programming algorithm: the sequence of four steps: 1. Characterize the structure of an optimal solution. 2. Recursively define the value of an optimal solution. 3. Compute the value of an optimal solution in a bottom-up fashion. 4. Construct an optimal solution from computed information. Naseebah Malaibari - CS page (18)

19 Matrix-chain multiplication: We can multiply two matrices A and B only if they are compatible: the number of columns of A must equal the number of rows of B. If A is a p q matrix and B is a q r matrix, the resulting matrix C is a p r matrix Counting the number of parenthesizations: Step 1: The structure of an optimal parenthesization Step 2: A recursive solution Step 3: Computing the optimal costs Step 4: Constructing an optimal solution Naseebah Malaibari - CS page (19)

20 Optimal binary search trees: Naseebah Malaibari - CS page (20)

21 Steps of Dynamic programming : 1. Characterize the structure of an optimal solution. 2. Recursively define the value of an optimal solution. 3. Compute the value of an optimal solution in a bottom-up fashion. 4. Construct an optimal solution from computed information. Summary: Dynamic programming is an optimization approach that transforms a complex problem into a sequence of simpler problems. Dynamic programming is effective when a given subproblem may arise from more than one partial set of choices Dynamic programming is applicable when the subproblems are not independent, that is, when subproblems share subsubproblems. Naseebah Malaibari - CS page (21)

22 Greedy algorithms: Like dynamic-programming algorithms, greedy algorithms typically apply to optimization problems in which a set of choices must be made in order to arrive at an optimal solution. The idea of a greedy algorithm is to make each choice in a locally optimal manner. Always makes the choice that looks best at the moment. They do not always yield optimal solutions, but for many problems they do. The greedy method is quite powerful and works well for a wide range of problems. Greedy versus dynamic programming: Backtracking: Algorithm structure: Iterative execute action in loop. Recursive reapply action to subproblem(s). Problem type: Satisfying find any satisfactory solution Optimization find best solutions Suppose you have to make a series of decisions, among various choices, where: You don t have enough information to know what to choose Each decision leads to a new set of choices Some sequence of choices (possibly more than one) may be a solution to your problem Backtracking is a form of recursion: The usual scenario is that you are faced with a number of options, and you must choose one of these. After you make your choice you will get a new set of options; just what set of options you get depends on what choice you made. This procedure is repeated over and over until you reach a final state. If you made a good sequence of choices, your final state is a goal state; if you didn't, it isn't. Conceptually, you start at the root of a tree; the tree probably has some good leaves and some bad leaves, though it may be that the leaves are all good or all bad. You want to get to a good leaf. At each node, beginning with the root, you choose one of its children to move to, and you keep this up until you get to a leaf. Suppose you get to a bad leaf. You can backtrack to continue the search for a good leaf by revoking your most recent choice, and trying out the next option in that set of options. If you run out of options, revoke the choice that got you here, and try another choice at that node. If you end up at the root with no options left, there are no good leaves to be found. Strategy: Starting at Root, your options are A and B. You choose A. At A, your options are C and D. You choose C. C is bad. Go back to A. At A, you have already tried C, and it failed. Try D. D is bad. Go back to A. At A, you have no options left to try. Go back to Root. At Root, you have already tried A. Try B. At B, your options are E and F. Try E. E is good. Well done! Naseebah Malaibari - CS page (22)

23 knapsack: 0-1 knapsack problem: Fractional knapsack problem A thief robbing a store finds n items; the ith item is worth v i dollars the setup is the same, but the and weighs w i pounds, where v i and w i are integers. He wants to take thief can take fractions of as valuable a load as possible, but he can carry at most W pounds in items, rather than having to his knapsack for some integer W. Which items should he take? (This is make a binary (0-1) choice for called the 0-1 knapsack problem because each item must either be each item. taken or left behind; the thief cannot take a fractional amount of an item or take an item more than once.) Huffman codes: A binary character code (or code for short) wherein each character is represented by a unique binary string. A fixed-length code, we need 3 bits to represent six characters. A variable-length code can do considerably better than a fixed-length code, by giving frequent characters short codewords and infrequent characters long codewords. o The table shows such a code; here the 1-bit string 0 represents a, and the 4-bit string 1100 represents f. This code requires ( ) 1,000 = 224,000 bits. Prefix codes (no codeword) Encoding is always simple for any binary character code; we just concatenate the codewords representing each character of the file. For example, with the variablelength prefix code of the table, we code the 3-character file abc as = , where we use to denote concatenation. Naseebah Malaibari - CS page (23)

24 We interpret the binary codeword for a character as the path from the root to that character, where 0 means go to the left child and 1 means go to the right child. Figure 16.4 shows the trees for the two codes of our example. Note that these are not binary search trees, since the leaves need not appear in sorted order and internal nodes do not contain character keys. Constructing a Huffman code: For our example, Since there are 6 letters in the alphabet, the initial queue size is n = 6, and 5 merge steps are required to build the tree. The final tree represents the optimal prefix code. The codeword for a letter is the sequence of edge labels on the path from the root to the letter. Line 2 initializes the min-priority queue Q with the characters in C. The for loop in lines 3 8 repeatedly extracts the two nodes x and y of lowest frequency from the queue, and replaces them in the queue with a new node z representing their merger. The frequency of z is computed as the sum of the frequencies of x and y in line 7. The node z has x as its left child and y as its right child. (This order is arbitrary; switching the left and right child of any node yields a different code of the same cost.) After n 1 mergers, the one node left in the queue the root of the code tree is returned in line 9. Naseebah Malaibari - CS page (24)

25 Correctness of Huffman s algorithm: Problem space consists of states (nodes) and actions (paths that lead to new states). When in a node can only see paths to connected nodes. If a node only leads to failure go back to its "parent" node. Try other alternatives. If these all lead to failure then more backtracking may be necessary. Summary: A greedy algorithm always makes the choice that looks best at the moment. Greedy algorithms do not always yield optimal solutions, but for many problems they do. Backtracking is a form of recursion. Backtracking - A scheme for solving a series of sub-problems each of which may have multiple possible solutions and where the solution chosen for one sub-problem may affect the possible solutions of later sub-problems. To solve the overall problem, we find a solution to the first sub-problem and then attempt to recursively solve the other sub-problems based on this first solution. If we cannot, or we want all possible solutions, we backtrack and try the next possible solution to the first sub-problem and so on. Backtracking terminates when there are no more solutions to the first sub-problem. Naseebah Malaibari - CS page (25)

26 File compression techniques: For reducing the number of bytes in an image file, but the effectiveness of particular compression method depend on the type of an image. Run Length Encoding LZW Encoding Huffman Encoding Repeating factor Pattern recognition Variable length code تكرار الرقم وجود النمط طول الكلمة Naseebah Malaibari - CS page (26)

27 Naseebah Malaibari - CS page (27)

28 Naseebah Malaibari - CS page (28)