1 5,9,2,7,6,10,4,3,8,1 The first number (5) is automatically the first number of the sorted list

Size: px

Start display at page:

Download "1 5,9,2,7,6,10,4,3,8,1 The first number (5) is automatically the first number of the sorted list"

Corey Green
5 years ago
Views:

1 Algorithms One of the more challenging aspects of Computer Science are algorithms. An algorithm is a plan that solves a problem. When assembling a bicycle based on the included instructions, in this case, the instructions are the algorithm. The expression goes, There are many ways to skin a cat, meaning there are multiple ways to solve pretty much any problem. The key is, to find the best algorithm or solution for a specific problem. Let s take a typical problem. I drop my contact list, consisting of a bunch of cards, on the floor, and now must pick them up and put them back in alphabetical order. What is the quickest way to do this? See figure below: Sorting a list of cards Insertion Sort Most people will take all the cards in their right hand and slowly build a sorted list in their left hand (or in one hand having my finger separate between the sorted and the unsorted list). As we move one card at a time from the unsorted to sorted list, we place the new card in its correct location in the sorted list, But, just how do we find the correct location? Well since we are inserting into a sorted list, we will see that we can find the correct spot using a binary search. Alternatively, we can do a simple traversal as we will now demonstrate. Assuming the list contains

2 the following numbers, 5,9,2,7,6,10,4,3,8,1. We will build a sorted list one number at a time. See below. Note: We have not specified how we find the location for the new number. Insertion Sort = Unsorted ---- = Sorted STEP LIST COMMENT Start 5,9,2,7,6,10,4,3,8,1 1 5,9,2,7,6,10,4,3,8,1 The first number (5) is automatically the first number of the sorted list 2 5,9,2,7,6,10,4,3,8,1 The next unsorted (9) is inserted. It is greater than the last sorted number so it becomes last number of the sorted list. 3 2,5,9,7,6,10,4,3,8,1 The next unsorted (2) is inserted. It is less than 9 and 5 so they are shifted right and 2 is placed in the beginning. 4 2,5,7,9,6,10,4,3,8,1 7 is added and 9 is shifted one place 10 1,2,3,4,5,6,7,89,10 List is sorted after tenth insertion Bubble Sort Another method is to find the card that should be last, e.g. it has the highest number, and place that last. Next we the next highest card and place that before the highest card and we slowly build a sorted list from highest to lowest. Each time we need to find a new card we must traverse the whole list of unsorted cards. A variation of this method is Bubble Sort. With the Bubble Sort algorithm, we do indeed always have the next highest card after each pass through the list,

3 but we bubble this value up. By bubbling, we partially sort all the numbers as we find the highest number. To explain, we ll use a list with ten numbers that require sorting. The list is 5,9,2,7,6,10,4,3,8,1. With Bubble Sort, we look only two numbers at time. First we examine the first two numbers, then the second and third, and so on until we have examined the ninth and tenth. Each time we examine two numbers, we see if they are in order, if they aren t we switch them. The first time through the list, the highest number will bubble out, and the second time the next highest number will bubble out. Each time we go through the list one less time, as our unsorted numbers decreased by one each time. We have a sorted list when we loop through the unsorted list and no switches have to be made. In the example below, the unsorted portion of the list is blue and the sorted portion is green = Unsorted ---- = Sorted Start - 5,9,2,7,6,10,4,3,8,1 Pass 1-9,2,5,6,7,4,3,8,1,10 Bubble Highest Pass 2-2,5,6,7,4,3,8,1,9,10 Bubble Next Highest Pass 3-2,5,6,4,3,7,1,8,9,10 Bubble Next Highest Pass 4-2,5,4,3,6,1,7,8,9,10 Bubble Next Highest Pass 5-2,4,3,5,1,6,7,8,9,10 Bubble Next Highest Pass 6-2,3,4,1,5,6,7,8,9,10 Bubble Next Highest Pass 7-2,3,1,4,5,6,7,8,9,10 Bubble Next Highest Pass 8-2,1,3,4,5,6,7,8,9,10 Bubble Next Highest Pass 9-1,2,3,4,5,6,7,8,9,10 Bubble Next Highest Alternating Bubble Sort As you can see Passes 6 through 9 are only to move the number one to the first position. Imagine if we had a million numbers all in order except the last one was a one, we would require a million passes just to sort one number! Below, I have looked to see what would happen if we Bubble Sort alternately from left to right,

4 to first find the highest number, and then from right to left to find the lowest number. As you can see, we have cut down the number of passes to from 9 to = Unsorted ---- = Sorted Start - 5,9,2,7,6,10,4,3,8,1 Pass 1 5,2,7,6,9,4,3,8,1,10 Bubble Highest Pass 2 1,5,2,7,6,9,4,3,8,10 Bubble Lowest Pass 3-1,2,5,6,7,4,3,8,9,10 Bubble Next Highest Pass 4-1,2,3,5,6,7,4,8,9,10 Bubble Next Lowest Pass 5-1,2,3,5,6,4,7,8,9,10 Bubble Next Highest Pass 6-1,2,3,4,5,6,7,8,9,10 Bubble Next Lowest Partition Sort This algorithm built around the concept that we can quickly sort something if we continuously partially sort the list of items to be sorted. If we partition a list into three parts, the part less than the partition, the partition, and the list of items larger than the partition. One of the nice things about partition sort is, after I partition a list I have two lists, one whose members are all less than the partition and one whose members are larger. Therefore, I can safely sort each list independently of the other and combine them together for one sorted list. This has huge implications. Nowadays, most computers have multiple processors usually four. A well written Partition Sort will take advantage of this, and will be able to sort four times as fast! For example, if we take the list 5,9,2,7,6,10,4,3,8,1, we can divide it into 3 parts, those numbers that are less than 6, the number 6, and the numbers greater than 6. See below.

5 Left Side Right Side Partition {5,9,2,7,6,10,4,3,8,1} = {5,2,4,3,1} {9,7,10,8} Each list in turn can again be further partioned. {2,1} {5,4} {} {9,10,8} And again. {} {2} {} {5} {} {8} {10} Merge Sort Merge Sort uses the fact that it is relatively easy to combine two lists if each list is already sorted. So we take our list, break it down to the smallest possible lists and then recombine them. What makes merge sort of interest, is that the number of steps required to sort the list is always known. It is a function of the size of the list. Specifically, it is Log2(N). So where N=10, we will need only 3 steps to recombine. The example below illustrates how Merge Sort works. {5,9,2,7,6,10,4,3,8,1} Original List {5,9} {2,7} {6,10} {3,4} {1,8} Step one break into lists of 1 or two 2 sorted items. {2,5,79} {3,4,6,10} {1,8} Step two recombine. {2,3,4,5,6,7,9,10} {1,8} Step three recombine. {1,2,3,4,5,6,7,8,9,10} Step four recombine. Comparing Algorithms The major aspect of any algorithm, is of course, the likelihood that our algorithm will be correct (or very close to correct). In most cases the assumption is that every algorithm compared will be 100% correct 100% of the time. The second aspect of algorithms is how long will this algorithm take to complete. To make comparisons of algorithms we usually say algorithm speed is equivalent to the number of computer operations needed by the algorithm to complete in the best, average, and worst case scenarios. Best and worst case scenarios are cases where the data analyzed affects the algorithm speed. For example, the

6 Bubble Sort algorithm will take very little time when the data is already sorted while Merge Sort will always take the same time any type of data. So to compare apples to apples, we usually have worst case, best case, and average case scenarios for algorithms. In fact, one interesting algorithm is; Start with Bubble Sort, if the data is a worst case scenario, switch to a sort which works well with all scenarios, like a Merge Sort. Big O Notation When we compare modes of transportation, we create categories. For example, we would never compare speed of a car with speed of an airplane. We may make speed categories like runner, four wheels, airplane, rocket ship, speed of light. Now let s take two runners, Runner A can run the mile in five minutes and Runner B can run a mile in 20 minutes. So we say, Runner A is four times faster than Runner B. Compared to an airplane, however, neither runner can be compared! People and airplanes are in two completely different leagues. To bring this comparison to algorithms, we must keep in mind that computers perform over one billion calculations a second. So Algorithm A may be four times faster than Algorithm B, but in the grand scheme of things, this isn t such a big deal. For example, 10 million operations or 40 million operations doesn t have much effect on a processor running at one billion calculations a second. What does makes an algorithm much slower is exponential growth, for example, number of calculations is N^2. This makes the difference whether an algorithm is a runner or an airplane. For example, if we sort a million numbers using bubble sort, in the worst case scenario it will make N^2 or one trillion operations. Merge sort will take 20 million operations. Translating this into computer time (at 1 billion calculations a second) we get ½ an hour vs less than 1 second. Another point, when discussing algorithms, is their order of growth. That is, to go one block, we don t really care if we walk, drive, or fly. The time differential is very small. What we are interested in is what happens if we have to travel 1000 miles. So what we are really interested in is the order of growth as the problem size increases. In Computer Science, this is known as Bog O Notation, with O referring to the Order of growth.

7 Below are common big O values. Big O classes in order of growth (from Big O Growth Name O(1) O(log(n)) O((log(n))^c) O(n) O(n^2) O(n^c) O(c^n) constant logarithmic polylogarithmic linear quadratic polynomial exponential Hard Problems Hard problems are those problems that can only be solved through brute force, there is no quick way to solve them. The most famous example of a hard problem is the Traveling Salesman Problem (or TSP). The Traveling Salesman Problem is as follows. A salesman needs to visit many cities and then return to his home city. Naturally, he would like to do this in the least amount of time, so he can return home and get more work done. Let s say the salesman wishes to visit 20 cities. On way to figure out the fastest route is to examine every possible combination of paths, and find which one is the shortest. Is this a good solution? Well to visit twenty cities there are 20 (20!) factorial possibilities which is equal to 2,432,902,008,176,640,000. A computer running at one billion calculations per second would take around 15 years to find the shortest route. Graphs

8 As we have mentioned elsewhere, many common situations can be represented as a graph. For example, the internet, tasks to be done, and maps can all be represented as graphs. So algorithms that deal with graphs will prove very useful. Our first problem is; We need to visit every vertex on the graph, only once, how do we accomplish this? We will cover two ways to conduct this traversal. Depth First Search (DFS) or Breadth First Search (BFS). Undirected Graph Depth First Search (DFS) In the figure above, we have Vertices A F. Let s pick A as the start vertex. The DFS then goes something like this: 1) Find the first unvisited ver Search Algorithms Searching for data If we are looking at disorganized data, there is only one way to find what we are looking for. That is, we look sequentially at the first item, and then the second item, etc. For example, if we go to a garage sale and are looking through a bunch of old baseball cards we go through them one by one. Or if we have a bunch of papers in a drawer we look for the document we are interested, paper by paper.

9 On the other hand, if we are looking for a pair of clean socks, in our bedroom, we know to first look in the sock drawer. That is because the clothes are organized. The same thing with data. If the data is organized, then we can perform an intelligent search. What are some intelligent ways to store data? Sorted array is probably the quickest to search but adding data is costly because we have to rearrange the data to always be sorted. Another data structure is a binary search tree. In this case the root has the middle value and all the children to a parents left are smaller or equal than their parent. The advantage of a binary tree is that adding items is pretty quick but the tree can become unbalanced and have to be rebalanced. The search time for either search is Binary Search Log(n) 32 nodes 5 (worst case) Searching for a path Sometimes I am not looking for a single element. Rather I am looking for a path. GPS - I am looking for the shortest/quickest path between two addresses. Traveling Salesman (TSP) - I am looking for the total shortest/quickest path between many cities. See TSP link for more about this problem. LinkedIn - I am looking for the total shortest/quickest connection between two people. These cases can be represented as graphs. A graph is basically a collection vertices and edges that connect them. In the first two examples, there is also associated with each edge, a value denoting the cost associated with each edge. The challenge with graph problems is that there are often millions of possibilities. Dijkstra's algorithm - calculates the shortest path.

10 Sorting algorithms Below are examples of sorting algorithms. They can be used to sort any object for which we can determine a greater than, less than, and equal operation. For simplicity, the examples below are using whole numbers or integers and we are sorting lowest to highest. Quick Sort (Partition Sort) Pick a value. Assume it is the middle value. Move everything lower to its left and everything larger to its right. Then do the same for the left hand side and the right hand side. Merge Sort Step A If there are only one member in the list then it is sorted. If there are two members, then switch them and the list is sorted. We are done sorting. Step B Break down list into two sorted child lists and then recombine. To create sorted child lists, divide list onto two halves and go to Step A. Insertion Sort Starts with a new list which is sorted, or an empty list. Taking one element at a time from the unsorted list, and add to the sorted list. A sorted list is built. Note: because list is always sorted, adding a new element can be done faster. Can you explain why? Bubble Sort This sort after the following property. The highest number is guaranteed to bubble out after the first pass. The second highest number is guaranteed to bubble out after the second pass and so on.

11 Each pass through the list, each element is compared with the element after it. If the first number is greater, then the two numbers are switched. The list is sorted after a pass has been made in which there are no switches. -first_search Depth-first search (DFS) for undirected graphs Depth-first search, or DFS, is a way to traverse the graph. The principle of the algorithm is quite simple: to go forward (in depth) while there is such possibility, otherwise to backtrack. Algorithm In DFS, each vertex has three possible colors representing its state: white: vertex is unvisited; gray: vertex is in progress; black: DFS has finished processing the vertex. NB. For most algorithms boolean classification unvisited / visited is quite enough, but we show general case here. Initially all vertices are white (unvisited). DFS starts in arbitrary vertex and runs as follows: 1. Mark vertex u as gray (visited). 2. For each edge (u, v), where u is white, run depth-first search for u recursively. 3. Mark vertex u as black and backtrack to the parent. Example. Traverse a graph shown below, using DFS. Start from a vertex with number 1.

12 Source graph. Mark a vertex 1 as gray. There is an edge (1, 4) and a vertex 4 is unvisited. Go there.

13 Mark the vertex 4 as gray. There is an edge (4, 2) and vertex a 2 is unvisited. Go there. Mark the vertex 2 as gray.

14 There is an edge (2, 5) and a vertex 5 is unvisited. Go there. Mark the vertex 5 as gray. There is an edge (5, 3) and a vertex 3 is unvisited. Go there.

15 Mark the vertex 3 as gray. There are no ways to go from the vertex 3. Mark it as black and backtrack to the vertex 5. There is an edge (5, 4), but the vertex 4 is gray.

16 There are no ways to go from the vertex 5. Mark it as black and backtrack to the vertex 2. There are no more edges, adjacent to vertex 2. Mark it as black and backtrack to the vertex 4. There is an edge (4, 5), but the vertex 5 is black.

17 There are no more edges, adjacent to the vertex 4. Mark it as black and backtrack to the vertex 1. There are no more edges, adjacent to the vertex 1. Mark it as black. DFS is over. As you can see from the example, DFS doesn't go through all edges. The vertices and edges, which depth-first search has visited is a tree. This tree contains all vertices of the graph (if it is connected) and is called graph spanning tree. This tree exactly corresponds to the recursive calls of DFS.

CSC Design and Analysis of Algorithms. Lecture 4 Brute Force, Exhaustive Search, Graph Traversal Algorithms. Brute-Force Approach

CSC Design and Analysis of Algorithms. Lecture 4 Brute Force, Exhaustive Search, Graph Traversal Algorithms. Brute-Force Approach CSC 8301- Design and Analysis of Algorithms Lecture 4 Brute Force, Exhaustive Search, Graph Traversal Algorithms Brute-Force Approach Brute force is a straightforward approach to solving a problem, usually