Algorithms (VI) Greedy Algorithms. Guoqiang Li. School of Software, Shanghai Jiao Tong University

Transcription

1 Algorithms (VI) Greedy Algorithms Guoqiang Li School of Software, Shanghai Jiao Tong University

2 Review of the Previous Lecture

3 Lengths on Edges BFS treats all edges as having the same length. It is rarely true in applications where shortest paths are to be found. Every edge e E with a length l e. If e = (u, v), we will sometimes also write l(u, v) or l uv

4 Alarm Clocks Set an alarm clock for node s at time 0. Repeat until there are no more alarms: Say the next alarm goes off at time T, for node y. Then: The distance from s to u is T. For each neighbor v of u in G: If there is no alarm yet for v, set one for time T + l(u, v). If v s alarm is set for later than T + l(u, v), then reset it to this earlier time. G: G :

5 Priority Queue Priority queue is a data structure usually implemented by heap. Insert: Add a new element to the set. Decrease-key: Accommodate the decrease in key value of a particular element. Delete-min: Return the element with the smallest key, and remove it from the set. Make-queue: Build a priority queue out of the given elements, with the given key values. (In many implementations, this is significantly faster than inserting the elements one by one.) The first two let us set alarms, and the third tells us which alarm is next to go off.

6 Dijkstra s Shortest-Path Algorithm DIJKSTRA(G, l, s) input : Graph G = (V, E), directed or undirected; positive edge length {l e e E}; Vertex s V output: For all vertices u reachable from s, dist(u) is the set to the distance from s to u for all u V do dist(u) = ; prev(u) = nil; end dist(s) = 0; H =makequeue(v)\\ using dist-values as keys; while H is not empty do u=deletemin(h); for all edge (u, v) E do if dist(v) > dist(u) + l(u, v) then dist(v) = dist(u) + l(u, v); prev(v) = u; decreasekey (H,v); end end end

7 An Alternative Derivation Initialize dist(s) = 0, other dist(.) to ; R = { }\\ the known region ; while R V do Pick the node v R with smallest dist(.); Add v to R; for all edge (v, z) E do if dist(z) > dist(v) + l(v, z) then dist[z] = dist[v] + l(v, z); end end end

8 Running Time Since makequeue takes at most as long as V insert operations, we get a total of V deletemin and V + E insert/decreasekey operations. The time needed for these varies by implementation; for instance, a binary heap gives an overall running time of O(( V + E ) log V )

9 Which Heap is Best Implementation deletemin insert/decreasekey Array O( V ) O(1) Binary heap O(log V ) O(log V ) d-ary heap d log V log V O( ) O( ) log d log d Fibonacci heap O(log V ) O(1) (amortized) V deletemin+( V + E ) insert O( V 2 ) O(( V + E ) log V ) (d V + E ) log V O( ) log d O( V log V + E )

10 Array The simplest implementation of a priority queue is as an unordered array of key values for all potential elements (the vertices of the graph, in the case of Dijkstra s algorithm). Initially, these values are set to 1. An insert or decreasekey is fast, because it just involves adjusting a key value, an O(1) operation. To deletemin, on the other hand, requires a linear-time scan of the list.

11 Binary Heap To insert, place the new element at the bottom of the tree (in the first available position), and let it bubble up. The number of swaps is at most the height of the tree log 2 n, when there are n elements. A decreasekey is similar, except the element is already in the tree, so we let it bubble up from its current position. To deletemin, return the root value. To then remove this element from the heap, take the last node in the tree (in the rightmost position in the bottom row) and place it at the root. Then let it shift down. Again this takes O(log n) time.

12 The Implementation of Binary Heap The regularity of a complete binary tree makes it easy to represent using an array. The tree nodes have a natural ordering: row by row, starting at the root and moving left to right within each row. If there are n nodes, this ordering specifies their positions 1, 2,..., n within the array. Moving up and down the tree is easily simulated on the array, using the fact that node number j has parent j/2 and children 2j and 2j + 1.

13 d-ary heap A d-ary heap is identical to a binary heap, except that nodes have d children. This reduces the height of a tree with n elements to Θ(log d n) = Θ((log n)/(log d)) Inserts are therefore speeded up by a factor of Θ(log d). Deletemin operations, however, take a little longer, namely O(d log d n).

14 Negative Edges Dijkstra s algorithm works in part because the shortest path from the starting point s to any node v must pass exclusively through nodes that are closer than v. This no longer holds when edge lengths can be negative. Q: What needs to be changed in order to accommodate this new complication? A crucial invariant of Dijkstra s algorithm is that the dist values it maintains are always either overestimates or exactly correct. They start off at, and the only way they ever change is by updating along an edge: UPDATE ((u, v) E) dist(v) = min{dist(v), dist(u) + l(u, v)};

15 Update UPDATE ((u, v) E) dist(v) = min{dist(v), dist(u) + l(u, v)}; Let s u 1 u 2 u 3... u k k be a shortest path from s to t. This path can have at most V 1 edges (why?). If the sequence of updates performed includes (s, u 1 ), (u 1, u 2 ),..., (u k, t), in that order (though not necessarily consecutively), then by 1 the distance to t will be correctly computed. It doesn t matter what other updates occur on these edges, or what happens in the rest of the graph, because updates are safe.

16 Bellman-Ford Algorithm SHORTEST-PATHS(G, l, s) input : Graph G = (V, E), edge length{l e e E}; Vertex s V output: For all vertices u reachable from s, dist(u) is the set to the distance from s to u for all u V do dist(u) = ; prev(u) = nil; end dists = 0; repeat V 1 times: for e E do UPDATE(e); end Running time:o( V E )

17 Graphs without Negative Edges There are two subclasses of graphs that automatically exclude the possibility of negative cycles: graphs without negative edges, and graphs without cycles. We already know how to efficiently handle the former. We will now see how the single-source shortest-path problem can be solved in just linear time on directed acyclic graphs. As before, we need to perform a sequence of updates that includes every shortest path as a subsequence. In any path of a dag, the vertices appear in increasing linearized order.

18 A Shortest-Path Algorithm for DAG DAG-SHORTEST-PATHS(G, l, s) input : Graph G = (V, E), edge length{l e e E}; Vertex s V output: For all vertices u reachable from s, dist(u) is the set to the distance from s to u for all u V do dist(u) = ; prev(u) = nil; end dists = 0; linearize G; for each u V in linearized order do for all e E do UPDATE(e); end end

19 Chapter V. Greedy Algorithms

20 Build a Network Suppose you are asked to network a collection of computers by linking selected pairs of them. 4 A C E B 4 D 6 F

21 Build a Network Suppose you are asked to network a collection of computers by linking selected pairs of them. This translates into a graph problem in which 4 A C E B 4 D 6 F

22 Build a Network Suppose you are asked to network a collection of computers by linking selected pairs of them. This translates into a graph problem in which nodes are computers, 4 A C E B 4 D 6 F

23 Build a Network Suppose you are asked to network a collection of computers by linking selected pairs of them. This translates into a graph problem in which nodes are computers, undirected edges are potential links, each with a maintenance cost. 4 A C E B 4 D 6 F

24 Build a Network The goal is to A 1 C E B 4 D F

25 Build a Network The goal is to pick enough of these edges that the nodes are connected, the total maintenance cost is minimum. A 1 C E B 4 D F

26 Build a Network The goal is to pick enough of these edges that the nodes are connected, the total maintenance cost is minimum. One immediate observation is that the optimal set of edges cannot contain a cycle. A 1 C E B 4 D F

27 Properties of the Optimal Solutions Lemma (1) Removing a cycle edge cannot disconnect a graph.

28 Properties of the Optimal Solutions Lemma (1) Removing a cycle edge cannot disconnect a graph. So the solution must be connected and acyclic: undirected graphs of this kind are called trees.

29 Properties of the Optimal Solutions Lemma (1) Removing a cycle edge cannot disconnect a graph. So the solution must be connected and acyclic: undirected graphs of this kind are called trees. A tree with minimum total weight, is a minimum spanning tree, MST.

30 Properties of the Optimal Solutions Lemma (1) Removing a cycle edge cannot disconnect a graph. So the solution must be connected and acyclic: undirected graphs of this kind are called trees. A tree with minimum total weight, is a minimum spanning tree, MST. Input: An undirected graph G = (V, E); edge weights w e

31 Properties of the Optimal Solutions Lemma (1) Removing a cycle edge cannot disconnect a graph. So the solution must be connected and acyclic: undirected graphs of this kind are called trees. A tree with minimum total weight, is a minimum spanning tree, MST. Input: An undirected graph G = (V, E); edge weights w e Output: A tree T = (V, E ) with E E that minimizes weight(t) = e E w e

32 Trees Lemma (2) A tree on n nodes has n 1 edges.

33 Trees Lemma (2) A tree on n nodes has n 1 edges. To build the tree one edge at a time, starting from an empty graph.

34 Trees Lemma (2) A tree on n nodes has n 1 edges. To build the tree one edge at a time, starting from an empty graph. Each of the n nodes is disconnected from the others, in a connected component by itself.

35 Trees Lemma (2) A tree on n nodes has n 1 edges. To build the tree one edge at a time, starting from an empty graph. Each of the n nodes is disconnected from the others, in a connected component by itself. As edges are added, these components merge. Since each edge unites two different components, exactly n 1 edges are added by the time the tree is fully formed.

36 Trees Lemma (2) A tree on n nodes has n 1 edges. To build the tree one edge at a time, starting from an empty graph. Each of the n nodes is disconnected from the others, in a connected component by itself. As edges are added, these components merge. Since each edge unites two different components, exactly n 1 edges are added by the time the tree is fully formed. When a particular edge (u, v) comes up, we can be sure that u and v lie in separate connected components, for otherwise there would already be a path between them and this edge would create a cycle.

37 Trees Lemma (3) Any connected, undirected graph G = (V, E) with E = V 1 is a tree.

38 Trees Lemma (3) Any connected, undirected graph G = (V, E) with E = V 1 is a tree. It is the converse of Lemma (2). We just need to show that G is acyclic.

39 Trees Lemma (3) Any connected, undirected graph G = (V, E) with E = V 1 is a tree. It is the converse of Lemma (2). We just need to show that G is acyclic. While the graph contains a cycle, remove one edge from this cycle.

40 Trees Lemma (3) Any connected, undirected graph G = (V, E) with E = V 1 is a tree. It is the converse of Lemma (2). We just need to show that G is acyclic. While the graph contains a cycle, remove one edge from this cycle. The process terminates with some graph G = (V, E ), E E, which is acyclic and, by Lemma (1), is also connected.

41 Trees Lemma (3) Any connected, undirected graph G = (V, E) with E = V 1 is a tree. It is the converse of Lemma (2). We just need to show that G is acyclic. While the graph contains a cycle, remove one edge from this cycle. The process terminates with some graph G = (V, E ), E E, which is acyclic and, by Lemma (1), is also connected. Therefore G is a tree, whereupon E = V 1 by Lemma (2). So E = E, no edges were removed, and G was acyclic to start with.

42 Trees Lemma (4) An undirected graph is a tree if and only if there is a unique path between any pair of nodes.

43 Trees Lemma (4) An undirected graph is a tree if and only if there is a unique path between any pair of nodes. In a tree, any two nodes can only have one path between them; for if there were two paths, the union of these paths would contain a cycle.

44 Trees Lemma (4) An undirected graph is a tree if and only if there is a unique path between any pair of nodes. In a tree, any two nodes can only have one path between them; for if there were two paths, the union of these paths would contain a cycle. On the other hand, if a graph has a path between any two nodes, then it is connected. If these paths are unique, then the graph is also acyclic.

45 A Greedy Approach

46 A Greedy Approach Kruskal s minimum spanning tree algorithm starts with the empty graph and then selects edges from E according to the following rule.

47 A Greedy Approach Kruskal s minimum spanning tree algorithm starts with the empty graph and then selects edges from E according to the following rule. Repeatedly add the next lightest edge that doesn t produce a cycle.

48 A Greedy Approach Kruskal s minimum spanning tree algorithm starts with the empty graph and then selects edges from E according to the following rule. Repeatedly add the next lightest edge that doesn t produce a cycle. Example: Starting with an empty graph and then attempt to add edges in increasing order of weight B C; C D; B D; C F; D F; E F; A D; A B; C E; A C A 6 5 C E A C E B 2 D 4 F B D F

49 The Cut Property Lemma: Suppose edges X are part of a minimum spanning tree (MST) of G = (V, E). Pick any subset of nodes S for which X does not cross between S and V\S, and let e be the lightest edge across this partition. Then X {e} is part of some MST.

50 The Cut Property A cut is any partition of the vertices into two groups, S and V\S. S V S e e

51 The Cut Property A cut is any partition of the vertices into two groups, S and V\S. The cut property says that it is always safe to add the lightest edge across any cut (that is, between a vertex in S and one in V\S), provided X has no edges across the cut. S e e V S

52 Proof of the Cut Property Edges X are part of some MST T;

53 Proof of the Cut Property Edges X are part of some MST T; if the new edge e also happens to be part of T, then there is nothing to prove.

54 Proof of the Cut Property Edges X are part of some MST T; if the new edge e also happens to be part of T, then there is nothing to prove. So assume e is not in T.

55 Proof of the Cut Property Edges X are part of some MST T; if the new edge e also happens to be part of T, then there is nothing to prove. So assume e is not in T. We will construct a different MST T containing X {e} by altering T slightly, changing just one of its edges.

56 Proof of the Cut Property Edges X are part of some MST T; if the new edge e also happens to be part of T, then there is nothing to prove. So assume e is not in T. We will construct a different MST T containing X {e} by altering T slightly, changing just one of its edges. Add edge e to T.

57 Proof of the Cut Property Edges X are part of some MST T; if the new edge e also happens to be part of T, then there is nothing to prove. So assume e is not in T. We will construct a different MST T containing X {e} by altering T slightly, changing just one of its edges. Add edge e to T. Since T is connected, it already has a path between the endpoints of e, so adding e creates a cycle.

58 Proof of the Cut Property Edges X are part of some MST T; if the new edge e also happens to be part of T, then there is nothing to prove. So assume e is not in T. We will construct a different MST T containing X {e} by altering T slightly, changing just one of its edges. Add edge e to T. Since T is connected, it already has a path between the endpoints of e, so adding e creates a cycle. This cycle must also have some other edge e across the cut (S, V\S).

59 Proof of the Cut Property Edges X are part of some MST T; if the new edge e also happens to be part of T, then there is nothing to prove. So assume e is not in T. We will construct a different MST T containing X {e} by altering T slightly, changing just one of its edges. Add edge e to T. Since T is connected, it already has a path between the endpoints of e, so adding e creates a cycle. This cycle must also have some other edge e across the cut (S, V\S). If we now remove e T = T {e}\{e } which we will show to be a tree.

60 Proof of the Cut Property Edges X are part of some MST T; if the new edge e also happens to be part of T, then there is nothing to prove. So assume e is not in T. We will construct a different MST T containing X {e} by altering T slightly, changing just one of its edges. Add edge e to T. Since T is connected, it already has a path between the endpoints of e, so adding e creates a cycle. This cycle must also have some other edge e across the cut (S, V\S). If we now remove e T = T {e}\{e } which we will show to be a tree. T is connected by Lemma (1), since e is a cycle edge.

61 Proof of the Cut Property Edges X are part of some MST T; if the new edge e also happens to be part of T, then there is nothing to prove. So assume e is not in T. We will construct a different MST T containing X {e} by altering T slightly, changing just one of its edges. Add edge e to T. Since T is connected, it already has a path between the endpoints of e, so adding e creates a cycle. This cycle must also have some other edge e across the cut (S, V\S). If we now remove e T = T {e}\{e } which we will show to be a tree. T is connected by Lemma (1), since e is a cycle edge. And it has the same number of edges as T; so by Lemma (2) and Lemma (3), it is also a tree.

62 Proof of the Cut Property T is a minimum spanning tree, since

63 Proof of the Cut Property T is a minimum spanning tree, since weight(t ) = weight(t) + w(e) w(e )

64 Proof of the Cut Property T is a minimum spanning tree, since weight(t ) = weight(t) + w(e) w(e ) Both e and e cross between S and V\S, and e is the lightest edge of this type.

65 Proof of the Cut Property T is a minimum spanning tree, since weight(t ) = weight(t) + w(e) w(e ) Both e and e cross between S and V\S, and e is the lightest edge of this type. Therefore w(e) w(e ), and weight(t ) weight(t)

66 Proof of the Cut Property T is a minimum spanning tree, since weight(t ) = weight(t) + w(e) w(e ) Both e and e cross between S and V\S, and e is the lightest edge of this type. Therefore w(e) w(e ), and weight(t ) weight(t) Since T is an MST, it must be the case that weight(t ) = weight(t) and that T is also an MST.

67 An Example of Cut Property (a) A 1 C 3 E 2 B D 4 F (b) A C E A C E Edges X: MST T: B D F B D F (c) The cut: A B C D e E F MST T : A B C D E F S V S

68 Kruskal s Algorithm KRUSKAL(G, w) input : A connected undirected graph G = (V, E), with edge weight w e output: A minimum spanning tree defined by the edges X for all u V do makeset (u); end X = { }; Sort the edges E by weight; for all (u, v) E in increasing order of weight do if find (u) find (v) then add (u, v) to X; union (u,v) end end

69 Kruskal s Algorithm makeset(x) create a singleton set containing x V find(x) find the set that x belong to 2 E union(x, y) merge the sets containing x and y V 1

70 Data Structure: Disjoint Sets

71 A Data Structure for Disjoint Sets Union by rank We store a set is by a directed tree.

72 A Data Structure for Disjoint Sets Union by rank We store a set is by a directed tree. Nodes of the tree are elements of the set, arranged in no particular order, and each has parent pointers that eventually lead up to the root of the tree.

73 A Data Structure for Disjoint Sets Union by rank We store a set is by a directed tree. Nodes of the tree are elements of the set, arranged in no particular order, and each has parent pointers that eventually lead up to the root of the tree. This root element is a convenient representative, or name, for the set.

74 A Data Structure for Disjoint Sets Union by rank We store a set is by a directed tree. Nodes of the tree are elements of the set, arranged in no particular order, and each has parent pointers that eventually lead up to the root of the tree. This root element is a convenient representative, or name, for the set. It is distinguished from the other elements by the fact that its parent pointer is a self-loop.

75 A Data Structure for Disjoint Sets Union by rank We store a set is by a directed tree. Nodes of the tree are elements of the set, arranged in no particular order, and each has parent pointers that eventually lead up to the root of the tree. This root element is a convenient representative, or name, for the set. It is distinguished from the other elements by the fact that its parent pointer is a self-loop. In addition to a parent pointer π, each node also has a rank that, for the time being, should be interpreted as the height of the subtree hanging from that node.

76 Union by Rank MAKESET(x) π(x) = x; rank (x)=0;

77 Union by Rank MAKESET(x) π(x) = x; rank (x)=0; FIND(x) while x π(x) do x = π(x); end return(x);

78 Union by Rank MAKESET(x) π(x) = x; rank (x)=0; FIND(x) while x π(x) do x = π(x); end return(x); MAKESET is a constant-time operation.

79 Union by Rank MAKESET(x) π(x) = x; rank (x)=0; FIND(x) while x π(x) do x = π(x); end return(x); MAKESET is a constant-time operation. FIND follows parent pointers to the root of the tree and therefore takes time proportional to the height of the tree.

80 Union by Rank MAKESET(x) π(x) = x; rank (x)=0; FIND(x) while x π(x) do x = π(x); end return(x); MAKESET is a constant-time operation. FIND follows parent pointers to the root of the tree and therefore takes time proportional to the height of the tree. The tree actually gets built via the third operation, UNION, and so we must make sure that this procedure keeps trees shallow.

81 Union UNION(x, y) r x =find(x); r y =find(y); if r x = r y then return; if rank(r x )>rank(r y ) then π(r y ) = r x ; end else π(r x ) = r y ; if rank(r x )=rank(r y ) then rank(r y )=rank(r y )+1; end

82 An Example Aftermakeset(A),makeset(B),...,makeset(G): A 0 B 0 C 0 D 0 E 0 F 0 0 G After union(a, D), union(b, E), union(c, F ): D 1 E 1 F 1 G 0 A 0 B 0 C 0

83 An Example After union(c, G), union(e, A): D 2 F 1 E 1 A 0 0 C 0 G B 0 After union(b, G): D 2 E 1 F 1 A 0 B 0 C 0 G 0

84 Properties Lemma (1) For any non-root x, rank(x) < rank(π(x)).

85 Properties Lemma (1) For any non-root x, rank(x) < rank(π(x)). Proof Sketch: By design, the rank of a node is exactly the height of the subtree rooted at that node. This means, for instance, that as you move up a path toward a root node, the rank values along the way are strictly increasing.

86 Properties Lemma (2) Any root node of rank k has least 2 k nodes in its tree.

87 Properties Lemma (2) Any root node of rank k has least 2 k nodes in its tree. Proof Sketch: A root node with rank k is created by the merger of two trees with roots of rank k 1. By induction to get the results.

88 Properties Lemma (3) If there are n elements overall, there can be at most n/2 k nodes of rank k.

89 Properties Lemma (3) If there are n elements overall, there can be at most n/2 k nodes of rank k. Proof Sketch:

90 Properties Lemma (3) If there are n elements overall, there can be at most n/2 k nodes of rank k. Proof Sketch: A node of rank k has at least 2 k descendants. Any internal node was once a root, and neither its rank nor its set of descendants has changed since then. Different rank-k nodes cannot have common descendants. Any element has at most one ancestor of rank k.

91 The Analysis With the data structure as presented so far, the total time for Kruskal s algorithm becomes

92 The Analysis With the data structure as presented so far, the total time for Kruskal s algorithm becomes O( E log V ) for sorting the edges (log V log E ),

93 The Analysis With the data structure as presented so far, the total time for Kruskal s algorithm becomes O( E log V ) for sorting the edges (log V log E ), O( E log V ) for the union and find operations that dominate the rest of the algorithm.

94 The Analysis With the data structure as presented so far, the total time for Kruskal s algorithm becomes O( E log V ) for sorting the edges (log V log E ), O( E log V ) for the union and find operations that dominate the rest of the algorithm. But what if the edges are given to us sorted? Or if the weights are small (say, O( E )) so that sorting can be done in linear time?

95 The Analysis With the data structure as presented so far, the total time for Kruskal s algorithm becomes O( E log V ) for sorting the edges (log V log E ), O( E log V ) for the union and find operations that dominate the rest of the algorithm. But what if the edges are given to us sorted? Or if the weights are small (say, O( E )) so that sorting can be done in linear time? Then the data structure part becomes the bottleneck!

96 The Analysis With the data structure as presented so far, the total time for Kruskal s algorithm becomes O( E log V ) for sorting the edges (log V log E ), O( E log V ) for the union and find operations that dominate the rest of the algorithm. But what if the edges are given to us sorted? Or if the weights are small (say, O( E )) so that sorting can be done in linear time? Then the data structure part becomes the bottleneck! The main question:

97 The Analysis With the data structure as presented so far, the total time for Kruskal s algorithm becomes O( E log V ) for sorting the edges (log V log E ), O( E log V ) for the union and find operations that dominate the rest of the algorithm. But what if the edges are given to us sorted? Or if the weights are small (say, O( E )) so that sorting can be done in linear time? Then the data structure part becomes the bottleneck! The main question: How can we perform union s and find s faster than log n?

98 Path Compression FIND(x) if x π(x) then π(x)=find(π(x)); return(π(x));

99 Path Compression A 3 A 3 B 0 C 1 E 2 B 0 C 1 E 2 F 1 I 0 D 0 1 F G 1 H 0 0 D G 1 H 0 J 0 I 0 J 0 K 0 K 0 3 A B 0 C 1 E 2 F 1 I 0 K 0 G 1 0 D H 0 J 0

100 Path Compression

101 Path Compression The benefit of this simple alteration is long-term rather than instantaneous and thus necessitates a particular kind of analysis.

102 Path Compression The benefit of this simple alteration is long-term rather than instantaneous and thus necessitates a particular kind of analysis. we need to look at sequences of find and union operations, starting from an empty data structure, and determine the average time per operation.

103 Path Compression The benefit of this simple alteration is long-term rather than instantaneous and thus necessitates a particular kind of analysis. we need to look at sequences of find and union operations, starting from an empty data structure, and determine the average time per operation. This amortized cost turns out to be just barely more than O(1), down from the earlier O(log n).

104 Prim s Algorithm

105 A General Kruskal s Algorithm X = { }; repeat until X = V 1; pick a set S V for which X has no edges between S and V S; let e E be the minimum-weight edge between S and V S; X = X {e};

106 Prim s Algorithm A popular alternative to Kruskal s algorithm is Prim s, in which the intermediate set of edges X always forms a subtree, and S is chosen to be the set of this tree s vertices.

107 Prim s Algorithm A popular alternative to Kruskal s algorithm is Prim s, in which the intermediate set of edges X always forms a subtree, and S is chosen to be the set of this tree s vertices. On each iteration, the subtree defined by X grows by one edge, namely, the lightest edge between a vertex in S and a vertex outside S. We can equivalently think of S as growing to include the vertex v S of smallest cost: cost(v) = min w(u, v) u S

108 The Algorithm PRIM(G, w) input : A connected undirected graph G = (V, E), with edge weights w e output: A minimum spanning tree defined by the array prev for all u V do cost(u) = ; prev(u) = nil; end pick any initial node u 0; cost(u 0) = 0; H =makequeue(v)\\ using cost-values as keys; while H is not empty do v=deletemin(h); for each (v, z) E do if cost(z) > w(v, z) then cost(v) = w(v, z); prev(z) = v; decreasekey (H,z); end end end

109 Dijkstra s Algorithm DIJKSTRA(G, l, s) input : Graph G = (V, E), directed or undirected; positive edge length {l e e E}; Vertex s V output: For all vertices u reachable from s, dist(u) is the set to the distance from s to u for all u V do dist(u) = ; prev(u) = nil; end dist(s) = 0; H =makequeue(v)\\ using dist-values as keys; while H is not empty do u=deletemin(h); for all edge (u, v) E do if dist(v) > dist(u) + l(u, v) then dist(v) = dist(u) + l(u, v); prev(v) = u; decreasekey (H,v); end end end

110 Huffman Encoding

111 MP3 Audio Compression 1 It is digitized by sampling at regular intervals, yielding a sequence of real numbers s 1, s 2,..., s T.

112 MP3 Audio Compression 1 It is digitized by sampling at regular intervals, yielding a sequence of real numbers s 1, s 2,..., s T. 2 Each real-valued sample s t is quantized: approximated by a nearby number from a finite set Γ This set is carefully chosen to exploit human perceptual limitations, with the intention that the approximating sequence is indistinguishable from s 1, s 2,..., s T by the human ear.

113 MP3 Audio Compression 1 It is digitized by sampling at regular intervals, yielding a sequence of real numbers s 1, s 2,..., s T. 2 Each real-valued sample s t is quantized: approximated by a nearby number from a finite set Γ This set is carefully chosen to exploit human perceptual limitations, with the intention that the approximating sequence is indistinguishable from s 1, s 2,..., s T by the human ear. 3 The resulting string of length T over alphabet Γ is encoded in binary.

114 Binary Encoding Let s look at a toy example in which T is 130 million and the alphabet Γ consists of just four values, denoted by the symbols A, B, C, D.

115 Binary Encoding Let s look at a toy example in which T is 130 million and the alphabet Γ consists of just four values, denoted by the symbols A, B, C, D. Q: What is the most economical way to write this long string in binary?

116 Binary Encoding Let s look at a toy example in which T is 130 million and the alphabet Γ consists of just four values, denoted by the symbols A, B, C, D. Q: What is the most economical way to write this long string in binary? The obvious choice is to use 2 bits per symbol.

117 Binary Encoding Let s look at a toy example in which T is 130 million and the alphabet Γ consists of just four values, denoted by the symbols A, B, C, D. Q: What is the most economical way to write this long string in binary? The obvious choice is to use 2 bits per symbol. codeword 00 for A, 01 for B, 10 for C, and 11 for D

118 Binary Encoding Let s look at a toy example in which T is 130 million and the alphabet Γ consists of just four values, denoted by the symbols A, B, C, D. Q: What is the most economical way to write this long string in binary? The obvious choice is to use 2 bits per symbol. codeword 00 for A, 01 for B, 10 for C, and 11 for D 260 megabits are needed in total.

119 Binary Encoding Let s look at a toy example in which T is 130 million and the alphabet Γ consists of just four values, denoted by the symbols A, B, C, D. Q: What is the most economical way to write this long string in binary? The obvious choice is to use 2 bits per symbol. codeword 00 for A, 01 for B, 10 for C, and 11 for D 260 megabits are needed in total. Q: Can there possibly be a better encoding than this?

120 Binary Encoding In search of inspiration, we take a closer look at our particular sequence and find that the four symbols are not equally abundant.

121 Binary Encoding In search of inspiration, we take a closer look at our particular sequence and find that the four symbols are not equally abundant. Symbol A B C D Frequency 70 million 3 million 20 million 37 million

122 Binary Encoding In search of inspiration, we take a closer look at our particular sequence and find that the four symbols are not equally abundant. Symbol A B C D Frequency 70 million 3 million 20 million 37 million Is there some sort of variable-length encoding, in which just one bit is used for the frequently occurring symbol A, possibly at the expense of needing three or more bits for less common symbols?

123 Binary Encoding In search of inspiration, we take a closer look at our particular sequence and find that the four symbols are not equally abundant. Symbol A B C D Frequency 70 million 3 million 20 million 37 million Is there some sort of variable-length encoding, in which just one bit is used for the frequently occurring symbol A, possibly at the expense of needing three or more bits for less common symbols? A danger with having codewords of different lengths is that the resulting encoding may not be uniquely decipherable.

124 Binary Encoding In search of inspiration, we take a closer look at our particular sequence and find that the four symbols are not equally abundant. Symbol A B C D Frequency 70 million 3 million 20 million 37 million Is there some sort of variable-length encoding, in which just one bit is used for the frequently occurring symbol A, possibly at the expense of needing three or more bits for less common symbols? A danger with having codewords of different lengths is that the resulting encoding may not be uniquely decipherable. {0, 01, 11, 001}

125 Prefix-Free Encoding We will avoid this problem by insisting on the prefix-free property: no codeword can be a prefix of another codeword.

126 Prefix-Free Encoding We will avoid this problem by insisting on the prefix-free property: no codeword can be a prefix of another codeword. Any prefix-free encoding can be represented by a full binary tree - a binary tree in which every node has either zero or two children.

127 Prefix-Free Encoding We will avoid this problem by insisting on the prefix-free property: no codeword can be a prefix of another codeword. Any prefix-free encoding can be represented by a full binary tree - a binary tree in which every node has either zero or two children. The symbols are at the leaves, and where each codeword is generated by a path from root to leaf, interpreting left as 0 and right as 1.

128 Prefix-Free Encoding We will avoid this problem by insisting on the prefix-free property: no codeword can be a prefix of another codeword. Any prefix-free encoding can be represented by a full binary tree - a binary tree in which every node has either zero or two children. The symbols are at the leaves, and where each codeword is generated by a path from root to leaf, interpreting left as 0 and right as 1. Decoding is unique: a string of bits is decrypted by starting at the root, reading the string from left to right to move downward, and, whenever a leaf is reached, outputting the corresponding symbol and returning to the root.

129 Prefix-Free Encoding 0 1 Symbol Codeword A 0 B 100 C 101 D 11 A [70] [23] [60] D [37] B [3] C [20]

130 Prefix-Free Encoding 0 1 Symbol Codeword A 0 B 100 C 101 D 11 A [70] [23] [60] D [37] B [3] C [20] The total size of the binary string drops to 213 megabits, a 17% improvement.

131 Optimal Encoding

132 Optimal Encoding Q: How do we find the optimal coding tree, given the frequencies f 1, f 2,..., f n of n symbols?

133 Optimal Encoding Q: How do we find the optimal coding tree, given the frequencies f 1, f 2,..., f n of n symbols? We want a tree whose leaves each correspond to a symbol and which minimizes the overall length of the encoding, n cost of tree = f i (depth of i th symbol in tree) i=1

134 Optimal Encoding Q: How do we find the optimal coding tree, given the frequencies f 1, f 2,..., f n of n symbols? We want a tree whose leaves each correspond to a symbol and which minimizes the overall length of the encoding, n cost of tree = f i (depth of i th symbol in tree) i=1 The number of bits required for a symbol is exactly its depth in the tree.

135 Optimal Encoding Q: How do we find the optimal coding tree, given the frequencies f 1, f 2,..., f n of n symbols? We want a tree whose leaves each correspond to a symbol and which minimizes the overall length of the encoding, n cost of tree = f i (depth of i th symbol in tree) i=1 The number of bits required for a symbol is exactly its depth in the tree. There is another way to write this cost function that is very helpful.

136 Optimal Encoding Q: How do we find the optimal coding tree, given the frequencies f 1, f 2,..., f n of n symbols? We want a tree whose leaves each correspond to a symbol and which minimizes the overall length of the encoding, n cost of tree = f i (depth of i th symbol in tree) i=1 The number of bits required for a symbol is exactly its depth in the tree. There is another way to write this cost function that is very helpful. We can define the frequency of any internal node to be the sum of the frequencies of its descendant leaves.

137 Optimal Encoding Q: How do we find the optimal coding tree, given the frequencies f 1, f 2,..., f n of n symbols? We want a tree whose leaves each correspond to a symbol and which minimizes the overall length of the encoding, n cost of tree = f i (depth of i th symbol in tree) i=1 The number of bits required for a symbol is exactly its depth in the tree. There is another way to write this cost function that is very helpful. We can define the frequency of any internal node to be the sum of the frequencies of its descendant leaves. The cost of a tree is the sum of the frequencies of all leaves and internal nodes, except the root.

138 Huffman Encoding We start constructing the tree greedily:

139 Huffman Encoding We start constructing the tree greedily: find the two symbols with the smallest frequencies, say i and j,

140 Huffman Encoding We start constructing the tree greedily: find the two symbols with the smallest frequencies, say i and j, and make them children of a new node, which then has frequency f i + f j. To keep the notation simple, let s just assume these are f 1 and f 2.

141 Huffman Encoding We start constructing the tree greedily: find the two symbols with the smallest frequencies, say i and j, and make them children of a new node, which then has frequency f i + f j. To keep the notation simple, let s just assume these are f 1 and f 2. By the second formulation of the cost function, any tree in which f 1 and f 2 are sibling-leaves has cost f 1 + f 2 plus the cost for a tree with n 1 leaves of frequencies (f 1 + f 2 ), f 3, f 4,..., f n.

142 Huffman Encoding We start constructing the tree greedily: find the two symbols with the smallest frequencies, say i and j, and make them children of a new node, which then has frequency f i + f j. To keep the notation simple, let s just assume these are f 1 and f 2. By the second formulation of the cost function, any tree in which f 1 and f 2 are sibling-leaves has cost f 1 + f 2 plus the cost for a tree with n 1 leaves of frequencies (f 1 + f 2 ), f 3, f 4,..., f n. The latter problem is just a smaller version of the one we started with. So we pull f 1 and f 2 off the list of frequencies, insert (f 1 + f 2 ), and loop.

143 The Algorithm Huffman(f ) input : An array f [1,..., n] of frequencies output: An encoding tree with n leaves Let H be a priority queue of integers, ordered by f ; for i = 1 to n do Insert(H,i); end for k = to 2n do i=deletemin (H); j=deletemin (H); create a node numbered k with children i, j; f [k] = f [i] + f [j]; Insert(H, k); end

144 Set Cover

145 The Problem A county is in its early stages of planning and is deciding where to put schools. (a) c (b) c d b a e f d b a e f k g k g i h i h j j

146 The Problem A county is in its early stages of planning and is deciding where to put schools. There are only two constraints: (a) c (b) c d b a e f d b a e f k g k g i h i h j j

147 The Problem A county is in its early stages of planning and is deciding where to put schools. There are only two constraints: each school should be in a town, and no one should have to travel more than 30 miles to reach one of them. (a) c (b) c d b a e f d b a e f k g k g i h i h j j

148 The Problem A county is in its early stages of planning and is deciding where to put schools. There are only two constraints: each school should be in a town, and no one should have to travel more than 30 miles to reach one of them. Q: What is the minimum number of schools needed? (a) c (b) c d b a e f d b a e f k g k g i h i h j j

149 The Problem This is a typical set cover problem.

150 The Problem This is a typical set cover problem. For each town x, let S x be the set of towns within 30 miles of it.

151 The Problem This is a typical set cover problem. For each town x, let S x be the set of towns within 30 miles of it. A school at x will essentially cover these other towns.

152 The Problem This is a typical set cover problem. For each town x, let S x be the set of towns within 30 miles of it. A school at x will essentially cover these other towns. The question is then, how many sets S x must be picked in order to cover all the towns in the county?

153 Set Cover Problem

154 Set Cover Problem Set Cover Input: A set of elements B, sets S 1,..., S m B Output: A selection of the S i whose union is B. Cost: Number of sets picked.

155 Set Cover Problem Set Cover Input: A set of elements B, sets S 1,..., S m B Output: A selection of the S i whose union is B. Cost: Number of sets picked. This problem lends itself immediately to a greedy solution:

156 Set Cover Problem Set Cover Input: A set of elements B, sets S 1,..., S m B Output: A selection of the S i whose union is B. Cost: Number of sets picked. This problem lends itself immediately to a greedy solution: Repeat until all elements of B are covered: Pick the set S i with the largest number of uncovered elements.

157 Set Cover Problem Set Cover Input: A set of elements B, sets S 1,..., S m B Output: A selection of the S i whose union is B. Cost: Number of sets picked. This problem lends itself immediately to a greedy solution: Repeat until all elements of B are covered: Pick the set S i with the largest number of uncovered elements. The greedy algorithm doesn t always find the best solution!

158 The Example c d b a e f k g i h j

159 Performance Ratio

160 Performance Ratio Lemma Suppose B contains n elements and that the optimal cover consists of k sets. Then the greedy algorithm will use at most k ln n sets.

161 Performance Ratio Lemma Suppose B contains n elements and that the optimal cover consists of k sets. Then the greedy algorithm will use at most k ln n sets. Proof

162 Performance Ratio Lemma Suppose B contains n elements and that the optimal cover consists of k sets. Then the greedy algorithm will use at most k ln n sets. Proof Let n t be the number of elements still not covered after t iterations of the greedy algorithm (so n 0 = n).

163 Performance Ratio Lemma Suppose B contains n elements and that the optimal cover consists of k sets. Then the greedy algorithm will use at most k ln n sets. Proof Let n t be the number of elements still not covered after t iterations of the greedy algorithm (so n 0 = n). Since these remaining elements are covered by the optimal k sets, there must be some set with at least n t /k of them.

164 Performance Ratio Lemma Suppose B contains n elements and that the optimal cover consists of k sets. Then the greedy algorithm will use at most k ln n sets. Proof Let n t be the number of elements still not covered after t iterations of the greedy algorithm (so n 0 = n). Since these remaining elements are covered by the optimal k sets, there must be some set with at least n t /k of them. Therefore, the greedy strategy will ensure that n t+1 n t n t k = n t(1 1 k ) which by repeated application implies n t n 0 (1 1 k )t

165 Performance Ratio A more convenient bound can be obtained from the useful inequality 1 x e x for all x with equality if and only if x = 0,

166 Performance Ratio A more convenient bound can be obtained from the useful inequality 1 x e x for all x with equality if and only if x = 0, Thus n t n 0 (1 1 k )t < n 0 (e 1 k ) t = ne t k

167 Performance Ratio A more convenient bound can be obtained from the useful inequality 1 x e x for all x with equality if and only if x = 0, Thus n t n 0 (1 1 k )t < n 0 (e 1 k ) t = ne t k At t = k ln n, therefore, n t is strictly less than ne ln n = 1, which means no elements remain to be covered.

168 SAT solver

169 What Is SAT Given a propositional formula in CNF, find an assignment to boolean variables that makes the formula true:

170 What Is SAT Given a propositional formula in CNF, find an assignment to boolean variables that makes the formula true: ω 1 = x 2 x 3 ω 2 = x 1 x 4 ω 3 = x 2 x 4

171 What Is SAT Given a propositional formula in CNF, find an assignment to boolean variables that makes the formula true: ω 1 = x 2 x 3 ω 2 = x 1 x 4 ω 3 = x 2 x 4 A = {x 1 = 0; x 2 = 1; x 3 = 0; x 4 = 1; }

172 Satisfiability The instances of Satisfiability or SAT:

173 Satisfiability The instances of Satisfiability or SAT: (x y z)(x y)(y z)(z x)(x y z)

174 Satisfiability The instances of Satisfiability or SAT: (x y z)(x y)(y z)(z x)(x y z) That is, a Boolean formula in conjunctive normal form (CNF).

175 Satisfiability The instances of Satisfiability or SAT: (x y z)(x y)(y z)(z x)(x y z) That is, a Boolean formula in conjunctive normal form (CNF). It is a collection of clauses (the parentheses),

176 Satisfiability The instances of Satisfiability or SAT: (x y z)(x y)(y z)(z x)(x y z) That is, a Boolean formula in conjunctive normal form (CNF). It is a collection of clauses (the parentheses), each consisting of the disjunction (logical or, denoted ) of several literals;

177 Satisfiability The instances of Satisfiability or SAT: (x y z)(x y)(y z)(z x)(x y z) That is, a Boolean formula in conjunctive normal form (CNF). It is a collection of clauses (the parentheses), each consisting of the disjunction (logical or, denoted ) of several literals; a literal is either a Boolean variable (such as x) or the negation of one (such as x).

178 Satisfiability The instances of Satisfiability or SAT: (x y z)(x y)(y z)(z x)(x y z) That is, a Boolean formula in conjunctive normal form (CNF). It is a collection of clauses (the parentheses), each consisting of the disjunction (logical or, denoted ) of several literals; a literal is either a Boolean variable (such as x) or the negation of one (such as x). A satisfying truth assignment is an assignment of false or true to each variable so that every clause contains a literal whose value is true.

179 Satisfiability The instances of Satisfiability or SAT: (x y z)(x y)(y z)(z x)(x y z) That is, a Boolean formula in conjunctive normal form (CNF). It is a collection of clauses (the parentheses), each consisting of the disjunction (logical or, denoted ) of several literals; a literal is either a Boolean variable (such as x) or the negation of one (such as x). A satisfying truth assignment is an assignment of false or true to each variable so that every clause contains a literal whose value is true. The SAT problem is the following: given a Boolean formula in conjunctive normal form, either find a satisfying truth assignment or else report that none exists.

180 2-SAT Given a set of clauses, where each clause is the disjunction of two literals (a literal is a Boolean variable or the negation of a Boolean variable). You are looking for a way to assign a value true or false to each of the variables so that all clauses are satisfied. that is, there is at least one true literal in each clause. (x 1 x 2 ) (x 1 x 3 ) (x 1 x 2 ) (x 3 x 4 ) (x 1 x 4 )

181 2-SAT Given a set of clauses, where each clause is the disjunction of two literals (a literal is a Boolean variable or the negation of a Boolean variable). You are looking for a way to assign a value true or false to each of the variables so that all clauses are satisfied. that is, there is at least one true literal in each clause. (x 1 x 2 ) (x 1 x 3 ) (x 1 x 2 ) (x 3 x 4 ) (x 1 x 4 ) Given an instance I of 2-Sat with n variables and m clauses, construct a directed graph G I = (V, E) as follows.

182 2-SAT Given a set of clauses, where each clause is the disjunction of two literals (a literal is a Boolean variable or the negation of a Boolean variable). You are looking for a way to assign a value true or false to each of the variables so that all clauses are satisfied. that is, there is at least one true literal in each clause. (x 1 x 2 ) (x 1 x 3 ) (x 1 x 2 ) (x 3 x 4 ) (x 1 x 4 ) Given an instance I of 2-Sat with n variables and m clauses, construct a directed graph G I = (V, E) as follows. G I has 2n nodes, one for each variable and its negation.

183 2-SAT Given a set of clauses, where each clause is the disjunction of two literals (a literal is a Boolean variable or the negation of a Boolean variable). You are looking for a way to assign a value true or false to each of the variables so that all clauses are satisfied. that is, there is at least one true literal in each clause. (x 1 x 2 ) (x 1 x 3 ) (x 1 x 2 ) (x 3 x 4 ) (x 1 x 4 ) Given an instance I of 2-Sat with n variables and m clauses, construct a directed graph G I = (V, E) as follows. G I has 2n nodes, one for each variable and its negation. G I has 2m edges: for each clause (α β) of I, G I has an edge from the negation of α to β, and one from the negation of β to α.

184 2-SAT Show that if G I has a strongly connected component containing both x and x for some variable x, then I has no satisfying assignment.

185 2-SAT Show that if G I has a strongly connected component containing both x and x for some variable x, then I has no satisfying assignment. If none of G I s strongly connected components contain both a literal and its negation, then the instance I must be satisfiable.

186 2-SAT Show that if G I has a strongly connected component containing both x and x for some variable x, then I has no satisfying assignment. If none of G I s strongly connected components contain both a literal and its negation, then the instance I must be satisfiable. Conclude that there is a linear-time algorithm for solving 2-SAT.

187 SAT Solver SAT solver decide whether a given propositional formula is satisfiable(i.e., whether there exists truth valuation for atomic propositions).

188 SAT Solver SAT solver decide whether a given propositional formula is satisfiable(i.e., whether there exists truth valuation for atomic propositions). 3-SAT (satisfiablity of boolean formula of length 3 clauses) is NP-complete; though often practically efficient.

189 SAT Solver SAT solver decide whether a given propositional formula is satisfiable(i.e., whether there exists truth valuation for atomic propositions). 3-SAT (satisfiablity of boolean formula of length 3 clauses) is NP-complete; though often practically efficient. SAT solver is to be said as the most successful formal tools.