22c:1 Algorithms Kruskal s Algorithm for MST Union-Find for Disjoint Sets 1
Kruskal s Algorithm for MST Work with edges, rather than nodes Two steps: Sort edges by increasing edge weight Select the first V 1 edges that do not generate a cycle
Walk-Through 5 H A 10 F 8 B 2 G 6 C 1 E D Consider an undirected, weight graph
5 H A Sort the edges by increasing edge weight F 10 C edge d v edge d v (D,E) 1 (B,E) 8 6 (D,G) 2 (B,F) B 2 G 1 E D (E,G) (C,D) (G,H) (C,F) (B,C) (B,H) (A,H) 5 (D,F) 6 (A,B) 8 (A,F) 10
5 H A Select first V 1 edges which do not generate a cycle F 10 C edge d v edge d v (D,E) 1 (B,E) 8 6 (D,G) 2 (B,F) B 2 G 1 E D (E,G) (C,D) (G,H) (C,F) (B,C) (B,H) (A,H) 5 (D,F) 6 (A,B) 8 (A,F) 10
5 H A Select first V 1 edges which do not generate a cycle F 10 C edge d v edge d v (D,E) 1 (B,E) 8 6 (D,G) 2 (B,F) B 2 G 1 E D (E,G) (C,D) (G,H) (C,F) (B,C) (B,H) (A,H) 5 (D,F) 6 (A,B) 8 (A,F) 10
5 H A Select first V 1 edges which do not generate a cycle F 10 C edge d v edge d v (D,E) 1 (B,E) 8 6 (D,G) 2 (B,F) B 2 G 1 E D (E,G) (C,D) (G,H) (C,F) (B,C) (B,H) (A,H) 5 (D,F) 6 (A,B) 8 (A,F) 10 Accepting edge (E,G) would create a cycle
5 H A Select first V 1 edges which do not generate a cycle F 10 C edge d v edge d v (D,E) 1 (B,E) 8 6 (D,G) 2 (B,F) B 2 G 1 E D (E,G) (C,D) (G,H) (C,F) (B,C) (B,H) (A,H) 5 (D,F) 6 (A,B) 8 (A,F) 10
5 H A Select first V 1 edges which do not generate a cycle F 10 C edge d v edge d v (D,E) 1 (B,E) 8 6 (D,G) 2 (B,F) B 2 G 1 E D (E,G) (C,D) (G,H) (C,F) (B,C) (B,H) (A,H) 5 (D,F) 6 (A,B) 8 (A,F) 10
5 H A Select first V 1 edges which do not generate a cycle F 10 C edge d v edge d v (D,E) 1 (B,E) 8 6 (D,G) 2 (B,F) B 2 G 1 E D (E,G) (C,D) (G,H) (C,F) (B,C) (B,H) (A,H) 5 (D,F) 6 (A,B) 8 (A,F) 10
5 H A Select first V 1 edges which do not generate a cycle F 10 C edge d v edge d v (D,E) 1 (B,E) 8 6 (D,G) 2 (B,F) B 2 G 1 E D (E,G) (C,D) (G,H) (C,F) (B,C) (B,H) (A,H) 5 (D,F) 6 (A,B) 8 (A,F) 10
5 H A Select first V 1 edges which do not generate a cycle F 10 C edge d v edge d v (D,E) 1 (B,E) 8 6 (D,G) 2 (B,F) B 2 G 1 E D (E,G) (C,D) (G,H) (C,F) (B,C) (B,H) (A,H) 5 (D,F) 6 (A,B) 8 (A,F) 10
5 H A Select first V 1 edges which do not generate a cycle F 10 C edge d v edge d v (D,E) 1 (B,E) 8 6 (D,G) 2 (B,F) B 2 G 1 E D (E,G) (C,D) (G,H) (C,F) (B,C) (B,H) (A,H) 5 (D,F) 6 (A,B) 8 (A,F) 10
5 H A Select first V 1 edges which do not generate a cycle F 10 C edge d v edge d v (D,E) 1 (B,E) 8 6 (D,G) 2 (B,F) B 2 G 1 E D (E,G) (C,D) (G,H) (C,F) (B,C) (B,H) (A,H) 5 (D,F) 6 (A,B) 8 (A,F) 10
5 H A Select first V 1 edges which do not generate a cycle F 10 C edge d v edge d v (D,E) 1 (B,E) 8 6 (D,G) 2 (B,F) B 2 G 1 E D (E,G) (C,D) (G,H) (C,F) (B,C) (B,H) (A,H) 5 (D,F) 6 (A,B) 8 (A,F) 10
F C Select first V 1 edges which do not generate a cycle edge d v edge d v 5 H A G B 2 1 E D (D,E) 1 (D,G) 2 (E,G) (C,D) (G,H) (C,F) (B,C) (B,E) (B,F) (B,H) (A,H) 5 (D,F) 6 (A,B) 8 (A,F) 10 } not considered Done Total Cost = d v = 21
Complexity of Kruskal s Algorithm 1. Sort all the edges T = {} 2. For each edge e, if T U {e} doesn t have a cycle, T := T U { e } Step 1 takes O(E lg E), where E is # of edges. Step 2 takes O(E) using Disjoint Sets
Equivalence Relations Relation R : For every pair of elements (a, b) in a set S, a R b is either true or false. If a R b is true, then a is related to b. An equivalence relation satisfies: 1. (Reflexive) a R a 2. (Symmetric) a R b iff b R a. (Transitive) a R b and b R c implies a R c 18
Equivalence Classes Given a set of things { grapes, blackberries, plums, apples, oranges, peaches, raspberries, lemons, bananas } define the equivalence relation All citrus fruit is related, all berries, all stone fruits, partition them into related subsets { grapes }, { blackberries, raspberries }, { oranges, lemons }, { plums, peaches }, { apples }, { bananas } Everything in an equivalence class is related to each other. 19
Determining equivalence classes Idea: give every equivalence class a name { oranges, limes, lemons } = like-oranges { peaches, plums } = like-peaches Etc. To answer if two fruits are related: FIND the name of one fruit s e.c. FIND the name of the other fruit s e.c. Are they the same name? 20
Building Equivalence Classes Start with disjoint, singleton sets: { apples }, { bananas }, { peaches }, As you gain information about the relation, take UNION of sets that are now related: { peaches, plums }, { apples }, { bananas }, E.g. if peaches R limes, then we get { peaches, plums, limes, oranges, lemons } 21
Disjoint Union - Find Maintain a set of pairwise disjoint sets. {,5,7}, {,2,8}, {9}, {1,6} Each set has a unique name, one of its members {,5,7}, {,2,8}, {9}, {1,6} 22
Union Union(x,y) take the union of two sets named x and y {,5,7}, {,2,8}, {9}, {1,6} Union(5,1) {,5,7,1,6}, {,2,8}, {9}, 2
Find Find(x) return the name of the set containing x. {,5,7,1,6}, {,2,8}, {9}, Find(1) = 5 Find() = 8 2
Example S {1,2,7,8,9,1,19} {} {} {5} {6} {10} {11,17} {12} {1,20,26,27} {15,16,21}.. {22,2,2,29,9,2,,5,6} Find(8) = 7 Find(1) = 20 Union(7,20) S {1,2,7,8,9,1,19,1,20 26,27} {} {} {5} {6} {10} {11,17} {12} {15,16,21}.. {22,2,2,29,9,2,,5,6} 25
Implementing Disjoint Sets n elements, Total Cost of: m finds, n-1 unions Target complexity: O(m+n) i.e. O(1) amortized per operation. O(1) worst-case for find as well as union would be great, but Known result: find and union can be done practically in O(1) time 26
Implementing Disjoint Sets Observation: trees let us find many elements given one root Idea: if we reverse the pointers (make them point up from child to parent), we can find a single root from many elements Idea: Use one tree for each equivalence class. The name of the class is the tree root. 27
Up-Tree for Uinon/Find Initial state 1 2 5 6 7 Intermediate state 1 7 2 5 Roots are the names of each set. 6 28
Find Operation Find(x) follow x to the root and return the root 1 7 2 5 Find(6) = 7 6 29
Union Operation Union(i,j) - assuming i and j roots, point i to j. 1 7 Union(1,7) 2 5 6 0
Simple Implementation Array of indices up 1 2 5 6 7 0 1 0 7 7 5 0 Up[x] = 0 means x is a root. 1 7 2 5 6 1
Union void Union(int[] up, int x, int y) { //precondition: x and y are roots Up[x] := y } Constant Time! 2
Exercise Design Find operator Recursive version Iterative version int Find(int[] up, int x) { //precondition: // x is in the range 0 to size-1??? }
A Bad Case 1 2 n 2 n Union(1,2) 1 1 2 n n Union(2,) : : Union(n-1,n) 2 Find(1) n steps!! 1
Now this doesn t look good Can we do better? Yes! 1. Improve union so that find only takes O(log n) Union-by-size Reduces complexity to Θ(m log n + n) 2. Improve find so that it becomes even better! Path compression Reduces complexity to almost O(m + n) 5
Weighted Union Weighted Union Always point the smaller tree to the root of the larger tree 1 2 1 7 W-Union(1,7) 2 5 6 6
Example Again 1 2 n 2 n Union(1,2) 1 1 2 Union(2,) n : : Union(n-1,n) 1 2 n Find(1) constant time 7
Analysis of Weighted Union With weighted union an up-tree of height h has weight at least 2 h. Proof by induction Basis: h = 0. The up-tree has one node, 2 0 = 1 Inductive step: Assume true for all h < h. Minimum weight up-tree of height h formed by weighted unions T W(T 1 ) > W(T 2 ) > 2 h-1 T 1 T 2 h-1 Weighted union Induction hypothesis W(T) > 2 h-1 + 2 h-1 = 2 h 8
Analysis of Weighted Union Let T be an up-tree of weight n formed by weighted union. Let h be its height. n > 2 h log 2 n > h Find(x) in tree T takes O(log n) time. Can we do better? 9
Worst Case for Weighted Union n/2 Weighted Unions n/ Weighted Unions 0
Example of Worst Cast (cont ) After n -1 = n/2 + n/ + + 1 Weighted Unions log 2 n If there are n = 2 k nodes then the longest path from leaf to root has length k = log 2 (n). Find 1
Elegant Array Implementation 1 2 1 7 2 5 6 up weight 1 2 5 6 7 0 2 1 0 1 7 7 5 0 2
Weighted Union void W_Union(int i,j){ //Pre: i and j are roots// int wi = weight[i]; int wj = weight[j]; if (wi < wj) { up[i] = j; weight[j] = wi + wj; } else { up[j] =i; weight[i] = wi + wj; } }
Path Compression On a Find operation point all the nodes on the search path directly to the root. 1 7 1 7 2 5 PC-Find() 2 6 5 6 8 9 10 8 9 10
Self-Adjustment Works PC-Find(x) x 5
Exercise: Draw the result of Find(e) c g a f h b d i e 6
Path Compression Find int PC_Find(int i) { int r = i; while (up[r]!= 0) //find root r = up[r]; if (i!= r) { //compress path// int k = up[i]; while (k!= r) { up[i] = r; i = k; k = up[k]; } } return r; } 7
Interlude: A Really Slow Function Ackermann s function is a really big function A(x, y) with inverse (x, y) which is really small. How fast does (x, y) grow? (x, y) = for x far larger than the number of atoms in the universe (2 00 ) shows up in: Computation Geometry (surface complexity) Combinatorics of sequences 8
A More Comprehensible Slow Function log* x = number of times you need to compute log to bring value down to at most 1 E.g. log* 2 = 1 log* = log* 2 2 = 2 log* 16 = log* 2 22 = (log log log 16 = 1) log* 6556 = log* 2 2 22 = (log log log log 6556 = 1) log* 2 6556 = = 5 Take this: (m,n) grows even slower than log* n!! 9
Disjoint Union / Find with Weighted Union and Path Compression Worst case time complexity for a W-Union is O(1) and for a PC-Find is O(log n). Time complexity for m n operations on n elements is O(m log* n) Log * n < 7 for all reasonable n. Essentially constant time per operation! 50
Amortized Complexity For disjoint union / find with weighted union and path compression. average time per operation is essentially a constant. worst case time for a PC-Find is O(log n). An individual operation can be costly, but over time the average cost per operation is not. 51
Cute Application Build a random maze by erasing edges. 52
Pick Start and End Cute Application Start End 5
Cute Application Repeatedly pick random edges to delete. Start End 5
Desired Properties None of the boundary is deleted Every cell is reachable from every other cell. There are no cycles no cell can reach itself by a path unless it retraces some part of the path. 55
A Cycle, not allowed Start End 56
A Good Solution Start End 57
A Hidden Tree Start End 58
Number the Cells We have disjoint sets S ={ {1}, {2}, {}, {}, {6} } each cell is unto itself. We have all possible edges E ={ (1,2), (1,7), (2,8), (2,), } 60 edges total. Start 1 2 5 6 7 8 9 10 11 12 1 1 15 16 17 18 19 20 21 22 2 2 25 26 27 28 29 0 1 2 5 6 End 59
Basic Algorithm S = set of sets of connected cells E = set of edges While there is more than one set in S pick a random, unused edge (x,y) from E u := Find(x); v := Find(y); if u v { Union(u,v); remove (x, y) from E } else mark (x, y) as used ; All remaining members of E form the maze 60
Example Step Pick (8,1) Start 1 2 5 6 7 8 9 10 11 12 1 1 15 16 17 18 19 20 21 22 2 2 25 26 27 28 29 0 1 2 5 6 End S {1,2,7,8,9,1,19} {} {} {5} {6} {10} {11,17} {12} {1,20,26,27} {15,16,21}.. {22,2,2,29,0,2,,5,6} 61
S {1,2,7,8,9,1,19} {} {} {5} {6} {10} {11,17} {12} {1,20,26,27} {15,16,21}.. {22,2,2,29,9,2,,5,6} Example Find(8) = 7 Find(1) = 20 Union(7,20) S {1,2,7,8,9,1,19,1,20 26,27} {} {} {5} {6} {10} {11,17} {12} {15,16,21}.. {22,2,2,29,9,2,,5,6} 62
Start Example Pick (19,20) 1 2 5 6 7 8 9 10 11 12 1 1 15 16 17 18 19 20 21 22 2 2 25 26 27 28 29 0 1 2 5 6 End S {1,2,7,8,9,1,19 1,20,26,27} {} {} {5} {6} {10} {11,17} {12} {15,16,21}.. {22,2,2,29,9,2,,5,6} 6
Example at the End Start 1 2 5 6 7 8 9 10 11 12 1 1 15 16 17 18 19 20 21 22 2 2 25 26 27 28 29 0 1 2 5 6 End S {1,2,,,5,6,7, 6} 6
Problem 6-6 Suppose we are given the minimum spanning tree T of a given graph G (with n vertices and m edges) and a new edge e = (u; v) of weight w that we will add to G. Give an efficient algorithm to find the minimum spanning tree of the graph G + e. Your algorithm should run in O(n) time to receive full credit, although slower but correct algorithms will receive partial credit.
Problem 6-21 6-21. [5] Let G = (V,E) be a weighted acyclic directed graph with possibly negative edge weights. Design a linear-time algorithm to solve the single-source shortest-path problem from a given source v.
Short Paths in a DAG If we update the edges in topologically sorted order, we correctly compute the shortest paths. Reason: the only paths to a vertex come from vertices before it in the topological sort. s 9 0 1 6 1 2
Code for Shortest Paths in DAG 1 DIJKSTRA(G, w, s) // DAG, weights, start vertex 2 for each vertex v in V[G] do d[v] p[v] NIL 5 d[s] 0 6 Q the topological order of V[G] starting at s 7 while Q is not empty do 8 u = EXTRACT-FIRST(Q) 9 for each vertex v in Adj[u] 10 d[v] = min{ d[v], d[u]+w(u, v)} 11 update p[v] if necessay
Shortest Paths in a DAG Theorem: For any vertex u in a DAG, if all the vertices before u in a topological sort of the dag have been updated, then d[u] is d(s,u), the shortest distance from s to u. Proof: By induction on the position of a vertex in the topological sort. Base case: d[s] is initialized to 0. Inductive hypothesis: Assume all vertices before u have been updated, and for all such vertices v, d[v] is d(s,v), the shortest distance
Proof, Continued Inductive case: Some edge (v,u) where v is before u, must be on the shortest path to u, since there are no other paths to u. When v was updated, we set d[u] to d[v]+w(v,u) = d(s,v) + w(v,u) = d(s,u) Running time: q(v+e), same as topological sort