Heuristics for Semi-External Depth-First Search on Directed Graphs

Transcription

1 Heuristics for Semi-External Depth-First Search on Directed Graphs Jop F. Sibeyn Institut für Informatik, Universität Halle, Germany jopsi James Abello ATT Labs Research, Florham Park, USA abello Ulrich Meyer Max-Planck-Institut für Informatik, Saarbrücken, Germany MPI Informatik Ulrich Meyer

2 The Problem: Depth-First Search (DFS) Graph: G = (V, E), V = n, E = m, directed, artificial source node. Find: Some valid DFS tree: 0 self loop backward tree edge backward cross edge 9 10 forward tree edge 4 forward cross edge 11 Standard implementation for internal memory: O(n + m) time. MPI Informatik Ulrich Meyer

3 The External Memory (EM) Model [VS94] CPU main memory of size M c secondary memory = D independent disks data is transfered in blocks of size Memory M up to D data per I/O step goal is to minimize the number of I/O steps scanning x items takes scan(x) := O( x D ) I/Os Disk 1 Disk i Disk D sorting x items takes sort(x) := O( x D log M/ x ) I/Os MPI Informatik Ulrich Meyer

4 Previous Work on External DFS Simulate IM Algorithm with EM Data Structures Fully-External (M n): 95 Chiang et al. O(n + M n scan(n + m)) I/Os. 96 Kumar/Schwabe O((n + D m ) log 2 n) I/Os, undirected 00 uchsbaum et al. O((n + D m ) log 2 n) I/Os, directed Special graph classes, e.g., planar graphs: O(sort(n + m)) I/Os. Semi-External (Θ(n) M m): O(n + m D ) I/Os. General sparse graphs nothing gained due to Ω(n) unstructured I/Os!! MPI Informatik Ulrich Meyer

5 Do we have a problem at all? Why DFS: - Central procedure to find Strongly Connected Components Identify communities in the web. - DFS can be used for topological sorting. ig graphs: WWW: n , m ; Telephone call graphs, GIS, etc. Slow disks: 1 I/O 10 msec; I/Os 230 days, infeasable!! Why not internal: - Economical reasons (m/n small) - Absolutely impossible (m/n huge) Constants matter: Less usefull if alg. requires M c n for some large c. MPI Informatik Ulrich Meyer

6 Key Difficulties in External-Memory DFS nodes are reached in unpredictable order: resulting unstructured access to adjacency lists seems to require one I/O per node. remembering visited nodes asks for a data structure to store the set of reached nodes; otherwise one I/O per edge. graph nodes s external adjacency lists the second problem disappears in semi-external setting: internal bit-vector: O(n + scan(n + m)) I/Os. our heuristics mostly overcome the first problem, too: times faster than n unstructured I/Os. MPI Informatik Ulrich Meyer

7 asic Idea for Semi-External DFS Maintain a tentative tree in internal memory. Gradually improve it towards a DFS tree using blocked I/O. Test for termination. RISK: Never converging to a final state. Next: First solution for undirected graphs. MPI Informatik Ulrich Meyer

8 MPI Informatik Ulrich Meyer EDGE-Y-EDGE (undirected) Edges from external memory are considered cyclically one-by-one. Cross edge found cutting and relinking in the tentative tree. Internal Memory A Internal Memory External Memory External Memory E G H C I L M Q D P O N J K A K L K O P O G C D H L K G L P P J E P L G K L A K O O G C D H M L D A E J K N O G P I Q C H Termination: Complete scan without any cross edge.

9 EDGE-Y-EDGE (cont d) Internal Tree: Construct on the fly during first scan. Each op. O(logn) amortortized time [ST85]. Performance: Processing x edges needs: O(x/) I/Os & O(x logn) time. 2 m x? x depends on: Modification rule: Initial order of edges: removing a cross edge may create new ones. may take long to find the next cross edge. MPI Informatik Ulrich Meyer

10 Modification rule & initial edge ordering "cut on lower side" "cut on higher side" 1 A A A 1 2 F F 2 3 D G C G C 3 4 C E D D 4 5 E E F 5 6 G 6 cut on the higher side : depths never decrease random cutting : no harm, no progress cut on the lower side : reached depths may decrease For each rule there are graphs and edge orders that require Ω() rounds Ω(m) I/Os. UT: Initial randomization and filtering of multi-edges can reduce # phases. MPI Informatik Ulrich Meyer

11 Edge-by-Edge Experiments Cutting on the higher side & edge randomization / filtering performs best. We never experienced more than log 2 n rounds cross edges m = 50 * n m = 25 * n m = 10 * n m = 5 * n m = 3 * n scanned edges * m UT: Only for undirected graphs. UT: Internal dynamic trees are relatively slow. UT: 20 integers/pointers storage per internal node, too much! MPI Informatik Ulrich Meyer

12 Omit complicated data structures. ATCHED PROCESSING At all times n edges giving a tree rooted at n. Internal edges maintained in adjacency lists, external edges as node pairs in initially random order. Repeatedly perform: read next n edges from file. insert new edges into adjacency lists. perform internal DFS on the 2 n edges. update set of internal/external edges. Termination when tree remains unchanged during a round, i.e., processing all m/n batches. MPI Informatik Ulrich Meyer

13 Tricks (1): Rearrangements of Adjacency Lists Problem: Unguided batched processing has slow convergence. Idea: Make the internal DFS search more goal directed; find some equivalent to the Cutting on the higher side rule. After each batch, rearrange order of the adjacency lists for internal DFS: give preference to edges already existing in the tentative tree. give preference to edges towards nodes that are roots of large subtrees. Internal tree trends to grow deep faster. # removed cross-edges trends to be larger than # newly created forw. cross-edges. MPI Informatik Ulrich Meyer

14 Example for Rearrangements DFS tree of a random graph is long and skinny. This is already true for very sparse graphs, i.e., after the first few batches. Wrong arrangements could destroy previous achievements. MPI Informatik Ulrich Meyer

15 Tricks (2): Reductions General Idea: After each full round compare tree with tree before the round and look at the edges on the file. Determine hereby passive nodes, having reached final positions. Reduce tree and edges on file accordingly. Aggressive Reduction: Throw out nodes and edges that are probably not needed anymore. Perform main loop until separate test shows that # cross-edges = 0. Gentle Reduction: Throw out nodes and edges that are certainly not needed anymore. Termination when # edges is small enough for internal processing. MPI Informatik Ulrich Meyer

16 Gentle Reduction Experience: Gentle reduction has proven to be best. Three sources of passive nodes: Stable initial part of the tentative tree (smallest preorder numbers). Stable final part of the tentative tree (highest preoder numbers). No passing edge. MPI Informatik Ulrich Meyer

17 Example for Reduction 1e+10 edges scanned so far edges scanned in round active nodes 1e+09 1e+08 1e+07 1e round... for a graph model of the WWW. MPI Informatik Ulrich Meyer

18 The program written in C. Internal-memory space requirements: ( 3 size(int) + 3 size(bool) ) n n bytes (reusing arrays; recomputing data). External-memory space requirements: 4 size(int) m 16 m bytes (core functionality: 2 size(int) m). Extra features: renumbering nodes consecutively; eliminating multiple edges and self loops; randomizing the input file; making the graph undirected; statistics & correctness checkers; input generators; MPI Informatik Ulrich Meyer

19 Performance (1) Random Graphs: After first round almost all nodes and edges eliminated; only about 10 times slower than internal algorithm. Web Graphs Models: For sparse graphs and growing n: slowly increasing # rounds, with strongly decreasing problem size; a bit worse on denser graphs. Worst Case: idirectional cycles. Additional preprocessing with randomization helps. Right now we do not know any natural class of graphs not efficiently tackled. Knowing what the program is doing and disallowing randomization, one can easily construct bad examples though. MPI Informatik Ulrich Meyer

20 Performance (2) For graphs with n = and m = n I/Os sec. r = # scanned and processed edges /m. Graph First DFS Second DFS r T tot [sec] T I/O [sec] r T tot [sec] T I/O [sec] RAND CYCLE GEOM-1D GEOM-2D CF-WE SIMPLE-WE OUT-STAR IN-OUT-STAR ACYC our approach is not I/O-bound; current gains: factor with increasing PU speed it will be even more efficient. MPI Informatik Ulrich Meyer

21 Influence of the average node degree: Depending on the graph class Performance (3) (a) dense graphs are more difficult: (b) dense graphs are easier: r 7 r OUT-STAR IN-OUT-STAR CF-Web Simple-Web n 1e m/n n e m/n MPI Informatik Ulrich Meyer

22 Conclusions Seen: Efficient heuristic for Semi-External DFS. Also tested on real world data: Input: Aggregated call graph from ATT; n 10 7, m Our heuristic on a Pentium III with 1 GHz: SCC ( 2 times DFS ) takes about 4 hours (2 n I/Os 55 hours). Some open problems: In [MehMey, ESA02] we show that FS can be solved using o(n) +O(sort(n + m)) I/Os in the worst-case. Can we obtain similar results for DFS? Is there an ω(sort(n + m)) I/O lower bound for sparse graphs? MPI Informatik Ulrich Meyer