CFinder The Community / Cluster Finding Program. Users' Guide

Transcription

1 CFinder The Community / Cluster Finding Program Users' Guide Copyright (C) Department of Biological Physics, Eötvös University, Budapest, 2005

2 Contents 1. General information and license Quick start Command line version (batch, no visualization) for Linux CFinder is very slow for my network, can you help? Theoretical background The Clique Percolation Method The outline of the community finding algorithm Download and Installation How to download and install CFinder How to start CFinder Directories and files A note for Mac OS X users with versions below 10.4 (Tiger) A note for Linux users How much memory and CPU time does CFinder need for my network? How do I increase the amount of memory used by CFinder? The user interface The File menu Open new network Output directory for the computed cliques, communities, etc Open communities (choose directory with previously computed data) The View menu View > Vertices: View the communities of a selected vertex The graph visualization panel View > Communities The Zoom and Walk buttons: Explore your network and navigate between the different views View > Cliques View > Stats: Statistics of the communities View > Graph of communities The Tools menu Tools > Export graph Tools > Stop calculation Tools > Settings Notes A note on the Export graph function How to contact us

3 1. General information and license Description How to cite CFinder is a software tool for network cluster (community) detection, based on the Clique Percolation Method, CPM, published in Palla et. al. Uncovering the overlapping community structure of complex networks in nature and society. Nature 435, (2005). Version CFinder 1.2 (Dec 15, 2005) Platforms Windows, Linux, Mac OS X Website Software requirements CFinder uses the Java Runtime Environment version 1.4 or higher (from Sun). On most computers this is installed. If you have a different Java engine, or only an earlier version of the same engine, you can download J2SE JRE from License Copyright (c) Department of Biological Physics, Eötvös University, Budapest, For non-profit users the permission to use, copy, and modify this software and its documentation for any purpose is hereby granted without fee, provided that the above copyright notice appears in all copies and that both the copyright notice and this permission notice appear in supporting documentation. All other users are kindly requested to contact the holders of the license (see below for contact information). The holders of the license make no representations about the suitability of this software for any purpose. It is provided as is without express or implied warranty. How to contact us cfinder@angel.elte.hu Web, regular mail: select People 3

4 2. Quick start How to start CFinder Input file format, example Output files of CFinder Compute the communities of your network Open already computed communities View the communities of a node Windows: open My Computer, change to the directory of the unzipped package, and double-click on the file start.bat Linux and Mac OS X: launch start.sh at the command prompt (xterm, Terminal, console) Mac OS X: double-click on the disk image CFinder.dmg File type Plain text file Comment lines Data lines Field separator Example Start with a # character Each data line has to contain the two nodes defining the link (two strings) followed by an optional non-negative number, the weight of the link. Self-links are ignored. One space or one TAB character ---file begins below this line--- # you can use comments (this line) # and empty lines node1 node node2 node node1 node file ends above this line----- CFinder writes all data (communities, cliques, statistics, etc.) into one output directory. This directory is at the same location (path) on your computer as the input file and its name is the name of the input file plus a _files suffix. All output files are plain text files. 1. File > Open new network > Browse, and select the input file. The demo files are in the demo subdirectory of the package. 2. You can chose the lower and upper threshold weights for links. If you do not need thresholds, leave these fields empty. 3. Run. File > Open communities, and select the directory containing the already computed communities of a network. The name of the directory ends with _files. 1. Compute the communities of the network (see above). Four panels will appear in the CFinder window. 2. The lower left panel lists all nodes of the network and each node will be followed by the number of its communities in brackets. 3. Select a node in the lower left panel. In the upper right panel a list of k values will appear. These are the k values (clique size values used for the community finding) for which the selected node is contained by at least one community. 4. Select a k value in the upper left panel: the communities of the selected node at this k will appear in the lower right panel in black. 5. In the upper right panel the communities are listed on sheets: you can bring a sheet to the front by clicking on its tab on the top. Each sheet contains the list of nodes and the list of links of one community. By default the sheet of the first community is on top. 6. The lower right panel shows the communities that the selected node is a member of. The community that you select in the upper right panel will be highlighted in the lower right panel. 4

5 Export network to Comment: node vs. vertex 1. IMAGE FILE: Select Tools > Export graph. Currently available image formats are: JPEG, EPS (Encapsulated PostScript) and PNG (Portable Network Graphics). 2. PAJEK.NET FILE: This option will be available soon under Tools > Export graph. (This is the input file format for Pajek.) We use the words node and vertex with identical meaning. Similarly: edge = link and graph = network Command line version (batch, no visualization) for Linux The executable CFinderBatch is the command line version of CFinder for Linux. It uses no visualization. To view a message on the usage of this program, type CFinderBatch h CFinder is very slow for my network, can you help? Yes. Please, contact us at cfinder@angel.elte.hu. Note that there can be several reasons for an unusually slow community finding. One of the most frequently occurring reasons is that the core (the most densely connected region) of the network contains several strongly overlapping cliques of size (or above). You can read further details about the amount of CPU time and memory used by CFinder in Sections 4.2 and Theoretical background 3.1. The Clique Percolation Method The Clique Percolation Method (CPM) used by CFinder is designed to locate the k- clique communities of unweighted, undirected networks. This community definition is based on the observation that a typical member in a community is linked to many other members, but not necessarily to all other nodes in the community. In other words, a community can be interpreted as a union of smaller complete (fully connected) subgraphs that share nodes. Such complete subgraphs in a network are called k- cliques, where k refers to the number of nodes in the subgraph, and a k-cliquecommunity is defined as the union of all k-cliques that can be reached from each other through a series of adjacent k-cliques. Two k-cliques are said to be adjacent if they share k-1 nodes. An illustration of these communities can be given by k-clique template rolling. A k- clique template can be thought of as an object that is isomorphic to a complete graph of k nodes. Such a template can be placed onto any k-clique of the network, and rolled to an adjacent k-clique by relocating one of its nodes and keeping its other k-1 nodes fixed. Thus, the k-clique-communities of a graph are all those subgraphs that can be fully explored by rolling a k-clique template in them but cannot be left by this template. 5

6 The notion of CPM was inspired by the fact that the k-clique communities also correspond to percolation clusters in the k-clique adjacency graph of the system. The nodes of the k-clique adjacency graph represent the k-cliques of the original network, and there is an edge between two nodes if the corresponding two k-cliques are adjacent. The advantages of the above community definition are the following: it is not too restrictive (unlike cliques that require each node to be connected to all other nodes), it is based on the density of links, it is local, it does not yield cut-nodes or cut-links (whose removal would disjoin the community), and it allows overlaps: (i) a node can be a member of several different communities at the same time, and (ii) communities can overlap with each other by sharing nodes. Naturally, the communities also constitute a network with these overlaps as their links. The number of such links of a community can be called its community degree The outline of the community finding algorithm The k-clique community finding algorithm implemented in CFinder first extracts all such complete subgraphs of the network that are not included in any larger complete subgraph. These maximal complete subgraphs are simply called cliques (the difference between k-cliques and cliques is that k-cliques can be subsets of larger complete subgraphs). Once the cliques are located, the clique-clique overlap matrix is prepared. In this symmetric matrix each row (and column) represents a clique and the matrix elements are equal to the number of common nodes between the corresponding two cliques, while each diagonal entry is equal to the size of that clique. The k-clique-communities for a given value of k are equivalent to such connected clique components in which the neighboring cliques are linked to each other by at least k-1 common nodes. These components can be found by erasing every offdiagonal entry smaller than k-1 and every diagonal element smaller than k in the matrix, replacing the remaining elements by one, and then carrying out a component analysis of this matrix. The resulting separate components will be equivalent to the different k-clique-communities. 4. Download and Installation 4.1. How to download and install CFinder CFinder can be downloaded from You can unpack the downloaded zip file with WinZip ( the standard Linux unzip utility, or Aladdin Stuffit Expander ( 6

7 CFinder uses the Java Runtime Environment version 1.4 or higher by Sun. Type java version (Windows users: at the DOS prompt, Linux and Mac users: at the command prompt) to see if you have this version installed on your computer. If an error occurs when executing the "java" command, or the version number is below 1.4, or you have a different Java engine, then please, download J2SE JRE from and install it. Having completed the installation of Java, you will need to make sure that your operating system can find it. On Windows systems this is carried out automatically, but on some Linux systems you need to append to your PATH environment variable the following: $JAVA_HOME/bin, where $JAVA_HOME is the name of the directory where you installed Sun s J2SE JRE How to start CFinder Windows: double-click on start.bat or execute it at the DOS command prompt. Linux and Mac: execute start.sh at the command prompt. Mac: double-click the disk image CFinder.dmg Directories and files These are the directories and files in the downloadable package of CFinder. demo/ doc/ lib/ start.bat start.sh CFinder.dmg README.txt LICENSE.txt Sample networks Documentation Libraries required for CFinder commfind.dll, libcommfind.so, libcommfind.jnilib: C++ routines of CFinder for Windows, Linux, and Mac community.jar: Java class files of CFinder jcommon pre2.jar, jfreechart pre2.jar, epsgraphics.jar: additional Java class files Start batch file for Windows (by double-clicking or from the command prompt) Start batch file for Linux and Mac (from the command prompt) Mac disk image file, double-click to start CFinder General information License 4.4. A note for Mac OS X users with versions below 10.4 (Tiger) If automatic updates are not enabled on your Mac, then please, update your system before using CFinder. Click on the Apple icon in the top left corner and select Software Update. Make sure to update both Java and Mac OS X to their most recent versions A note for Linux users On several Linux systems the default Java engine is not the Java Runtime Environment (by Sun) and/or there are various other Java engines available. 7

8 Java is meant to be platform independent. Unfortunately, the different engines running Java programs are not identical, and quite often these engines have unexpected differences in the way they run a given Java program. We suggest that you should use the Java Runtime Environment version 1.4 (by Sun) or higher to run CFinder. For the installation, follow these steps: 1. Download Sun s Java Runtime Environment (binary version) for your system into a temporary file called Sun_Java.bin. 2. Install Java locally: make a new directory called sun-java, and run Sun_Java.bin in that directory. 3. In the start.sh file replace java (the first word of the first line) with the java executable of the newly installed package, this is called usually something like./sun-java/j2re1.4.2_10/java How much memory and CPU time does CFinder need for my network? The amount of memory needed for the community finding in your network will be most strongly influenced by the number and the overlaps of cliques / dense regions. Of course, the number of nodes is also important, but the necessary CPU time and memory grow only slowly with the number of nodes. A short explanation follows here. (Jump to the next paragraph, if you want to receive practical advice.) A linear chain of nodes with a few loops on it contains only a small number of small cliques, e.g., triangles, and CFinder will need little memory and CPU time for this network. If the input network is a complete graph (each node is connected to all other nodes) is one clique, and resources of your computer needed for this network will be small again. Between these two extremes are those networks that need more memory and CPU time. These contain a high number of large, overlapping cliques: there are large dense regions in the network, but a typical node is not connected to all other nodes. (Remember that a clique is a complete subgraph not contained by any other complete subgraph.) In practice, the best way to run CFinder on a large network is to use confidence values for the links. The confidence (or weight ) of each link can be used for filtering the links in the New Community Finding dialog. At the first run, set a high value for the lower cutoff in the New Community Finding dialog. This will remove many links from the network and the remaining graph that CFinder receives for analysis will be sparse. Later, you can run CFinder again (File > Open new community) and decrease step by step the lower threshold to leave more and more links in the network How do I increase the amount of memory used by CFinder? If you are using a large network and/or a network with many overlapping dense regions, and CFinder stops with an error message explaining that more memory would be necessary, then in the start batch file increase the memory requested by CFinder from your operating system. In the start batch files (start.bat on Windows and start.sh on Linux and Mac) you will find two switches: -Xmx224m and -Xss32m. Increase both numbers (224 and 32) such that their sum remains smaller than the physical memory of your computer. 8

9 5. The user interface 5.1. The File menu Open new network Opens the New Community Finding dialog. In this dialog click on browse to open a data file containing the edges of a network. In the program package that you have downloaded and unpacked there are sample network files in the demo subdirectory. Figure 5.1. The main CFinder window with the New Community Finding and the Open file dialogs. Once you have selected the input file, use the fields to the left and right from the word weight to select a lower and upper cutoff for link weights. For the community finding only those links of the network will be used whose weights (as listed in the third column of the input file) are between the lower and the upper cutoff values. All other links of the network will be discarded before the community finding starts. If the input network file provides no weights for the links, then CFinder will ignore the cutoff values that you enter in the New Community Finding dialog. Click on run to start the community finding. While the communities of the network are computed, you will see five progress bars indicating the status of the community search Output directory for the computed cliques, communities, etc. To save the results, CFinder creates a new directory. This directory will be created at the same location as the analyzed input network file, and the name of the output directory will be the name of the input network file plus a _files suffix. If you select a lower and upper threshold for the links weights, then these two weights will be appended to the name of the output directory after _files, e.g., if the input file mynetwork.txt is analyzed with the lower and upper thresholds 0.2 and 0.5, then the output directory will be called mynetwork.txt_files_0.2_w_0.5. CFinder writes all the computed communities, cliques, and statistics into this directory. 9

10 Open communities (choose directory with previously computed data) If you have already run CFinder for a network and created an output directory, you can view the results again by selecting File > Open communities. Figure 5.2. The main CFinder window with the dialog for opening already computed communities. 5.2 The View menu After computing the communities of a network (or after loading the results of a previous community finding), CFinder can show you the results in various formats. The default view is Communities View > Vertices: View the communities of a selected vertex In this view the list of node names appears in the lower left panel. There is a search field at the bottom of the node name list. To find a node in the list, type its name into this search field, and press ENTER. Please, make sure to type the name with the correct spelling. You can also type the first few characters of the name to find the first name on the list that begins with these characters. In the list of nodes after the name of each node the number of its communities i.e., the number of communities that the node is contained by is given in brackets. If the node has no communities, then there is no number. The communities of a node can have different k values (k is the clique size parameter in the community search algorithm). Select a node in the lower left panel to view in the upper left panel the list of k values of the communities containing this node. In the upper left panel the k values of the communities of the selected node are listed. A lower k value means communities with lower link density (less stringent community search resulting in larger communities), and a higher k value means communities with higher link density (a smaller number of stronger, but more cohesive communities). In 10

11 the upper left panel select one of the k values to view the communities of the selected vertex at this k value. The communities will be displayed in the lower right panel. Figure 5.3. Communities of the protein Pwp2p in the DIP yeast core protein-protein interaction network. The proteins Krr1p, Pwp2p and Sik1p are colored red, because they participate in more than one of the displayed communities, i.e., they are the overlaps. Data source: Database of Interacting Proteins (DIP) The graph visualization panel You can adjust the graph displayed in the lower right panel the graph visualization panel in several ways. Some of them are listed in the table below. Please, check also this document about the dialog Tools > Settings (5.3.3). Zoom in / out Move entire graph Select one node Select nodes Move selected items Grab and move a node Use the scrollbar on the bottom Left-click on the background and drag Right click on the node With the right button select the rectangle containing these nodes Select the items, then left-click and drag Left-click on the node and move it Note that for large communities (above 1000 vertices) the computation of the graph s layout may take longer. If you decide not to wait until the layout routine finishes, you can select a different vertex and k (clique size) parameter value: the previous layout calculation will be cancelled and the newly selected communities will be displayed in the graph visualization panel. If you would like to browse through the communities of selected nodes, but you do not want to visualize them as a graph, select Tools > Settings, and uncheck the Display graph item. 11

12 View > Communities In this view you can have a look at the communities separately. However, if you would like to explore the connections of a community to other communities, then you can select a vertex in the graph and click on the walk button. This will bring you to the Vertices view where all communities of the selected node are displayed. Remember that at different k (community finding stringency) values the communities of a node can be different. If you start from a community with k=4, then pressing the walk button will bring you to the communities of the selected node at k=4. Figure 5.4. Community no. 6 at k=4 in the DIP yeast core protein-protein interaction network. In the left panel communities are listed by their k values (community finding stringency values). Click on one of the listed communities to view it as a graph in the lower right panel. To highlight a vertex or an edge in the graph, click on its name in the upper right panel. The cliques contained by the selected community are also listed in this panel The Zoom and Walk buttons: Explore your network and navigate between the different views The Zoom and Walk buttons help you to navigate between the various views. To learn what actions they will perform in your current view, just read their text labels. You can start by looking at the communities of a selected vertex, then highlight one community (use the tabs in the upper right panel) and press the zoom button to view the graph of communities around this community. In the network of communities you can select a different community and click on the walk button to see the neighbors of that community: you are walking on the graph of communities and CFinder shows you the neighborhood of the selected community. After your walk on the community graph, you can select some of the communities and press zoom to view their nodes. 12

13 View > Cliques This option lists the cliques detected by CFinder. Each clique has an index. To view the list of nodes in a clique, open the folder of the clique. A short reminder: a clique is a complete subgraph (each node is connected to all other nodes) not contained by any other complete subgraph. Figure 5.5. Clique view: the cliques of the DIP yeast core protein-protein interaction network are listed in the left panel View > Stats: Statistics of the communities. With this option you can view the community statistics in your network for a selected k- clique size. The log-log plot button changes both scales to logarithmic (and back to linear), while the cumulative distribution button switches between the original histogram of values and the plot with 1 P, where P is the cumulated probability density function. Four types of statistics are available for each k-clique size: Quantity Community size Community degree Communitycommunity overlap Node membership number Definition Number of nodes in a community Number of other communities overlapping with a selected community Number of nodes contained by two overlapping communities Number of communities containing the selected node For the visualization of the statistics, CFinder uses the Java package JFreeChart To zoom in, left click inside the chart, hold down the left mouse button while moving the 13

14 mouse. This way you can select the rectangular area that you would like to enlarge. To zoom out completely and to view the entire distribution, right-click inside the chart to open a menu of options, and then select Autorange Both Axes from this menu. To export a distribution, right-click on the distribution, select Export from the pop-up menu and enter a file name in the Save dialog. Currently, there is one available file format for exporting the distribution (PNG), please enter a file name ending with.png. Further information about JFreeChart can be found at the website of JFreeChart: Figure 5.6. Statistics of the communities in the cond-mat collaboration network or researchers; cumulative distribution of community size, community degree, vertex membership, and community-community overlap. Data source: View > Graph of communities In the graph window only a part of the community graph is displayed: the community selected in the left panel and its neighbors within a fixed distance. This fixed distance is 2 by default, and it can be changed in the Tools > Settings menu: set the Community graph depth variable to your preferred value. To view the original nodes of a group of communities, select these communities and press zoom. In the graph (network) of communities (i) nodes represent the original network s communities and (ii) two nodes are connected, if the corresponding communities in the original graph overlap (i.e., they share at least one node). 14

15 Figure 5.7. A part of the network of communities (k=4) around community no. 40 in the DIP yeast proteinprotein interaction graph. Community no. 40 and its neighbors at a maximum distance of 4 are displayed The Tools menu Tools > Export graph Exports the graph currently displayed in the graph visualization panel (in the lower right panel) into a file. Currently, the available output formats are JPEG, EPS (Encapsulated PostScript) and PNG (Portable Network Graphics) Tools > Stop calculation Use this option to stop the calculation of the graph layout Tools > Settings This dialog contains general settings, options for the communities and for the graph of communities. General settings Draw graph. Select whether in the lower right panel the graph should be displayed. (The graph shows the communities of a selected vertex, a clique, a community or the network of communities.) Uncheck this option, if you would like to see only the list of nodes in a large community, but not the community as a graph. Highlight color. Color of the selected nodes and links in the graph. Draw node label and Draw node. Check these options to draw node labels and nodes. Node label size and Node size. The size of a node s label and the node itself. 15

16 Radius for selection. To select an item (node or link) in the graph, you need to click within a circle of this radius from that item. Edge handle size. The size of the disk at the center of a link that can help you to select and move the link. Figure 5.8. In the Tools > Settings dialog you can modify the display properties of the graph visualization panel of CFinder. Communities Color of selected community. Set the color of the currently highlighted community. Draw overlap. Check this option to use a different color for the overlaps of communities, i.e., the nodes and links belonging to more than one community. This option is ON by default. Overlap color. Color of the nodes and links that belong to more than one community. Graph of communities Node label color, Edge color, and Node color. The color of the nodes labels, the links and the nodes themselves in the graph of communities. Community graph depth. Draw only those communities whose distance from the selected community is not larger than this distance. Label lines with edge weight. The weight of an edge connecting two communities is given by the number of nodes in which they overlap, i.e., the number of nodes belonging to both communities. Check this option to put a number on each link telling its weight. 16

17 Line width shows edge weight. Check this option to allow a different width for each line. The width of the line will show the weight of the connection between the two communities (the size of their overlap). Nodes have the same size. If the nodes in the graph of communities are allowed to have different sizes, then these sizes will show the number of nodes (of the original network) contained by each community. Node size multiplier. Set the size of all nodes with this multiplier. 6. Notes 6.1. A note on the Export graph function On some Linux platforms the Export Graph function does not work. If you encounter this problem, please contact us. We can help you with this How to contact us cfinder@angel.elte.hu Web and regular mail: select People Thank you for using CFinder. 17