netzen - a software tool for the analysis and visualization of network data about

Transcription

1 Architect and main contributor: Dr. Carlos D. Correa Other contributors: Tarik Crnovrsanin and Yu-Hsuan Chan PI: Dr. Kwan-Liu Ma Visualization and Interface Design Innovation (ViDi) research group Computer Science Department, UC Davis netzen - a software tool for the analysis and visualization of network data about netzen is a tool for analysis and visualization of network data. It combines a series of analysis tools, such as centrality and clustering, with sensitivity analysis and uncertainty quantification, and provides a variety of visualizations for social network data. netzen provides integrated visualization of node-link diagrams with scatterplots, parallel coordinate views and other visualizations of node and edge properties, including centrality metrics such as closeness, betweenness, eigenvector and Markov centralities. It also incorporates sensitivity analysis and uncertainty evaluation of diverse graph metrics. In addition, netzen provides unprecedented flexibility to layout and customize views for presentation-ready visualization. netzen code is now hosted in Google code. Please follow this link for instructions on how to download the latest code via svn: For more information, see the netzen website at: correac/netzen/index.html

2 Contents about 1 1 Introduction Principles Overview Data Preparation Data files Edge files Workspace file Script file Tutorial - Multivariate Data Load the Data Set Create the Views Scatter Plot Parallel Coordinates Practice - Scatter Plot Matrix Save Workspace and create a Script Tutorial - Network Data Load a Data Set Create Views Graph View Discrete Legend Create the Layout Force Directed Layout Filter Edges Re-layout the graph Save Workspace and Script Workspace Script Practice - Color the Edges with Centrality ii

3 Bibliography 27 iii

4 1 Chapter 1 Introduction netzen provides an integrated analysis and visualization of multi-dimensional data sets, including graph data. Unlike other similar products, netzen incorporates statistics of your data along with standard visualization tools, such as scatter plots and parallel coordinate views. netzen is also an open-source package and allows the creation of novel views and analytic tools for fast prototyping. 1.1 Principles netzen is designed with the following characteristics: General Purpose. netzen is a general-purpose visualization and analysis package. For this reason, we consider data in its most basic form, as a collection of records with multiple attributes. These records can also be connected in a graph-structure, with each connection also having multiple attributes. Our visualization tools are designed to be as general as possible. These include scatter plots, parallel coordinates and node-link diagrams. The exposed visual attributes, such as color and size, can be mapped to any property in the data. Integrated. Unlike other visualization tools, netzen seeks to integrate statistical

5 2 and data analysis tools with visualization tools. These tools include clustering and centrality, among others. Other tools, such as Principal Component Analysis, will be added in the near future. Presentation-ready. netzen is the first visualization tool designed to provide presentationready images. This is achieved with our shared canvas structure and flexible view layout. Extensible. netzen is open-source, which means that it can be extended and adapted in a number of ways. Examples of these include from the adaptation of simple views, such as the scatter plot, to more complex ones, such as the flow-based scatter plot [1]. 1.2 Overview netzen creates visualizations as a composition in views in what we call the shared canvas, the main view on the left of the window, controlled via the tool box on the right. To understand how netzen works, we must understand the three main components: views, tools and layouts. View. A view is a visualization space, designed to present a view of the data. These can be created using the view create menu, and modified in the Inspector tab on the tool box. Examples of views are scatter plots, parallel coordinates views, histograms and node-link diagrams. Color legends are also considered views. All views are controlled via properties, which can be accessed from the Inspector tab. Properties allows you to assign a data property to the exposed visual properties, such as color, size and labels. Tool. A tool is a statistic or data analytics tool to modify or filter your data. These can be accessed from the Analysis Tools tab. Similar to views, their parameters can

6 3 be controlled by setting properties. Once the parameters are set, clicking on the Go button runs the tool to produce the desired results. Layout. A layout is a mapping of certain properties of the data to the x- and y- position of the visual elements. At this moment, layouts are only used for graph data. A force-directed layout, commonly used for graphs is an example. Analogous to views, a layout can be created using the layout create menu and accessed via the Layouts tab. To assign a layout to a view (only available for graph views), use the inspector on the corresponding view and assign an existing layout via the Layout property. On the next sections, you will learn to prepare your data for use in netzen, create a simple interactive visualization of a multi-dimensional data set, and an interactive visualization of a network data set.

7 4 Chapter 2 Data Preparation netzen uses a number of files to load both data and visualizations. Here we discuss the format of these files. 2.1 Data files netzen assumes that your data can be represented as a table. The data format consists of tab-delimited files, where each row is a data object, and each column represents a property value. To indicate the semantics of these properties, all data files contains a twoline header, where the first and second line specify the type and name of each property, respectively. A property is the basic data element to store some information about an object. netzen defines the following types, to be specified in the first line of all data files: INT - an integer number, usually a nominal variable. STRING - a string of text. FLOAT - a general numeric value (real number) SIGNED - a signed numeric value (real number)

8 5 BOOL - a boolean value (only takes values true or false). An example of a data file is seen in Fig Note that in the data file, if the property NodeId does not exist, netzen uses the corresponding row number (except the first two header rows) to create the property. Note that the numbering of NodeId starts from zero. Figure 2.1: A data file example Edge files A special type of data file is an edge file, where each row represents a connection between two data points. These are widely used to represent graph data. As seen in Fig. 2.2, edge files are like other regular data files, except for the inclusion of two required INT properties, which must be named From and To, and represent the source and destination of a connection between the corresponding data points. The values in these properties are mapped to the property NodeId in the data file containing the data objects (or nodes). Figure 2.2: An edge file example.

9 6 2.2 Workspace file A workspace file contains the information regarding the views and visual mappings used during a netzen session. These files are written automatically by netzen when you use File Save Workspace, and can be loaded again in a new session using File Load Workspace. A workspace saves the visual mappings (found under the Mapping tab, such as the node color and the edge color), created views, and their configuration (listed under the Inspector tab) and layouts. Note that workspaces do not save the data or newly computed properties. An example of a workspace file can be found in /YourNetzenFolder/examples/vast/- vast.ws. It contains the visual mapping (e.g., edge color, node color, node size and so on), a GraphView and a Force Directed Layout for the network data. 2.3 Script file A script file is a series of commands that netzen uses to load both the data and the workspace files. Each line contains one the following commands: importtable - loads a tab-delimited file containing data objects loadworkspace - loads a workspace importedgetable - loads an edge file. By default, each edge is considered as bi-directional, i.e., the information in each row of the edge file is attached to both directions of a connection. importdirectededgetable - loads an edge file, but each edge is uni-directional. For a script example of a network data set, see /YourNetzenFolder/ examples//mit/mit.script. It contains three lines: first one uses importtable command to load the node file; second

10 7 Figure 2.3: Script file for a network data set. line uses importedgetable command to load the edge file; and the last one uses load- Workspace command to load the workspace file. This is the most common way to write a scirpt. To reload the data set and restore the views, just load this script file in netzen by File Open Script and then locate the script file in the pop-up file selection window.

11 8 Chapter 3 Tutorial - Multivariate Data This tutorial guides you in the creation of a few views of a multivariate data set. 3.1 Load the Data Set The Iris data set consists of 150 records and four variables including the length and width of the sepal and petal of the plant regarding the classification of a number of species of the iris plant. Four properties of nodes are of FLOAT type and the name of the properties are SepalLength, SepalWidth, PetalLength, and PetalWidth. (See /YourNetzenFolder/examples/UCI/iris.txt for the multivariate node file. To load the dataset, click File Import Table and locate the iris.txt file. After loading the data, Info tab on the right shows which data sources were used and how many nodes have been loaded. 3.2 Create the Views In this tutorial we want to create four scatter plots and a parallel coordinate view for the dataset. We will also demonstrate how netzen enables linked view exploration via global visual mappings such as node color, node mapping, and node size.

12 9 Figure 3.1: The first scatterplot of PetalLength-PetalWidth for Iris dataset Scatter Plot Click View Create Scatterplot. An empty scatterplot will appear. Now we have to assign properties to the x-axis and the y-axis of the view. To do so, click Inspector tab. Expand the list ScatterPlot and then click at Scatterplot 1. At the bottom of the right panel, a table of property names and values of the view will be shown. These properties define the specific appearance of the newly created view. To set up the x- and y- axis, click on the Value column of the first two rows of the table, Data.x and Data.y to assign one of the four variables of nodes which are SepalLength, SepalWidth, Petal- Length, and PetalWidth. For now, assign PetalLength to Data.x and PetalWidth to Data.y for a scatterplot PetalLength-PetalWidth. You can adjust the size of the scattered points to 5 to have a better view. See Figure 3.1 for the scatter plot just created.

13 10 Assign Color mapping to a view You can assign the color of data points to a particular property by setting color in the property table of a view. However, it is useful to keep a global mapping so that the same color coding is used across multiple views, this can be achieved using virtual properties and mappings. To assign the color mapping of a view to the global color Mapping control, follow the steps below. In the table of the view property at the bottom right, link the node color and the node colormap to the global color mapping by: set color as Virtual$node.color ; set colormap as Virtual$node.colormap. Then we set assign a real colormap to the global color mapping in the right hand side Mapping tab: set Virtual$node.color as PetalLength pick a colormap in Virtual$node.colormap such as warm.colormap Now, we have created a global color mapping Virtual$object.color associated to the data variable PetalLength. Any other view that uses the same virtual color will be automatically linked to the global color mapping. For a visualization to be informative, we must specify a color legend that indicates what color represents a large value of PetalLength and what color represents a small value of PetalLength. To do this, create a Node Legend as follows: Create a Node Legend by View Create Node Legend; Click in Inspector tab Color Legend Node Legend 1 to show the property table of Node Legend;

14 11 Figure 3.2: The first scatterplot of PetalLength-PetalWidth for Iris dataset. set property as Virtual$node.color ; set colormap as Virtual$node.colormap ; After this step, this Node Legend View is linked to the global color mapping, Virtual$node.color and Virtual$node.colormap. So we can see now for the selected global colormap (warm.colormap), light color (yellow) represents long Petal Length while dark color (brown) represent short Petal Length. And now in netzen we have two views (scatterplot 1 and NodeLegend 1) that are linked to the same global color mapping. See Figure 3.2 for the Node Legend just created and the Mapping tab for the global mapping. Changing the mapping in the Mapping tab allows you to dynamically change the property that is mapped to color or the color map simultaneously for both views Parallel Coordinates Now we want to create a parallel coordinates view to see how these four variables are related to each other. Follow the steps: Create a Parallel Coordinate View by clicking View Create ParallelCoords;

15 12 Figure 3.3: The Parallel Coordinate View for Iris dataset. Click Inspector tab Parallel Coordinates ParallelCoords 1 Set the coordinates for the view: in the table, select SepalLength, SepalWidth, PetalLength, and PetalWidth for Properties ; Link the colormap of this view to the global map: set color as Virtual$node.color ; set colormap as Virtual$node.colormap ; Adjust the line width to better see the mapped color: set LineWidth as 3; Set transparent of all three views as true to make it easier to see the same color map across the views. Rearrange and resize all views if necessary. See Figure 3.3 for a snapshot of the views we have created so far.

16 Practice - Scatter Plot Matrix To practice, you can create a four by four scatterplot matrix in which each cell is a scatterplot of a pair of two variables out of the four variable. You can skip the cell that the x-y variables are the same in the matrix. The created scatterplot matrix should look like Figure 3.4. Note that you should link all views to the global color mapping, so that a selection in one view (the green box in the parallel coordinate view) reflects in all other views, as shown in Figure Save Workspace and create a Script To save the workspace we have created so far, do the following: Save workspace: click File Save Workspace. In the pop-up window, name your workspace file as tutorial-iris.ws and select the following path to save: /YourNetzenFolder/ examples/ UCI Create a Script: create an empty text file under the path /YourNetzenFolder/ examples/ tutorial-iris.script. Edit the script file as the following two lines: importtable examples/uci/iris.txt importedgetable examples/uci/tutorial-iris.ws To test loading script to read data and create views at once, run netzen by the command line under the netzen directory:./netzen examples/tutorial-iris.script You should be able to see the same visualization you created before saving.

17 Figure 3.4: The scatteplot matrix. 14

18 Figure 3.5: Selection in the parallel coordinate view reflects on all other scatterplot views. 15

19 16 Chapter 4 Tutorial - Network Data This tutorial guides you in the use of workspace to recreate previously saved views of a network data set, and reveal the structure of the network by the analysis tools and layout tools such as Centrality and Force Directed Layout provided in netzen. To have a quick glance of what you will have after following this step-by-step tutorial, you can run the following command under the netzen directory:./netzen examples/mit2.script This tutorial will lead you to first create a graph view for the network data, as seen in the left of Figure 4.1. Then you will make use of centrality saved in the dataset to filter out less important edges in the view. At last you will re-layout the graph by these remained important edges to reveal the structure of the network, which should be like in the right of Figure Load a Data Set The MIT reality data set is comprised of 64 individuals with blue tooth devices. It is collected by monitoring if two devices are in close proximity to each other. If so, such a proximity relationship is reported to the server and is saved as a link in the data set.

20 17 Figure 4.1: Before and after: this tutorial will guide you from a highly connected graph (left) to a graph that reveals the structure (right). The data setis consist of a node table that stores information about these individuals such as their position, and an edge table that saves the proximity relationship detected by the blue tooth devices of two individuals. It is under the directory /YourNetzenFolder/ examples/mit/. To load the network data set, you have to load these table files in order: First, click File Import Table and locate the table file mit.nodes for nodes that represent individuals. (Note that in the pop-up window, if you do not see any file under the directory, click Tab seperated files at the bottom right and change to All Files. ) Second, click File Import Edge Table and choose mit.edges. After loading the data, Info tab on the right shows the data sources, the number of edges and edges that have been loaded. There are 64 nodes and 5,934 edges loaded for this data set, as shown in Figure 4.1. Note that Import Edge Table uses the data command importedgetable as mentioned in Chapter 2.3 Script file. It treats each row in the edge table as a bi-directional

21 18 edges by saving two inverse edges (e.g., A B and B A). Therefore, the edge number shown in the Info tab is the number of total edges saved, which doubles the number of edges in the edge table file. 4.2 Create Views In this tutorial, we want to create a graph view to show the whole network and a legend to show the color map on the nodes Graph View In this section we show how to create and set up a Graph View for a network data set. Click View Create GraphView, and a window GraphView 1 shows up in netzen canvas. Initially all nodes are positioned on a ring. Therefore all edges will be drawn inside the ring, which results in a lot of edge crossing, as shown in the left of Figure 4.2. Click Inspector tab and locate the property table of the view by clicking GraphView GraphView 1. Set NodeSize to Now we would like to set up the node color according to the position of the individual. In the property table of GraphView 1 we assign the color of node in the view as the global virtual color mapping: Set Node color to Virtual$node.color ; Set Node colormap to Virtual$node.colormap ; Then we set the global virtual color mapping in the Mapping tab: Set Virtual$node.color as Position ;

22 19 Figure 4.2: Created Graph View and Discrete Legend. Left: initial graph view. Middle: color individuals according to job position. Right: color legend of job position. Set Virtual$node.colormap as YIGn ; Then the Graph view should look like in the middle of Figure Discrete Legend It is helpful to create a color legend to show the mapping between colors and various values of the Position property. If you check the node table file mitnode.txt, you would see in the first two header lines that the property Position is a categorised variable of STRING type. The values are discrete texts. Therefore, instead of Node Legend, we need to use a special legend Discrete Legend to visualize the color mapping of such a discontinuous property. Follow the steps: Click View Create Discrete Legend. An empty view titled Discrete Legend 1 will pop up in the canvas. Click Inspector tab and then Discrete Legend Discrete Legend 1 to locate the property table of this view. In the table: Set Colormap as Virtual$node.colormap ;

23 20 Set Property as Virtual$node.color ; After you assign Property of the Discrete Legend view, it lists all possible values of the node property Position with colored bars and labels of job positions (e.g., 1styeargrad, 2ndyeargrad,newgrad,mIgrad, and sloan). You may want to resize this view so that these bars do not clutter together. Your Discrete Legend 1 view now should look like in the right of Figure 4.2. Note that now the view Discrete Legend 1 and the view GraphView 1 are linked to the global virtual colormap, so the color mapping is the same in these two views. In this color mapping, light yellow means sloan and all various green refers to job titles in the MIT Media lab. 4.3 Create the Layout In netzen some popular graph layout tools have been implemented, such as Force Directed Layout and LingLog Layout. In this section we show how to create the layout and apply it to the GraphView Force Directed Layout Create a Force Directed Layout and apply it to the graph by the following steps: Click Layout Force Directed in the tool bar. You can locate the property table of this layout tool by clicking ForceDirected Force Directed 1 under Layouts tab. This table lists all parameters that you can interactively modify the forces used in the layout, such as : Elasticity for attraction between nodes if there is an edge between them, Spring Rest for the setting of the spring that models the attracting forces,

24 21 Repulsion for the repulsion between all pair of nodes in the network, Temperature for the temperature of the force model, and Timestep for how long an iteration last in time. Relate this layout tool to GraphView 1: so that we can see how nodes move in the graph when we turn the forces in the layout interactively. Follow the steps: Locate the property table of GraphView 1 under the Inspector tab Set Layout as Force Directed 1 to assign the layout to this graph view. Run on the force directed layout and interact with it: Locate the property table of Force Directed 1 under the Layouts tab. Set Enabled as true. The nodes in GraphView 1 will start to move. We can try different force parameters in this table as well. For example, we can increase repulsion forces a bit so that the nodes do not clutter together in a small area by setting Repulsion to Nodes will stop moving until the the total energy from the forces reach the equilibrium. Now the network should look like Figure 4.3, where nodes are highly connected causing a hairball effect. 4.4 Filter Edges We can see from Figure 4.3 that majority of the Sloan group (light yellow nodes) is positioned on the left and the most of the MIT lab members (various green nodes) are in the right. However there is no clear distinction between them and the rest of the graph since nodes are highly connected. Therefore the next step is to make use of some edge properties to filter the edges of the graph.

25 22 Figure 4.3: the network after we apply the Force Directed Layout. In the edge tables file mitedge.txt, there are some pre-calculated properties for edges that were generated with the centrality tools under the Analysis Tools tab and saved to the data set, such as EigenCentrality.Derivative and MarkovCentrality.Derivative. For details about these centrality for edges, please see our paper [2]. In short, these centrality properties for edges describe the importance of these edges in the network with respect to different centrality evaluation of individuals in the network. We can remove those edges that have smaller importance to see if edge filtering distinguishes the Sloan group from the MIT group in the network. In this case, we use markov centrality property of edges. Follow the steps to filer edges in the graph: Locate the property table of the graph view under Inspector tab. Set EdgeThresholdProperty as MarkovCentrality.Derivative Increase the threshold value EdgeThreshold from 0.0 to 2.2. As we increase the threshold, more and more edges are removed. After edge removal we can see more clearly the distinction bewteen the yellow nodes on the left and the green nodes on the right of the graph, as shown in Figure 4.4.

26 23 Figure 4.4: the network after we filter out less important edges according to centrality. 4.5 Re-layout the graph Beside removing edge by the centrality property of edges, we can also lay out graph according to this edge property. That is, in laying out the graph, an edge will be considered (to add an attraction to the force model) only if its selected property is higher than a certain threshold we set. In this case, we set an edge threshold on the MarkovCentrality.Derivative so that in the force-directed layout only those important edges will be used in the graph layout. Follow the steps: Locate the property table of Force Directed 1 under Layouts tab. Set the following properties: set EdgeThreshodProperty as MarkovCentrality.Derivative ; set EdgeThreshold the same as the threshold value we set for edge removal; Then after we re-compute the layout based on this edge property, the two groups are clearer, as shown in Figure 4.5. The result highlights sub-clusters in the right that were not obvious before. Sloan group, on the left, remains a tightly connected group, while the group on the right separates into a rather loose collection of smaller clusters.

27 24 Figure 4.5: the network after we re-layout the graph by centrality. 4.6 Save Workspace and Script As discussed in the tutorial of multivariate data set, we can save the views, mapping and layout with a workspace file. Saving graph layout results is especially useful when we are dealing with a large graph and the layout tool has been running for a while. For example, before we change layout parameters to other values to test, we might want to save the current graph layout so that the graph can be regenerated later Workspace To save the workspace we have created so far, do the following: Before we save the workspace, rename the file that saves the lay out results. Locate the property table of Force Directed 1 under the Layouts tab. In the File row in the table, enter the file directory together with the file name to locate where we want to save the lay out result. We might want to change it to examples/mit/mit mytutorial.layout to save the result under the directory in /YourNetzenFolder/ exam to well organize layout results of different data set. By default the file name uses

28 25 the name of the tool, Force Directed 1, and it is saved under the directory where netzen executable is. Then we can save the workspace as described in the previous tutorial. Click File Save Workspace. Choose the directory /YourNetzenFolder/ examples/ mit and name it mytutorial-mit.ws Script Now create a script file in /YourNetzenFolder/ examples/ mytutorial-mit.script that contains the following three lines: importtable examples/mit/mitnodes.txt importedgetable examples/mit/mitedges.txt loadworkspace examples/mit/mytutorial-mit.ws We can start another execution of netzen by this command line in the console window (after switching to /YourNetzenFolder/):./netzen examples/mytutorial-mit.script and we should get the same graphview, discrete legend, and force directed layout as created in this tutorial. 4.7 Practice - Color the Edges with Centrality In this practice, please set color the edges in the graph according to the same edge property that have been used for filtering the graph and for re-laying out the graph. First, assign the global color mapping ( Virtual$edge.colormap and Virtual$edge.color ) to the properties of GraphView ( Edge color and Edge colormap ). Increase EdgeThickness and EdgeTransparency to highlight the edges better.

29 26 Figure 4.6: the network where edges are colored by centrality. Then create an Edge Legend and assign the global color mapping ( Virtual$edge.colormap and Virtual$edge.color ) to its properties colormap and property. In the Mapping tab, set Virtual$edge.color as MarkovCentrality.Derivative and pick a color map for Virtual$edge.colormap (e.g. rb.colormap ) Interact with the color control bars in Edge Legend. (Hint: you may move, delete or create color control bars). See if you can distinguish edges from important ones to less important one by colors. What is the threshold value to distinguish them? An example is shown in Figure 4.6

30 27 Bibliography [1] Y. Chan et al., Flow-based scatterplots for sensitivity analysis, in Visual Analytics Science and Technology (VAST), 2010 IEEE Symposium on. [2] T. Crnovsanin et al., Social network discovery based on sensitivity analysis, in Proceedings of International Conference on on Advances in Social Networks Analysis and Mining (ASONAM 2009).