Exploring Relationships between Annotated Images with the ChainGraph Visualization

Transcription

1 Exploring Relationships between Annotated Images with the ChainGraph Visualization Steffen Lohmann 1, Philipp Heim 2, Lena Tetzlaff 1, Thomas Ertl 2, and Jürgen Ziegler 1 1 University of Duisburg-Essen, Interactive Systems and Interaction Design, Lotharstr. 65, Duisburg, Germany 2 University of Stuttgart, Visualization and Interactive Systems, Universitätsstr. 38, Stuttgart, Germany {steffen.lohmann,lena.tetzlaff,juergen.ziegler}@uni-due.de {philipp.heim,thomas.ertl}@vis.uni-stuttgart.de Abstract. Understanding relationships and commonalities between digital contents based on metadata is a difficult user task that requires sophisticated presentation forms. In this paper, we describe an advanced graph visualization that supports users with these activities. It reduces several problems of common graph visualizations and provides a specific chain arrangement of nodes that facilitates visual tracking of relationships. We present a concrete implementation for the exploration of relationships between images based on shared tags. An evaluation with a comparative user studies shows good performance results on several dimensions. We therefore conclude that the ChainGraph approach can be considered as a serious alternative to common presentation forms. After a discussion of the limitations, we finally point to some application scenarios and future enhancements. Key words: graph visualization, interactive exploration, relationship discovery, exploratory search, user-annotated images, visual tracking, shared metadata, tagging, photo sharing, annotation. 1 Introduction Metadata is important in the organization, management, and retrieval of all kinds of digital contents. It also links the contents allowing for structured exploration and the discovery of relationships and commonalities. However, finding and following these links is often difficult for users, mostly due to the constraints of the presentation forms that are used to display the contents. Visualizations are needed that explicitly show relationships between digital contents based on shared metadata. This need becomes particularly apparent against the background of recent developments in the Web. Many providers of media sharing services use tagging as a specific form of user-generated annotation on their websites. Tagging-based systems enable users to annotate digital contents with multiple, arbitrary terms

2 2 Steffen Lohmann, Philipp Heim, Lena Tetzlaff, Thomas Ertl, Jürgen Ziegler in order to organize these contents for themselves and/or others. That way, large collections of tagging data emerge that are commonly known as folksonomies. These folksonomies are an unstructured form of metadata that helps users to browse media collections and find specific contents. For instance, an analysis of the tagging data from the photo sharing website Flickr 3 showed that each photo is annotated by almost three tags on average that consist usually of one, sometimes two, words and mostly refer to the contents of the images [4]. Since the same tags are usually assigned to multiple images, implicit relationships based on these shared tags result. 1.1 State-of-the-Art One popular visualization that supports browsing in folksonomies are tag clouds. Typically, a tag cloud presents a certain number of most often used tags where the tags popularity is expressed by their font sizes [7]. Although this visualization type allows easy access to digital contents, tag clouds are usually visualized separately from the contents in a specific area of the user interface (see Fig. 1a for an example from the multimedia sharing website ipernity 4 ). Links between contents that are based on shared tags are not explicitly shown making it hard to identify and follow them. Some more advanced tag cloud visualizations group similar tags [3] or even visualize implicit relationships between tags [9] based on certain criteria such as tag co-occurance. However, these visualizations merely improve the presentation of the tags themselves by showing their interrelations but provide no clue how the tags link the digital contents and what relationships between the contents exist. (a) (b) Fig. 1: a) tag clouds are simple browsing interfaces that show no relationships, b) common graph visualization tend to produce crossing edges and overlapping nodes

3 Exploring Relationships between Annotated Images with the ChainGraph 3 Especially for the visualization of image collections, many other approaches have been proposed. Some even use metadata to arrange the images in a meaningful way. For instance, Rodden et al. [11] propose a presentation form that arranges image thumbnails according to their mutual similarity based on lowlevel visual features and textual captions. Dontcheva et al. [1] present an interactive visualization that clusters images from Flickr according to their tags and provides several interaction mechanisms for browsing the clusters. Another popular example is the statistical clustering of photos in Flickr that is based on an analysis of tag co-occurrences. Only few approaches try to present images and their tags in a combined way. For instance, yahoo taglines 5 visualizes random images from Flickr in an animated tag river that simulates a timeline. However, such approaches allow no systematic exploration of the images but rather support free browsing and serendipitous discoveries. More generally, there is a lack of visualizations that display digital contents along with their metadata and explicitly show interrelations. In particular, relationships along multiple dimensions are hard to be followed in existing presentation forms. Graph visualizations, on the other hand, seem to be highly appropriate to address these user needs. One example for a graph visualization of user-annotated images is TagGraph 6. It displays relationships between images from Flickr by representing these images and their tags as nodes that are connected by edges (see Fig. 1b). However, common graph visualizations, such as TagGraph, have several drawbacks when displaying digital contents that are interrelated by shared metadata: Crossing edges and high densities hamper the visual tracking of relationships or even result in misinterpretations [10]. Overlapping nodes can result in an imperfect presentation of the digital contents and their metadata. Positioning of the nodes is often not optimal for the visual tracking of relationships. These drawbacks are clearly visible in the example given in Figure 1b where we used the TagGraph tool to visualize images that are highly interconnected by several shared tags: In some cases, the relationships are not clear due to high densities and crossing edges (e.g., for the tags nikon, philipp, or dcdead ); in others, nodes overlap image parts (e.g., the tags green or great ). Yet in other cases, the positioning of the tags is not optimal (e.g., the tags reflection or spiegelung are not placed in between the two images they connect). We developed an advanced graph visualization the ChainGraph that reduces these problems and provides better support for the exploration of relationships between digital contents based on shared metadata. Although the general ChainGraph approach is not limited to a specific application area, in the

4 4 Steffen Lohmann, Philipp Heim, Lena Tetzlaff, Thomas Ertl, Jürgen Ziegler following, we particularly focus on a concrete implementation of the ChainGraph that visualizes relationships between images based on shared tags. We introduce the general idea, present the implementation and describe a scenario in Section 2. We then report on a user evaluation where we compared the ChainGraph with a common graph visualization in Section 3. Finally, we discuss limitations of this approach and give an outlook on future work and possible application areas of the ChainGraph in Section 4. 2 The ChainGraph Approach The basic idea of the ChainGraph approach is best described by comparing it with a common way of visualizing linked contents in a graph (see Fig. 2). In common visualizations, each metadata instance is represented by exactly one node. If a metadata instance is shared by many content items, a force-directed layout [2] arranges the content nodes radially around this metadata instance (cp. tags winter, trees, and snow in Fig. 2a). In the ChainGraph, by contrast, each metadata instance connects two content items at most. This is realized by multiplying metadata nodes in the visualization: Every metadata instance that is shared by more than two content items is represented by several nodes arranged in a chain that connects the content nodes in a certain consecutive order (cp. the chains winter, trees, and snow in Fig. 2b). Changing the order of the nodes in, for example, the winter -chain in Fig. 2b would highly affect the quality of the resulting graph layout. We therefore apply a special algorithm to guarantee an optimal arrangement of the nodes in the chains. (a) (b) Fig. 2: Visualization of images and shared tags with a) a common graph and b) the ChainGraph visualization. The algorithm calculates a heuristic value, the constraintlevel, for all resources that have not yet been added to the graph and chooses the resource with

5 Exploring Relationships between Annotated Images with the ChainGraph 5 the highest constraintlevel to be added next. Whenever a resource is added to the graph, the constraintlevels are calculated anew until no resources are left. Given a set of chains C, the constraintlevel of a certain resource r is computed by the following formula: constraintlevel(c, r) = numsharedprops(c, r) (minnumconnectedres(c, r) + numalternatives(c, r)) The result is that if a resource shares many properties with the already drawn ones - numsharedp rops - but would still get connected to only a few of them by possibly multiple edges - minnumconnectedres - and has only few alternatives that are just as good - numalternatives - it is added as early as possible. A detailed explanation of the used algorithm is given in [5]. [ToDo: (Reviewer 2) Add some implementation detail (pros and cons of the algorithm e.g. when does it break?). When are two nodes placed next to each other in the graph (adjacent)?] [ToDo: Uebergang anpassen!] Although this multiplication increases the total number of nodes and edges in the graph, it significantly reduces the graph s density and energy level, i.e., attractive and repulsive forces are assigned rather parallel than opposite, resulting in a fewer number of crossing or overlapping edges compared to common graph visualizations. As a consequence, the connections within a chain must be interpreted in a transitive manner, i.e., all content items along one chain are related to each other via the same metadata instance, independently of their order. The parallel arrangement also leads to a placement of the metadata nodes in a position between the content nodes what further facilitates the identification of relationships and commonalities (cp. tags summer and sky in Fig. 2b). In order to support the correct interpretation and visual tracking of the chains, all nodes and edges that represent the same metadata instance are visualized in the same color. We developed an application prototype that demonstrates how the Chain- Graph approach can be used to visualize relationships between images based on shared tags. It is implemented in Adobe Flex 7. All nodes of the graph can be moved via drag&drop and images of interest can be enlarged simply by a click. How it supports the exploration of tag-based relationships and commonalities between images is best described by a small scenario. 2.1 Scenario Fig. 3 shows a screenshot of the application prototype as it can be used to browse a collection of annotated images of Paris 8. Assume a user is searching for a representative image of Paris that she would like to use as an illustration for a text about the French capital. She first glances at the image that shows the Eiffel The prototype is online accessible at ChainGraph/AnnotatedImagesDemo.swf

6 6 Steffen Lohmann, Philipp Heim, Lena Tetzlaff, Thomas Ertl, Jürgen Ziegler Tower in summer (Fig. 3, 1.). After enlarging this image, she is not satisfied with it as it looks a bit boring in her opinion. Therefore, she goes through further images by following the chain labeled with the tag eiffel tower. Another image of the Eiffel Tower catches her attention and she enlarges it (Fig. 3, 2.). She likes the monochromatic style of the image and decides to look for further images of this kind. Consequently, she follows the tag chain labeled with black & white. She recognizes that another chain, labeled with people, meets the black & white chain and runs in parallel with it (Fig. 3, 3.) she is on the right path, since people on an image help to make it lively and interesting what is in line with the goals of her search. Finally, she reaches a black-&-white image showing the subway of Paris with passengers inside (Fig. 3, 4.). Since she also reaches the eiffel tower chain again, this symbol of Paris is also on the image, visible in the background through the window of the subway. After enlarging the image, she recognizes that it perfectly meets her needs and copies it to her Weblog as illustration of her text about Paris Fig. 3: Using an ChainGraph implementation to search for a picture of Paris. As shown by the scenario, browsing with the ChainGraph is usually a combination of goal-oriented and exploratory search [8]. Relatively vague user needs can be iteratively refined by discovering and following tag chains of interest. If several images share more than one tag, the chains run in parallel making it

7 Exploring Relationships between Annotated Images with the ChainGraph 7 easy for the user to browse through related images and to select the one that fits best with her needs. Note that we visualized only shared tags in this example as these help to discover relationships between the images. Of course, tags that are assigned only to single images can also be shown in the ChainGraph implementation simply by adding a labeled node and connect it with the image. 3 Evaluation We performed a user study where we compared the ChainGraph approach with a common graph presentation. We were mainly interested in the understandability, user acceptance, and performance of the ChainGraph. The presentation of the graphs in the study was similar to the one shown in Fig. 2. However, we used a more abstract visualization in order to avoid biases resulting from personal preferences or distractions caused by certain images or tags. Therefore, the nodes and edges of the graph visualization consisted of numbered labels instead of actual images and tags in the user study. [ToDo: (Reviewer 2) How did the fact that two nodes where adjacent or not, affect the evaluation resutls? Why not just admit that ChainGraph proved not to be helpful in tasks 2 and 3, but was superb in task 1? ] 3.1 Study Design Overall, we generated three pairs of graph visualizations for the user study, whereas each pair consisted of one ChainGraph and one common graph that both showed exactly the same data. We kept the total number of content items and metadata instances constant (six each) but gradually increased the number of metadata connections between the content items for each evaluation pair. In this case, the minimum possible number of connections based on shared metadata instances is twelve (if each of the six metadata instances is shared by exactly two content items, cp. Table 1a) and the maximum number is 36 (if each of the six metadata instances is shared by all six content items, cp. Table 1c). The application of these extreme values in the user study makes no sense, since both graph types (ChainGraph and common graph) look the same for the minimum value; and no insights can be gained in case of the maximum value since all metadata is shared by all content items. Therefore, we choose three values in between (18, 24, 30) in order to test graphs with varying densities. We then generated random distributions for these values that we used to draw the nodes and edges for both graph types. Table 1b shows the random sample for the graphs with 24 shared metadata connections. We arranged the nodes of all graphs in a force-directed layout [2] and applied an optimization algorithm [5] for the ordering of the chains. For each graph type, the repulsion a factor that controls how strongly the nodes are pushed away from each other was defined in a way that the graph looks aesthetically pleasant and the edges were not becoming too long or short, leading to a generally lower

8 8 Steffen Lohmann, Philipp Heim, Lena Tetzlaff, Thomas Ertl, Jürgen Ziegler M 1 M 2 M 3 M 4 M 5 M 6 C 1 x x C 2 x x C 3 x x C 4 x x C 5 x x C 6 x x (a) M 1 M 2 M 3 M 4 M 5 M 6 C 1 x x x C 2 x x x x C 3 x x x x C 4 x x x x C 5 x x x C 6 x x x x x x (b) M 1 M 2 M 3 M 4 M 5 M 6 C 1 x x x x x x C 2 x x x x x x C 3 x x x x x x C 4 x x x x x x C 5 x x x x x x C 6 x x x x x x (c) Table 1: Distributions with a) 12 assignments (minimum), b) 24 assignments (random matrix) and c) 36 assignments (maximum) (C = content item, M = metadata instance) repulsion for the ChainGraph. Fig. 4 shows two visualizations from the user study with 24 shared metadata connections for both graph types, the common graph and the ChainGraph. (a) (b) Fig. 4: a) Common graph and b) ChainGraph visualization with 24 metadata connections (cp. Table 1b) Since we were particularly interested in how well the graph types support the visual tracking of metadata relationships and the identification of commonalities between digital contents based on shared metadata, we defined the following three user tasks for the comparative study: 1. Find the pair of resources that shares most properties. 2. Find all properties that are shared by a given pair of resources. 3. Find all properties that are shared by a given triple of resources.

9 Exploring Relationships between Annotated Images with the ChainGraph 9 In sum, we thus applied a 2x3x3 within-subject design with variables graph type (common graph vs. ChainGraph), task type (task 1, 2, 3), and shared metadata connections (18, 24, 30). 3.2 Procedure Twelve participants, mainly students, took part in the study, with an average age of 29 (ranging from 22 to 47). The general familiarity with graphs was given with an average of 7.7 (median of 8.5) on a scale of 1 to 10. All subjects reported normal or corrected to normal vision and no color blindness. We presented all three pairs of graphs along with the three tasks on a 17 TFT monitor with a screen resolution of 1280 x 1024 px to all participants. Each graph type and each task were introduced and explained by an example. To control learning effects, we interchanged the presentation order of the two graph types and randomly assigned the participants to one of the settings (group A started with the common graph, group B with the ChainGraph visualization). After completing all three tasks for all distributions of one graph type, the subjects were asked to fill out an evaluation sheet. The corresponding graph type had to be rated according to 23 pre-defined items on a scale of one to five. The items were then mapped to the four dimensions effectiveness, understandability, control, and attractiveness (5-7 items per dimension). In a final questionnaire, the subjects had to directly compare both graph types and indicate their general familiarity with graphs. Furthermore, we measured the time needed to fulfill the tasks and the accuracy of the answers by counting wrong answers. 3.3 Results Overall, the ChainGraph visualization performed very well in the user study. Nine of the twelve participants preferred using the ChainGraph to solve the tasks of the study. It also reached slightly better results in the evaluation sheets: Fig. 5a shows the user ratings on the four dimensions attractiveness, control, understandability and effectiveness that were generated from the items of the evaluation sheet (higher value = better rating). The participants quickly understood the ChainGraph layout and did not report on serious difficulties when using it to accomplish the tasks. The colored edges proved to be helpful in following the chains (cp. Fig. 2b). Though the compactness of the common graph was considered positive, the study participants complained about the high number of crossings edges in this visualization type (cp. Fig. 4b). Regarding the time that was needed to accomplish the tasks, the ChainGraph performed significantly better in the first task (see Fig. 5b). This indicates that the ChainGraph layout assists particularly in the identification of commonalities between contents based on shared metadata. With respect to the tasks of the user study, the ChainGraph generally showed a better performance than the common graph, independently of the graphs

10 10 Steffen Lohmann, Philipp Heim, Lena Tetzlaff, Thomas Ertl, Jürgen Ziegler Common graph ChainGraph effectiveness attractiveness understandability control Time needed (in sec) Common graph ChainGraph Increasing number of shared properties Task 1 Task 2 Task 3 (a) (b) Fig. 5: Results from the comparative study: a) user ratings for both graph types on the four evaluation dimensions, b) time needed to accomplish the tasks density (i.e., the number of metadata connections). Table 2 shows the mean values of the times needed to accomplish the tasks and the task errors for the three graph densities that were used in the study (18, 24, 30 see Section 3.1). Interestingly, the ChainGraph reached lower time and task error values for all three densities. These results indicate that the ChainGraph is a valuable alternative in any cases where relationships and commonalities between content items based on shared metadata instances are of interest. Of course, the results are limited to the kind of user tasks that were tested in the study. Further validation in real use cases with other user groups, scenarios, and tasks is needed. Density Repulsion Time Needed (sec) Task Errors Common Chain Common Chain Common Chain Table 2: Repulsion of the graphs, average time needed to accomplish the tasks, and avarage task errors for the three graph densities (= numbers of shared metadata connections) 4 Discussion With the ChainGraph, we introduced a new visualization approach for the exploration of relationships between digital contents based on shared metadata. Since the ChainGraph represents shared metadata instances by multiple nodes

11 Exploring Relationships between Annotated Images with the ChainGraph 11 it avoids an agglomeration of content nodes around metadata nodes and thus reduces the graph s general density. This also decreases the probability for crossing edges and overlapping nodes and tends to a placement of metadata nodes rather between the content nodes they connect than elsewhere. These modifications facilitate the exploration of relationships and commonalities between content items and ultimately result in a improved readability and usability for related user activities. To the best of our knowledge, the proposed ChainGraph is the first and only approach that multiplies nodes and arranges them in chains to better support the visual tracking of relationships and the identification of commonalities. Furthermore, we demonstrated the applicability of this approach with an implementation for the interactive exploration of tag-based relationships within a selected set of annotated images. 4.1 Limitations As illustrated in the evaluation, the ChainGraph provides no benefits for some extreme cases. For instance, it would be identical to a common graph visualization in distributions where all metadata instances are assigned to exactly two content items (cp. Section 3.1 and Table 1a). However, such extreme distributions are very unlikely in real application scenarios. Furthermore, it is important to notice that we developed the ChainGraph for the visualization of a limited set of annotated contents but not as a visualization for whole content collections. Usually, exploration with the ChainGraph is only one of the many activities of a corresponding search process. For instance, in the scenario given in Section 2.1 the presented ChainGraph could be a result of a user query for the tag paris. A general limitation of the ChainGraph is its relatively large size due to the multiplication of metadata nodes. Consequently, it needs more screen space than other presentation forms and is well applicable only on large displays with a high screen resolution. Although this is not a serious problem in times of Full HD, it might still restrict the application areas of the ChainGraph. 4.2 Application Scenarios and Outlook Several use cases are imaginable for the ChainGraph approach. Since the visualization is best viewed on large displays, we tested our implementation both on a projection screen and a multi-touch table with the dataset given in Section 2.1 (see Fig. 6). As expected, the interaction with the ChainGraph visualization displayed on the projection was experienced as more immersive than its presentation on a usual monitor. As a side effect, the default size of the images was already sufficient to get a fair impression of the image contents and hence fewer enlargements of images were necessary. Regarding the multi-touch interaction, we discovered several issues that might be beneficial for an efficient use of the ChainGraph visualization. For instance, users could drag the background with one hand while enlarging or rearranging images with the other; or they can use both hands to easily place images in a fixed position next to each

12 12 Steffen Lohmann, Philipp Heim, Lena Tetzlaff, Thomas Ertl, Ju rgen Ziegler (a) (b) Fig. 6: Exploring user-annotated images with the ChainGraph visualization a) on a large projection and b) on a multi-touch table other for a better comparison. Since multi-touch input is not supported by our current ChainGraph implementation, we work on this feature in order to get a better understanding of the opportunities multi-touch interfaces provide for the graph-based exploration of image collections (also cp. [6]). In this paper, we focused on the description and evaluation of the basic ChainGraph approach and the exploration of user-annotated images. Of course, many enhancements are imaginable: On the one hand, supporting further content formats (e.g., video or audio) might be an interesting extension; however, this also requires additional considerations about an adequate representation within the ChainGraph. On the other hand, the available metadata might go beyond simple tags. For instance, automatically extracted metadata might also be considered, such as low-level descriptors or context and media file information (e.g., the black & white tag of the scenario in Section 2.1 could have been automatically derived from the file information). Especially structured metadata raises many opportunities for extensions that allow aggregating and filtering of relationships or faceted exploration. However, structured metadata is not available in many situations. As we have shown in this paper, the ChainGraph offers a valuable alternative to classical presentation forms, already for the exploration of relationships in unstructured metadata, such as user-assigned tags. The only requirement is that the content items share certain metadata instances.

13 Exploring Relationships between Annotated Images with the ChainGraph 13 References 1. Dontcheva, M., Agrawala, M., Cohen, M.F.: Metadata Visualization for Image Browsing. UIST 2005 Demonstration (2005). 2. Fruchterman, T., Reingold, E.: Graph Drawing by Force-Directed Placement. Softw. Pract. Exper. 21(11), (1991). 3. Hassan-Montero, Y., Herrero-Solana, V.: Improving Tag-Clouds as Visual Information Retrieval Interfaces. In: Proceedings of the International Conference on Multidisciplinary Information Sciences and Technologies. INSTAC, Mrida (2006). 4. Heckner, M., Neubauer T., Wolff C.: Tree, funny, to read, google. What are Tags Supposed to Achieve?. In: Proceedings of the 2008 ACM Workshop on Search in Social Media, pp ACM Press, New York (2008). 5. Heim, P., Lohmann, S.: A New Approach to Visualize Shared Properties in Resource Collections. In: Proceedings of 9th International Conference on Knowledge Management and Knowledge Technologies. J.UCS, Graz (in press). 6. Kristensson et. al: InfoTouch: An Explorative Multi-Touch Visualization Interface for Tagged Photo Collections. In: Proceedings of the 5th Nordic Conference on Human-Computer Interaction (NordiCHI 08), pp ACM Press, New York (2008). 7. Lohmann, S., Ziegler, J., Tetzlaff, L.: Comparison of Tag Cloud Layouts: Taskrelated Performance and Visual Exploration. In: Proceedings of INTERACT Springer, Berlin/Heidelberg (in press). 8. Marchionini, G.: Exploratory Search: From Finding to Understanding. Commun. ACM 49(4), (2006). 9. Michlmayr, E.; Cayzer, S.: Learning User Profiles from Tagging Data and Leveraging them for Personal(ized) Information Access. In: Proceedings of the Workshop on Tagging and Metadata for Social Information Organization (2007). 10. Purchase, H.C.: Which Aesthetic has the Greatest Effect on Human Understanding?. In: Proceedings of the 5th International Symposium on Graph Drawing (GD 97), pp Springer, Berlin/Heidelberg (1997). 11. Rodden, K., Wojciech, B., Sinclair, D., Wood, K.: Does Organisation by Similarity Assist Image Browsing?. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI 01), pp ACM Press, New York (2001).