Integrating Text and Graphics to Present Provenance Information

Transcription

1 Integrating Text and Graphics to Present Provenance Information Thomas Bouttaz, Alan Eckhardt, Chris Mellish, and Peter Edwards Computing Science, University of Aberdeen, Aberdeen AB24 5UA, UK Abstract. As more and more systems are developed which capture and use provenance information, better mechanisms are required to present such information to end-users. In this paper we present two approaches for presenting provenance within ourspaces, a web-based virtual research environment. The first makes use of natural language generation technologies to produce texts describing provenance information. The second one adopts a more visual approach by building graphs. Our main contribution is a generic mechanism using a combination of visualisation modalities, allowing users not only to have a better understanding of the provenance of resources, but also to assist them with the creation of new provenance information. Keywords: provenance, NLG, visualisation, HCI 1 Introduction The provenance of a resource is defined by the W3C Provenance Incubator group as a record that describes entities and processes involved in producing and delivering or otherwise influencing that resource. Provenance thus provides additional documentation about the origin of a resource (or artefact). Goble [1] presents the 7 W s of Provenance : Who, What, Where, Why, When, Which, & (W)How. Each of these can be used to support assessment of the quality or trustworthiness of data. These assessments can be automated by computational agents, or performed by humans. Therefore presenting provenance information to end-users is an issue that needs to be addressed. The presentation of provenance should allow users to intuitively understand provenance information, without requiring any particular knowledge about the underlying model. In this paper, we discuss two ways to present provenance information, one based on text and the other a graphical visualisation. Our approach is motivated by the needs of multidisciplinary users within a web-based research environment, but is applicable in other contexts. Previous work has been carried out to compare the different performance characteristics associated with graphical and linguistic presentations [2 4]. While text can contain more explicit information, less expressive presentations such as

2 2 graphs can limit abstraction and therefore improve processability [2]. Moreover, people from different backgrounds may prefer different presentation modalities. To verify this hypothesis, we analysed 70 journal publications from 5 academic disciplines to find their presentation preferences. We identified that some disciplines tend to use more diagrams, while others prefer the use of text. Not only are different types of presentations better suited to different people, but also to the different types of tasks that they are performing [3, 4]. Therefore there is a need for different ways to present provenance data and to combine such mechanisms to satisfy a larger number of users. Several mechanisms have been developed to visualise provenance ([4 7]). When dealing with substantial amounts of data, it is important to carefully select what information should be presented, in order not to overwhelm users with unnecessary information. The authors in [5] rely on user views to provide the most relevant information to the user. By performing incremental querying, users are able to control how much information is displayed. Previous graphical visualisations ([4, 7]) present provenance entities as boxes, and connecting relationships as labelled edges. However, to the best of our knowledge, our mechanism is the first attempt to visualise provenance in a web environment, adopting a generic approach usable by other applications. Our implementation also makes use of several enhancements improving readability (e.g. emphasising the artefact in focus, visual distinction between artefacts, agents and processes), as well as allowing users to edit the provenance graph. Regarding the textual presentation of provenance, to the best of our knowledge, no similar mechanism has been previously described. Moreover our approach relies on both presentation modalities supporting each other. We have implemented our provenance visualisations within ourspaces 1, a web-based virtual research environment. By providing a centralised platform allowing its users to communicate and share their research artefacts (e.g. data, publications), ourspaces enables researchers from different backgrounds to better manage their projects. The underlying architecture of ourspaces is based on Semantic Web technologies (e.g. OWL 2, RDF 3 ) and associated reasoning mechanisms, allowing the use of domain knowledge in the form of ontologies, as well as the capture of the provenance of digital artefacts. In this context, the presentation of provenance to users is crucial to allow them to obtain a better understanding of what processes a scientific artefact went through, and to assess the quality of such an artefact. For instance a user might only rely on data that was collected by trusted agents, such as the members of his social network. 2 The ourspaces System At the heart of ourspaces is an OWL representation of the Open Provenance Model [8]. This ontology defines the primary entities of OPM as well as the causal

3 3 relationships that link them together. Figure 1 shows how the OPM model is integrated with other ontologies used in ourspaces. OPM is a generic model and as a result, our framework supports additional domain specific provenance ontologies that are created by extending the concepts defined in the OPM ontology. These domain specific ontologies describe scientific artefacts (e.g. data, papers, presentations), the processes involved in their creation (e.g. focus groups, questionnaires, experiments) and the relationships between these entities (e.g. involved). Using these ontologies it is possible, for example, to describe a physical research activity (e.g. an interview) as an opm:process, and how such an activity causes an opm:artifact to be generated (e.g. interview notes) and who was vre:involved in this process. In order to describe people and their social networks, we make use of the Friend-of-a-Friend 4 RDF vocabulary; a foaf:person is thus a subclass of opm:agent. To incorporate social networking activities, we have also integrated the SIOC 5 (Semantically-Interlinked Online Communities) ontology that provides a model to express user-generated content such as posting a message about an artefact. Domain Specific Social Networking and Project Management Interview Focus Group Paper Data sioc:post sioc:creatorof foaf:person vre:workswith Process hascause haseffect OPM Provenance CausalRelation hascause haseffect Artifact sioc:about vre:project vre:memberof vre:worksfor hasrole hascause haseffect vre:memberof vre:organisation Role Agent vre:group Fig. 1. Ontologies used within the ourspaces environment. ourspaces is composed of several services making use of these ontologies (see Figure 2). These support the creation, editing and querying of data, metadata and digital artefacts by accessing different repositories. The Provenance Service specifically manages provenance data by accessing the Provenance Repository. The Digital Artefact Repository is used to store the actual digital artefacts created by researchers (e.g. datasets, documents), while the Metadata and Provenance Repositories are used to store RDF triples. To allow the visualisation of this information, we have implemented two different services: The Graphical Visualiser that displays provenance information by building graphs. Since it only requires provenance information, it makes use of the Provenance Service to retrieve and update provenance data. A Natural Language Generation (NLG) service composed of the Text Formatter and the Text Generator, that generates textual descriptions of entities

4 4 ourspaces Web Interface Graphical Visualiser Web services Query Upload / Download Provenance Service Core Services Digital Artefact Management Metadata Access Data and Metadata Repositories Digital Artefacts Repository Metadata Repository Ontologies OWL Language Specifications Text Visualiser Text Formatter Text Generator Provenance Repository XML Fig. 2. Architecture of ourspaces. and their provenance. Since this service requires provenance information and other metadata, it uses the Metadata Access service to query the appropriate repositories (i.e. Metadata and Provenance Repositories). To illustrate the use of these components, consider an example where a user wants to upload a research artefact to share with other collaborators. Thanks to the Upload service, the user will be able to select his digital artefact, and will be asked for additional information, such as its type (corresponding to the domain specific ontology previously described), its title, when it was produced and so on. The artefact can also be associated with other entities by using the Query service to find related entities (e.g. persons that were involved in its creation, the project in which it was produced). As a result, this information will be uploaded and stored in the appropriate repositories. Moreover, the Provenance Service will generate basic provenance for this artefact by creating an upload process that was controlled by the appropriate agent, and stores this data in the Provenance Repository. As a result, the artefact will now be available on the ourspaces web interface through the artefact space, a page where all available metadata about an artefact is presented, as well as its provenance. Figure 3 shows an example of such a page for the artefact focus group data. The metadata is presented in table format at the top left, together with the same information described in natural language to the right. Related resources and a commentary about the artefact are also presented. Finally, at the bottom left is the graphical visualisation of the provenance graph focused on the uploaded artefact. On the right side is a textual description of the provenance graph. In the following sections, we will describe the implementation of both provenance visualisation mechanisms in more detail. 3 Natural Language Visualisation We have developed a NLG service based on the work of Hielkema [9] that generates short textual descriptions of OPM entities based on their associated RDF

5 5 Fig. 3. A screenshot of the ourspaces VRE. metadata. We have implemented two modes for this service: one for generating text about general information regarding an entity (e.g. type, title, date of creation), and one specifically about its provenance (i.e. how it relates to other OPM entities). 3.1 System Components Figure 4 presents an overview of the different components of the NLG service and how they interact with other parts of ourspaces. By using the Metadata Access service to query the repositories with the RDF ID of a particular entity, the Text Generator builds a rich model representing the information captured by the system. The axioms of that model are then translated into text by using the appropriate Language Specification files. These files, encoded in XML, are divided into two categories: Property Language Specification: contains the linguistic information required to structure the sentence corresponding to a property (e.g. syntactic category, verb tense). Class Language Specification: indicates which properties should be used to refer to a particular class (e.g. foaf:firstname and foaf:surname for a foaf:person).

6 6 Using these files, the Text Generator is able to construct sentences corresponding to RDF triples, and to aggregate the ones with similar syntactic structures. As a result, a short text describing an entity is generated. User Web Services Query Core Services Metadata Access Repositories Language Specifications Class Property ourspaces Web Interface Text Formatter Text Generator XML XML NLG Service components Fig. 4. Components of the text visualisation system. 3.2 User-Interface The NLG service is integrated across the ourspaces environment thanks to the Text Formatter that converts the plain text generated by the Text Generator into HTML. Figure 5 left shows an example of a text generated by the NLG service. This text contains general information about an artefact (focus group data) and can be further expanded by clicking on hyperlinks to related entities. This will call the service with the ID of that related entity, appending the resulting description to the original text. In the example shown in Figure 5 left, the user clicked on the hyperlink Thomas Bouttaz to obtain a description of that person. In order not to overload the description, similar properties are aggregated together (e.g. He is in contact with 17 other users ), which the user can disaggregate to obtain details about individual values. Fig. 5. Examples of generated texts containing general information (left) and provenance information (right).

7 7 3.3 Generating Provenance Texts The NLG service can be used to translate any RDF metadata associated with an entity into text, given the appropriate language specification files. Therefore this service can generate descriptions of provenance information (e.g. which agent was controlling an interview process). We have therefore implemented a specific mode for the NLG service that generates text only about the provenance of an artefact, providing users with a more transparent presentation of the underlying OPM data captured by the system. The OPM model represents causal relationships with edge classes (e.g. opm:used, opm:wasgeneratedby) that relate actual entities together with opm:cause and opm:effect properties. To be compatible with the model used by the NLG service that only translates direct properties, the OPM representation needs to be simplified. To this effect, we make use of rules that infer direct properties corresponding to the causal relationships linking the artefact being described to other entities. These rules are implemented using the SPIN API 6, by associating SPARQL 7 queries inferring direct properties from OPM edges. For example, the rule in Figure 6 shows how the opm:wasgeneratedby class is translated into an inferred property linking an artefact to the process that generated the artefact. CONSTRUCT {?this pggen:wasgeneratedbyinfer?process. } WHERE {?this a opm:artifact.?wasgeneratedbyedge a opm:wasgeneratedby.?wasgeneratedbyedge opm:effect?this.?wasgeneratedbyedge opm:cause?process. } Fig. 6. Rule inferring provenance properties from the OPM representation. Using the general mechanism previously described, the property inferred by this rule could be translated by the NLG service as: The interview transcript was generated by the transcribing process. We have integrated this functionality into the artefact space, by providing the user with a text describing how this particular artefact relates to other OPM entities (see Figure 5 right). By following anchors to these related entities, the user is able to further explore the provenance of this artefact, expanding the text generated. For example in Figure 5, the user has requested a description of the process Interview of Alan Eckhardt. Therefore this mechanism provides users with fine grain control over the amount of provenance information to be displayed

8 8 4 Graphical Visualiser The Graphical Visualiser 8 was developed to visualise provenance information using a graph displaying OPM entities as nodes and OPM causal relationships as connections between nodes. 4.1 System Components The Graphical Visualiser is implemented in HTML and Javascript. It is divided into two main components for communication and visualisation. The communication component makes use of the Provenance Service to query for provenance data, as well as to update the provenance graph. All communication with the service is done in the background using AJAX (Asynchronous JavaScript and XML) techniques. The visualisation component draws the provenance graph and allows user actions on the graph (e.g. dragging, zooming). It uses the Javascript library jsplumb 9 for creating connections between nodes. It is also used to manage the editing process, by checking the domain and range of new properties. Both components use jquery 10 for easier and more comprehensible manipulations of the DOM. 4.2 User-Interface The Graphical Visualiser uses rectangular nodes to display OPM entities - Agents, Artifacts and Processes, each type using a different colour, and with its own icon in the corner to ease recognition. Edges are represented as directed connections between nodes. The user can control the position of the nodes, as well as the level of magnification. Each node has three icons in its corners. At the top left is an icon specific to the node s type. At the lower right is an i icon, which redirects the browser to the page with detailed information about that node, i.e. the artefact space for an artefact, or a person s profile page for an agent. At the top right is a plus icon, that loads the provenance of this node when clicked, thus expanding the provenance graph. We decided to remove the plus icon from the Agents, because loading all the provenance information about an agent results in a substantial amount of data being displayed (e.g. all artefacts uploaded by the user, all editing processes), which overloads the graph. Figure 3 bottom left shows the provenance graph of the artefact focus group data, which is emphasized with bold letters, black border and darker colour, which enables easy identification of the focal point of the graph. The artefact was produced by a process Resource upload controlled by Thomas Bouttaz. Then the artefact was used by the process Interview of Alan Eckhardt. This interview process involved several people and produced an artefact (i.e. the recording of this interview). 8 The Graphical Visualiser is open source and available at:

9 9 The graph is initially arranged in chronological order, i.e. displaying the oldest process on the left and then aligning other processes left to right chronologically. Users are still free to rearrange the nodes according to their own preferences. Only the agents are not in the chronological order, as they are not associated with any temporal information relevant to the current provenance graph. They are displayed at the bottom middle part of the graph. The Graphical Visualiser conforms to several user requirements and Types presented in [4]. Some of these types include Timeline (chronological ordering) and Result (focus on the main artefact), which are captured by our approach. 4.3 Editing the Provenance Graph Providing users with an easy way to create provenance information in the system is a critical issue. For example in the graph of Figure 3, we can see that the artefact was used by the interview process. Since this fact cannot be inferred, the user has to provide the information explicitly. To this effect, we extended the Graphical Visualiser with functionality to permit the provenance graph to be edited. Fig. 7. Screenshot of interface for editing the provenance graph. Figure 7 shows the user interface for editing provenance. There are two modes within the interface - create or add existing nodes, and create new edges between nodes. The user can query for existing artefacts, agents or processes in the Metadata Repository using keyword search or he/she can create new processes (see Figure 7 left). In Figure 7, the user searched for the process Interview of Alan Eckhardt (step 1) and added it to the graph (step 2). To create a new edge, the user connects two nodes by dragging a handle, a grey circle in the lower part of the node (step 3), from one node to the other. To guide the user, potential

10 10 connection targets are highlighted and the name of the resulting edge is displayed in the top right corner of the node. Users can also delete edges between nodes. Each edge has a red x on the right side of its label. Clicking it triggers a confirmation dialog, and if the user accepts the edge between the two nodes is deleted. 5 Combining Visualisations We have combined the visualisation modalities in two different ways within the interface: by presenting them side-by-side in the artefact space, and by extending the functionality of the Graphical Visualiser. In the artefact space, both visualisations are presented, allowing users to explore them independently as some aspects of one compensate for the weaknesses of the other. For example, with the textual description it is not always clear which referring expression should be used to designate a particular entity (e.g. process), while the graphical presentation bypasses this issue. On the other hand, the NLG service provides an aggregated summary of the provenance (e.g. He controlled seven processes ), which can be disaggregated by the user. This results in a finer grained control over the amount of provenance information to be displayed. We have also integrated the NLG service within the graphical presentation, by allowing users to generate textual descriptions of the different entities present on the graph. Figure 8 shows how a user can generate the description of an entity (focus group data) by hovering on top of its icon. In this way, the user gets more information about entities without interrupting his current workflow. Fig. 8. Integration of textual and graphical visualisations. 6 Evaluation In order to evaluate this work, we ran a focus group with potential users of ourspaces. We ran three sessions, each with seven graduates from a range of

11 11 disciplines who were taking a course on human-computer interaction. We presented them with both presentations of the same provenance information, and asked them to discuss the main advantages and drawbacks of each. The participants felt that some form of temporal information would help them to better understand the provenance presentations. Therefore we adapted the graphical visualisation by reorganising the graph in chronological order (as described in Section 4.2). The participants preferred the use of active verbs to present relationships between entities, and would like the terminology to adapt to the context of use (e.g. referring to the agent that controlled an interview process as the interviewer, rather than with wascontrolledby). This suggests that more adaptation of the underlying representation of provenance information may be required to improve usability. For instance, the approach described by [6] adapts the OPM view by presenting artefacts as inputs or outputs of processes, depending on their causal relationships (e.g. used artefacts are inputs of processes). To better adapt the terminology, we could extend the OPM ontology by defining specific sub-properties with associated terminology for different processes, and use these properties depending on the context. For example we could define an interviewer property extending wascontrolledby, and use that property in the context of an interview process. This could be implemented by using our previous work [10], where we have described a policy mechanism allowing adaptation of the interface depending on the context. Overall the participants found that the combination of both graphical and textual presentations was useful to better understand the provenance of an artefact. 7 Discussion In this paper, we have introduced two different approaches for presenting provenance information to users. The design and implementation of both approaches was described in depth, as well as their integration within the ourspaces virtual research environment. This work differs from other provenance visualisation services by providing users with two presentation modalities supporting each other, as well as a mechanism to allow users to edit the provenance metadata. In order to further evaluate this work, we have implemented a logging mechanism recording how ourspaces users utilise these services (e.g. what type of presentation mode they prefer, how they create more provenance data with the Graphical Visualiser). After having gathered a substantial amount of data, we will be able to better assess the benefits of these services, and to identify further improvements. For the Graphical Visualiser, we have already concluded that we need to convert the labelling of the edges from passive to active form, e.g. was- GeneratedBy to generated. We could also implement other Types described by [4], such as Comparison (comparing the provenance of two artefacts, highlighting their differences) and Participation (emphasising the agents involved in the processes) which would be beneficial in the context of a collaborative environment such as ourspaces. Another improvement would be the ability to aggregate

12 12 parts of the provenance graph, presenting the user with a more abstract view of the provenance (see [7]). The users could then disaggregate these, to obtain more detailed information. Acknowledgments This work is supported by the UK Economic & Social Research Council (ESRC) under the Digital Social Research programme; award RES References 1. Goble, C.: Position statement: Musings on provenance, workflow and (semantic web) annotation for bioinformatics. In: Workshop on Data Derivation and Provenance. (2002) 2. Stenning, K., Oberlander, J.: A Cognitive Theory of Graphical and Linguistic Reasoning: Logic and Implementation. Cognitive Science 19(1) (1995) Mohar, T., Mak, D., Blumenthal, B., Leventhal, L.: Comparing the comprehensibility of textual and graphical programs: the case of petri nets. In Norwood, ed.: Empirical Studies of Programmers: Fifth Workshop. (1993) Kunde, M., Bergmeyer, H., Schreiber, A.: Requirements for a provenance visualization component, Springer-Verlag (2008) Biton, O., Cohen-Boulakia, S., Davidson, S.B., Hara, C.S.: Querying and managing provenance through user views in scientific workflows. In: Proceedings of the 2008 IEEE 24th International Conference on Data Engineering, Washington, DC, USA, IEEE Computer Society (2008) Anand, M., Bowers, S., Altintas, I., Ludäscher, B.: Approaches for exploring and querying scientific workflow provenance graphs. In McGuinness, D., Michaelis, J., Moreau, L., eds.: Provenance and Annotation of Data and Processes. Volume 6378 of Lecture Notes in Computer Science., Springer Berlin / Heidelberg, Springer Berlin / Heidelberg (2010) Cheung, K., Hunter, J.: Provenance explorer: customized provenance views using semantic inferencing. In: Proceedings of the 5th international conference on the Semantic Web. ISWC 06, Berlin, Heidelberg, Springer-Verlag (2006) Moreau, L., Clifford, B., Freire, J., Futrelle, J., Gil, Y., Groth, P., Kwasnikowska, N., Miles, S., Missier, P., Myers, J., Plale, B., Simmhan, Y., Stephan, E., den Bussche, J.V.: The open provenance model core specification (v1.1). Future Gener. Comput. Syst. 27 (June 2011) Hielkema, F.: Using Natural Language Generation to Provide Access to Semantic Metadata. PhD thesis, University of Aberdeen (2010) 10. Bouttaz, T., Pignotti, E., Mellish, C., Edwards, P.: A policy-based approach to context dependent natural language generation. In: Proceedings of the 13th European Workshop on Natural Language Generation, Nancy, France, Association for Computational Linguistics (September 2011)