REVEAL FP7-610928 REVEALing hidden concepts in Social Media Deliverable D5.4.2 Visualization and analytical tools Editor(s): Responsible Partner: Stuart E. Middleton University of Southampton IT Innovation Centre Status-Version: v1.2 Date: 28/04/2017 EC Distribution: Public (PU) Project Number: Project Title: FP7-610928 REVEAL Page 1 of 21
Title of Deliverable: Visualization and analytical tools Date of Delivery to the EC: 30/06/2015 Workpackage responsible for the Deliverable: WP5 - Modalities Analysis Framework Editor(s): Stuart E. Middleton (ITINNO) Contributor(s): ITINNO Reviewer(s): ATC Approved by: All Partners Abstract: Keyword List: This deliverable describes the visualization and analytics tools collectively grouped into a component called the REVEAL decision support system (DSS) framework. A set of situation assessment visualizations display a number of interactive multi-dimensional views on large volumes of real-time content items and person profiles associated with a news story or enterprise event. The overall visualization approach taken by the DSS framework is one of multidimensional views following a direct manipulation user interface (UI) metaphor. A software release accompanies this deliverable, installed and running on the WP6 project testbed. ITINNO is planning a TL6 demonstration service for the work in WP5 which is publically available called the Journalist Decision Support System (JDSS). This TL6 service provides a way to get direct user feedback and supports future exploitation paths by allowing potential customers to use the system for free in a limited support environment. Visualization, Trust, Credibility Page 2 of 21
DOCUMENT DESCRIPTION Document Revision History Version Date Modifications Introduced Modification Reason Modified by v0.1 20/06/2016 Setting up document, initial structure ITINNO v1.0 21/06/2016 Release candidate for partner QA ITINNO v1.1 27/06/2016 Version for coordinator QA ITINNO, ATC v1.2 28/04/2017 Public version ITINNO Page 3 of 21
CONTENTS 1 INTRODUCTION... 7 2 DECISION SUPPORT SYSTEM (DSS) FRAMEWORK... 8 2.1 PROTOTYPE DSS FRAMEWORK... 8 2.2 SITUATION ASSESSMENT VISUALIZATION... 9 2.3 JOURNALIST DECISION SUPPORT SYSTEM (DSS)... 18 2.4 HUMAN COMPUTER INTERACTION (HCI) EXPERIMENT... 18 3 MODALITY INNOVATION DESCRIPTION... 20 4 CONCLUSIONS... 21 5 REFERENCES... 21 Page 4 of 21
LIST OF FIGURES FIGURE 1: INFORMATION FLOW FOR DECISION SUPPORT SYSTEM FRAMEWORK... 8 TABLE 1: HTTP DSS FRAMEWORK INTERFACE... 9 FIGURE 2: MAP VIEW... 11 FIGURE 3: MAP VIEW DETAILS LAYER... 12 FIGURE 4: MAP VIEW WITH EXPLODING THUMBNAILS... 13 FIGURE 5: TEMPORAL VIEW... 15 FIGURE 6: TEMPORAL VIEW DETAILS LAYER... 16 FIGURE 7: TEMPORAL VIEW WITH EXPLODING THUMBNAILS... 17 Page 5 of 21
DEFINITIONS, ACRONYMS AND ABBREVIATIONS Acronym Title DSS Decision Support System DW Deutsche Welle, Germany HCI Human Computer Interaction HTML HyperText Markup Language HTTP HyperText Transfer Protocol ITINNO University of Southampton IT Innovation Centre, UK PM Person Month SINTEF SINTEF, Norway UI User Interface URI Uniform Resource Identifier WP Work Package Page 6 of 21
1 Introduction The visualization and analytics tools (i.e. software from D5.4.2) are collectively grouped into a component called the REVEAL decision support system (DSS) framework. The scope of this component is to provide an interactive view to any number of WP5 situation assessments running in real-time. Each situation assessment represents an aggregated collection of content filtered and annotated in the context of a specific news story or enterprise event to support verification decisions. The overall visualization approach taken by the DSS framework is one of multi-dimensional views following a direct manipulation user interface (UI) metaphor. This approach is motivated by the classic work of Ben Shneiderman [Shneiderman 1983] on Direct Manipulation Human Computer Interaction (HCI). Each visualization represents a different way to visualize, segment and explore the content items and people profiles behind each situation assessment database. Users (i.e. journalists or enterprise analysts) can visually explore each dataset, interactively changing filter and cluster settings to 'slice and dice' large datasets in real-time and zoom in to a small set of highly relevant content items that are worthy of manual inspection. All visualizations are based on a focus and context approach, with the contextual data displayed and clusters / sub-clusters adding layers on top of this context view. This avoids multi-screen switching and makes the interaction cognitively simpler and as natural as possible. The DSS framework visualizations are not pilot specific but will be integrated with the pilot user interfaces. Users will be able to drag and drop content items from the situation assessment visualizations and into the pilot UI's for subsequent end user specific processing and domain-specific use. Development work on the DSS framework component is a continuous process, with internal checkpoints along the way where end user feedback will be elicited to make sure the visualization work is on track with regards the use cases. A software release accompanies this deliverable, installed and running on the WP6 project testbed and is intended to elicit end user feedback. Page 7 of 21
2 Decision Support System (DSS) Framework 2.1 Prototype DSS Framework The DSS framework component is designed to continually pull data from the WP5 situation assessment framework (i.e. D5.3.2). A high-level view of the information flow is shown in Figure 1. User interactions via visualization HTML page - refresh data every few seconds - change visualization type - change visual dimensions displayed Situation Assessment (WP5) Database Decision Support Visualization (WP5) HTTP Endpoint End User Browser Data PULL Visualization HTML page + Javascript + D3 Pilot HTML page + Pilot web page + Control (e.g. crawl keywords, focus areas) + Link to each assessment s Viz HTML page Pilot UI (WP6) Web Server Figure 1: Information Flow for Decision Support System Framework The DSS framework consists of a HTTP endpoint that is able to (a) return HTML pages for each open situation assessment and (b) respond to interactive D3 Javascript HTTP requests for data. The visualization itself is rendered client side (i.e. in the client browser) using the open source D3 Javascript libraries 1. Interactive requests for data are issued by the D3 Javascript, which in turn are handled by the HTTP DSS framework endpoint. The HTTP interface is defined in Table 1. It supports the web page interface by providing data - end users do not see this interface. 1 http://d3js.org/ Page 8 of 21
Table 1: HTTP DSS framework interface HTTP Namespace Response Action GET /dss/api/ HTML landing page GET /dss/api/<assessment-id>/<viz-type> HTML page for selected visualization GET /dss/api/<assessment-id>/<viz-type>?<params> Top N clustered data items requested after applying filters Concept Description <assessmentid> Name of assessment (e.g. paris_2015) <viz-type> Name of visualization required = map timeline <params> {<type>:<value>} spatial, temporal and keyword filters to apply as well as cluster metrics and topn values needed The visualizations themselves are based on views of situation assessment aggregated datasets for each news story or enterprise event. For the situation assessment views feedback is limited to changing filter and cluster settings to visually explore and segment the crawled and annotated data. Many annotations (i.e. 15+) are supported for WP2/3/4/5 annotations such as hashtags, locations, mentioned URI s, damage reports, attributed entities etc. Users can interactively view content relevant as context for a verification decision. They can also export views to SVG to use as infographics in articles (e.g. news stories). The situation assessment framework is defined in detail in D5.3.2. Lastly details on the pilot UI's can be found in D6.3.2 and D6.4.2. 2.2 Situation Assessment Visualization In response to user feedback we have focussed on a map and temporal view, incrementally improving these visualizations over time to optimize them. We have not added additional views as our end users wanted to have a few well developed and refined views rather than many views that demonstrated different ideas but were not as refined and usable. We will until the end of the project be seeking user feedback and incrementally improving this work. Map view The map view is intended to display geospatial clusters at different levels of granularity. At the highest level of granularity content items are clustered by geographic region. When zooming in clusters for individual roads/rivers/etc. can be seen and lastly individual buildings/parks/etc. A set of filters (e.g. content type, time range, classifier results) and cluster settings (e.g. admin level for region clusters) are available to interactively change the view and help users explore the data available. The regions are colour coded based on mention count (i.e. brighter the colour the more mentions are in that region). Each location (e.g. a city) can be interactively selected and the top N posts for that location viewed in a details layer. The details layer renders posts as they would be in Twitter etc. ordered by created timestamp so the earliest mentions appear first. For verification end users wanted the earliest mentions to help with tracking down the original post of a media item (linked or embedded) and the author of this original post. A secondary cluster metric (e.g. URI, hashtag, damage report) can be specified and the top N metric values shown for all posts that contain a mention of the selected location. This is displayed in an Page 9 of 21
exploding thumbnail type layer and allows users to see the top N trending metrics (e.g. URI) for a given location. Each thumbnail can itself be selected and the posts mentioning its value will be displayed in the details layer. The view does not have any cached state, so changes in cluster or filter setting and/or selected locations will trigger another SQL lookup and re-rendering of the data. This stateless approach means the visualizations work in real-time with whatever data is available, and as new posts come along the view will change to reflect this. Examples of screenshots for the map view can be found in Figure 2, Figure 3 and Figure 4. The basic map view (i.e. Figure 2) presents administrative regions clustered at levels chosen interactively by the user such as countries, cities or boroughs. These can be selected and a details layer (i.e. Figure 3) shown for all content mentioning the selected location ordered by timestamp, earliest first. If a secondary metric is selected an exploding thumbnail view will show the top N metric values (e.g. URI s, hashtags, damage reports) from content items that contain a mention of the selected location (see Figure 4). These thumbnails can themselves be selected to display a details layer. Page 10 of 21
Figure 2: Map view Page 11 of 21
Figure 3: Map view details layer Page 12 of 21
Figure 4: Map view with exploding thumbnails Page 13 of 21
Temporal View The temporal view is intended to display a Gantt style view showing sampled sets of posts over a time window. A sample size can be chosen (e.g. 5 minutes, 1 hour or 1 day) and sample time is plotted on the x-axis of the view. A primary cluster metric (e.g. URI, hashtag, location, damage report) can be selected and this is plotted on the y-axis. A set of filters (e.g. content type, time range, classifier results) are available to interactively change the view and help users explore the data available. The sample cells are colour coded based on mention count (i.e. brighter the colour the more mentions are in that sample cell). Each cell on the Gantt chart can be selected and represents all the posts in that time sample that have the value of the chosen primary cluster metric (e.g. with a specific hashtag value). The details layer renders the top N selected posts as they would be in Twitter etc. ordered by created timestamp so the earliest mentions appear first. A secondary cluster metric (e.g. URI, hashtag, location, damage report) can be specified and the top N secondary metric values will be shown for all posts that contain a mention of the primary metric value and have a timestamp that falls within the selected sample cell. This is displayed in an exploding thumbnail type layer and allows users to see the top N trending metrics (e.g. URI) for a given temporal sample. Each thumbnail can itself be selected and the selected cluster posts mentioning its value displayed in the details layer. Examples of screenshots for the temporal view can be found in Figure 5, Figure 6 and Figure 7. The basic temporal view (i.e. Figure 5) presents sampled data on the x-axis and a primary cluster metric (e.g. URI, hashtag, damage report) on the y-axis. Each sample can be selected and a details layer (i.e. Figure 6) shown for all content mentioning the selected sample time and primary metric value ordered by timestamp, earliest first. If a secondary metric is selected an exploding thumbnail view will show the top N metric values from content items that contain a mention of the selected sample time and primary metric value (see Figure 7). These thumbnails can themselves be selected to display a details layer. Page 14 of 21
Figure 5: Temporal view Page 15 of 21
Figure 6: Temporal view details layer Page 16 of 21
Figure 7: Temporal view with exploding thumbnails Page 17 of 21
2.3 Journalist Decision Support System (DSS) ITINNO is planning to release a TL6 demonstration service for the work in WP5 which will be publically available and can be used for real. This is called the Journalist Decision Support System (JDSS) and will allow WP5 to get real-world feedback on the DSS and also provide engagement with potential customers to make exploitation easier post-project. The Journalist Decision Support Service will support a google login and allow end users to provide either (a) a Twitter list URI, (b) a Twitter collection URI or (c) a Twitter search string with keywords. A time range will be requested (within the 7 day limit from the current date imposed by Twitter) and the ITINNO crawler will be tasked to download a number of Tweets up to a sensible limit (e.g. 10,000 posts). All Tweets will be geoparsed, facts extracted and the eyewitness and fake classifiers run. The final data will be visualized using the WP5 visualizations (i.e. map and temporal views). End users will be able to interactively view and cluster the content found and identify key posts for help in their verification tasks. They will also be able to export views to SVG to add infographics to articles (e.g. news stories). This TL6 service provides a way to get direct user feedback and supports future exploitation paths by allowing potential customers to use the system for free in a limited support environment. 2.4 Human Computer Interaction (HCI) Experiment We are planning during the evaluation part of the project to run a Human Computer Interaction (HCI) experiment. This experiment will ask anonymous volunteers to try out the map and temporal view and a control view based on the Twitter keyword search interface. Results will help us evaluate the strengths and weaknesses of each view. The experiment outline is below. We have obtained ethical approval for this experiment plan via the University of Southampton ethical committee (ref ERGO/FPSE/18641). Pre-study Participants will be recruited with the help of DW and SINTEF. SINTEF will email students they run courses in Norway for asking for voluntary participation. DW will contact staff at DW (experienced journalists) to ask for voluntary participation. DW will use contacts (twitter invitation, email list invitation, web invitation) to ask for voluntary participation. ITINNO might also contact UoS staff involved in journalism courses to ask for voluntary participation. Participation is fully anonymous. No email addresses are requested or stored from participants. No financial incentives will be provided for participation and we expect participants to be motivated by the opportunity to engage with the REVEAL project and try out some of the tools that have been developed. Study procedure Task duration is 15 mins. A website will be setup to host the experiment. Recruitment material will include a URI to this website so participants can open the experiment webpage in a Chrome browser. Participants will be presented with a welcome page that includes a task description (see next), experiment details and a 'consent to participate' checkbox. The ERGO number will be displayed on this page for reference. Page 18 of 21
Participants will be asked to find the earliest mention of a target image from a corpus of social media content (Twitter, You Tube, Instagram) around the Nov 2015 Paris shootings. Each participant will be asked to find earliest mentions of 3 different target images. Each attempt will be performed with a different visualization (keyword search, spatial search, temporal search). They will have 5 minutes (representative of the type of search time a journalists might want to spend finding an original mention of a content item for a breaking news story) to find each image with a timeout if they do not manage in time. Social media content is pre-crawled 'raw' from Twitter, You Tube and Instagram. The social media datasets used for this experiment were obtained and stored in accordance with the social media T&C s. The dataset only stores links and metadata for content items. We will cache a set of thumbnails for content items which will be deleted after the trial. Links and thumbnails to content items are provided and participants can open these links to see the full content as rendered by the social media sites themselves. Therefore some content might be disturbing as it relates to a violent event (Paris shootings). No prior content filtering will be provided by us other than that provided by Twitter, You Tube and Instagram sites themselves (social media sites post warning on graphic content). The 3 target images for this task will not contain any graphic scenes (e.g. violent scenes) so participants can achieve the task without opening any social media URIs to content labelled graphic by social media sites. The web server will record data on (a) success or failure in finding first mention of target images (b) time to find first mentions (c) 'heat map' information about where mouse time was spend on the web page (d) click history on what was selected / browsed / searched during task. Post-study Metric data for each user will be analysed and presented in aggregate form in academic papers. Metric data will also be archived indefinitely so that results can in principle be reproduced on request (e.g. if an external researcher questions the validity of these results in some way). ITINNO will be the data controller for this metric dataset and will not provide access for any non-scientific reasons. Page 19 of 21
3 Modality Innovation Description Below is a module innovation description for components referred to in this deliverable. Module Name Decision Support System Framework Delivery date PM32 Module Overview The DSS framework provides an interactive view to any number of WP5 situation assessments running in real-time. Each situation assessment represents an aggregated collection of content filtered and annotated in the context of a specific news story or enterprise event. The DSS framework thus can render real-time annotations from WP2/3/4 and trust and credibility annotations from WP5 (e.g. facts extracted, fake & eyewitness classification). The DSS framework also acts as a host for the source bias analytics BLinD tool from D5.3.2. Based on existing work? (e.g. from other project or open source code) N/A Based on implementation of specific algorithms? (which? why?) Direct Manipulation User Interface Metaphor Innovation introduced The situation assessment UI is a component which supports end users. It is based on a classic HCI metaphor (direct manipulation). The innovation it provides is centred on spatial-temporalsemantic grounding and visualization of real-time evidence related to end user verification tasks. Is this considered a core innovation for the project? Why? Yes - Trust and Credibility analysis of social media content is a critical component to support the wider verification challenge REVEAL seeks to address. The DSS component allows visualization of the annotations associated with WP2/3/4 and trust and credibility annotations from WP5 (e.g. facts extracted, fake & eyewitness classification). What benchmarks will be used to evaluate the module performance? User interface response time (<1 second refresh rate) End user feedback (qualitative) Partners Involved and related WP/Task(s) ITINNO (T5.3 lead - development) ATC, SAG, DW (T5.3 contributor - user feedback) Page 20 of 21
4 Conclusions This deliverable describes the visualization and analytics tools (i.e. software from D5.4.2) which is collectively grouped into a component called the REVEAL decision support system (DSS) framework. The situation assessment visualization displays a number of multi-dimensional views on large volumes of real-time content items associated with a news story or enterprise event. The overall visualization approach taken by the DSS framework is one of multi-dimensional views following a direct manipulation user interface (UI) metaphor. A software release accompanies this deliverable, installed and running on the WP6 project testbed. The work in WP5 follows a continuous agile development process and this user feedback is continually sought and incremental updated made to improve the visualizations over time. ITINNO is planning a TL6 demonstration service for the work in WP5 which is publically available and can be used for real called the Journalist Decision Support Service (JDSS). This TL6 service provides a way to get direct user feedback and supports future exploitation paths by allowing potential customers to use the system for free in a limited support environment. 5 References Shneiderman, B. "Direct Manipulation: A Step Beyond Programming Languages", IEEE Transactions on Computers, Vol. 16, No. 8, August, pp. 57 69 (1983) Page 21 of 21