Deliverable D7.6. AXES HOME Digital Library System

Transcription

1 Project acronym AXES Project full title Access to Audiovisual Archives Project No Large-scale integrating project (IP) Deliverable D7.6 AXES HOME Digital Library System September 2014 SEVENTH FRAMEWORK PROGRAMME Objective ICT : Digital Libraries and Digital Preservation

2 PROJECT DELIVERABLE REPORT Project Grant Agreement number Project acronym: Project title: Funding Scheme: AXES Date of latest version of Annex I against which the assessment will be made: 30 June 2014 Access to Audiovisual Archives Large-Scale Integrating project (IP) Document Deliverable number: D7.6 Deliverable title AXES HOME Digital Library System Contractual Date of Delivery: 30/09/2014 Actual Date of Delivery: 30/09/2014 Author (s): Kevin McGuinness Reviewer (s): Max Kemman, Andy O Dwyer Work package no.: WP7 Work package title: Experiencing Digital Libraries Work package leader: Dublin City University Version/Revision: 1.1 Draft/Final: Final Total number of pages (including cover): 32 AXES Page2 of 32 D7.6

3 CHANGE LOG Reason for change Issue Revision Date Initial revision 1.0 6/09/14 Integrated changes from Max /09/14 AXES Page 3 of 32 D7.5

4 DISCLAIMER This document contains description of the AXES project work and findings. The authors of this document have taken any available measure in order for its content to be accurate, consistent and lawful. However, neither the project consortium as a whole nor the individual partners that implicitly or explicitly participated in the creation and publication of this document hold any responsibility for actions that might occur as a result of using its content. This publication has been produced with the assistance of the European Union. The content of this publication is the sole responsibility of the AXES project and can in no way be taken to reflect the views of the European Union. The European Union is established in accordance with the Treaty on European Union (Maastricht). There are currently 27 Member States of the Union. It is based on the European Communities and the member states cooperation in the fields of Common Foreign and Security Policy and Justice and Home Affairs. The five main institutions of the European Union are the European Parliament, the Council of Ministers, the European Commission, the Court of Justice and the Court of Auditors. ( AXES is a project partly funded by the European Union. AXES Page 4 of 32 D7.5

5 TABLE OF CONTENTS Change Log 3 Disclaimer 3 Table of Contents 5 Summary 6 Introduction 7 System Architecture 10 Key technologies 11 Server deployment 12 System API 12 Features and Functionality 13 Tablet-oriented interface 13 User management 13 Simplified search interface 14 Visual similarity search 14 Interacting with results 14 Content discovery 14 Social features 15 Deep links 15 User Interface Description 16 Home page 16 Sidebar navigation 17 News feeds 18 Explore popular 19 Explore topics 20 Search interface 21 Result list view 22 Asset view 23 Transcripts 24 Key frames and face tracks 25 User registration and login 26 User history and bookmarks 27 Conclusion 28 Appendix 29 Search 29 Assets 29 News 30 Interesting items 30 Popular items 30 Image upload and storage 31 User management 31 Administration 32 AXES Page 5 of 32 D7.5

6 SUMMARY This deliverable describes the AXES HOME digital library system. It describes the overall system architecture, the specific features and functionality in this version of the interface, and the user interface components themselves. The system is based on user requirements from D1.6 and design specifications in D7.3. It communicates with other AXES systems via the link management system and structured search system from WP6, which in turn consumes processed video content via the WebLab system in WP8. Other deliverables describe the specifics of these systems and the various content indexing and search systems (WP2-5); this deliverable describes the integrated system from the user interface perspective. AXES Page 6 of 32 D7.5

7 INTRODUCTION This document describes the AXES HOME system, the third system to be developed by the AXES FP7 project. It differs from previous iterations of AXES PRO (D7.1) and AXES RESEARCH (D7.2) by targeting users in a casual setting, providing a portal for users entertainment needs. Like the other AXES systems, the AXES HOME system is web browser based. However, unlike the other systems, the home system is specifically optimized for tablet interfaces. We have implemented the home interface using technologies that allow it to be deployed on numerous commercial tablets and handheld devices, as well as supporting traditional desktop/laptop web browsers. The design of user interface for the home system is based on several sources, including the AXES RESEARCH interface and the corresponding mock-ups and specifications described in D7.1/2 and D7.4, and on the user requirements, personas, and scenarios gathered and developed in WP1 for the home user and described in D1.6. Wireframes and initial mock-ups for the user interface design were described in detail in D7.3. However, since this deliverable, and based on feedback from user and technology partners, we have significantly overhauled AXES Page 7 of 32 D7.5

8 the design. The final design shown in this document differs significantly in look and feel from the original designs described in D7.3. The new design is simplified and places greater emphasis on novel features like exploring using news feeds, rather than trying to be as feature complete as AXES RESEARCH. The idea is to allow us to showcase AXES technology in a way that is easy to use and transparent to end-users. The new design re-uses many layout concepts from the previous design, incorporates several new features, and omits others. We have chosen to omit certain features, such as browse by location and mood, as they are difficult to implement due to lack of appropriate metadata in some of our target collections. The figure below shows the new and original asset view designs for comparison. As can be seen, the layout is similar, but the overall look and feel is radically different. Note that the image of the new design (right) is actually a screenshot of a real functioning system, whereas the older design (left) is just a mock-up. The increased focus on simplicity is even more apparent by comparing the old home and new home page design (left and right below). The remainder of this document is organized as follows. Section 2 gives a brief overview of the system architecture from the user interface perspective. This section describes the major components of the system and how it interacts with other components via the link AXES Page 8 of 32 D7.5

9 management and structured search system from WP6. It also outlines some of the key technologies used on both the client and server-side. Section 3 describes the features and functionality of the system from the user perspective, with a particular emphasis on features that have been added since the last deliverable, D7.4. Section 4 gives a walkthrough of the user interface itself, and serves as documentation of the interface components and an illustration of the new interface design. Section 5 concludes the deliverable. AXES Page 9 of 32 D7.5

10 SYSTEM ARCHITECTURE The architecture of the AXES HOME system is very similar to that of the AXES RESEARCH system. The user interface is client-server based. The client-side is a single-page web application (SPA) based on the standard browser technology stack: HTML5, CSS3, and JavaScript. The server-side of the UI is, like AXES RESEARCH, written in Python and exposes a HTTP-based REST API that can be accessed by the client. Communication between the client and server is achieved via AJAX, and JSON is used as the data format. The server maintains its own document-oriented database (MongoDB) for user profiles, bookmarks, history, query logs, and result caches. It communicates with the backend link management and structured search layer via a JSON-RPC based API. The link management and structured search layer is responsible for federated search using its internal databases and indices in combination with various (possibly distributed) external search services. These external services include the on the fly visual face, concept, and instance search services and the (instance and face) similarity search engines. AXES Page 10 of 32 D7.5

11 Key technologies AXES HOME is built on an updated version of the same underlying link management and structured search software from WP6 as AXES RESEARCH and uses the same JSON-RPC based communication protocols to communicate with this system. Many of the same technologies were used for developing AXES HOME as were used in AXES RESEARCH: the system is again based on single-page web application technology, uses a REST API and AJAX/JSON for communication between the client and server, and uses a similar set of server side technologies such as MongoDB for the database and Python as the implementation language. Several newer technologies were adopted for implementing AXES HOME, and some technologies used in AXES RESEARCH, such as Knockout.js and Sammy.js were dropped. The following are some of the key client-side technologies and frameworks that are used by the AXES HOME codebase. Angular.js: an open-source web application framework, maintained by Google and community, that assists with creating single-page applications, one-page web applications that only require HTML, CSS, and JavaScript on the client side. Its goal is to augment web applications with model view controller (MVC) capability, in an effort to make both development and testing easier. SASS: a stylesheet language to allow modular development of CSS heavy web applications. Adobe topcoat: an open source CSS library designed to help developers build web apps with an emphasis on speed with components specifically tailored for mobile and tablet interfaces. As with AXES RESEARCH, Python was used to implement the server-side of the UI. We reused several technologies that we found worked well with AXES RESEARCH, including the MongoDB database, Jsonrpclib, and supervisor (see D7.5). Several new server-side technologies were, however, also used when implementing the UI. Flask: a lightweight web application framework written in Python and based on the WSGI toolkit and Jinja2 template engine. This is a departure from our previous Django based system. Several flask components were also used to interface with the database (Flask-Pymongo), ease the creation of a REST interface (Flask-RESTful), and handle user logins (Flask-Login). The reason for the move from Django to Flask is that Flask turned out to be better suited to creating REST-based APIs (as opposed to general web APIs). AXES HOME uses the AngularJS framework on the client-side, which works makes working with REST style interfaces very convenient. Gunicorn: a Python WSGI HTTP Server for UNIX that we use to deploy the interface on a server. We had previously used Apache mod_wsgi for this, but have moved away from it since it causes issues with some libraries (most notably, numpy/scipy). Nginx: an open source reverse proxy server for HTTP, HTTPS, SMTP, POP3, and IMAP protocols, as well as a load balancer, HTTP cache, and a web server. We currently recommend deploying the user interface using nginx as a reverse proxy for gunicorn and to host the static files. It is also possible to use Apache2 as a reverse proxy. AXES Page 11 of 32 D7.5

12 Server deployment The diagram above illustrates a typical deployment setup for the AXES HOME user interface. The setup uses nginx as a HTTP server to host static UI files (HTML, CSS, and Javascript files) for any URLs starting with /axes/home. The server-side REST API is a Python WSGI application that is hosted on a Gunicorn server running on TCP port This is reverse proxied by nginx so that any HTTP requests to the paths /axes/home/api are forwarded to the gunicorn process. The server-side process uses MongoDB to store data and communicates with an externally running instance of LIMAS using the JSON RPC interface. Detailed information on how to deploy the user interface, including how to install the relevant dependencies, and the different containers in which the server-side software can be run, accompanies the project source code. It can also be viewed on the main page on the source code repository home page: (contact kevin.mcguinness@dcu.ie for access) System API All the features of the user interface can be used by any external application using the AJAX compatible HTTP JSON-based REST API. The API is structured around a set of hypermedia resources and implemented using the Flask RESTful framework. The appendix to this document summarizes the available hypermedia resources, endpoints, and supported HTTP methods. AXES Page 12 of 32 D7.5

13 FEATURES AND FUNCTIONALITY The following describes the main features and functionality of the user interface from the end-user perspective. The specific details of the techniques behind and technologies used to implement many of these components are omitted here, as further details can be found in the relevant deliverables. This section focuses on the features themselves; the next section of the deliverable shows how these features are realized in the user interface. Tablet-oriented interface Handheld devices have become synonymous with relaxation/leisure so, given the nature of the AXES HOME system and its focus on entertaining the user, we have designed the user interface to specifically target tablet based interfaces such as ipads. These devices offer the user an easy gateway into a world of content, both portable and easy-to-use, they will provide an excellent vehicle to give users access to the AXES HOME system. The new interface is web based and will also work well using a standard web browser on, for example, a laptop, but it is specifically optimized for tablet screens. With small modifications, it is also possible to compile the interface into an app using technologies like Apache Cordova, rather than running it in a web browser, and thus provide users with a native app feel. There are several major advantages of using a HTML5 based web-app and wrapping it using a technology like Apache Cordova over the alternative native app strategy. First, using this strategy we do not need to design, develop, and maintain a separate app for each target platform (e.g. Android, ios, etc.), which would be infeasible without a large development team. Second, the application can be developed using high-level open web languages (Javascript/CSS/HTML5) and technologies, instead of needing to develop on proprietary platforms and APIs and in lower-level languages like Java and Objective-C. Third, as the application uses open web standards, it is more robust to future changes in these proprietary APIs (e.g. the ios SDK, and the Android API) by their respective owners. Fourth, the application can be easily updated without needing to push changes through an App store and wait for customers to update. Fifth, it is possible for users to use the app via a web browser, without needing to install an app. Finally, it simplifies development and debugging many facets of the application can be tested from a standard web browser during development. User management To facilitate features personalized to each user and to control access to APIs, the system contains a complete user registration and authentication system. The user management system differs from AXES RESEARCH in that it does not require users to login users can still interact with the system anonymously. However certain features, such as user history and bookmarks, are only available after the user logs in. AXES Page 13 of 32 D7.5

14 Simplified search interface In AXES RESEARCH, the user was responsible for selecting the search modality, i.e. metadata search, search in speech, or visual search on categories, instances, or people. AXES HOME provides a significantly simpler text search interface that is more suitable for home users: a single unified search box. Such a simplified interface was experimented with in AXES RESEARCH, and positively evaluated by journalists (see D1.7). The search text is passed verbatim to the link management and structured search system with a flag indicating that the system should perform an automatic search. In this mode, the search text is analysed automatically and the most appropriate search modality is selected based on the query and the indexes in the system. Visual similarity search Like AXES RESEARCH, AXES HOME exposes visual similarity search to allow users to find video segments that are in some sense visually similar to a query image. Unlike AXES RESEARCH, however, this is now available at the video frame level. Users can pause the video at any time during playback and search using the currently displayed frame. To keep the interface simple, querying using multiple images and images from external sources is not supported in AXES HOME. Interacting with results Users can interact with videos and segments in many ways. Results can be viewed as thumbnails or in a more detailed list. Users can playback videos, browse metadata and transcripts, and browse a video by keyframe. Users may save interesting videos by bookmarking them. Content discovery In line with the differences in user requirements, AXES HOME places more emphasis on content exploration, browsing, and discovery than AXES RESEARCH and AXES HOME. Therefore, we have included several content-discovery features in AXES HOME. As with AXES RESEARCH, we also have related videos and related segments at the asset level to link content to other potentially interesting content, and provide soft linking by visual similarity search. However, AXES HOME provides two new content discovery mechanisms: browse by news feeds and browse by topic/entity. In the news feeds view, news is aggregated from various sources and displayed to the user; tapping on a news source or news item will use this source or item to find related content in the archive. This allows users to find interesting and topical content based on current affairs and other news sources such as recipes and travel blogs. Browse by topic/entity allows users to find content in the archive related to a particular topic. This is currently implemented by querying the link management system for a list of topics and related content, which, in our current implementation, correspond to pretrained ( uber ) classifiers and the corresponding top-ranked results. Finally, as with AXES HOME, users can browse content by popularity, allowing them to find videos that appear to be interesting to the wider community. AXES Page 14 of 32 D7.5

15 Social features AXES HOME includes some of the same social features as AXES RESEARCH, including the ability to track the amount of views a video has and allow users to like a particular video. AXES HOME also allows users to share links to content on social media (currently Twitter) directly from within the application. Deep links Like AXES RESEARCH, AXES HOME provides deep links for most pages. This means that, for example, if the user performs a complex query, once the results are displayed, the URL displayed in the browser will bring them back to that page. This allows users to bookmark results, use the browser back and forward buttons, and share links with others. Deep links are implemented for most of the major features, including (but not limited to) queries, result lists, asset views, and news feeds. AXES Page 15 of 32 D7.5

16 USER INTERFACE DESCRIPTION This part describes the user interface design and functionality. The final design deviates significantly in look and feel from the original designs described in D7.3. The overhaul in design was based on feedback from the partners and meetings clarifying the most important requirements and objectives for AXES HOME. The new design re-uses several layout concepts from the previous design, but incorporates several new features based around novel ways to explore the archive, such as exploration using news feeds and concepts. The following presents a screenshot tour of the user interface views and features. Home page The home page is the first page that the user sees when they access the system. It is prepopulated with a set of recommended videos, which can contain topical or popular content. The algorithm for determining which content is to be recommended is implemented by the link management and structured search system (LIMAS, WP6). Recommended content, in the form of video segments, is displayed as a grid of key-frames and corresponding titles and descriptions. The navigation bar at the top is always present and provides a simple text based search interface that allows the user to perform automatic searches, and access to a slide out navigation menu that allows users to access other features of the application. AXES Page 16 of 32 D7.5

17 Sidebar navigation AXES HOME is optimized for tablet based viewing and uses the well-known slide out menu user interface idiom commonly used for navigation on such devices. Tapping the menu button in the main navigation slides out the menu. From here the user can explore the content in various ways (such as by news feed) and see their profile, history, and bookmarks. AXES Page 17 of 32 D7.5

18 News feeds AXES HOME is designed to accommodate users that occasionally do not have particular information needs when they visit the archive, but rather just want to explore interesting and topical content. The news feeds view, accessible from the slide in menu, aggregates news from various sources and allows the user to find content related to feeds and articles. The news feeds view shows all news feeds stacked vertically, with the news items scrollable horizontally. Tapping or clicking on a feed name will search for content in the archive related to the entire feed. For example, clicking on Deutsche Welle in the above screen will search using the link management and structured search system for content related to all articles in the feed. Tapping or clicking on a feed item will search for content related to that particular news item. For example, clicking on the item entitled Tiananmen Square: A dangerous memory would search for video and video segments related to Tiananmen Square. AXES Page 18 of 32 D7.5

19 Explore popular The user interface keeps track of how many times a video is viewed by users of the system. The popular videos view shows all videos ranked by how many times they were viewed, providing users with quick access to the most popular content in the system. Tapping on a video segment will drill down to the asset view for the segment. In the current implementation, the videos shown are the most popular of all time. This could be relatively easily updated to include temporally popular videos too (most popular this week/month/year). AXES Page 19 of 32 D7.5

20 Explore topics Users can also explore videos by topic. The interface itself is currently agnostic to the method used to retrieve such topics, and simply queries the link management system for a list of topics. Like the news items, the topics are shown stacked vertically, and the top ranked video segments corresponding to the topic are scrollable horizontally. Tapping on a video segment will drill down to the asset view for the segment. AXES Page 20 of 32 D7.5

21 Search interface AXES HOME provides users with a very simple text search interface: a single text field in the upper right of the application. The system uses the user input to perform an automatic search using the link management and structured search system, assuming that the backend system can intelligently select the best search modality (metadata, visual concept, visual face, etc.) based on the input. The results from a search are displayed in a grid by default as illustrated in the screenshot above. Tapping or clicking on a result will drill-down to the asset view for that result. A bar at the top of the result list displays the number of video segments (clips) retrieved and the total number of full videos that these segments belong to. AXES Page 21 of 32 D7.5

22 Result list view The results bar also allows the user to group results by video or clip (top right) and view the results in a list view rather than a grid view (top left). The screenshot above shows the list view. AXES Page 22 of 32 D7.5

23 Asset view At any point when navigating the archive content, tapping or clicking on a video or video segment will bring up the asset view for that segment. The bar on the top of the asset view shows the video title. On the top left is the video playback area. As with AXES RESEARCH, if a video segment is selected, the video is automatically moved to the start of the relevant segment. The bar below this allows the user to interact with the video in various ways. From here, the user can bookmark the video for later viewing (if the user is logged in), like the video, share the video on social networks (for now, this allows the user to post a link on twitter), and search for visually similar content. The similarity search feature uses the current video frame as the query, allowing the user to find similar content to any frame by simply pausing the video at the point of interest and clicking search. The area below the video shows the number of times the video has been viewed, the number of times it has been liked, and various metadata associated with the video such as license, publication date, genres/categories, and description. The area on the right shows the top-10 related video clips, and, scrolling down, the top-10 related full videos. AXES Page 23 of 32 D7.5

24 Transcripts Scrolling down in the asset view allows the user to navigate the current video in various ways: by transcript, keyframes, and faces. The transcripts view displays any available transcript or ASR and allows the user to tap on speech content to automatically jump to the relevant point in the video. AXES Page 24 of 32 D7.5

25 Key frames and face tracks The key frames view allows the user to navigate the selected video by keyframe. Key frames are displayed chronologically in a grid, and annotated on the top-right with the time they occur in the video. Tapping on a keyframe will seek the current video to that keyframe. In addition to key frames, users can navigate the video by browsing the face tracks that occur in the video. Like key frames, these are ordered chronologically and annotated with the start time. Each face track is represented by the key face image associated with the track. Tapping on a key face will jump to the relevant portion of the video. AXES Page 25 of 32 D7.5

26 User registration and login Although, unlike AXES RESEARCH, users are not required to register or log into the system, they must do so if they wish to bookmark items for later use or have the system track user history. The system provides a simple registration and login mechanism for users that wish to use these features, as shown in the screenshots above. At present, there are no user customizable preferences in the system, but the user management system can be used to support this if needed in the future. User preferences can be included on their profile page (below). AXES Page 26 of 32 D7.5

27 User history and bookmarks Logged in users can access the full history of videos that they have viewed (above left) and a list of all videos they have bookmarked (above right). AXES Page 27 of 32 D7.5

28 CONCLUSION This deliverable has described the AXES HOME user interface developed as part of WP7. The user interface is based on the AXES interface mock-ups and specifications described in D7.3. The design has undergone significant changes since D7.3 based on feedback from user and technology partners. The user interface is built on top of the link management and structured search system developed in WP6, which in turn uses the integrated indexing components from WP8. The user requirements, personas, and scenarios from WP1 were used to guide the design decisions and choose features for implementation. The user interface is optimized for tablet use but also runs in all modern web browsers and has been tested on Safari, Firefox, and Google Chrome. AXES Page 28 of 32 D7.5

29 APPENDIX Search Hypermedia Resource: SimpleSearch Endpoint: /search Hypermedia Resource: ImageSearch Endpoint: /image-search Hypermedia Resource: AdvancedSearch Endpoint: /advanced-search Assets Hypermedia Resource: Asset Endpoint: /assets/<axes:uri> Hypermedia Resource: VideoStats Endpoint: /video-stats/<axes:uri>, PUT Hypermedia Resource: RelatedVideos Endpoint: /related-videos/<axes:uri> Hypermedia Resource: RelatedSegments Endpoint: /related-segments/<axes:uri> Hypermedia Resource: Keyframes Endpoint: /keyframes/<axes:uri> Hypermedia Resource: Transcript AXES Page 29 of 32 D7.5

30 Endpoint: /transcript/<axes:uri> Hypermedia Resource: FaceTracks Endpoint: /face-tracks/<axes:uri> News Hypermedia Resource: NewsSources Endpoint: /news-sources Hypermedia Resource: NewsItems Endpoint: /news-items/<axes:uri> Hypermedia Resource: NewsItemSearch Endpoint: /news-item-search/<axes:uri> Hypermedia Resource: NewsSourceSearch Endpoint: /news-source-search/<axes:uri> Interesting items Hypermedia Resource: InterestingItems Endpoint: /interesting-items Hypermedia Resource: InterestingTopics Endpoint: /interesting-topics Popular items Hypermedia Resource: PopularQueries Endpoint: /popular-queries AXES Page 30 of 32 D7.5

31 Hypermedia Resource: PopularVideos Endpoint: /popular-videos Image upload and storage Hypermedia Resource: ImageStore Endpoint: /image-store HTTP Methods: POST User management Hypermedia Resource: UserLogin Endpoint: /user/login HTTP Methods: POST Hypermedia Resource: UserProfile Endpoint: /user/profile Requires: user login Hypermedia Resource: UserLogout Endpoint: /user/logout HTTP Methods: POST Hypermedia Resource: UserRegistration Endpoint: /user/register HTTP Methods: POST Hypermedia Resource: UserHistory Endpoint: /user/history Requires: user login Hypermedia Resource: UserBookmarks Endpoint: /user/bookmarks, POST Requires: user login AXES Page 31 of 32 D7.5

32 Hypermedia Resource: UserBookmark Endpoint: /user/bookmarks/<axes:uri>, PUT, DELETE Administration Hypermedia Resource: ServiceInfo Endpoint: /service-info Hypermedia Resource: VersionInfo Endpoint: /version-info AXES Page 32 of 32 D7.5