D 4.8 PATHway platform validation and system usability report

Transcription

1 WP4 D 4.8 PATHway platform validation and system usability report Leading Author(s): Andrew Pomayanskyi (Nurogames GmbH) Status -Version: Version 0.9 Contractual Date: 31 th May 2017 M27 Actual Submission Date: 30 th May 2017 Code: D4.8 PATHway Platform Validation_V0.9.pdf ` 1 30/05/2017

2 Disclaimer This document contains material, which is the copyright of certain PATHway contractors, and may not be reproduced or copied without permission. All PATHway consortium partners have agreed to the full publication of this document. The commercial use of any information contained in this document may require a license from the proprietor of that information. The PATHway Consortium consists of the following members: Participant no. Participant Name Short Name Country 1 Dublin City University DCU Ireland 2 CERTH Information Technologies Institute & Institute of Applied Bioscience CERTH/ITI & CERTH/INAB Greece 3 University of Ulster UU UK 4 Electronic Record Services BV ERS Netherlands 5 Nurogames GMBH NG Germany 6 ENGINEERING- Ingegneria Informatica Spa ENG Italy 7 University of Leuven KU Leuven Belgium 8 University of Glasgow UGL UK 9 Mater Misericordiae Hospital limited Mater Ireland 10 University of Limerick UL Ireland The information in this document is provided as is and no guarantee or warranty is given that the information is fit for any particular purpose. The user thereof uses the information at its sole risk and liability. ` 2 30/05/2017

3 Contributors Name Timo Kästner Andrew Pomazanskyi Marco Polak Jens Piesk René Schippers Paul de Clercq Kieran Moran Catherine Woods Deirdre Walsh Haolin Wei Leonardo Gualano Petros Daras Argyris Chatzitofis Dimitris Zarpalas Dimitris Filos Andreas Triantafyllidis Paolo Zampognaro Fabio Melillo Veronique Cornelissen Jomme Claes Company NG NG NG NG ERS ERS DCU DCU DCU DCU DCU CERTH/ITI CERTH/ITI CERTH/ITI CERTH/INAB CERTH/INAB ENG ENG KUL KUL KUL Internal Reviewers Name Company Internal Approval Date Kieran Moran DCU 19/05/2017 René Schippers ERS 18/05/2017 ` 3 30/05/2017

4 Document Revision History Date Issue Author/Editor/Contributor Summary of main changes 10/11/ Andrew Pomazanskyi (NG) Initial table of contents 14/11/ Paolo Zampognaro [ENG] Table of content refinement 15/11/ Fabio Melillo [ENG], Paolo Zampognaro [ENG] 10/01/ Fabio Melillo [ENG], Paolo Zampognaro [ENG] 30/01/ Timo Kästner, Andrew Pomazanskyi (NG) Initial draft Sec Filled Sec with results and screenshot about acceptance testing for Usage Reporting FE application. Filled Sec with description and screenshot about the whole usability testing results. Section 3 description to requirements and project plan. Integrated the results of acceptance test for exercise application 20/02/ Fabio Melillo [ENG] Filled Sec with results and screenshot about acceptance testing for Usage Reporting FE application. 15/03/ Marco Polak, Andrew Pomazanskyi (NG) 05/04/ Andrew Pomazanskyi, Timo Kaestner, Marco Polak Section 4 contribution for test specification Integration of acceptance tests and usability feedback for second prototype, section /05/ Andrew Pomazanskyi Executive Summary, Introduction, Conclusion, GAP analysis 15/05/ Jomme Claes, Integration of usability testing procedures and feedback. 18/05/ René Schippers Integration of acceptance tests for first prototype, section and second prototype, section 23/05/ Andrew Pomazanskyi, ash Shekhawat Final integration of usability feedback, addressing feedback ` 4 30/05/2017

5 Table of contents Disclaimer... 2 Contributors... 3 Internal Reviewers... 3 Document Revision History... 4 Table of contents... 5 Abbreviations (alphabetically)... 7 List of Figures... 8 List of Tables... 9 Executive Summary Introduction Definition of PATHway Requirements Project Plan Gap Analysis Software Test Specifications & Cases Test Result Recording Recording Integration Tests Recording Acceptance Tests Recording Usability Test Test Exception Handling Testing results Prototype P1: M14_M Integration testing Acceptance Testing Exercising Application Usage Reporting & PA data Acquisition Front End Clinical & Research Front End Usability Testing Conclusions and Action Points for PATHway P ` 5 30/05/2017

6 5.2. Prototype P2: M22-M Integration testing Acceptance Testing Exercising Application Usage Reporting & PA data Acquisition Front End Clinical & Research Front End Notification (backend) Service Usability Testing Conclusions and Action Points beyond the Prototype Operations/Maintenance PATHway Updater Conclusion References Annex A ` 6 30/05/2017

7 Abbreviations (alphabetically) API DSS Application programming interface Decision support system M18 Month 18 PA P1 P2 RCT Physical Activity First prototype Second prototype Randomized controlled trial ` 7 30/05/2017

8 List of Figures Figure 1. PATHway system as a whole Figure 2. Involved concepts and their relationships from D Figure 3. Integration activity execution Figure 4. Integration status at M Figure 5. All issues view Figure 6. PATHway workflow management Figure 7. PATHway issue tracker example Figure 8. PATHway issue history Figure 9. Issue tracking and management via PATHway cockpit Figure 10. Lifestyle Dashboard completion update P Figure 11. Usability questionair template Figure 12. Compilation Guide Figure 13. Radar Chart of usability feedback for Home Platform Access Figure 14. Radar Chart of usability feedback for ExerClass Figure 15. Radar Chart of usability feedback for ExerGame Figure 16. Radar Chart of usability feedback for Exercise Practice Figure 17. Radar Chart of usability feedback for Preference Setting Figure 18. Radar Chart of usability feedback for Active Lifestyle Figure 19. Radar Chart of usability feedback for Usage Reporting Figure 20. Radar Chart of usability feedback for Assessment Figure 21. Radar Chart of usability feedback for Behavioural Change Figure 22. Radar Chart usability feedback average Figure 23. User feedback analysis Figure 24. Implementation Cockpit P Figure 25. Integration with agile software development methodology Figure 27. Usage and Reporting FE acceptance testing P Figure 28. Complition of UC testing for FE in cockpit Figure 29. Schematic setup of the room during PATHway usability testing Figure 30. Usability Testing P Figure 31. Pathway Component Completion Figure 32. PATHway Error Screen Figure 33. PATHway Updater flow Figure 34. PATHway Updater UI ` 8 30/05/2017

9 List of Tables Table 1. PATHway Home environment use cases Table 2. PATHway Clinical Environment Use Cases Table 3. PATHway GAP analysis Table 4. Acceptance test for exercise application P Table 5. Acceptance test for Usage Reporting Frontend P Table 6. Acceptance testing for Clinical Frontend P Table 7. Usability report per functionality Table 8. Prototype 2 user feedback prioritization Table 9. Exercise Application acceptance testing P Table 10. Acceptance test results for Clinical & Research Frontend P Table 11. Error Type Code Table 12. Component Code Table 13. Method Code ` 9 30/05/2017

10 Executive Summary This deliverable reports the results of the system validation tests that has been performed in two separate tests, for Prototype 1 and the final fully integrated system, presented as a Prototype 2 respectively. More specifically we report the process that the consortium partners have followed to ensure that the integration of the whole system was appropriate and as expected by the requirement documentation (D2.3 and D2.5) [1] [2], and design ensuring software reliability and usability. The validation procedure takes an iterative approach and consisted of integration validation, acceptance and usability tests, which were defined in the validation plan (D4.2) [3]. Such acceptance and usability tests are based on the functional specification described in D2.5 [2]. Therefore, the following deliverable is structured as follows: - SECTION 1: the introduction - SECTION 2: provides the definition of PATHway system following the requirement gathering process as well as the final PATHway requirements list with which the functionality has been validated, - SECTION 3: presents the project plan and the overall steps towards validation including the GAP analysis. - SECTION 4: briefly outlines the steps followed during the acceptance and usability tests - SECTION 5: reports the results of the performed tests for each of the testing periods for three applications (Exercise application, Clinical Interface, Lifestyle Dashboard) and one service (Notification backend service) with respect to prototype at M18 and the final integrated version in M27. - SECTION 6: provides an overview of how the consortium plans address the latest feedback coming from the acceptance and usability tests - SECTION 7: outlines the actions taken to further maintain the system in case some bugs are determined In summary, the system validation activities and this deliverable serve to verify the appropriateness of the design foundation, the interoperability/communication specification ` 10 30/05/2017

11 are correctly implemented (T4.1), the end user requirements are fulfilled (T2.1), and that the platform is ready for full-scale human intervention studies (T5.2). ` 11 30/05/2017

12 1. Introduction Acceptance tests and usability reports are the essential parts of PATHway validation process. It is an ongoing process that runs from the initiation of the project until its completion. PATHway system validation is meant to be performed at given cut-off dates for two prototypes of the system therefore the validation activities were respectively planned, specified [2], verified against the specification and finally reported; which is the body of this deliverable. It is important to note, that continuous monitoring and validation as well as integration testing has been performed on continuous bases throughout the project runtime and has been reported under the respective sections of the integration testing. Since PATHway system incorporates a number of components, the acceptance of software envisaged the analysis of the use cases within a specific usage context of the user (Exercise application, Lifestyle and Physical Activity Dashboard, Clinical Interface, Notification services) and included the evidence that all the components and software requirements have been implemented appropriately and are traceable to the system requirements. PATHway system validation is a critical and important tool to assure the quality of the developed product and services. This process is introduced to increase the usability and reliability of the applications, resulting in less software failures, fewer corrective actions and technical support, as well as less risk to the patients and clinical professionals. In addition to that, appropriate system validation reduces the long term costs in case modifications are required and certain functionalities have to be revalidated. System maintenance in many cases represents a decent share of the total costs of software life-cycle and an established and well-documented validation plan and process help to reduce the overall cost of subsequent validation of newer releases. ` 12 30/05/2017

13 2. Definition of PATHway The main purpose of PATHway is the creation of an exercise platform for CVD-patients to increase the adherence rate to the rehabilitation program and improve the overall quality of life. Figure 1. PATHway system as a whole The exercise programme is delivered through the exercise class and exercise game and based on patient multi-parametric modelling and decision support system (DSS). It monitors the patient s actions and physiological response in real time and provides feedback and instructions through a virtual coach (Avatar). A social interaction component offers the ` 13 30/05/2017

14 possibility for remote users to schedule events and exercise together at the same time interacting through vocal communication component therefore stimulating peer support and peer mentoring. The affective recognition module tracks and evaluates the enjoyment level of the participants while performing exercises. The physical activity assessment module offers an all-day monitoring of patient s physiological responses, also outside of the PATHway application, aggregates and abstracts this data over time to adapt the exercising program and provide feedback to the health professionals through the health data management system. The aggregated information on physical activity is presented through a dedicated Active Lifestyle web interface, which can be accessed via a desktop browser as well as viewed directly within the Home application. Such information is available to the physician through the Clinical & Research frontend. In addition to that the behavioural change module offers a holistic approach to improving healthy lifestyle with relation to dietary habits, smoking cessation, alcohol moderation, stress management, medication adherence and physical activity offering an initial assessment and goal management features. All input data is being validated before its transmission amongst components to prevent inconsistency and errors. 3. Requirements When performing a system validation tests, one of the first steps to consider is the design input requirements, which according to the validation plan, have to be verified. A requirement reflects the implied needs of the clinical partners that were captured during the runtime of the project and documented in the D2.3 [1] for the Prototype P1 released in M18 and D2.5 [2]for the final Prototype P2 released in M27. The requirement gathering process started with the definition of the usage scenarios that are designed to present user roles interacting with the system and describing a single path logic. It presents a real-world example that was further elaborated, formalized and plotted in a number of the use cases. A detailed list of captured requirements with their foreseen functionalities is detailed within the D2.5 [2] and summarised below once again, differentiating between the Home environment use cases and Clinical Environment Use Cases: Feature Use Case Id Home Platform Access Home Platform Access H_UC01 Home Exercising - Single User ExerClass Home Log-in Goals Change Proposal Confirmation [new] Single User ExerClass Patient s Avatar Visualization H_UC02 H_UC59 H_UC03 H_UC04 ` 14 30/05/2017

15 Wearing Sensors Instructions Pre-Screening [updated] Pre-Screening Instructions ExerClass Demonstration Motion Evaluation Visualization Vital Signs Visualization Real-Time ExerClass Adaptation Exercise Pausing Exercise Resuming Exercise Stopping Session Enjoyment & Difficulty Level Enquiry [updated] ExerClass Session Recording Next Session Adaptation [updated] Patient s Emotions Recognition and Recording ExerClass Duration Customization [updated] H_UC05 H_UC06 H_UC07 H_UC08 H_UC09 H_UC10 H_UC12 H_UC13 H_UC14 H_UC15 H_UC16 H_UC17 H_UC18 H_UC26 H_UC27 Home Exercising - ExerGame ExerGame H_UC19 Home Exercising - Exercise Practice Home Exercising - Active Lifestyle Activity ExerGame Instructions ExerGame Score Visualization Real-Time ExerGame Adaptation ExerGame Session Recording H_UC20 H_UC28 H_UC21 H_UC22 Other involved use cases are the same as for ExerClass: [H_UC04, H_UC05, H_UC06, H_UC10, H_UC13, H_UC14, H_UC15, H_UC16, H_UC18] Practice Individual Exercise Exercise Correctness Feedback (NT) Real-Time Practice Adaptation (NT) H_UC23 H_UC35 H_UC36 Other involved use cases are the same as for ExerClass: [H_UC04, H_UC09] Monitored Activity Data Acquisition [updated] Active Lifestyle Session Recording [updated] Not monitored Activity Data Acquisition [NEW] H_UC46 H_UC47 H_UC58 Usage Reporting Exercising Progress Visualization [updated] H_UC24 Behavioural Change Progress Visualization [updated] Periodic Notification for Exercising Engagement Periodic Behavioural Change Progress Notification Weekly Inactive Status Notification Next Session Notification Low Engagement Level Notification [NEW] Assessment Reminder Notification (NT) [NEW] H_UC32 H_UC25 H_UC33 H_UC34 H_UC45 H_UC69 H_UC70 Assessment Initial Patient Assessment [updated] H_UC29 ` 15 30/05/2017

16 Behavioural Change Readiness Assessment [updated] Exercise Programme Selection [updated] Behavioural Change Goal Progress Logging [updated] Periodic Patient Assessment Health and Fitness Assessment Health and Fitness Assessment Recording Health and Fitness Assessment Pre-Screening H_UC53 H_UC37 H_UC31 H_UC52 H_UC54 H_UC57 H_UC56 Behavioural Change Behavioural Change Goal Management H_UC30 Available Goals Visualization Good Habits Visualization Recommended Behavioural Goals Visualization (NT) Add Behavioural Change Personal Goal (NT) Remove Behavioural Change Personal Goal (NT) Personal Behavioural Goals Visualization (NT) H_UC38 H_UC39 H_UC40 H_UC41 H_UC42 H_UC44 Preference Settings Exercises Preferences Settings [updated] H_UC48 Home Exercising - Multi User Communication Useful Resources Access & Administration Calendar and Events Management Behavioural Change Notification Preferences Settings Exercising Notification Preferences Settings Class Event Reminder Notification [NEW] Chat Room Access [NEW] Social Interaction Vocal Analysis [NEW] Healthy Lifestyle Resources Visualization [NEW] Healthy Lifestyle Resources Management [NEW] Class Event creation [NEW] Class Event Invitation Acceptance [NEW] Class Event Reminder Notification Sending [NEW] Friend List Management [NEW] Table 1. PATHway Home environment use cases Feature Use Case Id H_UC43 H_UC11 H_UC66 H_UC67 H_UC68 H_UC64 H_UC65 H_UC60 H_UC61 H_UC62 H_UC63 Clinical Platform Access Clinical Platform Access C_UC01 Usage Reporting Patient Data Checking C_UC02 CRP Management Cardiac Rehabilitation Programme Definition C_UC03 Exercise Programme Creation C_UC04 Exercise Programme Editing C_UC05 Patient Management Add New Patient C_UC07 Remove Patient Edit Patient Patient List Visualization (NT) Patient Lifestyle Assessment C_UC08 C_UC09 C_UC10 C_UC14 ExerClass Exercises Administration Available Exercises Visualization (NT) C_UC15 Remove ExerClass Exercise C_UC12 ` 16 30/05/2017

17 Add New ExerClass Exercise Edit ExerClass Exercise Table 2. PATHway Clinical Environment Use Cases C_UC13 C_UC Project Plan The PATHway project plan envisioned an iterative development cycle that allowed close engagement between the technical and clinical partners. As per the plan of work, WP2 served to formalize the needs and requirements of the end users that later drove the development processes of the separate components of the system in WP3. Further WP4 envisioned a number of activities starting from architectural design of the whole system, continuing with development and integration and finally the validation of the system. The latter reflects the current stage of the project and this deliverable presents the validation process of the PATHway platform. It then will be deployed in hospital and home settings as part of WP5 activities therefore starting the human intervention studies that are going to be conducted in Ireland and Belgium Gap Analysis GAP analysis is used to determine the differences in the performance of a software application with intended usage of the system and, if not, what steps should be taken to ensure they are met successfully. Gap refers to the space between "where we are" (the present state) and "where we want to be" (the target state). A GAP analysis may also be referred to as a needs analysis, needs assessment or need-gap analysis. In the context of PATHway, GAP analysis can be seen as a comparison between the actual performance of the prototype and potential performance of the full system close to the so-called commercial product. It helps to establish a foundation for measuring investment of additional time and human resources that are required to achieve the outcome. Current State Future State The Gap Bridging the Gap Currently the system components are developed and integrated into a full system prototype and have been tested (integration, acceptance and usability tests) The system is ready for the upcoming randomized control trials and is set up to be able to support the end users and clinical team to provide with rapid updates, instructions and maintenance procedures described in section 7. Further tests are going to be The gap currently is due to the nature of the research project and lack of time, more long-term and large scale testing of the system needs to be performed to evaluate operation in different environments, as well as to see the Further testing of the system and integration of the components will bridge the gap. The re-evaluation of the requirements and feedback received from RCT will aid in this process. ` 17 30/05/2017

18 performed in parallel long term with RCT performance of the external services and rules, therefore optimal performance has not been evaluated to full extent. Table 3. PATHway GAP analysis 4. Software Test Specifications & Cases The main purpose of the system test specification is to appropriately manage the risks and maximize leverage with the end users. Therefore a validation plan for the system as whole has been defined. Its purpose was to set the activities, procedures as well as number of responsibilities among the component owners for establishing the adequacy of PATHway system. This plan (D4.2) [3] has been approved by the consortium members prior to commencing all the formal acceptance and usability procedures and served as a guide for continuous validation procedures. The plan defined the following: What activities needed to be performed during the validation process How the validation is going to be performed and which partner is responsible for the process What is the expected outcome of such validation How it will be maintained for the subsequent releases of PATHway system. Since the PATHway components are part of a larger and much more complex system, the validation of separate application and functionalities has been defined to include evidences that the requirements have been implemented correctly and are traceable to the system requirements. Figure 2. Involved concepts and their relationships from D4.2 ` 18 30/05/2017

19 In order to achieve the described objectives and ensure that validation and testing procedures convert the system specification as intended by the use cases, the consortium adopted a three-step process generating test cases, as outlined in D4.2 [3]: i) Generating a full set of use-case scenarios for each use case ii) Identification of at least one test case and the conditions under which it will be executed iii) For each test case identification of the data values with which the testing should be realized Software testing envisions running PATHway under known conditions with already defined inputs and well documented outcomes that later are compared to the expectations. These identify the schedule, environment, needed resources and methodologies, documentation Test Result Recording Since several different tests will be run on the system, different methods for result recording are used for each test. Below there is the recording method based on type and nature of the test Recording Integration Tests Integration tests validate that all the PATHway components, when functionally linked together, interact perfectly. Based on each use case a dependency chart was developed for the components to analyse the control flow and discover all interacting components and for each dependency a bottom-up approach with respect to control flow was highlighted and a test case prepared. Following the dependency chart, a method communication validation was done and checked by the call checkers to assess the possibility to reach the component method and compliance of the returned information to the expected information model. For recording the test results a Google excel sheet was developed which contained columns with the list of all components, related programmatic interfaces and methods, release due date, the use case identifiers, the implementation progress, Method call check to monitor the integration status of a method, monitoring dashboard and a final column for issues reporting Recording Acceptance Tests The Acceptance test considers the whole PATHway as a single unit. A three-step process is adopted for generating test cases. For recording the test cases, different scenarios are developed and pre conditions are set for testing. Based on the referred scenario, steps are ` 19 30/05/2017

20 developed for each scenario. For each step, input and expected result is reported as well as the end result of the test Recording Usability Test For recording, the usability questionnaires are distributed to a set of testers. A template is developed for recording the questionnaire results 4.2. Test Exception Handling Exceptions are events due to which a system ends abruptly without giving expected output. During internal testing procedures, exceptions are noted at each stage and handled by the partner responsible for the component or Test Case. Should an unexpected event occur, all the information about the behaviour of the application is stored locally in a log-file. All the participants have been informed that such log files must be sent to the component owner together with a detailed report on the time, stage and action following which the event has occurred. 5. Testing results Testing of the demonstrators must show that all the requirements and specification of the system are met and has been performed in planned phases. To better structure testing procedures a detailed plan was followed with clear responsibilities for creating, checking, approving and executing the test protocols as well as documentation of the errors and deviation as described in the following chapters Prototype P1: M14_M Integration testing During the Integration Test phase, all the involved partners tried to communicate with other components in order to check the communication and the correctness of the data transmitted. During that phase, the methods sheet of the Cockpit document (a centralized online tool that enables PATHway partners to see the dependencies between use cases, methods and components with development progress tracking capabilities) was the central point for reporting the advance both for the implementer and for the Call Checker (i.e. the client for the specific remote method). In the figures below, two examples in different timestamps are given: in the first one it is possible to see the Implementation Progress percentages with different values. From 0%- 79% means that the implementer is working on it. From 80% the Method Call Checker ` 20 30/05/2017

21 should test the method, and, if everything worked as expected, changes the status of the Method Call Check as Completed, otherwise the Call Checker should open an issue related to that particular method (see Issues management). Figure 3. Integration activity execution These two figures highlight also the dynamic messages (see column Monitoring Dashboard ) shown to the users for the different percentages; such messages are calculated upon an evaluation of the Due Date and the current date. In Figure 3. Integration activity execution it is possible to see a screenshot of the methods sheet just before the release of the first prototype. As shown at that time the Implementation Progress of all the methods was at 100% and the call checkers completed the integration check by setting the Method Call Check status to Completed for all the method which was the mandatory step before releasing and validating the prototype with Acceptance and Usability test (see Acceptance & Usability Testing on P1: M18-M22). ` 21 30/05/2017

22 Figure 4. Integration status at M18 In order to track the implementation activity and improve the general quality of the development process, it has been exploited the project issue tracker (based on the Redmine open source tool) to track/manage the issues identified during the integration steps. In Figure 5. All issues view a screenshot showing the final list of the issues at the end of the first prototype developing period (M18). Figure 5. All issues view The Redmine tool allows the definition of workflows to follow for upgrading the status of the issues. For example, an author of an issue, once they have completed all the needed fields, has to put the status as New, and the assignee is now able to determine to accept the issue, reject it or ask for some more details. ` 22 30/05/2017

23 From the New status it is possible to move towards the InProgress status, and then to all the other possible choices. The summary of the workflow is in Figure 6. PATHway workflow management. Figure 6. PATHway workflow management All the issues were resolved, and 22 of the 33 were also tagged as Closed for the first prototype. Most of those issues are related mainly to: a misunderstanding among the partners about the data to exchange, and the process of actual translation from the logic design signature to the concrete REST API and model. In the following figures, an example is given of realignment to the designed and agreed Data Model (Issue #38) and some difficulties for retrieving data from the REST API (Issue #26) ` 23 30/05/2017

24 Figure 7. PATHway issue tracker example ` 24 30/05/2017

25 Figure 8. PATHway issue history The process of issue tracking and management, is related to the other tool provided, the Cockpit, already presented in the D4.2 [3] A typical flow is present in the following images. ` 25 30/05/2017

26 ` 26 30/05/2017

27 Figure 9. Issue tracking and management via PATHway cockpit ` 27 30/05/2017

28 Acceptance Testing Acceptance tests serve to represent a certain expected result from the system or the application. In business or industrial projects, customers are usually responsible for verification of the correctness of the acceptance test. Within the PATHway context clinical partners can be viewed as the customers who verify, review the testing results and then make a decision on the priority of the failed tests. One might see the acceptance testing as regression tests as well. Apart from initially agreed testing procedures, several other approaches have been applied, namely: Internal Acceptance Testing is performed within the organization that developed the PATHway component (e.g. DCU) by those members who are not directly involved in the PATHway project and not part of the development team. External Acceptance Testing these are the tests that are performed by persons who are not affiliated to the organization that developed the software and in the case of PATHway it is the customers (clinical partners, end-user). The latter is presented in this section and the following steps were agreed to perform when running and performing the test: - acceptance testing - Each application owner has to report results of acceptance testing and comments gathered during the execution. - usability testing result (different degree of grain in terms of graphics) - Clinicians have to provide back the filled usability test excel template (for each patient) from which summary tables and charts will be extracted and reported here. - Conclusions about the main actions / feedback coming from the usability and acceptance campaign for the second prototype Exercising Application The accepting testing for the first prototype took place via Skype with clinical partners to ensure that all the captured requirements are covered in the integrated version of the first prototype. The acceptance testing used the template provided and specified within D4.2 and took place on with at least one representative from each of the clinical sides (DCU and KUL) and one from the technical side (application owner NG) (Table 4) ` 28 30/05/2017

29 Use Case Test Case Passed Not Passed Tester Names Execution Date [dd/mm/yyyy] Comments if Not Passed please provide details H_UC01 TC1 x Deirdre Walsh, Jomme Claes, Andrew Pomazanskyi H_UC01 TC2 H_UC02 TC1 x Deirdre Walsh, Jomme Claes, Andrew Pomazanskyi H_UC02 TC2 H_UC03 TC1 x Deirdre Walsh, Jomme Claes, Andrew Pomazanskyi H_UC03 TC2 H_UC03 TC3 H_UC04 TC1 x Deirdre Walsh, Jomme Claes, Andrew Pomazanskyi H_UC04 TC2 H_UC05 TC1 x Deirdre Walsh, Jomme Claes, Andrew Pomazanskyi H_UC05 TC2 H_UC06 TC1 x Deirdre Walsh, Jomme Claes, Andrew Pomazanskyi H_UC06 TC2 H_UC07 TC1 x Deirdre Walsh, Jomme Claes, Andrew Pomazanskyi H_UC08 TC1 H_UC09 TC1 x H_UC10 TC1 x Deirdre Walsh, Jomme Claes, Andrew Pomazanskyi Deirdre Walsh, Jomme Claes, Andrew Pomazanskyi H_UC11 TC1 x ` 29 30/05/2017

30 H_UC12 TC1 x Deirdre Walsh, Jomme Claes, Andrew Pomazanskyi Table 4. Acceptance test for exercise application P During the execution of the exerclass session DSS kept proposing the same exercise after third or fourth adaptation(fixed on ) After performing this exercise, the results are shared among the participants to summarize the results and cross-check the reported behaviour of the application. During the acceptance testing of the first prototype 34 Use-cases were tested and validated, comparing the results to the initial user requirements. As we can see from the Table 4, while testing the H_UC12 and related H_UC18 a small bug with the implementation of the DSS was identified where after a view adaptations it kept proposing the same exercise. Although the logic behind the implementation of this requirement was understood correctly and captured appropriately, an error has been identified and fixed in a timely manner. The UC was retested 3 days later with the issue resolved. All other use cases reported for the prototype 1 have been reported as Passed Usage Reporting & PA data Acquisition Front End The acceptance test execution has been carried out using an excel sheet as an operative instrument shared among teams. This sheet uses the same labels used in the D4.2 [3]for the columns UseCase and TestCase. Each component owner had to perform the test, together with the clinicians, for the actual acceptance of the implemented functionality (i.e. UseCase) reported in the requirements document (D2.3) [1]. For each test case, a test session was performed by the tester team reported in the column TesterNames, in the date present in the ExecutionDate column. The use cases were tested following all the related test cases, and for each of them it was reported if it Passed or Not. In case of comment from the clinical team, a note was kept in the Comments column. Use Case Test Case Passed Not Passed Tester Names Execution Date Comments H_UC24 TC1 x Deirdre, Ciara, Jomme, 26/10/2016 to show the FITT always in the DASHBOARD [like the card in the PhysicalActivity] ` 30 30/05/2017

31 Fabio H_UC31 TC1 x Deirdre, Ciara, Jomme, Fabio H_UC32 TC1 x Deirdre, Ciara, Jomme, Fabio H_UC44 TC1 x Deirdre, Ciara, Jomme, Fabio H_UC46 TC1 x Deirdre, Ciara, Jomme, Fabio H_UC46 TC2 x Deirdre, Ciara, Jomme, Fabio H_UC46 TC3 x Deirdre, Ciara, Jomme, Fabio H_UC47 TC1 x Deirdre, Ciara, Jomme, Fabio H_UC47 TC2 x Deirdre, Ciara, Jomme, Fabio 26/10/ /10/ /10/ /10/2016 MS Monitored session: to edit the content for explaining better the meaning of "monitore" [Deirdre should communicate the new content] 26/10/ /10/ /10/ /10/2016 H_UC58 TC1 x Deirdre, Ciara, Jomme, Fabio 26/10/2016 Table 5. Acceptance test for Usage Reporting Frontend P1 In order to monitor the advancement of the requirements testing (requirements testing coverage) in a synthetic manner the Cockpit requirement sheet was coherently updated as well (see Figure 10): ` 31 30/05/2017

32 Figure 10. Lifestyle Dashboard completion update P1 For the Usage and Reporting FrontEnd, all the test cases passed (10/10) but there was some comments about the layout of the web applications, which needed some quick fix or improvement. The use cases were marked as completed in the Cockpit, and the comments were addressed before the patients usability test sessions beginning Clinical & Research Front End Acceptance testing of the Clinical Use Cases of the first prototype started on December 7 th and was executed by Deirdre Walsh from DCU. After this initial testing a bug emerged that interfered with the acceptance testing of from KUL on December 8 th. After the removal of this bug, data and software versions being not in sync, the following overview of test results can be given (as presented in Table 6): Use Case Test Case Passed Not Passed Tester Names Execution Date [dd/mm/yyyy] C_UC01 TC1 x Deirdre Walsh, 07/12/ /12/2016 Comments if Not Passed please provide details C_UC01 TC2 x Deirdre Walsh, C_UC02 TC1 x Deirdre Walsh, C_UC03 TC1 x Deirdre Walsh, 07/12/ /12/ /12/ /12/ /12/ /12/2016 Assessed using whole of HDMS including Dashboard. Behaviour Change UI update (Paul0004 as example) needs to be fully implemented. Is data under CR Progress tab redundant with Dashboard? OK, but no confirmation message after an entry ` 32 30/05/2017

33 C_UC03 TC2 Deirdre Walsh, C_UC04 TC1 x Deirdre Walsh, C_UC05 TC1 x Deirdre Walsh, C_UC06 TC1 x Deirdre Walsh, C_UC06 TC2 Deirdre Walsh, C_UC07 TC1 x Deirdre Walsh, C_UC07 TC2 x Deirdre Walsh, C_UC08 TC1 x Deirdre Walsh, C_UC08 TC2 Deirdre Walsh, C_UC09 TC1 x Deirdre Walsh, C_UC09 TC2 Deirdre Walsh, C_UC10 TC1 x Deirdre Walsh, C_UC12 TC1 x Deirdre Walsh, C_UC12 TC2 Deirdre Walsh, C_UC13 TC1 x Deirdre Walsh, C_UC14 TC2 x Deirdre Walsh, C_UC15 TC2 x Deirdre Walsh, 07/12/ /12/ /12/ /12/ /12/ /12/ /12/ /12/ /12/ /12/ /12/ /12/ /12/ /12/ /12/ /12/ /12/ /12/ /12/ /12/ /12/ /12/ /12/ /12/ /12/ /12/ /12/ /12/ /12/ /12/ /12/ /12/ /12/ /12/2016 Can only be modified in Excel of Paul. Full implementation on new https supporting server for deployment. OK, but no message + not 'shown as flagged', but just deleted OK, but no message shown after an edit Can only be modified in Excel of Paul. Full implementation on new https supporting server for deployment. Can only be modified in Excel of Paul. Full implementation on new https supporting server for deployment. Can only be modified in Excel of Paul. Full ` 33 30/05/2017

34 Table 6. Acceptance testing for Clinical Frontend P1 implementation on new https supporting server for deployment. The comments made were dealt with on the new https supportive infrastructure for the second prototype. To monitor progress an intermediate acceptance test was executed on March 31 st by Jomme from KUL. All Clinical Use Cases to support Supervisors were accepted, except for those related to the management of Exercises to support the Administrator (C_UC s 6, 12, 13 and 15) Usability Testing The execution of usability testing was carried out by exploiting the template reported in D4.2 [2]. In order to speed up the process, templates were distributed to the patient, as an excel file, by the clinical teams Figure 11. Usability questionair template ` 34 30/05/2017

35 Figure 12. Compilation Guide For each feature, present in separate rows, and for each usability dimension, present in separate columns, there were several questions. For each question the patient could choose among a set of predefined answers. Based on those answers the sheet computes the percentage of success of the specific feature for the whole dimension. All those percentages were aggregated in the analysis sheet of the results (radar charts below). Each feature involves several software components and use cases; for that reason a Compilation Guide sheet was provided, in which the explanation of the features were broken down into Behaviours. Each behaviour was labelled to help the tester to report (in the comment) what behaviour there were issues with or unexpected errors. ` 35 30/05/2017

36 Figure 13. Radar Chart of usability feedback for Home Platform Access Figure 14. Radar Chart of usability feedback for ExerClass ` 36 30/05/2017

37 Figure 15. Radar Chart of usability feedback for ExerGame Figure 16. Radar Chart of usability feedback for Exercise Practice ` 37 30/05/2017

38 Figure 17. Radar Chart of usability feedback for Preference Setting Figure 18. Radar Chart of usability feedback for Active Lifestyle ` 38 30/05/2017

39 Figure 19. Radar Chart of usability feedback for Usage Reporting Figure 20. Radar Chart of usability feedback for Assessment ` 39 30/05/2017

40 Figure 21. Radar Chart of usability feedback for Behavioural Change Table 7 provides a summary of users feedback with respect to separate components of the system and even broken down to functionalities. The overall usability was rated very positive, although we can see that some of the functionalities were in the range between 60-70%, which means that certain features were still in development and the users could not engage with them in full or faced some errors. These results were then plotted in diagram (Figure 22) providing the overview on the average usability of the system. Effectiveness Efficiency Engaging Error Tolerance Ease of Learning Home Platform Access 98,2% 100,0% 100,0% 100,0% 100,0% Home Exercising - Single User ExerClass Home Exercising - ExerGame Home Exercising - Exercise Practice Home Exercising - Active Lifestyle Activ. 88% 99% 66% 93% 84% 85% 96% 64% 80% 87% 100% 98% 86% 97% 98% 60% 93% 85% 100% Usage Reporting 100% 100% 100% 100% ` 40 30/05/2017

41 Assessment 99% 92% 80% 63% 88% Behavioural Change 87% 88% 74% 100% 100% Preference Settings 100% 98% 93% 100% 97% Average 91% 96% 83% 90% 95% Table 7. Usability report per functionality Figure 22. Radar Chart usability feedback average Conclusions and Action Points for PATHway P1 In order to understand the rationale behind such results, the comments provided by the end users during the usability testing (see Figure 23) have been aggregated for a deeper analysis: ` 41 30/05/2017

42 Figure 23. User feedback analysis From such analysis a detailed APs list was identified and shared among the technical partners. It is worth mentioning that deeper analysis included the prioritization made by the technical team since the project has budget and time constraints. Hence, every feedback was evaluated and rated with respect to difficulty and effort. At the same time clinical partners have also provided feedback to state whether a given feedback is important for PATHway from clinical perspective in order to reach the ultimate objectives of the system. The final decision is then made in column Decision. ID Description Partner(s) Priority (clinicians) Priority - tech (1-10) Difficult to implement (1-10) Effort - time (1-10) Decision Remove which exercise band they.. Pre-screening once a day is enough No need to have verbal instruction When BP is asked for, the patient Need a variety of exercises for t NG ES NG, INAB ES ES NO, unless there is time NG NO Maybe NG ES INAB, NG ES ES depends on item#3 6 Next to using the INAB ES ES ` 42 30/05/2017

43 exercises label With the content we provided, Different exercises for frog game So all exercises with the feature We need to test this with patient Most found it difficult to unders Avatar coach which appears in the Participants found it difficult t INAB NO ES NG NO ES NG NO ES NG NO ES NG ES NO NG ES ES NG; ITI NO ES 14 The time setting NG, INAB ES ES Participants want some music Quality of exercise demonstration NG; ITI NO NO ITI NO ES 17 The exercises already implemented ITI; NG, INAB ES ES The system often picked up rep co (Major Health and Safety Issue). ITI ES ES ERS ES Es 20 Heart rate was not always correct DCU, INAB? ES NO 21 A variety of exercises using diff INAB ES ES 22 Speed of the exercise NG; ITI ES DONE ` 43 30/05/2017

44 execution b Remove nice job, great job after Some would like the option See separate feedback document.. When trying to access the dashboard Because of the above and the fact When logging outdoor activities All steps written out in the screen When executing the step exercise NG; ITI ES ES NG NO NO NG NO ES ENG NO ES ERS NO ENG, ERS ES ES NG ES ES ITI NO ES 31 The -HDMS COMMUNICATION TIME OUT ERS ES DONE (to be tested again) 32 When trying to access the lifestyle ENG ES MS Band did not always connect to When clicking on 'exit' button DCU ES ES NG ES ES 5.2. Prototype P2: M22-M27 Table 8. Prototype 2 user feedback prioritization Integration testing Following, with the integration testing for the prototype P2, the PATHway consortium followed a similar approach to the one described and adopted for Prototype P1 in section 5.1. Thus a PATHway Implementation Cockpit was further updated with the newly defined ` 44 30/05/2017

45 (or refined, based on clinical and end-user feedback) requirements and methods that the PATHway platform was required to support, where the call checker for a respective methods, as well as implementation due date, was newly defined as depicted in Figure 24: Figure 24. Implementation Cockpit P2 In addition to the already described methods, during the integration testing an agile software methodology was adopted in the form of weekly sprints. In product development, a sprint is a set period of time during which specific work has to be completed and made ready for review. Each sprint begins with a planning meeting. During the meeting, the product owner (the person requesting the work) and the development team agree upon exactly what work will be accomplished during the sprint. The development team has the final say when it comes to determining how much work can realistically be accomplished during the sprint, and the product owner has the final say on what criteria needs to be met for the work to be approved and accepted ` 45 30/05/2017

46 Figure 25. Integration with agile software development methodology Acceptance Testing - acceptance testing - Each application owner has to report results of acceptance testing and comments gathered during the execution. - usability testing result (different degree of grain in terms of graphics) - Clinicians have to provide back the filled usability test excel template (for each patient) from which summary tables and charts will be extracted and reported here. - Conclusions about the main actions / feedback coming from the usability and acceptance campaign for the second prototype Exercising Application As reported earlier, the same methodology for the acceptance testing of the second prototype P2 has been adopted. This time the tests were run independently by each of the clinical teams providing separate acceptance testing reports, where additional features and newly specified requirement stated in section 3 have been validated. The reports have then been combined in one single table for better presentation purposes: Use Case Test Case Passed Not Passed Tester Names Execution Date [dd/mm/yyyy] Comments if Not Passed please provide details ` 46 30/05/2017

47 H_UC01 TC1 x Deirdre Walsh, Ciara McCormack, Haolin Wei, Jomme Claes, H_UC01 TC2 H_UC02 TC1 x Deirdre Walsh, Ciara McCormack, Haolin Wei, Jomme Claes, H_UC02 TC2 H_UC03 TC1 x H_UC03 TC2 x Deirdre Walsh, Ciara McCormack, Haolin Wei, Jomme Claes, DW, CMC, Jomme Claes, Roselien Buys When logging in for the first time to the single user exerclass, we have been brought straight to the zensor instructions, 5minute wait. Also after the GHV, went straight into the exerclass, should go to home screen and then participant can select class. Went back out at accepted notification, this then created an error when we tried to enter the exerclass 'HDMS returned an invalid programme or it has not been set up. Please contact your supervisor'. Haolin fixed it so we could move forward with testing but needs to be fixed. Need to discuss the 'change ABOVE ISSUE fixed H_UC03 TC3 H_UC04 TC1 x Deirdre Walsh, Ciara McCormack, Haolin Wei, Jomme Claes, ` 47 30/05/2017

48 H_UC04 TC2 H_UC05 TC1 x Deirdre Walsh, Ciara McCormack, Haolin Wei, Jomme Claes, At first log in the zensor instructions came up here, not the Msband instructions. [FIXED] H_UC05 TC2 H_UC06 TC1 x Deirdre Walsh, Ciara McCormack, Haolin Wei, Jomme Claes, Need to say when in the 3rd step of blood pressure instruction that 'wait until the measurement shows on the screen'.zensor instructions shown again here-can remove[fixed] H_UC06 TC2 H_UC07 TC1 x Deirdre Walsh, Ciara McCormack, Haolin Wei, Jomme Claes, H_UC08 TC1 H_UC09 TC1 x H_UC10 TC1 x Deirdre Walsh, Ciara McCormack, Haolin Wei, Jomme Claes, Deirdre Walsh, Ciara McCormack, Haolin Wei, Jomme Claes, H_UC11 TC1 x H_UC12 TC1 x Deirdre Walsh, Ciara McCormack, Haolin Wei, Jomme Claes, During the execution of the exerclass session DSS kept proposing the same exercise after third or fourth adaptation (fixed on ) Deirdre Walsh, H_UC13 TC1 x Ciara McCormack, Haolin Wei, ` 48 30/05/2017

49 Jomme Claes, H_UC14 TC1 x H_UC15 TC1 x H_UC16 TC1 x Deirdre Walsh, Ciara McCormack, Haolin Wei, Jomme Claes, Deirdre Walsh, Ciara McCormack, Haolin Wei, Jomme Claes, Deirdre Walsh, Ciara McCormack, Haolin Wei, Jomme Claes, H_UC17 TC1 x H_UC18 TC1 x H_UC19 TC1 x H_UC20 TC1 x H_UC21 TC1 x Deirdre Walsh, Ciara McCormack, Haolin Wei, Jomme Claes, Deirdre Walsh, Ciara McCormack, Haolin Wei, Jomme Claes, At times the frog jumps when there is no movement at all and the participant is stationary, repetitions are remaining at 0 even though the movement is being captured. Need to add clock in game so that the overall time can be seen by the participant of the time they have left. [CLOCK ISSUE RESOVLED; FROG JUMPING ISSUE ON-GOING] Only certain exercises are used anyway [FIXED] ` 49 30/05/2017

50 H_UC22 TC1 x Deirdre Walsh, Ciara McCormack, Haolin Wei, Jomme Claes, H_UC22 TC2 H_UC23 TC1 x Deirdre Walsh, Ciara McCormack, Haolin Wei, Jomme Claes, H_UC24 TC1 x H_UC27 TC1 x H_UC28 TC1 x Deirdre Walsh, Ciara McCormack, Haolin Wei, Jomme Claes, Deirdre Walsh, Ciara McCormack, Haolin Wei, Jomme Claes, H_UC29 TC1 x H_UC30 TC1 x H_UC31 TC1 x H_UC32 TC1 x H_UC33 TC1 Deirdre Walsh, Ciara McCormack, Haolin Wei, Jomme Claes, Deirdre Walsh, Ciara McCormack, Haolin Wei,Jomme Claes, Roselien Buys Deirdre Walsh, Ciara McCormack, Haolin Wei Accessing the dashboard did not work for user 3073 [screenshot attached], tried user 3053 and it worked however. [08/05/17:FIXED] calendar option visible on the dashboard in the My healthy lifestyle section Separate testing for notification H_UC34 TC1 Separate testing for notification ` 50 30/05/2017

51 H_UC35 TC1 x H_UC36 TC1 x H_UC37 TC1 x DW,CMC, Jomme Claes, Roselien Buys DW,CMC, Jomme Claes, Roselien Buys FIXED FIXED H_UC38 TC1 x H_UC39 TC1 x H_UC40 TC1 x H_UC41 TC1 x H_UC42 TC1 x H_UC43 TC1 x H_UC44 TC1 x Deirdre Walsh, Ciara McCormack, Haolin Wei Deirdre Walsh, Ciara McCormack, Haolin Wei, Jomme Claes, Deirdre Walsh, Ciara McCormack, Haolin Wei, Jomme Claes, Deirdre Walsh, Ciara McCormack, Haolin Wei, Jomme Claes, Deirdre Walsh, Ciara McCormack, Haolin Wei, Jomme Claes, Deirdre Walsh, Ciara McCormack, Haolin Wei, Jomme Claes, Deirdre Walsh, Ciara McCormack, Haolin Wei, Jomme Claes, [FIXED] When added goals on the PATHway system we were then not able to log into it on Dashboard in the my healthy lifestyle section, error message appeared, (see attached error message) H_UC53 TC1 x ` 51 30/05/2017

52 H_UC55 TC1 H_UC56 TC1 x H_UC57 TC1 x Deirdre Walsh, Ciara McCormack, Haolin Wei, Jomme Claes, Deirdre Walsh, Ciara McCormack, Haolin Wei, Jomme Claes, Table 9. Exercise Application acceptance testing P2 The above table provides a summary of the use cases defined for the second and final prototype as well as the results of the performed acceptance tests. As we can see in some of the cases, during some earlier iterations and testing procedures, several use cases did not pass; therefore second testing was initiated when the issue was resolved. All in all, all the UC as they were specified in the requirement document for the exercise application were accepted, although it is important to note, additional feedback to certain functionalities has been provided and the actions will be taken described in this document in section 6. Since the acceptance tests were performed successfully a respective action has been taken to also update the PATHway implementation Cockpit as presented below: ` 52 30/05/2017

53 Usage Reporting & PA data Acquisition Front End Following similar procedure as during the P1 testing, the acceptance test execution was carried out using an excel sheet as operative instrument shared among teams. This sheet uses the same labels used in the D4.2 [3] for the columns UseCase and TestCase. Each component owner had to perform the test, together with the clinicians, for the actual acceptance of the implemented functionality (i.e. UseCase) reported in the requirements document [1]. For each test case, a test session was performed by the tester team reported in the column TesterNames, in the date present in the ExecutionDate column. The use cases have been tested following all the related test cases, and for each of them was reported if it Passed or Not. In case of comment from the clinical team, a note was kept in the Comments column. Use Case Test Case Passed Not Passed Tester Names Execution Date Comments H_UC24 TC1 x Deirdre, Jomme, Fabio H_UC31 TC1 x Deirdre, Jomme, Fabio H_UC32 TC1 x Deirdre, Jomme, Fabio H_UC44 TC1 x Deirdre, Jomme, Fabio H_UC46 TC1 x Deirdre, Jomme, Fabio H_UC46 TC2 x Deirdre, Jomme, Fabio H_UC46 TC3 x Deirdre, Jomme, Fabio H_UC47 TC1 x Deirdre, Jomme, Fabio H_UC47 TC2 x Deirdre, Jomme, Fabio H_UC58 TC1 x Deirdre, Jomme, Fabio 20/02/ /02/ /02/ /02/ /02/ /02/ /02/ /02/ /02/ /02/2017 ` 53 30/05/2017

54 H_UC64 TC1 x Deirdre, Jomme, Fabio H_UC65 TC1 x Deirdre, Jomme, Fabio 20/02/ /02/2017 Figure 26. Usage and Reporting FE acceptance testing P2 In order to monitor the advancement of the requirements testing (requirements testing coverage) in a synthetic manner, the Cockpit requirement sheet was coherently updated as well (see figure below): Figure 27. Complition of UC testing for FE in cockpit For the Usage and Reporting FrontEnd, all the test cases passed (12/12) without any comment. The use cases were marked as completed in the Cockpit Clinical & Research Front End On May 18 th 2017 KUL executed the acceptance testing of the second prototype on the https version of the Clinical and Research frontend (HDMS). In addition to that, the ENG private certificate was tested needed for the integration of the Usage Reporting & PA data Acquisition Front End in the HDMS while using https. During the testing it turned out that, when logged in as Administrator, a bug has been detected since the functionality to manage Messages was displayed instead of the functionality to manage Exercises (C_UC s 6, 12, 13 and 15), which was corrected. The test outcomes are presented below. Use Case Test Case Passed Not Passed Tester Names Execution Date [dd/mm/yyyy] C_UC01 TC1 x, Deirdre Walsh 18/05/2017 Comments if Not Passed please provide details C_UC01 TC2 x, Deirdre Walsh 18/05/2017 ` 54 30/05/2017

55 C_UC02 TC1 x, Deirdre Walsh C_UC03 TC1 x, Deirdre Walsh C_UC03 TC2, Deirdre Walsh C_UC04 TC1 x, Deirdre Walsh C_UC05 TC1 x, Deirdre Walsh C_UC06 TC1 x, Deirdre Walsh C_UC06 TC2, Deirdre Walsh C_UC07 TC1 x, Deirdre Walsh C_UC07 TC2 x, Deirdre Walsh C_UC08 TC1 x, Deirdre Walsh C_UC08 TC2, Deirdre Walsh C_UC09 TC1 x, Deirdre Walsh C_UC09 TC2, Deirdre Walsh 18/05/ /05/ /05/ /05/ /05/ /05/2017 Could not find were to edit an ExerClass Exercise, logged in as administrator. There is no "modify exercise" option? When clicking on "Manage", only BHC messages are shown. When trying to "add" an exercise under "Manage" I can only type numbers. 18/05/ /05/ /05/ /05/2017 OK, but no message + not 'shown as flagged', but just deleted 18/05/ /05/2017 OK, but no message shown 18/05/2017 ` 55 30/05/2017

56 C_UC10 TC1 x, Deirdre Walsh C_UC12 TC1 x, Deirdre Walsh C_UC12 TC2, Deirdre Walsh C_UC13 TC1 x, Deirdre Walsh C_UC14 TC2 x, Deirdre Walsh C_UC15 TC2 x, Deirdre Walsh R_UC01 TC2 x, Deirdre Walsh R_UC02 TC2 x, Deirdre Walsh 18/05/ /05/2017 Could not find were to edit an ExerClass Exercise, logged in as administrator. There is no "remove exercise" option? 18/05/ /05/2017 Could not find were to edit an ExerClass Exercise, logged in as administrator. There is no "add exercise" option? When clicking on "Manage", only BHC messages are shown. When trying to "add" an exercise under "Manage" I can only type numbers. 18/05/ /05/2017 Could not find were to edit an ExerClass Exercise, logged in as administrator. There is no "Exercise management" option? 18/05/2017 Able to download info 18/05/2017 Only filters present were: date fromto, include removed patients yes-no. No option to add more filters ` 56 30/05/2017

57 R_UC03 TC2 x, Deirdre Walsh R_UC04 TC2 x, Deirdre Walsh R_UC05 TC2 x, Deirdre Walsh 18/05/2017 Only.xml file automatically available. 18/05/ /05/2017 Table 10. Acceptance test results for Clinical & Research Frontend P2 The comment about the options to filter (R_UC02) needs further exploration, since the filters presented are in accordance with the examples shared with the partners during the Glasgow meeting on April 4 th but to better address the needs of the end-users further adjustments have to be made The export format in.xml (R_UC03) does not prevent opening the files in Excel as agreed/preferred format to be used by the researchers. The functionality to confirm entries made, has not been in place at the time of testing. Discussions about usability of the Frontend are an ongoing process since several different approaches have to be considered. Some prefer to have the data organized by visits (0, 3 and 6 months), whereas other clinicians want to edit the existing data when a patient has a follow up after 3 and 6 months (of course everything is logged into the system) and some clinicians want to confirm all data entries using a pop-up summary screen Notification (backend) Service The notification component depends on several other components mainly HDMS and DSS, as well as PATHway application offered by NG and the acceptance tests for such a service were introduced in second prototype only. Currently there are several testing cases based on different use cases: H_UC25 (Periodic Notification for Exercising Engagement), H_UC33 (Periodic Notification for the Behavioural Changing Engagement), H_UC34 (Weekly Inactive Status Notification) and Engagement Notification. All rules have been summarized in Annex A. For H_UC25, there are 10 rules. The message category is Physical Activity. There are 7 message types (Keep On Track, Booster, Advice, Arrange, Positive Reinforcement, WeeklyGoalNotAchieved and NonAdherent). For H_UC33, there are 8 rules, 5 for smoking and 3 for eating. There are no rules defined for stress, alcohol or medical. For smoking there are 5 message types (positive reinforcement, keep on track, booster, arrange and applaud). For eating there are 3 message types (booster, arrange and applaud). For H_UC34, there is 1 rule and the type is NonActive. For engagement, there is 1 rule and the type is Booster. Following are the steps to test the notification component: ` 57 30/05/2017

58 1. Creating a new patient and filling in the questionnaires, for different rules one can create different patients in order to test different rules simultaneously. For mobile phone number the country code needs to be included. As an example for Ireland, if the mobile phone is , the number you need input is Once a new patient have been created, the address will be automatically generated, the general format is pathway.lastnamexxxx@outlook.com, where x will be an number assigned based on the number existing patient with same first name start from User, who is setting up the system will need to go to and sign up a new outlook account using the address generated previously. 3. Use the newly created patient, log in to the PATHway system and select the behaviour change (BC). User can update the BC using goal management. 4. Set the notification preference for exercise and BC. 5. Exercise is based on different rules to generate different rules. Track the BC using the Lifestyle and Physical Reporting Frontend. Using the Google Spreadsheet track which rules have passed and which ones are not. The tester was able to report whether the use case has passed or not, and state the reason as presented in the Annex A. a. If a wrong message was sent, tester provided the date of the message, the message received, and an indication about the message that was supposed to be received and the patient ID. The same procedure had to be followed if no message was sent, providing the date, the message expected to receive and the patient ID Usability Testing KUL has performed usability testing with the PATHway build published on 07/03/2017. Participants were recruited amongst the friends and colleagues of the researchers. Eventually 10 healthy, young adults agreed to participate. Their "patient" profiles were set up in the HDMS a few days prior to the day of the usability testing. All values inserted were either based on information the participant gave us (age, gender, height, weight, etc.) or were completed as realistic as possible by the researcher based on the knowledge he had about the participant. On the day of the usability testing 4 PATHway platforms were running simultaneously in a room that had good natural and artificial lighting. Three other laptops were used to allow the participant to complete the HDMS questionnaires before starting with the PATHway system and to check the feedback from their session in the ENG dashboard. A schematic setup of the room is provided in Figure 27. ` 58 30/05/2017

59 DCU performed usability testing with the PATHway build published on 07/03/2017. PATHway was tested with the clinical team internally on the 10/03/2017. Further participants were recruited amongst MedEx for the 14/03/17 testing (n=4)). Five others were scheduled to take part of the 15/03/17, however these were cancelled due to repeated error screens on the first day of testing. The initial four "patient" profiles were set up in the HDMS prior to the usability testing. On the day of the usability testing one PATHway platform was running in a room that had artificial lighting. When participant came in, this was the flow of activities he/she had to complete: Read and sign an informed consent Complete the questionnaires in the HDMS Perform an Exerclass of 20 minutes and an Exergame of 10 minutes Check their feedback in the ENG dashboard Complete the System Usability Scale (SUS) and a form about the usability report of separate features of the system The ENG Excel file was completed by all participants insofar as possible given the elements of the platform which were functioning properly at the time. All participants completed the SUS online. Participants were told to report every error they encountered and these were written down by one of the researchers. The next pages give an overview of the encountered errors with pictures for clarification. ` 59 30/05/2017

60 Figure 28. Schematic setup of the room during PATHway usability testing The aggregated data about the user feedback about the usability of the system is presented in figure below, showing the overall improvement of the system for the second prototype. We do not provide a detailed analysis for each component as for the prototype 1 (depicted in chart diagrams) as this information is deemed meaningful and the improvement between the first prototype and the second one is a clear step forward for already fairly appraised system. Nevertheless, it is important to note, that the users had additional feedback for the development team on how to improve the system before the clinical trials start which were further evaluated, prioritized and, where required, rapid actions have been taken. ` 60 30/05/2017

61 Figure 29. Usability Testing P2 6. Conclusions and Action Points beyond the Prototype 2 The integration testing of prototype 2 showed that all the components developed in the WP3, and integrated as part of WP4 activities, are successfully tested by the component owners and call checkers, and fully integrated into the system. This statement is also in line with an aggregated view about the completion progress of the components within the PATHway implementation cockpit: ` 61 30/05/2017

62 Figure 30. Pathway Component Completion Although the official usability testing has been performed according to the plan set up by the DoA, as it is mentioned in the previous section, users had additional feedback to improve the system usability and encountered some errors. Technical teams carefully investigated the reports to provide the users with an improved version. In some cases it appeared that protected networks and slow Internet connection were sometimes the cause of errors. The overall goal of the integration team is to provide a stable flawless system beyond the prototype 2 and for this reason additional usability tests were performed with newer updated versions of PATHway system. Therefore additional tests were performed by KUL on 8/05/2017 and 12/05/2017 as well as DCU on 16/05/2017. The final report of DCU stated that they have encountered no errors within the system. KUL mentioned view errors but since the introduction of the error codes described in the Section 7, we can see that most of them were related to Internet connection (timeout errors). Additional feedback that was gathered does not reflect the system performance, rather provide additional improvements to graphical user interfaces and texts within the system (e.g. increase font size/colour/contrast). All the future changes and support that are required beyond the prototype 2 fall under maintenance and update activities described in the next section, whereas the integration and validation activities considered to be successfully performed and closed. ` 62 30/05/2017

63 7. Operations/Maintenance Once a PATHway system has been validated, it is important to be able to maintain its compliant state throughout its operational life. This can be achieved by several means: A well-documented process to overcome certain system errors that can be resolved either by the end user or the technical teams who were not involved in PATHway development The implementation of an appropriate maintenance system. For this reason, it was agreed to develop a number of supporting documentation that might help the clinical teams, end users and technical specialists to use the system appropriately and fix small errors that might occur without the involvement of PATHway technical support team. Namely the following documentation has been additionally produced: User Manual Guide, Troubleshooting Guide, FAQ, and Installation Guide. To appropriately operate the system, it is required that the user (both the end user and technical specialist) understands the core of the problem. For this reason, Error codes within the application were introduced (for example: Figure 31). Figure 31. PATHway Error Screen Here the following error code scheme has been implemented: <ErrorType>.<ComponentCode>.<MethodCode> and the tables below can indicate what this particular code stands for: ` 63 30/05/2017

64 Error Type Code Error Type Description 1 Default 2 Invalid data 3 Timeout 4 Service unavailable 5 Starting component failed 6 Invalid kinect data 7 Updater not found 8 Invalid cardiac rehabilitation programme 9 Invalid cpet test 10 Invalid patient details 11 No lifestyle assessments 12 Invalid phone number Table 11. Error Type Code Component Code Method Code 0 None 1 Calendar 2 Enjoyment recognition 3 Exercise evaluation 4 Health data management 5 Home decision support 6 Motion capture 7 Sensor home gateway 8 Vocal analysis 9 Updater Table 12. Component Code Component Description Method Description 0 None 1 AvailableExercises 2 BehaviouralChangeGoals 3 BehaviouralChangeProgramme 4 BehaviouralChangeQuestionnaire 5 CardiacRehabilitationProgress 6 CpetTest 7 DeliveredMessages 8 ExercisePreference 9 GetBehaviouralChangeNotificationPreferences 10 GetExerciseNotificationPreferences 11 GetPatientCRProgramme 12 HealthFitnessAssessmentPreScreeningQuestionnaire 13 LifestyleAssessments 14 PatientDetails 15 PatientLogin 16 PersonalizedParameters 17 PreScreeningQuestionnaire 18 PreviousPreScreeningInfos 19 SendBehaviouralChangeNotificationPreferences ` 64 30/05/2017

65 20 SendBehaviouralChangeProgramme 21 SendCardiacRehabilitationProgramme 22 SendDeliveredMessage 23 SendExerciseNotificationPreferences 24 SendExercisePreferences 25 SendExerClassSession 26 SendExerGameSession 27 SendHealthFitnessAssessmentSession 28 SendLifestyleAssessment 29 SendPatientStage 30 SendPersonalizedParameters 31 AfterSessionExcludedExercises 32 AfterSessionPersonalizedParameters 33 CustomExerciseProgrammes 34 GoodHabits 35 InitializeExerClassSession 36 NewExerciseProgramme 37 PatientStage 38 PreScreeningEvaluation 39 RealTimeEvaluation 40 RecommendedGoals 41 SessionEndAssessmentMessageAndAcceptability 42 TriggeredRules 43 HealthFitnessAssessmentEvaluation Table 13. Method Code 7.1. PATHway Updater All components will be updated if the necessity arises to fix bugs and prevent malfunctions of the system that would threaten the patients experience and motivation. Those located on the machine of the patient will be automatically updated using the PathwayUpdater. At every start of the PATHway application it connects to the update server to check if a new version is available. If so, the application will display a confirmation window to inform the user that a new version is available. After confirming the application will close and run the updater. ` 65 30/05/2017

66 Figure 32. PATHway Updater flow The updater connects to the update server using a System.Net.WebClient and starts downloading the zip-compressed archive containing the new version to a temporary folder. All information such as install location, server address and language (Language sent as parameter from the calling application will be prioritized) are read from a configuration file. The downloaded archive will then get unpacked and all contained files will be copied over to the application directory. Once the downloading and installing is finished the updater shows information to the user saying that the update was completed successfully before closing automatically. During all of these steps the progress of the recently performing step will be displayed to the user in the form of a progress bar (in case of the download additional information like download speed, file size and overall size are shown). Furthermore, all steps and errors that may occur are written to a log file. ` 66 30/05/2017

67 Figure 33. PATHway Updater UI ` 67 30/05/2017