Functional and Security testing of a Mobile Application

Transcription

1 Linköping University Department of Computer Science Bachelor thesis, 16 ECTS Information Technology 2017 LIU-IDA/LITH-EX-G--17/066--SE Functional and Security testing of a Mobile Application Funktionell och säkerhetstestning av en mobil applikation Johan Sjöstrand Sara Westberg Supervisor : Simin Nadjm-Tehrani Examiner : Nahid Shahmehri Linköpings universitet SE Linköping ,

2 Upphovsrätt Detta dokument hålls tillgängligt på Internet eller dess framtida ersättare under 25 år från publiceringsdatum under förutsättning att inga extraordinära omständigheter uppstår. Tillgång till dokumentet innebär tillstånd för var och en att läsa, ladda ner, skriva ut enstaka kopior för enskilt bruk och att använda det oförändrat för ickekommersiell forskning och för undervisning. Överföring av upphovsrätten vid en senare tidpunkt kan inte upphäva detta tillstånd. All annan användning av dokumentet kräver upphovsmannens medgivande. För att garantera äktheten, säkerheten och tillgängligheten finns lösningar av teknisk och administrativ art. Upphovsmannens ideella rätt innefattar rätt att bli nämnd som upphovsman i den omfattning som god sed kräver vid användning av dokumentet på ovan beskrivna sätt samt skydd mot att dokumentet ändras eller presenteras i sådan form eller i sådant sammanhang som är kränkande för upphovsmannenslitterära eller konstnärliga anseende eller egenart. För ytterligare information om Linköping University Electronic Press se förlagets hemsida Copyright The publishers will keep this document online on the Internet or its possible replacement for a period of 25 years starting from the date of publication barring exceptional circumstances. The online availability of the document implies permanent permission for anyone to read, to download, or to print out single copies for his/hers own use and to use it unchanged for non-commercial research and educational purpose. Subsequent transfers of copyright cannot revoke this permission. All other uses of the document are conditional upon the consent of the copyright owner. The publisher has taken technical and administrative measures to assure authenticity, security and accessibility. According to intellectual property law the author has the right to be mentioned when his/her work is accessed as described above and to be protected against infringement. For additional information about the Linköping University Electronic Press and its procedures for publication and for assurance of document integrity, please refer to its www home page: c Johan Sjöstrand Sara Westberg

3 Students in the 5 year Information Technology program complete a semester-long software development project during their sixth semester (third year). The project is completed in midsized groups, and the students implement a mobile application intended to be used in a multiactor setting, currently a search and rescue scenario. In parallel they study several topics relevant to the technical and ethical considerations in the project. The project culminates by demonstrating a working product and a written report documenting the results of the practical development process including requirements elicitation. During the final stage of the semester, students create small groups and specialise in one topic, resulting in a bachelor thesis. The current report represents the results obtained during this specialisation work. Hence, the thesis should be viewed as part of a larger body of work required to pass the semester, including the conditions and requirements for a bachelor thesis.

4 Abstract A mobile application has been developed to be used for assistance in crisis scenarios. To assure the application is dependable enough to be used in such scenarios, the application was put under test. This thesis investigates different approaches to functional testing and security testing. Five common methods of generating test cases for functional testing have been identified and four were applied on the application. The coverage achieved for each method was measured and compared. For this specific application under test, test cases from a method called decision table-testing scored the highest code coverage. 9 bugs related to functionality were identified. Fuzz testing is a simple security testing technique for efficiently finding security flaws, and was applied for security testing of our application. During the fuzz test, system security properties were breached. An unauthorized user could read and alter asset data, and it also affected the system s availability. Our overall conclusion was that with more time, creating functional tests for smaller components of the application might have been more effective in finding faults and achieving coverage.

5 Acknowledgments We would like to thank the members of our group - Sumia Al-Mufti, Rasmus Jönsson, Olav Nilsson and Filip Polbratt - for doing the excellent job of developing the system that has been put to test in this thesis together with us. Especially we want to thank our supervisor Simin Nadjm-Tehrani for the feedback and help during this project, and Mikael Asplund for the feedback during the start of the project. We also want to thank Rickard Hellenberg and Oskar Gustafsson for their opposition on this thesis. v

6 Contents Abstract Acknowledgments Contents List of Figures List of Tables iv v vi viii ix 1 Introduction Aim Problem statement Approach Delimitations Structure Software testing Software testing in general Android testing Functional testing Coverage Security testing Related work Test implementation The application under test Selected frameworks and approach Functional testing Security testing Results Functional testing Coverage reports Security testing Discussion Results Method The work in a wider context Conclusion 25 Bibliography 27 vi

7 A Appendix: Functional tests in Espresso - Source code 29 A.1 Authentication test A.2 Map test A.3 Contact test B Appendix: Fuzz test - Source code 42 B.1 Fuzz test

8 List of Figures 1.1 Application view over the map activity A pyramid of the testing levels Application view when creating a new pin Fuzz test flow chart Invalid database entries from the fuzzer viii

9 List of Tables 2.1 Decision table - Example Decision table - Map test, add pin Decision table - Map test, remove pin Identified test cases - Map test, add pin Identified test cases - Map test, remove pin Decision table - Authentication test Identified test cases - Authentication test Decision table - Call test Bug report from functional testing Coverage report - Whole application Coverage report - Map activity Coverage report - Authentication activity Coverage report - Service class Vulnerability report for the security testing ix

10 1 Introduction Smartphones are becoming more and more advanced and used in more advanced situations. Nowadays they are used not only in the entertainment sector but also in more critical domains. The increasing complexity makes testing mobile applications very hard [1], since the combination of possible inputs grows rapidly. Making sure that your application is dependable before pushing an update is not an easy task, but a necessary one. By testing the application s functionality, faults can be found and the application will become more dependable. However, there are many different testing methods to chose from. Therefore we will investigate how well different testing methods actually work. This will be done by performing tests on an application that has been developed alongside this thesis. The application under test (AUT) consists of an Android application, where the application depends on a server which is connected to a database. The purpose of the application is to be a mock up for a tool used by the Swedish Defence Forces for communication and information sharing during a crisis scenario. Figure 1.1 shows a view of the AUT, where pins are placed on a map, representing different tasks for the Swedish Defence Forces. There are many different ways to expose faults in the AUT, and there is no way of knowing when all of the faults have been found. But if the AUT passes tests that covers all of the source code with relevant variations of different input, it is a sign that most errors due to coding faults given input assumptions have been considered. Coverage is a metric that will be used in this thesis, which is often used in software testing and makes it possible to measure how much of the source code has been tested. A systematic way of performing automatic testing to cover all of the AUT s functionality needs to be identified. This is due to the low coverage that would be achieved when only end-user testing is used. This is seen in a study with 7- user testing of some popular apps that got a result of only 30% coverage of app screens and only 6% of the apps functionality [2], which is seen as a bad results. More advanced software also leads to more vulnerabilities in a system. These vulnerabilities can be exploited by an attacker who wants to get access to the system [3]. This type of attack is seen as a threat to the system since the system assets would be at risk, which calls for the need to identify and remove as many vulnerabilities as possible to increase the security of a system. 1

11 1.1. Aim Figure 1.1: Application view over the map activity 1.1 Aim In this thesis, we will test the AUT with respect to functionality and security. To test the functionality we will analyze different approaches of functional testing with the goal to gain higher coverage than 70%. When testing the security of the system we will focus on the assets and their exposure to breach of confidentiality and integrity. An unauthorized user should not be able to read or alter data. We will find and use approaches of constructing tests that make sure that the AUT is not compromising the confidentiality and integrity of the system. 1.2 Problem statement In this thesis we will: 1. Investigate how to construct functional test cases systematically to achieve a high enough coverage. 2. Identify an efficient way of testing a system s confidentiality and integrity. 3. Perform functional and security tests and apply on our system, then categorize and analyze the results from the implemented tests. We will try to answer what we mean by saying "high enough" in chapter 2, but for now, our aim is to achieve 70% coverage. 1.3 Approach To identify the faults on our application s functionality we have performed functional tests, aiming to get high enough coverage. The coverage was measured in terms of instructions run and branches taken. Random input generation tests were also performed to make a comparison between the different functional tests based on how much coverage they achieved. 2

12 1.4. Delimitations To identify vulnerabilities in the system we performed a simple penetration test, aiming to get access to or alter asset data as an unauthorized user. The need for an efficient method is because we have limited knowledge in security testing, thus need an easy to implement method that still can find flaws in a systems security. 1.4 Delimitations Software security is typically defined as the confidentiality, integrity and availability of a system. However, in this thesis we will only design test cases for the first two, confidentiality and integrity. 1.5 Structure The rest of the thesis is structured as follows. Chapter 2 will present background theory of testing and related work. Chapter 3 will describe the AUT as well as how the tests were constructed, and how the results were collected. Chapter 4 will present the results from the tests. In chapter 5 the results and method will be discussed. Lastly in chapter 6 the answers to the problem statement as well as our ideas for future work to be done will be given. 3

13 2 Software testing In this chapter, we will present some background theory of software testing. We will present some common methods for identifying test cases for functional testing, how to evaluate tests and cover security testing briefly. Lastly we will view related work for this thesis. 2.1 Software testing in general The definition of software testing can vary quite a lot depending on who you ask. One of the most popular definitions is Glen Myers, "Testing is the process of executing a program with the intent of finding errors". James Bach s definition is "Testing is questioning a product to evaluate it". The latter is more relevant to modern use of software testing, since testing can be used for quality assurance of a product [4]. The stakeholders of a product must know what quality it possesses, and to find that out empirical experiments are carried out [4]. Figure 2.1: A pyramid of the testing levels. Testing is usually performed at different levels, where the test cases have specified areas to cover. The most common levels used are unit testing, integration testing, system testing and acceptance testing [5], which can be seen in figure 2.1. A unit is the smallest piece of software that is tested. What a unit is can differ for different programming languages (for example a unit can be a class in java or c++, or a function in c), but if the tester thinks of it and tests it 4

14 2.2. Android testing as one "thing", it is a unit [4]. With integration testing, the smaller units are tested when assembled together [5]. System testing is the highest level of integration testing, which includes the complete product that is to be delivered [5]. Acceptance testing is a process that should determine if the stakeholder should accept the product [5]. System testing includes many testing techniques, to list a few: functional, usability, security, reliability and performance [5]. In this thesis, we will perform tests on the system level with the focus on functional and security testing. From now on we will focus on those subcategories. Popular software testing jargon often includes white box, grey box and black box testing. These represent strategies for constructing tests and are not testing technique in themselves. White box testing uses full knowledge of the inner workings of the code while black box uses requirements and specifications of the system to construct test [4]. Grey box is something in between. Unit testing is a form of white box test while functional testing usually is black box or sometimes grey box. When testing a system, it might have preconditions that does not allow invalid input and therefore has no need to test invalid input. This could be if the system only allows the user to press certain buttons to interact with it. In almost any system that accepts user-input, this is not the case. Then there might be a good idea to test invalid input, and hope that the system responds with some sort of error message. This is called negative testing [6]. Testing the system with valid input and expecting an action according to the specifications is called positive testing [6]. 2.2 Android testing When building an Android test, it will be based on the JUnit framework. The Android Testing Support Library provides a set of APIs which allows you to build and run test code for your applications, and has the ability to do functional user interface (UI) testing through Espresso. Espresso [7] is a testing framework that is integrated in the Android development environment, which has a very intuitive syntax that makes writing tests easy. The possibility to run tests directly in the Android Studio IDE makes it easy to integrate them into your development flow. During the starting process when developing an application that is to be tested, the structure of the code should be considered. To make a testable code architecture you need to split up parts of the application. An example of a common pattern for this is the Model- View-Presenter (MVP) architecture. The model provides and stores the internal data, the view handles the display of data, and the presenter coordinates the UI with the data and ensures that they are in sync [8]. The cycle for proper Android testing should start with testing the smaller units and move upwards towards integration testing. In the beginning, making sure that each component works as intended should be of focus. As tests succeed the confidence in the system increases and higher level of testing is to be used. The larger tests incorporating UI should be built upon the well-covering unit tests to get high coverage [9]. 2.3 Functional testing Since functional testing is a form of black box test the tester does not require full knowledge of the system to construct the test. The tester just needs to know what the system s output will be given a specific input. The test cases carry out interactions with the system that supposedly will fulfill a functionality that is intended. However, a system can usually take in a lot of different combinations of inputs and the amount of test cases rapidly gets out of hand. Even a simple program that takes an integer as input would mean a range from 2 31 to numbers in java, total of possible inputs [10]. And that only accounts the accepted 5

15 2.3. Functional testing inputs. What happens if the user tries to enter a larger number, or even a text-string? To minimize the amount of test cases while still assuring most faults are found makes up a challenge. To identify the most relevant test cases there are different methods that can be used [10]. Equivalence class testing is a technique that assumes test cases can be divided into equivalence classes. Test cases belong to the same equivalence class if the tester believes that [5]: They all test the same thing. If one test catches a bug, the others probably will too. If one test doesn t catch a bug, the others probably won t either. Consider a system that would accept an integer between 1-10 as input. If the integer is below 6 it returns false, otherwise true. Inputs 2 and 3 would then be in the same equivalence class, expecting the result to be false. The equivalence class testing technique implies identifying all possible equivalence classes and designing a test case for each [5]. In this example test cases could be input 3 expecting false and input 7 expecting true. Equivalence class testing is especially suitable if the input variables are independent of each other. It is also preferable if the variables refer to physical or logical quantities [10]. Boundary value testing is a technique based on equivalence class testing with the assumption that most faults occur at the boundaries of a system s possible inputs and equivalence classes. Therefore, test cases are created with input on the boundaries of the system. Like equivalence class testing, equivalence classes are identified and test cases are created on the boundaries, with one above and one below [5]. Using the same example as above there are two equivalence classes, inputs 1-5 and The inputs for the test cases would then be 1, 2, 4, 5, 6, 7, 9, 10. An addition to boundary value testing is Robustness testing, where you also add test cases slightly beyond the boundaries [10]. This adds negative test cases. In the example above test cases 0 and 11 would then be added. Decision table based testing is a method that identifies test cases by creating tables with conditions as inputs and actions as outputs [10][5]. When certain conditions are fulfilled the system is expected to do a certain action according to specifications. Each condition is labeled with a C and a number and can have the value true (T), false (F) and does not matter (-). Each action is labeled with an A and a number and is marked with an X if it is to be executed, and nothing if it should not be executed. Using the previous example, a decision table could be created like in table 2.1 with three conditions and three actions. One last action is added to indicate impossible combinations of conditions. Table 2.1: Decision table - Example C1: Number between 1-5? T T F F C2: Number between 6-10? T F T F C3: Number less then 1 or greater then 10? T F F T A1: Return false X A2: Return true X A3: Return error message X A4: Impossible combination X Decision table-based testing is an excellent method to cover most cases that are likely to happen without redundancy of test cases [10]. It is especially useful if the input variables are dependent on each other [10]. 6

16 2.4. Coverage Random testing is, as the name implies, a method that utilizes randomness in the generation of test cases. Instead of analyzing boundaries and normal values you just let a random input generator create the test cases. A positive effect of this approach is that any bias from a tester is reduced, and test cases that might not have been thought of can be found [10]. However there is a high risk that many redundant tests will be performed, and knowing when to stop testing is not obvious [10]. State-transition testing is like decision-table testing, an excellent method to create test cases. The method involves identifying various states and each of the possible transitions between the states by drawing a state-transition diagram. States are drawn as circles or boxes and transitions are drawn as arrows. From the diagram test cases can be designed by: 1. Visiting all states. 2. Taking all paths. 3. Visiting each transition. The first method is not optimal, since coverage usually will be low. The second method is impossible if the diagram has loops, since the possible paths would then be unlimited. The preferred method would be the third, since it can be achieved without much struggle and still generate test cases with a reasonable amount of coverage. For better coverage, the actual input values used in the transitions would have to be generated by methods above [5]. 2.4 Coverage Coverage is a metric often seen in software testing, used to describe the amount of the source code that has been executed during a test suit. It can also be described as a measurement of the adequacy of a test suit, and is often used as a quality evaluator [11]. Achieving a high coverage often means less bugs unnoticed, but most of all it means less untested parts in the source code. Instruction coverage provides information about the amount of byte code that has been executed or missed. When the source code of a program is compiled into instructions, it is easy to measure the total size and monitor how many of them are run in total. Even if a high instruction coverage is achieved, it does not necessarily mean that most paths of the program are taken. A single instruction branch could be left unchecked containing a big flaw and still score a high instruction coverage. Branch coverage examines all branches of the program, of each and every decision point. The aim of branch coverage could therefore be seen as a method in trying to execute all reachable code of the program at least once. Creating tests that cover all of the code is more likely to find faults than test that do not. But devoting time to develop meaningful test cases is probably more effective than writing tests that cover all of the code. Setting a coverage goal for a product release could be counter-productive, since humans are goal-oriented and might focus more on achieving high coverage rather than detecting defects [5]. 2.5 Security testing The attributes confidentiality, integrity and availability are central indicators of computer security [12]; thus, security testing covers the following concepts. 7

17 2.6. Related work The confidentiality of the system is ensured. The integrity of the system is maintained. Information is kept available for the authorized personnel. Security testing is a way to find security vulnerabilities in a system that could threaten these concepts. There are different types of security testing, one of them being penetration testing. Penetration testing is a way to test the vulnerabilities of a system by looking at security defences through the eyes of a hacker. In other words, the penetration testers use the same techniques and tools as a hacker would do to penetrate a system, but do so in a controlled manner [13]. Another popular method for security tests is vulnerability assessment, which is a technique to identify and quantify the security vulnerabilities in a given system. The vulnerabilities are then eliminated or mitigated to make the system more secure. Fuzz testing Fuzz testing, or fuzzing, is a brute force approach to find vulnerabilities in a system by feeding it with invalid or semi-valid input [14]. It is a form of random testing used to find weaknesses of systems. In other words, it is used to find flaws in a system which compromises the system s security, or cause any unwanted behaviour. The idea is simply to try semi-valid input against a system and observe how it responds. Semi-valid data should be just enough valid to enter through the system without immediately being discarded (by a parser for example, if you type in characters when an integer is expected), but still enough invalid to maybe cause problems [14]. Ultimately, fuzzing is usually a black box testing technique, but can still be used against software with accessible source code. To produce input data, fuzzers can use different techniques. The most common techniques are generating input or mutating existing valid input. To generate data, the fuzzer is given a set of rules or description of how the data should look and let it generate, often with random functions [14]. The mutation works in the way that some valid data is given to the fuzzer that changes parts of it in order to produce new, untested data [14]. 2.6 Related work Bångerius and Fröberg [15] performed a similar work of ours where functional tests were built using the Espresso framework. A recommendation in the thesis was that future testers could look for testing oracles or semi-oracles to use for testing, since running test on your own code is not optimal. They also recommended looking for an automatic exploration method to get more coverage. The intent of our project has been to follow in their footsteps by making more extensive tests, presenting coverage reports and testing the security. Some of the studies were clearly trying to promote their own tools and work in general, but could still be helpful to us when conducting or own work. In one study, Azim and Neamtiu [2] developed a tool for automatic exploration of Android applications. The tool was using two different algorithms, Depth-first- and Targeted Exploration, which were compared against each other measuring how much coverage they achieved. The Depth-first Exploration launched a start activity, extracted all UI-elements, and for each transition to another activity, it traversed to the new activity, making a new search and going back to the last activity visited when no new transition was found. This way the tool could systematically explore an application mimicking user interaction. Targeted exploration took a list of activities as argument and launched them all, as a fast way of exploring all the activities and also reaching activities that are not easily navigated to during normal use. The tool achieved 59-64% activity coverage and 30-36% method coverage on 25 popular Android applications. 8

18 2.6. Related work Unlike the mentioned work our thesis will present the coverage of one specific application using Espresso, while their approach was to test their own tool on several applications. Arnatovich, et al. [16] presents their own automated application UI exerciser DroidDEV, aiming to achieve high code coverage in automated functional UI tests. It uses a best-first search algorithm to discover all the application UIs and go through all their widgets, and according to the experiments conducted in the study it performs similarly to manual UI testing with regard to code coverage. The focus of this study was to illustrate the value of their own tool by comparing its code coverage with that of other testing tools. In our thesis however, the focus will be to find out how much coverage our tests provide and how to enhance the results. Inozemtseva and Holmes [17] did a study on the correlation between test suite effectiveness and coverage. The results suggest that there is no clear correlation between the two. However, they found that for larger java programs, there was a relative high correlation between number of test cases and test suite effectiveness. When the test suite size was ignored, they found a moderate to high correlation between suite effectiveness and coverage. Even though this study suggests that coverage might not be a good metric for software testing effectiveness in general, when the size of the test suite is ignored it might still be adequate. In our study the focus is on coverage and not on effectiveness. We will not limit the size of our test suite as well. 9

19 3 Test implementation In this chapter, the implementation of our tests will be presented. This will describe how our work has been carried out to provide some insight for others that intend to implement similar tests. First a short description of the AUT is presented, followed by the selected frameworks and approach, before the implementation is in focus. 3.1 The application under test The application mainly consists of a map functionality, where events can be placed on the map, and a voice- and video call functionality, which makes the communication and information sharing possible. An event is placed on the map in the from of a pin to describe a task for the Swedish Defence Forces, e.g. a broken bridge or a person in the need of help. The event information will only be accessible to a selected group of users. This makes a great deal of data confidential, and thereby seen as valuable assets. The application will also have a secure log-in functionality where a personal NFC id and password is required to become authenticated. This application needs to be dependable in order to work for its purpose. Figure 1.1, seen in chapter 1, shows the map functionality where 6 pins have been added to the map by the application users. 3.2 Selected frameworks and approach When performing automated tests the framework at hand should be well chosen to avoid problems during the testing phase. In our case, the criteria when choosing a framework was having an API that was easy to learn, and had a good documentation available. This is due to the fact that we had no earlier experience in any form of Android testing, and had a strict time limit. In a study conducted by Torchiano, et al. [18], five automated UI testing frameworks are presented and discussed, among one was the Espresso framework. They mention advantages of Espresso such as being able to use direct references to the GUI elements due to Espresso being integrated in the Android development environment, and how the API is easier to use in comparison with the four other frameworks that were presented. Esbjörnsson performed a study about open source Android GUI testing frameworks [19], in which an evaluation of different criteria was made involving four out of the five frameworks included in the work of Torchiano, et al., where Espresso was one of the four. The 10

20 3.3. Functional testing results show that Espresso has an easier API and a better logging support compared to the three others, as well as a presence of a good documentation of the framework. The aforementioned work provides evidence that Espresso would be a well suited framework, and has therefore been the selected framework when performing our functional tests. Also, given that the Android Studio IDE can provide a coverage report that matched our needs we found the framework good enough. A risk analysis of the system under test (SUT) was made in a separate work performed in parallel with this thesis. It was done in the early stages of the joint project presented in chapter 1, and thereby gave us a foundation for a security test. Having most threats and assets identified helped us find a suitable security test that could check the SUT and the possibility of these threats. The threats found in this risk analysis that were of interest to us were those which threatened assets connected to the concepts confidentiality and integrity. Studying the works of Goel [3] and Copeland [13] resulted in choosing a penetration test approach. A penetration test is formed to try and find and exploit the vulnerabilities of a system, much like we desired to do with our SUT. A simple yet effective way of finding vulnerabilities is through fuzzing, which implies that it could be a good technique for a penetration test [3]. Alongside inspection of binaries, fuzzing is the most popular method hackers use today to find vulnerabilities [20]. Given this knowledge the decision to use fuzzing as a security testing approach was made. During the project mentioned in chapter 1 we tried to do a vulnerability assessment with the help of open source software Metasploit and Nmap. Vulnerability assessments done using the aforementioned tools did not find any vulnerabilities. More knowledge in the security field would have been required to obtain any vulnerabilities. 3.3 Functional testing In order to ensure the user that the AUT is dependable, the application s functionalities have been tested. Three functionalities were put to test. The three functionalities that have been tested are the map, the authentication, and the call functionality. These tests have been made with the Espresso framework. Test cases were identified with the methods described in section 2.3 to compare how much coverage could be achieved. Each functionality was also put under a random test, measuring coverage for reference. When executing Espresso tests, the test will automatically detect when the main thread is idle before running the next instruction. This is a problem for some other testing tools when the test fail because the UI has not had enough time to render before the next instruction is run, requiring the tester to add thread sleeps between the instructions. When a test passes, the functionality works as desired. When a test fails, bugs can be found. In the case of a failed test, Espresso provides debugging information that simplifies the search of a bug. The paragraphs below describe the process of generating test cases for each functionality that was decided to put under test. After the test cases were identified they were implemented in Espresso and performed on the AUT. All of the Espresso tests source code is found in appendix A. Map test To ensure that a pin is added and removed correctly the map test was created. The map test consists of three parts; add pin, find pin and remove pin test. If all three parts of the test pass, the map test will pass. First, an add pin test performs a long click in order to add a pin on the map activity at a randomly generated position on the screen. Then it will add a title and description (with different inputs depending on test cases seen below). If a pin is successfully placed on the map, the first test will pass. Figure 3.1 shows the view when adding a pin, where the title and description fields are visible. 11

21 3.3. Functional testing Figure 3.1: Application view when creating a new pin After the pins are added, a find pin test will check the same position on the screen to see if the pin information will pop up and contain the same title that was recently added. If the title matches what is currently being displayed on the screen the test will pass. When the pin has been added and checked that it was created, a remove pin test starts. The same position on the screen will once again be clicked, this time to remove the pin. This is done by clicking the button "Archive". The position will then be clicked again to see if the pin actually has been removed. If the pin is gone, the test will pass. With the basic functions of the map test done, test cases were developed by the methods mentioned in chapter 2. Equivalence class testing: All combinations of inputs (title and description) of a map pin should result in adding a pin. Therefore all application actions after the confirm button are pressed should lead to a map pin is added, unless either title or description is too long. Since only positive test cases are identified in equivalence class testing, the only test case will be to add a map pin with a title and description within limits. When removing a pin, there is a similar case. The only thing that is supposed to happen is that the pin is removed, and therefore a test case for removing a pin and checking that it has actually been removed is added. Boundary value testing: It is hard to identify normal boundaries of the input since it is supposed to accept any. For robustness however, there is a limit of how long the description or title can be. The database has the limit of title length at 30 characters and the limit of description length at 200 character. The test cases for the boundaries would then be with a title of 29, 30 and 31 characters long and description of 199, 200, 201 characters long. Removing a pin has no interesting boundaries since there is no user input taken. The test case would be same as above. Decision table based testing: The conditions when either adding or removing a pin, in addition to those explained in methods above, is having a server connection and being au- 12

22 3.3. Functional testing thenticated. A decision table for adding a pin can be seen in table 3.1 and a decision table for removing a pin can be seen in table 3.2. Six test cases were found for adding a pin, and three for removing a pin. Table 3.1: Decision table - Map test, add pin C1: Title is correct format? T F T F - - C2: Description is correct format? T T F F - - C3: Connection to server established? T T T T F T C4: Authenticated to server? T T T T - F A1: Add pin X A2: Show error message X X X X A3: Return to log-in screen X Table 3.2: Decision table - Map test, remove pin C1: Connection to server established? T F T C2: Authenticated to server? T - F A1: Remove pin X A2: Show error message X A3: Return to log-in screen X The identified test cases for add pin tests are summarized in table 3.3 and the test cases for removing a pin are summarized in table 3.4. The test case column describes which functionality that has been tested and the method that was used to identify the test case. M A denotes the tested functionality was add pin on map, and M R denotes the tested functionality was remove pin on map. Label E i denotes the method used was equivalence class testing, B i denotes the method used was boundary value testing, and D i denotes decision table-based was used. The inputs for each test case was title, description, connection to server and if the user was authorized to the server. Column 2 denotes whether the input title is valid, if it is less than 30 characters. Column 3 denotes whether the input description is valid, if it has less than 200 characters. Column 4 denotes whether the application had a connection to the server and column 5 denotes whether the application was authenticated to the server. The value "-" means that the input is not relevant for test case. Table 3.3: Identified test cases - Map test, add pin Test case Title Description Connection Logged in Expected action M A E 1 valid valid yes yes pin is added M A B 1 valid valid yes yes pin is added M A B 2 29 characters valid yes yes pin is added M A B 3 30 characters valid yes yes pin is added M A B 4 31 characters valid yes yes error message M A B 5 valid 199 characters yes yes pin is added M A B 6 valid 200 characters yes yes pin is added M A B 7 valid 201 characters yes yes error message M A D 1 valid valid yes yes pin is added M A D 2 invalid valid yes yes error message M A D 3 valid invalid yes yes error message M A D 4 invalid invalid yes yes error message M A D no - error message M A D yes no launch log-in screen 13

23 3.3. Functional testing Table 3.4: Identified test cases - Map test, remove pin Test case Connection Logged in Expected action M R E 1 yes yes pin is removed M R B 1 yes yes pin is removed M R D 1 yes yes pin is removed M R D 2 no - error message M R D 3 yes no launch log-in screen Authentication test The authentication screen is the main activity of the application, as it is the first activity launched when the application is opened if there is no stored cache of previous sessions. It lets the user scan its NFC card, or enters the id manually, and then enter his/her password. The credentials are then sent to the server which checks if they are correct with the database and determines if the user is allowed to use the application or not. Unless the server responds with a token, the user will stay in the authentication screen. If the client receives a token, the map activity is launched and the user is given access to map information and contacts. With this information about the authentication functionality test cases can be designed. Equivalence class testing: The possible inputs for the application is NFC id and password. Valid combinations of these should launch the map activity, thereby can one equivalence class be using valid passwords and valid NFC ids as input. If the NFC id is valid but the password is invalid, the application is expected to stay in the authentication screen, creating one equivalence class. Vice versa, one equivalence class is made for invalid NFC id and valid password combination. One class is also added for both invalid password and invalid NFC id, creating a total of 4 test cases. Boundary value testing: When both password and NFC id is valid, you could say it is a boundary value since it is expected to be exact. One character above or below belongs to a different equivalence class, since it expects a different output. That would create test cases of input combinations: Valid NFC id, valid password Valid NFC id, password with one character off from being valid NFC id with one number off from being valid, valid password Invalid NFC id, invalid password Adding a case to test robustness would be to try a larger input than the database can handle. The password is hashed with SHA-512, making the stored hash a fixed length, disregarding actual length of password. However, the NFC id column only has 30 characters assigned per row. The robustness test case would be to try add a 31 length-long NFC id, expecting the application not to be authenticated. To be thorough, test cases for all users in the database would have to be done, but because of time limitations they were only made on one user. Decision table based testing: Four relevant conditions of the authentication functionality was identified, and is mapped with corresponding application action in table

24 3.3. Functional testing Table 3.5: Decision table - Authentication test C1: NFC id is correct? T F T F T F - C2: Password is correct? T T F F C3: NFC is correct format? T T T T F F - C4: Connection to server established? T T T T T T F A1: Authenticate user, launch map activity X A2: Do not authenticate user, stay on log-in screen X X X A3: Show error message X X A4: Impossible combination X All test cases generated to test the authentication functionality are summarized and described in table 3.6. The columns are like the previous table for map test. The test case column describes which functionality that has been tested and the method that was used to identify the test case. A denotes the tested functionality was authentication. Label E i denotes the method used was equivalence class testing, B i denotes the method used boundary value testing and D i denotes decision table-based was used.column 2 denotes whether the input NFC id is valid, if it exists in the database. Column 3 denotes whether the input password is valid, if it exists in the database. If both the NFC id and password is a valid, it is a valid combination in the database. Almost valid means that one character is off from being valid. Incorrect format of the NFC id means more characters than 31 as input. Table 3.6: Identified test cases - Authentication test Test case NFC id Password Connection Expected action AE 1 valid valid yes launch map AE 2 valid invalid yes stay on log-in screen AE 3 invalid valid yes stay on log-in screen AE 4 invalid invalid yes stay on log-in screen AB 1 valid valid yes launch map AB 2 valid almost valid yes stay on log-in screen AB 3 almost valid valid yes stay on log-in screen AB 4 invalid invalid yes stay on log-in screen AB 5 incorrect format - yes get error message AD 1 valid valid yes launch map AD 2 valid invalid yes stay on log-in screen AD 3 invalid valid yes stay on log-in screen AD 4 invalid invalid yes stay on log-in screen AD 5 incorrect format - yes get error message AD no get error message Call test The original plan was to test the contact list, but the contact list only exists to give the user the possibility to view which users are available to receive a call, and starting a call with a desired user. The contact list has no support to add, remove or alter contacts. The contacts that appear in the list are users of the application, and they can only be added directly in the database. Because of that, the test cases were designed to check if it is possible to call any of the available contacts. Without the call functionality the contact list would be unnecessary. Since no other phone is available to answer a call automatically, there is a difficulty in designing a test for the call itself. Equivalence class testing: The only action a user can perform in the contact list is to call a contact. Calling a contact should result in launching the call activity with a text showing whom is being called. Therefore the only test case will be to call a person in the contact list 15

25 3.4. Security testing and check if the call actually is started, and in that case make sure that it is started with the desired person. Boundary value testing: Since there is only a specific set of actions a user can do inside the contact list there are no interesting boundaries. There is no point in trying to call a contact that does not exist since the non-existing user would not appear on the list. The only test case possible would be the same as above, to call a person in the contact list. Decision table based testing: To call a person, the conditions to be authenticated and connected to the server has to be fulfilled. A decision table for the call test can be seen in table 3.7. Table 3.7: Decision table - Call test C1: Connection to server established? T F T C2: Authenticated to server? T - F A1: Call selected contact X A2: Show error message X A3: Return to log-in screen X In total, three test cases were identified for the call test. All three methods produced the same test case, call a contact, and the decision table method identified two additional ones. Coverage Coverage reports were generated for the test cases identified, one for each method. The reports were generated by Android Gradle plugin 2.3.1, and created by JaCoCo [21] which is a free coverage library for Java. From the reports one can read instruction coverage and branch coverage. There is also support for looking at which lines of code were actually run during the tests. To generate reports, the tool instrumented the bytecode of the application in order to inject tracing calls [21]. The tracing calls were registered during test execution in order to document which lines of code were run. To calculate coverage, the registered instructions run were compared to the total number of instructions in the java class files. The resulting coverage reports will be presented in section Security testing For security testing, a simple variant of a fuzz test was developed in Java. The goal of the fuzz test was to send messages to the server in an attempt to gain asset data, or alter the information in the database. To get higher chances of being successful in the task, the fuzzer was given a command builder that would generate valid commands. Since the goal was not to test robustness, all commands had the right amount of parameters and the correct format. If the server fails to parse an int for example, it will just discard the input and move on to the next iteration of the loop creating uninteresting test cases. The fuzzer did not have access to the correct combinations of NFC ids and passwords, but the possibility of brute-forcing to get it right was still there. The fuzzer generated random attempts to log in, use a random authorization token and all of the commands that an authorized user can send. The response it received from the server was compared to a list of commands associated with assets and then either discarded or saved. A flowchart of the test is shown below in figure

26 3.4. Security testing Figure 3.2: Fuzz test flow chart The source code of the fuzzer implementation can be seen in appendix B. It consists of three classes. One is the main class, deciding how many threads to be run and the code for starting them. The second class extends thread and establishes a TLS connection with the server to be able to transmit messages. It is also responsible to validate the responses that are received, whether the assets are affected or not. The third class is data generator which decides what to transmit. It generates almost valid commands at random, which the thread then can pass on to the server. Once started, the test would run until cancelled manually. During our testing, the fuzzer was run with 5 threads for 10 hours on 3 occasions. 17

27 4 Results In this chapter the results from all tests will be presented. First we present the bugs found during the functional tests. Then we will show how much coverage the tests achieved and lastly we will present the results from the security test. 4.1 Functional testing Out of the three functional tests made, only two were able to identify any bugs in the application. The authentication test found 5 different bugs and the map test 4, while the call test found none. After a bug was identified it was assigned a classification to specify the bug in other terms than just by what test it belonged to. The resulting bug report can be seen in table 4.1. In column 1 A i denotes the bug report i for authentication tests, and M i denotes the bug report i for map test. Table 4.1: Bug report from functional testing Report Bug Classification A 1 It is not possible to log in after a log out has occurred without restarting Control flow the application A 2 Two intents of the map activity starts when logging in to the application, Performance slowing down the application A 3 After a log out, seven intents in total are created when logging in again Performance A 4 When logging out, the unlock activity is started Control flow A 5 The more log-in/out in a row, the more the application is slowed Performance down. After 5 log-in log-out, a significant slow-down occurred. M 1 The map activity freezes when logging in for the first time after starting Functional the application M 2 The title should not be able to contain more than 30 characters, but Robustness accepts it anyways and cuts it M 3 If the description contains more than approximately 700 characters the Robustness description field takes up too much space which eliminates the "Close" and "Archive" buttons from the pin description view M 4 If the title is shown over the pin it is not possible to add or select a pin behind the title area Performance 18

28 4.2. Coverage reports 4.2 Coverage reports Coverage reports were retrieved after running the functional tests, one separate from each test case generation method; equivalence class, boundary value, decision table-based and random testing. The results from all four test cases can be seen in table 4.2. It is clear that the decision table-based test case performed the best as it achieved the highest amount of coverage in both instruction and branch coverage, where it got 59% and 43% respectively. Table 4.2: Coverage report - Whole application Test cases Missed instructions (coverage) Missed branches (coverage) Equivalence class 3,303 of 7,740 (57%) 306 of 510 (40%) Boundary value 3,255 of 7,740 (58%) 299 of 510 (41%) Decision table-based 3,198 of 7,740 (59%) 293 of 510 (43%) Random testing 3,290 of 7,740 (58%) 298 of 510 (42%) With respect to coverage for specific activities, the numbers are a little bit higher. The coverage for the map activity is shown in table 4.3 and the coverage for the authentication activity is shown in table 4.4. For the contact fragment, all tests achieved the same coverage with only one missed branch which occurred if the application had not received any contacts, which would result in showing a text saying there are no contacts. The biggest differences between the methods occurred in service, a class responsible for all background tasks like storage and server communication. A table of the service class coverage can be seen in table 4.5. Table 4.3: Coverage report - Map activity Test cases Missed instructions (coverage) Missed branches (coverage) Equivalence class 61 of 506 (88%) 5 of 14 (64%) Boundary value 61 of 506 (88%) 5 of 14 (64%) Decision table-based 61 of 506 (88%) 5 of 14 (64%) Random testing 61 of 508 (88%) 5 of 14 (64%) Table 4.4: Coverage report - Authentication activity Test cases Missed instructions (coverage) Missed branches (coverage) Equivalence class 214 of 641 (67%) 32 of 62 (48%) Boundary value 214 of 641 (67%) 32 of 62 (48%) Decision table-based 196 of 641 (69%) 29 of 62 (53%) Random testing 214 of 641 (67%) 30 of 62 (52%) Table 4.5: Coverage report - Service class Test cases Missed instructions (coverage) Missed branches (coverage) Equivalence class 672 of 2,195 (69%) 111 of 202 (45%) Boundary value 662 of 2,195 (70%) 107 of 202 (47%) Decision table-based 652 of 2,195 (70%) 106 of 202 (48%) Random testing 638 of 2,195 (71%) 106 of 202 (48%) 4.3 Security testing At one occasion the fuzz test was able to add map pins to the database without being authorized. This can be seen in figure 4.1, where entries with EventID 638 or higher is corrupt. When a map pin has been added from the application, the description and title gets converted 19

29 4.3. Security testing to base 64-encoding to easier be able to send, and also allow a larger set of characters. The map pins that have a title or description with characters that are not hexadecimal numbers have not been added from the application, but from somewhere else. Some of the pins were added as the user 0, which the server uses to identify who is not authenticated. The other pins were added by user 2, which means that the fuzzer managed to get authenticated as well. The breach occurred on the first run of the fuzzer. The breach occurred on the first run of the fuzzer. The results could not be reproduced on the next runs, and the exact reason for how it happened could not be figured out within the given time frame of the work. Figure 4.1: Invalid database entries from the fuzzer The security test was able to identify four different security flaws of the system in total. These were assigned a classification much like the functional test bugs, which can be seen in table 4.6. In column 1 S i denotes the bug report i for Security tests. Table 4.6: Vulnerability report for the security testing Test Security flaw Classification S 1 When the server was put under stress test, it was hard for Availability mobile users to get responses S 2 When a pin has been added with an incorrect format manually Availability (directly to the server, without the map functional- ity), the application will crash when trying to obtain the pin S 3 Pins could be added without authorization Integrity S 4 Pins could be fetched by an unauthorizad user Confidentiality 20