The Pennsylvania State University The Graduate School College of Engineering APPGRADER: AN APP QUALITY GRADING SYSTEM BASED ON CODE-LEVEL FEATURES

Transcription

1 The Pennsylvania State University The Graduate School College of Engineering APPGRADER: AN APP QUALITY GRADING SYSTEM BASED ON CODE-LEVEL FEATURES A Thesis in Computer Science and Engineering by Xi Li c 2018 Xi Li Submitted in Partial Fulfillment of the Requirements for the Degree of Master of Science August 2018

2 The thesis of Xi Li was reviewed and approved by the following: Sencun Zhu Associate Professor of Computer Science and Engineering Thesis Advisor Danfeng Zhang Assistant Professor of Computer Science and Engineering Chita R. Das Distinguished Professor of Computer Science and Engineering Department Head of Computer Science and Engineering Signatures are on file in the Graduate School.

3 ABSTRACT The current app ranking systems applied by app markets are mainly based on app rating and downloads. However, these systems have drawbacks in handling: (i) apps with abnormally high ratings and fake downloads; (ii) newly published apps with limited user feedback. Rankings of these apps may not accord with their actual quality, which will mislead users. Therefore, in an attempt to explore app ranking systems and change the under studied status quo of it, we propose AppGrader, a novel app quality grading system that ranks apps under the same category based on app functionality measured by code-level features. This system is inspired by the analysis on 18 millions app reviews which suggests that while giving ratings, most users may consider user interface and other features that can be extracted directly from app code. Therefore, our system statically analyzes app code and generates feature view graph for each app which encodes app codelevel features. For app ranking, we apply Graph Convolutional Network to cluster apps into different classes based on the complexity of their corresponding feature view graphs, where each class indicates one level of app quality. According to the system evaluation from two perspectives: system accuracy and label dissimilarity, AppGrader performs well on 1440 real world apps with average accuracy of around 72% and label dissimilarity of around 1, which indicates that AppGrader could be applied for evaluating apps with fake ratings and newly published apps. Keywords: Mobile App Ranking, App Quality Evaluation, App View, Graph Convolutional Network, Deep Learning iii

4 TABLE OF CONTENTS List of Figures List of Tables vi vii Chapter 1 Introduction 1 Chapter 2 Related Work 6 Chapter 3 Background 11 Chapter 4 Methodology System Overview App Code-Level Features User Experience of an App App Review Analysis System Architecture Code De-compiler Feature View Graph Generator Generate View Graph Generate Feature Vector Graph Convolutional Network iv

5 Chapter 5 Implement and Evaluation Data Collection and Processing Data Collection Label Assigning GCN Inputs Experiment Results and Analysis Efficiency Effectiveness System Accuracy Label Dissimilarity Synthesized Result Case Study Chapter 6 Discussion Limitation Future Work Chapter 7 Conclusion 50 Bibliography 51 Appendix A App Review Analysis 54 Appendix B Detailed Feature Vector 59 Appendix C Testing Result of AppGrader 63 v

6 LIST OF FIGURES 4.1 System architecture Interaction between a user and an app Distribution of top 500 high frequency terms Distribution of 50 topics among 18 million app reviews Proportion of topics indicating app features in total 50 topics Graph convolutional network for node classification Graph convolutional network for app classification Comparison result in terms of system accuracy under three groups of ground truth labels Comparison of propagation models of GCN (Thomas et al. 2017) Comparison result in terms of label dissimilarity under three groups of ground truth labels Comparison of test accuracy and label dissimilarity of the first test (weights: 33%, 33%, 33%) vi

7 LIST OF TABLES 4.1 Terms indicating app features in top 500 high-frequency terms Topics indicating app features Intent construction methods Android APIs for activity switching Android methods that listen user inputs Data overview Summary of results in terms of app classification accuracy Summary of results in terms of average label dissimilarity for false predictions Mis-estimated camera apps A.1 High-frequency terms counting result (top 500) B.1 Detailed feature vector C.1 Testing result of AppGrader on 123 apps in the first test (weights: 33%, 33%, 33%) vii

8 CHAPTER 1 INTRODUCTION Recently, as the rapid development of smart phones and tablets, mobile application markets such as Google Play for Android and App Store for ios are growing fast. Take Google Play for example, over 1 million apps were released in , and the number of downloads had exceeded 65 billions in the same year 2. Up to 2017, there have been more than 3.5 millions available apps 3, and the number of apps is still increasing. Not as simple as several clicks to download apps from app markets, the huge number of apps is overwhelming users: even under the same category, there are 200 thousands apps on average to choose. Therefore, an app quality evaluation system that ranks apps by app quality is indispensable to liberate users from countless apps. The current ranking strategies applied by app markets are mainly based on app ratings and downloads. Apart from the app ranking, users may also refer to the

9 reviews written by someone who has already tried this app. Nevertheless, these ranking strategies have drawbacks. Some app companies or developers may manipulate the ratings and reviews of their apps to get more downloads from users and further, to earn higher revenues. It is not uncommon to find deceptively positive reviews and abnormally high ratings. Besides, newly published apps have limited user feedback. As a result, rankings of these apps will deviate from real app quality, which may mislead users. The most effective way is to try each app on your own, but that is inconvenient and time-wasting. In the sea of mobile apps, it is hard for people to choose an appropriate one. Moreover, app developers are mired in predicament brought by ill-conceived ranking systems. For example, the app rankings offered by app markets merely inform developers the popularity of apps but not the popular reasons behind them, which finally impede the advent of excellent apps. However, there is few research about app ranking system. We only find ranking systems that offer app store optimization rather than optimizing the app itself, such as RankMyApp [1] and Appfigures [2]. These systems conduct sentiment analysis on app reviews to mine key words and popular topics by which they do app store optimization based on the mining result. That is, these systems analyze your app, your competitors apps and the app market, and give opinions on app name, app description, app icon, screenshots to improve your app visibility in a certain category. However, they just modify the external parts of an app rather than evaluating the app itself therefore cannot help users find high-quality apps as they expect. These systems are superficial and do not solve the primary problems. Compared with relatively rare app ranking systems, there are plenty of app recommendation systems. Nevertheless, these app recommendation systems do not solve the problems mentioned before either. Some systems take user privacy preference into considerations. For instance, Liu et al. propose a system that recommends apps to users which reach a trade-off between app functionality and user privacy preference [3]. But the fact is that most users lack safety awareness. When receiving a permission request to access sensitive data from an app, most users thoughtlessly click allow with less mind of their privacy. Thus, this type of system can not learn real user privacy preference and recommend appropriate apps 2

10 to them. Some other systems are personal recommendation systems which analyze user usage history to recommend apps that the user may like, such as [4, 5, 6, 7]. If a user is downloading an app under a specific category, their recommendations may not hit because of limited training samples. In this case, the user has to resort to app ratings and reviews again which, as we mentioned before, are not trustworthy. Given these observations, it is necessary to build an innovative app ranking system which is efficient and effective, and most importantly, can solve the problems we mentioned. This system must give a relatively impartial evaluation on app quality and rank apps by the level of quality. The most challenging part is what criterion should we choose to do quality evaluation on the apps. Since external ratings and reviews are not applicable, we decide to focus on app itself user interface and app structure. The inspiration comes from our observation towards the way in which people give rating to an app. As we know, Android apps are user interaction intensive thereby set user-friendliness as a benchmark to gauge apps. The rating of an app is mainly based on the user experience towards this app, or the interaction between the user and this app. The interaction between a user and an app is actually the interaction between the user and app interfaces and the overall app structure. So we decide to evaluate app quality based on code-level features like user interface (UI) and structure. This idea is supported by the app review analysis discussed in chapter 4, where we count term frequency and mine topics in about 18 millions app reviews covering all app categories. From the high-frequency terms and part of the review topics, we notice that while giving ratings, users care about the design of user interface such as picture and color, the widgets used in an interface, permissions an app may request like Wi-Fi and camera, and the advertisement contained in the app. Thus, it makes sense to evaluate apps by its code-level features, features that can be extracted directly from app code. Based on the observations above, this paper proposes a novel app ranking system based on app quality which is measured by app code-level features like user interface structure, aiming to help users to download apps which are worthy of their rankings. This system is an extension of ViewDroid [8] which is applied to detect app repackaging by app views. Since graph can preserve app nature as 3

11 much as possible, in our app quality grading system, we represent an app as a graph called feature view graph which encodes code-level features of an app. The code-level features such as UI and the overall app structure are extracted directly from app code. In this feature view graph, each node is an interface and has its own feature vector which encodes not only the widgets, colors, fonts, pictures, interaction methods used in the corresponding interface but also some app features. Each edge indicates a switch between two different interfaces. This feature view graph is generated by searching and tracking Android specific APIs in app code, such as startactivity() and setcontentview() which indicates interface switching and interface layout respectively. For app ranking, we will apply a classification model to cluster apps into different classes, where each class indicates one level of app quality. The reason why we apply classification model rather than regression model is that there is barely difference between apps in the same quality level. For example, it is hard to tell the superiority of twitter over Facebook. As for the classification model, we apply graph convolutional network (GCN) [9] on these feature view graphs to estimate quality of their corresponding apps. GCN is a classification model for graph-based data. We represent an app as a feature view graph and app structure is also an important feature, so GCN fits our system well. It assigns label (the level of app quality) to each app by which we can rank apps. By AppGrader, we can evaluate app quality by app functionality and make recommendations on apps to users, as well as give feedback to app developers on app designing. Our paper makes the following contributions: 1. Our system extracts code-level features from an app and generates feature view graph for each app which preserves the nature of an app, such as user interface and app structure. 2. This paper implements AppGrader, a novel app quality grading system based on code-level features like user interface and app structure. As far as we know, it is the first system that takes advantage of app itself to evaluate apps. Besides, it combines the newly proposed graph convolutional network to do the app quality grading. 4

12 3. Evaluation: AppGrader is trained and tested on over 1440 real world apps from Google Play, covering 14 app categories. The performance evaluation of AppGrader is grounded in two perspectives: system accuracy and label dissimilarity. AppGrader is a very efficient app quality grading system with average accuracy of 72.54% and label dissimilarity of around 1. The remainder of the paper is organized as follows. Chapter 2 talks about related work. Chapter 3 introduces the background of Android platform and apps, as well as the graph convolutional network. The design of AppGrader is described in chapter 4, followed by the implementation and evaluation in chapter 5. Chapter 6 presents discussion and we conclude our work in chapter 7. 5

13 CHAPTER 2 RELATED WORK The systems that help users choose appropriate apps consist of app ranking systems and app recommendation systems. Research on app ranking system for app markets is rare now, and we only find the systems that help app developers improve their apps visibility in app markets like RankMyApp [1] and Appfigures [2]. While there are numerous app recommendation systems that provide users with personal recommendations on what apps to choose, such as [10, 11, 12, 3, 13, 4, 5, 6, 7]. RankMyApp [1] targets on app developers. It offers App Store Optimization (ASO) and Search Engine Optimization (SEO) to improve app visibility in app markets by modifying keywords inserted in app name and description. Besides, it also implements app review analysis by periodically checking app reviews, replying quickly to user queries and converting user feedback into suggestions. Appfigures [2] is similar to RankMyApp, differing only in this that it takes users as their target clients at the same time. For app developers, Appfigures offers mobile app store analytics, ranking and review monitoring. It shows rankings of your app by hours to get a real-time view of the performance of your app compared with your competitors. And it monitors all your app reviews to reply directly and record bugs and suggestions quickly. Besides, it compares the ad revenue and spend through all the ad networks to help you make more money. For app users, it offers

14 comprehensive filters that match users needs from app name and price, to SDKs, demographics, performance, and so on. It even can combine different filters in one search by AND/OR to provide precise searching results whose views are customized to firstly present the apps that the users are mostly interested in. However, these systems have essential problems. Firstly, they do not solve the primary problems that causes app unpopularity and cannot improve the app quality radically. They just modify the external parts of an app rather than improving the app itself. Secondly, although they consider many aspects of an app, the app reports offered by these systems only focus on the app they are tracking, ignoring the other apps in the same category, especially the popular apps. The other related systems that help users choose appropriate apps are app recommendation systems. Traditional app recommendation systems like [11, 12, 10] aim to cater to user interest. Some of them try to find the relationship between user interest and app functionality, and recommend apps whose functionality matches user interest most, such as the systems proposed by Koren et al. [11] and Salakhutdinov et al. [12]. They encode user interest and app functionality into vectors and compute the distance between them. If the distance is short, the corresponding app will be recommended to the user. This type of recommendation system is successfully applied to online shopping [14], book [15], music [16] and movie [17] recommendation. But it is not appropriate for app recommendation because user interest towards an app is not only based on app functionality, but also from the app design, the advertisement contained in the app, the permissions requested by the app and so on. Some other recommendation systems like the one proposed by Bae et al. [10] analyze the user s app usage history to make predictions on user interest thereby recommend similar apps to the user. They propose a graph-based technology for app recommendation under Android OS and solve limitations exists in the previous recommendation systems, such as cold start and domain disparity. However, the information extracted from user usage history is limited. So such recommendation system offers no help when users facing hundreds of thousands of apps. Based on usage history, this system will recommend several apps over many app categories at one time. If a user want to search for apps under a certain category, the recommendations may not hit because of the small amount of training 7

15 data. Along with the exploration of app recommendation system, novel benchmarks for recommendation are proposed such as user privacy preference. Liu et al. put forward an app recommendation system that emphasizes the importance of user privacy preference, considering the fact that apps often access user s sensitive data and diverse user privacy preferences [3]. Besides to identify apps matching the user interest, it also measures whether the app s behavior will meet user s requirements of privacy. It assumes that there are two values used for app recommendation: one is called functionality match which denotes the degree of match between app functionality and user interest, and the other is privacy respect which shows the degree to which the app accords with user privacy preference. This system will recommend apps to a user which can strike a balance between functionality match and privacy respect. Zhu et al. also focus on user privacy preference and propose an app recommender system with security and privacy awareness [13]. Different from the systems we mentioned before, it is not a personalized system. It is not so much an app recommendation system as an app ranking system. In addition to app popularity which is usually measured by app rating and downloads, this system provides users with other evaluation metrics like security and hybrid of popularity and security. Further, when choosing security as evaluation metric, users can select different level of security to get different app ranking results under different granularities. These research proposes novel ideas on app recommendation. But the fact is that most of the users lack safety awareness. When receiving a permission request to access sensitive data from an app, most users may dismiss it and click allow. Thus, this type of system is unable to truly recommend appropriate apps to users. Yin et al. propose an innovative benchmark on app recommendation [4]. They treat apps as continuous consumption and assume that the apps which a user has already downloaded will prevent him from downloading similar apps. But the fact is that some users may have several similar apps on their phones. Thus, to predict whether a user will download a new app, they propose two values for apps: (1) the existing apps have satisfactory value which measures whether they can satisfy the user s need; (2) the apps in app market have tempting value which estimates the 8

16 satisfactory value of the candidate apps. This system regards app replacement as a contest between existing apps satisfactory value and candidates tempting value. If the tempting value wins, the system may recommend the user to download new apps, otherwise, no change is needed. There is another personal app recommendation system based on spatio-temporal app usage log developed by Han et al. [5]. Unlike the systems we discussed before which evaluate apps by discrete features, it suggests to recommend apps to users based on continuous random variables such as location and time. It recommends apps that are appropriate to user s current time and location. This system is actually a variation of the system based on user usage history. It treats app recommendation as a probabilistic problem, where (1) an app is actually a distribution of topics extracted from app descriptions; (2) the user s preference towards an app is also topic distribution impacted by time and location. It collects the user usage history and encodes app usage log into a series of topics as user preference, then it analyzes new apps and recommends them if they topically accord with the user preference at the current time and location. Similar to Han et al., Lin et al. also regard apps and user usage history as a set of topics [6]. While in the latter system, the same app may have more than one set of topics as the app is not a static item anymore. Apps are evolving continuously and new features are introduced in the latest version. Thus, each version of an app has its own set of topics which encodes its particular features. Features are extracted from app description and version introduction and encoded in new set of topics for the current version. The system builds personal profiles for users. After generating topic distribution for apps, it searches user profile to recommend apps which topically accord with user preference. If the topic distribution of an app and user preference have a lot of overlap, the app is recommended to the user. As a result, the app that was not appropriate for a user in the past may be recommended to the user after version update. Most users may think only explicit information like app ratings have impact on the apps that they choose. However, there is research shows that implicit information also affects mobile app users behaviors [18] and app usage [19]. Thus, apart 9

17 from time and location, Karatzoglou et al. also consider implicit mobile context information such as activity and social interactions to facilitate app recommendations [7]. They propose Djinn model: a novel context-aware collaborative filtering method for implicit data that is based on tensor factorization [7]. They give up explicit user feedback and build a model of user preference based on implicit contextual information. After that, this system will do as what the previous systems do: recommend apps that fits user interest most. Systems [4, 5, 6, 7] are personal recommenders and all are variants of recommendation system based on user usage history or something related to it. As we mentioned before, the biggest drawback of such system is the limited number of training samples. Thus, imagine that app users who are using this type of system with limited recommendations on a specific category, they are largely left on their own to select an appropriate app among countless apps. In a word, all the systems we discussed in this chapter take great pains to mine the relationship between the external factors of an app and user interests. They care about anything about an app but the user experience and the app per se. These systems fail to recommend apps that are commensurate with their rankings to users and externalize features of popular apps to app developers. 10

18 CHAPTER 3 BACKGROUND Android apps are published and downloaded in a package file format, Android Application Package (APK). Like ZIP format package, APK file is one type of archive file. It contains an app s manifest file (such as AndroidManifest.xml), resource files, and executable program code (such as.dex files) [20]. The manifest file declares the Android version that this app can be compatible with, the permissions requested by this app, the activities registered and so on. The resource folder contains the resource files needed by the app, like pictures, media files, and xml files that describe the layout of interfaces. The executable program code has all the classes that implement the functionality of this app. The APK file also contains files like the libraries included in this app. Such files are unrelated to our system so we omit them. Since the APK file only offer unreadable Dalvik Executable code, we need to de-compile.dex files to get a human-understandable code, smali code. Smali code is the intermediate code between Java source code and Dalvik Executable code. App components serve as the cornerstone of an Android app which comprise Activity, Services, Broadcast receivers and Content providers. Activity provides the entry point of an app for users to interact with. There is only one main activity in an app, but there can be as many activities as possible. Usually, each

19 activity has its own screen view, or interface. Service is used to conduct long-timerunning works. It runs in background and can also keep an app running in the background. Broadcast receiver helps an app receive the system-wide notices and make responds. Content provider manages shareable data stored in file system, SQL database and so on. In our system, we only analyze activities to generate the feature view graph for an app, since only activity offer interaction with users. When an app is launched, the main activity is first started which is announced in Android manifest.xml. Then the user may trigger certain events to switch to different activities. This activity switching is done by invoking some Android APIs like startactivity(), startactivityforresult() in activities. These methods take Intent as parameter which indicates the target activity. And the caller activity is usually the source activity. The layout file which contains view components and describes activity layout, is loaded to initialize user interface when an activity is created. This is accomplished by invoking Android APIs like setcontentview() in method oncreate() in activity. Hence, by statically analyzing the program code and tracking these specific APIs, we can extract activity switching graph from activities and generate feature vector for each activity from its layout file. Then the aforementioned activity switching graph and activity feature vectors will be integrated into one feature view graph which encodes the user interface and structure of an app. For the classification model, we will apply Graph convolutional network [9]. GCN is a scalable approach for semi-supervised learning on graph-structured data. Initially, it is proposed to classify nodes (such as documents) in a graph (such as a citation network), where labels are only available for a small subset of nodes [9]. It takes a N N adjacency matrix and a N F feature matrix of a graph as inputs and outputs a N L label matrix which contains the one-hot label for each node (Figure 4.6). But here we want to do graph classification since we treat an app as a graph. Heuristically, to achieve graph classification, we can regard each graph as a subgraph and integrate them into one graph and put this huge graph into GCN (Figure 4.7). The output label matrix still indicates node labels. However, the nodes from the same graph should have similar labels. Thus, we need to combine the labels of homogeneous nodes to get graph label. 12

20 CHAPTER 4 METHODOLOGY 4.1 System Overview Confronting with countless apps, users need an efficient and effective app quality evaluating system to help them choose appropriate apps. Unfortunately, the current app ranking systems applied by app markets have drawbacks in handling: (i) apps with abnormally high ratings and fake downloads; (ii) newly published apps with limited user feedback. Besides, the existing related systems, including app ranking system and app recommendation system, fail to solve the problem. As a result, it is necessary to introduce an innovative app quality evaluating system. This paper proposes AppGrader to evaluate app quality based on app itself rather than the external ratings, or downloads which can be forged. Our system extracts app static features directly from app code, such as user interfaces and overall app structure. This is inspired by our observation towards the way in which people give ratings and the result of app reviews analysis which will be detailed later. To represent an app by its UI and structure, we conceive of graph which is called feature view graph. In this graph, each node is an app interface. Edges indicate interface switching relationships. To better capture the app features, we introduce feature vector for each node which encodes both the activity features and app features.

21 To insure scalability, we extract app features and structure directly from app code by statically analyzing the de-compiled app code and tracking specific Android APIs and encode them into feature view graph. As for app quality grading, we will apply a classification model to cluster apps into different classes, where each class denotes one level of app quality. Here we apply graph convolutional network for classification. GCN is developed for handling graph-based data and fits our system well since in AppGrader, apps are represented as graphs. App 1 Code decompiler Smali code Resource files Feature view graph generator Feature view graph Graph convolutional network App labels App n Code decompiler Smali code Resource files Feature view graph generator Feature view graph Figure 4.1: System architecture Figure 4.1 shows the overall architecture of our system. AppGrader consists of three components: code de-compiler, feature view graph generator and graph convolutional network. We will talk about our system and its components in detail later. Aiming to further illuminate our methodology, we propose the following definitions: DEFINITION 4.1 (View) View is a user interface which a user can interact with. Each view is related to an activity where the methods handling user inputs and making responds are defined. It also associates to a layout file which declares the view components, such as widgets, shown in that interface. DEFINITION 4.2 (View Graph) View graph is essentially a directed graph G (V, E), where V is a set of vertices, each of which corresponds to a view, E is a set of edges v 1, v 2, such that v 1 V, v 2 V, and v 1, v 2 is a switch from view v 1 to view v 2 triggered by some event. 14

22 DEFINITION 4.3 (Feature Vector) Feature vector encodes the app code-level features and is assigned to each view in view graph. It includes: (1) view feature which encodes the view components contained in the current view; (2) app feature which stores app meta data. The events that trigger switches between views are regarded as edge features and usually are the event listener functions (e.g., onclick(), onlongclick(), ontouch(), etc.) that listen user generated events. However, the graph convolutional network we apply later cannot deal with edge feature. Therefore, instead of edge feature, we encode these event listener functions into the feature vector of source view. This makes sense since these functions that process user inputs can be treated as interaction functions which are stored in the view feature vectors. DEFINITION 4.4 (Feature View Graph) Feature view graph combines the view graph G (V, E) and feature vectors. It attaches a feature vector to its corresponding node in view graph. 4.2 App Code-Level Features Before processing data and evaluating apps, we need to determine what features we should extract from an app. As mentioned in chapter 1, we will preserve user interface and app structure in feature view graph. This is inspired by the observation that while giving ratings, users will make decisions based on their experience towards the app. Besides, we also conduct app review analysis to verify our idea and explore more possible code-level features User Experience of an App Android apps are user interaction intensive which requires user-friendliness as an essential factor of outstanding apps. The rating of an app is mainly based on user experience which is the interaction between user and app. According to our observation, the interaction can be divided into 4 stages, as is shown in Figure 15

23 User Interface 4.2. A user uses an app by traversing different interfaces and touching the widgets on the screen. Therefore, the Structure interaction between a user and an app is actually the interaction between the user and each interface, further, is the interaction between the user and the Functionality widgets the current interface. Hence, widgets are the cornerstone of an app which serve as the first stage of interaction. Then the user may switch to other interfaces and get to know the app structure (stage 2) Performance and overall functionality (stage 3). At last, some expert users may care about app performance (stage 4), such as CPU overhead and battery consumption. Button Listview Interface 1 Imageview.... Interface Interface n App CPU battery Webview User Interface Structure Functionality Performance Figure 4.2: Interaction between a user and an app The first two stages which a user can directly interact with, can be represented as static features extracted from the app code directly. The last two stages are dynamic features which cannot be taken into consideration in fast, large scale experiment. Thus, based on the first two stages, our system extracts the overall app structure and the design of user interfaces, such as the number of each type of widgets used, the number of colors, pictures and fonts applied, as app features. 16

24 4.2.2 App Review Analysis To verify our observation and figure out what factors else impact the rating a user gives to an app, we analyze 18 millions app reviews covering all app categories. We do both term frequency counting and topic mining on app reviews to find out what users are discussing about the apps so as to extract possible app features. First of all, we preprocess the raw data. We replace all the contractions in the reviews since contractions like don t are easy for human to understand but hard for computer. Then we break texts into tokens by general purpose English tokenizer provided by library sklearn. At last we remove the stop words. Words like articles and conjunctions that do not convey a content meaning as a token are known as stop words. Here we apply terrier stop words library [21]. Table 4.1: Terms indicating app features in top 500 high-frequency terms Term Frequency Term Frequency Term Frequency crashes bugs figure graphics photos camera screen access tap features text design crashing word internet feature view connection ads pics error crash picture connect touch wifi color pictures click battery button photo speed interface bug Before mining topics among these reviews, we do simple term frequency counting. There are over one million different words appearing in the reviews. Although we have removed meaningless stop words, in the result, there still are large amount of words irrelevant to our research, such as [ app, ], [ game, ], [ fun, ], [ awesome, ] (The numbers in the bracket is word frequency), and lots of nosies, like wrong-spelling words. After filtering out these irrelevant terms, we get the words related to app features in top 500 high-frequency words and show them in Table 4.1 (the whole table of top 500 high-frequency 17

25 terms is shown in appendix A). We divide the top 500 high-frequency terms into five classes: noun, verb, adjective and adverb, number and symbol, terms indicating app features. And the app feature is divided into static feature and dynamic feature. Figure 4.3 shows the proportion of the five classes of terms from which we can learn that the terms irrelevant to app features, such as verb (31%), adjective and adverb (23%), number and symbol (4%), account for 58% of the high-frequency words. And among the remaining nouns ( ), the words related to app features ( ) take up 12%. So, it is safe to say that the words in Table 4.1, or the app features indicated by these words, matter to users when they rate. While among these words indicating app features, there are terms, such as [ crash, ], [ bug, 73533] and [ battery, 50480], cannot be considered since they represent dynamic features which are too hard to capture and are not appropriate for fast, large scale experiment. adjective & adverb 23% numbers & symbols 4% non-featured noun 37% noun 42% verb 31% features 5% verb adjective & adverb numbers & symbols non-featured noun features Figure 4.3: Distribution of top 500 high frequency terms Then we mine the topics in app reviews by LDA model [22] which is one kind of topic model. It posits that each document is a mixture of a small number of topics and that each word s creation is attributable to one of the document s topics [22]. We set the number of topics to 50 (default setting) and list the 18

26 topics whose key words indicate possible app features in Table 4.2 (Here we only list the key words related to app features in each topic and put the whole topic mining result in appendix A). As we mentioned before, there are lots of irrelevant terms and noises in our dataset, so the topic mining result is not as much ideal as we thought. Besides, due to the huge size of our dataset and the limitation of computing environment, we did not conduct topic mining for many times to get the optimized number of topics, which is also a reason contributing to the unclear topics. Table 4.2: Topics indicating app features Topic Key words Frequency Topic 2 version photo latest Topic 11 update crashes problem crash video Topic 15 picture color light Topic 18 ads crashed database Topic 28 interface Topic 32 pic finger Topic 38 new graphics updated Topic 44 photos words Topic 49 pictures button Figure 4.4 shows the frequency of each topic, where y-axis denotes the topic frequency and x-axis denotes topic. The topics indicating app features, such as topic 2, 11, 15, 18, 28, 32, 38, 44, 49, are in solid fill. This histogram shows that: (1) topic 11 is the third popular topic in the 18 millions reviews; (2) topic 2, 18, 28, 44, 49 are average; (3) topic 15, 32, 38 are less considered by users. To emphasize the importance of these topics related to app features, we present the proportion of these topics in the total 50 topics in Figure 4.5, where these topics take up 16% of the overall app reviews. It seems that there is only nine topics related to app features that only account for 16% in the reviews, which is too diminutive to be convincing. However, there are facts that we cannot ignore. First, most of the users write simple reviews like good game, nice app which don t contain useful information about the app quality. Second, among the remaining analyzable reviews, users may not discuss about the same topic. Besides, among the possible app features, we only focus on the parts which, from our perspective, can be extracted directly from app code and are appropriate for app quality evalua- 19

27 tion. Thus, 16% is big enough to demonstrate the importance of the app features indicated by these topics topic 0 topic 1 topic 2 topic 3 topic 4 topic 5 topic 6 topic 7 topic 8 topic 9 topic 10 topic 11 topic 12 topic 13 topic 14 topic 15 topic 16 topic 17 topic 18 topic 19 topic 20 topic 21 topic 22 topic 23 topic 24 topic 25 topic 26 topic 27 topic 28 topic 29 topic 30 topic 31 topic 32 topic 33 topic 34 topic 35 topic 36 topic 37 topic 38 topic 39 topic 40 topic 41 topic 42 topic 43 topic 44 topic 45 topic 46 topic 47 topic 48 topic 49 Figure 4.4: Distribution of 50 topics among 18 million app reviews Ignoring the meaningless words like nice, game, think, lol, we synthesize the results of term frequency counting and topic mining and notice that users care about the part that they can interact with when giving ratings, such as the design of user interface (picture and color), the widgets used in an interface (button), permissions an app may request (wifi and camera) and the advertisement contained in an app (ads). This result accords with our observation. Although some app features like whether the app will crash while being used (crash) are also concerned by users, they are not considered since they are dynamic features and not appropriate for fast, large scale experiment. Based on our observation and the app review analysis, the feature vector for each view contains the following code-level features (The detailed feature vector is shown in appendix B): For each view (activity): the number of each type of widgets used in this view, the number of colors, pictures, fonts and interaction methods used in this view. 20

28 other topics 84% topic 2 2% topic 11 3% topic 15 1% topic 18 2% topic 28 2% topic 32 1% topic 38 1% topic 44 2% topic 49 2% other topics topic 2 topic 11 topic 15 topic 18 topic 28 topic 32 topic 38 topic 44 topic 49 Figure 4.5: Proportion of topics indicating app features in total 50 topics For each app: the number of each type of permissions requested by this app, the proportion of advertisement in this app, the number of views and switches between views in the view graph, app size and the number of files contained in this app. 4.3 System Architecture As shown in Figure 4.1, AppGrader comprises three components: code de-compiler, feature view graph generator and graph convolutional network. Code de-compiler de-compiles app APK file into human-readable smali codes and resource files for further analysis. Feature view graph generator analyzes resulting smali codes and resource files to extract feature vectors and view graph which are encoded into feature view graph. At last, graph convolutional network evaluates all apps feature view graphs and outputs a matrix indicating labels (level of app quality) of each app. The detailed explanations of these components are expanded in the rest of this section. 21

29 4.3.1 Code De-compiler The code de-compiler is actually the apktool [23] developed by Google for reversing Android APK files. Android apps are released and downloaded in APK format, which only contains unreadable Dalvik Executable code. For further analysis, the APK file should be de-compiled into human-readable codes by apktool. The resulting files include Android manifest file, smali codes, resource files and so on. Android manifest file stores app meta data and declares all registered activities in the app. Smali code, an intermediate code between Java source code and machine recognizable code, preserves Android specific APIs by which we can restore the app view graph and extract view features. The resource files include the layout files of each interface, pictures and other media resources used by the app Feature View Graph Generator To generate feature view graph, we start from generating app view graph and view feature vectors. In this graph, each node is an app view. Edges denote app view switching relationships. In terms of app view, it consists of activity and layout.xml file. Methods that process user inputs and make responds are defined in activity, including Android specific APIs which remain unchanged in different codes even written by different developers. Therefore, we can extract nodes (app views) and edges (view switches) by tracking Android APIs related to view switching in activity to restore view graph. Layout.xml file from which we can extract view features declares view components shown in its corresponding view. With regard to app features, we can explore the Android manifest file which records the app meta data Generate View Graph The view graph of an app presents all app views and view switching relationships. To restore view graph, our system searches and tracks the Android APIs related to app view switching. As mentioned before, a view has its corresponding activity 22

30 which is de-compiled in smali code and contains necessary methods for app running. Usually, an app may have more than one view. If we want to traverse among different views, we have to resort to Intent in activity which is used to deliver necessary information for component communication. Some of its construction methods which are used for activity switching are shown in Table 4.3. In Android coding level, activity is preferred than view therefore is used in the following discussion. Table 4.3: Intent construction methods Intent () Intent (Intent o) Intent (String action) Intent (String action, Uri uri) Intent (Context packagecontext, Class? cls) Intent (String action, Uri uri, Context packagecontext, Class? cls) As the last two construction methods in Table 4.3 show, there are two parameters related to the activity switching: Context which denotes the context of source activity that initiates the switching and Class which indicates the target activity. There are several Android APIs taking Intent as parameters used for traversing among activities, as listed in Table 4.4. Table 4.4: Android APIs for activity switching startactivity (Intent) startactivityforresult (Intent, n) startactivityifneeded (Intent, n) startnextmatchingactivity (Intent) startactivityfromchild (ChildActivity, Intent, n, Bundle) startactivityfromfragment (Fragment, Intent, n, Bundle) Since Android APIs remain unchanged in different codes even written by different developers, we can search and track these APIs to dig out the words indicating source and target activities thereby restore nodes and edges in app view graph. The pseudocode for generating app view graph is shown in Algorithm 1. Specifically, take startactivity() for example, we go through all activities and only keep the 23

31 instructions containing this API. Then for each instruction, we search for the nearest Intent construction method to it and analyze the key words indicating an activity name, such as String, to get the name of target activity. As for source activity, it is usually the caller of startactivity(). The method that invokes startactivity() is also recorded as edge feature and encoded in source activity s feature vector. Algorithm 1: Generating view graph Input: apk, de-compiled APK file func.list, a list of all Android APIs related to activity switching Output: ViewGraph, a file stores app view graph Function View Graph(apk, func.list): for each API i in func.list do src ret all instructions invoking i in all activities in apk for each instruction j in src ret do Go to the activity contains j source activity the current activity m the nearest Intent construction method of j target activity activity indicated in method m Add (source activity, target activity) into f ile V iewgraph end end return ViewGraph So far, what we introduced is explicit Intent, which indicates the target activity with certainty. However, sometimes app developers may use implicit Intent, such as the first four construction methods in Table 4.3, to do activity switching. Node and edge extraction based on implicit Intent are similar to that based on explicit Intent, with only little difference though. The third and fourth Intent construction method in Table 4.3 only specify action in an Intent. An action is used for informing the system to invoke appropriate activities and declared in activity intent filters which are announced in the Android manifest file. When implicit Intent is processed, it is the Android system that analyzes the action in Intent and chooses appropriate target activities by matching intent filter. The target activity can be either in or out of this app. If there are more than one candidate activities, the user has the right to choose. In light of our generator, it selects possible target activities 24

32 in the way of Android system and further generates an edge for each of them (all the edges end up to the same source activity). Especially, it generates one node others for all the target activities outside this app. The first method in Table 4.3 initializes an empty Intent object. In this case, we search methods such as setclass(), setcomponent() and setaction() around this Intent construction method rather than searching key words indicating activity name. With regard to the second method which copies an existing Intent object to a new one, we trace the old Intent object and do as what we talked before to get the target activity. What s more, fragment which is very similar to activity and usually embedded in one or more activities can also call these activity switching APIs, therefore will be extracted as a node in view graph. To avoid redundancy, the generator replaces all fragments in the resulting edges with their corresponding activities Generate Feature Vector Feature vector consists of view feature and app feature. The pseudocode for generating feature vector is shown in Algorithm 2. In terms of view feature, it involves widgets, colors, pictures, fonts and interaction functions used in a view. As what is done in generating nodes and edges of the app view graph, we also extract view features through specific Android APIs. To load widgets, colors, pictures and fonts in a view, APIs such as setcontentview() and inflate() are called in Android. Similarly, we go through the activity of the current view and keep all the instructions invoking these APIs to get the layout file of the view in which we can extract the number of each type of widgets used in the view, as well as the colors, pictures and fonts. Besides, there are other Android APIs in activity by which we can extract colors, pictures and fonts, such as settypeface(), setbackgroundresource(), setimagebitmap() and setcolor(). With regard to interaction methods, they are usually user-triggered events that listen user inputs and make responds, such as the ones shown in Table 4.5. What s more, as mentioned before, the events that trigger switches among views are parts of the methods listed in Table 4.5. As a result, it makes sense to store the edge feature as one dimension in the view feature vector. 25