User Behaviour and Platform Performance. in Mobile Multiplayer Environments

Transcription

1 User Behaviour and Platform Performance in Mobile Multiplayer Environments HELSINKI UNIVERSITY OF TECHNOLOGY Systems Analysis Laboratory Ilkka Hirvonen 51555K

2 1 Introduction As mobile technologies advance and more games and applications are designed for mobile devices, a need for methods to analyse data gathered by mobile application platforms has arisen. This study discusses this field and gives several examples of how statistical methods can be used to draw conclusions about user behaviour based on simple data gathered by server-based multiplayer application platforms. The focus is on multiplayer games. Due to the nature of mobile environment, games running on mobile platforms are subject to several restrictions. Because game device is mobile, the user can be physically at any location. Network failures might afflict the user and latencies can vary significantly. Compared to fixed networks latencies are larger and data transfer speeds are lower. Costs of data transfer may vary depending on the operator s tariffs. These factors, among others, need to be taken into account when designing mobile games and applications as well as analysing data gathered by mobile platforms. There have been studies on game platforms and user behaviour in fixed network environments, but this study transfers the focus to mobile environment and mobile game devices. (See Henderson et al. 2001, McCoy et al. 2004) The objective of this study is to find patterns in user behaviour and to give recommendations how these patterns can be used for marketing purposes and for improving mobile game design in order to develop games that better serve these segments. The objective can be divided into two sub-objectives: 1. Segmentation of users; Importance of this analysis stems from the needs of marketing and game design. Information on user segments can be used directly to focus marketing efforts as well as for designing more attractive games that conform to the characteristics of different segments. Users behavioural patterns are rendered from characteristics of users play sessions by means of simple statistical key figures and autocorrelation analysis. 2. Forecasting future server load; Estimates of future server loads help optimise server capacity and avoid unnecessary performance problems. First some useful statistical methods are described. After that, applications of those methods are sketched. Focus is on simplicity of implementation of the methods, as the analysing

3 software may be running real-time. Finally, a case example is presented. Multiplayer Publishing Environment (MUPE) is an open-source application platform for context-aware multiplayer mobile applications (see The platform is released under the Nokia Open Source Licence (NOKOS). From a data set logged by MUPE some key figures are calculated with intention to segment users and to keep track on e.g. cumulative amounts of data sent per user and average latencies the users are experiencing. All this information can be used to develop better games and other multiplayer applications. Also, if the available data is adequate and the internal structure of the application is known, conclusions about e.g. optimal game strategies can be drawn. The data could well be used for forecasting purposes and for analysing platform s performance. However, these areas are beyond the scope of this study, but may provide fruitful topics for future research. 1.1 Objectives of segmentation User segmentation is the process of dividing users into different groups depending on a particular user s position on certain segmentation dimensions, for example how frequently the user is using the mobile application (Kotler and Keller, 2005). Two partly interrelated objectives of segmentation discussed in this study are (i) (ii) segmentation for marketing purposes, and segmentation for game design purposes. For companies that develop or offer mobile games marketing costs are usually significant. To efficiently focus marketing efforts it is imperative to have a good understanding of user segments and their characteristics. One dimension on which users can be segmented is temporal, i.e. for how long, how often and when the users are using the applications. Information on the user segments is also useful for game developers. Game design is very different if the users were to play a game for one minute ten times a day, compared to a game that is played only a few times a day but for longer at a time.

4 Answers to these questions can be mined from the data gathered by the application platform. Methods depend on the structure and content of the data; in this study sets of statistical key figures, autocorrelations and covariances are used. 1.2 Objectives of forecasting The number of users of mobile applications is not constant around the day and on different days of the week. Server loads also differ respectively. Therefore, forecasting the number of users online (or value of any similar variable that affects server load significantly) may prove useful when deciding how much server capacity should be reserved for the mobile applications. Lack of sufficient server capacity would appear as higher latencies and as a generally unsatisfying user experience. 2 Elements of logged data In this section some variables that provide useful input for data analysis are proposed. Without information carried by these variables relevant results are difficult, if not impossible, to be derived. Typically an application platform writes down data whenever events caused by users or the system occur. For simplicity, it is assumed that the platform (the server) logs data every time a user triggers a method call. When a method call occurs, the platform writes down a set of values of different attributes or variables. By mining this data it should be possible to distinguish some fundamental key elements or objects, at least different users and the users play sessions with some internal characteristics. User is an entity that uses a certain application that runs on the platform. Usually users are human beings, though artificial intelligence technologies could in some cases be considered as a user. A play session belongs to a single user. Also with multi-player games each user has its own play sessions; naturally, different users play sessions can overlap. A play session has a beginning and an end, which together determine the duration of the play session. During the play session user causes events and these different events shape the characteristics of the play session in question.

5 Life cycle of a play session is a series of three states. This way, the start and end criteria of a play session can be defined according to the requirements of the case in question. State Trigger START A certain method call indicating a new play session Any method call from a new user SESSION Automatically from START state END A certain method call indicating an end of the session No method calls in SESSION state for N minutes (e.g. for 5 minutes) Loss of connection Figure 1 Play sessions of four different users. Play sessions have a certain duration and some of the play sessions might overlap with other users'sessions. (Source: Riku Suomela, Nokia Research Center)

6 This study is restricted to platforms and contexts where it is logically relevant to split the logged data into users and play sessions of these users. 2.1 Required variables For the analysis, each data set stored by the logger should contain at least values of the following attributes: - User identification number; to identify the originator of the method call and to which play session the data set belongs, - Timestamp of the method call; to determine the time when the method call was initiated, - Server processing time; how long it took for the server to process the method call, - Latency; the delay between making the previous method call and the time user receives a reply from the platform for that method call, - Size of the message sent by the user, - Size of the response sent by the server, - Name of the method called. Unique identification numbers are used to distinguish different users. Latencies provide evidence of how real-time and fluent the user s game experience has been. Long latencies are generally unpleasant to the user. Message sizes indicate how much data has been transferred over the network. This is interesting because i) it is one of the factors showing how much load there is on the network and ii) operators providing networking services usually bill according to the amount of data transferred. It is also assumed that there are plenty of data sets available. With very limited data, it is not reasonable to perform some of the analysis or, with insufficient data, the results might at least be biased. For example, time series analysis is not considered applicable with series shorter than 100 periods.

7 3 Methods of data analysis 3.1 Statistical key figures A set of simple statistical key figures to describe a play session (or a set of play sessions) on a general level could include at least - arithmetic mean, - variance or standard deviation, - correlations, and - minimum and maximum values. The logged data is divided into finite sequences, one for each of the stored parameters such as latencies, processing times and data packet sizes. The previously described key figures are calculated from these sequences with formulae presented in Laininen (1998). Arithmetic means are used to find sequence s average value, e.g. average length of a play session. Variances or standard deviations show how constant the values in the sequence have been. Correlation coefficient is used to find interdependencies between variables. Minimum and maximum values of a sequence show the range or peak values of the sequence. Taken these statistical key figures, appropriate variables give information on the distinguishing characteristics of users play sessions, thereby depicting users behaviour as well. 3.2 Autocorrelations A time series is a sequence of data points with uniform time intervals, for example daily temperature measurements, opening prices of a share of stock or amount of data transferred per hour over a two-month period. With appropriate methods the future values of these data points can be predicted. (Pindyck and Rubinfeld, 1998) In this case autocorrelation is the correlation of a discrete time series with its time-shifted version. It is usually calculated for different time shifts (lags) and plotted as a bar graph (histogram) with lags on the x-axis and the correlations on the y-axis. The time series has to

8 be stationary, meaning that the probability density function of the random variable of which the series consist of does not change over time. Autocorrelations R(k) of a time series X t, k being the lag, are calculated with the following formula: [( µ )( µ )] E X ( ) = i X R k i+k. 2 σ The formula is similar to normal correlation coefficient, the only difference being that variable Y is replaced with variable X s time-shifted version. Autocorrelations can be used to identify repeating patterns in time series. High correlation on a certain time shift indicates a repeating pattern occurring with respective intervals. A heart rate would be a good example of this phenomenon: if the patients heart rate is 120 there would be a peak in the autocorrelation graph at 0.5 seconds. High but declining values in the beginning of an autocorrelation graph tell that changes in the original series are rather small and the new values are quite near to (i.e., highly correlated with) the previous ones. 3.3 Time series forecasting Time series forecasting is an attempt to predict future values of data points of a time series. To obtain justified estimates of future values one can use a mathematical model based on previous values of data points. Previous data points contain valuable information on the underlying process and that information can be used to select and fit an appropriate mathematical model. The process of analysing the previous data points and finding the appropriate model is called identification. (Pindyck and Rubinfeld, 1998) There are two widely used classes of forecasting models: autoregressive models (AR) and moving average models (MA). These two models can be combined, producing a family of ARMA models. ARMA(p,q) refers to an ARMA model with an autoregressive part of order p and a moving average part of order q. For some applications the ARMA model can be, if needed, developed further by adding a seasonal part (SARMA) and an integrated part (SARIMA) to obtain stationarity. However, those models are not covered in this study.

9 Autoregressive model of order p is simply a sum of a constant c (sometimes the constant is omitted), weighted sum of previous values X t-i and an error term t : X t p = c + φ i X t i + ε t. i= 1 Autoregressive model is a linear regressive model which can be fitted to data in hand with a least squares algorithm. Moving average model, linear as an AR(p) model, of order q is written as X t q = ε + θ ε, t i= 1 i t i where t, t-1, are error terms. By combining these two models we obtain an ARMA(p,q) model: p i= 1 t i + X t = ε + φ X θ ε. t i q i= 1 i t i The variables 1, q, and 1,, p are the parameters of the model. The error-minimising values of these parameters are usually found with a least squares algorithm; this is called model fitting. Choosing the correct orders p and q are often crucial for fitting the model. As a rule, the smallest values of p and q that produce adequate fit to the data are chosen. In this context seasonality is highly probable. It is quite clear that there are more users playing the games during daytime than at night or very early in the morning. That characteristic in mind, the data might probably require a seasonal ARIMA model which uses, in addition to the parameters described above, seasonal components with seasonal lag of S. An educated guess for the length of the season would be 24 (hours). More information on advanced time series forecasting and fitting the models can be found in the literature. Tools for time series analysis can be downloaded, e.g., from the home page of U.S. Census Bureau (see References).

10 4 Applications of data analysis methods 4.1 User segmentation Sets of statistical key figures One of the simplest ways to analyse data stored by a data logger might be to implement a feature that calculates sets of statistical key figures. From a programmers point of view this should not be a time-consuming task. Sets could include key figures calculated from either variables related to user s play sessions (user behaviour) or from variables related to server s performance. In these cases the figures could be exploited for example as follows: - Arithmetic means and standard deviations (or variances) could be calculated to obtain average values for server processing times, latencies and sizes of messages sent between users and the server. In this case e.g. large deviation of latencies could be a sign of an unreliable network connection. - Correlations between e.g. latencies, message sizes and server processing times might reveal useful information about the internal performance of the application. Also correlation between the number of users online and latencies and/or server processing times could prove useful information about server s capability to handle larger strain. - Minimum and maximum values can be used to find peak values, for example largest latencies users have experienced; this value among average latencies can be used as one method to measure users perceived quality. - Average number of user s method calls per play session or number of play sessions per day; according to this information the user could be classified e.g. as a heavy user or a random user Temporal segmentation with autocorrelations Besides their important role in identifying time series models, autocorrelations can be used independently to find certain characteristics of user behaviour. In this section a way to use autocorrelations for temporal segmentation of users is described. Users are segmented depending on the time of day when they use the application: in the morning, during afternoon

11 coffee breaks, in the evening or evenly during the day. Numbers of method calls triggered by a user per hour are used to conclude how active the user has been during that time of day. As stated in previous sections, a time series is a sequence of data points with uniform intervals. The problem in this application is that users do not trigger method calls regularly with uniform time intervals (Figure 2). Therefore, values of the attributes stored by the data logger cannot be handled as time series without manipulating the data properly. One way to obtain a feasible time series from the data at hand would be to count the method calls or cumulative values of variables during some limited time intervals. For example, total data transferred per one hour or number of method calls of a user triggered in one hour could be used as sources for data points (Figure 3). Figure 2 Uneven distribution of method calls. Method calls occur randomly instead of uniform time intervals, and thus require data manipulation to form a feasible time series.

12 Time Figure 3 Number of method calls as a histogram. Number of method calls during a certain time interval (one hour in this example) has been counted and visualised as a histogram. This data can now be used as input for analysis methods. With a small number of users one can draw conclusions visually from histograms, but a more useful way would be to analyse the data automatically using autocorrelations. For this analysis to be reliable, the amount of data has to be large enough. Practically more than 20 days should be enough, but with more data the analysis will be more accurate. The steps to implement a segmentation feature to an analyser application are as follows: 1. For each user, count the total number of method calls during a constant time period, for example one hour, over the total observation period, for example 30 days, thus obtaining a time series with 24 * 30 data points (time series A). User identification numbers and time stamps of the method calls are needed complete this task. 2. For each user, calculate autocorrelations of the time series A for lags between 1 to 24.

13 3. Analyse the autocorrelations: a. If there is a significant peak around the lag 24 the user is using the application daily around the same time of day. What is considered to be a significant peak can be determined e.g. by comparing single autocorrelations to their arithmetic mean or by selecting a predetermined threshold value. In that case, e.g. values over 0.50 are considered significant. (See Figure 4.) b. If autocorrelations seem to be nearly same for all lags the user is using the application randomly during the day and the user is segmented as a random user. Now skip the following steps and go to step 2 to analyse next user. 4. To find out the time of day which caused the high autocorrelation, count the total numbers of method calls that have occurred during a certain hour of day over the whole observation period. The result is a series of 24 data points (time series B). 5. If the values of the data points before noon are larger than the values in the afternoon or in the evening, i.e. if their difference from the arithmetic mean is more than a certain predetermined threshold, then the user is segmented as a morning user (or afternoon user or evening user respectively).

14 Correlation 1 0,9 0,8 0,7 0,6 0,5 0,4 0,3 0,2 0, Lag Figure 4 Autocorrelations of a user s time series A (from simulated data). Correlation is significantly high around lag 24, which implies that there is a pattern occurring with an interval of 24 hours. In other words, the users seems to be sending method calls regularly around the same time of day. Deeper inspection of autocorrelations can reveal more information on user s behaviour. For example, if the user is using the application twice a day, first around 7-8 a.m. and then again around a.m., there would be, in addition to lag 24 s peak, two slightly lower peaks around lags 3 and 21. However, finding and deciphering this information might be too complicated, but together with other means of analysis could give useful results Two users simultaneous play sessions To find out if two players play sessions are often overlapping an accurate algorithm could be designed. However, while that kind of algorithm might be rather easy to design, it may be time-consuming to execute due to complexity issues. Comparisons that need to be done will be (n 2 n) / 2, where n is the number of users. On the contrary, calculating covariances is

15 quite a simple task. In that sense, a statistical method is more desirable, albeit slightly less accurate. Covariance is calculated between two users time series As (from step 1 in the previous list). If the covariance is significant (larger than a certain threshold), the users are often sending similar numbers of method calls around the same time of day, which may imply that they are using the application simultaneously to play multi-player games together. However, there is a risk of false results because of at least two factors: The users may play the games with different strategies. User A s strategy might demand sending a lot of method calls, while user B is sending less method calls. In that case covariance is small and the simultaneity would be overlooked. Counting play sessions instead of method calls could reduce this risk. High covariance coefficient between two users time series does not automatically imply that they play the game together, even though they are playing the game around the same time of day. A combination of statistical analysis and a tailored algorithm with required built-in features of the platform s data logger would give optimal results. 4.2 Forecasting There are several elements of mobile games and user behaviour that can be forecasted. These elements include e.g. amounts of data to be transferred in the near future or number of players online on the next Saturday evening. As the process of selecting the appropriate model and fitting it is very complex, it might prove difficult to implement a built-in time series forecasting feature to the analyser. A more useful way to exploit time series forecasting would be to use ready-made statistical applications with time series features. The ready-made application with built-in tests can be used to select the model. If the structure of the time series seems stable enough over time, it may be appropriate to use the same model constantly once it has been identified. In that case it would be enough to fit

16 the parameters again when needed. If the re-selection of the model is unnecessary, it would be in within the limits of reasonable amount of programming work only to implement a least squares fitting algorithm to obtain new parameter values and to run it regularly, for example once a week to get the next week s estimates. More information on least squares algorithms is available in literature. 4.3 Case MUPE MUPE stands for Multiplayer Publishing Environment. MUPE is an open source platform for context-aware multiplayer mobile applications. It is released under the Nokia Open Source Licence (NOKOS). The model described above is customised and used to evaluate usage of MUPE. The data logger of MUPE stores a line of text every time a user sends a method call. The variables stored are timestamp, user identification number, server processing time, lag of the previous method call (-1 if the method call was user s first one), size of the message sent by the user and size of the response message Suggested analysis methods Useful methods of analysis include at least statistical key figures and autocorrelation analysis. There are two goals: the first goal is to get a view of single users play sessions and to be able to compare them to one user s previous play sessions and other users play sessions in general. The second goal is an attempt to segment users based on the time of day they are usually playing mobile games. The set of statistical key figures described in section 3.1 Statistical key figures is directly applicable to case MUPE. For one play session the set could include Start time, end time and duration Total amount of data sent and received during the session Highest, lowest and average latency and standard deviance of latencies

17 Highest, lowest and average server processing times and their standard deviations By comparing these sets of statistical key figures game developers can draw conclusions to some extent about users perceived experiences of quality and users behaviour. For all play sessions the set could include Start time, end time and duration of the observation period Number of players Number of play sessions Average number of play sessions per user and average time between play sessions Minimum, maximum and average values of latencies, server processing times etc. Average amount per session and total amount of data sent and received Correlation between number of open play sessions and server processing times and/or latencies Analysing these sets might provide useful information on both performance of the MUPE platform and users behaviour. Autocorrelation analysis and covariance analysis, as described in previous sections, can be exploited as such to segment users. However, it is crucial that there is enough data available for the analysis. Recommended minimum duration of the observation period is at least one month. Without enough data the results cannot be considered reliable. 5 Discussions The methods described in this study are not yet tested and validated with real data and therefore should be considered as a suggestion how this field of study could be approached. Further, autocorrelation and covariance analysis are not absolutely exact methods and results obtained from them are merely approximate; this should be noticed when exploiting the results.

18 Time series analysis requires an external application for the model selection and fitting process. The process is rather time consuming and hard to automate or to implement into an analyser application as such, but performed manually every once in a while it may give useful results. Regression analysis could also provide sound estimates of future server load. Most importantly, selection of variables that are stored has a strong impact on what analyses can be performed. Many of the calculations could be replaced with built-in features of server s data logger. For example, keeping record of online users directly by the server itself might require additional computing capacity, but that way exact results could be obtained. However, if the data logger is rather simple and all information must be derived from very limited data, the methods described in this study might well give adequate results.

19 References Multi-User Publishing Environment (MUPE), official web site, (Link verified on December 21, 2005) Laininen, Pertti (1998). Todennäköisyys ja sen tilastollinen soveltaminen. Otatieto, book number 586. Pindyck, R.S. and Rubinfeld, D.L. (1998). Econometric Models and Economic Forecasts. McGraw & Hill, New York. U. S. Census Bureau. The X-12-ARIMA Seasonal Adjustment Program, available at (Link verified on December 21, 2005) Kotler, P and Keller, K.L. (2005). Marketing Management, 12 th edition. Prentice Hall. Henderson, T. and Bhatti, S. (2001). Modelling User Behaviour in Networked Games. In Proceedings of ACM Multimedia, Ottawa, Canada. McCoy, A., Delaney, D., McLoone, S. and Ward, T. (2004). Towards Statistical Client Prediction Analysis of User Behaviour in Distributed Interactive Media. In Proceedings of the 5 th Game-On International Conference, MS Campus, Reading, UK.