Dynamic update of binary logistic regression model for fraud detection in electronic credit card transactions

Transcription

1 Dynamic update of binary logistic regression model for fraud detection in electronic credit card transactions Fidel Beraldi 1,2 and Alair Pereira do Lago 2 1 First Data, Fraud Risk Department, São Paulo, Brazil 2 University of São Paulo, Department of Computer Science, São Paulo, Brazil ABSTRACT In this paper we develop Dynamic Model Averaging (DMA) models regarding electronic transactions coming from e-commerce environment which incorporate the trends and characteristics of fraud in each period of analysis. We have also developed logistic regression models in order to compare their performances in the fraud detection processes. For the experiment, a dataset was provided by an e-commerce company in Brazil to develop the models and compare their results. 1 Introduction Regarding technological and economic development, which made communication process easier and increased purchasing power, credit card transactions have become the primary payment method in brazilian and international retailers. 1 In this scenario, as the number of transactions by credit card grows, more opportunities are created for fraudsters to produce new ways of fraud, resulting in large losses for the financial system. 2 Fraud indicators have shown that e-commerce transactions are riskier than card present transactions, since those do not use secure and efficient processes to authenticate the cardholder, such as personal identification number (PIN). 1

2 2 Methods Due to the fact that fraudsters quickly adapt to fraud prevention measures, statistical models for fraud detection need to be adaptable and flexible to change over time in a dynamic way. Fraud scoring models can be updated sporadically or continuously over time, which raises the question of dynamic update of the model parameters to detect fraud. Raftery et al (2012) developed a method called Dynamic Model Averaging (DMA) 345 which implements a process of continuous updating over time. The DMA methodology combines some existing ideas: weighted Bayesian models (Bayesian Model Averaging - BMA), Markov chains and forgetting factor in shaping state-space. All those characteristics make DMA models a better option for fraud scoring models. For the experiment, an e-commerce company provided the transactions data which have been performed by its payment system from July 2009 to January The data analysis was performed following a non-disclosure agreement, as recommended by PCI Data Security Standard. 6 Data analysis made possible to compare the DMA methodology against the classical logistic regression model, which is often used in fraud detection process. The following steps were taken in the experiment: Step 1 - Original dataset: it was created the structure for access to tables and fields to export data. The structure was made in the SQL database management system. Step 2 - Tables and fields selection: at this stage, all analyzed tables and fields developed along with the business team were selected and exported. At total, 28 tables were exported leading to 354 fields. Step 3 - Data filters and structure: all credit card transactions approved from July 2009 to January 2014 were selected. For this period, the records identified as fraudulent, either by cardholder or the company s internal analysis, were identified in the database. The entire selection of fields and tables relationship was performed. Thus, the originality of the data stored in the fields of exported tables could be preserved. Step 4 - Development of derived variables: based on the original variables, derived variables were created for the modeling process. 2/7

3 Step 5 - Final dataset for sampling: finally the final dataset of records and variables for sampling and modeling. Additionally, for validation purposes, some records in the database on the internal system of the company were selected in order to verify if the data of variables collected were equal to the original. The database for the experiment is composed of 7,716,09 records of credit card transactions, distributed from July 2009 to January For each extracted record, there are 52 independent variables (original and derived variables) and one dependent variable (fraud/non-fraud transaction). Denied transactions, regardless of the reason of negative, were excluded from the database. Thus, to the modeling process are considered only approved transactions. Fraudulent transactions were identified in two ways: through chargebacks information and the analysis of the company s internal team. However, both ways of detection resulted in a classification of fraud transactions in the final database. Transactions not classified as fraud were identified as non-fraud. For the modeling process, we sampled 428,256 records, with 22,615 fraud and 405,641 non-fraud transactions. This implies a ratio of about 1 fraud record for each 18 non-fraud transaction. This ratio of fraud and non-fraud is very close to the values adopted for this kind of problem, as noted in previous experiments. 789 After collecting the sample, for the modeling process, the sampled records were split into 80% (342,605) for model development and 20% (85,651) for validation. 3/7

4 3 Results Evaluating the performance of logistic regression model and DMA model separately on the table 1, the Stepwise Modified and the DMA 95 (λ =α = 0.95) showed the best performance indicators in its categories. However, in general, it is noted that DMA models have better performance for all indicators in relation to logistic regression models, except the detection rate, which has approximately a difference of 10% to compare the Stepwise and DMA 95. Models Indicators Logistic Regression DMA Model Stepwise Stepwise DMA 99 DMA 95 Modified (λ =α=0,99) (λ =α=0,95) Classification Performance Detection Rate 62,9% 66,2% 50,8% 56,5% Specificity 83,1% 83,9% 96,3% 97,9% False Positive Rate 16,9% 16,1% 3,7% 2,1% Ratio Non-Fraud/Fraud 4,8 4,4 1,3 0,7 Precision (Fraud) 17,2% 18,7% 43,2% 60,1% Precision (Non-Fraud) 97,6% 97,8% 97,2% 97,6% Model Performance KS 48,1% 52,1% 61,3% 70,4% Accuracy 82,1% 83,0% 93,9% 95,7% Area under ROC curve 81,7% 83,9% 88,3% 92,9% F-measure 27,0% 29,2% 46,7% 58,2% Bold values indicate the best results in the indicator. Table 1. Adjusted Models Performance. Figure 1 shows AUC values for the four models developed. Adopting the Stepwise for comparative purposes, the Stepwise Modified, DMA 99 and DMA 95 have relative performance of 3%, 8% and 14% 4/7

5 better, respectively. Figure 1. ROC curves of adjusted models. As we are interested in the detection rate and accuracy of developed models, we can use the F-measure to compare the performance among them. In the table 2, we observed which the DMA models showed better performance and, in particular, the DMA 95. Model F-measure DMA 95 (λ=α=0,95) 58% DMA 99 (λ=α=0,99) 47% Stepwise Modified 29% Stepwise 27% Table 2. Adjusted models F-measures. 5/7

6 4 Conclusion The experiment shows that DMA models present better results than logistic regression models in respect to the analysis of the area under the ROC curve (AUC) and F measure. The F measure for the DMA was 58% while the logistic regression model was 29%. For the AUC, the DMA model reached 93% and the classical model reached 84%. Considering the results for DMA models, we can conclude that its updating over time characteristic makes a large difference when it comes to the analysis of fraud data, which undergo behavioral changes continuously. Given all that, its application has been proved to be appropriate for the detection process of fraudulent transactions in the e-commerce environment. References 1. Bolton, R. J. & Hand, D. J. Statistical fraude detection: A review. In Statistical Science 17, (2002). 2. Chan, P. K., Fan, W., Prodromidis, A. L. & Stolfo, S. J. Distributed data mining in credit card fraud detection. IEEE Intelligent Systems 14, (1999). 3. Raftery, A. E., Karny, M. & Ettler, P. Online prediction under model uncertainty via dynamic model averaging: Application to a cold rolling mill. American Statistical Association 52 (2010). 4. McCormick, T. H., Raftery, A. E., Madigan, D. & Burd, R. S. Dynamic logistic regression and dynamic model averaging for binary classification. Biometrics (2012). 5. Madigan, D. & Raftery, A. E. Model selection and accounting for model uncertainty in graphical models using occam s window. American Statistical Association 89, (1994). Washington. 6. PCI-DSS. Payment card industry (pci) - data security standard (2013). URL pcisecuritystandards.org/documents. 7. Chan, P. K. & Stolfo, S. J. Toward scalable learning with non-uniform class and cost distribution: A case study in credit card detection. In Proceeding of the Fourth International Conference on Knowledge Discovery and Data Mining (1998). 6/7

7 8. Gadi, M. F. A. Uma comparação de métodos de classificação aplicados à detecção de fraude em cartões de crédito. Dissertação de Mestrado - Instituto de Matemática e Estatística da Universidade de São Paulo (2006). 9. Gadi, M. F. A., Wang, X. & do Lago, A. P. Comparison with parametric optimization in credit card fraud detection. Seventh International Conference on Machine Learning and Applications (2008). 7/7