Spam Detection ECE 539 Fall 2013 Ethan Grefe. For Public Use

Transcription

1 Detection ECE 539 Fall 2013 Ethan Grefe For Public Use

2 Introduction is sent out in large quantities every day. This results in inboxes being filled with unwanted and inappropriate messages. These spam s often have very similar characteristics allowing them to be detected using various machine learning algorithms. Detecting and removing spam from inboxes saves people time and frustration. Some of the most effective existing spam filters use Naïve-Bayes Classifiers and Support Vector Machines (SVMs) in order to detect spam. Naïve-Bayes looks at how frequently certain words are found in spam and non-spam s. It then determines the probability that an is spam based off of these frequencies. Classifiers that use SVMs take features from an that typically differ between spam and non-spam. s that have been classified by an expert are used to create support vectors. The resulting SVM is used to classify future s as spam or non-spam. Work Performed For this project, I decided to design an SVM classifier for detecting spam. and testing data for the project was taken from the CSDMC2010 SPAM corpus found on From this dataset, I only used the pre-classified s from the TRAINING directory. This directory contains 4327 eml files out of which there are 2949 non-spam messages (HAM) and 1378 spam messages (SPAM). The dataset also came with a python script for extracting the subject and body of each . To create an SVM classifier for these s, I first needed to extract features to differentiate HAM and SPAM . I wrote a Java class to extract four different features from each , then to write these features along with their corresponding label on a single line of the file features.txt. After examining the sample s and researching typical characteristics of spam, I decided to extract from each the percentage of letters that are capitalized, the percentage of punctuation that uses exclamation marks, the amount of HTML usage, and the average length of words. Other features were also tested, but did not yield productive results. After extracting these features, I created an SVM in the file spamsvm.m to classify them using Matlab s svmtrain function. The function svmclassify was then used to obtain classifications for the testing data. Various types of kernel functions were tested with the radial basis function performing most effectively. I used varying percentages of the data for training and testing with the data sorted at random each run. Best results were found when about one quarter of the data was used for training and three quarters for testing. Results Using individual features to train SVM: Feature 1 - letters that are capitalized

3 RBF 35.00% 31.62% 89.15% 60.39% Linear 35.00% 33.61% 87.17% 60.39% Quadratic 35.00% 29.83% 90.71% 60.27% Feature 2 - punctuation that uses exclamation marks RBF 35.00% 65.39% 73.70% 69.55% Linear 35.00% 46.42% 86.32% 66.37% Quadratic 35.00% 49.60% 83.60% 66.60% Feature 3 - Average length of words RBF 35.00% 53.19% 84.39% 68.79% Linear 35.00% 2.66% 99.92% 51.29% Quadratic 35.00% 47.94% 87.20% 67.57% Feature 4 - Amount of HTML usage RBF 35.00% 53.66% 96.92% 75.29% Linear 35.00% 46.50% 96.32% 71.41% Quadratic 35.00% 53.14% 97.10% 75.12% Using all features to train SVM: RBF 5.00% 78.45% 91.92% 85.19% RBF 15.00% 79.88% 91.93% 85.90% RBF 25.00% 79.16% 92.99% 86.08% RBF 35.00% 80.06% 92.33% 86.20% RBF 45.00% 82.28% 83.00% 82.64%

4 RBF 55.00% 80.43% 91.82% 86.13% RBF 65.00% 85.10% 71.52% 78.31% RBF 75.00% 84.81% 80.74% 82.77% RBF 85.00% 86.48% 71.01% 78.74% RBF 95.00% 85.17% 77.38% 81.28% Linear 5.00% 71.42% 90.55% 80.99% Linear 15.00% 76.69% 82.40% 79.54% Linear 25.00% 80.65% 72.71% 76.68% Linear 35.00% 78.69% 80.66% 79.67% Linear 45.00% 83.27% 62.52% 72.90% Linear 55.00% 95.90% 16.00% 55.95% Linear 65.00% 94.57% 21.07% 57.82% Linear 75.00% 96.78% 15.35% 56.07% Linear 85.00% 96.50% 15.10% 55.80% Linear 95.00% 97.05% 14.52% 55.79% Quadratic 5.00% 75.61% 94.71% 85.16% Quadratic 15.00% 76.99% 94.54% 85.77% Quadratic 25.00% 83.18% 74.52% 78.85% Quadratic 35.00% 82.75% 84.85% 83.80% Quadratic 45.00% 86.50% 64.22% 75.36% Quadratic 55.00% 89.88% 53.43% 71.66% Quadratic 65.00% 89.30% 54.13% 71.72% Quadratic 75.00% 88.70% 61.33% 75.01% Quadratic 85.00% 93.88% 36.09% 64.98% Quadratic 95.00% 96.62% 23.75% 60.18% Discussion The type of kernel used seemed to very dramatically help or hurt results. Although the data seemed linearly separable to some extent, the use of a Quadratic or RBF kernel function improved the results very notably. The amount of data used to train also seemed to affect the results more than I had initially expected. The best balance between successful spam classification and ham classification seemed to occur when about 35% of the data, or about 1514 feature vectors, were used. The use of less data than this may have resulted in not enough data to clearly differentiate spam and ham . The use of more may result in over fitting. More ham s than spam s were always used to train the SVM. A better ratio may result in somewhat better classification of testing data. After reviewing my classifier s results, I have concluded that training on approximately 35% of the data and using the radial basis function for the SVM s kernel produces the best results. This resulted in approximately 92% of HAM being correctly classified and 80% of SPAM being correctly

5 classified. Additional features may help this classifier yield even better results, but the additional features tested thus far have not produced useful results. While all of the features are correlated to extent, each of them seems to add some amount of additional information. The above results show classification using individual features gives a classification rate between 60% and 75%. The combination of all of these features has given me the best results. Removing any of these features hurt the overall classification rate. In the future, improvements may be made to this classifier by using additional features. Word frequency is a very commonly used feature that I would like to examine using a Naïve Bayes classifier. The data extracted thus far could easily be used as input to a number of other algorithms. I would like to try using my current set of features in other classifiers such as K-NN and MLP. Provided these classifiers produce meaningful results, the resulting classifications could also be used as inputs to a Mixture of Experts classifier. There are a great number of possible approaches to the spam detection problem. For those who are interested, there is a great deal of research that can be done surrounding this problem.