BUPT at TREC 2006: Spam Track

Transcription

1 BUPT at TREC 2006: Spam Tac Zhen Yang, Wei Xu, Bo Chen, Weian Xu, and Jun Guo PRIS Lab, School of Infomation Engineeing, Beijing Univesity of Posts and Telecommunications, , Beijing, China Abstact. This epot summaizes ou paticipation in the TREC 2006 spam tac, in which we conside the use of Bayesian models fo the spam filteing tas. Fistly, ou anti-spam filte, Kidult, is biefly intoduced. And then we ty to use weighted adjustment of sepaating hypeplane and selective classifies ensemble to impove the filteing pefomance. Finally, we summaize the elevant esults fom the official evaluation. 1 Intoduction In 2005, a new tac on spam filteing was intoduced to TREC, whose goal was to povide a standad evaluation of cuent and poposed spam filteing appoaches. The 2006 tac epises the 2005 expeiments with new filtes and data, and also investigate delayed feedbac and active leaning. [1] Thee ae two tass: 1. Online filteing - enhancement to TREC 2005 tas; 2. Active leaning - completely new tas. In this yea, we focus on the online filteing tas. Fo the pilot tas, active leaning, the only diffeence is that the un.sh is eplaced by the active leaning shell active.cpp with andom selection in the jig. So the emainde of this pape is stuctued aound online filteing tas. Section 2 outlines an biefly oveview of the idult anti-spam famewo. Some impovements ae poposed in Section 3. In Section 4, we summaize the elevant esults fom the official evaluation. The majo conclusions that can be dawn fom the evaluation ae pesented in Section 5. 2 System Oveview The idult is an anti-spam solution with self-dependence intellectual popety, which is developed by Pis Lab of Beijing Univesity of Posts and Telecommunications. The esulting technology of idult has been successfully eleased in ou TREC 2005 and TREC 2006 spam tac system [2]. The pocessing pocedue of the idult system is same as the geneal pocessing famewo of TREC 2006 spam tac. Ou system uses Bayesian models fo classification. The Bayesian classifie is a pobability based appoach, which is often used in text classification applications and expeiments fo its simplicity and effectiveness. The following subsections descibe ou methods in geate detail. 2.1 Pepocessing

2 Some common o often poposed initial tansfomations ae: looalie tansfomations, HTML deobfuscation, MIME nomalization, chaacte set folding, case folding, wod stemming, stop wods list, featue selection [3]. Discussed in ou 2005 spam tac epot [2] and CRM114 s notes [4], it would be fa bette if the leaning machine itself eithe made these tansfomations automatically o used all the featues. In this liteatue, in this wo, we only use HTML deobfuscation and MIME nomalization. 2.2 Chinese Wod Segmentation Usually, the basic unit fo text pocessing is wod. It is natual fo English, but fo Chinese language text, wods ae not demacated in a sentence. Thus, wod segmentation must be pefomed fist in most natual language pocessing (NLP) applications, which is necessay but time-consuming. We adopted a POC-NLW based HMM segmente, as descibed in [5], to implement the pepocessing of the context of an . Howeve, in ode to meet the constaints on pocessing time, only a simplest segmentation model was used, which was a puely chaacte-level tagge based on the POC-NLW template without any wod-level infomation. This model only need to load fewest featues and the loading can be accomplished in fa less than one second, while othe moe complex models cost a few seconds on featue loading. Howeve, this simplification may lead to decay on the oveall pefomance. As pesented in [5], detailed expeimental esults show that such a simplified model pefoms much wose than those complex ones. 2.3 Toenization Usually, the wod is used as the basic pocessing unit. The basic idea is to bea of the input text steam into a seies of toens. The boost [6] Toenize pacage povides a flexible and easy to use way to bea of a sting o othe chaacte sequence into a seies of toens, by which we can choose how the sting gets boen up using diffeent Toenize function. In this wo, we bea up the input text sting based on a supeset of comma sepaated value lines (such as space, punctuation, customize escaped list sepaato and offset sepaato). 2.4 Naive Bayes Spam Filteing Famewo The Bayesian classifie is a pobability based appoach, which is often applied to text categoizations tass. Fo spam detection, suppose each instance M is descibed by a conjunction of wod attibute values < w1, w2,..., wn >. And L is the numbe of taget classes ( Ci, i = 1,..., L). The basic concept of Bayesian classifie is to find whethe an is spam o not by looing at which wods ae found and which wods ae absent fom the message. In the liteatue, the Bayesian appoach to the new is to assign the most pobable taget label: HMAP = ag max P( Ci w1, w2,..., wn ) i L. = ag max PC ( ) Pw (, w,..., w C) i L i 1 2 n i To maes the estimation of paametes tactable, the Naive Bayes assumption is used, which suppose that the attibute values ae conditionally independently, then H = ag max PC ( ) Pw ( C). (2) NB i i i L (1)

3 Fo the situation of spam detection, attibute values < w1, w2,..., wn >is the wods in one message (fo Chinese copus, wod segmentation is needed), whee L is the numbe of taget classes C i (e.g. C + spam/c - ham). In pactice,log-lielihood is computed as following:. (3) scoe(m)= log PC ( ) + log Pw ( C) (log PC ( ) + log Pw ( C)) + + Theefoe, if scoe(m) > 0, the will be assigned to C+, and C- othewise. In ou expeiments, n-gam model shows good pefomance. But with the incease of n, n-gam suffeed fom data spaseness and ealtime limitation, which maes highe ode model cannot be used in ou submitted systems. 2.5 Add-One Smoothing Algoithm and Kill-One Stategy The statistical appoaches fo spam filteing ae often Bayesian and seveal distibution models (such as multi-vaiants Benoulli model, Poisson Naive Bayes model, and the multinomial model) ae assumed. The diffeence between these models is the ways of calculating P(w C i ). In this wo, multinomial model is used fo its supeio pefomance [7]. One benefit of the multinomial appoach is the numbe of available smoothing methods to handle unhappened toens. In Bayesian models, accoding to the pinciples of symmety, the toens have no othe chaacteistics in addition to the numbe of toen. Then toen with the same counte has the same pobability value. Suppose n is the numbe of special toen occued as often as in taining copus. N is the total numbe of toens, then: n = N (3) Based on the Maximum Lielihood (ML) estimation model, the numbe of w in taining copus is Nw ( ) =. Then P ( w ) = / N, subject to: ML np ML( w ) = 1 (4) Fo simplicity we use add-one fomula fo smoothing [8], which use the =1 to estimate the unhappened toen: P = P = 1/ N (5) 0 1 ML ML On one hand, fo ou pepocessing stategy, many insignificant and meaningless toens ae often poduced, which incease the system load. By using the add-one smoothing algoithm, we can discad the toens with =1, which doesn t deceasing the filteing pefomance. It is so-called ill-one stategy. In pactice, the toens with =1~3 ae usually discaded. On the othe hand, toens discading is tiggeed by setting conditions (such as un time limitation, memoy). The effection on pecision of ou system still needs to be obseved. 3 Impovements

4 3.1 Weighted Adjustment of Sepaating Hypeplane In ou 2005 spam tac, we discussed some impovements based on sepaating haypeplane weighted adjustment [2]. The official evaluation esults of TREC 2005 show that the modification is effective. So in the 2006 tac, we epise the 2005 methods with new tass and data. 3.2 Selective Classifies Ensemble Last yea, we discuss the Bagging-based method fo spam filteing. In this yea, we use selective ensemble to impove the pefomance of classifying. Afte analysis of the elationship between the ensemble and its component, some eseaches [9,10,11] eveal that it may be bette to ensemble many instead of all of the classifies at hand. Selective classifies ensemble is thought an impoved method fo Bagging aggegate, in which mutual infomation weighted method is widely used [9,10,11]. Fo this yea s tac, we discuss two aggegate stategies: 1) selective ensemble based on mutual infomation of each classifie; 2) selective ensemble based on mutual infomation shaing with the optimal classifie. 4 Expeiments In this section, we epot the elevant esults fom the official evaluation. The basic statistics fo these datasets ae given as following: MX2 (9032 ham, spam), SB2 (9274 ham, 2751 spam). The pefomance of idult anti-spam solution is given in Table 1-Table 2. Results ae included fo 2 copoa, with immediate feedbac, delayed feedbac, and active leaning as denoted by the un tag suffix: x2 (MX2 copus, immediate feedbac), x2d (MX2 copus, delayed feedbac), x2a (MX2 copus, active leaning), b2 (SB2 copus, immediate feedbac), b2d (SB2 copus, delayed feedbac), b2a (SB2 copus, active leaning). Table 1. Immediate/delay feedbac esults Run tag Ham Misc% Spam Misc% Lam% (1-ROCA)% KB3S1x ( ) 0.68 ( ) 2.66 ( ) ( ) BASS2x ( ) 0.56 ( ) 2.52 ( ) ( ) B53S3x ( ) 0.65 ( ) 2.55 ( ) ( ) KB9S4x ( ) 0.58 ( ) 2.53 ( ) ( ) KB3S1x2d ( ) 0.71 ( ) 3.27 ( ) ( ) BASS2x2d ( ) 0.74 ( ) 3.01 ( ) ( ) B53S3x2d 9.13 ( ) 1.90 ( ) 4.22 ( ) ( ) KB9S4x2d ( ) 0.68 ( ) 3.15 ( ) ( ) KB3S1b ( ) 3.27 ( ) 2.74 ( ) ( ) BASS2b ( ) 3.16 ( ) 2.58 ( ) ( )

5 B53S3b ( ) KB9S4b ( ) KB3S1b2d 3.69 ( ) BASS2b2d 3.64 ( ) B53S3b2d 3.86 ( ) KB9S4b2d 4.83 ( ) 3.56 ( ) 3.02 ( ) 5.45 ( ) 5.45 ( ) 5.63 ( ) 4.54 ( ) 3.05 ( ) 2.83 ( ) 4.49 ( ) 4.46 ( ) 4.67 ( ) 4.69 ( ) ( ) ( ) ( ) ( ) ( ) ( ) Table 2. Active leaning esults Run tag Ham Misc% Spam Misc% Lam% (1-ROCA)% KB3A1x ( ) ( ) ( ) ( ) KB3A1x ( ) 1.20 ( ) 2.38 ( ) ( ) BASA2x ( ) ( ) ( ) ( ) BASA2x ( ) 1.15 ( ) 2.23 ( ) ( ) KB9A3x ( ) ( ) ( ) ( ) KB9A3x ( ) 2.09 ( ) 2.24 ( ) ( ) WEIA4x ( ) ( ) ( ) ( ) WEIA4x ( ) 1.58 ( ) 2.21 ( ) ( ) KB3A1b ( ) ( ) ( ) ( ) KB3A1b ( ) 1.05 ( ) 1.70 ( ) ( ) BASA2b ( ) ( ) ( ) ( ) BASA2b ( ) 1.26 ( ) 1.82 ( ) ( ) KB9A3b ( ) 5.89 ( ) ( ) ( ) KB9A3b ( ) 3.37 ( ) 3.66 ( ) ( ) WEIA4b ( ) ( ) ( ) ( ) WEIA4b ( ) 1.05 ( ) 1.75 (nan - nan) ( )

6 5 Summay Fo the un time limitation of spam tac, filtes that use moe than 2 seconds pe message will be illed and the esult will be ecoded as "class=ham scoe=0" fo any unpocessed messages. This maes us use simplified algoithms. In expeiments, some methods with good pefomance but time-consuming can not be applied. Moe impotantly, the impovement of ou system moe and moe depends on the details, such as wod segmentation, HTML deobfuscation, MIME nomalization, chaacte set folding, etc., which aleady have depatue fom the oiginal goal of TREC in some degee. 6 Acnowledgements This eseach is patially suppoted by NSFC (National Natual Science Foundation of China) unde Gant No and No , Key Poject of Chinese Ministy of Education unde Gant No and the Foundation of Chinese Ministy of Education fo Centuy Spanning Talent. 7 Refeences Yang, Z., Xu, W.R., Chen, B., Hu, J.N., Guo, J.: PRIS Kidult Anti-SPAM Solution at the TREC 2005 Spam Tac: Impoving the Pefomance of Naive Bayes fo Spam Detection. Poceedings of Fouteenth Text REtieval Confeence (2005) 3. Yeazunis, W., Chhaba, S., Siefes, C., Assis, F., Gunopulos, D.: A Unified Model of Spam Filtation Assis, F., Yeazunis, W., Siefes, C., Chhaba, S.:CRM114 vesus M. X: CRM114 Notes fo the TREC 2005 Spam Tac. Poceedings of Fouteenth Text REtieval Confeence (2005) 5. Chen, B., Peng, T., Xu, W.R., Guo, J.: POC-NLW Template fo Chinese Wod Segmentation. Poceedings of the Fifth SIGHAN Woshop on Chinese Language Pocessing (2006) Kim, Y.H., Hahn, S.Y., Zhang, B.T.: Text Filteing by Boosting Naive Bayes Classifies. In SIGIR Confeence on Reseach and Development (2000) 8. Nicolas, G., Domingo, O.: Impoving Multiclass Patten Recognition by the Combination of Two Stategies. IEEE Tansactions on Patten Analysis and Machine Intelligence, vol. 28(2006) Zhou, Z.-H., Wu, J.-X., Tang, W.: Ensembling Neual Netwos: Many Could Be Bette Than All. Atificial Intelligence (2002) Zhou, Z.-H., Wu, J.-X., Tang, W., Chen Z.-Q.: Selectively Ensembling Neual Classifies. Poceedings of the Intenational Joint Confeence on Neual Netwos (2002) Stehl, A., Ghosh, J.: Cluste Ensembles - A Knowledge Reuse Famewo fo Combining Multiple Patitions. Jounal on Machine Leaning Reseach (2002)