Study on effective detection method for specific data of large database LI Jin-feng

Transcription

1 Iteratioal Coferece o Automatio, Mechaical Cotrol ad Computatioal Egieerig (AMCCE 205) Study o effective detectio method for specific data of large database LI Ji-feg (Vocatioal College of DogYig, Shadog 25709, Chia) Keywords: data detectio; database; mappig; Abstrt: i the process of detectig specific data of large database, whe the traditioal detectio method is utilized for detectig specific data, it is vulerable for iterferece of mass iformatio, which maes the specific data detectio process time-cosumig, ad of low efficiecy. For this, a effective detectio method for specific data of large database is proposed based o improved TFIDF algorithm, the iformatio etropy betwee the specific data features of large database ad the iformatio etropy withi the features are viewed as the weighted ftor for specific data detectio, oliear mappig ability of eural etwor is adopted to hieve calculatio of weights ad fuzzificatio of TFIDF algorithm, thus solvig the detectio problem for specific data of large database. The experimetal results show that, improved algorithm for effective detectio of specific data i large databases, ca effectively reduce time cosumed for detectio of specific data, esure the detectio quality of specific data to meet customer requiremets. Itroductio With the rapid icrease of database maagemet techology, specific data detectio techology have bee widely used i database maagemet of various idustries, ad plays a more ad more importat role []. Therefore, how to process effective detectio for specific data i large database cordig to the eeds of users [2], has become the core problem to research i the field of database maagemet [3]. The curret stage, the mai detectio methods for specific data i large database icludes detectio method for specific data i large database based o improved support vector mhie algorithm [4, 5], detectio method for specific data i large database based o Gauss's model [6] ad detectio method for specific data i large database based o fuzzy clusterig algorithm [7]. Oe of the most commoly used is detectio method for specific data i large database based o improved support vector mhie algorithm [8]. Because detectio methods for specific data i large database play a irrepleable role i the field of database maagemet, therefore, has a broad prospect for developig [9, 0], ad become the focus studied by may experts. 2 Priciple of detectio method for specific data i large database based o improved TFIDF algorithm 2. descriptio of improved TFIDF algorithm With the give probability distributio P = ( p, p,..., p ) of data i large database, the iformatio etropy of distributio trasfer is defied as: H( P) = pi log2 pi () I the specific data collectio D of a large databases, cordig to data types are divided ito class, deoted as C, C 2,..., C, defies the probability distributio cotaiig features t as P= ( / N, / N,..., / N). 2 I the detectio process of specific data i large database, higher specific data distributio uiformity idex of a charteristic item cotaied i large database, shows that the distributio etropy H betwee eh class after specific data detectio is bigger, amely the system 205. The authors - Published by Atlatis Press 784

2 cotributio is small. Therefore, the improved TFIDF method, based o the traditioal method, combies the distributio etropy of eh class ad separate iformatio etropy withi eh class, to form the ew ifluece parameter, ew iformatio etropy ftor betwee classes is defied as: H ah ( ) = max( H ) + l (2) max( H ) = log2 Wherei: max( H ) is the maximum of eh feature class iformatio etropy of correlated feature item after extrtig the feature of specific data, is the class umber of data i large database, l is a costat coefficiet. Thus, based o the iformatio etropy distributio of specific data feature items betwee class ad withi class i large-scale database, TFIDF weightig method is defied as: IDF Wi ( d) = ah ( ) IDFcost N IDF = tfi ( d) log( + 0.0) (3) 2 N 2 IDFcost = ( tfi ( d)) [log( + 0.0)] The defiitio is optimized with the above formula, cotributio of classifyig of charteristics of specific data i large database ca be show obviously. 2.2 weight calculatio based o eural etwor The improved TFIDF method after weightig wors better for cosiderig detailed distributio situatio of eh specific data charteristics i large database, whe specific data quatity is very low, the maual way is adopted to calculate weights of eh specific data set, while, whe specific data quatity i large databases is eormous, maual calculatio is ot realistic, thus, it is ecessary to adopt a ew data fusio processig method for large amout of data to calculate the weight. BP eural etwor is a multilayer feedforward eural etwor cordig to the error b-propagatio algorithm traiig, is oe of the most widely used eural etwor models. BP etwor ca lear ad store a lot of iput ad output scheme mappig relatio, without revealig the mathematical equatios to describe this mappig relatio i adve. Its learig rule is based o the method of steepest descet, costatly adjust the etwor weight of etwor. BP eural etwor model topology structure icludes iput layer, hidde layer ad output layer. The basic idea of BP eural etwor is to obtai a set of optimal weights after traiig with a large umber of data, for utilizatio afterwards, for a set of specific iformatio, cordig to the optimal weights which are traied before, the output iformatio of predictio is obtaied. BP eural etwor have great iput ad output oliear mappig relatioship, ca hieve self-learig ad update, which is t applicable for the probability ad statistics method. The relatioship betwee iput ad output is defied as: yj = xi* aij + δ (4) m z = y * b + δ 2 (5) j j j= Amog them: xi ( i =, 2,, ) The elemet of iput layer, is the umber of elemets of output layer; aij ( i =, 2,, ; j =, 2,, m) The weighted value of iput layer to the hidde layer; m the umber of elemets of hidde layer; δ The threshold value of iput layer to the hidde layer; y ( j =, 2,, m) The elemet value of hidde layer; j 785

3 b ( j=, 2,, m ; =, 2,, p) The weighted value of hidde layer to the output layer; j p Numbers of hidde layer elemets; z ( =, 2,, p) The elemet value of output; δ 2 The threshold value of hidde layer to the output layer. BP eural etwor is used for weight calculatio, through iput large amout of traiig data of large database sample distributio to the eural etwor to obtai the collectio of weight feature. The specific data feature database is established, afterwards, through the weighted calculatio for ew data, the results is compared to specific data feature database, if it is cosistet with the charteristic, appropriate weight will be give. I this paper, BP eural etwor is used to calculate weight, with the followig advatages: () the algorithm has strog ati-jammig ability; (2) self-adaptive to various eviromets; (3) do ot eed tedious statistical model. 3 Experimet results ad aalysis I order to verify the effectiveess of improved algorithms, there is the eed for a experimet, the experimetal eviromet is Visual C For the experimet coducted with large database, the umber of all data cotaied is P, the species umber of all special data is p, all special data collectio is { b, b2,, b P } { c, the data set costituted of all special data attributes is, c2,, c p }, bj the probability of specific data belogig to attribute c is λ. The followig formula was employed to calculate the time cosumed by specific data detectio i large database: j = 2 bj bj c (6) Specific data detectio cosumig time i large database, is a importat idex to measure the specific data detectio method. The umber of all data samples i large database was 000, all species of data is 5. The data sample data ad attribute of the large database is orgaized ad aalyzed, so as to obtai the table as show below: Table experimetal samples data NO. Attribute Quatity of data(millio) Residetial buildig.5 2 Dwellig area Property Warm oeself Water supply Power supply.9 7 Gas Pesio Medical care Commuity 3.6 Boos Library Supermaret 4. 4 Shoppig Tourism 5.7 Whe the sample complexity was high, the traditioal algorithm ad the improved algorithm were adopted separately to detect specific data i large database, the detectio results ca be 786

4 described i the followig table: Table 2 experimetal results whe sample complexity is high The umber of experimets time cosumig of the traditioal algorithm(ms) time cosumig of the improved algorithm(ms) Accordig to above table, it ca be leart that by usig the improved algorithm for specific data detectio i large database, ca avoid the defects of the traditioal algorithm, therefore, improves the cury of detectio for specific data i large database. 4 Coclusio Aimig at the problem happeed i the process of detectig specific data of large database, whe the traditioal detectio method is utilized for detectig specific data, it is vulerable for iterferece of mass iformatio, which maes the specific data detectio process time-cosumig, ad of low efficiecy. For this, a effective detectio method for specific data of large database is proposed based o improved TFIDF algorithm, the iformatio etropy betwee the specific data features of large database ad the iformatio etropy withi the features are viewed as the weighted ftor for specific data detectio, oliear mappig ability of eural etwor is adopted to hieve calculatio of weights ad fuzzificatio of TFIDF algorithm, thus solvig the detectio problem for specific data of large database. The experimetal results show that, improved algorithm for effective detectio of specific data i large databases, ca effectively reduce time cosumed for detectio of specific data, esure the detectio quality of specific data to meet customer requiremets. Refereces [] Re Li a, He Qig. A ew classificatio method of mass data [J]. Computer egieerig ad applicatios, : [2] Fa Bo, Li Haigag, Guo Qiog. Research o spatial data classificatio for customer segmetatio [J]. Joural of systems egieerig, 2008.: [3] Wu Fei, [J]. Classificatio method of real etwor data ad aoymous address. Joural of Chagjiag Uiversity atural scieces: Polytechic volume, based o 2008.: [4] Liu Hogya, Che Jia, Che Guoqig. Review o data classificatio algorithm i data miig [J]. Joural of Tsighua Uiversity: Natural Sciece Editio, : [5] Ni Xiaju. Research o the tehig method based o data miig classificatio techology [J]. Sciece techology ad egieerig, : [6] Li Guagda, Chag Chu, Zheg Huaiguo, Ta Cuipig, Zhao Jigjua. Study o the classificatio ad search method of agricultural scietific data of foreig etwor [J]. Ahui Agricultural Scieces, : [7] Gao Hogbig, Li Fegbi, Wag Ji, Liu Yu. SCS data classificatio coversio ito Shape documet based o VBA [J]. Moder surveyig ad mappig, : [8] Wag Defe, Gao Jiaqiag, Li Li. Research o [J]. Data classificatio based o media PCA ad weighted PCA, iformatio techology, 204.2:4-8. [9] Zhag Haifeg. The iput method of effectiveess sequece the multi-level classificatio data i 787

5 Excel [J]. Applicatio of computer system, : [0] Hu Zhaolig, Li Haiqua. Extrtio ad classificatio of SAR image texture feature [J]. Joural of Chia Uiversity of Miig ad Techology, 2009, 38 (3):