Study on the Application Analysis and Future Development of Data Mining Technology

Transcription

1 Study on the Application Analysis and Future Development of Data Mining Technology Ge ZHU 1, Feng LIN 2,* 1 Department of Information Science and Technology, Heilongjiang University, Harbin , China 2 Information and Network Construction Management Center, Heilongjiang University, Harbin, , China *Corresponding author(feng LIN, lf1861@163.com) Abstract Data mining technology is a new hot-spot in database research, which combines traditional data analysis technique with relatively sophisticated algorithms for dealing with mass data, exploring and analyzing new data type and processing old data type with new method. This article explains the background of data mining technology, neural network, decision tree, genetic algorithm and discusses the basic process of data mining based on various types of data, and finally pointing out the application field and future trend of data mining technology. Keywords: Data mining; Analysis technology; Support service 1. INTRODUCTION With the rapid development of the Internet and database technology, the huge mount of data was stored in the database, which makes difficult to find the needed information. The data mining technology is the exact method to solve this problem and to look for the needed data in mass data, thus can help company dig the valid data and reduce unnecessary expense. Recently, many companies have already adopted the data mining technology to identify the most valuable customer groups, and then revise their promotion strategy which aims to get higher return with lower investment. The data mining is also called data exploration, mining of data, which generally refers to the process of extracting the hidden, unknown, but potentially valuable information among large number of noisy, ambiguous, incomplete and random data. This article will firstly talk about the data mining analysis technique. 2. COMMONLY USED ANALYSIS TECHNOLOGY OF DATA MINING Generally speaking, according to the specific type of data and model, the more technology of data mining system has been used, the more reliable conclusion will be get. In terms of function, the analysis technology of data mining can be classified into following techniques: (1) Association Analysis Technique The purpose of association analysis is to dig out the correlation among the large amounts of data. Given a set of Item and a record collection, the association analysis is to know the correlation among Item through analyzing the characteristics of record collection. For example, 30% is the record including Item A, Item B and Item C, which also includes Item E and Item F, the 30% is referred to the credibility of rule for the record including Item A, Item B and Item C, which also includes Item E and Item F, and Item A, Item B and Item C are called the antithesis to Item E and Item F. This article will use a typical example of goods sale for easy discussion. The goods retailer keeps all the transaction record which includes transaction number, customer number and product number, etc. Now it has association analysis on customer s buying behavior, and the object of association analysis is a group of product of different goods type. The goods retailer saves the transaction record and forms a record collection for association analysis, each record, as shown in Table 1, is composed of transaction number, customer number, product number and date of every transaction. Table 1. Transaction Record Collection Transaction Number Customer Number Product Number Date T1 1 A,B,C 3/3/05 T2 2 A,C,D 4/3/05 T3 3 A,B,D 3/4/05 T4 4 A,C 4/4/05 T5 5 D 4/5/05 439

2 Before doing association analysis, it needs to preset two parameters: 1) the minimum degree of confidence, whose role is to filter out the possibility of smaller rules, and in this case, you can set a minimum confidence level of 0.2; 2) the minimum support, whose role is to represent the probability of occurrence of a specific rule, and the credibility value will be 0.2 based on the minimum confidence level. According to the above description, it can set rule as customer who buys product X will also buys product Y, and setting the confidence level as C for this rule, so doing support level as S. Using formula to obtain: C= (the transaction number of both purchase product X and product Y)/ (the transaction number of only purchase product X), S= (the transaction number of both purchase product X and product Y)/ (total transaction number). Using the data in Table 1, getting the confidence level C and support level S of each rule through the above formula, the result is shown in Table 2. By the first row of data in Table 2, it can be seen that the confidence level C of the customer who buys product A will also buys product B is 0.5, and the support level is 0.2. Thus the market staff will know that the customer who buys A oven will has 50% chances to buy the B. This usage is relatively simple for the ordinary market staff who will master it after some training, the results have a certain reference value. This method does not involve complex algorithm, but from which can understand the basic method and reference value of association analysis. Table 2. Record Form of Association Analysis Item 1 Item 2 confidence level C support level S A B A C A D B C A,B D A,C D (2)Sequential Pattern Analysis Technique The Sequential pattern analysis technique is also to dig out the relevance of large amounts of data, the purpose is somewhat similar to association analysis technique, but sequential pattern analysis focuses on the causal relationship of data mining. With the above examples of data and uses the customer number, date, product number to rebuild the table (see Table 3). Table 3. Classification by Timetable Customer Number date product number /3/05 A,B 3/4/05 C 4/5/05 D 3/3/05 A,C 3/4/05 A,B,C 3/3/05 A 3/4/05 B Similarly, it needs to preset the minimum support count S before doing the sequential pattern analysis. As in this example, presetting the minimum support count as 2. Sequential pattern mining is to find all the frequent sub sequences, that is, the occurrence frequency of the sub sequence in the sequence is not less than the minimum support count. Table 4. Record Form of Sequence Analysis Sequence Support Count B,C 2 B,D 2 As shown in the Table 4, we can know that the customer who buys the B, it is likely to buy C or D. Using sequential model to analyze the above data, the product retailer will find the potential way for shopping of the customers, for example, which product does the customer usually buy before purchasing the electromagnetic 440

3 oven, and then has the sales strategy for the product. The sequential model analysis technique can also be used in stock analysis, for example, it can find some regular pattern through sequential model analysis technique, in medical insurance industry it also has a good effect, the insurance company can use this new technique to predict the most common medical approach and then adjust the insurance policies. (3) Classification Analysis Technique The classification analysis technique needs to assume the record collection has a set of mark which needs to have the characteristics to distinguish from other categories. In preparation of classification analysis, it needs to define a mark for every record firstly, which means to classify the records according to mark when doing the record, then check out those records with marks, finally, describing the characteristics of those records with marks. The description method can be different, it can be explicit way, such ac directly define a set of rule or characteristic: and it can be implicit, such as the definition of a mathematical formula of a mapping method. Now, many classification model has started to adopt classification analysis technique to analyze and predict. For example, the database of major banks save the records of various customers, and have classification according to personal credit degree(namely the mark) which can be divided into: good, fair, poor, very poor, etc. This classification method is classification analysis technique. Banks can use classification analysis technique to give an explicit description of the credit rating as: the fair credit consumers generally refers to ones who have monthly income of 5000RMB, the age between 25 to 35, and mostly living in certain area, (4) Cluster Analysis Technique Clustering is the process of classifying the collection of physical or abstract objects and compose the multiple categories of similar objects. The input set of cluster analysis technique is a group of marked records, which is not classified, and the purpose is to divide the record collection reasonably according to specified rules, then adopting explicit or implicit rule to classify. Generally, cluster analysis technique can adopt various algorithms, so the same record collection may has different classification result, and the above classification analysis substantially applies to cluster analysis. In fact, the classification analysis technique and the cluster analysis technique is complementary in many aspects, such as, during the early data analysis, the analyst can label and classify the data according to experience or general rule, then analyze the data with classification analysis technique, getting a general description of each category, then using the description as a new classification rule to re-divide the data, thus can get a better division effect, and analyst can cycle use the two analysis techniques to obtain a satisfactory result. In terms of data mining system, sometimes it needs integrated utilization of four methods that described above. To take product sales as example, now it needs to have market position for a certain product, the data mining system may comprehensively use four methods: 1) using association analysis technique to sum up the product which is usually bought with the certain product; 2) using sequential analysis technique to find the customer group of purchasing the certain product, and summarize the sequence of shopping; 3) using classification analysis technique to sum up the shopping mode of the certain product according to the result of sequential analysis technique; 4) using the above shopping mode as the rule of cluster analysis technique, and adopting cluster analysis method to find the potential customer with the same shopping mode and having a promotion for the potential. 3. COMMONLY USED TECHNOLOGY OF DATA MINING (1) Neural Network Neural network is also known as connection model, which is the algorithm mathematical model of an intimation of animal nervous feature, and having distribution parallel information processing which is usually used to solve the classification and regression problems. Neural network is considered as a set of connected input or output unit, each of which is connected to a power. In the learning phase of neural network, people can adjust the weights of the neural network so that it can accurately predict the category number of samples for learning. (2) Decision Tree Decision tree has an extensive usage, now it can be used to judge the condition under which the value can be get with certain rule method. For example, when bank staff dealing with the loan, it needs to judge the risk of the loan applicant. The decision Tree is shown as Figure

4 Income>4000 0? Work Time>5 High Liability Low Risk High High Risk Figure 1. Decision Tree Risk of Loan Low Risk For example, bank employees can use the decision tree as shown in Figure 1 to decide whether granting or refusing the loan applicant, and using the size of loan to determine the loan risk. Such as income > 40,000 and high liability consumer will be treated as high risk, otherwise, "income <40,000" and "working time> 5 years," user will be considered as "low risk "and get the loan. The decision tree which used in data mining technology can be used in analyzing data and predicting, the decision tree which is shown in Figure 1 can be used to predict. The commonly used algorithms for predicting are: CHAID, Quest, CART and C5.0, etc. As long as scanning the data in database, then creating a decision tree which can easily handle cases with variable. But, the decision tree can not be too huge to effect the predicting result, setting the maximum layer of decision tree can be used to limit the size of decision tree, another method is to preset node which contains the smallest record, when the node contains the record that is smaller than the smallest record, it is the time to stop the division. Sometimes it can let decision tree to grow freely, then prune it to the desired size, but the accuracy of the decision tree need to be maintained during the pruning, When dealing with the node division of the decision tree, it can adopt greedy algorithm whose idea is: all division are based on sequential order, the rationality is not considered after division of each node, which means each division is dependent on the rationality of the last time division. Since the limitation of the greedy algorithm, it is seldom used. Relatively speaking, the decision tree is good at dealing with non-numerical data, such as bank information; while neural network is good at processing numerical data. (3) Genetic Algorithm Genetic algorithm borrows the biological evolution law(such as survival of the fittest, survival of the fittest genetic mechanism) and evolves into a kind of algorithm which has ability of random searching for the optimal solution. Its characteristic is to operate directly on the object without the limitation of deviation and continuous function, and having the inherently implicit parallelism and overall optimization capability; the adoption of probabilistic optimizing method can automatically obtain and optimize searching field, and have self-adapt adjustment of searching direction. The genetic algorithm consists of three basic operators: 1) reproduction (selection), namely choosing a more vital individual from the parental generation (former population), thus generating a new population; 2)crossing (recombination), which means to select two different parts of individual to have genetic crossing and recombination, thus generating new individual; 3) variation (mutation), i.e. having mutation treatment on some parts of some individuals, thus generating a new individual. Genetic algorithm can play a role in improving offspring, which has played a key role in optimizing calculation and classified machine learning 4. PROCESS OF DATA MINING Now to have a further discussion on basic steps of data mining process(as shown in Figure 2), data mining steps include:1) determine the research object, this determination process is the foundation of whole mining process; 2) establish the database, through the establishment, it can collect, describe, and select the data in advance; 3) analyze the data, it can use the commonly used analysis methods which introduced in this article (such as association analysis technique, sequential mode analysis technique, etc.) to analyze the data and classify the characteristic of the data; 4) prepare the data, which includes choosing the variables, recording and transforming the variables, etc. 5) establish the model, which needs to select suitable mining algorithm to mine on the transformed data; 6) evaluate and explain, which needs to evaluate the established model and has some necessary explanations; 7) implementation (knowledge assimilation), applying the analysis result to the similar system. 442

5 XML data textual data data cleaning data transformation data check XML database other data Figure 2. Data pre-treatment process 5. APPLICATION OF DATA MINING TECHNOLOGY 1) Data Mining Technology Aimed at Bio-medical and DNA Data Analysis At present, many bio-medical researches mainly focus on data analysis of DNA sequence, which has found causes of many diseases and disabilities genes. Using association analysis technology to identify the gene sequence which occurs simultaneously; by means of path analysis technique to find the gene sequence of disease at different stage; and it can also use visualization tools and genetic data analysis technique to display the complex structure and sequence of gene. 2) Data Mining Technology Aimed at Finance Banks and financial institutions all provides various kind of savings business, credit business, investment, financial service business and insurance business. Aimed at the characteristics of the business and data, there are several typical data mining techniques: 1. design and construct the database for the characteristic of multidimensional data; 2. Using loan repayment predicting technique and consumer credit policy analysis method to analyze data; 3. Having classification and cluster analysis on specified target market customers. 3) Application of Data Mining Technology on Retail Business Retail Business is the main field for the research and application of data mining, retail business has the accumulation of long-term sales data, such as the purchase record of customer, consumer, service record and goods purchase record, etc. The data mining of retail business can help sellers to identify the shopping behavior and buying mode of customer, as well as improving the service quality, increasing the goods sales ratio and reducing the cost The related data mining techniques are: design and construct database according to data mining technology; multidimensional analysis on sales, customers, goods, time and area; analysis of customer s shopping loyalty, etc. 6. FUTURE TREND OF DATA MINING TECHNOLOGY Since the variety of data, data mining task and data mining method, which gives many research direction of data mining technology, such as the application development of data mining technology; the data mining method with flexibility; data mining technology, database system and integration of Web database system; standardization of data mining language; visualized data mining tool; new mining technique for complicated data type; privacy protection of data mining process and information security, etc. 7. CONCLUSION Data mining technology uses artificial intelligence and XML-related technologies to analyze the data in certain database, in order to dig out its rule, which involves various analysis techniques, commonly used algorithms, and the application field of data mining technology is quiet extensive, the development potential is huge. ACKNOWLEDGMENTS This research is supported by Natural Science Foundation of Heilongjiang Province of China (No.F201432). References Brin S, Page L. The anatomy of a large-scale hypertextual Web search engine, International Journal of Approximate Reasoning, 1998, pp

6 Jianwu Yang, Songlin Wang. Extended VSM for XML document classification using frequent subtrees, Proceeding INEX'09 Proceedings of the Focused retrieval and evaluation, and 8th international conference on Initiative for the evaluation of XML retrieval.2010.pp Jiawei Han,Micheline Kamber, Data Mining Concepts and Techniques, Second Edition, China Machine Press, Xinfeng Lai, Changxuan Wan,Dexi Liu. XML Node Semantic Weight Model Based on VSM, Proceeding ICMECG '08 Proceedings of the 2008 International Conference on Management of e-commerce and e-government, 2008, pp Zuobing Xu, Ram Akella. Improving probabilistic information retrieval by modeling burstiness of words, Information Processing and Management: an International Journal.2010.pp