PRODUCT SEARCH OPTIMIZATION USING GENETIC ALGORITHM

Transcription

1 International Journal of Computer Engineering and Applications, Special Edition ISSN PRODUCT SEARCH OPTIMIZATION USING GENETIC ALGORITHM Pramod Kumar, Sadique Nayeem Department of Computer Science, RVS College of Engineering and Technology Jamshedpur, Jharkhand, INDIA ABSTRACT: Now a day s E-Commerce is very common. Among different types of E-Commerce, B2B E- Commerce is very popular. These web portals have a very massive amount of data related to its product. Simultaneously they generate a large amount of data every day. So, there is an urgent need for very efficient and optimized solution to retrieve information from such a massive data. Most importantly, the online buyers should be able to search their product efficiently. In this paper we have developed an online information retrieval system using genetic algorithm. First of all, a web crawler has been used for gathering and extracting related product information from the e-business websites. In the next stage, preprocessing of data by indexing in vector space model is done and finally genetic algorithm was used to retrieve most relevant optimized product search result for the user. Keywords: Hidden Web Crawler, Query Optimization, Search engines, Metadata, document frequency, term weights, E-Commerce. INTRODUCTION Business commonly referred to as "e-business" or an internet business, defined as the application of information and communication technologies (ICT) in support of all the activities of business, Such as buying and selling of the products. Example of e-business web sites are Amazon.com, ebay.com, flipcart.com, Snapdeal.com and many more. In such a huge, fragmented and unstructured information collection today s greatest problem is to find relevant information. For this online information retrieval system we are using the machine learning such as genetic algorithm to find relevant information. Information retrieval (IR) is finding material (usually documents) of an unstructured nature (usually text) that satisfies an information need from within large collections (usually stored on computers).as defined in this way, information retrieval used to be an activity that only a few people engaged in: reference librarians, paralegals, and similar professional searchers. A web search engine is a software system that is designed to search for information on the World Wide Web. The search results are generally presented in a line of results often referred to as search engine results pages (SERPs). A search engine operates in the order of Web crawling and Indexing in vector space modelling.

2 Web crawling is the process by which we gather pages from the Web, in order to index them and support a search engine. The objective of crawling is to quickly and efficiently gather as many useful web pages as possible, together with the link structure that interconnects them. Web crawler; it is sometimes referred to as a spied. The representation of a set of documents as vectors in a common vector space is known as the vector space model and is fundamental to a host of information retrieval operations ranging from scoring documents on a query, document classification and document clustering. GENETIC ALGORITHM Genetic Algorithm is search algorithm based on the mechanics of natural selection and natural genetics. They combine survival of the fittest among string structures with a structured yet randomized information exchange to form a search algorithm with some of the innovative flair of human search. Genetic algorithm was developed by John Holland and his colleagues in the University of Michigan. A simple genetic algorithm that yields good results in many practical problems is composed of three operators: REPRODUCTION: is a process in which individual string are copied according to their objective function values, f (biologists call this function the fitness function). CROSSOVER: May proceed in two steps 1. Members of the newly reproduced strings in the mating pool are mated at random. 2. Each pair of strings undergoes crossing over as follows: a. An integer position k along the strings uniformly at random between 1 and the string length less one [1, l-1]. b. Two new strings are created by swapping all characters between position k+1 and l inclusively. MUTATION: In artificial genetic systems, the mutation operator protects against such an irrecoverable loss. In simple genetic algorithm, mutation is the occasional (with small probability) random alteration of the values of string position simply means (changing 1 s or 0 s and vice versa). OBJECTIVE: Implement vector space model. Optimize the Query using the genetic algorithm. Retrieve the result using optimized query. RESEARCH METHODOLOGY QUERY: Our goal is to develop a system to address the ad hoc retrieval task. This is the most standard IR task. To assess the effectiveness of an IR system (i.e., the quality of its search results), a user will usually want to know two key statistics about the system s returned results for a query: Precision: What fractions of the returned results are relevant to the information need? Recall: What fraction of the relevant documents in the collection was returned by the system? BUILDING IR SYSTEM

3 The proposed system is based on Vector Space Model (VSM) in which both documents and queries are represented as vectors. Firstly, to determine documents terms, we used the following procedure: Extraction of all the words from each document. Elimination of the stop-words from a stop-word list- Stemming the remaining words using the porter stemmer that is the most commonly used stemmer in English. After using this procedure, the final number of terms that described all documents of the collection, we assigned the weights by using the following formula which proposed by Salton and Buckley. a ij = tf ij ( ) log N max tf n i ( tf ij max tf )2 (log N ) 2 n i Where a ij is the weight assigned to the term t j in document D i, tf ij is the number of times that termt j appears in document D i, n i is the number of documents indexed by the term t j and finally, N is the total number of documents in the database. RESULTS AND DISCUSSION QUERY OPTIMIZATION SEARCH SYSTEM EXAMPLE DATABASE: contains these documents Shipment of gold damaged I a fire D1.txt Shipment of gold arrived in a truck D3.txt Delivery of silver arrived in silver truck D2.txt Shipment of coal arrived in a truck D4.txt Fig: 1 Documents used for experiment Documents go under pre-processing. And index is built in vector space model Terms Q D1 D2 D3 D4 df D/df IDF Wq Wd1 Wd2 Wd3 Wd4 Arrived /3= Coal /1= Damag ed /1=

4 Deliver /1= y Fire /1= Gold /2= Shipme /3= nt Silver /1= Truck /3= Table.1 Vector Space Index A document vector (Doc) with n keywords and a query vector with m query terms can be represented as Doc = (term 1, term 2, term 3,., term n ) Query = (qterm 1, qterm 2, qterm 3,, qterm m ) We use binary term vector, so each term i (orqterm j ) is either 0 or 1. term i is set to zero when term i is not presented in document and set to one when term i is presented in document.for example, user enters a query into our system that could retrieve 4 documents. These documents are D1 = {shipment, gold, damaged, fire} D2 = {delivery, silver, arrived, truck} D3 = {shipment, gold, arrived, truck} D4 = {shipment, coal, arrived, truck} All keywords of these documents can be arranged in the ascending order as Arrived, coal, damaged, delivery, fire, gold, shipment, silver, truck Encode in the chromosome representation as, D1 = D2 = D3 = D4 = Q = From our example the length of each chromosome is 9 bits. D 1 = = = D 2 = = = D 3 = = = D 4 = = =

5 2 D i = i w i,j. (1) Q = = = Q = i w Q,j (2) Compute all dot products (zero products ignored): Q *D 1 = * = Q* D 2 = * * = Q* D 3 = * * = Q* D 4 = * = Q* D i = i w Q,j w i,j.. (3) Calculate the similarity value: Cosine θ D1 = Q D 1 Q D 1 = = Cosine θ D2 = Q D 2 Q D 2 = = Cosine θ D3 = Q D 3 Q D 3 = = Cosine θ D4 = Q D 4 Q D 4 = = Cosine θ Di = Sim(Q, D i ) Sim(Q, D i ) = i w Q,j w i,j w 2 Q,j 2 j i wi,j FITNESS EVALUATION: Fitness function is a performance measure or reward function which evaluate how good each solutions. Cosine θ Di = Sim (Q, D i ) Sim (Q,D i ) = i w Q,j w i,j w 2 Q,j 2 j i wi,j (4) Result from these fitness functions are interval 0 to 1. By 1means document and query is sameness.0 mean documents and query are less relevant. Values evaluate from fitness functions are called fitness. SELECTION: After we evaluate population s fitness, the next step is chromosome selection. Selection embodies the principle of survival of the fittest. CROSSOVER: Crossover technique includes one point crossover, two point crossover and multiple point crossover. If the structures are represented as binary strings, crossover can be implemented by

6 choosing a point at random, called crossover point, and exchanging the segments to the right of this point. For example, two chromosomes are crossover between position 4 and The resulting crossover yields two new chromosomes MUTATION : Mutation involves the modification of the values of each gene of a solution with some probability (mutation probability). For example: randomly mutate chromosome at position 6. The result will be: PROCESS OF OUR SYSTEM TEST CASE FORMULATION: This experimentation tests for queries with fitness function: cosine coefficient. Fitness function tests with set of parameters: probability of crossover (Pc = 0.8), and probability of mutation (Pm =0.01, 0.10, 0.30) to compare the efficiency of retrieval system The information retrieval efficiency measures from precision P, recall R, test accuracy F1. Result relevant documents documents retrieved P = relevant documents R = relevant documents documents retrieved documents Retrieved F1 = 2 PR P + R Percentage Total Docs Technique P R F1 100 Without GA 100% 100% 100% With GA 100% 100% 100% 200 Without GA 97.2% 51.02% 67.08% With GA 96.00% 96.15% 96.07% 300 Without GA 92.16% 52.04% 66.52% With GA 85.71% 87.50% 86.60% 400 Without GA 92.16% 52.04% With GA 85.71% % 500 Without GA 92.16% 52.04% 66.52% With GA 85.71% 87.50% /5 Without GA 94.88% 61.43% 73.33% With GA 90.63% 91.73% 91.17% Table 2 value of precision, recall and F1.

7 % 80.00% 60.00% 40.00% Without GA With GA 20.00% 0.00% Precision Recall F1 Fig. 2 The average percentage result for P, R and F1. Fig. 2 shows the average result of precision, recall and F1 for e-business topics. From the results, the precision of GA (90.63%) are lower than the precision, P without GA (94.88%). It means only some of the documents that are relevant to the user search. However, the recall, R result with GA is 91.73% compared to 61.43% without GA. It means that 91.73% of the documents are successfully search by the system based on the query selected by the user. The F1with GA (91.17%) are also higher than the result without GA (73.33%). From this result, we believed that the searched document based on the GA have higher accuracy rate rather than the result without A % % 50.00% 0.00% withou t GA % % 90.00% with 80.00% GA 70.00% without GA with GA Fig: 3. Precision Fig: 4. Recall % % 50.00% 0.00% without GA with GA Fig: 5. Value of F1 As shown in fig.3, we found that the precision values with GA and without GA are decreased when total number of documents increased. This is caused by keyword expansion from GA process that making the result after that is not accurate to user search but relevant by the system search. However the recall and F1 value with GA as shown in figure 4 and figure 5 is higher than recall and F1 value without GA.

8 CONCLUSION In this paper we have used crawler, which retrieve documents from E-business web pages. These web pages go under information retrieval process. And finally the proposed Query Optimization Search System is a two stage approach: First uses genetic algorithm to obtain the set of best combination of terms in the first stage. Second stage uses the output which is obtained from the first stage to retrieve more relevant results. The proposed information retrieval system is more efficient within a specific domain as it retrieves more relevant results. This has been verified using the evaluation measures, precision and recall REFERENCES [1] Christopher D. Manning, Prabhakar Raghavan and Hinrich Schütze, Introduction to Information Retrieval, Cambridge University Press [2] David E. Goldberg, Genetic Algorithms in Search Optimization and Machine Learning, Publisher: Addison-Wesley. [3]H. Chen, Machine learning for information retrieval: neural networks, symbolic learning, and genetic algorithms. Journal of the American Society for Information Science, 46(3), 1995, pp [4]Ahmed A. A. Radwan, Bahgat A. Abdel Latef, Abdel Mgeid A. Ali, and Osman A. Sadek, Using Genetic Algorithm to Improve Information Retrieval Systems World Academy of Science, Engineering and Technology [5] Eman Al Mashagba, Feras Al Mashagba and Mohammad Othman Nassar Query Optimization Using Genetic Algorithms in the Vector Space Model IJCSI International Journal of Computer Science Issues. [6] Cristina Lo pez-pujalte, Vicente P. Guerrero-Bote, Fe lix de Moya-Anego n Order Based Fitness Functions for Genetic Algorithms Applied to Relevance Feedback. Journal of the American Society for Information Science and Technology, 54(2): , 2003 [7] A.S.Siva Sathya, B.Philomina Simon, A Document Retrieval System with Combination Terms Using Genetic Algorithm. International Journal of Computer and Electrical Engineering, 2010 [8]Detelin Luchev, Abdelmgeid A. Aly, Applying Genetic Algorithm in query improvement problem International Journal "Information Technologies and Knowledge" Vol.1 / [9]Philomina Simon, Two Stage Approach to Document Retrieval using Genetic Algorithm. International Journal of Recent Trends in Engineering, Vol. 1, No. 1, May 2009 [10] Abdelmgeid A.Aly, Enhancing Information Retrieval by using Evolution Strategies,Information theories and applications Vol. 15, , 2008 [11]Andrew T., "an Artificial Intelligence Approach to Information Retrieval", Information Processing and Management, 40(4): , [12]D. Vrajitoru, Crossover improvement for the genetic algorithm in information retrieval, Information Processing& Management, 34(4), pp , 1998.

9 [13] Ilmério R. Silva, João Nunes Souza, Karina S. Santos, Dependence among Terms in Vector Space Mode. Proceedings of the International Database Engineering and Applications Symposium (IDEAS 04) [14]Hai-yan Kang, Yan-fang, Gui-fa Teng, Xiao-zhong Fan, Xiao-yang He, Research on Natural Language IR System based on Genetic Algorithm and VSM, Proceedings of the Third International Conference on Machine Learning and Cybernetics, Shanghai, August 2004.