MEASURING THE PERFORMANCE OF SIMILARITY PROPAGATION IN AN SEMANTIC SEARCH ENGINE

Transcription

1 ISSN: (ONLINE) DOI: /isc ICTACT JOURNAL ON SOFT COMPUTING, OCTOBER 013, VOLUME: 04, ISSUE: 01 MEASURING THE PERFORMANCE OF SIMILARITY PROPAGATION IN AN SEMANTIC SEARCH ENGINE S. K. Jayanthi 1 an S. Prema 1 Department of Computer Science, Vellalar College for Women, Inia ayanthiskp@gmail.com Department of Computer Science, K.S.R College of Arts an Science, Inia premashanmuga11@gmail.com Abstract In the current scenario, web page result personalization is playing a vital role. Nearly 80 % of the users expect the best results in the first page itself without having any persistence to browse longer in URL moe. This research work focuses on two main themes: Semantic web search through online an Domain base search through offline. The first part is to fin an effective metho which allows grouping similar results together using BookShelf Data Structure an organizing the various clusters. The secon one is focuse on the acaemic omain base search through offline. This paper focuses on fining ocuments which are similar an how Vector space can be use to solve it. So more weightage is given for the principles an working methoology of similarity propagation. Cosine similarity measure is use for fining the relevancy among the ocuments. Keywors: Semantic Web, BookShelf Data Structure, Similarity Propagation, Cosine Similarity measure, Vector Space Moel 1. INTRODUCTION Information Retrieval is a problem of selecting the relevant information from a ocument atabase in response to search queries given by a user. Information Retrieval Systems (IRSs) eal with ocument atabase that usually consist of textual information an process user queries to provie the user with access to relevant information within a reasonably acceptable time interval. After retrieving the results it is essential for a search engine to rank-orer the ocuments matching a query. To perform this, the search engine calculates, for each matching ocument, a score with respect to the given query. This research work initiate the stuy of assigning a score to a (query, ocument) pair. Two types of moe are iscusse in this research work. One is an online web search an another is an offline search. For both the moe of research the similarity between the retrieve ocuments are calculate using the cosine similarity measure. In summary, this architecture gives much higher quality results yet with similar response time to other systems. The rest of the paper is organize as follows: relate work is iscusse in Section an provies the preliminaries on web search engines. Section 3 provies an iea for fining similarity propagation among ocuments.it also focus on vector space moel an cosine similarity measure. Term frequency an Inverse ocument frequency for similarity calculation is iscusse. Section 4 explains the experimental results of Online an Offline research. Arranging the retrieve results in BookShelf Data Structure is also reporte in this section. Finally, Section 5 gives the concluing remarks.. RELATED WORK Qingtian Han an Xiaoyan Gao [1] analyses a Web mining algorithm base on usage mining. A novel an efficient approach for the etection of nearest uplicate web pages was esigne by V. A. Narayana et al. [].The ocuments with similarity scores greater than a threshol value are consiere as near uplicates. J. Akilaneswari an N. P. Gopalan [3] analyze an architectural framework of a crawler for locating eep web repositories using learning multi-agent Systems. Crawling an Page Rank Algorithms for Internet Searches are esigne by Animesh Tripathy an Prashanta K Patra [4]. An architecture an implementation prototype of web ata mining system base on web service was evelope by Chunying Chen et al. [5]. This approach takes a source-centric perspective on the information-seeking process, aiming to ientify trustworthy sources of relevant information from within the user's social network by Tom Heath [6]. Where an iniviual encounters a problem or task for which their current knowlege is inaequate, they may engage in information-seeking in orer to change their knowlege state (Belkin) [7]. Louis S. Wang [8] presents a moifie vector space moel for measuring similarity between the query an the ocument. Huilian Fan et al. [9] esigne a new crawling strategy which combine the avantages of hyperlinks structure an web content strategies. Topic keywors base VSM is use to evaluate iniviual fitness, an imports new URLs to implement crossover an mutation, an the URLs that have the same prefix are regare as niche. Mehran Sahami et al. [10] propose a similar kernel function, for measuring the similarity between short text snippets. Sean A. Golliher [11], two classes (on-page an off-page variables) of search engine ranking factors an their possible implications for ranking web ocuments are iscusse. Albert Bifet et al. [1] analyze the result rankings for several queries of ifferent categories using statistical methos in Google (via its API) as testbe. Chowhury Farhan Ahme et al. [13] esigne an efficient mining of utility-base web path traversal patterns. Hazem Elmeleegy et al. [14] provies a novel technique for extracting tables from lists. The technique is omain-inepenent an operates in a fully unsupervise manner. The link structure of a web site can be visualize in a link hierarchy consisting of web pages on multiple conceptual levels for user navigation [15]. Duran an Kahn [16] evelope a system calle MAPA to extract a hierarchical structure from an arbitrary web site for navigation. 667

2 S K JAYANTHI AND S P REMA: MEASURING THE PERFORMANCE OF SIMILARITY PROPAGATION IN AN SEMANTIC SEARCH ENGINE 3. SIMILARITY PROPAGATION The main contribution of this paper is to fin the similarity propagation among the ocuments. The approach iscusse in this section is common for both the online web search an offline omain-centric approach. To workout this approach one shoul have the basic iea about the following: tf-if calculation, Vector space moel, Query as vector, tf-if weight calculation an cosine similarity measure. All those funamental concepts are iscusse in the following sections. The weight of the components of a ocument vector can be represente by Term Frequency or combination of tf an if. Number of occurrences of a term t in the ocument D is referre by tf. Number of ocuments, where a particular term t occurs is note as f. In if i.e. log (n/f) wor with rare occurrences has more weight.tf-if is calculate as tf-if= tf if t Fig.1. Documents as vector Documents D are points or vectors in this space. Terms are axes of the space. Documents that are close together in vector space talk about the same things. View each ocument as a vector with one component corresponing to each term in the ictionary, together with a weight for each component that is given by tf-if t, = tf t, if t. For ictionary terms that o not occur in a ocument, this weight is zero as in Fig VECTOR SPACE MODEL t The representation of a set of ocuments as vectors in a common vector space is known as the vector space moel an is funamental to a host of information retrieval operations ranging from scoring ocuments on a query, ocument classification an clustering. The tf-if values can now be use to create vector representations of ocuments. Each component of a vector correspons to the tf-if value of a particular term in the corpus ictionary. Dictionary terms that o not occur in a ocument are weighte zero. Using this kin of representation in a common vector space is calle vector space moel, which is not only use in information retrieval but also in a variety of other research fiels like machine learning (e.g. clustering, classification). A vector space provies the possibility to perform calculations like computing ifferences or angles between vectors. Since ocuments are usually not of equal length, simply computing the ifference between two vectors has the isavantage. The ocuments of similar content but ifferent length are not regare as similar in the vector space. Each term from the collection becomes a imension in an n- imensional space. A ocument is a vector in this space, where term weights serve as coorinates. It is important for scoring ocuments for answering queries, Query by example, Document classification an Document clustering (Prabhakar) [17]. Formalizing vector space proximity oesn t use Eucliean istance because it is large for vectors of ifferent lengths. But at the same time using cosine similarity will result in ranking the ocuments in increasing orer of cosine (query, ocument). 3. RELEVANCY MEASURE The vector space representation of text is an increibly powerful tool. Any text can be treate as a vector in a V- imensional vector-space (Jaime Arguello) [18]. Documents are matche with a query base on their similarity. If a ocument is similar to the query, it is likely to be relevant. Non-binary weights for inex terms in queries an ocuments are use in the calculation of egree of similarity. Decreasing orer of this egree of similarity for the retrieve ocuments gives the ranke ocuments with partial match (Manwar et al.) [19]. For the vector moel, the weight w i, associate with a pair (k i, ) is positive an non-binary. Further, the inex terms in the query are also weighte. Let w i,q be the weight associate with the pair(k i, q),where w i,q 0. Then, the query vector q is efine as q = (w 1,q, w,q,. w n,q ) where n is the total number of inex terms in the system. The vector for a ocument is represente by = (w 1,, w,,. w n, ) by Manwar et al. The main obective is to retrieve more ocuments like those labele relevant an fewer ocuments like those labele irrelevant. The combination of tf an f is the most popular weight use in case of ocument similarity exercises. The weight is high when t occurs many times within a small number of ocuments. The weight is low, when the term occurs fewer times in a ocument or in many ocuments. A vector V can be expresse as a sum of elements such as, V = a 1 v i1 + a v i a n v in where, a k are calle scalars or weights an v in as the components or elements. Consier two ocument vectors 1, an a query vector Q. The space contains terms t 1, t, t 3,,t n ).The ocument 1 has components t 1, t 3,.. an has components t, t 4,..So V( 1 ) is represente closer to axis t 1 an V( ) is closer to t.the angle represents the closeness of a ocument vector to the query vector. Its value is calculate by cosine of. 3.3 COSINE SIMILARITY MEASURE To avoi the bias cause by ifferent ocument lengths, a common way to compute the similarity between the two ocuments is using the cosine similarity measure. The inner prouct of the two vectors is ivie by the prouct of their vector lengths. This has the effect that the vectors are normalize to unit length an only the angle, more precisely the cosine of the angle, between the vectors account for their similarity. Documents not sharing a single wor get assigne a similarity value of zero because of the orthogonality of their 668

3 Y-axis ISSN: (ONLINE) ICTACT JOURNAL ON SOFT COMPUTING, OCTOBER 013, VOLUME: 04, ISSUE: 01 vectors while ocuments sharing a similar vocabulary get higher values. Because a query can be consiere a short ocument, it is of course possible to create a vector for the query, which can then be use to calculate the cosine similarities between the query vector an those of the matching ocuments. Finally, the similarity values between the query an the retrieve ocuments are use to rank the results. For any two given ocuments an k, their similarity is: n. w i i w k 1, i, k sim, k n n k w w i1 i, i1 where, w i is a weight of the ocuments. The most popular way to measure the similarity between two frequency vectors (raw or weighte) is to take their cosine value as in Fig.. It is necessary to compute the cosine angle between A (the query) an each ocument an sort these in ecreasing orer of cosine angles. This treatment can be extene to the entire collection of ocuments. C(x 0, y 0 ) Fig.. Cosine angle 3.4 TF AND IDF WEIGHT FOR SIMILARITY CALCULATION i, k Cosine similarity between two ocuments is given by, v sim1, v 1. v v If the vector 1 has component weights w 1, w, w 3 an vector has component weights u 1, u, then the ot prouct = w 1 *u 1 + w *u. Since there is no thir component, w 3 *null = 0. 1 w1 w w3 Eucliean length of. Consier there are 3 ocuments, D 1 = The file contains operating concepts D = My laptop is operating uner winows operating system D 3 = This system is not working properly Q = Operating System Number of ocuments = 3; inverse ocument frequency IDF = log (D/f i ) is calculate as in Table.1. Calculating the vector lengths Eucliean length of the Vector is D W i, A(x 1, y 1 ) X-axis 1 B(x, y ) i D D D Q Term Table.1. tf-if calculation tf i Weights=tf i*if i f i D/f i if i Q D1 D D3 Q D1 D D3 concepts contains file is laptop My not operating properly system The This uner Winows Calculate the ot proucts of the query vector with each Document vector, Q. Di WQ, * Wi, Q.D 1 = * = Q.D = * * = Q.D 3 = * = Cosine value calculation Cosine ( 1 ) = Q.D 1 / Q * D 1 = /(0.489*0.970)=0.18 Cosine ( ) = Q.D / Q * D =0.099/(0.489*0.930) = Cosine ( 3 ) = Q.D 3 / Q * D 3 = /(0.489*0.8630)=0.144 Document D is more similar to the query. Cosine formula gives a score which can be use to orer ocuments. Documents with a partial match are also ientifie an the problem is that positional information about the terms is missing. The implementation of cosine value calculation is one in Java an the source coe is: 669

4 S K JAYANTHI AND S P REMA: MEASURING THE PERFORMANCE OF SIMILARITY PROPAGATION IN AN SEMANTIC SEARCH ENGINE if(len1>len) for(int p=0;p<len;p++) System.out.println(" if ss1 "+ss1[p]+" "+ss11[p]+" "+ss[p]+" "+ss[p]); 1=(Double.parseDouble(ss1[p])); =(Double.parseDouble(ss11[p])); 3=(Double.parseDouble(ss[p])); 4=(Double.parseDouble(ss[p])); ouble if1=(1/3); ouble if=(/4); ouble sf=(double.parsedouble(s5[p])); ouble sf1=(double.parsedouble(s6[p])); if(if1>if) sim=(if1*sf*sf1); if(if1<if) sim=(if*sf*sf1); int t=(e+1); String fit=w+" "+t+" "+sim; System.out.println("Dis "+fit); System.out.println("insert into Similarity values("+tt+",'"+w+" "+t+"',"+sim+")"); //b.st.executeupate("insert into Similarity values("+tt+",'"+w+" "+t+"',"+sim+")"); tt++; res.a(fit.trim()); ta.appen(fit+"\n"); else boolean bt=true,bt11=true; for(int p1=0;p1<len1;p1++) System.out.println("ss1 "+ss1[p1]+" "+ss11[p1]+" "+ss[p1]+" "+ss[p1]); 1=(Double.parseDouble(ss1[p1])); //=(Double.parseDouble(ss11[p1])); 3=(Double.parseDouble(ss[p1])); 4. EXPERIMENTAL RESULTS 4.1 ONLINE SEARCH Online results for the search query operating system are consiere for the experimental ataset. The similarity propagation for the retrieve web ocuments are calculate following the proceure as in section 3. The probability measure for all the retrieve ocuments are consiere like, (0,1),(0,),(0,3)(0,4),(0,5),(0,6),(0,7),(0,8),(0,9),(0,10), (1,0),(1,),(1,3)(1,4),(1,5),(1,6),(1,7),(1,8),(1,9),(1,10),.,(10,0),(10,1),(10,),(10,3) (10,4),(10,5),(10,6),(10,7),(10,8),(10,9) For every pair of ocument similarity, automatic upation of SQL table is create with column specification for ocument i of two ocuments an similarity value an the similarity propagation measure is given in Fig.3. String qur="create table Hierarchical (i varchar (500), oc1 numeric (18, 10),"; for (int k=0;k<(out.length-1);k++) if(k<(out.length-)) else qur+="oc"+(k+)+" "+"numeric(18,10)"+","; qur+="oc"+ (k+) +" "+"numeric (18, 10)"+")"; Fig.3. Similarity Propagation Measure Similarity values are automatically calculate for each an every web search, i.e., ynamically store in the Java file for further reference as in Fig.4. The Fig.5 represents the plots of clustering quality against the number of clusters for the given query in online search using the similarity propagation representation. In each of the graphs, the curves corresponing to the two similarity measures are 670

5 ISSN: (ONLINE) ICTACT JOURNAL ON SOFT COMPUTING, OCTOBER 013, VOLUME: 04, ISSUE: 01 shown (Jayanthi an Prema) [1]. It can be clearly seen that the quality of clustering increases monotonically with the number of clusters. Fig.7. Cluster measure Fig.4. Automatic Backup of the Similarity Propagation File 4. OFFLINE SEARCH Two types of moes are focuse in an offline search. One is an IT professional search an the other is a non-professional search. Professional search is base on omain-centric system. Document collection an keywors focuse in this section are all base on computer science omain. Main an sub omain sample ocuments collecte for professional omain-centric search is in Table.. Sample ocuments for non-professional search with their keywors are in Table.3. Table.. Professional Domain-Centric Clustering Fig.5. Documents versus Similarity measure Initially, as expecte, the increase in quality of clusters is rapi. It can also be observe that the curve for the quality of clusters flattens out somewhat for the cluster numbers close to the natural number of clusters for the corresponing ataset. The corresponing screenshots are given in Fig.6 an Fig.7. Main omain Sub-omain keywors Operating System apple, winows, unix, kernel Network "communication"," protocol","topology","layer" DBMS "bms","sql","oracle" Data Structure "ata structures","array","list","vector" "Software Testing, Software SOFT Design","Implementation","Software evelopment" Table.3. Non-Professional ocuments Clustering Main omain winow apple mouse Sub-omain keywors "oor","winow","esign","interior" "history","apple","fruit","season" "history","mouse","home" The moe of search i.e., professional or non-professional is to be ecie by the user. Professional search is focuse on computer science omain. Non-professional search is focuse on general concept. 5. CONCLUSION Fig.6. Clustere Results There is a limite average time they will spen before giving up, or becoming very upset with the search technology available for them. It is foun that people will not search for long on the 671

6 S K JAYANTHI AND S P REMA: MEASURING THE PERFORMANCE OF SIMILARITY PROPAGATION IN AN SEMANTIC SEARCH ENGINE web. To spee up the web search, an alternative approach is iscusse in this paper. Two moes of searching options are provie in this research work. One is an online web search an another one is an offline search. Instea of isturbing all the ocuments, the web search with the most similar propagation is consiere. The highly correlate ocuments are clustere using Agglomerative hierarchical clustering. The clustere ocuments are arrange in BookShelf Data Structure for an easy access (Jayanthi an Prema) [0]. For users, seeing clustere search results has the following benefits such as Bringing those search results into easy view that otherwise woul remain invisible because they are far own the list. Allows users to examine nearly ouble the number of relevant ocuments than in the case of result lists of commercial search engines. Leas to effortless knowlege iscovery as the user learns the types or subtopics of available information relating to the query. Provies context by placing the relate ocuments within a single foler for oint viewing. All of these factors have significant impact on the user s search prouctivity. REFERENCES [1] Qingtian Han an Xiaoyan Gao, Research of Distribute Algorithm Base on Usage Mining, Proceeings of Secon International Workshop on Knowlege Discovery an Data Mining, pp , 009. [] Narayana V.A, P. Premchan an A. Govarhan, A Novel an Efficient Approach For Near Duplicate Page Detection in Web Crawling, Proceeings of IEEE International Avance Computing Conference, pp , 009. [3] Akilaneswari J an Gopalan N. P, An Architectural Framework of a Crawler for Locating Deep Web Repositories using Learning Multi-agent Systems, Proceeings of Thir International Conference on Internet an Web Applications an Services, pp , 008. [4] Tripathy A, Patra P. K, A Web Mining Architectural Moel of Distribute Crawler for Internet Searches Using Page Rank Algorithm, Proceeings of IEEE Asia-Pacific Services Computing Conference, pp , 008. [5] Chunying Chen, Xiongwei Zhou an Jianzhong Zhang, Web Data Mining System Base on Web Services, Proceeings of Ninth International Conference on Hybri Intelligent Systems, Vol. 3, pp. 16-0, 009. [6] Tom Heath, Information-seeking on the Web with Truste Social Networks from Theory to Systems, Ph.D Thesis, Department of Computer Science, 008. [7] N. J. Belkin, Helping People Fin What They Don't Know, Communications of the ACM, Vol. 43, No.8, pp , 000. [8] Louis S. Wang, Relevance Weighting of Multi-Term Queries for Vector Space Moel, IEEE Symposium on Computational Intelligence an Data Mining, pp , 009. [9] Huilian Fan, Guangpu Zeng an Xianli Li, Crawling Strategy of Focuse Crawler Base on Niche Genetic Algorithm, Proceeings of Eighth IEEE International Conference on Depenable, Autonomic an Secure Computing, pp , 009. [10] Mehran Sahami an Timothy D. Heilman, A Web base Kernel Function for Measuring the Similarity of Short Text Snippets, Proceeings of the 15 th International Conference on Worl Wie Web, pp , 006. [11] Sean A. Golliher, Search Engine Ranking Variables an Algorithms, Sem.org, Supplement Issue, Vol. 1, 008. [1] Albert Bifet, Carlos Castillo, Paul-Alexanru Chirita an Ingmar Weber, An Analysis of Factors Use in Search Engine Ranking, AIRWeb 005: Available at: [13] Ahme C. F, Tanbeer S. K, Byeong-Soo Jeong an Young- Koo Lee, Efficient Mining of Utility-Base Web Path Traversal Patterns, Proceeings of International Conference on Avance Communication Technology, Vol. 3, pp , 009. [14] Hazem Elmeleegy, Jayant Mahavan an Alon Halevy, Harvesting Relational Tables from Lists on the Web, The International Journal on Very Large Data Bases, Vol. 0, No., pp. 09-6, 009. [15] Jianhan Zhu, Mining Web Site Link Structures for Aaptive Web Site Navigation an Search, Ph.D Thesis, University of Ulster at Joranstown, 003. [16] Davi Duran an Paul Kahn, MAPA: A System for Inucing an Visualizing Hierarchy in Websites, Proceeings of the ninth ACM Conference on Hypertext an hypermeia: links, obects, time an space- structure in hypermeia systems, pp , [17] Jaime Arguello, Vector Space Moel, INLS 509: Information Retrieval, Lecture Notes, 011. [18] A. B. Manwar, Hemant S. Mahalle, K. D. Chinchkhee an Vinay Chavan, A vector space moel for information retrieval: a matlab approach, Inian Journal of Computer Science an Engineering, Vol. 3, No., pp. -9, 01. [19] S. K. Jayanthi an S. Prema, Facilitating Efficient Integrate Semantic Web Search with Visualization an Data Mining Techniques, Proceeings of International Conference on Information an Communication Technologies, pp , 010. [0] S. K. Jayanthi an S. Prema, CIMG-BSDS: Image Clustering Base on Bookshelf Data Structure in Web Search Engine Visualization, Proceeings of International Conference on Recent Trens in Computing, Communication an Information Technologies, pp ,