Performance Evaluation of search engines via user efforts measures

Transcription

1 IJCSI International Journal of Computer Science Iue, Vol. 9, Iue 4, No, July Performance Evaluation of earch engine via uer effort meaure Raeh Kumar Goutam 1 and Sanay K. Dwivedi 1 Department of Computer Science, Babaaheb Bhimrao mbedkar Univerity, Lucknow, Uttar Pradeh, India Department of Computer Science, Babaaheb Bhimrao mbedkar Univerity, Lucknow, Uttar Pradeh, India btract Many metric exit to perform the tak of earch engine evaluation that are either looking for the expert udgment or believe in earcher deciion about the relevancy of the web document. However, earch log can provide u information about how real uer earch. Thi paper explain, our attempt to incorporate the uer earching behavior in formulation of uer effort centric evaluation metric. We alo incorporate two dimenional uer travering approach in the ERR metric. fter the formulation of the evaluation metric, author udge it goodne and found that preented metric fulfill all the requirement that are needed for a metric to be mathematically accurate. The finding obtained from experiment, preent a complete decription for earch engine evaluation procedure. Keyword: Information retrieval, Search engine performance, Search engine evaluation, Correlation baed Ranking. 1. Introduction The ie of World Wide Web i continuouly expanding rapidly. Thi i becaue of world wide move to migrate the information from online reource. To retrieve ome information from the web, earch engine are eentially required. When thee earch engine receive the querie, return a lit of document which are ranked on the bai of their quality. Normally, earch engine preent thouand of page in repone of a ingle query. Practically, thi i not poible to acce all thee document at all. With the help of our literature urvey, we conclude that a normal earcher browe approximately firt ten reult o it i eential for a relevant document to get a place in top ten poition. Search engine prepare ranking with the help of their evaluation algorithm. Each earch engine ue it own algorithm. web i open to all, hold no retriction to upload the document, reult expanion in web ie. It eem impoible for a earch engine to crawl all the web page a quickly a thee are getting uploaded. So it i a quick requirement to develop an evaluation metric that can evaluate the web page in fatet way.. Background and Related work Information i a vital a it wa thouand year ago. number of reearcher contributed with their valuable and unforgettable effort to convert the low traditional ource of information to vat and fat reource of information. Now, there are variou ource of information are available. From thee reource of information, web ha been accepted a very fat and primary reource of information. It ha amaing power to atify it uer with all kind of information intantly. Search engine are eentially required tool, ued to migrate the information from the web. Variou organiation have launched their earch engine with different functionalitie. Now, the ituation i very critical a thouand of information retrieval ytem are exiting and each i claiming for it uperiority and accuratene. So, the evaluation of the earch engine performance i done to decide their efficiency and accuratene. Chu and Roenthal [1] evaluated the capabilitie of ltavita, Excite and Lyco earch engine on the bai of their performance. They ued five criteria to perform the tak of evaluation. Thee criteria were compoition of web indexe (Coverage), Search Capability, Retrieval performance, output option (preentation) and uer effort. lthough, the author planned an effective trategy to perform the evaluation tak but their evaluation proce wa low a expert udgment were required. Suri [] preented a earch engine evaluation metric in which uer travering approach among the citation ha been ued. In thi paper, the requirement of a good metric have alo been dicued. O. Chapelle et. al. [3] ued the cacade model and ued a metric ERR (Expected Reciprocal Rank). In thi metric the document are udged for relevancy with probability of relevance. Thi approach eem to be ome un-appropriate becaue ame information can be irrelevant for a peron which i relevant to ome other peron while both ubmit the ame query. Cleverdon [4] uggeted ix criteria for earch engine evaluation. Thee criteria are web coverage, dwell Copyright (c) 01 International Journal of Computer Science Iue. ll Right Reerved.

2 IJCSI International Journal of Computer Science Iue, Vol. 9, Iue 4, No, July time, recall, preciion, preentation and uer effort. We explored the uer effort in the form of eion duration, Ranked Preciion and Click hit. We do not ue the preciion and recall a evaluation criteria becaue of ome problem. Evaluation baed on preciion and recall i not difficult to compute but thee meaure are conidered bit incomplete. Preciion aume that probability of randomly elected and retrieved web-page become relevant. It alo aume that frequently earch engine preent the mot relevant reult in the top poition in ranking ytem. Preciion compute the exactne of the retrieving relevant document in the information retrieval proce. It alo compute how many document are relevant in total retrieved web document. It doe not care if we are not retrieving all the relevant web-page but we uffer if we are retrieving non-relevant web document. 3. Metric Formuliation To meaure the performance of the earch engine, We have derived a metric named Ranked Preciion (RP) which i baed on two dimenional uer travering approach [6,7] among the retrieved citation. Thi metric return a number between 0 and 1. In thi metric, we divided all the web document in four categorie: Mot Relevant, Partially relevant, omewhat relevant and completely relevant. Different relevance core are aigned to different categorie of document. Initially, we divided the relevance core for web document in two part S and W. where S i relevance core for ub-link and W i the core for root-link. For the calculation of total relevance core of ub-link, we hall um up the total relevance core of all ub-link. During the calculation of total ub-link core, we have preumed that uer can viit up to m th link. It i not neceary at all that each earcher will have to viit m th link. If the earcher find the atifactory information in intermediate link then he/he can exit. One notable iue with the calculation of relevance core of ub-link, i it decrement in ucceive way, a the earch length increae. In thi way, total ub-link relevance numeric = m core for ingle root link i = 1 dding the relevance numeric core (w ) for the root link to = m we got the term = 1 1 = m = ( + w ) = 1 fter incluion of dead link b, we get the term a follow: (1) = m = ( + w ) * b () = 1 In the equation () b i the variable that hold only two numeric value 1 and 0. If the uggeted citation (by earch engine) i not alive then b hold the value 0 otherwie 1. the earcher i viewing the root link one by one from the top of lit to the bottom of lit. So we hall multiplying the with the term ( ( n + 1) r ). To concrete the concept, we uppoe that there are n root link and the rank of the th document i r. The term ( ( n + 1 ) r ) help in reducing the relevance core of the root link gradually a the earch length increae. So 3 = ( ( n + 1) r ) * ( + w ) * b (3) = m = 1 3 i the relevance core of ingle root link ( th ) and it ub-link. Extending the equation (3) for the n number of root link. The equation (3) take the form = n = m (4) 4 = [ ( ( n + 1 ) r ) * ( + w ) * b ] = 1 = 1 We divide the equation (4) by the term n ( n + 1) to find the Ranked Preciion (RP). Thi term i ued to calculate the bet cae and wort cae of the RP metric. R P = (5) 4 n ( n + 1 ) Finally, Putting the value of the 4 from the equation (4) into the equation (5) we found the metric R P = [ { ( ( 1) ) } * ( ) * ] = n = m n + r + w b = 1 = 1 (6) n ( n + 1) Table 1. Score for root link. Root link Relevance Score (w = 0.50 maximum ) The mot Relevant Partly Relevant Somewhat Relevant 0 Not Relevant at all Table. Score for Sub-link. Sub-link Relevance Score ( = 0.50 maximum ) The mot Relevant Partly Relevant Somewhat Relevant 0 Not Relevant at all Copyright (c) 01 International Journal of Computer Science Iue. ll Right Reerved.

3 IJCSI International Journal of Computer Science Iue, Vol. 9, Iue 4, No, July In our metric a hown in equation (6), the different numeric core (S and W ) are aigned by earcher to web page that depend upon the quality of information publihed on it. In the table 1 and table, the range for relevancy about the document are decribed. Searcher normally prefer to earch only few top citation to find the deired information. Silvertein et. al. [5] preented an tudy in which it wa highlighted that approximately 85% of the earcher viit only top ten reult. We conidered thi fact in our conideration and derived a metric in which uer can fix the top range for the document election. In the equation (6), n i number of document exiting on top poition, are required to be examined for relevancy. lthough, equation (6) i capable to evaluate the earch engine and differentiate them but it working depend upon earcher udgment. Searcher can aign the highet relevance core to the document which are irrelevant. Cranfield tyle of evaluation ha gained much popularity in pat two decade. ccording to thi method, the relevancy of the reult decreae from top to bottom gradually. The principle of the cacade model conider thi approach. The cacade model conider that the relevancy of retrieved document become in decending order. It alo conider that earcher top the earching a he/he find the reult. Olivier Chapelle et. al. [3] ued the cacade model and ued an ERR (Expected Reciprocal Rank) metric for earch engine evaluation. For thi evaluation metric, the author uggeted ome extenion to improve the reult. The Olivier chapelle et. al. [3] ued the ERR a follow. E R R R = n r (7) r = 1 r In the Equation (7), the R r i defined a the probability or relevance and r i the rank of document. In our method of earch engine evaluation, we ued the ame metric (Equation 7) a it wa ued by author ( Olivier chapelle et. al., 009). The main difference lie in computing the probability of relevance. In our method of earch engine evaluation, R r i calculated with the help of correlation between ix parameter: eion duration, dwell time, Ranked Preciion (RP), Click Hit, uer atifaction with quality of reult and uer atifaction for preentation of reult. Correlation CR1 i calculated between Seion duration and Dwell time a thee are poitively correlated. In other word, variation in eion duration time reult the correponding increment or decrement in the dwell time of WebPage. It i important to know here that how we organie the reult according to eion duration and dwell time. The document for which the eion duration i minimum, i kept on the top in the furnihed lit while the document for which the eion duration i maximum i kept on the lowet poition in the lit. Converely, the document i poitioned at the top for which the dwell time i maximum while the document hold minimum dwell time i kept on the lowet poition in the lit. In both the cae the document poition may change or identical. Correlation CR i etablihed between the Ranked preciion (RP) and Click Hit becaue thee two parameter are indirectly correlated. It could be eaily concluded that Ranked Preciion (RP) i directly dependent to earch length [ 6 ]. Variation in the depth of relevant reult will increae or decreae the correponding click Hit. During the CR calculation, we form the firt lit in uch a way that the maximum RP i poitioned on the top thereafter the ucceive decrement begin. In the econd lit, maximum click hit correponding a query are kept on the top in the lit after that the ucceive decrement tart until the organiation of all reult get completed. Correlation CR3 i calculated between uer atifaction with the preentation of reult and uer atifaction with the quality of reult. We organie the numeric core about uer atifaction with the preentation and quality of reult in decending order. Suppoe, a earch engine i preenting the low quality reult in the top while the relevant reult are poitioned in bottom of the lit. It i alo poible that earch engine can preent ambiguou reult correponding particular query. In both the cae, the uer atifaction with the preentation will degrade. Some other correlation pair are till poible with the help of thee ix parameter. Seion duration can be correlated with Ranked Preciion (RP) a the increment in the earch length reult the correponding increment in the eion duration and vice vera. Similarly, the eion duration can be correlated with click hit but it i very difficult to correlate eion duration with the uer atifaction with the quality and preentation of reult becaue thee two parameter are not dependent on eion duration. In our opinion, Dwell time cannot etablih the correlation with the Ranked Preciion (RP) becaue it i concerned with the time which i pent on a ingle document. It i not dependent on the earch length. Correlation between the dwell time and click hit can be formed a expanion in the quality aement time normally invite more click hit. 4. Metric Characteritic For the validation of the evaluation tak, the author (P.K. Suri et. al. 005) realied all the requirement for a good metric. We alo validate our metric with ame Copyright (c) 01 International Journal of Computer Science Iue. ll Right Reerved.

4 IJCSI International Journal of Computer Science Iue, Vol. 9, Iue 4, No, July requirement and found that our evaluation metric meet all the requirement that are needed to decide a metric a mathematically good. (1) Empirically and intuitively peruaive: The metric reult hould rie and fall appropriately under variou ituation. Both metric extended ERR and RP return a value between 0 and 1. It can be eaily een that the value of RP become 1 when all the reult retrieved are highly relevant and RP become 0 if all the retrieved reult are irrelevant. () Conitent and Obective: Both the metric RP and extended ERR are capable to yield relevant reult. It i alway eentially required that if a peron derive ome reult with a metric, it hould alway be poible to derive ame reult in ame ituation by another peron. For thi purpoe, we include three uer effort baed ignal uch a eion duration, dwell time and click hit o that the deciion of a particular earcher could not affect the end reult of the metric. (3) Programming language independent: our metric of earch engine evaluation i not derived for any particular language or particular platform o it can be programmed in any language for evaluation tak. (4) n effective mechanim for quality feedback: Number of click-hit help the earch engine developer to collect information that can be ued by them to evaluate the effectivene of their product and ubequently make eaier development of a higher quality product. (5) Poibility for extenion: extended ERR metric i extenible with ome other earch engine evaluation parameter uch a query formulation time and web coverage a well. we dicued, our evaluation metric fulfill all the requirement that are neceary for the goodne of any metric. So on the bai of thee ix reaon, we can ay that ERR i a good metric for earch engine evaluation. 5. Experimental Reult oftware to record the eion duration in minute and dwell time in econd. With thi oftware we count the total number of click-hit, web document are receiving. Beide of Mouotrom 5.0, we alo ued macromedia Dreamweaver CS5.5 oftware to validate the HTML webpage. The relevance core for the web-page, which i decided with uer interaction with brower i further integrated with earcher own udgment about the quality and preentation of reult. Thi i done becaue the relevance udgment, collected automatically can produce the bia reult a few web-ite incorporate the attractive advertiement on which few earcher make hit unnecearily. To reduce the impact of thi biane, we combined the reult derived automatically with earcher own udgment derived manually for quality of reult and preentation of reult. We apply our newly derived metric over a et of 150 TREC querie. The finding of the teting are hown in the table 3. In the table 3, on the bai of ix uer effort meaure and three correlation pair, we computed the average correlation value for all three elected earch engine. On the bai of thee correlation value, all the elected earch engine are compared. In our reult, we found that Google i mot efficient earch engine than ret two earch engine. Our tatitic decide MSN a le ignificant earch engine than Google and Yahoo ytem. From the teting of reult, we can conclude that approximately all the earch engine conider all thee ix parameter becaue none of the correlation pair attain a value near to ero or ero. if a correlation pair attain a numeric value ero it mean the poition organied for the querie for firt lit, are aigned poition in exactly revere order in econd lit. For the firt correlation pair CR1, approximately eventy three querie changed their poition in econd lit in Google earch engine. Similarly, in the correlation pair CR in Google earch engine, approximately eventy eight querie changed their poition from the firt lit and in the correlation pair CR3 We tet the efficiency of our extended metric ERR with the 150 TREC pattern querie. We ued Mouotrom 5.0 Search Engine Seion Duration Dwell Time RP Table 3. Search Engine comparion Click- Hit Score for Quality Score for Preentation CR1 CR CR3 verage* (ERR) Google Yahoo MSN Copyright (c) 01 International Journal of Computer Science Iue. ll Right Reerved.

5 IJCSI International Journal of Computer Science Iue, Vol. 9, Iue 4, No, July Search Engine Correlation Pair Range for kipped Citation (0-30) Table 4. Range for kipped Citation. Range for kipped Citation (31-60) Range for kipped Citation (61-90) Range for kipped Citation (91-10) Range for kipped Citation (11-150) Total variation querie Poition Google CR CR CR Yahoo CR CR CR MSN CR CR eighty three querie changed their poition. The numeric value for the correlation i not dependent only on the querie poition varying in both lit but alo depend upon the number of citation being kipped. In other word, the correlation value i converely proportional to the number of citation that are being kipped in query organiation in econd lit in any pair. The finding in table 4, how variation range in querie poition in all the correlation pair in all elected earch engine. In the table 4, maximum mall variation in querie poition for Google earch engine are found in all the elected correlation pair. Therefore, the average correlation value in table 3 for Google earch engine become large. For the Yahoo earch engine, comparatively ome large variation in querie poition are found than Google o the average correlation value in the table 3 become mall for Yahoo than Google earch engine. In our reult, extremely large variation are found in the querie poition in the all the correlation pair for the MSN earch engine o comparatively mall correlation value i found for MSN earch engine than ret two earch engine. 6. Concluion CR evaluation metric. Finally, we tet the performance of our method for evaluation with 150 TREC pattern querie. On the bai of the average relevance core, we elected the Google a mot efficient earch engine from a et of three earch engine. Mot of the evaluation metric for earch engine evaluation are baed upon unrealitic aumption that the uer viit only the root link. However, the author ue the uer two dimenional earching approach and believe that earcher not only viit the root-link but alo hit to ub-link to find the deired and atifactory information. In thi paper, we preent the extended ERR metric that incorporate the ix uer action dependent ranking parameter to evaluate the earch engine. Furthermore, we focued on the characteritic of newly formed metric. The author validate their metric with the characteritic which are required to be udged for the goodne of the Reference [1] Chu, H., Roenthal, M., Search engine for the world wide web: a comparative tudy and evaluation methodology,. Proceeding of the nnual Conference for the merican Society for Information Science, 1996, pp [] P.K Suri, Rakeh Kumar, R.K Chauhan, Search Engine Evaluation, DESIDOC Bulletin of Information Technology, 005, pp [3] O. Chapelle, D. Metler, Y. Zhang and P. Grinpan, Expected Reciprocal Rank for Graded Relevance, In Proceeding of the 18th CM conference on Information and knowledge management New York, US, pp [4] Cleverdon, C.W., Mill, J., and Keen, E.M.., n inquiry in teting of information retrieval ytem, Cranfileld, U.K.: lib Cranfield Reearch Proect, College of eronautic, 1966, pp [5] Silvertein, C., Heninger, M., Marai, J. & Moric, M., nalyi of a very large lta Vita query log, Technical Report , COMPQ Sytem Reearch Center, Palo lto, Ca, US, [6] Raeh Kumar Goutam and Sanay K. Dwivedi, Search Engine Evaluation uing uer effort In Proceeding of the nd International Conference on Computer and Communication Technology (ICCCT). 011, llahabad, India, pp [7] Sanay K. Dwivedi and Raeh Kumar Goutam, Evaluation of Search Engine uing Search Length, In Proceeding of the International Conference of computer Modeling and Simulatione, 011, Mumbai, India, pp Sanay K. Dwivedi ociate profeor, Department of computer cience at Babaaheb Bhimrao mbedkar Univerity, Lucknow 605 (U.P.) India. Hi reearch interet i in rtificial Intelligence, web Mining, NLP and ene diambiguation etc. He ha 16 year of experience of teaching and reearch and ha handled/involved in ome government funded reearch proect. in Copyright (c) 01 International Journal of Computer Science Iue. ll Right Reerved.

6 IJCSI International Journal of Computer Science Iue, Vol. 9, Iue 4, No, July He ha publihed a large number of reearch paper in reputed international ournal and conference. Raeh Kumar Goutam Reearch Scholar in Department of computer Science at Babaaheb Bhimrao mbedkar Univerity, Lucknow (U.P.) India. Hi reearch interet i earch engine and it performance evaluation, and web technology. Copyright (c) 01 International Journal of Computer Science Iue. ll Right Reerved.