CORE: A Tool for Collaborative Ontology Reuse and Evaluation

Transcription

1 CRE: A Tool for Collaboratve ntology Reuse and Evaluaton Mram Fernández, Iván Cantador, Pablo Castells Escuela Poltécnca Superor Unversdad Autónoma de Madrd Campus de Cantoblanco, Madrd, Span {mram.fernandez, van.cantador, pablo.castells}@uam.es ABSTRACT ntology evaluaton can be defned as assessng the qualty and the adequacy of an ontology for beng used n a specfc context, for a specfc goal. In ths work, a tool for Collaboratve ntology Reuse and Evaluaton (CRE) s presented. The system receves an nformal descrpton of a semantc doman and deteres whch ontologes, from an ontology repostory, are the most approprate to descrbe the gven doman. For ths task, the envronment s dvded nto three man modules. The frst component receves the problem descrpton represented as a set of terms and allows the user to refne and enlarge t usng Word- Net. The second module apples multple automatc crtera to evaluate the ontologes of the repostory and detere whch ones ft best the problem descrpton. A ranked lst of ontologes s returned for each crteron, and the lsts are combned by means of rank fuson technques that combne the selected crtera. A thrd component of the system uses manual user evaluatons of the ontologes n order to ncorporate a human, collaboratve assessment of the qualty of ontologes. Categores and Subect Descrptors H.3.3 [Informaton Storage and Retreval]: Informaton Search and Retreval nformaton flterng, retreval models, selecton process. eneral Terms Algorthms, Measurement, Human Factors. Keywords ntology evaluaton, ontology reuse, rank fuson, collaboratve flterng, WordNet.. INTRDUCTIN The Semantc Web s envsoned as a new flexble and structured Web that takes advantage of explct semantc nformaton, understandable by machnes, and therefore classfable and sutable for sharng and reuse n a more effcent, effectve and satsfactory way. In ths vson, ontologes are proposed as the backbone technology to supply the requred explct semantc nformaton. Developng ontologes from scratch s a hgh-cost process that requres maor engneerng efforts, even for a medum-scale ontology. In order to properly face ths problem, we beleve effcent ontology reuse and evaluaton technques and methodologes are needed. The lack of approprate support tools, and the lack of automatc measurement technques for certan ontology features are often a barrer for the mplementaton of successful ontology reuse methods. In ths work, we present CRE, a Collaboratve ntology Reuse and Evaluaton system. Ths tool provdes automatc smlarty measures for comparng a certan problem or olden Standard to a set of avalable ontologes, retrevng not only those most smlar to the doman descrbed by the olden Standard, but the best rated ones by pror ontology users, accordng to the selected crtera. For smlarty assessment, a user of CRE selects a subset from a lst of comparson technques that the tool provdes. Based on ths, the tool retreves a ranked lst of ontologes for each crteron. Fnally, a unque rankng s defned by means of a global aggregated measure whch combnes the dfferent selected crtera, usng rank fuson technques []. nce the system retreves those ontologes closely related to the olden Standard, t supports an addtonal step n the evaluaton process, by mplementng a Collaboratve Flterng approach [2][4][9]. Snce some ontology features can only be assessed by humans, the last evaluaton step takes nto consderaton the manual feedback provded by users of the ontologes, to re-rank the lst of ontologes, thus retrevng not only the ontologes that best ft the olden Standard, but the most qualfed ones accordng to human evaluatons. The paper s organzed by the followng structure. In Secton 2 we present the relevant work related to our research; secton 3 descrbes the system archtecture; sectons 4 and 5 present the automatc evaluaton measures used by the system; and some conclusons are gven n secton RELATED WRK ur research addresses problems n three dfferent research areas, where we draw from pror related work. These are: ontology evaluaton and reuse, whch s the prmary goal of our work; rank fuson, whch s used to combne the ratngs provded by dfferent ontology evaluaton crtera; and collaboratve flterng, by whch we get further evaluaton measures for ontology features that are better assessed by human udgment. 2. ntology Evaluaton Dfferent methodologes for ontology evaluaton have been proposed n the lterature consderng the characterstcs of the ontologes and the specfc goals or tasks that the ontologes are ntended for. An overvew of ontology evaluaton approaches s presented n [2], where four dfferent categores are dentfed: Those that evaluate an ontology by comparng t to a olden Standard, whch may tself be an ontology [] or some other knd of representaton of the problem doman for whch an approprate ontology s needed.

2 Those that evaluate the ontologes by pluggng them n an applcaton, and measurng the qualty of the results that the applcaton returns [6]. Those that evaluate ontologes by comparng them to unstructured or nformal data (e.g. text documents [3]) whch represent the problem doman. Those based on human nteracton to measure ontology features not recognzable by machnes [0]. In each of the above approaches, a number of dfferent evaluaton levels mght be consdered to provde as much nformaton as possble. Several levels can be dentfed n the lterature: The lexcal level [3][][2] whch measures the qualty by comparng the words (lexcal entres) of the ontology wth a set of words that represent the problem doman. The taxonomy level [] whch consders the herarchcal connecton between concepts usng the s-a relaton. ther semantc relatons besdes herarchcal ones [6][8]. The syntactc level [7] whch consders the syntactc requrements of the formal language used to descrbe the ontology. Context or applcaton level [4] whch consders the context of the ontology, such as the ontologes that reference or are referenced by the one beng evaluated, or the applcaton t s ntended for. The structure, archtecture and desgn levels [0] whch take nto account the prncples and crtera nvolved n the ontology constructon tself. Table summarzes all these approaches [2]. Table. An overvew of approaches to ontology evaluaton Level Lexcal entres, vocabulary, concept, data Herarchy, taxonomy ther semantc relatons Context, applcaton Approach to evaluaton olden Standard Applcaton based Data Drven Assessment by humans X X X X X X X X X X X X Syntactc X X Structure, archtecture, desgn In the present paper, two novel evaluaton measures are proposed. The frst one s based on a olden Standard approach and the lexcal level measure proposed by Maedche and Staab []. The second one s based on assessment by humans n a collaboratve flterng approach. X X X 2.2 Rank Fuson Rank fuson has been a wdely addressed research topc n the feld of Informaton Retreval [][5][9]. ven a set of rankngs whch apply to a common unverse of nformaton obects, the task of rank aggregaton conssts of combnng ths lst n a way to optmze the performance of the combnaton. Examples where rank fuson takes place nclude, for nstance, metasearch [8] dstrbuted search from heterogeneous sources, personalzed retreval, classfcaton based on multple evdence, etc. Fuson technques typcally brng better recall, better precson, and more consstent performance than the ndvdual systems beng combned []. Fuson technques can be characterzed by: The nput data they requre: ranks, scores, or full nformaton of the obects. Whether or not tranng data s used, whch usually conssts of manual relevance udgments on the nformaton obects. The degree of overlap between the sets of rated obects, rangng from total overlap (a.k.a. data fuson), to completely dsont sets (a.k.a. collecton fuson), and arbtrarly overlappng. The applcaton level, whch can be a) external, f autonomous ratng systems are ntegrated nto a new meta layer, or b) nternal, f the combnaton takes place at heart of a retreval system, where dfferent subsystems collect evdence from several sources or dfferent crtera. In our work, rank fuson technques are used to combne the ndvdual ontology lsts retreved by partal evaluaton crteron nto an aggregated ontology rankng. Ths can be understood as a metasearch problem where a) the nput data are the evaluaton ratngs from dfferent crtera, b) no tranng data s used (there are no pror manual ratng or reference udgements for comparson), c) the overlap s complete (all the evaluaton crtera are appled on the same ontology repostory), and d) the level of applcaton s nternal (the ratng sources are components wthn the CRE system). 2.3 Collaboratve Flterng Collaboratve flterng strateges make automatc predctons (flter) about the nterests of a user by collectng taste nformaton from many users (collaboratng). Ths approach usually conssts of two steps: ) look for users that have a smlar ratng pattern to that of the actve user (the user for whom the predcton s done), and 2) use the ratngs of users found n step to compute the predctons for the actve user. These predctons are specfc to the user, dfferently to those gven by more smple approaches that provde average scores for each tem of nterest, for example based on ts number of votes. Collaboratve flterng s a wdely explored feld. Three man aspects typcally dstngush the dfferent technques reported n the lterature [4]: user profle representaton and management, flterng method, and matchng method. User profle representaton and management can be dvded nto fve dfferent tasks: Profle representaton. Accurate profles are vtal for the content-based component (to ensure recommendatons are approprate) and the collaboratve component (to ensure

3 that users wth smlar profles are n fact smlar). The type of profle chosen n ths work s the user-tem ratngs matrx (ontology evaluatons based on specfc crtera). Intal profle generaton. The user s not usually wllng to spend too much tme n defnng her/hs nterests to create a personal profle. Moreover, user nterests may change dynamcally over tme. The type of ntal profle generaton chosen n ths work s a manual selecton of values for only fve specfc evaluaton crtera. Profle learnng. User profles can be learned or updated usng dfferent sources of nformaton that are potentally representatve of user nterests. In our work, profle learnng technques are not used. The source of user nput and feedback to nfer user nterests from. Informaton used to update user profles can be obtaned n two dfferent ways: usng nformaton explctly provded by the user, and usng nformaton mplct observed n the user s nteracton. ur system uses no feedback to update the user profles. Profle adaptaton. Technques are needed to adapt the user profle to new nterests and forget old ones as user nterests evolve wth tme. Agan, n our approach profle adaptaton s done manually (manual update of ontology evaluatons). Flterng method. Products or actons are recommended to a user takng nto account the avalable nformaton (tems and profles). There are three man nformaton flterng approaches for makng recommendatons: Demographc flterng: Descrptons of people (e.g. age, gender, etc) are used to learn the relatonshp between a sngle tem and the type of people who lke t. Content-based flterng: The user s recommended tems based on the descrptons of tems prevously evaluated by other users. Content-based flterng s chosen approach n our work (the system recommends ontologes usng prevous evaluatons of those ontologes). Collaboratve flterng: People wth smlar nterests are matched and then recommendatons are made. Matchng method. Defnes how user nterests and tems are compared. Two man approaches can be dentfed: User profle matchng: People wth smlar nterests are matched before makng recommendatons. User profle-tem matchng: A drect comparson s made between the user profle and the tems. The degree of approprateness of the ontologes s computed by takng nto account prevous evaluatons of those ontologes. In CRE, a new ontology evaluaton measure based on collaboratve flterng s proposed, consderng user s nterest and prevous assessments of the ontologes. 3. SYSTEM ARCHITECTURE In ths secton we descrbe the archtecture of CRE, our Collaboratve ntology Reuse and Evaluaton envronment. Fgure shows the overvew of the system. We dstngush three dfferent modules. The frst one, the left module, receves the olden Standard defnton as a set of ntal terms and allows the user to modfy and extend t usng WordNet [3]. The second one, represented n the center of the fgure, allows the user to select a set of ontology evaluaton technques provded by the system to recover the ontologes closest to the gven olden Standard. The thrd one, or rght one, s a collaboratve module that re-ranks the lst of recovered ontologes, takng nto consderaton prevous feedback and evaluatons of the users. Fgure. CRE archtecture 3. olden Standard Defnton Phase The olden Standard Defnton module receves an ntal set of terms. These terms are supposed to be obtaned by an external Natural Language Processng (NLP) module from a set of documents related to the specfc doman n whch the user s nterested. Ths NLP module would receve the repostory of documents and return a lst of pars (lexcal entry, part of speech), that roughly represents the doman of the problem. Ths phase s part of future work. Here n our experments, the lst of ntal (root) terms has been manually assgned. The module allows the user to expand the root terms usng WordNet [3] and some of the relatons t provdes: hypernym, hyponym and synonym. The new terms added to the olden Standard usng these relatons mght also be extended agan and added to the problem defnton. The fnal representaton of the olden Standard can be defned as a set of terms T(L, PS, L P, R, Z) where: L s the set of lexcal entres defned for the olden Standard. PS corresponds to the dfferent Parts f Speech consdered by WordNet: noun, adectve, verb and adverb. L P s the set of lexcal entres of the olden Standard that have been extended. R s the set of relatons between terms of the olden Standard: synonym, hypernym, hyponym and root (f a term has not been obtaned by expanson, but s one of the ntal terms). Z s an nteger number that represents the depth or dstance of a term to the root term from whch t has been derved.

4 Fgure 2. CRE user nterface Example: T ( pzza, noun,, RT, 0). T s one of the root terms of the olden Standard. The lexcal entry that t represents s pzza, ts part of speech s noun, t has not been expanded from any other term so ts lexcal parent s the empty strng, ts relaton s RT and ts depth s 0. T 2 ( pzza pe, noun, pzza, Synonym, ). T 2 s a term expanded from T. The lexcal entry t represents s pzza pe, ts part of speech s noun, the lexcal entry of ts parent s pzza, t has been expanded by the synonym relaton and the number of relatons that separated t from the root term T s. The left part of Fgure 2 shows the nterface of the olden Standard Defnton phase. In the top level we can see the lst of root terms. The user s allowed to manually nsert new root terms gvng ther lexcal entres and selectng ther parts of speech. The correctness of these new nsertons s controlled by verfyng that all the consdered lexcal entres belong to the WordNet [3] repostory. In the bottom level we can see the fnal olden Standard defnton: the fnal lst of (root and expanded) terms that represent the doman of the problem. In the ntermedate level t can be seen how the user can make a term expanson. The user selects one of the prevous terms from the olden Standard defnton and the system shows hm all ts meanngs contaned n WordNet [3]. After he chooses one, the system automatcally presents hm three dfferent lsts wth the synonyms, hyponyms and hypernyms of the term. The user can choose one or more elements of these lsts and they wll automatcally be added to the expanded terms lst. For each expanson the depth of the new term s ncreased by one unt. Ths wll be used later to measure the mportance of the term wthn the olden Standard: the greater the depth of the derved term wth respect to ts root term, the less ts relevance wll be. 3.2 System Recommendaton Phase In ths phase the system should retreve the ontologes that better conceptualze the olden Standard doman. The mddle module of Fgure represents the structure of the recommendaton phase of the system. Frstly, the user selects a set of evaluaton crtera to be performed. After consderng the selected crtera and takng nto account the olden Standard and the ontologes of the repostory, the system retreves a ranked lst of ontologes (ordered by ther smlarty to the olden Standard) for each crteron. Then, all these lsts are merged usng rank fuson technques [] to obtan a global measure. The mddle part of Fgure 2 represents the user nterface of the System Recommendaton module. In the upper level we dstngush the crtera selecton phase. By now, two content evaluaton crtera can be selected to retreve the most smlar ontologes: ) the lexcal crteron, whch measures smlarty between the lexcal entres of the olden Standard and the lexcal entres of the ontologes and, 2) the taxonomc crteron, whch evaluates the herarchcal structure between them. The user can also select the relevance of each crteron n the rank aggregaton process, usng a range of dscrete values [, 2, 3, 4, 5], where symbolzes the lowest relevance value and 5 the hghest. Moreover, dfferent knds of lexcal and taxonomc smlarty measures have been mplemented and tested usng ths tool. These may now be selected n ths phase. These measures wll be explaned n secton 4 of ths document. The ntermedate level of Fgure 2 shows a dfferent ranked lst for each crteron and the fnal fused lst. In each of these tables, two dfferent ratngs are dsplayed for each ontology. The frst one refers to the smlarty between the ontology and the olden Standard. The second ratng, score, shows the smlarty value normalzed by the sum of all the values. The score measure exhbts the dstrbuton of the ratngs and allows us to better evaluate the dfferent technques.

5 nce the fnal ranked lst has been retreved, the system allows the user to select a subset of ontologes that he consders adequate for the Collaboratve Evaluaton Phase. 3.3 Collaboratve Evaluaton Phase Ths module has been desgned to confront the challenge of evaluatng those ontology features that are by ther nature, more dffcult for machnes to address. Where human udgment s requred, the system wll attempt to take advantage of Collaboratve Flterng technques [2][4][9]. Some approaches for ontology development [20] have been presented n the lterature concernng collaboraton technques. However to our knowledge, Collaboratve Flterng strateges have not yet been used n the context of ontology reuse. The collaboratve module ranks and presents the best ontologes for the user, takng nto consderaton prevous manual evaluatons. Several ssues have to be consdered n a collaboratve system. The frst one s the representaton of user profles. The type of user profle selected for our system s a user-tem ratng matrx (ontologes evaluatons based on specfc crtera). The ntal profle s desgned as a manual selecton of fve predefned crtera [5]: Correctness: specfes whether the nformaton stored n the ontology s true, ndependently of the doman of nterest. Readablty: ndcates the non-ambguous nterpretaton of the meanng of the concept names. Flexblty: ponts out the adaptablty or capablty of the ontology to change. Level of Formalty: hghly nformal, sem-nformal, semformal, rgorously-formal. Type of model: upper-level (for ontologes descrbng general, doman-ndependent concepts), core-ontologes (for ontologes descrbng the most mportant concepts on a specfc doman), doman-ontologes (for ontologes descrbng some doman of the world), task-ontologes (for ontologes descrbng generc types of tasks or actvtes) and applcaton-ontologes (for ontologes descrbng some doman n an applcaton-dependent manner). The above crtera can be dvded n two dfferent groups: ) the dscrete crtera (correctness, readablty and flexblty) that are represented by dscrete numerc values [0,, 2, 3, 4, 5] where 0 ndcates that the ontology does not fulfll the crteron, and 5 ndcates the ontology completely satsfes the crteron and, 2) the boolean crtera (level of formalty and type of model) are represented by a specfc value that s ether satsfed by the ontologes, or not. The collaboratve system does not mplement any profle learnng technque or relevance feedback to update user profles. But, the profles may be modfed manually. After the user profle has been defned, t s mportant to select an approprate type of flterng. For ths work, a contentbased flterng technque has been chosen; ths means, ontologes (our content tems) are recommended based on prevous users evaluatons. Fnally, a type of matchng must also be pcked out for the recommendaton process. In ths work, a novel technque of User Profle-Item matchng s proposed. To evaluate the levels of relevance of the ontologes, the technque wll make comparsons between the user s nterests and the ontology s evaluatons stored nto the system. Ths wll be explaned n secton 5. The rght porton of Fgure 2 shows the Collaboratve Evaluaton module. At the top level the user s nterest can be selected as a subset of crtera wth assocated values representng thresholds that manual evaluatons of the ontologes should fulfl. For example, when a user sets a value of 3 for the correctness crteron, the system recognzes that he s lookng for ontologes whose correctness value s greater than or equal to 3. nce the user s nterests have been defned, the set of manual evaluatons stored n the system s used to compute whch ontologes ft hs nterest best. The ntermedate level shows the fnal ranked lst of ontologes returned by the Collaboratve Flterng module. To add new evaluatons to the system, the user must select an ontology from the lst and choose one of the predetered values for each of the fve aforementoned crtera. The system also allows the user to add some comments to the ontology evaluaton n order to provde more feedback. ne more acton has to be performed n order to vsualze the evaluaton results of a specfc ontology. Fgure 3 shows the user s evaluaton module. n the left sde we can see the summary that the system provdes of the exstng ontology evaluatons wth respect to the user s nterest. In Fgure 3, 3 of 6 evaluatons of the ontology have fulflled the correctness crtera, 5 of 6 evaluatons have fulflled the readablty crtera, and so on. n the rght sde, we can see how the system enables the user to observe all the evaluatons stored nto the system about a specfc ontology. Ths may be of nterest snce we may trust some users more than others durng the Collaboratve Flterng process. Fgure 3. CRE user s evaluatons 4. CNTENT NTLY EVALUATIN In order to obtan smlartes between the olden Standard and the stored ontologes, two dfferent content ontology evaluaton levels have been consdered, the lexcal and the taxonomc. Several measures have been developed and tested for each level.

6 In the followng sectons we present the approaches that have shown better performance. 4. Lexcal Evaluaton Measures The lexcal evaluaton assesses the smlarty between the doman of the problem as descrbed by the olden Standard and an ontology by comparng the lexcal entres, or words that represent them. A new lexcal evaluaton measure based on Maetche and Staab work [] wll be explaned n ths secton. Some defntons must frst be ntroduced. Defnton (Lexcal entry). A lexcal entry l represents a strng or word. Defnton 2 (olden Standard Lexcon). The olden Standard Lexcon, L s defned by the set of lexcal entres extracted from the terms of the olden Standard, where each term has a sngle lexcal entry that represents t. Defnton 3 (ntology Lexcon). The ntology Lexcon, s defned as the set of lexcal entres extracted from the Concepts of the ntology. Each concept s represented by one or more lexcal entres that are extracted from the concept name, the rdfs:label property value, or other propertes that could be added to the lexcal extracton process consderng the characterzaton of each ontology. Defnton 4 (Levenshten dstance). The Levenshten dstance, ed( l, l ) between two lexcal entres l and l measures the mum number of token nsertons, deletons and substtutons to transform l nto l usng a dynamc algorthm. Example: ed(" pzzape"," pzza _ pe ") = Maedche and Staab [] propose a lexcal smlarty measure for strngs called Strng Matchng. Ths method compares two lexcal entres l, takng nto account the Levenshten dstance aganst the shortest lexcal entry. ( l, l ) ed( l, l ) SM ( l, l ) = max(0, ) [0,] ( l, l ) SM returns a degree of smlarty between 0 and, where 0 s a null match and represents a perfect match. Example: SM( pzzape, pzza_pe ) = 7/8. Based on the Strng Matchng they propose a lexcal smlarty measure to compare an ontology to a olden Standard, by computng the average strng matchng between the set of olden Standard lexcal entres and the set of ontology lexcal entres: SM ( L, L ) = max (, ) SM l l L l L l L SM ( L, L ) s an asymmetrc measure that deteres the extent to whch the lexcal level of the olden Standard s covered by the lexcal level of the ntology. Future work must be done n order to penalze those ontologes whch contan all the strngs of the olden Standard but also many others. L There s one prncple dfference between that approach and ours; Maedche defnes the olden Standard as an ontology, whle we use our own model. Ths fact provdes us wth the capablty to use all the addtonal nformaton stored n the olden Standard n order to mprove content evaluaton measures. When a doman s modeled as a set of lexcal entres, some lexcal entres have greater relevance when defnng the semantcs than do others. Assug ths characterstc we have decded to dstngush the mportance of the olden Standard terms. The root terms are consdered the most representatve ones whle the relevance of the expanded terms depends on the number of relatons that separate them from a root term. Wth ths modfcaton we emphasze the man semantcs and relegate the complementary ones nto the background. In ths work we defne the olden Standard Lexcal weght measure to evaluate the mportance of each term. Defnton 5 (olden Standard lexcal weght). ven a lst of lexcal entres L = { l } expanded from a common root lexcal entry, we defne the weght of l L as: max ( Depth( l)) Depth( l) + [, 2] f L > wl () = max ( Depth( l)) 2 otherwse The value returned s represented as a degree of relevance between (the farthest dstance to the root lexcal entry), and 2 (no dstance to the root lexcal entry). If the root lexcal entry has not been expanded we assgn t the weght value 2. Fgure 4 shows an example of ths measure, where T s the root term and consequently has the greater weght. T 3 s the most remote term and t has the smaller weght. The ntermedate terms lke T 2 have a weght between the maxmum and the mum relatve to ther dstance from the root term. Fgure 4. olden Standard lexcal weght measure In our approach, we have modfed the prevous lexcal measure takng nto account the weght or relevance of each term to represent the semantcs of the doman. SM ( L, L ) = max SM( l, l) w( l) L l Lo l L Through our experments, ths new measure has been shown to better dscrate the ontologes, gvng a hgher smlarty value to the ontologes that are closer to the olden Standard and lower ratng to the ontologes that worst ft the problem doman. Future work s needed n order to gve more or less relevance to the derved terms of the olden Standard usng not only ther dstances to the root terms but also, the knd of relaton from whch they have been derved, synonym, hypernym or hyponym.

7 4.2 Taxonomc Evaluaton Measures The taxonomc evaluaton assesses the degree of overlappng between the herarchcal structure of the ontology, defned by the s-a relaton and the olden Standard structure, defned by the dervatons of terms to complete the doman representaton. The followng notatons and defntons wll be used to defne our measure: T C T represents a olden Standard term. C represents an ntology concept. Defnton 8 (Semantc Cotopy of a olden Standard Term). The Semantc Cotopy of a olden Standard term SC( T ) s defned as the set of lexcal entres of the terms derved from the same root term as T, ncludng the lexcal entres of T. Defnton 9 (Semantc Cotopy of an ntology Concept). The Semantc Cotopy of an ntology concept SC( C ) s defned as the set of lexcal entres of the concepts related wth C n the ontology wth a drect relaton, ncludng the lexcal entres of C. ven Maedche and Staab [] measure, an adaptaton s performed to obtan a smlarty between an ontology Concept and a olden Standard Term relatve to the new olden Standard defnton. The smlarty s computed as the ntersecton between the Semantc Cotopy of the olden Standard term and the Semantc Cotopy of the ontology concept normalzed by the total possble overlap. SC( T ) SC( C ) TS( T, C ) = SC( T ) SC( C ) In order for two lexcal entres to be consdered a match, ther smlarty must be greater than a threshold emprcally estmated as 0.2. For smlartes below ths value we have observed there s no sgnfcant morphologcal resemblance between terms. The taxonomc smlarty measure consders all the overlaps between the ntology and the olden Standard. In order to optmze the evaluaton, only a subset of terms and concepts are used to assess the taxonomc smlarty. Ths subset s obtaned through the lexcal measurement, ths s done by selectng only the terms and concepts that have matched wth a smlarty value greater than 0.2. TS( T, C ) = TS( T, C ) T T T, C C ^ l T, l C : SM( l, l) > CLLABRATIVE FILTERIN FR NTLY REUSE In ths secton, a new automatc evaluaton measure that explots the advantages of Collaboratve flterng s proposed. It wll match the set of ontologes or tems that better fulfll the user s nterest explorng the set of manual evaluaton stored nto the system. As we explaned n secton 3 user s evaluatons are represented lke a set of fve defned crtera and ther respectve values manually detered by the user who makes the evaluaton. n the other hand, user s nterests are expressed lke a subset of those crtera, and ther respectve values, meanng a threshold or restrcton to be satsfed by user s evaluatons. Two man steps are presented for ths measure. The frst one descrbes how the smlarty degree between a user s evaluaton crteron and a user s nterest threshold for the same crteron s assessed. The second one descrbes how calculate the fnal rankngs of the ontologes. 5. Collaboratve Evaluaton Measures As we explaned n secton 3.3, the user evaluaton about a specfc ontology s made consderng fve dfferent crtera. These fve crtera are dvded n two dfferent groups: ) the dscrete crtera (correctness, readablty and flexblty), whch take dscrete numerc value [0,, 2, 3, 4, 5], where 0 means that the ontology does not fulfll the crteron, and 5 means the ontology completely satsfy the crteron, and 2) the boolean crtera (level of formalty and type of model) that are represented by specfc values that can be or not satsfed by the ontology. User s nterests are defned lke a subset of those crtera and ther respectve values representng a set of thresholds that the ontologes should fulfll. The user s nterests are szed up aganst the respectve values of those crtera n the user s evaluatons or user s profles. For the boolean case, a value of 0 s returned f the value of the crteron n n the evaluaton m does not fulfll the user s requrements for that crteron, and 2 otherwse. smlarty bool ( crteron ) = 0 f evaluaton 2 f evaluaton threshold = threshold For the dscrete case, the measure ncludes dfferent aspects: a smlarty assessment and a penalty assessment. The smlarty assessment s based on the dstance between the value of the crteron n nto the evaluaton m, and the threshold specfed n the user s nterest for that crteron. The more the value of the crteron n n evaluaton m overcomes the threshold specfed for ths crteron, the greater the smlarty value s. The penalty assess consders how dffcult s to surpass ths threshold. The more dffcult to surpass the threshold, the lower the penalty value s. smlartynum( crteron) = * = + smlartynum( crteron) penaltynum( threshold) Ths measure also returns values between 0 and 2. The consderaton of retrevng a smlarty value between 0 and 2 has taken from other collaboratve matchng measures [9] to not manage negatve numbers. In the case of ths collaboratve measure, negatve smlarty values would be returned when the value of the crteron n the user s evaluatons does not surpass the threshold requred n the user s nterests. 5.2 Collaboratve Evaluaton Rankng The user s nterests and the user s profles, or evaluatons of the ontologes stored nto the system, are used to make the fnal rankng of the ontologes. The smlarty between an ontology evaluaton and the user s requrements s measured as the average of ts N crtera smlartes. N smlarty( evaluaton = m ) smlarty( N crteron ) n=

8 The smlarty of a specfc ontology to the user s requrements s measured as the average of the M evaluatons smlartes for that specfc ontology. M smlarty( ontology) = smlarty( evaluatonm) M m= M N = smlarty( crteron) MN m= n= In case of tes, the fnal collaboratve rankng sorts the ontologes takng nto account not only the average smlarty between the ontologes and the evaluatons stored nto the system, but also the total number of evaluatons of those ontologes, provdng more relevance to those ontologes that have been rated more tmes. M M total smlarty ( ontology ) 6. CNCLUSINS AND FUTURE WRK In ths work a new tool for ontology evaluaton and reuse have been presented, ncludng some nterestng features lke a new olden Standard model, new lexcal evaluaton crtera, the use of rank fuson technques to combne dfferent content ontology evaluaton measures, and the use of a novel Collaboratve flterng strategy to take advantage of user s opnons n order to automatcally evaluate features that only can be assessed by humans. Some ntal experments, not explaned n ths paper, have been developed usng a set of ontologes form the Protégé WL repostory [7] obtanng postve results, but a more detaled and rgorously expermentaton must be done n order to acheve relevant conclusons. 7. ACKNWLEDMENTS Thanks to Enrco Motta, the great drector of the SSSW2005, Aldo angem and Elke Mchlmayr, also members of ths fantastc School, for our 2 a.m. conversatons about ontology evaluaton. Thanks to Alexander omperts for hs help revewng the paper, and specal gratefulness to Denny Vrandecc for encourage us to do ths work. Ths research was supported by the Spansh Mnstry of Scence and Educaton (TIN ). 8. REFERENCES [] Aslam, J. A., Montague, M. Models for metasearch. 24th Annual Internatonal ACM SIIR Conference on Research and Development n Informaton Retreval (SIIR 200). New rleans, Lousana, 200, pp [2] Brank J., robelnk M., Mladenć D. A survey of ontology evaluaton technques. SIKDD 2005 at multconference IS 2005, 7 ct 2005, Lublana, Slovena. [3] Brewester, C. et al. Data drven ontology evaluaton. Proceedngs of Int. Conf. on Language Resources and Evaluaton, Lsbon, [4] Dng, L., et al., Swoogle: A search and metadata engne for the semantc web. Proc. CIKM 2004, pp [5] Fox, E. A., Koushk, M. P., Shaw, J., Modln, R., Rao, D. Combnng evdence from multple searches. st Text REtreval Conference (TREC ). athersburg, Maryland, March 992, pp [6] angem, A., Catenacc, C., Caramta, M., Lehmann, J. A Theoretcal Framework for ntology Evaluaton and Valdaton. In Proceedngs of SWAP2005. [7] omez-perez, A. Some Ideas and Examples to Evaluate ntologes. In Proceedng of the th Conference on Artfcal Intellgence. Los Angeles, CA, February, pp , 995. [8] uarno, N., Welty, C. Evaluatng ntologcal Decsons wth ntoclean. Communcatons of the ACM. 45(2):6-65. New York: ACM Press, [9] Lee, J. H. Analyses of multple evdence combnaton. 20th ACM SIIR Conference on Research and Development n Informaton Retreval (SIIR 97). New York, 997, pp [0] Lozano-Tello, A., ómez-pérez, A. ntometrc: A method to choose the approprate ontology. J. Datab. Mgmt., 5(2): 8 (2004). [] Maedche, A., Staab, S. Measurng smlarty between ontologes. Proc. CIKM LNAI vol. 2473, pp [2] Masthoff J. roup modelng: Selectng a Sequence of Televson Items to Sut a roup of Vewers. User Modelng and User-Adapted Interacton 4: 37-85, [3] Mller,. WordNet: A lexcal database. Communcatons of the ACM, 38(): [4] Montaner M., López B., De la Rosa J.L. A Taxonomy of Recommended Agents on the Internet. Artfcal ntellgence Revew 9: , [5] Paslaru, E. Usng Context Informaton to Improve ntology Reuse. Doctoral Workshop at the 7th Conference on Advanced Informaton Systems Engneerng CASE'05. [6] Porzel, R., Malaka, R. A task-based approach for ontology evaluaton. ECAI 2004 Workshop nt. Learnng and Populaton. [7] Protégé WL ontology Repostory. protege.stanford.edu/plugns/owl/owl-lbrary/ndex.html [8] Renda, M. E., Stracca, U. Web metasearch: rank vs. score based rank aggregaton methods. ACM symposum on Appled Computng. Melbourne, Florda, 2003, pp [9] Resnck P. et al. rouplens: An pen Archtecture for Collaboratve Flterng of Netnews. Internal Research Report, MIT Center for Coordnaton Scence, March 994. [20] Sure Y., Erdmann M., Angele J., Staab S., Studer R., Wenke D. ntoedt: Collaboratve ntology Development for the Semantc Web. ISWC [2] Velard, P., et al. Evaluaton of ntolearn, a methodology for automatc learnng of doman ontologes. In nt. Learnng from Text: Methods, Evaluaton and Applcatons, IS Press, 2005.