Gold-standard evaluation of a folksonomy-based ontology learning model

Transcription

1 Journal of Physics: Conference Series PAPER OPEN ACCESS Gold-standard evaluation of a folksonomy-based ontology learning model To cite this article: E Djuana 2018 J. Phys.: Conf. Ser View the article online for updates and enhancements. Related content - Local ontology for a dual-rail qubit Pawel Blasiak - The ontology in description of production processes in the Industry 4.0 item designing company A V Gurjanov, D A Zakoldaev, A V Shukalov et al. - Ontology to relational database transformation for web application development and maintenance Kamal Mahmudi, M M Inggriani Liem and Saiful Akbar This content was downloaded from IP address on 06/12/2018 at 18:05

2 Gold-standard evaluation of a folksonomy-based ontology learning model E Djuana Computer System Laboratory, Electrical Engineering Department, Faculty of Industrial Technology, Trisakti University, Grogol, Jakarta 11440, Indonesia edjuana@trisakti.ac.id Abstract. Folksonomy, as one result of collaborative tagging process, has been acknowledged for its potential in improving categorization and searching of web resources. However, folksonomy contains ambiguities such as synonymy and polysemy as well as different abstractions or generality problem. To maximize its potential, some methods for associating tags of folksonomy with semantics and structural relationships have been proposed such as using ontology learning method. This paper evaluates our previous work in ontology learning according to gold-standard evaluation approach in comparison to a notable state-of-the-art work and several baselines. The results show that our method is comparable to the state-of the art work which further validate our approach as has been previously validated using task-based evaluation approach. 1. Introduction Collaborative tagging is a collective process whereby web users annotate web resources with their own keywords (tags) as users defined metadata for those resources (Golder and Huberman, 2006; Marlow et al., 2006). Because of this process, there is emerging a kind of informal categorization system for searching and browsing of web resources which is defined by users themselves. This categorization system is known as folksonomy or folks generated taxonomy (Peter, 2009). Folksonomy has been acknowledged for its potential in improving categorization and searching of web resources (Peter, 2009; Bischoff et al., 2008; Robu et al., 2008). Also, there are studies which discover the potential of folksonomies for building semantic resources such as lightweight ontologies or taxonomies (Heymann and Garcia-Molina, 2006; Schmitz, 2006; Mika, 2007; Garcia-Silva, 2012). However, folksonomy may contain inherent semantic ambiguities e.g. synonymy, polysemy and generality problem (Golder and Huberman, 2006). Besides that, it has no explicit structural and semantic relationships among tags (Djuana et al, 2012). Nonetheless, since tags are contributed by users, there are many personal tags which may only be meaningful to themselves such as chapter 1, 101, etc (Bischoff et al., 2008). All these challenges may hinder folksonomy s potential for improving search, browsing and other potential applications such as recommendation. There are many attempts to overcome these challenges, one of which is by associating tags with semantic entities such as lexical resources, dictionary, taxonomy or ontology to make the meaning of tags explicit (Mika, 2007; Garcia-Silva, 2012). Other stream of approaches (Garcia-Silva, 2012) is using clustering based on similarity-based approach (Heymann and Garcia-Molina, 2006) or settheoretical approach (Schmitz, 2006) to consolidate tags into one structure such as taxonomy or Content from this work may be used under the terms of the Creative Commons Attribution 3.0 licence. Any further distribution of this work must maintain attribution to the author(s) and the title of the work, journal citation and DOI. Published under licence by Ltd 1

3 ontology. In this context, how to evaluate the accuracy and effectiveness of the built structures and relationships becomes a crucial issue for these entities to be useful. This paper presents an attempt to evaluate our ontology learning method from folksonomy which has been conducted and presented in previous papers (Djuana et al, 2012, Djuana et al, 2013, Djuana et al, 2014). Previously, the result of this ontology learning method has been evaluated using taskbased evaluation approach or evaluation by application which is in tag recommendation scenario. In this paper, we focus on formal evaluation using gold-standard evaluation approach (Dellschaft and Staab, 2008). The motivation of this further evaluation is to evaluate coverage of this built ontology as it presents the usefulness of this built ontology for wider application as well as to evaluate intrinsic quality of it. The other motivation is to evaluate our method in comparison to other people s method in ontology learning including state-of-the-art method, how well our method is improving over baseline (classic) method in comparison to state-of-the-art s improvement over the baseline against gold standard universal ontology. We compare this built ontology to one notable state-of-the-art work by Liu et al. (2010) which has been compared to two notable baseline methods: Heymann and Garcia-Molina (2006) which represents similarity-based approach and Schmitz (2006) which represents settheoretical approach. This paper is structured as follows. In Section 2 we discuss the key concepts in ontology learning, with specific emphasis in ontology learning from folksonomy, and ontology evaluation. Section 3 discusses related works in terms of ontology learning and ontology evaluation, which discusses the two baselines and the state the art work as the benchmark. Section 4 summarizes our proposed approach from our previous papers. In Section 5 we discuss the evaluation settings and in Section 6 we discuss the evaluation results. Finally, Section 7 concludes this paper with a conclusion and direction for future work. 2. Key Concepts In this section, we provide brief introduction on ontology and ontology learning, specific class of ontology learning from folksonomy and ontology learning evaluation approaches Ontology and Ontology Learning Gruber defined that ontology is formal description and explicit specification of a shared conceptualization (Gruber, 1992). Depending on the types of stored knowledge, ontology can be differentiated in two types: domain ontology and general ontology (Navigli et al., 2003). General ontology defines concepts that are general for all domains while domain defines specific concepts that forms the core knowledge for one specific domain. Construction of ontology from Web contents is one important task in Web intelligence according to Zhong and Hayazaki (2002). This work points out that ontologies serves the Semantic Web by providing a controlled vocabulary of concepts, each with explicitly defined and machine-processable semantics. However, manual ontology construction is time consuming and very costly. Therefore, automatic and semi-automatic ontology constructions have been eagerly studied over the last decade (Maedche and Staab, 2001). One stream of approach relies on machine learning and automated languageprocessing techniques to extract concepts and ontological relations from structured or unstructured data such as database and text (Navigli et al., 2003) Ontology Learning from Folksonomy Mika stated that folksonomy which is emerging from collaborative tagging has been acknowledged as potential source for constructing ontology. As it captures vocabulary of users which may be aggregated to produce emergent semantics, people may develop lightweight ontologies (Mika, 2007). In this context, folksonomies can be referred to as a new data source for ontology learning that can be 2

4 analyzed using techniques already used in this area such as clustering, natural language processing, and formal concept analysis (Garcia-Silva et al, 2012). Construction of ontology from Web contents is one important task in Web intelligence according to Zhong and Hayazaki (2002). This work points out that ontologies serves the Semantic Web by providing a controlled vocabulary of concepts, each with explicitly defined and machine-processable semantics. In our previous work we have taken Garcia-Silva et al (2012) s view which describes the most relevant approaches described in the literature whose main objective is either to extract ontologies from tags in folksonomies or to associate tags to external semantic entities to make explicit the meaning of those tags. They have identified three group of approaches which are based on 1) clustering techniques i.e. to cluster tags according to some relations among them (statistical techniques); 2) ontologies i.e. aiming at associating semantic entities e.g. WordNet, Wikipedia, to tags to formally define their meaning; 3) hybrid approach i.e. mixing clustering techniques and ontologies (Djuana et al, 2012) Ontology Learning Evaluation In this section, we summarize the strategies for evaluating ontology learning approaches according to comprehensive evaluation strategies proposed by Dellschaft and Staab (2008) Preliminaries. Dellschaft and Staab (2008) have introduced two scenarios in ontology learning evaluation which are 1) evaluation of the learning algorithm in the broader context of an automatic or semi-automatic approach to ontology engineering whereby not only the learning algorithm influences the results but also the choice of the correct corpus which must contain information relevant for the task; and 2) evaluation of the quality of the learning algorithm itself. There are two dimensions of ontology which needs to be evaluated: functional and structural. The functional dimension of an ontology is related to its conceptualization while the structural dimension is related to the representation of an ontology as a graph. While the structural evaluation may pinpoint areas where the problems could exist, it is the functional evaluation that more crucial for evaluating the usefulness or effectiveness of the learned ontology. Dellschaft and Staab (2008) have argued that for the first scenario, the functional dimension of an ontology should be evaluated by means of an extrinsic, task-based evaluation, i.e. in the running application for which the ontology is engineered while in the second scenario, an intrinsic or taskneutral evaluation by means of a gold-standard based evaluation is usually the better choice. Task-based approaches (among other approaches such as corpus-based and criteria-based) is trying to measure in how far an ontology helps to improve the results of a certain task. A task-based evaluation is influenced by many aspects which must be kept constant during all evaluations so that changes in the results can be put down to the changes in the used ontologies (Dellschaft and Staab, 2008). Gold-standard based approaches (as supposed to manual evaluation by human experts) compare the learned ontology with a previously created gold standard which represents an idealized outcome of the learning algorithm. A learning algorithm is better when the learned ontology has a high similarity with the gold standard (Dellschaft and Staab, 2008) Gold-standard evaluation approach. The gold-standard based evaluation approach for evaluating ontologies may involve several measures. They can be distinguished between measures which only evaluate the lexical layer of an ontology, the ones which also take the concept hierarchy or taxonomic layer into account and the ones which evaluate the non-taxonomic relations contained in an ontology (Dellschaft and Staab, 2008). In this paper, we will concentrate on the measures for evaluating the lexical and the taxonomic layer. 3

5 The lexical layer or also known as coverage measure are often used for comparing the terms from the reference and the learned ontology based on an exact match of strings. Examples for this kind of measure are the Term Precision and Term Recall or also known as Lexical Precision and Recall. The taxonomic layer or also known as relationships measure compares the similarity of the positions of two concepts in the learned and the reference hierarchy as the local measure. The global measure is then computed by averaging the results of the local measure for concept pairs from the reference and the learned ontology. It is usually calculated using local taxonomic over-lap which compares two concepts based on the set of all their super- and sub concepts. These two measures will be described in more details in Section Related Works In this section, we describe related body of works in ontology learning from folksonomy data for evaluating our proposed method (Djuana et al, 2012, Djuana et al, 2013, Djuana et al, 2014) using gold-standard approach in addition to previous evaluation using task-based approach (tag recommendation). First, we describe two classic baseline approaches by 1) Heymann and Garcia-Molina (2006) and 2) Schmitz (2006); which are well known for their effectiveness and ease of implementation in Section 3.1. Then, we describe related state-of-the-art approaches and we specifically describe a method by Liu et al. (2010) which we are using as benchmark for our ontology validation in Section Similarity-based approach The algorithm published by Heymann and Garcia-Molina (2006) is classified as similarity-based approach according to classification by Liu et al (2010). The algorithm works on tag vectors whose index is equal to the number of times that a tag annotates an object. Then it calculates the similarity between tags using the cosine similarity between tag vectors to build tag similarity graph where each tag is represented by a vertex, and two vertices are connected by an edge if the similarity of the nodes they represent is above some set threshold. In the explanation provided by Liu et al (2010), for building up the hierarchy of tags or taxonomy, the algorithm starts with a single node tree whose only node is the root" node representing the top of the tree. Then, it adds each tag in the tagging system to the tree in decreasing order of how central the tag is to the similarity graph described above. It decides where to put each candidate tag by computing its similarity to every node currently present in the tree, keeping track of the most similar node. The candidate tag is then either added as a child of the most similar node if its similarity to that node is greater than some threshold, or it is added to the root node if there does not currently exist a good parent for that node (Liu et al, 2010) Set-theoretical approach The algorithm published by Schmitz (2006) is classified as set-theoretical approach according to classification by Liu et al (2010). It is an extension of the algorithm published by Sanderson and Croft (1999). The algorithm is based on subsumption model which partially order the concepts with the pairwise subsumption relations. The subsumption relation of two concepts is usually derived by a settheoretical method according to the inclusion relation between their specific attribute sets, such as the occurrences of terms in documents. It is originally defined as follows, for two terms, x and y, x is said to subsume y if the following two conditions hold: P(x y) = 1, P(y x) < 1 In other words, x subsumes y if the documents which y occurs in are a subset of the documents which x occurs in. Because x subsumes y and because it is more frequent, in the hierarchy, x is the parent of y. Although a respectable number of term pairs were found that adhered to the two subsumption conditions, it was noticed that many were just failing to be included because a few 4

6 occurrences of the subsumed term, y, did not co-occur with x. Subsequently, the first condition was relaxed and subsumption was redefined as: P(x y) 0.8, P(y x) < 1 as is described by Liu et al (2010). Schmitz (2006) adjusting Sanderson and Croft (1999) s statistical thresholds to reflect the ad hoc usage and adding filters to control for highly idiosyncratic vocabulary as follows: P(x y) t, Dx Dmin, Ux Umin, P(y x) < t, Dy Dmin, Uy Umin Where: t is the co-occurrence threshold, Dx is the # of documents in which term x occurs, and must be greater than a minimum value Dmin, and Ux is the # of users that use x in at least one annotation, and must be greater than a minimum value Umin State of the Art Approaches According to framework proposed by Garcia-Silva et al (2012), our proposed work falls into the second group which is based on ontologies. It will be discussed in detail in Section 4. Outside the list from survey conducted by Garcia-Silva et al (2012), we have described previously in Djuana et al (2012) that there are several recent works which tried to extract ontological structures from user tagging systems. Lin, Davis and Zhou (2009) extracted ontological structures by exploiting low support association rule mining supplemented by WordNet. Trabelsi, Jrad and Yahia (2010) focused more on extracting non-taxonomic relationships from folksonomies using triadic concepts with external resources: WordNet, Wikipedia and Google. Tang et al. (2009) and Liu et al. (2010) represents state-of-the-art work for generating ontology from folksonomy based on generative probabilistic models i.e. tag-topic model and set-theoretical approach i.e. to produce tag subsumption graph respectively. We chose state-of-the-art work by Liu et al. (2010) as a benchmark to evaluate our proposed approach as it is a major improvement to the subsumption models based on set-theoretical approach, by proposing to reduce noisy subsumptions including irrelevant and inconsistent paths. It is also chosen for its comprehensive evaluation which has been conducted against the classic baselines (Heymann and Garcia-Molina, 2006; Schmitz, 2006) according to evaluation framework proposed by Dellschaft and Staab (2008). The approach by Liu et al (2010) consists of 3 steps. In the first step, it identifies subsumption tags with a set-theoretical method. Since the subsumption relations discovered in this step may be inconsistent, it then resorts to ranking the tags by generality to settle this problem. Therefore, in the second step, it constructs a tag subsumption graph to compute the generality scores of tags with a random walk based procedure. In the last step, it uses an agglomerative clustering approach, which leverages the result of the tag generality ranking procedure, to generate the concept hierarchy. 4. Proposed Model To describe our proposed approach, we present several definitions below. This approach has been summarized here as a background for the evaluation purpose and the details has been published in previous papers (Djuana et al, 2012, Djuana et al, 2013, Djuana et al, 2014) Definitions Collaborative Tagging System. A collaborative tagging system contains three entities: users, tags, and items, which are described below: Users U = {u 1, u 2,.. u U } contains all users in an online community who have used tags to annotate their items; 5

7 Tags T = {t 1, t 2,.. t T } contains all tags used by users in U. Tags are typically arbitrary strings which could be a single word or short phrase. In this respect, a tag is defined as a sequence of terms. For t T, t =< term 1, term 2,, term m >, a function tagset(t) = {term 1, term 2,.. term m } is defined to return the terms in a tag; Items I = {i 1, i 2,.. i I } contains all domain-relevant items or resources. What is considered by an item depends on the type of collaborative tagging system, for instance, in Delicious the items are mainly bookmarks; Based on these three entities, a collaborative tagging system is formulated as Folksonomy which consists of 4-tuple: F = (U, T, I, Y) where U, T, I are finite sets, whose elements are the users, tags and items, respectively. Y is a ternary relation between those elements, i.e. Y U T I, whose elements are called the tag assignments, whereby an element (u, t, i) Y represents that user u annotated item i using tag t General Ontology. The general ontology is defined as a 2-tuple GeneralONTO = (C, R). where C = {c 1, c 2,.., c C } is a set of concepts; R = {r 1, r 2,.., r R } is a set of relations representing the relationships between concepts. A concept c in C is a 3-tuple c = (id, synset, category) where id is a unique identification of concept c; synset is a synonym set containing synonymic terms which represent the meaning of the concept c; and category is a taxonomic category to classify this concept c. A relation r in the relation set R is a 3-tuple r = (type, x, y), where type {is_a, }; x, y C are the concepts that hold the relation r. Specifically, there are the set of synonyms representing c which represented as synset(c) and the category of c as category(c). For each term w in synset (c), w is represented as a 2-tuple (w, freq c (w)) where w is a synonym term of the concept c; freq c (w) is the frequency as an indication of how frequently this term has been used to represent the meaning of the concept c based on the accompanying corpus. For a term w, the set of concepts for which w is a synonymic term is defined as con(w) = {c (w, f) synset(c)} Domain Ontology. The domain ontology is defined as 2-tuple DomainOnto = (TC, TR). where TC = {tc 1, tc 2,.., tc TC } is a set of tag-concepts, i.e., TC C 2 T, and TR = {tr 1, tr 2,.., tr TR } is a set of tag relations. Each element in TC is a pair of a concept c and a set of tags {t 1, t 2,.. t n }, i.e., tc = (c, {t 1, t 2,.. t n } ) TC, which represents that each tag in {t 1, t 2,.. t n } can be mapped to concept c. TR is defined as: r R, TR = {r = (type, c 1, c 2 ) Concept_Tag(c 1 ), } Concept_Tag(c 2 ) 4.2. Ontology Learning Process From the backbone ontology, it was expected by conducting ontology learning process; domain ontology which represents a tag collection can be generated. It is expected that this domain ontology will contain sub ontology of the backbone ontology which contextualized to the tag vocabulary and possibly personalized to users tag usage in that collection. This sub ontology is expected to have tag to concept mapping and the taxonomic relationships between tags which extracted from concept to concept relationships in the backbone ontology. The lexical knowledge base WordNet (Fellbaum, 1998) was chosen as the backbone ontology as it has wide coverage of concepts (over 200,000) and richness of relationships such as semantic relationships is-a, part-of, lexical relationships synonymy and antonymy as well as availability of accompanying corpus and other facility for disambiguation process. We have summarized the 3 stages in the domain ontology generation process which are mapping tags to concepts, mapping disambiguation and relationships extraction. 6

8 Mapping Tags to Concepts. One tag may contain one or more terms. It is possible that a tag can map directly to one of synonym terms of a concept in the backbone ontology. In other cases, only part of a tag that can map to one of synonym terms. These cases where handled by three mapping approaches which are (1) whole mapping where by whole tag string can be mapped to a synonym terms in a concept; (2) partial mapping where by partial tag string, after phrase identification stage, can be mapped to a synonym terms in the concept; and (3) term mapping where by each individual term in tag string is mapped to a synonym terms in a concept. This mapping is represented as tag to concept mapping and one tag may map to more than one concepts. Overall, for t T, the tag to concept mapping is defined as follows: Tag_Concept whole (t), directly mapped Tag_Concept(t) = { Tag_Concept partial (t), partially mapped Tag_Concept term (t), term mapped Mapping Disambiguation. After all the possible mappings are found, the next stage was mapping disambiguation to choose the most appropriate concept from mapped concepts to represent the meaning of tag for this tag collection. Two disambiguation strategies were performed which are (1) disambiguation by frequency which comes from an expert point of view about general meaning of tags. This mapping strength comes from frequency in a representative corpus of documents which indicate how frequent one synonym terms would be used to represent the meaning of concept that contains these terms; (2) disambiguation by tag relevance which comes from users point of view about a personal meaning in the tags collection. This mapping strength comes from the tag relevance in relation to similar users understanding and usage of tags. Given a related tag that has been used for an item, this mapping is chosen according to the relevance to other tags. After mapping disambiguation, each tag t will be mapped to one and only one concept. This can be defined by a one to one disambiguation mapping M γ : T C, M γ (t) = argmax (T C [t, c] ) (1) γ c Tag Concept(t) where matrix T_C[t i, c j ] n is defined to represent the strength of the mapping between tags and concepts, where m= T and n= C and γ is a mapping disambiguation strategy. On the other hand, multiple tags may also be mapped to one concept. The following function defines the mapping from a concept to tags: Concept_Tag: C 2 T, Concept_Tag(c) = {t t T, M γ (t) == c} At the end, the confirmed mapping according to two disambiguation strategies were: M frequency (t) and M relevance (t) Relationships Extraction. Once mapping tags to concepts and mapping disambiguation processes are completed, each tag will map to a concept on the backbone ontology. Based on tag to concept mappings, available relationships ( is-a relation) among concepts in general ontology were extracted to form the domain ontology. 5. Evaluation 5.1. Evaluation Method To compare the intrinsic quality of our proposed method we use the gold standard evaluation approach as discussed in Section For task-based evaluation results please refer to our previous papers (Djuana et al, 2012, Djuana et al, 2013, Djuana et al, 2014). 7

9 Following gold-standard ontology chosen by Liu et al. (2010), we use the concept hierarchy from Open Directory Project (ODP) 1. ODP is a free, user-maintained hierarchical web directory. Each node in the ODP hierarchy has a topic label (e.g. Sports or Arts) and a set of associated URLs. ODP is generated by collaborating users. In the following the simplified definition of a core ontology will be used. This definition of an ontology only contains the lexical layer and the concept hierarchy. In Dellschaft and Staab (2008) a core ontology is defined as follows: The structure O: = (C, root, c) is called a core ontology. C is a set of concept identifiers and root being a designated root concept for the partial order c on C. This partial order is called concept hierarchy or taxonomy. The equation c C: c root holds for this concept hierarchy. Given a computed core ontology O C and a reference ontology O R, the lexical precision (LP) and lexical recall (LR) are defined as follows: LP(O C, O R ) = C C C R (2) C C LR(O C, O R ) = C C C R C R To compare between the gold standard and the learned ontology, each sub tree which starts with different ODP topic label will be compared to its corresponding ODP sub tree using lexical precision and lexical recall. The higher the value of precision and recall the more accurate the learned ontology to the gold standard ontology Experiment Setup Dataset and Experiment Run. One public folksonomy dataset was used for the experiment. We use the subset of Delicious dataset provided by Wetzker et al (2008) which contains all public bookmarks of users posted on delicious.com between September 2003 and December In our experiment, we use the data between September 2003 and July We also perform a filtering for the dense part of the folksonomy using p-core calculation according to Batagelj and Zaversnik (2002). This p-core 15 calculation reduces the folksonomy population to tags, user, and items which appear in at least 15 posts. The overall statistics is presented in Table 1. (3) Table 1. Delicious dataset statistics All Filtered by p-core=15 #users 75,245 24,562 #items 3,158,4 45,793 #tags 456, ,718 #posts 7,698,6 53 1,436,52 7 We have implemented the two baselines by Heyman and Garcia-Molina (2006) and Schmitz (2006) and ran their algorithms with our dataset and produced two versions of learned ontology structures. We have run our proposed algorithm with the same dataset and produced a version of learned ontology structure. For each of this learned ontology we are calculating lexical precision and recall against the gold standard. We did not implement Liu s method, but instead we are using results from Liu et al (2010) s paper directly since they are also comparing against the same two baselines. To compare with Liu s, we are measuring the percentage of improvement from our results in comparison to the two baselines using our dataset with the percentage of improvement in Liu et al (2010) s paper. By comparing the percentage of improvement, we are indirectly comparing the accuracy of the two algorithms

10 Table 2. Improvement Results between the Proposed Method to Baselines Lexical Precision Lexical Recall Sub tree Ours Heymann Schmitz Ours vs Ours vs Ours Heymann Schmitz Ours vs Ours vs Heymann Schmitz Heymann Schmitz sport science news program history book culture computers game education resources media shop graphic health all average Table 3. Improvement Results between the State-of-the-Art Method to Baselines Lexical Precision Lexical Recall Sub tree Liu Heymann Schmitz Liu vs Liu vs Liu Heymann Schmitz Liu vs Liu vs Heymann Schmitz Heymann Schmitz sport science news program history book culture computers game education resources media shop graphic health all average

11 6. Results and Discussion The results for our proposed method are presented in Table 2 while the results from Liu et al (2010) s paper are copied in Table 3 for easy comparison. The average value is the average over all the sub trees but not include the all root. The all root is when all of sub trees are connected which means for all overlap sub trees, the overlapped part will be merged. Overall results for the proposed method are better against the two baselines with the improvement over Heymann and Garcia algorithm is higher than the improvement over Schmitz algorithm. In this dense part of folksonomy, the proposed method is also following the trend shown by Liu et al s method because the absence of all idiosyncratic and personal tags. Overall results also show that the proposed methods in average and all situations are better in terms of coverage which are shown by higher value of lexical precision and recall. The improvement in recall value are higher than in precision which shows that the proposed method is better in coverage against Liu et al s method. However, as we compare the percentage of improvement over Schmitz algorithm, there are several sub trees such as program (programming), computers and game which has lower improvement of precision value than Liu et al. s method, although comparing to Heymann algorithm the improvement is still better. It is suspected that since the proposed method is partly based on WordNet which is contains more general vocabularies in comparison to ODP, these more technical sub trees were not performed that well. In this evaluation, we haven t included the taxonomic evaluation in terms of taxonomic precision and recall which evaluates the closeness of relationships structure of the learned ontology to the gold standard. At this stage, we can conclude that based on coverage alone, the proposed method is better and for overall evaluation, the proposed method may be comparable. This is subject to taxonomic evaluation to be conducted as future work. 7. Conclusions We have presented an attempt to evaluate the proposed ontology learning approach using goldstandard evaluation approach, which previously evaluated using task-based evaluation approach. The lexical evaluation (coverage) show a positive indication that the proposed method is better than notable state-of-the-art method. This is subject to a taxonomic (relationships) evaluation which is to be conducted as a near future work. References [1] Batagelj, V. and Zaversnik, M Generalized cores, Arxiv preprint cs/ [2] Bischoff, K., Firan, C.S., Nejdl, W. and Paiu, R Can all tags be used for search? In Proceedings of the 17th ACM Conference on Information and Knowledge Management, ACM, New York, NY, [3] Dellschaft, K. and Staab, S Strategies for the evaluation of ontology learning, In Proceedings of the 2008 conference on Ontology Learning and Population: Bridging the Gap between Text and Knowledge, IOS Press, Amsterdam, The Netherlands, [4] Djuana, E., Xu, Y. and Li, Y Learning personalized tag ontology from user tagging information. In Proceedings of the Tenth Australasian Data Mining Conference (Sydney, Australia, December 05-07, 2012). AusDM ACS, Sydney, Aus, [5] Djuana, E., Xu, Y., Li, Y and Cox, C Personalization in tag ontology learning for recommendation making. In Proceedings of the 14th International Conference on Information Integration and Web-based Applications & Services (Denpasar, Bali, December 03-05, 2012). iiwas '12. ACM, New York, NY, [6] Djuana, E., Xu, Y., Li, Y., Josang, A. and Cox, C An ontology based method for sparsity problem in tag recommendation. In Proceedings of the 15 th International 10

12 Conference on Enterprise Information Systems (Angers, France, July 04-07, 2013). ICEIS SCITEPRESS, INSTICC, Portugal, [7] Djuana, E., Xu, Y., Li, Y., and Josang, A A Combined Method for Mitigating Sparsity Problem in Tag Recommendation. In Proceedings of the 47th Hawaii International Conference on System Sciences (Waikoloa, HI, USA, January 6-9, 2014). IEEE Computer Society 2014, [8] Fellbaum, C. (ed.), WordNet: An Electronic Lexical Database, Cambridge, MA: MIT Press. [9] García-Silva, A., Corcho, O., Alani, H. and Gómez-Pérez, A Review of the state of the art: discovering and associating semantics to tags in folksonomies. The Knowledge Engineering Review. 27, 1 (Feb. 2012), Cambridge University Press, [10] Golder, S.A. and Huberman, B.A Usage patterns of collaborative tagging systems. J. Inf. Sci. 32, 2 (Apr. 2006), [11] Gruber, T.R A translation approach to portable ontology specifications. Knowledge Acquisition, 5, 2, [12] Heymann, P. and Garcia-Molina, H Collaborative creation of communal hierarchical taxonomies in social tagging systems. Technical Report Stanford University. [13] Lin, H., Davis, J. and Zhou, Y An integrated approach to extracting ontological structures from folksonomies, The Semantic Web: Research and Applications, Springer, [14] Liu, K., Fang, B. and Zhang, W Ontology emergence from folksonomies, In Proceedings of the 19 th ACM International Conference on Information and Knowledge Management, ACM, New York, NY, [15] Maedche, A. and Staab, S Ontology learning for the semantic web. IEEE Intelligent Systems, 16, 2, (March 2001), IEEE, NJ, US, [16] Marlow, C., Naaman, M., Boyd, D. and Davis, M HT06, tagging paper, taxonomy, Flickr, academic article, to read, In Proceedings the Seventeenth Conference on Hypertext and Hypermedia, ACM, New York, NY, [17] Mika, P Ontologies are us: A unified model of social networks and semantics. Web Semantics: Science, Services and Agents on the World Wide Web. 5, 1 (March. 2007), [18] Navigli, R., Velardi, P. and Gangemi, A Ontology learning and its application to automated terminology translation. IEEE Intelligent Systems, 18, 1, (Jan 2003), IEEE, NJ, US, [19] Peters, I Folksonomies. Indexing and Retrieval in Web 2.0. De Gruyter Saur, Berlin, Germany. [20] Robu, V., Halpin, H., and Shepherd, H Emergence of consensus and shared vocabularies in collaborative tagging systems. ACM Trans. Web. 3, 4 (Sep. 2009), 14:1-34. [21] Sanderson, M. and Croft, B Deriving concept hierarchies from text. In Proceedings of the 22nd Annual International ACM SIGIR conference on Research and Development in Information Retrieval. SIGIR'99, [22] Schmitz, P Inducing ontology from flickr tags. In Proceedings of the Collaborative Web Tagging Workshop at WWW'06, (Edinburgh, Scotland, May 23-26, 2006). [23] Tang, J., Leung, H., Luo, Q., Chen, D. and Gong, J Towards ontology learning from folksonomies, In Proceedings 21 st International Joint Conference on Artificial Intelligence, [24] Trabelsi, C., Jrad, A.B., and Yahia, S.B Bridging folksonomies and domain ontologies: Getting out non-taxonomic relations, In Proceedings IEEE International Conference on Data Mining Workshops, IEEE, [25] Wetzker, R., Zimmermann, C. and Bauckhage, C Analyzing social bookmarking systems: A del.icio.us cookbook, In Proceedings of European Conference on Artificial Intelligence (ECAI). 11

13 [26] Zhong, N. and Hayazaki, N., Roles of ontologies for web intelligence. In M.-S. Hacid, Z. Ras, D. Zighed & Y. Kodratoff (Eds.), Foundations of Intelligent Systems, 2366, Springer Berlin, Heidelberg,