ه عا ی Semantic Web Ontology Engineering and Evaluation Morteza Amini Sharif University of Technology Fall 93-94
Outline Ontology Engineering Class and Class Hierarchy Ontology Evaluation 2
Outline Ontology Engineering Class and Class Hierarchy Ontology Evaluation 3
What Is Ontology Engineering? Ontology Engineering: Defining terms in the domain and relations among them. Defining concepts in the domain (classes) Arranging the concepts in a hierarchy (subclass-superclass hierarchy) Defining which attributes and properties (slots) classes can have and constraints on their values Defining individuals and filling in slot values 4
Ontology Development Process here: determine scope consider reuse enumerate terms classes properties constraints create instances In reality - an iterative process: determine scope consider reuse enumerate terms consider reuse classes enumerate terms classes properties classes properties constraints create instances classes create instances consider reuse properties constraints create instances 5
Determine Domain and Scope determine scope consider reuse enumerate terms classes properties constraints create instances What is the domain that the ontology will cover? For what we are going to use the ontology? For what types of questions the information in the ontology should provide answers? 6
Consider Reuse determine scope consider reuse enumerate terms classes properties constraints create instances Why reuse other ontologies? to save the effort to interact with the systems and tools that use other ontologies to use ontologies that have been validated through use in applications 7
What to Reuse? Ontology libraries DAML ontology library (www.daml.org/ontologies) Ontolingua ontology library (www.ksl.stanford.edu/software/ontolingua/) Protégé ontology library (protege.stanford.edu/plugins.html) Upper ontologies IEEE Standard Upper Ontology (suo.ieee.org) Cyc (www.cyc.com) General ontologies DMOZ (www.dmoz.org) WordNet (www.cogsci.princeton.edu/~wn/) Domain-specific ontologies UMLS Semantic Net GO (Gene Ontology) (www.geneontology.org) 8
Enumerate Important Terms determine scope consider reuse enumerate terms classes properties constraints create instances What are the terms we need to talk about? What are the properties of these terms? What do we want to say about the terms? 9
Define Classes and the Class Hierarchy determine scope consider reuse enumerate terms classes properties constraints create instances A class is a concept in the domain a class of books a class of authors a class of romances A class is a collection of elements with similar properties Instances of classes Thinking on the Web book 10
Class Inheritance Classes usually constitute a taxonomic hierarchy (a subclass-superclass hierarchy). A class hierarchy is usually an IS-A hierarchy: an instance of a subclass is an instance of a superclass. If you think of a class as a set of elements, a subclass is a subset. E.g., Apple is a subclass of Fruit. Every apple is a fruit 11
Levels in the Hierarchy Person (Top) Female Top level Middle level Parent Woman Mother Mother-having-only-female-kids Mother-having-only-daughters Bottom level Grandma Great-grandma 12
Modes of Development Top-Down the most general concepts first and then specialize them. Bottom-Up the most specific concepts and then organize them in more general classes. Combination the more remarkable concepts first and then generalize and specialize them. 13
Documentation Classes (and Properties) usually have documentation Describing the class in natural language Listing domain assumptions relevant to the class definition Listing synonyms Documenting classes and slots is as important as documenting computer code! 14
Define Properties (Slots) of Classes determine scope consider reuse enumerate terms classes properties constraints create instances Properties in a class definition describe attributes of instances of the class and relations to other instances Each book has title, author, ISBN, etc. 15
Properties (Slots) Types of properties intrinsic properties: type and color of book extrinsic properties: ISBN of book Simple and complex properties simple properties (attributes): contain primitive values (strings, numbers) complex properties: contain (or point to) other objects (e.g., a publication as a publisher of a book) 16
Property Constraints (Facets) determine scope consider reuse enumerate terms classes properties constraints create instances Property constraints (facets) describe or limit the set of possible values for a property The title of a book is a string The book publisher is an instance of Publication A publication has exactly one location 17
An Example: Domain and Range DOMAIN class slot RANGE allowed values When defining a domain or range for a slot, find the most general class or classes. Consider the book-type slot Domain: Physics, Chemistry, Mathematics, Engineering Domain: Scientific Consider the publishes slot for a Publication: Range: PhysicsBooks, ChemistryBooks, MathematicsBooks Range: FundamentalScienceBooks 18
Create Instances determine scope consider reuse enumerate terms classes properties constraints create instances Create an instance of a class The class becomes a direct type of the instance Any superclass of the direct type is a type of the instance Assign slot values for the instance frame Slot values should conform to the facet constraints 19
Outline Ontology Engineering Class and Class Hierarchy Ontology Evaluation 20
Defining Classes and a Class Hierarchy The things to remember: There is no single correct class hierarchy But there are some guidelines The question to ask: Is each instance of a subclass an instance of its superclass(es)? 21
Transitivity of the Class Hierarchy The IS-A relationship is transitive: B is a subclass of A C is a subclass of B C is a subclass of A A direct superclass of a class is its closest superclass. 22
Multiple Inheritance A class can have more than one superclass. A subclass inherits slots and facet restrictions from all the parents. Parent Woman Mother 23
Disjoint Classes Classes are disjoint if they cannot have common instances. Disjoint classes cannot have any common subclasses either. Man and Woman are disjoint. Parent and Woman are not disjoint. Man Parent Person Woman 24
Avoiding Class Cycles Danger of multiple inheritance: cycles in the class hierarchy. Classes A, B, and C have equivalent sets of instances By many definitions, A, B, and C are thus equivalent. 25
The Perfect Family Size If a class has only one child, there may be a modeling problem. If a class has more than a dozen children, additional subcategories may be necessary However, if no natural classification exists, the long list may be more natural. 26
Classes and Their Names The class name can change, but it will still refer to the same concept. Synonym names for the same concept are not different classes. Many systems allow listing synonyms as part of the class definition. Class names should be either all singular all plural (Class) Books (Instance) SW Book 27
Content: Top-Level Ontologies What does top-level mean? describes very general concepts that are the same across all knowledge domains. objects: tangible, intangible processes, events, actors, roles agents, organizations spaces, boundaries, location time IEEE Standard Upper Ontology effort (SUMO) Goal: Design a single upper-level ontology Process: Merge upper-level of existing ontologies 28
CYC: Top-Level Categories (OpenCyc Project) 29
WORDNET 30
Sowa s Ontology (by John Sowa) 31
Ontology Evaluation Key factor which makes a particular discipline or approach scientific is the ability to evaluate and compare the ideas within the area. In most practical cases ontologies are a non-uniquely expressible. One can build many different ontologies which conceptualizing the same body of knowledge. We should be able to say which of these ontologies serves better some pred criterion. 32
Categories of Ontology Evaluation Those based on comparing the ontology to a "golden standard (an ontology). Those based on using the ontology in an application and evaluating the results of it. Those involving comparisons with a source of data (e.g. a collection of documents) about the domain that is to be covered by the ontology. Those where evaluation is done by humans who try to assess how well the ontology meets a set of pred criteria, standards, requirements, etc. 33
Different Levels of Evaluation Lexical, vocabulary, or data layer Hierarchy or taxonomy Other semantic relations Context or application level Syntactic level Structure, architecture, design Multiple-criteria approaches 34
A: Lexical, Vocabulary, or Data Layer The focus is on which concepts, instances, facets, etc. have been included in the ontology, and the vocabulary used to represent or identify these concepts. Evaluation on this level tends to involve comparisons with various sources of data concerning the problem, as well as techniques such as string similarity measures (e.g. edit distance). MAEDCHE AND STAAB (2002). Concepts are compared to a Golden Standard set of strings that are considered a good representation of the concepts. Golden standard Another ontology Taken statistically from a corpus of documents Prepared by domain experts. 35
B: Hierarchy or Taxonomy An ontology typically includes a hierarchical is-a or subsumption relation between concepts. BREWSTER et al. (2004) used a data-driven approach to evaluate the degree of structural fit between an ontology and a corpus of documents. Cluster the documents and make topic representing documents Each concept C of the ontology is represented by a set of terms including its name in the ontology and the hypernyms of this name, taken from Wordnet. Y is a hypernym of X if every X is a (kind of) Y. Measure how well a concept fits a topic results from the clustering step. Indicate that the structure of the ontology is reasonably well aligned with the hidden structure of topics in the domain-specific corpus of documents. 36
C: Context Level An ontology may be part of a larger collection of ontologies, and may reference or be referenced by various definitions in these other ontologies. In this case it may be important to take this context into account when evaluating it. Swoogle search engine uses cross-references between semantic-web documents to a graph and compute a score for each ontology in a manner analogous to PageRank used by the Google web search engine. The resulting ontology rank is used by Swoogle to rank its query results. An important difference in comparison to PageRank is that not all links or references between ontologies are treated the same. If one ontology s a subclass of a class from another ontology, this reference might be considered more important than if one ontology only uses a class from another as the domain or range of some relation. 37
D: Application Level It may be more practical to evaluate an ontology within the context of particular application, and to see how the results of the application are affected by the use of ontology in question. The outputs of the application, or its performance on the given task, might be better or worse depending partly on the ontology used in it. One might argue that a good ontology is one which helps the application in question produce good results on the given task. 38
E: Syntactic Level For manually constructed Ontologies. The ontology is usually described in a particular formal language and must match the syntactic requirements of that language (use of the correct keywords, etc.). This is probably the one that lends itself the most easily to automated processing. 39
F: Structure, Architecture, Design This is primarily of interest in manually constructed ontologies. Assuming that some kind of design principles or criteria have been agreed upon prior to constructing the ontology, evaluation on this level means checking to what extent the resulting ontology matches those criteria. Must usually be done largely or even entirely manually by people such as ontological engineers and domain experts. 40
G: Multiple-Criteria Approaches Selecting a good ontology from a given set of ontologies. Techniques familiar from the area of decision support systems can be used to help us evaluate the ontologies and choose one of them. Are based on defining several decision criteria or attributes; for each criterion, the ontology is evaluated and given a numerical score. A weight is assigned to each criterion. An overall score for the ontology is then computed as a weighted sum of its per-criterion scores. 41
Example (G): Ontology Auditor Metrics Suite Metric Attributes Description Syntactic Quality Lawfulness Richness Correctness of syntax used Breadth of syntax used Semantic Quality Pragmatic Quality Social Quality Interpretability Consistency Clarity Comprehensibility Accuracy Relevance Authority History Meaningfulness of terms Consistency of meaning of terms Average number of word senses Amount of information Accuracy of information Relevance of information for a task Extent to which other ontologies rely on it Number of times ontology has been used 42
Example-2: Overall Quality Metric Overall quality (Q) is a weighted function of its constituents: Q = (c1 S) + (c2 E) + (c3 P) + (c4 O) where S = syntactic quality E = semantic quality P = pragmatic quality O = social quality, and c1+c2+c3+c4 = 1 The weights sum to unity, and currently, are set by the user, the application, or else assumed equal. 43
Example-3: Syntactic Quality (S) Measures the quality of the ontology according to the way it is written. Lawfulness refers to the frequency of syntactical errors. Richness refers to the proportion of features in the ontology language that have been used in an ontology Syntactic Quality (S) S = b 1 S L + b 2 S R Lawfulness (S L ) Richness (S R ) Let X be total syntactical rules. Let X b be total breached rules. Let N S be the number of statements in the ontology. Then S L = X b / N S. Let Y be the total syntactical features available in ontology language. Let Z be the total syntactical features used in this ontology. Then S R = Z/Y. 44
Example-4: Semantic Quality (E) Evaluates the meaning of terms in the ontology library. Interpretability refers to the meaning of terms in the ontology Consistency whether terms have consistent meaning Clarity whether the context of terms is clear Semantic Quality (E) Interpretability (E I ) Consistency (E C ) Clarity (E A ) E = b 1 E I + b 2 E C + b 3 E A Let C be the total number of terms used to classes and properties in ontology. Let W be the number of terms that have a sense listed in WordNet. Then E I = W/C. Let I = 0. Let C be the number of classes and properties in ontology. C i, if meaning in ontology is inconsistent, I+1. I = number of terms with inconsistent meaning. Ec = I/C. Let C i = name of class or property in ontology. C i, count A i, (the number of word senses for that term in WordNet). Then E A = A/C. 45
Example-5: Pragmatic Quality (P) Refers to ontology s usefulness for users or their agents, irrespective of syntax or semantics. Accuracy whether the claims an ontology makes are true. Comprehensiveness measure of the size of the ontology. Relevance whether ontology satisfies the agent s specific requirements. Pragmatic Quality (P) P = b 1 P O + b 2 P U + b 3 P R Comprehensiveness (P O ) Let C be the total number of classes and properties in ontology. Let V be the average value for C across entire library. Then P O = C/V. Accuracy (P U ) Relevance (P R ) Let N S be the number of statements in ontology. Let F be the number of false statements. P U = F/N S. Requires evaluation by domain expert and/or truth maintenance system. Let N S be the number of statements in the ontology. Let S be the type of syntax relevant to agent. Let R be the number of statements within N S that use S. P R = R / N S. 46
Example-6: Social Quality (O) Reflects that agents and ontologies exist in communities. Authority number of other ontologies that link to it History number of times the ontology is accessed Social Quality (O) Authority (O T ) History (O H ) O = b 1 O T + b 2 O H Let an ontology in the library be O A. Let the set of other ontologies in the library be L. Let the total number of links from ontologies in L to O A be K. Let the average value for K across ontology library be V. Then O T = K/V. Let the total number of accesses to an ontology be A. Let the average value for A across ontology library be H. Then O H = A/H. 47
Any Question... amini@sharif.edu 48