Vocabulary Harvesting Using MatchIT. By Andrew W Krause, Chief Technology Officer

Transcription

1 July 31, 2006 Vocabulary Harvesting Using MatchIT By Andrew W Krause, Chief Technology Officer Abstract Enterprises and communities require common vocabularies that comprehensively and concisely label/encode, define, and relate information that must be shared or applied by people and computers. This paper demonstrates how Revelytix MatchIT software can be used to discover, extract, and disambiguate vocabulary terms from existent metadata found in relational databases and.xsds. MatchIT s powerful underlying technologies for tokenization, semantic ontologies, and graphical representation can also be applied to related schema matching and ontology development challenges. Organizations increasingly need to share and integrate their data and make software applications interoperable to stay competitive and exploit new opportunities. This is critical challenge facing development of web services, Service-Oriented Architectures (SOA), and netcentric computing and applications ranging from regulatory compliance to Homeland Security. A typical data sharing and interoperability example is diagrammed in Figure 1 below. Relational Databases File Systems Web Content ODBC/JDBC JDBC SOAP Information sharing Location_Type Location_ID XML Document <a> <b> </b> </a> Application views of Information Logical, target data model Interoperability Bldg_type Bldg_id Depot_Number SITENUM Facility_ID Authoritative sources Figure 1: Schematic example of sources and targets requiring mediation. Vocabulary Harvesting Using MatchIT by Andrew W Krause Page 1 of 5

2 In essence, we have to be able to recognize a customer or supplier in each data source we need to share. This is the job of Vocabulary Management which develops and maintains transforms of metadata syntax (format) and semantics (meaning) across disparate data sources and maps them to controlled terms that can be reused interoperably. Vocabulary Management encompasses many different possible approaches, frameworks, and tools, which are reviewed in the whitepaper, entitled Vocabulary Management: The Big Picture, also available from Revelytix. Vocabularies can be systematically created from the top down, discovered from the bottom up or imported from other domains and standardized sources. Typically, vocabularies are surveyed and extracted from the metadata associated with existing databases such as fieldnames, schemas, XML namespaces, business rules, or ontological entities. Done manually, this is an extremely laborious, costly, and time-consuming process. Fortunately, this process can now be automatically facilitated using Revelytix MatchIT, the first product to offer sophisticated pattern matching, tokenization, and semantic search algorithms that can rapidly and accurately extract candidate vocabulary terms from metadata and XSD namespaces. The terms can then be graphically displayed and navigated in an Eclipse GUI along with their source(s), potential schema-level matches across disparate databases, and definitions derived from MatchIT s internal English language knowledgebase. Other languages or knowledgebases can also be imported from simple acronym lexicons to complex ontologies. Moreover, the semantic distance (similarity) between different candidate terms can also be graphically displayed to help the user grasp and understand term-namespace relationships over all data sources. All the available definitions found in the MatchIT knowledge-base Results of the automated tokenization All the usage instances where each term was used in any of the sources A comprehensive list of terms automatically discovered across all sources Figure 2. Automated term discovery and interpretation Vocabulary Harvesting Using MatchIT by Andrew W Krause Page 2 of 5

3 MatchIT s vocabulary discovery from federated databases is illustrated in Figure 2. The index shows all the candidate vocabulary terms discovered and extracted from XML and database field names such as ArticleAmount or ArticleID. Each term s possible definition and sense is given along with usage information to trace source and type. Figure 3 shows how the example metadata field called ArticleAmount is tokenized and related to other terms that may be present in other databases. Type-of Assets Synonym Sum Amount ArticleAmount Article Automated term tokenization Creation Automated semantic linking using the default knowledge-base contained within MatchIT Figure 3: Tokenization and contextualization of metadata for vocabulary extraction and semantic exploration to assist in disambiguation. MatchIT not only facilitates vocabulary discovery but also term disambiguation by enabling sense and context matching. MatchIT s internal knowledgebase can show each sense of a term and can be augmented by user-defined terms and senses as well. Furthermore, the context can be viewed from instance-level data by several means including integration of MatchIT with Eclipse-based data-federation tools such as MetaMatrix Modeler EII (Enterprise Information Integration) product. For example, this term sense and context visualization could be essential to figuring out whether the discovered term program refers to a computer program, TV program, or government weapons program. MatchIT enables terms across vocabularies to be visualized, contextualized, and semantically mapped in order to facilitate vocabulary refinement and standardization as well as enable schema matching across disparate databases and models. This is illustrated in Figures 4 and 5. MatchIT s ability to contextualize and refine term definition can also be used to bootstrap ontology development to enable automated inferencing and machine reasoning. Vocabulary Harvesting Using MatchIT by Andrew W Krause Page 3 of 5

4 Abstraction Evidence Figure 4: Semantic matching and translation mediated by internal MatchIT ontology. Target element for matching Automatically calculated semantic distance between terms Helps facilitate rapid decision making Source candidate for matching Figure 5: Facilitating decision making for schema matching and vocabulary disambiguation. Vocabulary Harvesting Using MatchIT by Andrew W Krause Page 4 of 5

5 Figure 4 also illustrates several practical aspects of using MatchIT for vocabulary discovery and profiling. First, not all matches are necessarily relevant. Rather the purpose is to facilitate human definition and selection through the ability to interpret related context. MatchIT s built-in knowledgebase assists in finding more distant potential matches that are actually more relevant. Figure 5 shows how MatchIT can also be used for schema matching and ontological mediation, which can be conducted independently or in parallel with vocabulary discovery and disambiguation. MatchIT is easy to use and enables much more rapid development of controlled vocabularies at dramatically lower cost. Ambiguous or highly interrelated information types with seemingly vague boundaries can be better visualized for taxonomic classification as the vocabulary develops. Term nomenclature conventions can be developed or adapted during this process as the scope and specificity requirements come into focus. Complex large-scale projects that were previously intractable can now be conquered and carried to productive application. Architecture and Interface MatchIT runs on MS Windows 2000 or Windows XP as a stand-alone or plugin Eclipse 3.1 application that can flexibly work with any ODBC-compliant data source and application. It can be integrated with popular third party data integration, manipulation, and management tools and applications. MatchIT is organized into projects, files, and objects and employs wizards to guide, assist, and simplify project creation, source data/models selection, vocabulary discovery, importing lexicons or ontologies, making match recommendations, and exporting results. Match results can be mapped to XSDs, RDF and OWL models for schema display or export. MatchIT is also compliant with OSGI and JDBC standards and can import schema metadata from relational database sources using the database vendor's JDBC driver. For more information on this or related topics please contact info@revelytix.com. Vocabulary Harvesting Using MatchIT by Andrew W Krause Page 5 of 5