Information Intelligence: Metadata for Information Discovery, Access, and Integration

Transcription

1 Information Intelligence: Metadata for Information Discovery, Access, and Integration Randall Hauch, Alex Miller, Rob Cardwell MetaMatrix St. Louis, Missouri {rhauch, amiller, Abstract Integrating enterprise information requires an accurate, precise and complete understanding of the disparate data sources, the needs of the information consumers, and how these map to the semantic business concepts of the enterprise. We describe how MetaMatrix captures and manages this metadata through the use of the OMG s MOF architecture and multiple domain-specific modeling languages, and how this semantic and syntactic metadata is then used for a variety of purposes, including accessing data in real-time from the underlying enterprise systems, integrating it, and returning it as information expected by consumers. 1. Turning Data Into Information Information access and integration is the process of locating and extracting data in real-time from multiple disparate sources, relating it, and producing a unified representation matching the syntactic and semantic expectation of the recipient. This process is achieved by leveraging metadata that describes the syntax and semantics of each data source and how the data is related. This definition contains four critical facets: Locating and extracting data in real-time from multiple disparate sources For decades we have been deconstructing the information that we use to communicate and interact with each other and persisting it in many different forms. Most of those systems were built to support specific applications, and thus intentionally deal with a subset of the enterprise s data. Business needs require that the data in these disparate silos be accessible beyond the systems that were originally designed to use it, and that this data be available for reuse by the newer enterprise applications and processes. An important aspect of this is that many of these disparate data silos change continuously, making real-time access to that data a requirement. Using and leveraging metadata describing the syntax and semantics of each data source Metadata attempts to define and capture the syntax and semantics of how information is represented as data. In other words, metadata helps us understand the technical specifics of how data is represented or Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. SIGMOD 2005 June 14-16, 2005, Baltimore, Maryland, USA Copyright 2005 ACM /05/06 $5.00. stored, how it can be accessed, and perhaps most importantly, what the data means and how it relates to concepts that people (within a certain context) understand. A primary challenge of information integration is capturing metadata that is accurate, complete and precise, while also making the maintenance of that metadata straightforward and simple for users. Relate, join and synthesize the data Extracting data from multiple sources can be challenging, given the variety of information storage systems, interfaces, and languages used to interact with those systems. Further combining and synthesizing data from these systems requires the use of algorithms that are resilient in the face of varying data formats, poor data quality, and semantic value mismatches. Finally, users expect to work with large volumes of data with performance comparable to a single source, regardless of the difficulties of the integration process. Produce a unified representation matching the syntactic and semantic expectation of the recipient This is perhaps the most challenging part of information integration, and it is the part that distinguishes information integration from federated query or mediation. Each information consumer has their own unique context, requirements, and information needs. The more closely the data can be provided in the form expected by the recipient, the less work they must perform to extract value. The key is abstracting the recipient from how and where the data is obtained and transformed into information deemed useful by that recipient. Abstraction reduces the time and effort required to connect multiple systems to the information integration service, thereby increasing its cost effectiveness. Why is information integration even necessary? An enterprise that is able to break down the barriers surrounding the disparate silos can extract maximum benefit from its enterprise data, providing unified views of business-critical data (such as customers, products, orders, etc.) and leveraging existing information in new ways. In addition, breaking down the barriers allows the business to detect redundancy and appropriately manage their operational costs. Removing redundancy also allows a business to more quickly respond to change as one system must be changed rather than many. Information integration can be achieved a number of ways. Data warehouses and data marts perform the steps a priori to construct a centralized representation that is closer to the recipient, allowing the recipient to access and process the copied and cleansed data to find the information they are seeking. One substantial benefit of this approach is typically fast access, since the information is all collocated and preprocessed. Another advantage is that transactional systems are isolated from decision support systems, 793

2 allowing them to be managed independently. However, businesses today need ever-fresher data, increasing the cost and complexity of maintaining a warehouse. Additionally, warehouses are often structured and optimized for specific types of processing, not necessarily for providing the information in the form expected by one or more recipients with different needs and requirements [1]. Enterprise Information Integration (EII) performs the integration in real time on an as-needed basis. Performing integration in real time means that the EII system must often provide a complete bidirectional abstraction, not just a stylized layer for retrieval in a certain context. This capability includes transactional inserts, updates, and deletes. 2. Historical Perspective Large and medium size enterprises have data stored in different systems using a variety of technologies. One major reason for this is that new systems are often developed in conjunction with their own new data stores, yet systems can remain in production for decades. Additionally, changes in corporate structure lead to new data stores for an enterprise, often with similar purposes as existing systems. Finally, new technologies such as the Internet, , and wireless provide benefits to users that drive new kinds of stores. The result is that the typical enterprise has many sources of data that are disjoint, dissimilar, overlapping, and even duplicate. Integrating data that exists in these different types of systems requires understanding what data is managed by each system, and how the data relates to the semantic terms and concepts expressed by the information consumer. 2.1 Modeling Technology In the 60s, 70s, and 80s, the technology to efficiently persist, manage, access and update large amounts of data matured. This included hierarchical, relational, and object-oriented database management systems. As these systems could be configured to support different data schemas, tools were needed to help data architects design data structures. But describing the different data schemas required different concepts, different terminology, and different notations to best match the underlying systems that would host the data. Mylopoulos [2] provides a brief history of several of the major modeling approaches, including: Physical models used the same data representations and structures used by the software applications (b-trees, arrays, lists, variables, etc.), and consequently required the programmer/modeler to deal expressly with the conflicting concerns of computational efficiency and application and information simplicity. Logical models offered abstract mathematical constructs (sets, arrays, relations) to hide implementation details, but the constructs were limited and flat, and still required mapping the semantic concepts that people deal with into the mathematical constructs. Conceptual models used more expressive techniques to capture the same semantic terminology (customer, account, employee, etc.) used by people in the enterprise to model the application, and were able to organize the information using abstraction mechanisms (generalization, aggregation, classification, normalization, parameterization, etc.). 2.2 Data Management and Integration There are many good surveys and histories of the different technologies involved in data management, and consequently we only wish to provide a brief synopsis. Data management systems (mainframes and hierarchical systems) initially enabled large enterprises to persist storage for large enterprises. With the advent of relational database management systems that could be run on smaller and cheaper hardware, many small- and medium-sized companies were able to take advantage of this technology. Until the late 90s, attempts to integrate decoupled and disjoint systems ( silos of data ) often followed one of two approaches. Data warehousing extracts the data from its master locations, transforms it into a common schema, and loads the newly transformed data into a single, centralized store used for highvolume read-only decision support activities. As such, data warehouse schemas are optimized for such queries, often at significant expense to updates. Data marts are similar to data warehouses, but are often smaller scale and involve data for only parts of an enterprise. The other common approach to integrate disparate silos was Enterprise Application Integration (EAI), which attempted to integrate the software applications that sit on top of the data stores. Early integration architectures used point-to-point communication that was extremely complicated with even moderate numbers of systems. Subsequently, hub and spoke integration architectures required the creation of a common representation that was mutually agreed upon, but arriving at a common representation that accurately and completely satisfied the structure and semantic requirements of all systems involved proved costly and difficult. Perhaps more fundamentally, integrating the software applications lost sight of the fact that the enterprise data is what needs to be exchanged between the systems, and involving the software applications only complicated the integration efforts. 3. Disparate Data and Disparate Metadata Perhaps the most fundamental reason why data sources have proliferated is because people have different information needs brought about by the different roles they play. One solution does not fit all, and probably never will. Thus, people have continually looked to new approaches and new technologies that can help them solve their specific information needs in ways that are more efficient, effective, and effortless. So why, then, do people expect their metadata to be expressed and managed in a single, common and ubiquitous language? In fact, metadata is merely another form of data, and thus people bring to metadata management the same types of bias, context, and expectation as they do to data management. Consider the many modeling systems that use a single modeling language (ER, UML, XSD, Object-Role modeling, etc.). While such tools are often very good at modeling a specific type of system, users become familiar with the tool and techniques and eventually attempt to use it to model other types of systems. Unfortunately, if the system has different constructs and concepts, modeling that system will require the person to adapt the modeling language in ways that often not easily repeatable or sharable. The resulting models form isolated silos of metadata that are not only less accurate, precise and complete, but also less usable and subject to disagreement. 794

3 Figure 1. Systems have their own semantic concepts that are best modeled and represented with domain-specific languages. Therefore, if the lessons of data management and integration can be applied to metadata representation and management, then metadata, like data, will continue to be disparate and disjoint. Consequently, effective definition and management of metadata requires the ability to describe the metadata using the semantic constructs and terminology most appropriate for the system being modeled. In other words, if different types of systems are to be modeled effectively, accurately and precisely, then practical metadata management must have a foundation with support for different and domain-specific modeling languages, or metamodels (See Figure 1). Some systems import metadata expressed in any metamodel and transform it into a single, generic representation or metamodel [3,4]. This is often the case for systems that, following import, automatically use and process this unified and homogenized metadata the homogeneous metadata representation makes it Figure 2. The MetaBase Modeler, an integrated model development environment. significantly easier to process. However, one major disadvantage of this approach when used for metadata management is that once the metadata is imported, its representation is no longer in the domain-specific language that captures the specific structure and semantics of the system, and so it becomes difficult for users to view, interpret and maintain the metadata. 4. Model Development and Integration The MetaMatrix MetaBase product provides an integrated environment for modeling different types of data and information systems. MetaBase includes a modeling workbench, called the MetaBase Modeler (see Figure 2), where models of different systems can be viewed, edited, validated and related. MetaBase also includes a repository that provides standard configuration management capabilities, and a virtual catalog that can integrate disparate metadata repositories to enable searching, reporting, and discovery of metadata across the enterprise (see Figure 3). The Modeler is an integrated model development environment where models of different types of systems can be created, viewed, manipulated and managed in common ways. For example, the Modeler provides integrated model validation, searching across all models, support for a number of diagram types, undo/redo, events, and various wizards to import, update or export metadata. The Modeler also provides integrated support for refactoring to change models (delete, rename, move, etc.) and automatically view and accept changes to dependent models. Also, the Modeler provides a common mechanism for managing descriptions. However, the Modeler does not support these features with one or even a couple of modeling languages. The Modeler supports multiple domain-specific languages (or metamodels), including relational, XML documents, XML Schema Documents (XSD) 1,

4 Figure 3. MetaBase: modeling all aspects of information, and using metadata for real-time integration. web services, UML2 (object-oriented) 2, and relationships. The Modeler uses the OMG s Meta Object Facility (MOF) 3 architecture, which defines how multiple modeling languages can be defined so that they are interoperable (see Figure 4). The Modeler then adds on top of the MOF architecture the ability to drive the runtime behavior of the modeling environment with these different modeling languages. As such, new domain-specific modeling languages can be added at any time, and the runtime behavior adapts to include the new languages throughout the environment. Additionally, the metamodels can be extended to add custom properties on each of the language s constructs. Because the Modeler natively supports many domain-specific languages in a single user interface workbench environment, users can find, view, relate and transform metadata that exists in separate models and that is expressed in different modeling languages. For example, a relational model of a database can be related to the UML conceptual/logical model, and to the XML schemas used to represent the data in an XML format. In other cases, a relational model may be generated, for example, from a UML or ER conceptual representation that was imported from IBM Rational Rose, Popkin System Architect, or ERwin. These types of general relationships are useful to track dependencies, usage, realization, refinements, etc., and in fact the types of relationships can be extended. This ability to capture mappings between models that form different contexts of the enterprise information provides substantial flexibility and power to define how information can be transformed from one form to another. These transformation definitions are detailed mapping relationships that address, among other things, data type mismatches, schema and structural mismatches (e.g., combining data from two columns into one), joins between sources, unions of multiple sources, procedural logic, and bi-directional transformations (querying and updates). The most common approach is to create different layers of metadata, each with different purposes. Physical layer The lowest layer directly and accurately models the data sources themselves. Because these models directly correlate with the data source, the metadata can be used to identify and understand the data source itself. Business concept layer The next layer sits above the physical layer, and represents the business constructs and terminology. To each of these business concepts are added the transformation detail that defines how the data in the physical layer is related to the business concept. Because these business constructs represent the semantic terminology of the enterprise and because they are mapped with executable transformations to the underlying enterprise data, these are essentially reusable data components. If desirable, the transformations can be defined as supporting updates, meaning that inserts, updates and deletes may be pushed to the data sources through the business layers. Consumer layer This layer builds upon the reusable semantic constructs of the business layer and defines views that are specific to different categories of information consumer. Thus, this layer maps the semantic enterprise concepts back into a particular literal representation needed by the consumer and may take the form of relational tables, relational procedures, XML documents, or web services. In fact, the relational-to- XML mapping technology is able to transform any relational structure into any XML document structure (see Figure 1). Again, the transformations can be defined as supporting updates. 2 and and 796

5 Figure 4. OMG s MOF Architecture and support for multiple metamodels The Modeler is built on top of the Eclipse platform 4, which is an open-source tool-integration framework used by numerous commercial and open-source projects. This plug-in based architecture encourages decoupling various software components, provides standard and reusable components for common functionality (file management, repository integration, editors, view management, software updates, etc.), and provides a mechanism by which new functionality and new components (new importers, exporters, wizards, metamodels) can be added to the platform. 5. Using the Metadata The MetaBase metadata management system can be used to model many different types of systems and multiple views of those systems within a single environment with multiple modeling languages. These different and disparate models can be validated, related, shared, controlled, reused, automatically transformed into alternative metadata representations, and assembled into consistent and complete packages of metadata. These metadata packages are easily managed and deployable, and can be leveraged by a variety of tools for a number of purposes and activities. 5.1 Information Discovery This metadata, once captured, can be used to search, discover and understand what data is available and how the enterprise data is related. Being able to compare dissimilar data structures provides the ability to identify similarities and to provide harmonized views or representations. Additionally, the metadata can be used to help rationalize the existing data assets to find redundancies and eliminate duplicate silos of data. 5.2 Metadata Access and Integration The packages of complete and consistent metadata can be leveraged by applications that drive their behavior with metadata. 4 These applications come in many forms. For example, data access portals may use the metadata to completely define the behavior and presentation of the portal. Metadata-aware applications consume and integrate metadata and data from a variety of sources, incorporating and relating the metadata from enterprise sources with the information available on the enterprise intranet or the Internet. A single and extensible framework for modeling and managing the metadata for multiple types of systems makes it possible for each of these applications to use custom views and representations of the integrated metadata. 5.3 Information Access and Integration The MetaMatrix Server is an advanced integration engine that provides unified access to integrated information by consuming metadata about data sources, consumer-specific views of information, and the relationships between them. A metadata package is simply deployed to a MetaMatrix Server to create a virtual database (VDB), which appears to client applications as a typical database or web service. Multiple virtual databases can be deployed simultaneously, making it possible to support information consumers with different expectations and views of the same underlying enterprise data. Clients connect to and communicate with the virtual databases through standard mechanisms like Java Database Connectivity (JDBC) 5, Open Database Connectivity (ODBC), Simple Object Access Protocol (SOAP) 6 over HTTP 7, or SOAP over Java Message Service (JMS) 8, and interact with the VDB through SQL, XQuery 9 or web service operations. As such, client applications are abstracted away from the underlying data sources and how to communicate and interact with them

6 Driving the information integration with metadata has a number of advantages: Optimization is left to the engine The models declare what data exists and how the business concept and consumer layers relate to the data. The MetaMatrix integration engine is responsible for determining the actual process to execute the integration activity. Optimizations may include the use of a variety of algorithms, the removal of parts of the abstraction inconsequential to the result, and the enlistment of underlying data sources to perform the integration work in the optimal location for minimal data flow. Caching policies are declarative Different consumers have different expectations about performance and the liveliness of data. MetaMatrix can provide caching policies that vary between users to satisfy their requirements. Security Access to information can be controlled through flexible security policies that, again, are independent of the metadata. MetaMatrix can leverage and integrate with existing security infrastructure (single sign-on, LDAP directories, and data source specific mechanisms). Safety and control MetaMatrix can be configured to constrain and limit the loads on the underlying sources. This may include caching, but also may involve defining limits on the connectors to the underlying data sources, such as allowed access paths. Traceability MetaMatrix provides several ways to track usage of information by user and by data source. This information can be used to understand and view access histories and to enable financial policies such as billing. 6. Case Study A large financial services company, like many organizations, has numerous and often-duplicate data sources tightly bound to specific applications. Without metadata management, the organization was challenged to understand the data it had or where it resided. In particular, the organization wanted to improve the availability of vital reference data, including somewhat static information describing assets and account entities used in trade processing transactions, business intelligence, risk management, and reporting. Inconsistent and incomplete reference data can cause trade delays and trade failures, and problems caused by faulty reference data result in increased operational risk, lost revenue opportunities, and expensive manual trade duplication and reconciliation processes. Having many legacy silos of reference data complicates the process of data retrieval, normalization and aggregation. The organization used MetaMatrix technology to move from data silos and batch processing to centralized information access in real-time. The organization modeled the existing silos of reference data and created virtual models with specific views of the reference data needed by transaction processing, reporting, and business intelligence applications. These models were then packaged as virtual databases and deployed to the MetaMatrix Server integration engine, resulting in the ability to eliminate inefficient batch processing jobs and giving desktop trading applications ondemand access to integrated information. Processing that previously occurred in a nightly batch operation now occurs in MetaMatrix on both an ad-hoc and a batch request basis. Frequently changing information is available in real-time while static data is updated only as needed, leveraging existing data stores and data warehouses. 7. Conclusion Integrating disparate enterprise data sources is one of the most pressing problems facing enterprises today. These different sources of information contain different structures and contexts of data that have their own semantics, but the different types of sources and different views of information needed by different consumers require an adaptive and flexible approach to modeling information. MetaMatrix employs a modeling environment that supports multiple domain-specific languages that are appropriate for different types of systems, and that effectively model the available information, the enterprise business concepts, and the different views needed by consumers. This metadata is then deployed to the MetaMatrix Server and used to model-drive the on-demand access and integration of information. 8. Acknowledgements The authors would like to thank Bob Scanlon, Shawn Curtiss, and Michael Lang for their helpful comments. 9. References [1] Hellerstein, J., Stonebraker, M., and Caccia, R., Independent, Open Enterprise Data Integration. IEEE Data Engineering Bulleting, 22(1): 43-49(1999) [2] Mylopoulos, John, Information Modeling in the Time of the Revolution. ACM Information Systems, 23, 3-4 (May 1998), [3] Bernstein, P., Melnik, S., Petropoulos, M., Quix, C., Industrial-Strength Schema Matching, ACM SIGMOD Record, 33, 4, (Dec 2004), [4] McComb, D., Semantics in Business Systems, Morgan Kaufman Publishers,