SHARED DATA: THE ACHILLES HEEL OF SERVICE- ORIENTED ARCHITECTURES

Transcription

1 SHARED DATA: THE ACHILLES HEEL OF SERVICE- ORIENTED ARCHITECTURES INTRODUCTION Service-oriented architectures (SOAs) are a significant advance in improving the flexibility of business logic. However, they don't address the challenge of ensuring the business data used by this logic has the expected meaning. In fact, they exacerbate the problem of using consistent definitions of business entities such as customer, product, and order. Because different services rely on these entities to execute their distributed business logic, fully leveraging the power of this paradigm requires an explicit strategy for reconciling alternative definitions. SOAs encourage the distribution of business logic. Different services may apply their respective logic to the same business entities and may even completely duplicate each other. The power of the paradigm lies in its ability to give service consumers a consistent mechanism for combining related business operations and choosing between alternative implementations. However, information technology (IT) departments are on their own when it comes to guaranteeing the quality, consistency, and usability of the data required by overlapping and substitutable services. While SOAs deliver interoperable business logic with high levels of flexibility and robustness, there is no similarly flexible and robust technology that delivers interoperable business data. This chasm between logic and data explains why up to 70 percent of integration project costs stem from manually implementing data manipulation code. For SOAs to achieve mainstream adoption enterprises must embrace an enhanced data architecture to support them one that does for the data what SOAs do for the logic. The Pantero Exchange Modeling Technology fills this void by linking business rules for data to the message structures that contain them. Using these rules, it is possible to generate metadata-driven services that provide flexible data manipulation throughout an enterprise's SOA. The Pantero Shared Data Services Suite implements this technology to deliver smooth deployment of reusable business entities that capture and reconcile data usage. By implementing data interoperability through reconciliation, this suite enables enterprises to maximize the return on investment in their SOAs and ultimately reduce integration project costs by 50 percent or more. SOAS IGNORE UNDERLYING DATA Web services provide a standards-based mechanism for client applications to access business logic on an as-needed basis. Development tools make it relatively simple to put new interfaces on top of existing data sources to deliver such services. However, these new services are forced to manipulate data at too fine a level of granularity. As a result, they require hand- 1

2 coded rules to aggregate pieces of data into relevant business entities, enforce constraints among data elements, and achieve semantic agreement across applications in different functional areas. This approach creates a brittle set of data services that scatter semantics across the entire servic e network. It also creates an architecture that is not amenable to change. Changing a single logical rule sets off a domino effect in which every application service that has implemented this rule must also be changed assuming each instance of the rule can even be found. Consider the apparently simple concept of "order status". This concept is probably vital the correct execution of many business processes within the enterprise, but what are the true semantics of the term? Is it the status of placing the order itself such as issuing a request for quote the quote, or the purchase order? What about the status of a credit check on the customer? Perhaps it s the manufacturing, shipping, delivery, return, or payment status. In practice, it's probably all of these things. Applications in different functional areas will have different interpretations of "status" and different databases will store different "status" values. Each database and application within a given functional area of responsibility such as order management, manufacturing, or shipping obviously has to use a locally consistent definition of "status". However, when an enterprise begins using an SOA to integrate processes across these areas, source systems must convert their local definition to the local definition of the target systems. Canonical messages in formats like XML ensure that the target system can extract the "status" data, but they do not ensure that the source and target systems share the same underlying definition. The problem increases with the number of target systems order management systems may send messages to both manufacturing systems and shipping systems, each with a different definition of "status". Moreover, the enterprise probably has a set of rules about the relationships among the different kinds of order status. No other type of status may even be available until "order status" is "accepted". "Shipping status" may be equal to "manufacturing status" until "manufacturing status" is "finished". "Delivery status" may be "not shipped" until "shipping status" is "shipped." What happens when the rule for "shipping status" changes so that it hides the "manufacturing status" behind a "manufacturing in progress" value until the manufacturing process completes? Addressing common cases of dispersed semantics such as "status" usually causes handcoding and one-off development to devour the lion s share of project budgets. Data manipulation code is not reusable from project to project and every policy change sets off a cascade of additional labor-intensive coding. There is a critical need for a new kind of middle -tier data architecture that is focused on the exchange model for data interoperability in SOA applications. Exchange models are often derivatives of existing canonical models used for interchange and integration, such as ACORD for insurance and MISMO for mortgages. These canonical models are rigorously specified data formats, often in terms of XML, that serve as a contract between sender and recipient. However, exchange models are enriched versions of canonical models. They link business rules to the canonical message structures. These rules can be defined at three different levels: (1) on the canonical message format itself, (2) on data sources that provide the underlying data used to generate messages, or (3) on the Web services generated for accessing the 2

3 messages. Together, these three levels of rules reconcile the usage of data throughout the SOA. So an exchange model would reconcile the usage of "status" by specifying that a canonical message sent from an order management system to a manufacturing system would use one representation of "status" while a canonical message sent from an order management system to a shipping system would use a different representation. With this approach, enterprises can put the mechanisms in place to accommodate differences in message formats and data semantics in the SOA without impacting the source systems. They can also create reusable rules and semantics that radically reduce the cost of future integration projects. Finally, because the exchange model approach effectively encapsulates all semantics and rules, the barrier to changing policies is much lower. This proposition is fundamentally different from enterprise information integration (EII). EII gives applications a common, high performance mechanism to get data from data sources while exchange models focus on ensuring that SOAs have semantically consistent data throughout business processes. EII increases data access performance and decreases the cost of adding new applications. Exchange models increase business process efficiency and decrease the cost of adapting to business change. ATTEMPTED SOLUTIONS Enterprises currently utilize a range of products, technologies, and strategies to coordinate data semantics and usage in distributed applications. However, to date, no approach has successfully overcome the key issues plaguing these projects: agreeing on the meaning of data, hand coding of policy enforcement logic, and the high cost of changing policies in every component that enforces them. The most common approaches include: EAI Packages: Many enterprises already rely on enterprise application integration (EAI) packages for certain types of integration. Unfortunately, these packages are only a first step in data interoperability. They focus on reliably moving data among enterprise applications and performing the minimal transformation necessary to ensure data access. They don't really attempt to reconcile data semantics across a widely distributed set of services. Canonical Messages: Many organizations have adopted canonical message formats as a solution to transferring data among applications and exposing business entities via services. The primary benefits of implementing canonical formats are ensuring that the structure and syntax of messages are always in accordance with the canonical format, so systems receiving these messages will not encounter errors because data arrives in formats they cannot understand. However, even when data is transformed into a single format, there is still no guarantee that the data is valid within the context of how the target system will use it. Accommodating the semantics of the message, the source systems, and the context in which this message will be consumed by applications still requires manual coding. Consider how much effort it would require to change the rules associated with "status" in every order management, manufacturing, and shipping application. 3

4 While none of these attempts fully addresses the shared data challenge, each does possess aspects that enterprises would like to see in a true solution. It should be robust like EAI packages, and facilitate agreement like canonical messages. THE REAL SOLUTION: THE PANTERO SDS SUITE While the solution should incorporate aspects of current practice, it requires a fundamentally different architecture: a centrally managed network of shared data services with fully distributed execution that seamlessly reconciles data usage. Pantero s Exchange Modeling Technology delivers this architecture by using a model-driven approach to capture and codify relationships, semantics, and usage as metadata. This architecture means that services within the SOA need only access the shared data services for data, rather than having to go to multiple individual data sources via tightly coupled integration channels. Enterprises use Pantero s Shared Data Services (SDS) Suite to extend existing canonical models so that they become exchange models capturing the semantics and usage of data as well as its syntax and structure. Moreover, it automates the deployment of exchange models as rich sets of shared data services that formalize and automate the interaction with data across an SOA ensuring data interoperability and data quality. The Pantero SDS Suite is built on three fundamental tenets: reconciliation, centralization, and change. 4

5 Reconciliation of Differences The most critical aspect of building and deploying reusable middle -tier data services is determining what data model to expose and specifying the semantics of that model. Because it is extremely difficult to achieve global agreement on data structure and semantics, Pantero s SDS Suite is built on the premise that enterprise-wide agreement is not realistic or even desirable. Forcing such agreement creates a situation where there are winners and losers in the enterprise, with the losers having to redefine their business processes to accommodate the official enterprise data model. Pantero avoids this pitfall and fosters win-win interoperability by focusing on the exchange model. This model captures the structure and semantics of each source and target system as well as the relationships among these systems. Users of the SDS Suite can attach semantics to each stage of a data models source, exchange and target. This staged modeling approach enables all systems within the SOA to cooperate through data exchanges while still maintaining the independent data views that best support their respective requirements. This flexibility means that IT departments can create middle-tier data architectures that facilitate cooperation but do not dictate implementation. So for the case of the "status" rule discussed earlier, a developer would create a set of rules on canonical messages sent from order management, manufacturing, and shipping systems. These conditions would fire based on the source system, the target system, and the canonical message type. In cases where a transformation is necessary, the fired condition would apply a mapping that describes how to translate among the various definitions of status. Because the conditions and mapping are described as metadata, the developer can change them dynamically at runtime and have the new policy applied to the very next message that meets one of the conditions. Distributed Execution with Centralized Control Currently, service-oriented applications interact with data at all levels of abstraction, primarily because the operations for retrieving, transforming, and updating data are hand coded into different services. This approach to interacting with data creates a brittle set of data services that are not reusable and that scatter semantics across the entire application stack. Pantero s Exchange Modeling Technology enables enterprises to define, build, and manage an exchange model that consolidates all important data interactions into a coherent network of shared data services. Because they operate as a network, shared data services can easily accommodate distribution of both the data and business logic. However, distribution is only desirable for runtime execution. To ensure consistency, enterprises have to centrally manage the design process and production updates. Central management confers two important benefits. First, developers can make changes in a single location and have them take effect throughout the 5

6 enterprise without exception. Second, combining centralized management with metadata-driven execution makes it possible to adapt the entire enterprise in real time. Together, these benefits simultaneously reduce development costs, improve data quality, and increase production flexibility. Affinity for Change The same hand coding that makes reuse difficult also creates a middle-tier data architecture that does not tolerate change well. Middle-tier data architecture is fundamentally different from back-end data architecture. Back-end data architecture is concerned with eliminating duplication, enabling concurrent access, and enforcing type and range constraints. Middle-tier data architecture is concerned with the semantics of data as it flows from one application to another within a business process. For example, back-end data architecture strives to make sure only one official ship date and one official delivery date exist, all the applications that need these dates can get them, and that the data records are formatted as date types. Middletier data architecture strives to guarantee semantic consistency such as the "delivery date" can't occur before the "ship date". 6

7 Traditionally, developers must implement such middle-tier data consistency rules in many different services. If that rule needs to be changed, how can they find every service that implements that policy? How can they change the rule consistently across all services at once? How will the rule change impact other services and systems in the enterprise? Pantero s modeldriven design approach supports the critical need for change in enterprise application architectures by enabling two critical change management concepts: (1) the incremental and dynamic deployment of rules and (2) the ability to assess the cost of changes. With Pantero s SDS Suite, developers no longer have to worry about finding and changing every application service whenever change is required. Moreover, its impact analysis capability enables them to fully understand the ramification of any changes before implementing them. The deployment of Pantero-based shared data services radically reduces the need for oneoff development, thus slashing the costs of application integration project costs. It also significantly reduces SOA maintenance costs through its affinity to change, support for impact analysis, and ability to improve data quality. PANTERO: FILLING THE VOID IN THE SOA Pantero provides a comprehensive solution for enterprises that need shared data services to realize the full benefits of their SOAs. Pantero s SDS Suite includes both design-time and runtime components. The design-time component includes model-driven capabilities enables developers to capture semantics, desig n the exchange model, generate code, and perform change analysis. The run-time component provides a centrally managed and distributed execution infrastructure for exposing source data as business entities, applying integrity and policy constraints, and logging all interactions with data. SDS Editor The SDS Editor is a design-time modeling environment that supports the creation of Pantero Exchange Models. These models capture the relationships among source systems, canonical messages, data service payloads, and data policies. The SDS Editor can compile an exchange model into a set of Pantero Shared Data Services by generating code for corresponding Web services. Because the resulting interfaces are based on Web services standards, an enterprise can deploy them anywhere within its SOA. IT staff can save, version, and share exchange models. They can even use them as a means of unambiguous communication between data architects and developers. The SDS Editor support all of the following modeling operations: Canonical Message Import: Many enterprises already have a canonical messaging format for transferring data among their constituent organizations. These message formats are often expressed in XML or as a model in a modeling tool. The SDS Editor can directly import into these models. Source System Import: To understand the usage of data within the enterprise, a Pantero Exchange Model must understand the schemas used by 7

8 all source systems. The SDS Editor can directly import these schemas. Then developers can graphically specify the relationships among canonical models and source schemas. Map Editor: After specifying the existing relationships among canonical models and source schemas, developers can enrich them to create exchange models. This enrichment includes specifications for how to aggregate data structures from canonical models and source schemas to form business entities and how to transform business entity definitions used by one part of the enterprise into those used by other parts. The SDS Editor renders all this vital information as a design-time and run-time metadata. Rules Editor: The rule editor allows the declarative definition of data integrity, aggregation, and business rules. Developers then associate these rules with the relevant parts of the exchange model. A rule can apply to the source systems, the canonical message, or the shared data service payload. Like the exchange models, the SDS Editor renders these rules as metadata. The result of this modeling process is a comprehensive description of all the existing data sources within the scope of the model and a definition of how data should flow among them. Because Pantero Exchange Models involve both the data definitions and the application components that consume them, it understands all the touch points where data requires aggregation and transformation. Based on the data captured during the modeling process, the Code Generator can create the shared data services that weave these touch points into a seamless data fabric. Code Generator Given an exchange model, the SDS Editor supports the creation of shared data services through full, incremental, and dynamic code generation via the Code Generator. By supporting all these modes, it enables enterprises to smoothly accommodate the level of change it requires. These modes are applicable under the following levels of change: Full Deployment for Substantial Change. When first deploying an exchange model or making a major revision, the enterprise should generate 8

9 shared data services from a clean slate. The Code Generator creates the Java code and XML metadata for every shared data service that the enterprise can deploy to a J2EE execution platform. Incremental Deployment for Gradual Change. Updates to accommodate new source systems, canonical messages, or major policies will obviously occur on occasion. The Code Generator creates only the Java code and metadata components that have changed since the previous version. Dynamic Deployment for Continuous Change. Many enterprises have certain processes that continuously evolve based on rapidly changing businesses conditions. The Code Generator accommodates this level of change by allowing any declarative rule to be deployed as metadata and automatically implemented by the Code Generator already in place. Enterprises can use the generation mode most appropriate for the particular change. However, it s critical to understand the impact of a change before making this decision. The SDS Editor supports these needs by providing impact-analysis reporting. Architects and project managers can determine the impact of making changes by using the running scenarios and inspecting their inventory of current shared data services. For example, a change to a data-integrity or business rule may change the behavior of a single data service or dozens of data services. With the SDS Editor, the impact of that single rule change can be quickly determined by running a report. That single rule change can also be dynamically deployed to a test environment and then to a production environment once it receives full validation. The SDS Editor runs as an extension of current Interactive Development Environments (IDEs) today, such as BEA's WebLogic Workshop environment. By extending these environments with the SDS Editor components, developers do not need to learn a new IDE or switch to a new environment to leverage the Pantero solutions. SDS Engine Once an enterprise uses the SDS Editor to generate a set of shared data services, it deploys them to a J2EE server. Because an SOA approach to building and deploying systems is both distributed and dynamic by nature, the Pantero run-time architecture explicitly supports distributed execution, dynamic change, and comprehensive auditing. The SDS Engine shared data services are metadata-driven, stateless, and J2EE compliant. Each service can be exposed as an EJB or a Web service. When using Bea's WebLogic Workshop, developers also have the option of deploying services as a Java Custom Control. Calling applications can invoke such services synchronously or asynchronously. Typically, an enterprise will have an asynchronous messaging bus or an enterprise services bus for managing the flow of business processing. Source systems publish messages to 9

10 topics to which target systems subscribe. Deploying shared data services in this environment is simply a matter of inserting them into the processing pipeline. For the "status" example discussed previously, the enterprise would simply change the message publishing of source syste ms so they no longer published directly to the "order management," "manufacturing," or "shipping" topics. Instead, they would publish to the "fulfillment validation" topic. The shared data service would pull messages from this topic, apply the "status" rule as well as any others that applied, and then publish the validated message to the "order management," "manufacturing," or "shipping" topic as specified in the message addressing. The SDS Metadata Repository supports all the shared data services in the exchange model. This repository holds all the metadata required to implement the operations of every shared data service. It supports dynamic change by allowing a new data integrity or business rule to be deployed as metadata and propagated to individual services so they will begin automatically enforcing it on the fly. All Pantero Shared Data Services support comprehensive auditing through sophisticated logging. The logging system records typical statistics such as how many times a service has been invoked, which rules were fired, how many times a rule was triggered, which violations were captured, and the severity levels of each rule violated. The SDS Editor can read these logs and generate reports useful for measuring improvements, impact analysis, and project planning. With these components in place, the Pantero SDS Suite provides IT departments with the capabilities required to create a middle -tier data architecture that is as flexible, modular and adaptive as the rest of the SOA. This flexibility greatly reduces development and maintenance costs and, more importantly, enables the SOA to deliver on its promise as the next generation of IT infrastructure. THE CHALLENGE OF CHANGE A fundamental aspect of data reconciliation stems from change. The "status" example was a case of different systems evolving different meanings for the same term. But what about the case where an enterprise or group of partners can start fresh with common definitions? Unfortunately, no matter how much such an enterprise or group might wish that it could codify interactions once and for all, change is inevitable. Evolving conditions necessitate adaptation. But even from the first day of codification, the pull of change is irresistible. Different organizations will see advantages in adding custom semantics that give them an advantage. So just as the different applications in the same enterprise need to reconcile their alternative views of "status", different members of the same group need to reconcile their alternative evolutions of standards. Consider the case of mortgage credit checks. Mortgages are a well-understood financial instrument. Credit checks are a well-established procedure. There's a set of standardized canonical message formats in XML called CreditRequest and CreditResponse maintained by the Mortgage Industry Standards Maintenance Organization (MISMO). There are only three active 10

11 credit check providers that implement them. Nevertheless, there is a significant amount of variation and extension of these formats, which is unfortunate, because credit checks are one of the primary drivers of the mortgage process. The CreditRequest and CreditResponse formats enumerate the types of credit loans an applicant may have held. However, there are a number of new types supported by some or all of the credit check providers, such as deferred student loans, construction loans, bi-monthly mortgages, and semi-monthly mortgages. There are also other types of credit loans where different providers encode them differently, such as second mortgages. Implementing each of them requires dynamic changes to the canonical formats in five different places. Other changes aren't so bad. One provider includes an account balance date for each loan and this addition only requires changes in three places. A different provider includes a balloon payment due date that also only requires changes in three places. Of course, one consumer of these services wants a summary of late payments made within a series of time intervals and implementing it requires changes in four different places. A few small customizations with only two message types required fifteen different changes. Making these changes would require some effort, but remain manageable if the changes were consistent across all message instances of the same type and only a few client services consumed the messages in question. Unfortunately, these changes are dependent upon the particular service provider. So the network of interacting services must operate on the data differently depending on the credit bureau. While only a few client services may initially interact with the credit bureau, they may pass this information through an arbitrarily long chain of downstream services. So every potentially affected service must accept the altered data. Because the Pantero SDS Suite accounts for the initial data source in its exchange model, it can dynamically adapt the structure of messages to account for source-specific differences. Moreover, because it also models all the data producers and consumers, it knows how upstream adaptations necessarily dictate downstream adaptations and implement the appropriate downstream processing. If this relatively contained and standardized mortgage credit check operation includes this many potential points of flexibility, one can only imagine the flexibility required to support an entire process that is both highly complex and highly customized. Under such circumstances, only the Pantero SDS Suite can deliver. CONCLUSION As demonstrated with the "status" and MISMO examples, even small differences in the usage of data within a relatively stable business process can affect a substantial number of problems in an SOA. IT organizations typically compensate for these differences by writing significant amounts of code to facilitate consistent usage and to enforce data interpretation rules, but this code is expensive to develop and resistant to change. 11

12 Pantero's SDS Suite solves this problem by reconciling the inconsistencies between diverse data sources and targets for agile and cost-effective business integration. By managing the message models as well as the rules that govern their usage and interpretation, it delivers global control of information exchange and facilitates full data interoperability. As a result, IT organizations maximize the return on investment in their SOAs by significantly reducing development and maintenance costs as well as ensuring better overall data quality across the enterprise. 12