A New SOA Data-Provenance Framework

Transcription

1 A New SOA Data-Provenance Framework Wei-Tek Tsai, Xiao Wei, Dawei Zhang, Ray Paul*, Yinong Chen, Jen-Yao Chung! Department of Computer Science and Engineering Arizona State University, Tempe, AZ * Department of Defense, Washington, DC! IBM T. J. Watson Research Center, USA Abstract Due to the dynamic nature, such as runtime composition and evaluation, it is critical for an SOA system to consider its data provenance, which concerns security, reliability, and integrity of data as it is routed in the system. In a traditional software system, one can focus on the software itself to determine the security, reliability, and integrity; however, in an SOA system, one also needs to consider the origins and routes of data and their impact, i.e., data provenance. This paper first analyzes the unique natures and characteristics of data provenance in an SOA system. Then it proposes a new framework to classify data provenance and collect data in SOA systems. Finally, this paper uses an example to illustrate these concepts and techniques Introduction Service-Oriented Architecture (SOA) and the corresponding Service-Oriented Computing (SOC) [20] have received significant attention recently as major computer and software companies, such as IBM, Intel, Microsoft, HP, SAP, and Sun Microsystems, as well as government agencies, such as U.S. Department of Defense (DoD) [15], have embraced SOA/SOC. DoD has adopted SOA for their C2 applications, including NCES, JBMC2, FORCEnet, JBI, FCS and GIG-ES. SOA has the following unique features: Standard-based interoperability Dynamic composition via discovery Dynamic governance and execution 1 This work is partially supported by US Department of Defense and by Department of Education, Award No. P116B Data provenance (also referred as pedigree or lineage) techniques emerge in SOA-based systems to address the data reliability, security, and integrity problems. The importance of data provenance in SOA can be seen from the Net-Centric Checklist [7] developed by OASD NII DCIO, such as "Can all potential consumers of all of the data available from your program determine the data pedigree (i.e., derivation and quality), security level, and access control level of your data?" Data provenance is not a new problem as it has been studied before in other fields such as business, medicine, science, and technology. As reported in [19], data provenance is recorded in different forms, such as scientific publications, GIS standards (spatial data transfer standard), citations, and annotations. Miles and his colleagues [14] presented a variety of use cases on recording and using provenance data of scientific experiments and analyzed those use cases to determine the technical requirements of a generic and application-independent architecture. Different researchers have different definitions of data provenance depending on their application domains. For example, Buneman [4] defined data provenance in the context of database systems as the description of the origins of data and the process by which it arrived at the database. Lanter [13] defined it in the context of GIS as information that describes materials and transformations applied to derive the data. Greenwood extended Lanter s definition to metadata recording the process of experiment workflows, annotations, and notes about experiments. Clarke and Clark [6] defined data provenance as Lineage of a data product encompasses data acquisition and compilation methods, conversions, transformations, and analyses, along with the assumptions and criteria applied at any stage of the

2 data product life cycle. Bose and Frew [3] did a comprehensive survey on traditional data provenance in scientific fields. Two important parts in data provenance are ancestral data products and process of transformation of these ancestral data products. Provenance information enables researchers to trace the data. Applications of provenance information are summarized by Goble [9] as data quality, audit trail, replication recipes, attribution, and informational. Groth and others [10] stated the requirements for a provenance system as verifiability, accountability, reproducibility, preservation, scalability, generality, and customizability. Rajbhandari and Walker [17] stated two additional requirements for a SOA system: A provenance system should be able to collect and archive the provenance of the transformation of data during the data processing by web services. The provenance data should be accessible and viewable by web browsers and query interfaces. A Microsoft study partitions SOA provenance data into four categories: reference data, activity data, resource data and service Interaction data [12]. Provenance in SOA is often classified into two categories in terms of granularity: Fine-grain provenance: This refers to tracing the data movement in the system, including where the data come from and go to, the rational for the data, and the time of data creation, manipulation, and termination. Coarse-grain provenance: This refers to data generated through processing a work flow. 2. Issues of SOA Data Provenance In SOA, data are passed around in an SOA system, processed by multiple workflows and services. Data may have a long history before they arrive at their final destinations. A sample SOA data lifecycle including the following activities: 1) Creation: a data is created by a sensor in observing an external event, created by a service or workflow, or by an input device. 2) Routing: the data is routed within the SOA system by SOAP protocol and other protocols such as those for discovery, composition, governance, and collaboration. 3) Processing: the data is processed by workflows and services as it is routed. 4) Storage: the data may be saved at multiple places including the source node, intermediate node, and the destination node. 5) Arrival at the final destination: the data will be presented to the user after processing. The data may be stored in a database, and go through the entire process again. A model-driven SOA development process has been proposed to address the system engineering issues [22][25], as shown in Figure 1. But this Service- Oriented System Engineering (SOSE) process has not addressed data provenance yet. Actually system properties critically depend on data provenance. If the data is not good, even if all the routing, participating services and workflows, interoperability protocols, such as SOAP, related database managements, work fine, the system still generates incorrect outputs. The following data provenance issues need to be addressed: Figure 1. The Development & Execution Environment Data Security: DoD ( iaso/lesson01.htm) defined data security as the protection of data from unauthorized (accidental or intentional) modification, destruction, or disclosure. Kernochan [11] listed four impacts on security in SOA: 1) disguising data; 2) erasing data; 3) placing (old) data in secure facility; and 4) controlling data access. Tan and others [21] discussed security issues in the SOA-based provenance system, and the issues include access control of provenance information, provide trustworthy access of provenance information with accountability with proper authorization. Data Reliability: According to the United States General Accounting Office (GAO) [8], data reliability refers to the accuracy and completeness of computer processed data, given the intended purposes for use. Data source, routing, processing, and storage in the SOA system can be a source for reliability concern. Incorrect, inaccurate, outdated, and imprecise data can cause the system to be unreliable. Data Integrity: Integrity is also a security concern because it also requires the assurance that data can only be accessed and altered by those authorized to do so. In this sense, secure data means confidential and integral data. Two well-known integrity models are

3 Biba Integrity model and Clark-Wilson Integrity model [2]. In an SOA system, each service can have an integrity level, and data produced by a high-integrity service can go into a low-integrity service, but not the other way around. This is opposite to the security management, where low-level data can go to a highlevel process, but high-level data cannot go to a lowlevel process. The services and their composition structure determine the performance of SOA-based application. Thus, the data integrity of the SOA-based application depends on that of all constituent services and the composition structure. Dynamic Composition: In an SOA system, a new application can be composed by specifying a workflow with reusable services, and services can be selected at runtime. As data are routed in an SOA system, they will be executed by various services selected at runtime. Thus, it may not be possible to know which service will be selected for execution of the data a priori, and thus the quality of processing by these services may not be known before execution. Dynamic composition affects almost all the issues in data provenance, including data security, reliability and integrity. Data Volume: An enterprise SOA system may process a large number of data at the data source, at the intermediate nodes, and at the data destination. Furthermore, some of these data can be multimedia presentation data which have large storage requirements. The large volume of data puts the SOA system under stress and affects data reliability and integrity significantly. Long-Life Data: Data in an SOA system can have a long life as they may be processed and stored in the system for an extended period of time. This may cause several issues. First, each new processing and storage of data will increase the probability of data corruption. Second, the longer the history requirements are, the more storage will be needed. Furthermore, certain data provenance algorithms require a reliable and complete history to be effective. However, this may not be feasible. Third, according to Gambler s Ruin theory in stochastic processes [18], a long-life data will have a probability of one of being corrupted eventually. Note that assuming each processing by a service or a workflow has a probability of p for being corrupted or compromised, and thus 1-p of being intact. Then after n processing, the probability of being intact is only (1-p) n, and if n is large, the probability of being intact goes to 0. Furthermore the more processing it has, the higher probability of being corrupted. Note that this is true even if p is very small. For example, if p is 0.001, but if n is 10,000, the probability of being intact is practically 0. Thus, unless data are eventually ascertained, validated, or corrected during the process, long-life data will be corrupted (i.e., security compromised, low reliability, and low integrity) eventually according to the Gambler s Ruin theory. The reliability of long-life data is a significant factor affecting the reliability of the service where those data reside on. The reliability of the service can be represented in the following formula: RelService = RelData RelOthers m = ( RelDatai) Rel i= 1 m i = ( (1 pi) ) Rel i= 1 n Others Others Where RelService represents the reliability of a service, RelData represents the reliability of data on this service, RelOthers represents other factors that affect the reliability of the service. Suppose there are m data on this service, RelDatai denotes the reliability of the i th data. p i is the probability of the i th data to be corrupted or compromised, and n i is the number of processing on the i th data. From the above formula one can see that, assuming other factors remain the same, long-life data have larger effect than short-life data because short-life data has smaller n i n and therefore its reliability (1 pi) i is closer to 1 when p i is very small. On the contrary, the n reliability of long-life data (1 pi) i is closer to 0 because they tend to have large n i. Real-time Data: Some data may be needed in a real-time basis, e.g., during discovery and matching, during SOA system reconfiguration [23], and during real-time command and control. It may be difficult to retrieve the relevant data in a timely manner to fit the real-time requirements unless the SOA infrastructure including the data provenance system is designed to meet the real-time requirements. The real-time requirements affect all of three data properties. Data Characteristics: Data may have specific properties that make them difficult to management. For example, multilevel secure data need a multilevel SOA infrastructure. Furthermore, secure data must be stored in a different location than unclassified data. Care must be taken to eliminate or minimize the number of potential covert channels. Message Routing: Data are routed in an SOA system for processing, and data can be routed by a communication backbone such as ESB (Enterprise Service Bus) [5]. If the communication backbone is faulty, it may corrupt the data.

4 Dynamic Data Classification: Not all data will be tracked in an SOA system and thus only important data will be tracked. This implies that there is a need to classify data according to their criticality with respect to provenance. As an SOA system is dynamically changing, data may need to be re-classified to meet the changing requirements. An unimportant data may become important due to an unexpected event. For example, tracking terrorists in an airline passenger list may not be important before, but it became important after the 911 event. Table 1 compares the differences between traditional data provenance and SOA data provenance. Dynamic Composition Reliability Security Integrity Table 1. Traditional Data Provenance vs. SOA Data Provenance Traditional Data Provenance SOA Data Provenance Not applicable Services may be selected during dynamic composition, and the workflow may be dynamically updated Data reliability needs reliable data In addition to data reliability in traditional provenance, creation, recording, and storage. If SOA data provenance also needs to deal with software is involved, also need to workflow and service reliability during dynamic deal with software reliability. composition and data routing in an SOA environment. Traditional data security mechanisms, including security kernel, access control list, covert channels, encryption, intrusion detection, digital signature, and authorization Both software and data have integrity considerations. Must deal with the same issue in the context of dynamic composition (service discovery), dynamic reconfiguration, and data routing Both data and service have integrity consideration, and need to address this issue in the context of dynamic composition and reconfiguration Data Volume Can be huge Can be huge Data Life Can be long Can be long Real-Time Data Yes Yes Data Routing In communication networks Also in SOA communication backbone such as ESB Data Classification Mostly static 3. A New Data Provenance Framework This section proposes an integrated SOA data provenance framework, as illustrated in Figure 2. Different from traditional data provenance, the proposed framework has the following new features: Need to be dynamic in case of dynamic composition. Figure 2. SOA Data Provenance Framework Embedded in an SOA System: The data provenance mechanisms including data classification, collection, and analysis are embedded in an SOA system using the services provided by the SOA infrastructure for implementation. From Figure 2 we can see that the proposed data provenance framework collects and analyzes data from all of the three SOA layers data layer, IS layer and SOA layer. Agile Data Classification: Not all data need to be tracked, and even for those data that need to be tracked, only selected aspects need to be tracked. The proposed framework provides a mechanism to classify data into different provenance data categories according to the type of the SOA system and to select data for tracking. Dynamic Analysis: The framework can provide a variety of analyses on SOA applications: o Security Policy Checking Service (SPCS): Allows security policy editing and runtime enforcement for both individual service and hierarchical integrated SOA applications. o Integrity Estimation Service (IES): Apply structure-based integrity estimation model on both individual service and integrated SOA.

5 o Reliability Analysis Service (RAS): Provide based on data flow analysis in the system data provenance reliability analysis on SOA system. o Other services are also possible and can be added into the framework. The provenance collection and analysis process starts from system requirements. The first step models the system with PSML-S [23]. All the workflows, from specifications. The next step decides which data, or which aspects of data need to be tracked using data provenance classification techniques. After that, data collection services are used to collect the provenance data. Multiple analyses can be done on the collected data, including security analysis, integrity analysis and reliability analysis. the top level to the bottom level will be modeled in PSML-S. After that, data flow diagrams are generated Analysis results may be used to enhance the original system. One can replace the faulty services by another service with the same specification using dynamic recomposition [24]. The proposed SOA data provenance framework can be used to collect and analyze the provenance data in the SOSE process as illustrated in Figure 3. The following subsection presents the classification and collection module of the data provenance system. Figure 3. Apply Data Provenance Framework on SOSE 3.1 Data Provenance Classification As not all data needed to be recorded or analyzed, the data provenance system needs to classify data into different provenance categories: Maximum provenance: This is for priority data, and a complete history from the creation of data will be recorded. A complete history will include all the IDs of services and workflows that process the data as well as the time the data was processed. Time-based provenance: This is for data for certain period of time. For example, data related to a specific transaction need to be traced from two weeks before the transaction to two weeks after the traction to ensure the completion of transaction, and its effects can be observed adequately. Event-based provenance: This is for data associated with a specific event, e.g., data related to a specific execution of a service for a given location. Actor-related provenance: This is for data related to a specific actor or agent in a system, an actor can be a process or human agent that initiates certain actions. For example, an SOA system will track every action issued by the CEO. Scenario-based provenance: Certain data related to a specific scenario or process will be collected once certain conditions are activated. For example, a terrorist plot scenario alert will trigger all the relevant data about the participants in the scenario. They will be stored and processed with a high priority.

6 Minimum provenance: This is for low priority data, and only certain aspects of data will be tracked, e.g., sender, receiver, intermediate service names, and time of message delivery. No provenance: This is for useless data that have no value and can be safely discarded. The data provenance classification module in the proposed framework includes two components: a dynamic Data Classification Service (DCS) and a Data Provenance Manager (DRM). Using the DCS, a Data Provenance Manager (DPM) can decide which data to track in the SOA system. Once the data has a new classification, e.g., from no provenance to maximum provenance, the corresponding tracking mechanism will be activated. Furthermore, the DCS will alert DPM about other dependent data as they may be reclassified as well to meet the new environment. For example, if the passengers of a flight need to be tracked, their friends and families may need to be tracked. Two types of data dependencies can be used to do automatic classification in DCS: pre-defined data dependencies and runtime data provenance history. The former includes all the static data dependencies arising from the workflow logic, while the latter includes all the runtime data dependencies identified during the execution. More specifically, these two sources may identify the following data dependencies: Pre-defined data dependencies include: o Input-input dependencies: both data are input of a service call. o Output-output dependencies: both data are output of a service call. o Input-output dependencies: one of the data is the input of a service call, and the other is the output of the same service call. Runtime data provenance history identifies the data dependencies including: o Common source dependencies: both data come from the same source service. o Common destination dependencies: both data arrive at the same destination service. o Common intermediate node dependencies: both data pass at least one common intermediate service. 3.2 Data Collection The framework allows a variety of data collection strategies: Actor-based: An actor (a system or service) can have a monitoring service or agent that track the data communicated. Message-based: Instead of depending on monitoring agents to collect data, the data may carry its provenance, e.g., in its XML file. Each service that uses the XML file can leave a trace in the file so that data provenance can be tracked. Actor-based data collection has a disadvantage of collecting information about specific data. If the provenance system needs to locate all the information about specific data, it needs to query or process the data stored by the monitors of actors. As the data volume can be large and data can have long life, processing these data may take significant time and effort. In the message-based approach, it is necessary to apply digital signatures to ensure that services or systems can overwrite the data during the subsequent computation or communication. Furthermore, if the data has a long life, the message can be rather long as each computation or routing will add more information to the message. Thus, it is cumulative. Another disadvantage is the required digital signatures, as a new signature will be added for each service that manipulate the data, the list of digital signatures can only increase as time progress. The processing of these digital signatures can be problematic. A hybrid solution is possible also. For example, actor-based approach will track only those maximum provenance data, while message-based approach will be used for short-lived data, i.e., those data will expire within 24 hours. The architecture of the proposed data collection module is shown in Figure 4. Figure 4. Data Collection Architecture

7 The data collection mechanism works as following: 1. Due to the hierarchical nature of SOA applications, the data collection mechanism is also hierarchical. 2. On each hierarchy, there is a global data collection agent (global monitor) which controls all local data collection agents on this hierarchy to collect data for analysis. For example, the business process shown in Figure 4 is composed of two component business processes. Besides the local data collection agents that belong to those two component business processes, this two-level hierarchy has another two global data collection agents: composite business process data collector and component business process data collector. This is efficient in the dynamic changing situation, because even if a component business process is recomposed, the composite business process data collector may not need to be changed as long as the data collected by and passed from the component business process data collector are the same. 3. Because the data flow may occur not only in one particular layer of the SOA system, but also between different layers, ESB [5] can be used in the data collection module as the data routing path in and between SOA layers. 4. Note that the communications between SOA system components and corresponding data collectors, as well as the communications between the data collectors and the ESB are both bidirectional. Therefore, a data collector can collect data either from the corresponding SOA system component, or from the ESB, whose source may be SOA system components at the same SOA system layer, or at different layers. In this way, the proposed data collection module manages to collect all the data flows in the SOA system. 4. Case Study The proposed SOA data provenance framework has been applied to an enterprise Service-Oriented Production Planning System (SOPPS) for a food processing company [16]. SOPPS makes production plans following three steps. First, the Monthly Production Planning Service (MPPS) retrieves annual production plans from other department of the company and makes monthly production plans with the key service -- aggregate capacity planning (ACP) service. It also collects customer orders and passes them to the weekly planning service. Then the Aggregation of Production Sections Planning Service (APSPS) takes monthly production plans and customer orders as the input to generate weekly production plans for all the product lines in company. Finally, Shop Floor Production Planning Service (SFPPS) creates daily production plans for each individual production line. It specifies which materials to be processed, and at which production line. Table 2 shows the data provenance classification results following the approach introduced in section 3.1. From the tables we can see that the data provenance framework tracks only three data with maximum provenance: MPP, MPS, and DPO. Five of the rest of the data are decided upon not to be tracked. The rest data will be tracked normally. Table 2. Data Classification Results in SOPPS System Data Name Life Time Track Priority Data Name Life Time Track Priority CO 1 month Normal SWSF 1 week Normal APP 1 month None SI N/A None MDFR 1 month Normal MPS 1 week Maximum SBR 1 month Normal MSC 1 day None MPP 1 month Maximum FSF 1 day None SMPP 1 month Normal DPO 1 day Maximum WSFR 1 week Normal CBS 1 day None Table 3. Partial Data Collectors in SOPPS System Data Collector Level Services/System Production Planning System Data Collector 4 Whole Production Planning System Customer Order Management Department Data Collector 3 COMD MONTHLY PRODUCTION PLANNING SERVICE Data Collector 2 MPPS Customer Orders Collecting Service Data Collector 1 COCS Annual Production Plans Retrieving Service Data Collector 1 APPRS As proposed in section 3.2, the data collectors in SOPPS system are hierarchically categorized into four levels, part of which are shown in Table 3. The data collectors with higher level rankings are used to

8 monitor the data input/output of the higher level composite services. The collected data are used for data integrity estimation and other data provenance analysis. 5. Conclusion This paper discussed the issues related to SOA data provenance, focusing on security, reliability, and integrity of data as they are routed in an SOA system. To address the related issues, this paper proposed a dynamic framework for data provenance classification and data collection. The proposed framework is applied in an enterprise production planning system to illustrate the process and techniques involved. 6. References [1] D. Bell and L LaPadula, Secure Computer System: Unified Exposition and Multics Interpretation, Technical Report, MITRE Corporation, March [2] M. Bishop, Computer Security: Art and Science, [3] R. Bose, and J. Frew, Lineage Retrieval for Scientific Data Processing: A Survey, ACM Computing Surveys, Vol. 37, No. 1, March 2005, pp [4] P. Bunemaaan, S. Khanna, and W. C. Tan, Why and Where: A Characterization of Data Provenance, in ICDT, [5] D. Chappell, Enterprise Service Bus, O Reilly Media, [6] D. G. Clarke and D. M. Clark, Lineage, in Elements of Spatial Data Quality, S. C. Guptill and J. L. Morrison, Eds. Oxford: Elsevier Science, 1975, [7] DCIO, DOD OASD NII, Net-Centric Checklist, version 2.1.2, March 31th, [8] GAO, Assessing the Reliability of Computer- Processed Data, External Version 1, October [9] C. Goble, Position Statement: Musings on Provenance, Workflow and (Semantic Web) Annotations for Bioinformatics, in Workshop on Data Derivation and Provenance, Chicago, [10] P. Groth, M. Luck and L. Moreau, A Protocol for Recording Provenance in Service-Oriented Grids, Proc. of 8th International Conference on Principles of Distributed Systems (OPODOS 04), [11] W. Kernochan, Mainframe Security Changes as Web Services Arrive. [12] D. Kiely, How SQL Server 2005 Enables Service-Oriented Database Architectures, Microsoft TechNet, January 16th [13] D. P. Lanter, Design of a Lineage-Based Meta- Data Base for GIS, in Cartography and Geographic Information Systems, vol. 18, [14] S. Miles, P. Groth, M. Branco, and L. Moreau, The Requirements of Recording and Using Provenance in e-science Experiments, Technical Report, Electronics and Computer Science, University of Southampton. [15] R. Paul, DoD Towards Software Services, Proc. of 10th IEEE International Workshop on Objectoriented Real-time Dependable Systems (WORDS 05), February 2005, pp [16] V. Portougal, Business Processes: Operational Solutions for SAP Implementation, Idea Group Pub, December [17] S. Rajbhandari and D. W. Walker, Support for Provenance in a Service-based Computing Grid, UK e-science All Hands Meeting, [18] S. Ross, Applied Probability Models with Optimizing Applications, Holden-Day, San Francisco, Ca, 1970, [19] Y. L. Simmhan, B. Plale, and D. Gannon, A Survey of Data Provenance in e-science, SIGMOD Record 34(3): (2005). [20] M. P. Singh and M. N. Huhns, Service-Oriented Computing, John Wiley & Sons, [21] V. Tan, P. Groth, S. Miles, S. Jiang, S. Munroe, S. Tsasakou, and L. Moreau, Security Issues in a SOA-based Provenance System, In Proceedings of the International Provenance and Annotation Workshop 2006 (IPAW 2006). [22] W. T. Tsai, "Service-Oriented System Engineering: A New Paradigm", IEEE International Workshop on Service-Oriented System Engineering (SOSE), Beijing October 2005, pp [23] W. T. Tsai, R. Paul, B. Xiao, Z. Cao, and Y. Chen, "PSML-S: A Process Specification and Modeling Language for Service Oriented Computing", The 9th IASTED International Conference on Software Engineering and Applications (SEA), Phoenix, November 2005, pp [24] W. T. Tsai, W. Song, R. Paul, Z. Cao, and H. Huang, Services-Oriented Dynamic Reconfiguration Framework for Dependable Distributed Computing, COMPSAC, September 2004, pp [25] W. T. Tsai, X. Wei, R. Paul, J. Xu, Q. Huang, and B. Xiao, Single Model, Multiple Analyses (SMMA) for Service-Oriented System Engineering (SOSE), to appear in Proc. of IEEE FTDCS 2007.