Concurrent View Maintenance Scheme for Soft Real-Time Data Warehouse Systems

Transcription

1 JOURNAL OF INFORMATION SCIENCE AND ENGINEERING 23, (2007) Concurrent View Maintenance Scheme for Soft Real-Time Data Warehouse Systems NAMGYU KIM AND SONGCHUN MOON * Department of Business Information Technology Kookmin University Seoul, Korea * Department of Management Engineering Korea Advanced Institute of Science and Technology Seoul, Korea The data warehouse should be refreshed periodically or aperiodically in order to prevent OLAP transactions from reading extremely stale data. This refreshing process is referred to as view maintenance. There have been many researches on ways to preserve data consistency during the process of view maintenance. The simplest approach among them is to perform view maintenance in the nighttime while OLAP transactions are allowed to be active only in daytime. However, this approach cannot be applied to globalized corporations because no common idle time is available for such corporations. In this paper, we propose a new algorithm for performing view maintenance without blocking OLAP transactions. Our algorithm improves data freshness by allowing one or more maintenance transactions and OLAP transactions to be executed concurrently. We introduce the details of algorithm with some examples, present proofs of devised scheme, and then evaluate performance gains by means of simulation method. Keywords: concurrency control, database management, data warehouse, information storage and retrieval, real-time systems, view maintenance 1. INTRODUCTION OLAP [1] transactions usually issue long-lived queries like complicated decision support processes. Multiple, distributed, and heterogeneous databases are integrated to support such information-intensive OLAP queries. The integration methods can be classified into on-demand approach and in-advance approach [2]. In the former approach, information is not extracted from the source databases until queries are actually issued. In the latter one, on the contrary, the information of interest is extracted from each source database in advance, and then it is stored in a logically centralized repository. When a query is actually issued, it is analyzed directly at the repository without accessing operational database (ODB). This approach is commonly referred to as data warehousing and the integrated repository itself is referred to as data warehouse (DWH). Data flows from ODB into DWH [3] in DWH environment. Value of data in DWH cannot be updated directly by write operations because only read operations of OLAP transactions are allowed to access data in DWH. Data in ODB could be, on the contrary, updated directly by write operations of OLTP transactions. Therefore, the data value in DWH and ODB would be getting different with the passage of time. It implies that data value in DWH Received June 7, 2005; revised December 12, 2005 & January 17, 2006; accepted March 20, Communicated by Ming-Syan Chen. 723

2 724 NAMGYU KIM AND SONGCHUN MOON should be replaced with recent value of data in ODB in order to prevent OLAP transactions from reading excessively stale data from DWH. This refreshing process is frequently referred to as view maintenance and is performed by maintenance transactions generated by DWH management system. In the literature, there have been many researches [2, 4-8] on ways to preserve data consistency [9] during the view maintenance. The simplest approach among them is to execute maintenance transactions in the nighttime. In this approach, every maintenance transaction is supposed to be executed only in the nighttime and no OLAP transaction is allowed to execute its read operations in this period. On the contrary, no maintenance transaction is allowed to execute its write operations in daytime. View maintenance can be performed without damaging data consistency in this manner. This approach has revealed two kinds of limitations. First of all, there is no common nighttime for a globalized corporation. If any, it is hard to pledge view maintenance transactions to be completed in the nighttime because maintenance transactions are usually long-lived ones. In this sense, there have been many efforts to enable both OLAP and maintenance transactions to be performed concurrently without damaging data consistency. Most of them assumed that two or more maintenance transactions are not allowed to be executed concurrently. This restriction seems to be unduly strict when it is considered that each ODB might be owned by mutually different owners. Moreover, the restriction would degrade transaction concurrency and data freshness [10] because it disallows concurrent execution of maintenance transactions even when they are independent of each other. By relaxing the restriction, therefore, the data freshness of DWH could be improved without raising cost of version management. Our view maintenance scheme proposed in this paper could be characterized as follows. At first, the scheme tries to minimize the cost of space management by maintaining only one version for OLAP transactions and the other one for OLTP ones. Additionally, the proposed scheme could improve data freshness of DWH by allowing concurrent execution of multiple maintenance transactions. To explain our algorithm simply by transactional approach [4], we regard OLAP transactions as read-only ones and OLTP transactions as write-only ones. In this paper, we discuss details of the algorithm with some examples, present proofs of devised scheme, and then evaluate performance gains by means of simulation method. The rest of this paper is organized as follows. Section 2 presents overviews and limitations of previous works on view maintenance. In section 3, we propose conflict order-based (COB) view maintenance scheme for soft real-time data warehouse environment. Section 4 provides proofs of correctness for COB. Performance gains of COB are analyzed in section 5. Finally, section 6 concludes this paper. 2. RELATED WORKS In order to preserve data consistency between ODBs and DWH, traditional view maintenance schemes [6] allowed only one maintenance transaction to be executed at a time. However, the serial execution of maintenance transactions would degrade the data freshness. In [6], two-version no locking (2VNL) algorithm is proposed for enhancing data availability by allowing OLAP transactions and a maintenance transaction to be executed concurrently. 2VNL achieves the concurrency by offering two alternative data

3 VIEW MAINTENANCE SCHEME FOR DATA WAREHOUSE SYSTEMS 725 versions to OLAP transactions. 2VNL maintains three types of database versions such as a future version, a current version, and a previous version. When a maintenance transaction is running, there exists only a future version and a current version. On the contrary, only a current version and a previous version can exist when there is no active maintenance transaction. Maintenance transactions always operate on future versions while OLAP transactions could read data value from either previous versions or current versions. Each OLAP transaction is associated with exactly one version that was a current one when the transaction began, and continues to read the version all through its lifetime. Fig. 1 illustrates a simple scenario for view maintenance in 2VNL. Maintenance Transactions T U1 T U2 T U3 OLAP Transactions T R1 T R2 T R3 T R4 T R5 V 0 Database Versions V1 V 2 Time 8 am 9 am 8 am 9 am Fig. 1. Performing view maintenance without blocking OLAP transactions. In Fig. 1, OLAP transactions T R1, T R2, T R3, T R4, and T R5 are continuing their execution without being blocked during execution of maintenance transactions T U1, T U2, and T U3. To support this concurrency, 2VNL provides two database versions to OLAP transactions. In Fig. 1, V 0 is an initial version of database while V 1 and V 2 represent new versions which are created by maintenance transactions T U1 and T U2, respectively. Let us suppose that every maintenance transaction starts at 9 a.m. and finishes its execution at 8 a.m. on the next morning. A new database version is created as soon as the maintenance transaction finishes. This version is maintained from then and destined to be expired at 9 a.m. on the next day. In Fig. 1, OLAP transactions T R1 and T R2 are destined to access only V 0 which was a current version when they started. In particular, it should be noticed that T R2 should still read V 0 even after 8 a.m. although another version V 1 is newly created and is currently available to T R3. A 2VNL scheduler disallows T R2 to read V 1 in order to prevent T R2 from reading inconsistent data from two versions, V 0 and V 1. Therefore the version V 0 should be still offered to T R3 till 9 a.m. although it is not a current version any longer. To prevent the number of versions from increasing infinitely, the previous versions ought to be expired at a predefined deadline (i.e. 9 a.m.). In the same manner, T R3, T R4, and T R5 are destined to access V 1 all through their lives. T R4 reads V 1 even after 8 a.m. because of the same reason in the case of T R2. We should be attentive to execution of T R5. T R5 will be

4 726 NAMGYU KIM AND SONGCHUN MOON aborted finally because it reads data from V 1 and is still active after the expiration of V 1. It implies that the more OLAP transactions would be aborted by 2VNL scheduler when the larger size of OLAP transactions is executed. In 2VNL, an OLAP transaction should be aborted if it is still running even after its associated version has been expired. Frequent aborts of OLAP transactions could cause critical overheads because OLAP transactions usually have long execution time. It implies that only two versions might be insufficient for view maintenance in DWH management systems [11]. In this sense, [6] proposed not only 2VNL but also nvnl algorithm which maintains more than two database versions. However, nvnl algorithm suffers from a limitation that it is not easy to estimate the sufficient number of required versions exactly. Cost of version management, moreover, increases as the number of versions increases. On the other hand, 2VNL and nvnl enforce only one maintenance transaction to be executed at a time. This strict restriction could degrade transaction concurrency because it disallows multiple maintenance transactions even in the case that they do not conflict with each other at all. 3. CONFLICT ORDER-BASED VIEW MAINTENANCE SCHEME 3.1 Notion of Publish Ordering To execute multiple maintenance transactions concurrently without damaging data consistency, COB maintains a data version for ODB (VODB) as well as a version for DWH (VDW). Only one VODB is maintained globally instead of being maintained locally by each ODB. Let us suppose that T Ui represents an OLTP transaction which updates data in VODB while T Ri is an OLAP transaction. Every data updated by T Ui cannot be directly reflected in VDW so it should be stored in VODB for a while. On the contrary, T Ri s read operation retrieves data value directly from VDW. In summary, every OLAP transaction reads data value from VDW while every OLTP transaction updates new data value into VODB. In order to prevent OLAP transactions from reading excessively stale data, therefore, data in VDW should be replaced with recent data in VODB by maintenance transactions. In this paper, a term publish stands for the replacement process and a notation Pub( ) denotes its operation. Every data in VDW could be refreshed with recent data in VODB only by performing publish operations. For example, publish of every data written by a committed transaction T Ui is represented as Pub(T Ui ). For the purpose of preserving data consistency, COB utilizes publish order graph (POG) in scheduling conflict-free publish order. Conflicts are not occurred between OLAP transactions because OLAP transactions never issue any write operation. Every conflict occurred during the view maintenance process would be only the following patterns: (1) a conflict between a read operation of an OLAP transaction and a write operation of a maintenance transaction, (2) a conflict between write operations of maintenance transactions. We will refer the first case as inter-conflict and the other as intra-conflict. In the next section, some rules will be presented to enable COB protocol to generate a conflict-free schedule. Each maintenance transaction is generated from exactly one OLTP transaction. To create a maintenance transaction, COB only appends publish operation to the end of the transaction. Some transactions could execute publishes immediately after their commits while others should

5 VIEW MAINTENANCE SCHEME FOR DATA WAREHOUSE SYSTEMS 727 be delayed. By arranging publish order of the maintenance transactions in accordance with the precedence of POG, COB scheduler could preserve data consistency during concurrent execution of maintenance transaction. 3.2 View Maintenance Using Publish Order Graph When a transaction is committed, some links would be appended to POG if the transaction has executed any conflicting operations with other transactions. To produce a one-copy serializable view maintenance schedule [9], every updated data in VODB should be published in the topological order of POG. In the following scenario, T Rd is an OLAP transaction while T Ui and T Ud are view maintenance transactions each of which is generated by attaching a publish operation to the end of an OLTP transaction. A link T Ud T Ui on POG implies that Pub(T Ui ) should be performed after Pub(T Ud ). Similarly, Pub(T Ui ) cannot be performed until T Rd is committed when T Rd T Ui exists in POG. In this subsection, we define Rules 1 and 2 for appending links to POG and Rules 3 and 4 for publishing data in one-copy serializable order. When each transaction is committed, the rules are triggered in the sequential order of Rules 1, 2, 3 and 4. OLAP Transaction: T R1, T R2 OLTP Transaction: T U1, T U2, T U3 T R1 r(x) r(y) c R1 T R2 r(y) r(z) c R2 T U1 w(x) w(y) c U1 Pub(T U1 ) T U2 w(z) c U2 Pub(T U2 ) T U3 w(y) w(z) c U3 Pub(T U3 ) Fig. 2. Data inconsistency caused by immediate publishes of T U1, T U2, and T U3. Data consistency might be threatened if a transaction publishes its updated data immediately after its commit (Fig. 2). T R1 and T R2 are reading data from VDW while T U1, T U2, and T U3 are updating data on VODB. Every data updated by OLTP transactions could become accessible to OLAP transactions only by Pub( ) operations. In the first place, let us take only T R1 and T U1 into consideration. The data x and y updated by T U1 is not accessible to T R1 until Pub(T U1 ) is performed. Therefore, T R1 reads old value of x while it reads new value of y which has been updated and published by T U1. It results in inconsistent data retrieval because T R1 is affected by partial results of T U1. To be a one-copy serializable schedule, every transaction should be affected by all or no result of other transactions. We can detect inconsistent data retrieval if any cycle in serialization graph (SG) [9] is observed. In the SG for T U1 and T R1, we can detect a cycle between them which consists of links T R1 T U1 on data x and T U1 T R1 on data y. COB regulates the following Rule 1 to prevent this type of cycle from being formed. In the following rules, T Ui, T Ud are maintenance transactions while T Ri, T Rd are OLAP ones. They are assumed to be participating in contention on data x. Operation Pub(T Ui ) is initially issued only after T Ui s commit. The operation could be executed immediately or should be de-

6 728 NAMGYU KIM AND SONGCHUN MOON layed according to the following rules. In the following rules, a symbol < H describes that an operation located in the left side of the symbol precedes the other one located in the right side of it in the generated schedule. Rule 1 Direct dependency preservation between inter-conflicting transactions: When Pub(T Ui ) is issued, append T Rd T Ui to POG if T Rd is running and r Rd (x) < H w Ui (x) or w Ui (x) < H r Rd (x) is detected in a schedule. By Rule 1, Pub(T U1 ) appends a link T R1 T U1 to POG in Fig. 2. Therefore, T U1 s publish should not be performed until T R1 s publish is finished. Data consistency is not violated because both of x and y read by T R1 have not been affected by T U1. Again, let us take not only T R1 and T U1, but also T U3 into consideration. At the stage of T U3 s commit, there is no active OLAP transaction which has executed read operations on y or z. Therefore, T U3 can be published immediately after its commit because no transaction precedes T U3 on POG when only Rule 1 is considered. Nevertheless, a different type of cycle could be formed in SG when T R1 executes a read operation on y which has been written by T U3. In this case, T R1 s read operation on data x indirectly influences T U3 via dependencies of T R1 T U1 and T U1 T U3. It results in an unserializable schedule among T R1, T U1, and T U3. To prevent the dependency between T R1 and T U1 from being delivered to T U3, T U3 s publish should not be performed until T U1 is published. Rule 2 is devised for this purpose and it is invoked by every OLTP transaction s commit. Rule 2 Indirect dependency preservation between intra-conflicting transactions: When Pub(T Ui ) is issued, append T Ud T Ui to POG if, for a committed transaction T Ud, w Ud (x) < H w Ui (x) is detected in a schedule and a node for T Ud already exists in POG. Let us reconsider T R1, T U1, and T U3 in Fig. 2. If Ru1e 2 has not been defined, T U3 s publish may be performed as soon as it is committed while T U1 s publish should be delayed after T R1 s commit by Rule 1. It means that the final write operation on y would be executed by T U1 in VDW while it has been executed by T U3 in VODB. Therefore, Rule 2 as well as Rule 1 ought to be defined to preserve data consistency between DWH and ODB. By Rule 2, T U3 s publish should not be performed until T U1 s publish is executed. In the case of T U2, it can perform publish immediately after its commit because it does not participate in any type of conflict mentioned in Rules 1 and 2. To improve data freshness of DWH, COB scheduler permits some OLTP transactions to be published immediately after their commits if and only if they are participating in neither inter-conflicts nor intraconflicts. The improvement of freshness could be explained by observing T R2 s read operations on y and z. Owing to T U2 s immediate publish, T R2 can read fresh value of data z which has been updated by T U2. However, T R2 still reads somewhat stale value of data y which has been updated by T U3 but not published yet. By applying Rules 1 and 2, we can modify the schedule in Fig. 2 into the one in Fig. 3. Further discussion on the data y in Fig. 3 is required to examine committed but not published OLTP transactions in detail. The value of data y updated by T U1 cannot be published immediately because T U1 has its precedence (i.e. T R1 ) on POG. It means that updated value of x and y are not reflected in VDW yet due to delayed publish although the values are written in VODB immediately after T U1 s commit. Note that the value of y in

7 VIEW MAINTENANCE SCHEME FOR DATA WAREHOUSE SYSTEMS 729 OLAP Transaction: T R1, T R2 OLTP Transaction: T U1, T U2, T U3 T R1 r(x) r(y) c R1 T R2 r(y) r(z) c R2 T U1 w(x) w(y) c U1 Pub(T U1 ) T U2 w(z) c U2 Pub(T U2 ) T U3 w(y) w(z) c U3 Pub(T U3 ) Fig. 3. One-copy serializable schedule with delayed publishes. T U2 T R1 T U1 T U3 T R2 Fig. 4. Serialization graph for the schedule in Fig. 3. VODB is overwritten by T U3 before it is reflected in VDW by Pub(T U1 ). When Pub(T U1 ) is executed, therefore, data y in VODB contains only the value written by T U3 while data x does the value written by T U1. In this sense, the exact definition of Pub(T Ui ) will be publishing current value of every data which has ever been written by T Ui whether the value is actually written by T Ui or not. COB maintains only recently updated value of every data in VODB rather than manages multiple data versions for temporary results of each maintenance transaction. To prevent overwritten data from damaging data consistency, COB organizes timings of publish operations by referring topological order of POG. The committed but not published data value could be stored in buffer for better performance although this is not considered in this paper. The modified schedule is one-copy serializable because there is no cycle in its SG (Fig. 4). For instance, the schedule in Fig. 3 is identical to a serial schedule T U2 T R2 T R1 T U1 T U3. It should be noticed that T R1 could precede T U1 and T U3 in the SG because the data y read by T R1 is stale one which is affected by neither T U2 nor T U3. More detailed proofs are provided in section 4. The POG for the schedule in Fig. 3 is constructed as T R1 T U1 T U3 at the stage of T R1 s commit. And then, publishes of T U1 and T U3 could be performed based on this order. We defined Rules 1 and 2 for ordering publishes. The rules are invoked when every OLTP transaction T Ui is committed. When an OLTP transaction T Ui is committed, Pub(T Ui ) can be performed immediately if there is no node for it on POG. For instance, T U2 in the schedule in Fig. 3 can perform publish immediately after its commit because there is no node for T U2 on POG. We define Rule 3 for such an immediate publish. Additionally, T Ui has another chance to be published even though there is a node for it on POG. T Ui can be published only if there is no incoming link into the node for it. Rule 4 is defined to manage this situation. Differently from the Rules 1, 2, and 3, Rule 4 is invoked by commit of an OLAP transaction. When Rule 4 is applied, some OLTP transactions may have chance to perform their delayed publishes.

8 730 NAMGYU KIM AND SONGCHUN MOON The delayed publishes could invoke recursive publishes of other OLTP transactions which have been waiting for the OLAP transaction s commit. Therefore, Rule 4 should be defined with recursion for handling the cascading publishes. Rule 3 Immediate publish: When Pub(T Ui ) is issued, perform Pub(T Ui ) immediately if there is no node for T Ui on POG. Rule 4 Delayed publish: When T Ri is committed, remove a node for T Ri and its outgoing links from POG. In resultant POG, find any node which has no incoming link. If any, publish a transaction represented by the node. And then, remove the node and its outgoing links from POG. Repeat the process until there is no node with no incoming link. Let us apply the rules to the schedule in Fig. 3. By Rule 1, a link T R1 T U1 is appended to POG after T U1 s commit. On the contrary, T U2 can be published immediately after its commit by Rule 3. When T U3 is committed, a link T U1 T U3 should be appended to POG by Rule 2. Consequently, the resultant POG becomes T R1 T U1 T U3. The situation implies that T U1 and T U3 cannot perform their publishes until T R1 is committed. T R2 s commit does not affect POG because no OLTP transaction is waiting T R2 s commit. On the contrary, Rule 4 is invoked when T R1 is committed. Rule 4 removes a node for T R1 and its outgoing link from POG. After the removal, POG contains only one dependency, T U1 T U3. Recursive application of Rule 4 enables T U1 and T U3 to perform their publishes in topological order. In result, total order of publishes becomes T U2 T U1 T U3. By performing publishes in compliance with the order, COB can serialize the schedule in Fig. 3 like T U2 T R2 T R1 T U1 T U3. It is a matter of course that the scenario in Figs. 2 and 3 illustrates somewhat unrealistic situation. In real-world applications, the number of OLAP transactions and the size of them are expected to be larger than those of OLTP ones. The unreality nevertheless does not seem to weaken the worth of the example because the example is devised for the purpose of explaining four rules of COB as concise as possible. If the more OLAP transactions run concurrently, the fewer fresh data in VDW will be read by OLAP transactions because of the more delayed publishes. On the contrary, it should be noticed that only some OLTP transactions conflicting with active OLAP transactions should delay publishes while irrelevant OLTP ones could execute publishes immediately after their commits. The frequency of the data conflicts is expected to be low when the size of DWH is large enough. When COB is applied to real-world applications, the freshness will vary with the number of concurrent transactions, a ratio of OLAP transactions, and the size of DWH. The effect of environmental parameters on data freshness will be analyzed in section Pseudo Algorithm for Ordered Publish Rules 1, 2, 3, and 4 could be summarized in the following pseudo-algorithm. In the algorithm, procedure Pub(T U ) is called by operation Pub(T U ) of OLTP transaction T U while procedure Trigger(T R ) is called by OLAP transaction T R s commit. The last procedure Recursive_Pub(T i ) is called in the procedure Trigger( ).

9 VIEW MAINTENANCE SCHEME FOR DATA WAREHOUSE SYSTEMS 731 Procedure Pub(transaction T U ); BEGIN FOR EACH active OLAP transaction T d IF r d (x) < H w U (x) or w U (x) < H r d (x) exists in a schedule H THEN append T d T U to POG; //Rule 1// END IF END FOR FOR EACH committed OLTP transaction T d on POG IF w d (x) < H w U (x) exists in a schedule H THEN append T d T U to POG; //Rule 2// END IF END FOR IF there is still no node for T U on POG THEN CALL Pub(T U ); //Rule 3// ELSE delay Pub(T U ) after its precedence s publish; END IF END Procedure Trigger(transaction T R ); BEGIN IF there is a node for T R on POG THEN CALL Recursive_Pub(T R ); //Rule 4// END IF END Procedure Recursive_Pub(transaction T i ); BEGIN FOR EACH transaction T k such that T i T k on POG remove a link T i T k from POG; IF there is no incoming link to T k THEN CALL Pub(T k ); END IF IF T k has any outgoing link THEN CALL Recursive_Pub(T k ); ELSE remove a node for T k from POG; END IF END FOR remove a node for T i from POG; END 4. CORRECTNESS PROOFS In the previous section, we examined that every OLAP transaction could read consistent data from VDW in the schedule produced by COB. It means that every schedule produced by COB scheduler is one-copy serializable even though COB allows concurrent execution of multiple OLAP and OLTP transactions. We here present some properties of COB scheduler and proofs for them in a semi-formal manner.

10 732 NAMGYU KIM AND SONGCHUN MOON Lemma 1 Conflict-serializability among OLTP transactions: All OLTP transactions are conflict-serializable. Proof: COB has no additional restriction on scheduling operations of OLTP transactions. Note that four rules in section 3 affect only timing of publish after the OLTP transaction itself has been committed. It implies that the rules are irrelevant to either commit or read/write operations. Therefore, traditional concurrency control schemes can be directly applied to produce conflict-serializable schedule among OLTP transactions. More formal proofs could be found in [9]. Property 1 Conflict-serializability among all committed transactions: All committed transactions are conflict-serializable. Proof: Let us suppose a schedule which consists of OLAP transactions T R1, T R2,, T Rn and OLTP transactions T U1, T U2,, T Um. Because any conflict cannot be occurred only among OLAP transactions, the Property 1 could be validated enough by observing conflict-serializability among only one OLAP transaction and several OLTP transactions. Sequential order of T U1 T U2 T Um could be assumed without damaging generality because all OLTP transactions are conflict-serializable by Lemma 1. A schedule can be regarded to be conflict-serializable if its corresponding SG has no cycle. Now let us prove the Property 1 by the method of reduction to absurdity. If an OLAP transaction T Ri could form a cycle with OLTP transactions, the only possible type of the cycle in SG whould be T Ri T Ut T Uk T Ri (for 1 i n, and 1 t < k m). In this cycle, T Uk T Ri describes the situation that T Ri reads some data written and published by T Uk. However, any data written by T Uk cannot be published until T Ri s commit according to the dependencies T Ri T Ut and T Ut T Uk on POG. This makes contradiction. It implies that T Ri cannot read the data written by T Uk. Hence, the cycle T Ri T Ut T Uk T Ri cannot be formed if every publish is performed in compliance with the Rules 1-4. In conclusion, we can guarantee that all committed transactions are conflict-serializable in any schedule produced by COB scheduler. Property 2 Write order preservation: COB guarantees that a sequence of publishes for a data is performed in the same order of writes on the data in VODB. Proof: Let us suppose that there are OLTP transactions T U1,, T Ut,, T Uk,, T Um (for 1 t < k m). The two transactions T Ut and T Uk are assumed to have made conflicts by executing write operations on the same data. Sequential order of T U1 T Ut T Uk T Um could be assumed without damaging generality because all OLTP transactions are conflict-serializable by Lemma 1. It implies that order of write operations between T Ut and T Uk on the conflict data is T Ut T Uk. In this situation, let us assume again that Pub(T Uk ) is performed earlier than Pub(T Ut ). It means that POG contains a link T Uk T Ut. By Rule 2, the link represents relative order of write operations between the two OLTP transactions (i.e. T Uk T Ut ). Resultant SG would contain a loop between T Uk and T Ut. This is contradictory to Lemma 1 that all OLTP transactions are conflict-serializable. In conclusion, we can guarantee that a sequence of publishes for the same data is performed in the same order of write operations on the data in VODB.

11 VIEW MAINTENANCE SCHEME FOR DATA WAREHOUSE SYSTEMS 733 In this paper, we used many abbreviations for data structures and algorithms. We summarize the abbreviations in a table for better readability (Table 1). Table 1. Summary of all abbreviations (listed in an order of appearance). Abbreviation Full Name Description ODB operational database database for OLTP transactions DWH data warehouse integrated repository of many ODBs COB conflict order-based scheme new view maintenance scheme proposed here 2VNL two-version no locking scheme traditional view maintenance scheme [6] VODB version for operational database every update operation is occurred here VDW version for data warehouse every read operation refers this POG publish order graph ordered graph for scheduling publish order 5. PERFORMANCE EVALUATION In this section, a performance of COB is evaluated under the various workloads by means of simulation approach. Among a few of researches, 2VNL is chosen to be compared with COB since it is not only a representative view maintenance technique for data warehouse environment but also it has a similar data structure (i.e. two data versions) to COB. This simulation has been performed by means of Microsoft Visual C compiler and CSIM version 18 [12] on the Windows XP operating systems. 5.1 Simulation Parameters and Performance Indices Simulation parameters and their value are listed in Table 2. We have set the value in accordance with [13] for accurate comparison to previous researches. There may be a gap between the value in Table 2 and real value in advanced modern systems. However, it should be noticed that the goal of this simulation is to analyze relative discrepancy in performance between COB and 2VNL. We believe that the setting does not weaken the meaning of this simulation. In the simulation, multiple OLAP transactions and multiple OLTP transactions are assumed to be executed concurrently. Data in VODB can be updated directly by OLTP transactions. Changed value of the updated data should be transmitted to VDW only by maintenance transactions. In the case of 2VNL, at most one maintenance transaction can be running at a time. The maintenance transaction is composed from results of multiple OLTP transactions and it is executed in a batched fashion to replace stale data in VDW with fresh data in VODB. In the case of COB, on the contrary, each OLTP transaction produces its own maintenance transaction by attaching publish operation after its commit. It is well known that performance of algorithms is affected not only by data contention but also by resource contention such as CPUs and disks. To provide a reasonable level of data and resource contention, we set the number of data in data warehouse, DW_size, to be relatively small in comparison with that of real system in industry. Primary performance indices observed in our experiments are fresh_read_ratio and deadline_missed_ ratio. They are defined as follows:

12 734 NAMGYU KIM AND SONGCHUN MOON Table 2. Simulation parameters. num_cpus num_disks cpu_cc_delay obj_io_delay S min, S max DW_size num_terms tr_size_min OLTP_size_max OLAP_size_max OLAP_ratio 2 CPUs 4 DISKs 7.5 milliseconds 35 milliseconds 4, 10 System Parameters Database and Transaction Parameters 1,000 to 10,000 pages 20 to 200 terminals 3 pages 6 pages 50 pages 10 to 90 % The number of CPUs The number of DISKs CPU time for concurrency control I/O time for accessing an object Minimum and Maximum slack factor The number of objects in data warehouse The number of terminals Size of the smallest transaction Size of the largest OLTP transaction Size of the largest OLAP transaction Ratio of OLAP to total transactions 1. fresh_read_ratio: a ratio of the number of read operations which have read the most recent value from VDW to the number of total read operations. 2. response_time: average response time for each transaction (from begin to commit). 3. deadline_missed_ratio: a ratio of the number of transactions which have missed their deadlines to the number of total committed transactions. We used fresh_read_ratio and deadline_missed_ratio for the purpose of analyzing real-time properties. The value of fresh_read_ratio indicates freshness of data read by OLAP transactions. deadline_missed_ratio is defined as an average ratio of the number of committed transactions which could not meet their deadlines to the number of total committed transactions. The deadline of transaction is calculated by Definition 1. Definition 1 Deadline of transaction: A deadline of transaction T i is defined as follows: DL(T i ) = ARR(T i ) + SLACK EXP(T i ) where DL(T i ) is a deadline of T i, ARR(T i ) is arrival time of T i, and SLACK is a factor for slack. In our simulation, SLACK is statically chosen from uniform distribution in the range between [S min, S max ]. EXP(T i ) is expected execution time of T i and it is calculated as follows: EXP(T i ) = (the number of operations in T i ) (cpu_cc_delay + obj_io_delay). 5.2 Simulation Results and their Implication We evaluate performance gains of COB and 2VNL in respect of fresh_read_ratio and deadline_missed_ratio with DW_size, num_terms, and OLAP_ratio varied. num_ terms is the number of terminals and it is therefore identical to multiprogramming levels. We used 2,500 transactions for each run. We used 2,500 transactions for each run. Each run is performed ten times with various random seeds. Average value of the ten runs is used for performance analysis.

13 VIEW MAINTENANCE SCHEME FOR DATA WAREHOUSE SYSTEMS Analysis of fresh_read_ratio The higher fresh_read_ratio implies that the more read operations have read the most recent value of data from VDW. To compare performance of 2VNL and COB in respect of data freshness, we performed experiments for fresh_read_ratio with DW_size, OLAP_ratio and num_terms varied. At first, we examined effects of DWH s size on the data freshness. In the first experiment, value of DW_size is varied from 1,000 to 10,000 in steps of 1,000 (Fig. 5). fresh_read_ratio (%) COB 2VNL DW_size Fig. 5. fresh_read_ratio with DW_size varied (num_terms = 100, OLAP_ratio = 50%). Fig. 5 shows that fresh_read_ratio of COB increases as DW_size grows. When the number of concurrent transactions is fixed, increasing DW_size would lower a possibility of two or more transactions concurrent access to the same object. In result, the increasing DW_size could lower the number of committed but not published OLTP transactions by lowering the frequency of data conflicts between OLAP and OLTP transactions. It implies that the more OLAP transactions could expect to read the most recent data value when the larger DW_size is available. On the contrary, 2VNL s fresh_read_ratio does not fluctuate with DW_size because version change in 2VNL is performed periodically irrespectively of the level of contention among transactions. The result in Fig. 5 reveals that COB could outperform 2VNL with regard to the data freshness only when the DW_size is larger than 3,000. Because the size of DWH in real-world application is usually assumed to be large enough, we expect that COB could provide better freshness to OLAP transactions than 2VNL in real-world environments. fresh_read_ratio of COB is primarily affected by the timing of publish. The execution of publish operation will be delayed longer when the more active OLAP transactions are running concurrently. We performed the next experiment to examine how the ratio of OLAP transactions to all transactions affects data freshness. In this experiment, value of OLAP_ratio is varied from 10 to 100 in steps of 10 (Fig. 6 (a)). Far from our expectations that the freshness will be more damaged as the number of OLAP transactions grows, fresh_read_ratio of COB in Fig. 6 (a) shows small improvement as OLAP_ratio grows. It should be noticed here that the parameter used in this experiment is not the number of OLAP transactions itself but a ratio of the number of OLAP transactions to the number

14 736 NAMGYU KIM AND SONGCHUN MOON fresh_read_ratio fresh_read_ratio OLAP_ratio num_terms COB 2VNL Fig. 6. (a) fresh_read_ratio with OLAP_ratio varied (DW_size = 5,000, num_terms = 100). COB 2VNL Fig. 6. (b) fresh_read_ratio with num_terms varied (DW_size = 5,000, OLAP_ratio = 50%). of total transactions. In other words, the experiment is performed with only OLAP_ratio varied while the number of total concurrent transactions is fixed. The higher OLAP_ratio, therefore, means that the smaller number of OLTP transactions are concurrently executed, which causes the number of delayed publishes to decrease. When only a few OLTP transactions are issued due to high OLAP_ratio, most of data in VDW would have the same value as VODB s one because the data in VODB cannot be updated frequently. Therefore, OLAP transactions could have the more chance to read the most recent data when the higher OLAP_ratio is maintained. The same reasoning could be applied to 2VNL s case. Because the experiment in Fig. 6 (a) is performed with only varying the ratio of OLAP transactions while the whole workload is fixed, we could not observe the pure effect of increasing number of OLAP transactions on delayed publishes. To see the effect of the number of transactions on data freshness, we performed another experiment with varying num_terms and fixing OLAP_ratio. In this experiment, value of num_terms is varied from 20 to 200 in steps of 20 (Fig. 6 (b)). The result reveals that the fresh_read_ ratio of COB decreases as num_terms grows. The phenomenon might be caused by the fact that the increasing number of concurrent transactions would deepen the data contention among transactions. The heavy contention would degrade the fresh_read_ratio of COB by increasing the number of delayed publishes. In the case of 2VNL, on the contrary, fresh_read_ratio is revealed to be hardly affected by variance of num_terms because version change in 2VNL could be performed irrespectively of the level of contention among transactions. COB shows better freshness than 2VNL through most of workload except when the number of transactions is extremely large Analysis of response_time and deadline_missed_ratio We performed an experiment to observe average response time for each transaction. In the experiment, value of num_terms is varied from 20 to 200 in steps of 20 (Fig. 7). In the both cases of COB and 2VNL, as expected, response_time increases as num_terms increases. The reason can be explained from the viewpoint of data and resource contention. In Fig. 7, COB shows higher response_time than 2VNL when more than 100

15 VIEW MAINTENANCE SCHEME FOR DATA WAREHOUSE SYSTEMS 737 response_time num_terms COB 2VNL Fig. 7. response_time with num_terms varied (DW_size = 5,000, OLAP_ratio = 50%). deadline_missed_ratio deadline_missed_ratio num_terms OLAP_ratio COB 2VNL COB 2VNL Fig. 8. (a) deadline_missed_ratio with num_terms varied (DW_size = 5,000, OLAP_ratio = 50%). Fig. 8. (b) deadline_missed_ratio with OLAP_ratio varied (DW_size = 5,000, num_terms = 100). terminals work simultaneously. This could be because COB produces a larger number of maintenance transactions according as a larger number of terminals work while 2VNL produces only one maintenance transaction regardless of the number of working terminals. However, COB s time consumption for handling publish operations is not revealed not be serious. This could be because delay between commit and publish of a transaction is not included its response time. Because shorter response time is not always proportioned to lower deadline misses, we performed another experiment to observe the number of transactions which missed their deadlines. In the experiment on effects of the level of contention on deadline misses, value of num_terms is varied from 20 to 200 in steps of 20 (Fig. 8 (a)). Fig. 8 (a) shows that deadline_missed_ratio increases as num_terms grows in both cases of the two schemes. This could be resulted from the fact that transactions would spend more time in waiting queue during scheduling when the contention among transactions becomes heavier. Note that transaction s deadline is not considered at all during the scheduling of 2VNL while COB assigns higher priority to a transaction with an earlier deadline. The slight difference between the two schemes deadline_missed_ratio might be induced by the difference in priority assignment policies.

16 738 NAMGYU KIM AND SONGCHUN MOON The last experiment we have performed was to analyze an effect of OLAP transactions ratio on deadline misses. In this experiment, value of OLAP_ratio is varied from 10 to 100 in steps of 10 (Fig. 8 (b)). Note that read operations of OLAP transactions are never blocked during scheduling in both cases of the two schemes. It implies that most of deadline misses would occur by blocked OLTP transactions. Remember that the number of concurrent OLTP transactions would decrease as OLAP_ratio grows when the number of total transactions is fixed. In conclusion, increasing OLAP_ratio could lower deadline_missed_ratio by decreasing the number of concurrent OLTP transactions. Fig. 8 (b) show that, as expected, deadline_missed_ratio decreases as OLAP_ratio increases in both cases of the two schemes. The slight difference between the two schemes deadline_missed_ratio here could be explained similarly to the case of Fig. 8 (a). 6. CONCLUSIONS AND FUTURE WORKS In this paper, we proposed new algorithm so called COB for concurrent view maintenance in DWH environment. COB maintains one data version (i.e. VODB) for OLTP transactions and the other version (i.e. VDW) for read-only OLAP transactions. Data value in VDW could be updated only by maintenance transactions which are generated by scheduler and executed concurrently. Some rules for preserving data consistency during the view maintenance are also introduced in this paper. By allowing concurrent execution of multiple maintenance transactions, COB could improve freshness of data and reduce the number of deadline misses. Because OLAP transactions usually require excessively long execution time, traditional two version-based techniques suffer from low throughput due to repeated aborts of the transactions. Therefore, they cannot be directly applied to soft real-time OLAP applications such as fraud-detection or stock market analysis. On the contrary, COB could meet deadline constraints for OLAP transactions because COB never aborts read-only OLAP transactions. Simulation results have indicated that COB has adequate characteristics to be applied to soft real-time OLAP transaction management. REFERENCES 1. S. Chaudhuri and U. Dayal, An overview of data warehousing and OLAP technology, ACM SIGMOD Record, Vol. 26, 1997, pp J. Widom, Research problems in data warehousing, in Proceedings of the 4th International Conference on Information and Knowledge Management, 1995, pp W. H. Inmon, Building the Data Warehouse, 2nd ed., Wiley Computer Publishing, J. Chen and E. A. Rundensteiner, TxnWrap: a transactional approach to data warehouse maintenance, Technical Report No. WPI-CS-TR-00-26, Worcester Polytechnic Institute, S. Kulkarni and M. K. Mohania, Concurrent maintenance of views using multiple versions, in Proceedings of the International Database Engineering and Applications Symposium, 1999, pp

17 VIEW MAINTENANCE SCHEME FOR DATA WAREHOUSE SYSTEMS D. Quass and J. Widom, On-line warehouse view maintenance, ACM SIGMOD Record, Vol. 26, 1997, pp A. Vavouras, S. Gatziu, and K. R. Dittrich, Modeling and executing the data warehouse refreshment process, in Proceedings of the International Symposium on Database Applications in Non-Traditional Environments, 1999, pp Y. Zhuge, H. G. Molina, J. Hammer, and J. Widom, View maintenance in a warehousing environment, ACM SIGMOD Record, Vol. 24, 1995, pp P. A. Bernstein, V. Hadzilacos, and N. Goodman, Concurrency Control and Recovery in Database Systems, Addison-Wesley, D. Theodoratos and M. Bouzeghoub, Data currency quality factors in data warehouse design, in Proceedings of the International Workshop on Design and Management of Data Warehouses, 1999, pp M. Teschke and A. Ulbrich, Concurrent warehouse maintenance without compromising session consistency, in Proceedings of the 9th International Conference of Database and Expert Systems Applications, 1998, pp H. Schwetman, CSIM Users Guide for Use with CSIM Revision 16, Microelectronics and Computer Technology Corporation, Texas, R. Agrawal, M. J. Carey, and M. Livny, Concurrency control performance modeling: alternatives and implications, ACM Transactions on Database Systems, Vol. 12, 1987, pp Namgyu Kim ( ) received the B.S. degree in Computer Engineering from Seoul National University in 1998 and Ph.D. degree in Management Engineering from Korea Advanced Institute of Science and Technology (KAIST) in He has been working for Kookmin University since then. His current research interests include enterprise data modeling and data mining. Songchun Moon ( ) received his Ph.D. degree in Computer Science from the University of Illinois at Urbana- Champaign in He has been working for Korea Advanced Institute of Science and Technology (KAIST) since then. He has developed a multi-user relational database management system, IM, which is the first prototype in Korea in 1990 and a distributed database management system, DIME, first ever in Korea in His research interests include enterprise data modeling, security, privacy, piracy, and data warehousing.