3. SCHEMA MIGRATION AND QUERY MAPPING FRAMEWORK

Transcription

1 35 3. SCHEMA MIGRATION AND QUERY MAPPING FRAMEWORK 3.1 Prologue Information and Communication Technology (ICT) trends that are emerging and millions of users depending on the digital communication is leading to significant rise in the quantum of data being stores in the systems. Review of social media applications [40], emphasize the fact that the quantum of user-generated data that is fostered in the process is turning out to be very high. RDMS has been one of the most effective database management systems that has catered to the requirements of managing the information systems data in structured manner. The efficacy of RDBMS solutions are about the simplicity of the system, despite of its robust performance and compatibility in terms of data management and processing. However, taking in to consideration the emergence of the business intelligence solutions, and its need to focus upon the gathering and processing of unstructured data, RDBMS solutions are not able to address the issues. The limitations that are envisaged in the RDBMS towards processing huge volumes of real-time unstructured data have led to the emergence of No-SQL [41]. The increasing demand for BI solutions implementation and many companies focusing on adapting analytics for generating insights in their business model, the need for robust information systems with scalability, ability for real-time processing of huge volumes of both structured and unstructured data is turning out to be a key factor. Data portioning and data replication are the main characteristic difference that differentiates NoSQL to RDBMS. Though, there are many BI developers are opting for NoSQL solutions for relative impact, still many companies are only depending on RDBMS solutions due to the migration process related issues.

2 36 Migrating huge volumes of data is one of the complex issues that has to be addressed in the RDBMS solution as scalability in RDBMS solutions are limited. The other key challenge that constitutes the process is the data redundancy system in RDBMS, which also plays a vital role in NoSQL models. It is very essential that the new data model has to be semantically identical to the actual data model, ensuring that there is no loss or distortion of data. The associated costs of data model development and migration of data from one platform to the other is also one of the perceived challenges in the process of data modeling. Review of literature provides solutions in terms of semi-automatic migration process or some adjustments carried out to the existing relational models to ensure that existing system shall deliver essential support in handling large volumes of data [42]. Despite of such developments and solutions proposed, there are significant gaps that are envisaged in the process. In this paper, the proposed solution is about Schema Migration and Mapping Framework, that enables in automatic, transparent way of data and model migration to NoSQL database system from an RDBMS system. The key objective of the proposed model is about retaining the semantics of actual database in terms of data modeling, but also in terms of how the codification is done to query the database. The framework that is adapted offers abstract layers which allow the software application to handle the data in NoSQL model more transparently, without any need for changing the existing query structure. The Schema Migration and Mapping Framework supports such solution by process the entire data migration by ensuring that the actual database is maintained and the data is stored without much structural changes, in a NoSQL model. The key SQL functions like insert, update and delete, that are captured in the actual application is capture in the framework and shall be converted in to requests in the NoSQL database. The generated results are again converted to standard SQL format and

3 37 the response to query is processed. The proposed model shall ensure that the efficacy of the existing semantics for the system is managed without much structural changes. The evaluation of the model is carried out both in qualitative and quantitative perspective. From the qualitative testing, the efficacy of the proposed system for retaining the same semantic data modeling despite migration to NoSQL solutions can be envisaged. In an case scenario of transition of data and the data modeling from RDBMS system to NoSQL systems are carried out and the various SQL operations that play a vital role in the process is executed using the Schema Migration and Mapping Framework, and the results depict that the results are identical to the actual process. For the quantitative evaluation, the objective is to evaluate the kind of overhead that is presumably caused by the introduction of Schema Migration and Mapping Framework layers. An experimental set-up with large quantum of data is considered for the analysis. By evaluating the response time for the query processing in both the actual RDBMS solution and also as per the framework model has been considered. From the test results it is imperative that the problems that are caused by the Schema Migration and Mapping Framework is significant when the volume of data involved in the operation is small, but with the rising queries, the output in terms of data processed from the solutions has raised to the desired levels, reducing the overhead time. 3.2 Grounding Research There are many case scenarios of comparisons pertaining to the data models and also the NoSQL models [41]. Some of the key differences amid the RDMBS and NoSQL are about method of storing and retrieving of information using the queries. In RDBMS, there is structured schema that is used for storing the data, whereas, in the case of the NoSQL models, there is hardly any such pre-defined schema, thus leading to the fact that attributes of a record shall not be same in both the cases. In [43], the authors have performed comparison among the various key models of NoSQL, like Key-Value [43],

4 38 which are used in systems like Redis, Riak, Project Voldemort; The document oriented [44], that is used in NoSQL CouchDB, and SimpleDB systems that are adapted in some major systems like the Google Big Table and Cassandra systems. Among the key findings from the evaluation, the limitation of RDBMS in terms of handling the database with the rising volume of data is a key constraint. In the recent past, some of the studies were also focusing on handling the large volume of data in RDBMS solutions. One of the methods proposed is about creating an abstraction layer in between SQL and NoSQL Databases [45]. The authors of the study [45] emphasize that while the results for certain kind of data sets are processed better by RDBMS, and the other kind of sets are processed better in NoSQL. Analysis of the request and the output is analyzed to find the suitable model for every kind of request and such process could be much time consuming process. In the proposed model, the original structure of database is stored in NoSQL database, and the queries that are arising from the applications are converted during the runtime to NoSQL, without any necessity of human intervention. In [46], the authors present a model for addressing the issues related to unstructured data processing, in which a framework is provided to integrate the data from varied sources. The model has the structured data managed in a relational database, while the unstructured data is managed in varied files even with different types like CSV, XML, Doc. etc. and the model has intermediate module connected to the application system for processing the queries. Verification of data sources and consolidation of the results are some of the key developments forwarded back to the application. The other significant process depicted in the model is about interceptor/mediator protocol/controller used for regulating the whole process.

5 39 The interceptor performs the task of sidetracking in the information flow amidst the application and the data source, ensuring that application is not modified in the instance of changes to data source. However, the proposal is about complete migration of the system from relational database model to NoSQL models, and in terms of managing only one data source, and in terms of eliminating the need for verifying the data source; Also, such a process shall support in keeping the application unchanged and also in terms of considering the data that is stored in relational model. Towards achieving such goals, the key challenge is to ensure complete abstraction of the relational model towards NoSQL model, and due to the differences in the respective differences in the structures, the ways of using it as an abstraction layer is quite feasible. 3.3 Schema Migration Layer Horizontal scaling is a major limitation with the RDBMS models, despite the fact that it gets well-structure data and also shall support in query processing. Hence, it can be stated that in some conditions, the traditional forms of such data is about Relational Database schema's and then migrate into autonomously NoSQL Database Reverting Normal Forms and Row-key Selection It is imperative from the review that there is hardly any model in terms of standard column-oriented NoSQL database for creating the NoSQL structured schema as a column-oriented schema. The column-oriented NoSQL database shall only constitute DDI design concepts like the duplication, de-normalization, and smart secrets in order terms etc. The process of de-normalization and replication are adapted for re-aggregating related table of SQL in to NoSQL. Despite that some information could be significantly duplicate or redundant when the data is reviewed in NoSQL table.

6 40 In the process, in every row a key that is unique for identification has to be selected, but there is hardly any need for cross-table query in the NoSQL database, as predominantly all the essential SQL tables are merged in to one big table. The proposed table is certainly a traditional design that allows the articles to have majority of the rows, and the table supports in having few rows and majority of the articles on the other hand. Proposed method shall support in building the NoSQL database as there tall-narrow design pattern, as the RDBMS solutions can split in terms of line boundary. In order to achieve the outcome of tall-narrow design, the relativity and the factor of keeping the cardinality at the high levels is important. The line type defined in NoSQL shall be concatenation of some of primary factors proposed in the model, as some of the inputs could be acknowledged each row of SQL tables. In the process of table integration to SQL tables, if Table A is related to Table B with a foreign key relativity, one SQL table is created as Table. However, in the instance where Table B is also related to Table C, in such instances, the formation of SQL table shall be with Table C as the key row reference. Predominantly, the line key shall be concatenation of various major keys of multiple tables in SQL tables in the form of chained system. In terms of traditional schema design, the tall-narrow design, the pattern allows the table has only some columns but many rows. In the model of flat-wide design pattern allow the tables to have few rows but many columns, as the RDDMS that is considered in the paper has the scope for splitting at the row boundary, the model depicted shall work on crating NoSQL database that has tall-narrow pattern of design. For accomplishing such outcome, the selected row must ensure that there is high level of cardinality. There is need for concatenation of many primary keys in the NoSQL table as predominantly the primary keys are used for identifying each row in the SQL tables.

7 41 Similarly, in the case of relationship among the vivid range of SQL tables are also taken in to consideration. In an illustrative scenario, there is relativity with a foreign key between two SQL tables Table A and Table B, and Table B is also related to Table C. In such conditions, the primary key of the corresponding table Table C shall have higher cardinality, and thus the Primary key of Table C s shall be selected as the row key in NoSQL table. If there are more than one table is identified with the same relationship, in such conditions, the row key shall be conglomerate of SQL s primary keys. Figure 3.1: Schema de-normalization

8 42 In the figure 3.1, the inputs depict the illustrative scenario of proposed schema of de-normalization pertaining to DDI design principle. In the case scenario illustrated, Relational Database called Hush from Hbase is constituted with four SQL tables 1) Users 2) URI, 3) CLICK and SHORTURI. CLICK table is related to SHORTURI which is related to USER and URI tables. And in such relativity, as per the aforesaid conditions, the key for selected row concatenation of USER and URI tables which are considered as PK-1 and PK-2 as primary keys as depicted in Figure 3.2., however the crux is, as all the four tables are related, respective columns in all four tables are aggregated as one NoSQL table Schema Migration In Figure 3.2, the proposed schema migration that is depicted is autonomous, with only the NoSQL schema that automatically works on the SQL schema analysis. Information schema is one classified table in which RDBMS stores all the SQL schemas, and hence it s much easier to parse and extract primary and foreign keys of each table from the relational database solution.

9 43 Figure 3.2: The flowchart of the proposed schema migration The process flow of schema migration: 1) For conducting schema analysis of SQL, one table is selected from RDBMS. 2) In order to store the results of SQL schema analysis, a linked list id generated, with the main functionality of the list to calculate the chained relationship. 3) As same column names might exist in varied tables, the proposed migration should comprise each table name as one column name for clear identification. 4) The PK (Primary Key) of a table is stored as Row Key candidate in a NoSQL table. 5) Detection of the existence of any Foreign Key (FK) shall be determined by the proposed migration solution.

10 44 a) If true, go to Step 6. b) If false, go to Step 9. 6) To aggregate the related table, foreign key (FK) is stored 7) Depending on the FK, the related tables in the scenario are also added to the link list, whilst incrementing the list length by one. 8) The related table is selected as the next table and the process is repeated till the SQL schema analysis ensures that all the tables in the link list are added. 9) In the case of the proposed migration, any deviation in terms of analysis is inspected. a) If true, go to Step 2. b) If false, go to Step ) In ref. to section III.A, the level of migration determines the Row Key on the basis of the Primary Keys that are stored. 11) In reference to Section III.A, all columns are aggregated into one NoSQL table by the proposed migration Data Mapping Module In this module the emphasis is on abstraction layer amid the NoSQL DBMS and the application modules. The key goal of the proposed module is to focus on seamless range of database migration which can prevent any kind of change in the application code whilst changing the data module adapted. Also, the application developers shall create the queries for the RDBMS model, but the data shall be fetched from a NoSQL database, ensuring that advantages for performance and scalability that is managed in the system.

11 45 Figure 3.3: Data Mapping Module Working Model Among the preliminary tasks considered in the module, intercepting the queries issued by the application to the DBMS and redirects them to the suitable NoSQL DBMS. A sub-module called Mediator is developed using the proxy of Relational RDBMS [48], with an open source toll which shall perform the Relational DBMS server and client application. In terms of using the proxy using the LUA language [49] to conduct the process of data manipulation, however, the inputs comes with certain rang of predefined functionalities for intercepting the queries for refining the results and also in terms of sending the signals pertaining to queries that are performed effectively or the ones with errors. Using LUA, relational DBMS proxy has been changed in order to ensure that the operations are rightly intercepted and is forwarded to a second module that is responsible for query conversion. LauSoap library [50] shall be used for implementing the communication amdist mediator and the request conversion using SOAP (Simple Object Access Protocol) and the XML files that shall constitute the information pertaining to operations. Covert is the conversion sub-module that is opted in terms of addressing all the mediator requests that are moved to the relational database and works on converting them

12 46 to queries with NoSQL. Using the Java language, this sub-module was developed as WebService, and it shall not wait for any requests that are sent by Mediator as XML files. Among the XML files that are used as request communication, the first attribute of xmlns indicates that the class of WebService is to be adapted as query interceptor. Also, two of the key parameters that are essential as query and querytype indicate the intercepted request and its type. QueryType as a parameter is evaluated by Convert for identifying the next procedure that could be executed as per the function that has to be incorporated. For each of the function like Insert, Update, Delete, Modify, there shall be a corresponding method which shall be responsible to perform the operation for NoSQL Database and reverting the results from the executed function. Despite the fact that there could be some deviations, for majority of the instance of functions, the process described below is the pattern adapted.

13 47 Extracting Information about Query: The process evaluated the parameters of the query to ensure that the details of SQL operations like the tables, attributes andother such relevant information has been gatheredadditionally apart from the criteria that is used in the clauses and the method is adapted using Javb library JSQL Parser [51]. The key factor that differentiates the individual methods is about the information that each one exploits. For instance, in the case of a corresponding operation to SQL select methods has to focus treating the involved tables, and the necessary functions towards sorting and grouping. But in the case of the insert operation, the function must address the attributes and the tables that are involved in the operations. Generating and Implementing Suitable Operation in NoSQL: In this step, the correspondence is more about translating the SQL operations towards its equivalent NoSQL factors. Also, the translation process is completely based on official NoSQL from to the operation mappings in SQL [52]. Metadata collection stores the relationship of the original database to its correspondence in NoSQL Database, which is very important. But in the end the new operation is only executed on NoSQL and the results are processed in furtherance. Mapping Return Results: The results returned by NoSQL shall be sent to the Mediator as it is accountable for forwarding the results to the application. In the operational process, the header is built with inputs like the identification tables, attributes, elements that are related to the results. In the sequential process, each of the records that are returned by NoSQL is mapped, after the header is built. A XML is built in pertinent to the proposed solution and is sent to Mediator by the Covert. An illustrative XML built is depicted below. The proposed method of mapping also supports for SQL nested transactions like the Delete process, where the items to be removed from the process purely depends on the Select. The proposed model of Schema Migration Model intrigues the method that is more suitable to most external operating.

14 48 In the process of Parser execution, the nested transaction is identified in terms of corresponding methods that has to be adapted for the operation which is also triggered to completely resolve the issue. However, the necessary information for conducting the external operation is gathered from the process. In the case of Delete function, parser execution maps and identifies the select function and invokes the method which is essential to handle such conditions. Once the process is accomplished, the results from the process are directed to the method Delete. The proposed scheme use recursive calls for processing. However, one of the limitations in process is about the hardware resources availability for the process. In the final step, Mediator handles the XML dispersed by Convert, and it is accountable for converting the XML inputs to the RDBMS structure using the proxy, to ensure suitable structure as depicted in the query.

15 Experimental setup and Evaluation In order to assess the RDBMS-TO-NOSQL Conversion by schema migration and mapping, focusing on two types of analysis: qualitative and quantitative. While in the qualitative evaluation our goal is to present a proof of concept by showing the Schema Migration and Mapping Framework execution in practice, in the quantitative one we aim to verify whether the use of NoSQL, with our framework, leverages the system performance Experimental Setup Two relational databases are used in the analysis ( See figure 3.4). The First one is gathered from W3Schools a website of a reputed web developer. Despite the fact that there is simple structure that is adapted, still the information that is captured in terms of characteristics are turning out to be quite different to the ones that are typically adapted in the varied applications, relationships and vivid range of data types. The second model is more of a simple database model which collects, stores from the users posts in Twitter, a popular social networking site. Such data has high volume of data stored, which could help in achieving quantitative assessment of Schema Migration and Mapping Framework. The implementation of the proposed model is done using java 8 on a computer with i5 processor, 4GB ram. The open source data storage system MongoDB is used as target NoSQL database and JSQLParser was used to parse the SQL queries. The RDBMS models HBase and Mysql database schemas were used as input to the proposed model.

16 50 a. W3schools b. Twitter Figure 3.4: Database models used in the evaluation. Using the Relational RDMS both the databases are implemented, and also an application has been developed using Java for each of the database, with the clear purpose of executing varied set of SQL operations. For every arrangement of application/database, another version is developed using Schema Migration and Mapping Framework, in order to perform the entire range of migration towards the relational database to NoSQL.

17 51 In the modeling of the system, Schema Migration and Mapping Framework Scenario adapting, the Schema Migration and Mapping Framework was also developed for capturing the operations that are requested by the applications and also NoSQL. Hence, using the process of quantitative and qualitative comparisons for the results is achieved using the scenario. Despite the fact that several SQL operations were tested using the development process, because of space constraints, the emphasis in the present model is about the implementation of the functions as depicted in Table.2 But in terms of operations that are used for various set of operators certain factors like MAX functions, ORDERBY are some of the critical factors that are adapted.

18 52 Table3.1: Operations performed in the evaluations

19 Qualitative Evaluation The key purpose of the analysis is to ensure that the use of Schema Migration and Mapping Framework that requires the functionality of the applications for performing numerous SQL operations. The tests were conducted in two scenarios as discussed below. To ensure that the results have significant inputs, the quantum of records that were selected or affected by the functions has also been changed. Scenario 1: Operational Queries: In the case of every operation performed, the results were depicted in the plain text files. And sequentially the applications that identify the differences between files that are executed are taken in to account and the result was stored in a corresponding file. The queries that are depicted in table 3.1 are considered for tests and the inputs for the same are empty, which means that no anomaly transactions has occurred and significantly Schema Migration and Mapping Framework had sorted in an effective manner. Scenario-2: Database States are changed by Operations: Towards execution of an operation for the changes in the database state, a dump of tables or collections and the results that are stored in the text files, as per the operation performed has been chosen for purview. Also, in terms of evaluated differences, as per the result files, no significant deviation is found, which emphasize that the Schema Migration and Mapping Framework is functioning as per the expected processes.

20 Qualitative Assessment on CMS Systems In terms of proposed autonomous RDBMS to-nosql schema migration which is assessed using two of the popular content management system WordPress and Joomla. Initially, we migrate the Relational Database schemas of Joomal and WordPress to NoSQL Database directly using the Schema Migration and utilizing Mapping Framework for conducting cross-table query. In the other way, focusing on migrating the Relational Database Schemas using the proposed schema migration mechanism for cross table querying is also achieving using the mapping module. In the case of each CMS, an effective Big Data tool is used to generate close to 10 million transactions for analysis. Figure 3.5: The performance comparison over the WordPress CMS

21 WordPress WordPress is predominantly used to develop websites, and as per reports on the active websites in CMS form; around 23% of the existing top websites are on WordPress themes. WordPress s Relational Database is very much is constituted with 11 SQL tables. After conducting the schema analysis, WordPress s SQL tables are classified in to two groups, with one group comprising 3 tables and the other with 6 tables. The other 2 tables from the classified eleven are independent, as there is no foreign key existence in such tables. In the figure 3.5, the performance comparison over the CMS WordPress themes; From the results, it is imperative that when data size increases, both the mechanisms require more time to provide results to the query. But the proposed mechanism requires much lesser time for the data size is around 10 million transactions. This clearly depicts that in the proposed solution, there is more than 45% of query time savings, on an average. Figure3.6: The performance comparison over the Joomla CMS

22 Joomla Joomla is the other CMS used for publishing the contents in the website and is second most widely used CMS system. For the chosen tests, Joomla s Relational Database has more SQL tables than the WordPress, with total of 34 tables considered for the study. From the process of SQL schema analysis, Joomla s SQL tables are categorized in to three groups. First group with 17 SQL tables, thus with Relational Database has complex kind of relationships. Figure 3.6 reflects upon the performance comparison of Joomla over CMS. In terms of WordPress s results, two mechanisms require more time for completion of the query as and how the data size increases. In terms of Joomla based tables evaluation, the proposed mechanism was able to save around 48% of the query time on an average Schema Migration Analysis CMS are popular internet system for publishing the web contents, and some of the extended functionalities are also feasible in the system. Despite such developments, the fact is that majority of the CMS sites are dependent on traditional Relational Databases. As NoSQL Database offers better range of scalability and support for parallel processing for handling the big data solutions. Considering such factors, in this paper, the emphasis is on autonomous RDBMS-to-NoSQL schema levels of de-normalization. The key outcome from the proposed model is that, current CMS sites need not redesign their database schemas to handle NoSQL system. The test results that are derived for the process depicts the query time has reduced significantly by the range of 45%-48% for query time response. In the future works the emphasis shall be on dividing the columns to different column families.

23 Quantitative Evaluation In the quantitative evaluation, the efficiency of the proposed model is assessed, by evaluating whether the model is beneficial in terms of performance compared to the traditional RDBMS, even in the case of defining a layer between the application and NoSQL Database. From the set of experimental study that is carried out to evaluate the runtime of various SQL transactions using the following scenarios. In Table 2, the inputs pertaining to SQL functions that are considered in the study has been depicted, and to make the test results more emphatic, the quantum of data processed for each of the function is also varied. In the processing of queries, for every query the volume of records that has to be returned by the system after processing the query has been altered to test the efficacy of proposed model. Data insertion to the application database has been carried out to ensure that the data is retrieved from the process, and the similar strategy is used for UPDATE and DELETE functions too. Even for the INSERT data, the quantum of data to be inserted has undergone change. The time of execution has been taken in to count from the moment the application performed the request, to the levels of the results were received by the application system. The inputs and measurements has been considered using the Relational DBMS Scenario, and the model of Schema Migration and Mapping Framework Scenario (the time estimations were till the response from Schema Migration and Mapping Framework). Each of the operations has been performed 10 times and the average of the performance has been given as the estimated execution times. The tests were carried out using an computation system with the configuration of Intel Core i3 2.5GHz, 4GB RAM and a 500GB disk space. Operating system of the system used for analysis is Ubuntu bit.

24 58 The results of the analysis are present in Figure 3.7. The log scale is adapted as execution time in y-axis and the results pertaining to tests of four select operations and the one for each changing operation is depicted below. (a) Select1 (b) Select2 (c) Select3 (d) Select4

25 59 (e) Delete1 (f) Insert1 (g) Update1 Figure 3.7: Execution time of different operations From the inputs evaluated, it is imperative that the selected functions, Relational DBMS responds in much lesser time for low data volumes, rather than the proposed model. But with the rising volumes of total number of records to be retrieved increases, the response time for the process also increases. In a peak response time of RDBMS, there is excessive use of CPU to deal with high volumes of data. RDBMS starts a new processing unit resulting in addressing the issues pertaining to synchronization among distinct CPUs resulting in substantially increasing the response time.

26 60 The results depict that with the rising overhead which is generated by database abstraction layer, the model of Schema Migration and Mapping Framework shall be more efficient in terms of relational database as and how the data volume rise. However, taking in to account the distributed processing features of NoSQL, which is not explored in this study, there could be certain overridden factors. In terms of analyzing the plots pertaining to the changing database state, many of them have recorded much lower response time with the proposed model compared to the execution in RDBMS. The primary reason that could be attributed to such behavior is that in NoSQL Databases, there is absence of integrity constraints. Some advancements in the modeling of Schema Migration and Mapping Framework shall be able to address the integrity constraints that are envisaged in the process for the database.