Combining Relational and Semi-Structured Databases for an Inquiry Application

Transcription

1 Combiig Relatioal ad Semi-Structured Databases for a Iquiry Applicatio Marius Ebel ad Marti Huli Uiversity of Applied Scieces Ravesburg-Weigarte, Weigarte, Germay {ebelma,huli}@hs-weigarte.de Abstract. The popularity of NoSQL databases keeps growig ad more compaies have bee movig away from relatioal databases to orelatioal NoSQL databases. I this paper, the partitioig of a relatioal data model of a iquiry system ito semi-structured ad relatioal data for storage i both SQL ad NoSQL databases is show. Furthermore, the CAP theorem will be applied to categorize the storig of the correct parts of the model ito their correspodig databases. As a result of these reorgaizatios ad the itroductio of additioal histogram data, overall performace improvemets of about 93% to 98% are achieved. Keywords: NoSQL, Documet Stores, MogoDB, semi-structured data, iquiry system Itroductio Not Oly SQL (NoSQL) reitroduced by Evas i 2009 [7] stads for a variety of ew data stores which ow are gaiig populatio o the market [0]. Log time fully relatioal data stores have bee sufficiet for most purposes. But sice compaies ad orgaizatios have started to collect huge amouts of data, such as customer, sales ad other data for further aalysis, several types of orelatioal data stores are preferred over the relatioal oes [0]. Traditioal SQL databases come with the eed of a fixed schema orgaized i tables, colums ad rows, which caot hadle the chaged eeds for today s database applicatios. NoSQL does ot ecessarily mea schema-free. There are also NoSQL databases for structured data like Google s Bigtable [4]. Amog the schema-free NoSQL databases there are umerous data stores like the key-value-stores Redis [] ad the documet stores like Amazo SimpleDB [5], Apache CouchDB ad MogoDB. NoSQL data stores themselves provide a variety of sophisticated techiques like MapReduce [6] ad better mechaisms for horizotal scalability [3] amog others. NoSQL documet stores form oly oe group of this large variety of NoSQL data stores. These databases store ad orgaize data as collectios of documets, rather tha as structured tables with uiformsized fields for each record. With these databases, users ca add ay umber of fields of ay legth to a documet [0, p. 3]. They orgaize data i as the ame idicates documets. Documets are treated as objects with dyamic properties, which meas that a

2 2 Marius Ebel, Marti Huli documet ca be iterpreted as a dyamic list of key-value pairs. The approach described i this paper makes use of the absece of schemas i documet-orieted NoSQL databases to store dyamic, semi-structured data. SQL databases mostly use proprietary coectio drivers to hadle commuicatio with the server, where data trasfer is just a black box. NoSQL databases are differet i this case. May of them use Represetatioal State Trasfer (REST) [8,4] where ofte a commuity-developed coectio driver has to be used (Redis, Apache CouchDB), but some also come with a proprietary coectio driver (MogoDB). I most projects there is either o time for developig a coectio driver or it is usafe to rely o commuity projects. Therefore, the presece of a proprietary coectio driver ca be a very importat factor for choosig the appropriate NoSQL database. The approach to use semi-structured data structures is basically ot ew. For istace, the database system CDS/ISIS is a type of semi-structured NoSQL database, which is used sice the 980 s by a vast amout of academic libraries [3]. But the way of usig semi-structured data described i the followig sectio focuses o reorgaizig previously relatioal data ito semi-structured record sets ad o the optimizatio of computatioal effort of statistical data. Furthermore, we will show a approach to affect data orgaizatio through dyamic semi-structured data ad the subsequet queryig, which is also applied i the iquiry system INKIDU. 2 Methodology The use case of the methods described i the sequel is the olie iquiry platform INKIDU. INKIDU provides the user with the ability to desig questioaires, which ca be statistically evaluated after completig the iquiry. Questioaires i INKIDU are user-defied sequeces of questios. Each questio ca be of a differet type such as ratig questios, sigle ad multiple choice questios, free text questios, etc. After submittig a questioaire, the submitted data has to be stored for further result aalysis. A questioaire therefore has two factors affectig the possible amout of data, which ca be produced by each user: The umber of questios ad for each questio its subsequet type. The effective structure ad amout of aswers of a aswer set is fially determied by the questios, which the user decided to give a aswer for. Hece, there is o way to predict the exact amout ad structure of data, which will be produced by each user. Coclusively oly a rough estimate of the structure is kow: A list of key-value-pairs, i.e. the key is the questio idetifier ad the value is the usergive aswer. Cosiderig the variable legth of such a list ad ot kowig the data type of each value, oly semi-structured data is give. I a system meat for strictly structured relatioal data, the presece of just semi-structured data leads to a coflictig situatio of storig a set of semi-structured data with a upredictable umber of fields ad data types.

3 Combiig Relatioal ad Semi-Structured Databases 3 2. Curret Data Orgaizatio The curret way of orgaizig aswer sets is show i Figure, where aswer sets are orgaized vertically. A aswer set cotaiig all of a user s aswers belogig to a questioaire is split ito tuples cosistig of the madatory primary key, the iquiry this aswer belogs to, the aswered questio, ad a field referecig the first primary key value of all aswers belogig to the same aswer set. The last field is created by the eed of makig the assigmet of aswers recostructable, e.g. for data export ad for allowig further statistical cross-aalysis. As the data model shows, the redudat referece to the iquiry is ot ecessary, but is held to elimiate the eed of a table joi ad therefore to keep query times at a reasoable level, whe e.g. retrievig all aswer sets belogig to a certai iquiry. These bottleecks such as redudat foreig keys, complex queries ad huge idex data, arise from the RDBMS eed of a fixed table schema, which requires a predefied ad therefore predictable structure of the data. iquiries id INT questios iquiry INT id INT questiooptios id INT questio INT value VARCHAR value aswersets id INT iquiry INT questio INT value VARCHAR assig INT assig Fig.. Excerpt from the curret fully relatioal data model of INKIDU beig reorgaized. I order to demostrate the curret data orgaizatio ad the approaches described i the followig, a questioaire with oe ratig questio ad oe multiple choice questio with three optios is take as a referece example. There are two aswer sets to be show i differet maers of data orgaizatio: (ratig=, multiple choice=[optio, optio 2]) (ratig=4, multiple choice=[optio, optio 2, optio 3]) Applyig the fully relatioal approach o the referece example data, aswer sets, show i Table 2., are produced.

4 4 Marius Ebel, Marti Huli id iquiry questio aswers assig remarks ratig value ratig value optio optio optio optio optio 3 Table. Aswer sets which result from storig the above described example data i the fully relatioal data model. The values of the colum aswers for optio 3 are foreig keys referecig record sets of the the table questiooptios. 2.2 Orgaizig Aswer Sets I the followig, the approach of reorgaizig the above data structure is described. With the emergece of schema-free databases that do ot deped o a predefied structure of data records, the data reorgaizatio ca be doe ot oly much more ituitively, but also more efficietly. This meas that the costructio of the data records requires less effort such as data redudacies ad idex data ad produces a structure which is easy to read for machies ad humas as well. Semi-structured data is ofte explaied as schemaless or self-describig, terms that idicate that there is o separate descriptio of the type or structure of data. Typically, whe we store or program with a piece of data, we first describe the structure (type, schema) of that data ad the create istaces of that type (or populate) the schema. I semi-structured data we directly describe the data usig a simple sytax [, p. ]. I this case we use JavaScript Object Notatio (JSON) to describe a aswer data set, e.g. i terms of NoSQL a documet such as the followig record sets, which result from the referece example data. { // first aswer set: // (ratig=, multiple choice=[optio, optio 2]) iquiry: 8722, data : { : : [ , ] } } { // secod aswer set: // (ratig=4, multiple choice=[optio, optio 2, optio 3]) iquiry: 8722, data : {

5 Combiig Relatioal ad Semi-Structured Databases 5 } } : : [ , , ] The use of JSON otatio is proposed by MogoDB, which is used for this project. Biary JSON (BSON) is the format used by MogoDB to store documets ad to trasfer them via etwork [2,3]. As show above a aswer set, formerly to be divided ito several records, ca ow be grouped ito oe sigle, semi-structured record, which uses a key-value list mappig questio ids to the user-give aswers. Hece, a covetio of a basic loose structure of a aswer set ca be decribed as the followig: a referece to the iquiry the user participated ad key-value pairs of the data mappig as described before. The advatage of a o-structured NoSQL-Database is ow clearly visible: All the data belogig to the same aswer set ca be kept i oe sigle record that ca be easily queried by the iquiry field ad still does ot eed to fulfill the requiremet of a predefied umber of fields or data types. This way of storig data simplifies data maagemet ad is easier to uderstad. 2.3 Data Codesatio The approach described above ca be refied to group all aswer sets ito a sigle record, i.e. to codese data from multiple record sets ito a sigle oe. The referece example data would result i a record set show below. { } iquiry: 8722, data : [ { // first aswer set: // (ratig=, multiple choice=[optio, optio 2]) : : [ , ] }, { // secod aswer set: // (ratig=4, multiple choice=[optio, optio 2, optio 3]) : : [ , , ] } ] Such a codesed record set ca grow very large, which is a disadvatage especially for real-world applicatios. It icreases the memory cosumptio of aswer processig outside the database resultig i severe scalability problems

6 6 Marius Ebel, Marti Huli for the whole applicatio. But with smaller amouts of data or less strict scalability requiremets, such approach is still a better way of data modelig with comfortable hadlig. Databases use B-Trees for idexig data [5]. The previously fully relatioal data model required three fields per aswer record set (i.e. for every aswer to every questio) to be idexed i order to keep query times at a reasoable level. The B-Trees idexig oly the aswer sets of INKIDU cosume more storage space tha the data itself. Usig the semi-structured approach the idex data is miimized, because oly oe field per collectio (iquiry) ad less record sets have to be idexed. This is true whether the aswer sets are stored i multiple records or are codesed to oe record. 2.4 Histograms The price to pay for less idex data ad easier queries is that some fuctios formerly used i SQL queries are t available aymore. Examples for some of these fuctios are miimum/maximum, average, stadard deviatio, etc. Keepig i mid that a iquiry applicatio also icludes the statistical aalysis of the retrieved aswers, the absece of these fuctios is fatal. These fuctios have to be regaied ad ufortuately have to be implemeted outside the database. Usig the situatio to our beefit, the computatio of statistical data ca be doe more efficietly by itroducig aother semi-structured dyamic data structure, which is described i the followig. For better uderstadig the formula of the arithmetic mea has to be cosidered first: x aive = i= x i = x + x 2 + x x i This aive formula implies that every elemet x X has to be processed for calculatig x aive. This meas that the computatios of the arithmetic mea ad the stadard deviatio result i severe scalability problems for larger or growig amouts of aswers. The computatioal effort for e.g. the average grows liearly by O() with as the umber of record sets. Therefore these calculatios have to be optimized, which is ow doe by storig additioal data. The liear growth of computatioal effort by the umber of record sets is especially i the described sceario a bottleeck. To tackle the challege of reducig the computatioal effort, data is reorgaized to histograms. Per defiitio, a histogram is a graphical represetatio of the distributio of data. I the above described sceario, a histogram represets the distributio of aswers to a sigle questio holdig the absolute frequecy of every optio. A optio ca be either a foreig key referecig a questio optio of a multiple/sigle choice questio or a ratig value of a ratig questio. Hece, the histogram ca be represeted as a set of optio-frequecy pairs, which ca be used to do a much more efficiet calculatio of the arithmetic mea ad the stadard deviatio. The followig schema shows a represetatio of a histogram, which ca be easily projected oto a semi-structured NoSQL documet. I this schema ()

7 Combiig Relatioal ad Semi-Structured Databases 7 o stads for optio ad f stads for frequecy, where a list of these optiofrequecy mappigs is the actual data part of a histogram, H iquiry, questio, without the iquiry ad questio iformatio. o, f o 2, f 2 data(h iquiry, questio ) = o 3, f 3.,. o k, f k e.g. =, 594 2, 453 3, 203 4, 34 5, 43 As show i Eqs. 2+3, every optio is weighted by its absolute frequecy, which meas that the set of data is ot iterated over every record set, but over every questio optio istead. This makes the computatioal effort still grow liearly by O(k) where k is the umber of available questio optios (revise followig setece), but rather by the umber of available questio aswers tha by the umber of give record sets. x hist = k i= (o i f i ) = o f + o 2 f 2 + o 3 f o k f k The correctess of Eq. 3 is give by the fact that the sum of all frequecies is equal to the umber of give aswers: k i= f i =. Now the umber of differet questio optios is a predefied umber, which is idepedet from the growig umber of icomig aswers ad therefore makes the computatioal effort predictable ad as small as possible: It is idepedet of the umber of users takig part i a questioaire. Of course, the complete computatio time for calculatig the mea remais the same. However it is moved from a time critical part of the applicatio (statistical aalysis) to a ucritical part (storig ew aswer sets). Additioally the computatioal effort is distributed over every user submittig aswers to the system, which results i shorter processig times for visualizig ad displayig statistical aalysis reports (see Sectio 3.3). The disadvatage of storig redudat frequecy values is justified by the performace gai of the statistical aalysis, a core feature of iquiry applicatios. As metioed above a documet i terms of a documet-orieted NoSQL database is a list of estable key-value pairs. The semi-structure of a histogram data set ow ca be modeled as follows; cosistig of a fixed ad a dyamic part. The fixed part is the idetifyig header, which icludes the iquiry id ad the associated questio id. The dyamic part is the actual mappig from questio optio oto absolute frequecy. (2) (3) 3 Software Architecture I this sectio the realizatio of the above cocepts is described. The applicatio is, as already metioed, the olie iquiry platform INKIDU, which allows the user to create ad evaluate user-defied questioaires.

8 8 Marius Ebel, Marti Huli iquiry questio iquiry id questio id fixed part / header data optio frequecy optio 2... frequecy... dyamic part / payload optio k frequecy Fig. 2. Structure of a semi-structured histogram record. The upper part ( header ) is the fixed part, which ca be foud i every histogram record. The lower part ( payload ) is the dyamic part, which is variable i its umber of key-value-pairs. 3. System Overview The cocept of itegratio i case of INKIDU icludes the parallel usage of a traditioal SQL database ad a NoSQL documet store. The NoSQL documet store is provided by MogoDB 2 while MySQL 3 is used as the data store of the relatioal data model. RDBMS (MySQL) NoSQL Documet Store (MogoDB) iquiries id INT questios iquiry INT id INT questiooptios id INT questio INT value VARCHAR m V(data) K(data) aswersets iquiry INT questio INT data { INT -> STRING } histograms iquiry INT questio INT data { INT -> INT } Fig. 3. Excerpt of the data model of INKIDU showig which parts belog to the relatioal ad o-relatioal data model

9 Combiig Relatioal ad Semi-Structured Databases 9 Cosistecy ad referetial itegrity are importat features i relatioal data models. With the features of mass data storage, database scalability ad high availability NoSQL databases ofte do ot provide ative support for ACID trasactios. NoSQL databases are able to offer performace ad scalability ad keep strog cosistecy out of their fuctioalities, which is a problem for several types of applicatios [0]. For istace, i case of INKIDU there are parts of the data model which do require ACID trasactios ad strog cosistecy as well as parts which do ot require this fuctioality. Therefore, as Figure 3 shows, the data model is distributed over the two differet databases. So the decisio to make it a hybrid system cosistig of relatioal ad orelatioal databases is explaied by two differet factors. The first factor is, that the approach of icreasig system performace by reorgaizig data works also o sigle server setups. Other mechaisms icreasig performace by scalig a system horizotally (e.g. load balacig) eed a multiple server setup to work. The secod factor is defied by the differet requiremets towards cosistecy ad referetial itegrity. However, there are approaches to esure cosistecy withi a o-relatioal data model [6], which for ow exceeds the resources of this project. The hierarchy cosistig of iquiries, questios, questiooptios, etc. has to be cosistet at ay time. This icludes o oe had ACID trasactios ad o the other had referetial itegrity, such that e.g. o orphaed records ca exist. This meas that the absece of foreig key checkig ad the presece of evetual cosistecy as achieved by MogoDB is ot sufficiet. Therefore MySQL is still used to store these structured data. NoSQL database maagemet systems ca store data, which chage dyamically i size much better tha relatioal oes; for istace odes ca be added dyamically to icrease horizotal scalability. Cosiderig the three guaratees cosistecy, availability ad partitio-tolerace of the CAP theorem [2,9], cosistecy ad availability are most importat for the relatioal part. Availability ad partitio-tolerace are most importat for the o-relatioal of INKIDU due to the fact that it is ot tolerable that data is ot accessible or gets lost i case of a database server ode crashig. 3.2 Implemetatio There exist a variety of differet NoSQL database cocepts ad implemetatios: key-value stores, documet stores ad extesible record stores [3]. To choose the appropriate NoSQL database maagemet system for INKIDU three key requiremets were cosidered: Extesibility: The possibility to migrate the full data model ito the NoSQL database has to be preserved. This meas the database has to be capable of storig objects of higher complexity tha histograms ad aswer sets. Idexig: Besides the uique ID of every record set a secod additioal field has to be idexed.

10 0 Marius Ebel, Marti Huli Reliability: The database itself ad the driver have to work well with PHP. Especially the driver has to be well-egieered for simple ad reliable use. The requiremets to store more complex objects ad idexig more tha oe field led to the decisio to use documet stores. I order to fid the right documet store fulfillig all of the above key requiremets Apache CouchDB ad MogoDB were ivestigated. CouchDB offers some drivers developed by commuity projects, but without support. MogoDB offers oe well developed proprietary driver, which is kow to work well with PHP. Therefore MogoDB was chose for this project. A poit ot liked to the key requiremets but still otable is the presece of atomic operatios i MogoDB. Atomic operatios allow chages to idividual values. For istace to icremet a value the modifier $ic ca be used i a update commad, $push adds a value at the ed of a array ad $pull removes it. These updates occur i place ad therefore do ot eed the overhead of a server retur trip [3]. This is especially of advatage for the updatig of histograms, where ofte oly oe sigle value has to be icreased. 3.3 Performace Results I order to measure the performace of the recostructed system the fully relatioal ad the hybrid implemetatio were compared agaist each other i two differet scearios: Firstly, the geeratio of the visualized histograms. This icludes the queryig of the histogram, the calculatio of the average ad the stadard deviatio ad the rederig of a histogram image displayed to the user. Secodly, the export of all aswer sets of a iquiry, which icludes the queryig of all aswer data belogig to a iquiry ad formattig it ito a CSV file. These scearios were chose, because they are the most affected parts of the recostructio. The test procedures were supplied with 00 differet aoymized real-life iquiries with oe to 200 questios. The iquiries itself had participats withi the rage from 0 to 000. Durig the test scearios the followig two values were measured: Query Timigs: The time eeded to query the database itself. Processig Timigs: The time the whole process eeded to complete icludig the query time. The reaso to distiguish betwee queryig ad processig timigs is that we also have to take ito accout the time cosideratio for the optimizatio of the computatio processes outside the database. Table 3.3 shows the query timigs ad Table 3.3 shows the processig timigs. The measuremets show that the average queryig time was reduced by about 30% to 95% ad the average processig time by about 93% to 98%. Histograms could have also bee realized usig a relatioal database. However all reasos of Sectio 2.2 to store aswer sets i a NoSQL database are true for histogram data as well. These data are also semi-structured ad chage

11 Combiig Relatioal ad Semi-Structured Databases i size dyamically. Therefore the overall performace improvemets are the result of the combiatio of the relatioal database with a NoSQL database. Aother possible approach - usig a object database - was ot ivestigated i this project. Histogram Queryig (µs) Mi Avg Max Relatioal Hybrid 2 24 ( 30%) 67 Data Export Queryig (µs) Mi Avg Max Relatioal Hybrid ( 95%) 879 Table 2. Query timigs for histogram ad data export queryig. Histogram Geeratio (µs) Mi Avg Max Relatioal Hybrid ( 98%) 6779 Data Export (µs) Mi Avg Max Relatioal Hybrid ( 93%) 9334 Table 3. Processig timigs for histogram geeratio ad data export. 4 Coclusio This paper illustrates a approach to distribute a fully relatioal data model ito a relatioal ad a o-relatioal part i order to deal with differet requiremets cocerig cosistecy, availability ad partitio-tolerace. Before partitioig a relatioal data model ito multiple parts, the choice of the database(s) has to be made carefully. Especially with regard to NoSQL databases, there is a large variety of differet types of data stores available. I this project, the decisio to select MogoDB as NoSQL database maagemet system has bee strogly iflueced by the presece of a reliable driver for database coectio ad atomic operatios, which provide the ability to make simple update operatios as fast as possible.

12 2 Marius Ebel, Marti Huli The data model of the INKIDU iquiry applicatio is separated i two parts: The structured part is still stored usig the relatioal DBMS MySQL, which guaratees ACID trasactio cotrol. The semi-structured part, the aswer sets of a questioaire, is stored i the documet store MogoDB. This reduces the amout of stored data by more tha 50% because three of the four idexes ca be omitted. I a secod step histograms are stored to icrease the performace of a questioaire s statistical aalysis - a core feature of a iquiry applicatio. MogoDB is used to store histogram data because they are semi-structured, too. The time the queries take to retrieve the histogram ad export data has bee reduced by about 30% ad 95%. The overall performace for the processes of geeratig histograms ad exportig data has icreased by 98% ad 93% respectively. Ackowledgemets We gratefully ackowledge Wolfgag Ertel for collaboratio ad Michel Tokic for his valuable feedback o this paper. Refereces. Abiteboul, S., Buema, P., Suciu, D.: Data o the Web: From relatios to semistructured data ad XML. Morga Kaufma Publishers Ic., Sa Fracisco, CA, USA (2000) 2. Brewer, E.A.: Towards robust distributed systems (abstract). I: Proceedigs of the ieteeth aual ACM symposium o Priciples of distributed computig. p. 7. PODC 00, ACM, New York, NY, USA (2000) 3. Cattell, R.: Scalable SQL ad NoSQL data stores. SIGMOD Record 39(4), 2 27 (200) 4. Chag, F., Dea, J., Ghemawat, S., Hsieh, W.C., Wallach, D.A., Burrows, M., Chadra, T., Fikes, A., Gruber, R.E.: Bigtable: A distributed storage system for structured data. I: Proceedigs of the 7th Coferece o USENIX Symposium o Operatig Systems Desig ad Implemetatio. vol. 7, pp (2006) 5. Comer, D.: The Ubiquitous B-Tree. ACM Computig Surveys, 2 37 (979) 6. Dea, J., Ghemawat, S., Ic, G.: MapReduce: Simplified data processig o large clusters. I: Proceedigs of the 6th Symposium o Opeartig Systems Desig & Implemetatio. USENIX Associatio (2004) 7. Evas, E.: Eric Evas Weblog html (may 2009), retrieved: Fieldig, R.T.: Architectural styles ad the desig of etwork-based software architectures. Ph.D. thesis, Uiversity of Califoria (2000) 9. Gilbert, S., Lych, N.: Brewer s cojecture ad the feasibility of cosistet, available, partitio-tolerat web services. SIGACT News 33, 5 59 (Ju 2002) 0. Leavitt, N.: Will NoSQL Databases Live Up to Their promise? Computer 43, 2 4 (February 200). Lerer, R.M.: At the Forge - Redis. Liux Joural 200(97) (Sep 200), http: // (Retrieved: )

13 Combiig Relatioal ad Semi-Structured Databases 3 2. MogoDB: BSON. retrieved: Ramalho, L.: Implemetig a Moder API for CDS/ISIS, a classic semistructured NoSQL Database. I: Todt, E. (ed.) Forum Iteracioal do Software Livre, XI Workshop sobre Software Livre, Porto Alegre. vol., pp (200) 4. Riva, C., Laitkorpi, M.: Desigig Web-Based Mobile Services with REST. I: Di Nitto, E., Ripeau, M. (eds.) Service-Orieted Computig - ICSOC 2007 Workshops, Lecture Notes i Computer Sciece, vol. 4907, pp Spriger Berli / Heidelberg (2009) 5. Robiso, D.: Amazo Web Services Made Simple: Lear how Amazo EC2, S3, SimpleDB ad SQS Web Services eables you to reach busiess goals faster. Emereo Pty Ltd, Lodo, UK, UK (2008) 6. Xiag, P., Hou, R., Zhou, Z.: Cache ad cosistecy i NOSQL, vol. 6, pp IEEE (200)