NoSQL-- Hadoop/ HBase/ MongoDB Database Management System II Computer Science Department

Transcription

1 NoSQL-- Hadoop/ HBase/ MongoDB Database Management System II Computer Science Department Dr. Karne

2 Description: In the pass decades, the volume of data is increasing explosively. There are 5-V characteristics [1] to describe the contemporary data. With the advent of advanced electric equipment, more data are collected through smart phone, sensors. These data are not being able to handle by relational database(rdbms) due to the scale and structure. RDBMS need users to design well-formed, strict schema to store the data. However, most of the data nowadays are semistructure or even non-structure hence can t follow the schema. This become a challenge to RDBMS and obviously, it s not suitable for data analyst and data miner. To overcome this difficulty, people turn to other architecture like Hadoop or MongoDB. They both belong to NoSQL (Not only SQL) database design (Here we specify the HBase which runs on top of HDFS). NoSQL database is schema-less and thus having more flexibility than RDBMS to handle various type of data. In this paper, we will introduce Hadoop (HBase) and MongoDB two famous DBMS and compare to RDBMS with their architecture and performance. However, none of them are designed to replace RDBMS, it depends on the user requirement. RDBMS is a mature product and robust in handling online transaction processing (OLTP) and small size of data online analysis processing (OLAP). Hadoop and MongoDB is better for OLAP with big data due to its scalability. Outline: Introduction to Hadoop/ HBase/ MongoDB. Comparison of infrastructure/ Cons and Pros. Example of company database in these different databases. Compare the performance in terms of querying/loading data. How to migrate from SQL to NoSQL Introduction to Hadoop (HBase) and MongoDB: Hadoop: Hadoop is a framework that is useful for big data analysis. The core of Hadoop is composed of two parts: HDFS and MapReduce [6]. HDFS provides reliable, scalable and fault tolerate data storage. The idea of MapReduce originates from divide-and-conquer. First, map data into clusters of nodes, each node is responsible for part of the data computation. Finally, each node return its result back to client, aggregate all the results then merge to a single result. This allows parallel implementation hence reduce the time for analysis. The architecture of Hadoop is shown below. It contains numbers of data nodes to store and process data, one name node monitors all the data node. Every data node has a task tracker to keep track its own schedule and report result once finished. One job tracker monitors all the task trackers.

3 Hadoop architecture Map-Reduce paradigm HBase: HBase is a NoSQL database runs on top of HDFS [5][7]. It s a key-value column-oriented storage. It doesn t require each key-value pair has same number of columns. Moreover, it also allows each column has several column members hence we call column family. Each record contains a timestamp to keep different versions. The data looks like the tables in RDBMS, those tables are split into regions and distributed to various nodes. Similar to HDFS, HBase has three different kind of servers in the architecture. One is region server responsible for storing data and accepting request from clients. Second one is master server which is to assign region server and create, delete and update tables. The last one is Zookeeper, it s used to maintain which servers are live or unavailable. HBase allows user to define row key just we define primary key in the RDBMS. Accessing specific record can through key quickly. With this feature, HBase can handle real-time data processing but still can t replace RDBMS to process complex transactions. HBase architecture ZooKeeper monitor all nodes with heartbeats MongoDB: MongoDB is a free and open-source cross-platform document-oriented database program developed by MongoDB, Inc. It is classified as a NoSQL database program which uses JSON-like documents is vary in schemas. [8] The aim of MongoDB is to enhance the high-performance of data storage solutions with providing WEB applications. [9] MongoDB is database product with the character between SQL and NoSQL. It is one of the popular NoSQL database with abundant functions. It also supports various of different formats such as JSO-like and BSON-like documents that stored complex data types. One of its advanced characteristic is the big support to powerful query language. MongoDB is similar with the object-orient query language which can achieve

4 most functions like in RDBMS and support data indexing. When we using MongoDB, we can create records without defining the structure first, that means we can just simply change the format or structure by adding new fields or deleting existing ones. MongoDB help us use a much easier way to represent hierarchical relationship, to store arrays, and other more complex structures. MongoDB features MongoDB architecture Comparison of infrastructure, Cons and Pros with RDBMS: HDFS/HBase V.S. RDBMS: Characteristics HDFS HBase RDBMS MongoDB Property Distributed file Distributed Relational system NoSQL database database Index Schema N/A Flexible Required & fixed Flexible Data Size Petabytes Petabytes Terabytes Byte Data Storage Files Key-value Roworiented column-oriented BSON files Data Type Semi/Un Semi/Un structured structured Structured JSON/BSON Scalability Scale-out Scale-out Scale-up Scale-up Scale Cost Low Low High High Normalization No need No need Required Doesn t support joins Index N/A Support primary Support index. No various secondary indexing indexing various Random accessing Update No support Support Support Support Write once, read many times Read/Write many times Read/Write many times Read/Write many times

5 Best fit OLAP Batch processing Massive data storage Handle massive data but also need random read/write OLTP Complex transactions Acceptable OLAP NoSQL New SQL Big Data MongoDB V.S. RDBMS: Compare with RDBMS (SQL), MongoDB is schema less that one collection holds different documents. The structure of a single object is clear and easy to understand. There are no complex joins. Also with rich queries it can support dynamic queries on documents using JSON document based query language almost has the same power with SQL. MongoDB uses internal memory to store the working set that makes the access of data faster. Company database example: er ce 21 James E Borg DEPARTMENT HBase: RDBMS: When using RDBMS, we need well-designed schema and normalized tables like below. Both employee and department information are stored in different tables, if we want to know the department name to which employee belongs, Don is used as foreign key and joined with two tables to get the relations. EMPLOYEE Fname Min it Lnam e Ssn Bdate Address Se x Salar y Super_s sn Dn o John B Smith Fondren, M Houston, TX 0 Frankl T Wong Voss, Houston, M in TX 0 Jennif S Walla Berry, Bellaire, F TX 450 Stone, Houston, TX 0 M 00 0 null 1 Dname Dnumber Mgr_ssn Mgr_start_date Research Administration Headquarters As we mentioned above, HBase is a key-value column-oriented database. Each row has a row key just like primary key in RDBMS and a collection of column families and column members. So, we don t need to normalize our table, in fact, it s usually de-normalized. HBase allows user to add arbitrary column members under each column family. The company example in HBase is like below, notice that it allows no value stored in the cell like Mgr_ssn in the row key 88866, namely, HBase stores sparse table. When we read data from table, the format looks like

6 KEY VALUE Row Key Family Member Timestamp Value Name F T1 John Row key Name Address Dept. Personal Info. F M L stre et John B Smith Frankl in Jennif er T Wong S Walla ce James E Borg city Houst on Houst on Houst on Houst on Stat e nam e Dn o TX Rd 5 TX Rd 5 TX Adm in 4 Mgr_ssn Se x M M TX Head 1 M When clients want to read data from HBase, if it s their first time, they need to get a META table which holds the region server location stored in Zookeeper, then query the corresponding location of records and visit the region server. This information will be cached in the clients. When clients issue a write operation, they write the data to a write-ahead log (WAL) located in the MemStore which is a write cache. The WAL can be used for recovery. There is one MemStore per column family per region. When MemStore accumulates enough data, it will be written into files in the disk. HBase store data in HFiles which is indexed as B+ tree style. This allows clients to read specific data without having to scan all the files compared with HDFS only. MongoDB: Nowadays, NoSQL such as MongoDB is widely used in many companies and organizations. Such as, Google, ebay, Federal communication commission, Saks Fifth Ave, and CitiGroup etc. different areas like government, high-tech, and financial services. For retail store, they can build sync what the customer see with MongoDB, to create seamless, endless window shopping. For government, it is easier to use MongoDB to store and analyze fast-moving data. MongoDB provide a rich source of querying. For high-tech company like Expedia, MongoDB supports for flexible document model and any attributes that can analyze better applications or networks. Performance comparison: To understand how NoSQL database benefit to large data analysis, we study some performance related paper to compare RDBMS with Hadoop [2][3]. Performance metrics include data loading time and query execution time. MySQL is used to compare with Hive where it s a data warehouse can analyze, query and extract data and work on top of Hadoop. It provides SQL like syntax and transform query into Map-Reduce automatically. The experiment data set are of three different size. As we can see, the loading time difference get larger when data set size increases. This is because RDBMS needs to check data F Sala ry Bda te

7 integrity when loading data, however, NoSQL database doesn t follow strict schema so it can postpone this step until reading data. Query comparison is based on TPC-DS (Transaction Processing Performance Council- Decision Support) which is a benchmark for evaluating decision support systems. A complex query contains joining 5 tables, 4 aggregation operations, 1 group by operation and 1 order by operation is selected for experiment. The result is shown on the right-hand side, Hive outperforms MySQL especially when data set goes up to 390 GB. This results from the attribution of Map- Reduce and paralleling of Hadoop. Concurrent query performance is also evaluated through the same configuration. Below graphs show the result for comparison of execution time of concurrent query with consecutive query. Table 7 has 4 queries and each query is concurrently running with 5 times so total 20 queries are executing concurrently. The last column represents run 5 times concurrently for each individual query. Obviously, the total concurrency performance is worse than solo concurrency. This is because Hive initially is not designed for such kind of purpose. Hence, such transaction schedule or locking mechanism over distributed system may not optimized as RDBMS cause poor performance for concurrent queries. Migrate SQL to NoSQL: Many of current enterprises use RDBMS face the problem of explosive increasing data. As we mentioned before, NoSQL database is not designed to replace RDBMS. Instead, we can take advantage of NoSQL and combined with RDMBS. The former can be used as backend data storage or off-line data analysis and the latter can handle small/medium data size and complex transactions. Thus, the problem is how to integrate them as a hybrid database [4]. The idea is to use middleware to solve this problem. One paper proposed a data adapter system to integrate heterogeneous databases. The architecture is shown as below. Traditional, user behind application interacts with RDB directly. However, we need one DB adapter to parse and

8 translate SQL command to make it works on NoSQL as well, one DB converter to transform data from RDB to NoSQL in the hybrid database scenario. DB converter is a crucial part because it needs to keep data consistency all the time. When user sends a query request, the query executes on both RDB and NoSQL. At the same time, DB converter may also work on transform data from RDB to NoSQL so it s possible user can get different results from both side. The author takes the idea of synchronization from operation system, blocking some query on NoSQL side when transformation is working and try to minimize the blocking time. RDBMS is a mature product and can handle very complex real-time transaction and many companies still use it for management, it s not practical to develop one NoSQL to replace RDBMS. However, with the existence of middle ware, people can take features of two types of database and reduce the learning rate for employee. How to perfectly integrate them to work on real world is still an interesting topic to explore.

9 Reference: [1] Ishwarappa and J. Anuradha, "A Brief Introduction on Big Data 5Vs Characteristics and Hadoop Technology", Procedia Computer Science, vol. 48, pp , [2] K. Gadiraju, M. Verma, K. Davis and P. Talaga, "Benchmarking performance for migrating a relational application to a parallel implementation", Future Generation Computer Systems, vol. 63, pp , [3] A. Alekseev, V. Osipova, M. Ivanov, A. Klimentov, N. Grigorieva and H. Nalamwar, "Efficient data management tools for the heterogeneous big data warehouse", Physics of Particles and Nuclei Letters, vol. 13, no. 5, pp , [4] Y. Liao, J. Zhou, C. Lu, S. Chen, C. Hsu, W. Chen, M. Jiang and Y. Chung, "Data adapter for querying and transformation between SQL and NoSQL database", Future Generation Computer Systems, vol. 65, pp , [5] "HBase and MapR-DB: Designed for Distribution, Scale, and Speed MapR", Mapr.com, [Online]. Available: [Accessed: 06- May- 2017]. [6] B. Hedlund, "Understanding Hadoop Clusters and the Network", Bradhedlund.com, [Online]. Available: [Accessed: 06- May- 2017]. [7] "HBase Tutorial", [Online]. Available: [Accessed: 06- May- 2017]. [8] Banker, Kyle (March 28, 2011), MongoDB in Action (1st ed.), Manning, p. 375, ISBN [9] Hawkins, Tim; Plugge, Eelco; Membrey, Peter (September 26, 2010), The Definitive Guide to MongoDB: The NoSQL Database for Cloud and Desktop Computing (1st ed)