Enable Spark SQL on NoSQL Hbase tables with HSpark IBM Code Tech Talk. February 13, 2018

Transcription

1 Enable Spark SQL on NoSQL Hbase tables with HSpark IBM Code Tech Talk February 13, >> MARC-ARTHUR PIERRE LOUIS: I want to welcome you to another of our tech talk series. Today is a great presentation again about technology. My name is Marc-Arthur Pierre Louis, your friendly host. Last week we spoke about [inaudible] connector that allows you to do streaming. Some requirement imposed by modern technology on an older technology JDDC. It's great, allows you to do streaming and nonblocking which is required today to have responsive websites. Today we have something else in the realm of database technology, HSpark enabling you to do SQL on a database that is not SQL. That is HSpark, the project that allows you to leverage Hbase on the Spark technology. To talk about this interesting technology is Bo Meng from the HSpark team who is going to take us through what is HSpark and how you can do SQL on [inaudible] database on the Spark. >> BO MENG: Hi, thank you for having me. This is a project I was working on before and this project is still going on. Let me introduce myself. My name is Bo Meng. I'm a senior software engineer in IBM, and also I'm working the open technology such as Spark, hadoop. My major interest is focus on big data, hadoop and the Hbase and Spark. Also I have my colleague here, a big data architect. He is lead architect for this project. He has hadoop [inaudible] years experiencing database domain. He will be my company for today. (distorted audio). After this talk. Let's get started. Today's agenda will be like this. First we will give a short introduction of what Hbase, what is Hbase, and how it, what it is and what it is not. Also we are talking about Hbase and traditional... (echoing). Traditional the comparison, because between the Hbase and the SQL database, and also I'm going to show some key features between why the Hbase has some performance advantage. Also it will help us to understand how we design the HSpark. After that

2 we will switch to the HSpark part. We are talking about its data, how it is mapping the traditional data type to HSpark data type. Also we will discuss iteration of HSpark. Finally we will give a short demo to show how to use the HSpark. This will be a review, but I will go through it and add comments along with as we do. Hbase is a open source technology, which right now is the top level project in Apache on the website. The Hbase is distributed and sorted map actually. This model is the original idea is from Google big table. The Hbase has Apache 2.0 license. Right now it's part of the hadoop ecosystem. You can see also there is Hbase is widely used by some big companies such as hadoop, Adobe, Facebook. Facebook is, if you are using the messaging system in Facebook, they are backed by the Hbase. I checked a few days ago the current stable version is That is the stable version for the Hbase. From the user perspective, actually Hbase is just a key value store and use HDFS for actual storage. As I said, it's modeled by the Google big table and it's column oriented. It has the version support which means you can have the data for the same column but has the multiple versions for that. Compared to the traditional SQL database Hbase is not a SQL database, it does not have any join, query engine, data types and SQL query, we will see a little bit later. It also does not have the schema and it will not have the traditional DBA will be have usable. That is Hbase, it is and is not. Compared to the traditional database, Hbase has some advantage. The first category is it's high performance. The key features here is rowkey is always sorted. Whenever you want to retrieve the data by its row key or by certain range of the row key, since the row key will be sort, is already sorted, it is very fast to retrieve the data. So it's very efficient. Also the row key can be controlled by different filters. The filters can control how you can scan the rowkey. Also the columns or families can be used any, also can be filtered. It also can support different kind of filters. Because it's built on the hadoop system, it uses hadoop and zookeeper so it has high scale abilities. The other advantage for the Hbase is you have the consistency, realtime, kind of realtime performance and also the other things that can be advantage from the hadoop system. Right now, we can do comparison about the SQL and Hbase. Suppose we have a table here, which has the, just the normal user table. It has some user information need to be stored. Some of the user information like name, age and gender. So you can see the upper table is the SQL table. Usually you can find

3 something like this. The second row here I want to show is if somebody wants to change the name from Bob to Andrew, so basically you have to update for that column. But sometimes if this happens, it may lost the information here, because if we want to trace the history, it may get a loss. Another thing here is how you design something from the Hbase. As I said, Hbase has rowkey. So row key basically is the binary array to represent the row. It's always ordered, as I said. For example, you have something like a row key ordered here, then you have the column key to represent the actual column, then you have a time stamp to give the version of each row, and also a value associated with the row. >> MARC-ARTHUR PIERRE LOUIS: Question for you, Bo. On the row key, are those, they are not unique. They can be multiple keys that are the same? >> BO MENG: Actually, in the, this is just for illustration perspective. Actually the row key will be, it will have one row key that has column keys in one row. >> MARC-ARTHUR PIERRE LOUIS: Row keys [inaudible] okay. >> BO MENG: It's like one row, for example, if the row number, then you have the three row key, column key here. Then you have the values. The second row may have the row key 02, then you have the two column keys here. >> MARC-ARTHUR PIERRE LOUIS: Got it. >> BO MENG: So, it means the Hbase, the row, the column could be varying, it is not a fixed. You can add column -- (overlapping speakers). >> MARC-ARTHUR PIERRE LOUIS: I failed to see one row that had several columns. You were describing each of these rows were actually part of one row. >> BO MENG: Yeah, okay. Yeah. Another thing I want to mention here is how you change the name. For the second row you change name from Bob to Andrew, actually the time stamp changed. Basically if you want to retrieve the information, you can retrieve the history to see how this value got changed. >> MARC-ARTHUR PIERRE LOUIS: Got it. >> BO MENG: Hbase also has some tool called Hbase shelf. It provides the DDL. The DDL is how you create the tables and are also how you retrieve the information. So we can simply walk through that. First line is how you create the table, and also create the column name cf. The first test is the table name, cf is the column name. Then we put the three records into this table. The first is the table name test and the row name is row1. Also, we put the column name cf column name and the value is Bob. Another thing is we want to put the column row2 is Tom and row3 is the Andrew. This will basically list all the tables in the current

4 namespace. Here basically it will get the test and the tables. Scan test will list all the records in the test. Get will get the, all the [inaudible] in the test table then row key equal to row 1. Delete will, you can complete the test table, row 1, then the column name, column name equal to the cf column name. >> MARC-ARTHUR PIERRE LOUIS: Quick question for you there. Once you create a row, and you abend the row it doesn't delete and if you don't get row, it will get all the rows and you are going to use the time stamp to find out what one is the most current one. (cavernous audio). >> BO MENG: In the default Hbase give you the latest version. So it's the latest, the value of that row but you also config how many records you want to track, meaning if you want to track last record, it is okay for -- >> MARC-ARTHUR PIERRE LOUIS: Got it. Interesting, yeah. >> BO MENG: Disable, basically disable because it's [inaudible] you need to first disable the tables, make sure the [inaudible] has all the tables disabled, then you can drop the table. That is a little bit different. Here basically you want to see is there is quite difference from the traditional SQL perspective how you create a table, how you get all the record, because it's not, Hbase is not traditional database. SQL is not supported. The next step is about how you map the, so from the SQL perspective how you can map the terminology of the Hbase to the SQL. So per the table here so it's for you to understand how you map the concept from SQL to the Hbase perspective. It's namespace is kind of equal to the database, and table same and row key is Hbase, placeholder is the key columns in the SQL term. The important, the point here is the key column could have multiple columns, which in the SQL term is called the composer key, like multiple keys to be one, to be the key column. In Hbase a column base and qualifier can combine [inaudible] (distorted audio). In SQL terms it could be [inaudible] in the SQL it can be whatever the data type is, like int and the float, double. But in Hbase, the only data type is the byte array. That is the only type, so meaning whatever the SQL, the value of the SQL type you need to convert the type into the byte array to store the data. Here is another example. In the SQL we want to create a table customer which has the ID name and age, age and salary and list the information and primary key is actually the ID and the name. This is composed key. ID name will combine as the

5 primary key. In the Hbase, if we want to create a table, we want to create the table name could be called customer. That is the logical name. In the physical storage we can call it H customer or whatever the name in the Hbase, and the row key will be the ID plus name in the byte array format, and the column family plus qualifier will represent the nonkey column which will be the age and the salary. Now we can switch our focus to the HSpark. Why this project can be fit into this scenario, so because the HSpark is a combination of Spark optimizer and the Hbase, the filtering, pruning capabilities. In the large collateral basically we want to use the Spark to optimize the query and also in the physical layer we want to push down the predicate to the Hbase, then leverage its filtering pruning capabilities for performance. From this perspective, HSpark looks very fitful for this scenario. It also fits into the Spark ecosystem, because we can, user can still use the data set to carry the database. Also from the Hbase user perspective, we provide the Spark SQL and data set so it's easy to use SQL to carry those unstructured data in structured way. Also in the market there are dimension, there are some similar technologies. One of this is the Apache Phoenix, which is built SQL layer on the Hbase, but it's not using, leverage the Spark. So from the performance perspective, we are not leveraging any Spark logical level optimizer. So the performance won't be that good. Spark has very advanced optimization in the logical level. Another thing is the, in the Hbase there are also Spark connectors. But also, they are not leveraging the performance power for the physical pushdown into the Hbase. So, here is the summary of the HSpark project. HSpark basically is running Spark on Hbase. We are leveraging the Spark's framework as the, such as the parser, optimizer and execution engine. While you are doing DDL or query, the parser is already there. We are leveraging the Spark parser. We are also leveraging the logical level optimizer from Spark, and push down the physical execution to the Hbase. The Hbase will act as the actual data source, and for the metadata information, we create additional extra table in the Hbase to store notes information like the mapping, the logical table name mapped to the physical table name, something like that. And also for the predicates, we prune the predicate and based on its, the region information, and also some of the data will be included in the baseline information. So additional filter can be used to avoid the full scan in the Hbase. Another feature in the HSpark is the bulk load.

6 Data, a lot of scenario is you try to load the historical data into your Hbase to do the data analysis. Bulk load is another feature we support. We want to allow users to load the historical data into the Hbase by using the HSpark. The data types, currently we are supporting a subset of the HSpark, Spark data types, gives type listed, string, byte, the float, double, boolean, date and time stamp. The reason we are not supporting every data type in Spark is that because every data type, as I said before, the data is actually stored in the Hbase is the binary array. So every data type, when we want to support, we need to provide our own algorithm to convert data from the actual value to the binary array, and also because the row key is ordered so we want to preserve the ordering when it convert to the binary array. Let me give you an example. For example, if you have a integer data type, you have two values. One value is the minus 1. One value is plus 1. By simply convert to the binary array, the minus 1 may have a bit, the higher bit as 1, like 10001, then the plus 1 may convert to So in the binary format, the ordering may be reversed, instead of the correct order. It will provide the actual algorithm to convert to the binary array. Also we want to keep it original ordering. That is one of the challenge. That is why some of the data type we are still working on. Another point is when we convert the actual data value to the binary array, we want to use the last byte as much as possible. So that means we want to use less storage to store the data. That will also improve the performance, because when the data shuffles they are traveling through the network. It will be less data if we use less bytes. So another example how you can improve the performance by using less data, so suppose we have three columns, one column is date. One column is string. The third column is the boolean. We want three columns as composed key to convert to a row key which is binary array, as I said. One solution is we use something like a index. First we provide the column length which is 3. We provide the offset of the first value, then we provide the length of the first column, then we provide the offset of the second column. Then we provide the length of the second column as a feature. In this way we basically use the index to provide those information and also provide the location of the actual data. Then we convert the whole thing into the binary array. The second solution is we just provide the data, because it's a date, we can convert to a full [inaudible] second is the string, because string could be any length, so we add the 00 to the end

7 of the string, which will be the ending indicator, and then we add the third data, this is the boolean, we can just use the one byte. So here, compared solution 2 and solution 1, we got much less the bytes. In this way we can save a lot of byte and also save a lot of storage. Here is just an example of how we can implement the conversion. Another thing in HSpark is, we want to, how we can converting the predicates into the, push down to the Hbase. One thing is the parser at the logical level it will be automatically optimized by the Spark. At the [inaudible] level the predicate will be divided into two groups. One is the things that can be handled by the Hbase, and the rest is cannot be handled by Hbase. For the Hbase doable predicate, we will first eliminate the not, then reduce the predicates by the Hbase region. Then use the different filter to handle the nonkey and the key column. Then we use the filter to scan the data there. Eventually we will construct a final result based on the scan, because there was also a part left Hbase cannot be handled. Currently, the HSpark has the syntax similar to the Spark. It has create a database, create a table, then query and drop kind of DDL and the queries. The key thing here is we want to leverage the Spark parser as much as possible. So some of the Hbase specific informations will be added as the table parses. We will see shortly in the demo how it works with the current Spark parsers. The SQL query is always supported, it is nothing changed. When the table is created, so all SQL query will be supported. >> MARC-ARTHUR PIERRE LOUIS: Bo, I have a quick question for you. The HSpark team, you are part of that team. Is it made of IBMers only? Or do we have external people in that team? >> BO MENG: Right now it's IBM only. >> MARC-ARTHUR PIERRE LOUIS: What release are you guys in right now? >> BO MENG: I will talk about shortly. >> MARC-ARTHUR PIERRE LOUIS: Good. >> BO MENG: For the HSpark, we also in the release, we also have the developer or the user to quickly doing the test. So we provide two tools to help them. One is the HSpark SQL shell. This shell is if you use Spark you know how the shell works. You can try the SQL in the shell. It will automatically give you the response, like the result, how if you do the query, if you do the creation, the table creation, it will give you the result. And also we provide Python shell to help people familiar with

8 Python language. Here is the some statistics of the HSpark. HSpark project is right now in the GitHub. It's code version is same as Spark version, because as Spark [inaudible] some code changed. We want to make sure HSpark will work with certain version of Spark. So the current release version is It will work with same Spark version. It worked with same environment like JDK and Scala in the GitHub. Developers, right now, is the Yan Zhou and me. In the Read me you can follow the read me to set up the environment and run the test and do all these things from spark. This project is published on the Spark package. The Spark package is like a host for all the third party for all the contribution to Apache Spark. You can find that project also. >> MARC-ARTHUR PIERRE LOUIS: Quick question that I have for you is, and that is question when rubber meets the road where these tech talks are concerned, if somebody in the audience wanted to kick the tire of HSpark, how easy would it be for them to set it up and see what you are talking about? >> BO MENG: I think we also have the, as the base presentation, along with the pattern is to help you to set up the environment and doing a quick test for this project. I think follow that pattern will be quite easy to help you to understand how to set up the environment. >> MARC-ARTHUR PIERRE LOUIS: Excellent. It's nice to be talking to the developers themselves. This project is your baby, right? >> BO MENG: This project can be used as Spark plan for developers to use Hbase on, to use the Hbase by through the Spark. So it can be used as the data scientists to query the information using the SQL. But the data can be reside on the Hbase. >> MARC-ARTHUR PIERRE LOUIS: Good. >> BO MENG: Any contribution, any tester, any improvement and it's just very welcome to this project. But our main communication channel is the GitHub. So you can file any through the GitHub. For to-dos, we still, there are some ongoing performance testing on the clusters, to compare to other approach. We want to support the different data paths, as I said each data path need algorithm to convert the actual data into the binary array. Another big part is the Hbase has coprocessor, went to support in the future release. We want to improve the documents and also we want to improve the code. As I said, as I showed in the last page, we have the actual, the code, and also we have the test code. The test coverage is pretty good right now. But we still want to improve it.

9 Then the final thing is to update to the Spark package to the latest version, the link to the GitHub. So, here I'm going to show a demo how to use HSpark shell to assess the data, how to create the table and import data and do the query. It will be a video, but I will comment along with the video. Another thing I want to mention is this demo is actually you can find this demo in the pattern, IBM pattern Web Page with the -- let me show you. We have a pattern here, it's called use Spark [inaudible] Hbase tables. The link here actually you can view the video. You can work through how you set up the environment, how to do the query. But I will briefly show this video. Suppose you follow that pattern to set up the environment. You can invoke the HSpark which is the HSpark shell. This shell will be released along with the HSpark version. In the folder you can find this in the shell, it will give you this environment. This is the shell. In the shell, we want to first create some data table. So I put some schemas down below to show the table. But basically we want to use the TPPPS which is benchmark for this. I show the table schema here to allow you to understand each table, and also the relations between tables. Here is the table. Also you can find the actual script in the, in this pattern. There was a folder, that already has the table creation script. So in the script folder, you have the create table text. It will have the, first it has the commented, which is the formatted script. Then you have just one line script, because if you want to cut and paste, you want just one line to paste into this shell. Here I paste the shell to the shell, then it will execute it. Then you can see this script executed, create a table. Follow this script, you create like five tables. Each table has a different key column, nonkey column. So after you create the tables, you will basically want to take a look at the tables, how the table schema gets created. And because the HTPS and the actual may have different data paths you may want to take a look. The second thing is load the data into the structure. As I said in the presentation, HSpark has bulk load. We have syntax here, load the data into the table. So one thing you want to make sure is that in the path parameter you want to make the change to the actual, you know, the location of the data. I put the location, actual location. You will see data is load to the table, shell's table. After the data is loaded we can query the table by using the standard SQL query, SQL language. In the script we also have the, some examples, how you query

10 a table. The nice thing here is, I put the SQL here, and also I put the expected results. You can verify if the result is expected, because sometimes if it's something wrong, you may get a different result but the result may not be as expected. For first query, how many lines of the table, it's got a hundred. Also we can try the other things like the query, query 3, will be more complicated. So you get results here. So you can compare the result with the actual result, which will be shown in the examples. Another thing I want to mention is if you are a developer, you want to use the codes to query the table, you can go to the actual project at GitHub. There is a text folder. You have a lot of test codes how you can, show you how you can use the actual code to access the HSpark, use the HSpark. So it will help you to get started, if you are developers. >> MARC-ARTHUR PIERRE LOUIS: I got a quick question for you, Bo. >> BO MENG: Yes. >> MARC-ARTHUR PIERRE LOUIS: As we do in JDBC and we have a connector that we can use to issue [inaudible] (cavernous audio). In HSpark, a connector? >> BO MENG: Yeah, in the Hbase it's also, you need a connection to connect to the H master. In the test code you will see how you can set up a connection to the H master, that is for the HSpark shell, it is supposed to connect to your local H master so it just connect with your local. But yeah. But in the code, you can configure to, configure whatever the machine that is the H master. >> MARC-ARTHUR PIERRE LOUIS: Thanks. >> BO MENG: That is the demo. That will also conclude my presentation for the HSpark. And any questions? I think we still have some time to answer the questions. I also mentioned before we have Yan Zhou here, he can answer the questions if there are any coming. >> MARC-ARTHUR PIERRE LOUIS: I'm looking into the chat room here. I don't see remaining questions. Do you have any questions that we answer [inaudible] >> You are asking if I have any questions? >> MARC-ARTHUR PIERRE LOUIS: In the chat room, I'm wondering if there are some questions that are left to be answered or if all of them have been answered already. >> I answered a few, a couple questions, regarding the use of, to load data directory into HSpark is question 1. There is another question regarding data unique list, how is that guaranteed. The third question I'm not sure, it's why are we parsing cf name for delete.

11 >> MARC-ARTHUR PIERRE LOUIS: Yeah, no. What I was wondering, I was wondering if there were any other questions left. It doesn't seem there are any questions left in the chat room. >> I'm not seeing that. I'm only seeing three questions. >> MARC-ARTHUR PIERRE LOUIS: Okay. All right. Well, so we thank you, Bo and Yan for this presentation. It is certainly an interesting technology that people can use, take a look at and use for their applications. I have a question for you, Bo. >> BO MENG: Yeah. >> MARC-ARTHUR PIERRE LOUIS: What about bindings, are they only available -- you said Python shell, but how about Java and other languages? Are there bindings available? >> BO MENG: Yeah, basically it's HSpark can be used as, we also have to test the code to show you how to use the Java to assess the Hbase, by using the HSpark. Just take a look. Currently I think we support... (buzzing). -- the Java and Scala to -- >> And Python. >> MARC-ARTHUR PIERRE LOUIS: Yes. Python. Okay. I thank you. Certainly people can take a look at the pattern if they want to try it. You will see how you can use this technology. They don't need to have a, do they need to have a Spark account or can they load that on their laptops? >> BO MENG: It can be load on just single laptops. You don't have to use the clusters. But in the real environment you may need a cluster. But for test purposes, you can just use one machine, just laptop is okay. >> MARC-ARTHUR PIERRE LOUIS: Very good. Thank you. People can certainly take a look at the pattern if they want to replicate what you have done. There is always use for new language and new technology to access Spark. Thank you for your time. Looking forward to see how [inaudible] this presentation. We got three things that I want to discuss real quick before we let you go, the audience. Index is next week, the 20th through the 22nd at the [inaudible] center in California in San Fran. We invite you to be there. So far we have about 1,000 people registered. We need more than that. It's a great conference. Go register and be there and learn from developers, put together by developers, so that is going to be great, Index first of its kind at IBM. The next thing that we want to talk about is our pattern, next week we have a pattern, it's about how you can use Watson natural language understanding and NLTK natural language toolkit and DSX to gain insights on data. That is going to be a presentation that shows you how to use these to gain insights

12 [inaudible] world. That is going to be at 1:00 next week. Then the week after, the week of the 26th we are going to start, we are going to have [inaudible] tech talks, that which can be open source week. It is going to start the week after the 26th, week of the 26th. We are going to have four days of great tech talks that are going to be offered in blocks of two hours, and each two hours are going to have four tech talks. We are going to talk about OpenWhisk, other open source projects, that are going to be there for you, OCI, ODPI, all those great projects are going to be there, Java, Python, Jupyter Notebook, Jupyter and all of that great stuff is going to be there for that week of the 26th. We invite you to come and [inaudible] tech talks. We have a lot lined up for you next week and the week after. Some of you are going to be at Index. While you are at Index we are going to do one tech talk. The week after we have a bunch of tech talks for you, the week of the 26th. Thank you for your time and we hope to see you next week [inaudible] coalition tech talk. Thank you for your time and we will see you next week. Bye-bye. >> BO MENG: Thank you, bye-bye. (end of call at 12:51 p.m. CST)