In this exercise, you will import orders table from MySQL database. into HDFS. Get acquainted with some of basic commands of Sqoop

Transcription

1 Practice Using Sqoop Data Files: ~/labs/sql/retail_db.sql MySQL database: retail_db In this exercise, you will import orders table from MySQL database into HDFS. Get acquainted with some of basic commands of Sqoop Import. Connect to MySQL $ mysql u root mysql> create database retail_db; Query OK, 1 row affected (0.00 sec) mysql> quit Bye Open Another Terminal $ cd ~/labs/sql $ mysql u root retail_db < retail_db.sql Exploring MySQL

2 Follow below command to view database list, changing the database and to list the tables. $ mysql -u root mysql> show databases; mysql> use retail_db; mysql> show tables ; mysql> describe orders; mysql> select * from orders limit 5; mysql> select count(*) from orders; mysql> quit Bye Exploring Sqoop Commands To know various options run sqoop help on the terminal prompt. see sqoop help COMMAND for information on a specific command. $ sqoop help To list the databases available in your database server $ sqoop list databases connect jdbc:mysql://localhost/r etail_db username root

3 Import Data into HDFS from MySQL To list the tables in retail_db database: $ sqoop list tables connect jdbc:mysql://localhost/reta il_db username root Import orders table from retail_db database to HDFS $ sqoop import connect jdbc:mysql://localhost/retail_db \ username root \ table orders \ target dir /360dt/orders Status of above job can be checked in HUE in the Job Browser. List contents in HDFS directory Verify the output at the target directory in HDFS using : $ hdfs dfs ls /360dt/orders Verify the content at the target directory in HDFS using: $ hdfs dfs cat /360dt/orders/*3

4 Incremental Load in Sqoop Updates Data inside retail_db : $ mysql -u root mysql> show databases; mysql> use retail_db; mysql> show tables; Write the Insert query mysql> mysql> insert into orders values (100001, CURRENT_DATE, , 'COMPLETE'); mysql> insert into orders values (100002, CURRENT_DATE,99 995, 'PENDING'); mysql> Above query is used to insert the data in orders table in retail_db databases. Append in import command for tables where rows only get inserted. $ sqoop import connect jdbc:mysql://localhost/retail_db \ username root \ incremental append \

5 null non string '\\N' \ table orders \ target dir /360dt/data \ check column order_id \ last value Verify the output at the target directory in HDFS using : $ hdfs dfs ls /360dt/data Verify the content at the target directory in HDFS using: $ hdfs dfs cat /360dt/data/part * Importing from the query $ sqoop import connect jdbc:mysql://localhost/retail_db username root \ query "SELECT * FROM orders o \ WHERE o.order_id >= \ AND \$CONDITIONS" \ target dir /data/ \ as textfile \ split by order_date Verify the output at the target directory in HDFS using :

6 $ hdfs dfs ls /360dt/data Verify the content at the target directory in HDFS using: $ hdfs dfs cat /data/part * The Sqoop Export Tool Sqoop s export process will read a set of delimited text files from HDFS in parallel, parse them into records, and insert them as new rows in a target database table. The following are sqoop export arguments: export-dir: the directory in HDFS that contains the data to export. input-fields-terminated-by: the input field delimiter. A comma is the default. update-mode: specify how updates are performed when new rows are found with non matching keys in database. Values are updateonly (the default) and allowinsert. :one: Create a simple file named id.txt using the cat cmd and the data like number from 1 12 $ cat > id.txt

7 :two: Upload the file into HDFS $ hdfs dfs put id.txt :three: Go to MYSQL $ mysql -u root :four: Select database retail_db; mysql> use retail_db; :five: Inside retail_db create a single row table name poem mysql> create table poem (poem_id int); :six: Insert the data into poem mysql> insert into poem values (10); :seven: Exit MYSQL or Open another Terminal :eight: Type Sqoop Export Query: $ sqoop export connect jdbc:mysql://localhost/retail_db

8 username root \ export dir id.txt \ table poem :nine: Then Go to MYSQL and verify your data mysql> select * from poem; :keycap_ten: Create a STORE PROCEDURE SQL mysql> delimiter // mysql> CREATE PROCEDURE simpleproc (IN param1 INT) BEGIN update poem set poem_id = 4; END // mysql> delimiter; :one::one: Go to Another Terminal Write SQL Export cmd $ sqoop export connect jdbc:mysql://localhost/retail_db username root \ export dir id.txt \ call simpleproc Now in MYSQL you will see that your data replaced and look like

9 Results You have become familiar will import orders table from MySQL database into HDFS. And gotten acquainted with some of basic commands of Sqoop Import.