Contents Kaivos User Guide 1 1. Getting a database account 2 2. MySQL client programs at CSC 2 2.1 Connecting your database..................................... 2 2.2 Setting default values for MySQL connection.......................... 3...................................................... 4 2.3 Graphical Interfaces........................................ 4 2.4 Following the disk usage in kaivos.csc.fi............................. 4 Example: Creating a table.................................... 4 3 Data import and export 5 3.1 Importing and exporting data between CSC servers and Kaivos................ 5 Example: Importing data with mysqlimport............................. 6 4. Using MySQL client through batch job system 7 Example : Using MySQL database from a batch job script in Taito................ 7 5. Using kaivos.csc.fi directly from your local computer 8 5.1 Tunneling MySQL port with ssh or plink............................ 8 5.2 Tunneling MySQL port with Scientist s User Interface..................... 9 5.3 Remote MySQL connections................................... 10 6. Using Perl MySQL API at CSC 11 Write a MYSQL database access script............................. 11 Running the MYSQL database access script.......................... 11 The statements issuing methods................................. 12 Transaction............................................ 12 7. Using Python MySQL API at CSC 12 Write a MYSQL database access script............................. 12 Running the MYSQL database access script.......................... 13 The statements issuing methods................................. 14 Transactions........................................... 14 Kaivos User Guide Relational database service at kaivos.csc.fi is intended for the users, who want to use their own MySQL databases in CSC computing environment. In this guide you can find information about getting started with the database service and instructions for using it. The actual SQL command language is not discussed in this document in detail. Please use the MySQL manual or some of the many SQL guides published for an introduction to SQL databases. 1
1. Getting a database account The kaivos.csc.fi database service can be accessed only from the computing servers of CSC. Thus, in order to use the database service, the project manager as well as all the users should have a CSC user account at Taito. To open a database to kaivos database service, the project manager needs to print out the application form for database service at: Applying for user accounts When the application is accepted, a new empty database is created to the database server (kaivos.csc.fi). The database service is used through MySQL client programs or other MySQL interfaces. The project manager will receive three database user accounts with passwords to access and manage the database. The database user accounts are based on the database name defined in the application form. The three database user accounts have different roles: The databasename_admin user account is intended for database administration. This user account can create new tables and indexes for the database. This user account can also remove records and tables from the database. However, the admin user account can t create new databases or database user accounts. This user account has all rights to the database except the GRANT OPTION. The databasename_user user account is intended for everyday usage of the database. This user account can read, write, change and delete data from the database tables. However, this user account can t change the structure of the database i.e. create or drop tables. This user account has the following rights to the database tables: SELECT, INSERT, UPDATE, DELETE. The databasename_read user account is intended for users who are allowed to do only queries to the database. This user account can t change the database in any level. This user account has only the SELECT right to database tables. The project manager should distribute the database user accounts with passwords to the users of the database service. The databasename_read user account can be given to users who are not members of the computing project. One research group can have several databases. Each database has proprietary database user accounts and passwords. 2. MySQL client programs at CSC Below you can find instructions on how to use the MySQL client programs in CSC s computing environment. The SQL language is not covered, but you can find a lot of documentation and guidance elsewhere (e.g., MySQL user manual). 2.1 Connecting your database Once the database user accounts have been received from CSC the user can start to create tables and store data into the empty database. The database service can be used through MySQL client programs available in Taito ( including Taito-shell ). In command line usage, users open the connection from Taito to the kaivos.csc.fi server, which provides the database service. To be able to use MySQL commands you need to first load the environment: module load mysql After this you can start the MySQL client program or execute some of the MySQL commands. A MySQL command line client session to kaivos.csc.fi is opened with the command: 2
mysql -u db_user_account p -h kaivos.csc.fi --local db_name This starts an interactive MySQL session which you can use to execute SQL commands for your database. In the command above options -u and -h define your database user account and the database server name (kaivos.csc.fi). The -p option defines that password is used for authentication. The local option is not always necessary, but it is useful as it indicates to the MySQL client that the MySQL connection comes from a remote host. Instead of giving the SQL commands interactively, you can also write the SQL commands into a file and execute them with the command: mysql -u db_user_account -p -h kaivos.csc.fi --local db_name \ < commands.sql > output.txt or mysql -u db_user_account p -h kaivos.csc.fi --local \ --skip-column-names --quick db_name < commands.sql > output.txt In the latter command, the skip-column-names option is used to print out only the data produced by the SQL commands. Otherwise names of the selected columns would be printed too. The quick option makes the client to print each row as it is received instead of storing it to the cache first. 2.2 Setting default values for MySQL connection In the mysql commands above, the client program would ask for the user name and password every time when a mysql command is executed. It is however possible to define default values for the database name, user name and password, that will be used for the connection if no password or user name is given. Setting the default values is useful especially in cases where the user uses mostly just one database. The default values for the MySQL connections is defined in a file called.my.cnf (note the dot in the beginning of the file name) that locates in the user s home directory. This definition file can be constructed in two alternative ways: in Taito or Taito-shell with a normal text editor or with the command: kaivos_mysql_cnf Below is shown the basic structure of the.my.cnf file: [client] user = MySQL_username password = MySQL_password host = kaivos.csc.fi [mysql] database = db_name You can also store the settings for MySQL connection to some other file name, and apply these settings by using option defaults-extra-file=settings_file. For example, if we would like to use MySQL connection configuration that is stored to file db_conn2.def, we could execute the previously used MySQL query by using command: mysql --defaults-extra-file=db_conn2.def --local db_name \ < commands.sql > output.txt 3
2.3 Graphical Interfaces Graphical MySQL interfaces have not been installed to Taito. However graphical database interfaces are very efficient when you need to get familiar and administrate a complex database that contains a large number of tables. If you wish to use your database through a graphical user interface, we recommend that you install the interface program to your local computer and create a remote connection to Kaivos as described in chapter 5. 2.4 Following the disk usage in kaivos.csc.fi Each user has only limited disk space available in the kaivos.csc.fi server. If the database reaches the disk quota, the database users can no longer write to the database. In these cases the users should clean up the database to reduce the size or apply more disk space from CSC. You can check the database quota and usage with the Kaivos Status tool or with command: or mysql_quota mysql_quota u db_user_account p db_password The situation can also be checked during the MySQL terminal session. kaivos.csc.fi, give the MySQL command: To see the disk quota in CALL quotadb.quota(); The current size of the database can be checked with the MySQL command: CALL quotadb.usedquota(); Example: Creating a table In the following example we create a new table called results into an empty database DB_A. First we connect to the database with the MySQL client: mysql u DB_A_admin -h kaivos.csc.fi p DB_A Next we create a new table that contains three columns: id, value and comment. The id column is in this case defined to be a unique integer, the value column contains floating point numbers and the comment column text data (a non binary string with max. 30 characters). Note that in real life you normally define many other features like the primary key column and auto filling etc. when you create a new table. CREATE TABLE results (id INT UNIQUE, value FLOAT, comment VARCHAR(30)); Note the semicolon (;) that is used as end character in SQL commands. You can now use SQL command SHOW TABLES to see which tables your database contains. 4
mysql> SHOW TABLES; +----------------+ Tables_in_DB_A +----------------+ results +----------------+ 1 row in set (0.01 sec) Data can be inserted to the table with the SQL command INSERT INTO. Below we insert three new lines to the table: mysql> INSERT INTO results (id, value, comment) VALUES (1, 27.45, "Test case"); mysql> INSERT INTO results (id, value, comment) VALUES (2, 12.33, "Another"); mysql> INSERT INTO results (id, value, comment) VALUES (3, 25.33, "Value2"); When the table contains data, we can now do SQL queries with the SELECT command: mysql> SELECT value FROM results WHERE id=2; +-------+ value +-------+ 12.33 +-------+ The MySQL client session is closed with the command EXIT: mysql>exit 3 Data import and export 3.1 Importing and exporting data between CSC servers and Kaivos In the CSC computing environment we recommend command mysqlimport for loading datasets into the database. This command reads in a delimited text file into a table already existing in the database. To load a large text file from a CSC computing server to a database in Kaivos, you can use the command syntax: mysqlimport -h kaivos.csc.fi -u db_user_account --local \ --compress --password database_name input_file.table Mysqlimport strips any extension from the input file name and uses the result to determine the name of the table into which to import the file s contents. The user must make sure that the database contains a previously created table with data columns that match the data in the input data file. The user may also have to change the name of the input file if it is not compatible with a database table. The local option defines that the data file locates in the machine where the client is used and not in the actual database server. Thus in the case of mysqlimport, this option is obligatory in the computing servers of CSC. Downloading complete tables or databases from the database can be done with command mysqldump. 5
This command was developed for making backup copies of MySQL databases. In the case of kaivos.csc.fi, backup copies are not needed as the database is automatically backup copied by CSC. Instead mysqldump offers an easy way to make a copy of your database so that you can move both the data content and structure of your database to another SQL server. You can make a copy of the whole database: mysqldump --set-gtid-purged=off u db_user_account -h kaivos.csc.fi p \ database > database_dump.txt or just from one or more tables: mysqldump --set-gtid-purged=off u db_user_account -h kaivos.csc.fi p \ database table_name > table_dump.txt When mysqldump is used with the default settings, the result files contain MySQL commands that are needed to create and populate the selected database tables. Mysqldump locks the table in the beginning of the copying. Because of this only the dbname_admin user account of the database can launch the command by default. In the case of other user accounts (dbname_read or dbname_user) skip-locktables option should be added to the mysqldump command. You can import the database_dump.txt file as follows: mysql u database_user_account -h kaivos.csc.fi p database < database_dump.txt Example: Importing data with mysqlimport In this example we import a data set into the result table created in the example in chapter 2. The data to be imported to the database locates in file: data_to_import.txt. This file contains data rows like: 1-419.557 STRUCTURE1.PDB 2-479.662 STRUCTURE2.PDB 3-517.019 STRUCTURE3.PDB 4-450.922 STRUCTURE4.PDB 5-421.991 STRUCTURE5.PDB 6-507.076 STRUCTURE6.PDB 7-444.598 STRUCTURE7.PDB 8-444.552 STRUCTURE8.PDB 9-414.492 STRUCTURE9.PDB 10-444.549 STRUCTURE10.PDB 11-463.394 STRUCTURE11.PDB 12-430.548 STRUCTURE12.PDB etc To import the data to the results table, we must first copy the data to a file with a name that is compatible with the table name (results). cp data_to_import.txt results.table Then use mysqlimport to import the data mysqlimport -h kaivos.csc.fi --local --compress p u DB_A_admin DB_A results.table 6
4. Using MySQL client through batch job system The MySQL client program can be used in the batch job systems in the same way as in interactive client usage. The only difference is that in the batch jobs, the database password can t be given interactively. Instead it should be given by using MySQL configuration file (.my.cnf) in the home directory. Below is a sample MySQL script for Taito. First we need to create a MySQL connection configuration file that locates in the home directory. In this case we use user account mydb1_admin, whose password is abc123. The file, named as.my.cnf, would now look like following: [client] user = mydb1_admin password = abc123 host = kaivos.csc.fi Then we create the actual batch job script. The script below reserves 12 h time and 1 GB memory to run the MySQL query that is defined in the file query_commands.sql. The query is made to database mydb1 and the connection parameters are read from file.my.cnf. The results are written to the results.txt file. #!/bin/bash -l #SBATCH -J mysql_job #SBATCH -o output_%j.txt #SBATCH -e errors_%j.txt #SBATCH -t 12:00:00 #SBATCH -n 1 #SBATCH --mail-type=end #SBATCH --mail-user=kkayttaj@uni.fi #SBATCH -p serial #SBATCH --mem-per-cpu=1024 module load mysql cd $WRKDIR/my_data mysql --local mydb1 <query_commands.sql > results.txt Example : Using MySQL database from a batch job script in Taito The MySQL database in kaivos.csc.fi is intended for cases where computing servers of CSC use the MySQL database to store and analyze data. In these cases the database is normally not used interactively, but the MySQL client is used automatically from a shell or program script. Below is a a sample mysql session where database called DB_A is accessed using the database user account DB_A_admin and password abc123. This user account information is first stored into.my.cnf in home directory: [client] user = DB_A_admin password = abc123 host = kaivos.csc.fi Below is a sample batch job script, called as kaivos.bash, queue system. that utilizes kaivos.csc.fi within the batch 7
#!/bin/bash -l #SBATCH -J mysql_job #SBATCH -o output_%j.txt #SBATCH -e errors_%j.txt #SBATCH -t 12:00:00 #SBATCH -n 1 #SBATCH -p serial #SBATCH --mail-type=end #SBATCH --mail-user=kkayttaj@uni.fi #SBATCH --mem-per-cpu=1024 #load mysql environment module load mysql #go to the right directory cd datadir # run the analysis my_program < inputfile30.data > results.30 #feed the data to the database mysqlimport --local --compress DB_A results.30 #change the status value in the dataset_table mysql --local DB_A <<EOF update dataset_table set status="done" where name="inputfile30.data" ; EOF #remove the original results file rm f results.30 The sample script above first analyzes a file called inputfile30.data with program my_program. The results are first written to file called results.30. The data in this file is then imported to a database with mysqlimport command. Note that the script assumes that a table called results already exists in the database DB_A and that the columns in the results file are in the same order as in the database table. After importing the data to the database the script launches another MySQL command. The second command modifies an existing table called dataset_table. The mysql command changes the status value in this table so that in the row where the name column contains the value inputfile30.data, the status column gets the value: done. The kaivos.bash script, described above, can be submitted to the batch job system of Taito with command sbatch kaivos.bash 5. Using kaivos.csc.fi directly from your local computer 5.1 Tunneling MySQL port with ssh or plink The MySQL databases in kaivos.csc.fi can be directly accessed only from the computing servers of CSC. However, you can make the database visible to your own computer using your CSC user account and port forwarding through an ssh tunnel. In linux and MacOSX machines an ssh tunnel from your local computer to kaivos.csc.fi via taitoshell.csc.fi can be done for example with the command: 8
ssh -l taito_username -L 3306:kaivos.csc.fi:3306 taito-shell.csc.fi -N The N option in the end of the connection command blocks the command shell after the connection is established. Once the connection is closed also the port forwarding becomes disabled. Note that the ssh command above uses your CSC user account and password, not the database user account. In Windows machines you can use e.g. plink program to open the tunnel. Plink can only be used through the command prompt. Below is a sample command for using plink: plink -L 3306:kaivos.csc.fi:3306 taito_username@taito-shell.csc.fi 5.2 Tunneling MySQL port with Scientist s User Interface If you don t have ssh or plink program available in your windows computer you can use the GSI SSH console tool in the Scientist s User Interface to do the secure server port forwarding. First log into the Scientist s User Interface service by clicking the link below https://sui.csc.fi Once you have entered to the service, launch the SSH Console by clicking the icon in the Services Desktop or by selecting the tool from the Services Menu ( under the Computing Environment sub-menu). When the console is launched, it asks for your user account and the name of the server you want to connect. The password will be asked only later on. For example: Username: csc_username Remote Host: taito-shell.csc.fi And then launch the SSH console. When the Java based console program starts you may need to accept launching the process in you local machine and allow the process connect CSC. When the console is running it will ask you to choose the authentication method to be used. Select: password and press Proceed. Figure 1: Selecting authentication for SSH Console. After this a Password Authentication window opens. Type in you password and press OK. 9
Figure 2: Password authentication for SSH Console. You can use the console to use the CSC servers, but it contains also other tools than just the command line client. You can use it for example for file transport or secure tunneling of server ports. In the case of port tunneling go to the Tools menu of the client window and select: Tools Secure Tunneling To open the MySQL tunnel, press the New Tunnel button in the Tunnel management window. To use Kaivos, you should define a tunnel with following settings Name: Mysql Tunnel type: Outgoing Linstenig Port(local): 3306 Destination Host: localhost Destination Port: 3306 5.3 Remote MySQL connections The commands in capter 5.1 and 5.2 define that the communication to port 3306 (the default port of MySQL) in you local computer is transported to the MySQL port of kaivos.csc.fi through taito-shell.csc.fi server. As long as the connection to taito-shell.csc.fi is active the database in kaivos.csc.fi can be accessed from your local computer using the same MySQL commands as described above for Taito (assuming that you have the MySQL client program installed in your local computer). The only difference compared to previous command examples is that the host section (-h) should now point to host 127.0.0.1 that refers to your local host, instead of kaivos.csc.fi. So for example the syntax to open an interactive MySQL session would now be: mysql -u db_user_account p -h 127.0.0.1 --local database_name And the syntax for mysqlimport: mysqlimport -h 127.0.0.1 -u db_user_account --local --compress \ --password database_name input_file.table In the same way, you can make your databases visible in your local computer using locally installed graphical MySQL interfaces, like MySQL Workbench. The only major limitation with the port forwarding is that normally ssh tunnels are not very stable. You may need to reopen the ssh connection every now 10
and then and you should not trust that the tunnel remains usable for several hours. For the security reasons we recommend that you always close the ssh connection when you stop using the database. 6. Using Perl MySQL API at CSC The Perl MySQL API is available in Taito at CSC, but it is not officially supported. The following tasks are usually performed by a Perl database script: Perl DBI module is imported The connection to the MySQL server is opened The statements are executed and their results sets are retrieved The server connection is closed The following guidance assumes that you have a database user account to the database service at CSC. If you are accessing another MySQL server replace the server name (kaivos.csc.fi) in script the server name you are using. Write a MYSQL database access script Use your favourite text editor to create a named script file e.g. mydb_script.pl. Then copy the following text to the script. # mydb_script.pl script to show MySQL server version use strict; use DBI; my $dbh = DBI->connect ("DBI:mysql:your_database_name:kaivos.csc.fi", "your_database_user_account", "your_database_password") or die "Cannot connect:". $DBI::errstr; my $sth = $dbh->prepare ("SELECT VERSION()") or die "Cannot prepare:". $dbh->errstr(); $sth->execute () or die "Cannot execute: ". $sth->errstr(); while (my @row = $sth->fetchrow_array()) { print "@row\n"; } $dbh->disconnect (); The connection to the database is established by invoking the connect() method with the connection parameters. These parameters are: the database to use, database server, database user account and database password. Replace these values corresponding your database, database user account and database password. The prepare() method prepares the SQL statement and execute() method sends statement to the database server. The fetchrow_array() method retrieves rows from the result set in a loop, and the resulting rows are printed. Finally the connection is closed by disconnect() method. Running the MYSQL database access script Run the script from the command line with Perl interpreter. environment, because it contains the required modules. We recommend using bioperl in CSC module load biokit perl mydb_script.pl 11
or add following to the beginning of the script: #!/appl/bio/bioperl/5.16.3/bin/perl Then make the script executable and run it directly: chmod +x mydb_script.pl./mydb_script.pl The statements issuing methods The prepare() method is for preparing the SQL statement and execute() method is for issuing SQL statements. However, you can use the do() method for non repeated non-select statement (e.g. INSERT, UPDATE, DELETE), because no data is returned from the database: $rows_affected = $dbh->do("update your_table SET foo = foo + 1"); Transaction By default AutoCommit mode is on. You do not need to use commit() method while making transactions. Only InnoDB storage engine is transactional. The default MyISAM is a non-transactional storage engine. 7. Using Python MySQL API at CSC The Python MySQL API is available in Taito at CSC, but it is not officially supported. taito.csc.fi and write a Python script that access to the database server. The following tasks are usually performed by a Python database script: Login to MySQLdb module is imported The connection to the MySQL server is opened The statements are executed and their results sets are retrieved The server connection is closed The following guidance assumes that you have a database user account to the database service at CSC. If you are accessing another MySQL server replace the server name (kaivos.csc.fi) in script the server name you are using. Write a MYSQL database access script Use your favourite text editor to create a named script file e.g. mydb_script.py. Then copy the following text to the script. 12
# mydb_script.py script to show MySQL server version import sys import MySQLdb try: conn = MySQLdb.connect (host = "kaivos.csc.fi", user = "your_database_user_account", passwd = "your_database_password", db = "your_database_name") except MySQLdb.Error, e: print "Error %d: %s" % (e.args[0], e.args[1]) sys.exit (1) try: cursor = conn.cursor () cursor.execute ("SELECT VERSION()") row = cursor.fetchone () print "server version:", row[0] cursor.close () except MySQLdb.Error, e: print "Error %d: %s" % (e.args[0], e.args[1]) sys.exit (1) conn.close () First of all the script imports modules sys and MySQLdb. The module MySQLdb is a driver to the MySQL database, and it must be imported before connection to database can be created. The sys is needed for error handling by using the try statement and an except clause that contains the error-handling code. The MySQLdb.Error exception class stores the error information to the variable e. If an exception occurs, the error information is printed from e.args as a two-element tuple. The first value is numeric error code and the second value is a string describing the error. If an exception does not occur, the connection to the database is established by invoking the connect() method with the connection parameters. These parameters are the database server, database user account, database password and database use want to use. Replace user, passwd and db values corresponding your database user account, database password and database. The connect() method returns a connection object, which invokes cursor() method to create cursor object. The execute() method of cursor object sends SQL statement to the database server and fetchone() method receives results of the SQL statement. The SELECT VERSION() statement contains a single value as a tuple, which the script prints. Finally the cursor and connection are closed by close() method. Running the MYSQL database access script Run the script from the command line with Python interpreter: python mydb_script.py or add following to the beginning of the script: #!/appl/opt/python/2.7.3-gcc-shared/bin/python Then make the script executable and run it directly: chmod +x mydb_script.py./mydb_script.py 13
The statements issuing methods The execute() method is for issuing SQL statements e.g. cursor.execute ("DROP TABLE IF EXISTS table_name") cursor.execute (""" CREATE TABLE table_name ( col1_name col1_type, ) """) col2_name col2_type If your SQL statement invoked with execute() method returns a result set, you must retrieve the result set with different methods. The method fetchone() is used to get the rows one at a time, and the method fetchall() is used to get rows all at once. The result set retrieval using fetchone() cursor.execute ("SELECT col1, col2 FROM table") while (1): row = cursor.fetchone () if row == None: break print "%s, %s" % (row[0], row[1]) The result set retrieval using fetchall() cursor.execute ("SELECT col1, col2 FROM table") rows = cursor.fetchall () for row in rows: print "%s, %s" % (row[0], row[1]) The rows can be fetched as dictionaries, but dictionary access requires a different kind of cursor. When using dictionaries you can access the column values by names. cursor = conn.cursor (MySQLdb.cursors.DictCursor) cursor.execute ("SELECT col1, col2 FROM table") result_set = cursor.fetchall () for row in result_set: print "%s, %s" % (row["col1"], row["col2"]) Transactions If you are using the default MyISAM storage engine, you do not need to worry about transactions, because MyISAM is a non-transactional storage engine. However, if you are using transactional storage engine like InnoDB, commit your data changes with commit () method. Commit updates after cursor closing but before closing connection to the database. cursor.close () conn.commit () conn.close () Use your favourite text editor to create a named script file e.g. mydb_script.py. Then copy the following text to the script. 14