PROJECT INFRASTRUCTURE AND BASH INTRODUCTION MARKUS PILMAN<

Transcription

1 PROJECT INFRASTRUCTURE AND BASH INTRODUCTION MARKUS PILMAN<

2 ORGANIZATION Tutorials on Tuesdays - Sometimes, will be announced In General: no exercise sessions (unless you get an from your TA) Mapping Student -> TA is published on course web page - find your TA there On Thursdays: Office hour: write an before you show up

3 DRYAD CLUSTER Use this cluster only for testing You can connect to it any time with ssh 16 machines (dryad01 - dryad16) ssh nethzname@dryad01

4 THE PROJECT No Team-Work Use: Java PostgreSQL Ant (no Eclipse/Netbeans/IntelliJ project files for submission!) Not allowed are: Usage of external libraries (except Postgresql JDBC driver) JPA, RMI Running anything else on the DB-machine than Postgresql (no Java-Proxy)

5 WHAT ABOUT JUNIT? Your end product is not allowed to use anything but JDK+Postgresql BUT: you are free to use any other library/framework for testing and benchmarking So feel free to use JavaScript for testing and Brainfuck to configure your machines

6 THE COMPUTER SYSTEMS YOU SHOULD USE You have three Infrastructures at your fingertips: Your private machine (or ETH-machines in the PC rooms), for development and simple testing. A cluster provided by the Systems Group for distributed testing The Amazon Computing Cloud for benchmarks

7 BASH TUTORIAL

8 BASICS Bash is the command line interface used by most linux systems On Windows: Use Putty to connect to a remote machine (Cluster and/or Amazon) Most Bash-Commands are simple programs that get executed

9 RUNNING A COMMAND command arg1 arg2 # starts a command - prompt returns as soon as the command finished command arg1 arg2 & # runs the command in the background command arg1 arg2 > out # runs the command, write standard out to file called "out" command arg1 arg2 command2 arg # runs the command and "pipes" the output into command2

10 EXAMPLES ls # prints files and directories in current directory ant run & # starts your middleware server as a background progress java -jar MyTests.jar > testout.txt # runs a java program and writes the test results in a file ls grep Test # lists all files containing the string "Test" find. -name \*.java xargs grep "class MyClass" # Searches for all java source files which contain # "class MyClass" within the file (not file name!)

11 CONDITIONAL BRANCHES MYVAR=foo # set variable MYVAR to foo export MYVAR=bar # set MYVAR, child processes will see it as well if [ $MYVAR = foo ] then echo "MYVAR is foo" elif [ $MYVAR = bar ] then echo "Bar" else echo "Something is odd" fi

12 LOOPS There are several ways of looping in Bash, some are: for i in $( ls ); do echo item: $i done for i in `seq 1 10`; do echo $i done COUNTER=0 while [ $COUNTER -lt 10 ]; do echo The counter is $COUNTER let COUNTER=COUNTER+1 done

13 function quit { exit } function e { echo $1 } e "hello" quit e "world" # does not get executed FUNCTIONS

14 ABOUT USING BASH FOR AUTOMATION Bash is great for simple tasks If you forget a space somewhere, you might get weird results Bash scripts are horrible to debug If it get's complicated, you might consider something else: Python should be installed on all machines Perl might be an alternative (should also be available everywhere) Tcl with the expect extension is great for the stuff you will be doing You can also use Java for automation tasks. BUT: don't over engineer! Doing stuff manually with Cluster SSH and Screen might be good enough for this project.

15 SCREEN AND TMUX When you log out of a machine, your running processes might get killed This can screw up experiments (you might lose a connection, your notebook might crash...) You might want to use more than one session on a remote machine: Use screen or tmux Short demo!

16 SSH SSH (secure shell) is a software that allows you to remotely log in to another machine scp can copy files from one machine to another machine ssh mpilman@optimus.ethz.ch # Log in to machine called optimus.ethz.ch as user mpilman ssh mpilman@optimus.ethz.ch ls # execute command "ls" on optimus as user mpilman # This is great for scripting!!! scp project.jar mpilman@optimus.ethz.ch:~/asl # Copy project.jar to ~/asl on optimus

17 PUBLIC KEY AUTHENTICATION Writing the password is often a problem if you want to run scripts that execute commands on several machines. It is therefore a good idea, to generate a key-pair and place the public key on the machines. For Amazon, this is required anyway. ssh-keygen -t rsa -C "mpilman@inf.ethz.ch" # Creates a new ssh key with a given label Generating public/private rsa key pair. Enter file in which to save the key (/Users/you/.ssh/id_rsa): [Press enter] Enter passphrase (empty for no passphrase): [Press enter] Enter same passphrase again: [Press enter again] # A passphrase would be better for security, # but will make benchmarking more complicated for you. # Now you need to copy the PUBLIC key to the servers: cat ~/.ssh/*.pub ssh mpilman@optimus.ethz.ch 'umask 077; cat >>.ssh/authorized_keys'

18 CLUSTER SSH

19 INSTALLING AND RUNNING POSTGRESQL Use Postgresqls documentation to figure out about users and configuarion On your local machine, you can install Postgresql with an installer (Windows) or with your package amanger (Linux). For OS X go to use Mac Ports, or homebrew On the dryad machines, you have to compile postgresql manually and install it into a directory where you have write access to.

20 COMPILING POSTGRESQL mkdir /mnt/local/username # create a new directory # download postgresql source and copy it with scp to the dryad machine tar xjf postgresql tar.bz2 cd postgresql-9.3.5/./configure --prefix="/mnt/local/username/postgres" # the prefix tells the system where you want to install postgres make # this compiles postgresql - it might take a while make install # install it

21 RUNNING LOCALLY INSTALLED POSTGRESQL LD_LIBRARY_PATH=/mnt/local/username/postgres/lib export LD_LIBRARY_PATH # make sure Linux can find the required shared libraries /mnt/local/username/postgres/bin/initdb -D /mnt/local/username/postgres/db/ # this will create a new database # make sure to create it on a local disk and NOT in NFS (not your home directory) /mnt/local/username/postgres/bin/postgres -D /mnt/local/username/postgres/db/ -p PORTNUMBER \\ -i -k /mnt/local/username/ # PORTNUMBER should be a random number bigger than it might fail if another user # runs postgres on that port already - just use another port number # The -k parameter let's postgres write the Unix socket into another directory (default is /tm # if this is not set, postgres might fail because another user wrote a unix socket already.

22 QUESTIONS?