Using & Installing So:ware on Odyssey
|
|
- Zoe Matthews
- 5 years ago
- Views:
Transcription
1 Using & Installing So:ware on Odyssey Bob Freeman, PhD Dir. Research Technology Opera1ons (HBS) formerly RC Facilitator Slide 1
2 Goals Objec&ves Feel knowledgeable about compute & so:ware environment Understand and create Lmod module files Know how to handle different types of programs & scripts Customize libraries for common scrip1ng languages Understand basics on so:ware development Know common prac1ces for compiling so:ware Strategically Work smarter, bever, faster... and to Think Differently Enable you to be successful with your research! Slide 2
3 Overview 1. Environment basics Login scripts & environment variables 2. Module system How they work, what they look like, and what they do for you 3. Crea1ng your own modules 4. Java, Python, & Perl applica1ons 5. Upda1ng local modules & packages 6. Version control 7. Using FASRC / precompiled libraries 8. Compiling So:ware Methods Flags, op1ons, and environment variables Troubleshoo1ng Slide 3
4 Environment basics When you login, Unix executes certain steps for your interac1ve sessions Startup files are read Command prompts are set up Aliases expanded. Startup files set up default values for your environment /etc/profile.bash_profile.bash_login.profile.bashrc The only things that really need to be in.bash_profile are environment variables and their exports and commands these aren t defini1ons but actually run or produce out- put when you log in. Op1on and alias defini1ons should go into the environment file (.bashrc) Slide 4
5 Environment basics ~]$ cat ~/.bash_profile #.bash_profile # Get the aliases and functions if [ -f ~/.bashrc ]; then. ~/.bashrc fi # User specific environment and startup programs How shells behave: Slide 5 a b c Shell: source alice hatter gryphon Shell: Subshell: Shell: Subshell: alice alice & Figure 4-1. Ways to run a shell script hatter hatter gryphon gryphon Chapter 3: Learning the bash Shell (O'Reilly) source.bash_profile & more, keeps all sekngs in same context main shell waits; subshell gets.bashrc only, knows about exports; all is lost a:er exit same as b, but main shell can keep doing work with other in bkg
6 Environment Basics ~]$ cat ~/.bashrc #.bashrc # Source global definitions if [ -f /etc/bashrc ]; then. /etc/bashrc fi # User specific aliases and functions command aliases alias l.='ls -d.*' alias lt='ls -alt' alias lthead='ls -alt head' alias llhead='ll head' alias which='alias /usr/bin/which --tty-only --read-alias --show-dot --show-tilde' alias get_interact1='srun --pty --x11=first --mem=1000 -p interact -t 0-6:00 /bin/bash' alias ssqueue='/usr/bin/squeue -o "%.18i %.9P %.8j %.8u %.2t %.10M %.6D %R %C"' alias sacct_ram="sacct --format=jobid,jobname,reqmem,maxrss,elapsed" alias cd_regal='cd /n/regal/rc_admin/bfreeman' export PATH=$PATH:$HOME/bin export PATH=$PATH:$HOME/apps/parafly/bin source new-modules.sh module load fasrc export EDITOR=nano adding executables to where systems looks op1ng in to new modules sekng default editor # setup local Perl environment export LOCALPERL=$HOME/apps/perl export PERL5LIB=$LOCALPERL:$LOCALPERL/lib/perl5:$PERL5LIB export PERL_MM_OPT="INSTALL_BASE=$LOCALPERL" export PERL_MP_OPT="--install_base $LOCALPERL" export PATH="$LOCALPERL/bin:$PATH" module load perl-modules # setup for local Python environment Slide 6 personal perl modules
7 Common Environment Variables You may run across these in your Unix & cluster endeavors... For variables that include filesystem loca1on (paths), these are colon (':') delimited, and usually do not contain a trailing '/'. Order is important! Pre- pending your local paths will supersede system executables of the same name Variable PATH Explana&on Ordered loca1ons that the system looks for programs you wish to execute HOME TEMP TMP TMPDIR DISPLAY EDITOR Your home folder. Usually not a good idea to change Local directory used for temporary file crea1on Used to direct X11 GUI data to the appropriate drawing screen The name of the user's preferred text editor. Used by command- line programs such as git LIBRARY_PATH LD_LIBRARY_PATH CLASSPATH PWD USER PAGER List of directories in which library files should be searched for List of directories for Java jar files / class files Your current directory Your account / login name Name of the u1lity used to display long text by commands such as man Slide 7
8 Overview 1. Environment basics Login scripts & environment variables 2. Module system How they work, what they look like, and what they do for you 3. Crea1ng your own modules 4. Java, Python, & Perl applica1ons 5. Upda1ng local modules & packages 6. Version control 7. Using FASRC / precompiled libraries 8. Compiling So:ware Methods Flags, op1ons, and environment variables Troubleshoo1ng Slide 8
9 Lmod Module System So:ware is loaded incrementally using modules, to set up your shell environment (e.g. PATH, BLASTDB, and other environment variables) Using the Harvard- modified, TACC module system Lmod: New system is s1ll opt- in. Strongly suggested reading: hvp://fasrc.us/rclmod source new-modules.sh module load fastqc/1.0.0-fasrc01 module load fastqc # for opt-in folks # recommended!! # most recent version module-query fastqc # recommended!! module-query --full-text fastqc # gives more detail module spider fastqc # find details on software module avail 2>&1 grep -i fastqc # find titles/defaults So:ware search capabili1es similar to module-query are also available on the RC Portal! Module loads best placed in SLURM batch scripts: Keeps your interac1ve working environment simple Is a record of your research workflow (reproducible research!) Keep.bashrc module loads sparse, lest you run into so:ware and library conflicts Slide 9 hvp://fasrc.us/rclmod
10 Lmod Module System LMod: Environmental Modules System from hvps:// development/tacc- projects/lmod Provide a convenient way to dynamically change the users' environment through modulefiles (Lua- based scrip1ng file). This includes easily adding or removing directories to the PATH environment variable. A modulefile contains the necessary informa1on to allow a user to run a par1cular applica1on or provide access to a par1cular library. dynamically changes environment without logging out and back in. applica1ons modify the user's path to make access easy. Library packages provide environment variables that specify where the library and header files can be found. Packages can be loaded and unloaded cleanly through the module system. All the popular shells are supported: bash, ksh, csh, tcsh, zsh. Also available for perl and python. It is also very easy to switch between different versions of a package or remove it. Slide 10 hvp://fasrc.us/rclmod
11 Modules: How do they work?(1) ~]$ module load trinity/2.0.6-fasrc01 ~]$ which Trinity /n/sw/fasrcsw/apps/core/trinityrnaseq/2.0.6-fasrc01/trinity ~]$ ll /n/sw/fasrcsw/apps/core/trinityrnaseq/2.0.6-fasrc01 total 974 drwxr-xr-x 12 root root 514 May drwxr-xr-x 7 root root 163 Oct 30 12:51.. drwxr-xr-x 5 root root 118 May Analysis drwxr-xr-x 4 root root 171 May Butterfly drwxr-xr-x 9 root root 1488 May Chrysalis drwxr-xr-x 2 root root 695 May galaxy-plugin drwxr-xr-x 3 root root 540 May hpc_conf drwxr-xr-x 4 root root 501 May Inchworm -rw-r--r-- 1 root root 1490 May LICENSE -rw-r--r-- 1 root root 1492 May LICENSE.txt -rw-r--r-- 1 root root 1737 May Makefile -rw-r--r-- 1 root root 2113 May modulefile.lua -rw-r--r-- 1 root root 87 May notes drwxr-xr-x 7 root root 1384 May PerlLib -rw-r--r-- 1 root root 105 May README -rw-r--r-- 1 root root 76 May README.md -rw-r--r-- 1 root root May Release.Notes drwxr-xr-x 12 root root 444 May sample_data -rwxr-xr-x 1 root root May Trinity drwxr-xr-x 11 root root 620 May trinity-plugins drwxr-xr-x 7 root root 634 May util [bfreeman@sa01 ~]$ pushd!$ # cool: save current dir and use last param of last cmd pushd /n/sw/fasrcsw/apps/core/trinityrnaseq/2.0.6-fasrc01 /n/sw/fasrcsw/apps/core/trinityrnaseq/2.0.6-fasrc01 ~/source_tarballs ~ Slide 11
12 Modules: How do they work? (2) fasrc01]$ cat modulefile.lua local helpstr = [[ trinityrnaseq fasrc01 De-novo assembler from RNA-Seq from Broad Inst. et al. ]] help(helpstr,"\n") whatis("name: trinityrnaseq") whatis("version: fasrc01") whatis("description: De-novo assembler from RNA-Seq from Broad Inst. et al.") info for module- query & help program metadata ---- prerequisite apps (uncomment and tweak if necessary) for i in string.gmatch("bowtie/1.1.1-fasrc01 samtools/1.1-fasrc02 perl-modules/ fasrc11","%s+") do if mode()=="load" then a = string.match(i,"^[^/]+") if not isloaded(a) then load(i) end end end ---- environment changes (uncomment what is relevant) prepend_path("path", "/n/sw/fasrcsw/apps/core/trinityrnaseq/2.0.6-fasrc01") setenv("trinity_home", "/n/sw/fasrcsw/apps/core/trinityrnaseq/2.0.6-fasrc01") we load dependencies for you Magic! we change your environment as needed --prepend_path("cpath", --prepend_path("fpath", --prepend_path("infopath", --prepend_path("ld_library_path", --prepend_path("library_path", --prepend_path("ld_library_path", --prepend_path("library_path", --prepend_path("manpath",... "/n/sw/fasrcsw/apps/core/trinityrnaseq/2.0.6-fasrc01/include") "/n/sw/fasrcsw/apps/core/trinityrnaseq/2.0.6-fasrc01/include") "/n/sw/fasrcsw/apps/core/trinityrnaseq/2.0.6-fasrc01/info") "/n/sw/fasrcsw/apps/core/trinityrnaseq/2.0.6-fasrc01/lib") "/n/sw/fasrcsw/apps/core/trinityrnaseq/2.0.6-fasrc01/lib") "/n/sw/fasrcsw/apps/core/trinityrnaseq/2.0.6-fasrc01/lib64") "/n/sw/fasrcsw/apps/core/trinityrnaseq/2.0.6-fasrc01/lib64") "/n/sw/fasrcsw/apps/core/trinityrnaseq/2.0.6-fasrc01/man") Slide 12
13 Modules: Why Use Hierarchies? Use of groupings is important for proper func1oning programs! Libraries built with one compiler need to be linked with applica1ons with the same compiler version. For HPC there are libraries called Message Passing Interface (MPI) that allow for efficient communica1ng between tasks on a distributed memory computer with many processors. Parallel libraries and applica1ons must be built with a matching MPI library and compiler. Instead of using a flat namespace, we can use module hierarchies! Simple technique because once users choses a compiler and MPI implementa1on, they can only load modules that match that compiler and MPI implementa1on. FASRC follow's TACC's conven1on: $MODULEPATH_ROOT/{Core,Comp,MPI} # /n/sw/fasrcsw/modulefiles Core: Comp: MPI: compiled so:warelibraries that should work in all environments so:ware/libraries 1ed to a specific compiler and version compiler- specific & MPI- specific flavors of so:ware/libraries Slide 13
14 Modules: Hierarchies modulefiles]$ module-query raxml raxml Description: RAxML (Randomized A(x)ccelerated Maximum Likelihood) is a program for sequential and parallel Maximum-Likelihood-based inference of large phylogenetic trees. It can also be used for postanalyses of sets of phylogenetic trees, analyses of alignments and, evolutionary placement of short reads. It has originally been derived from fastdnaml which in turn was derived from Joe Felsentein s dnaml which is part of the PHYLIP package. Versions: raxml/8.1.5-fasrc03... Comp raxml/8.1.5-fasrc02... MPI raxml/7.7.6.bib-fasrc02... Core raxml/7.3.0-fasrc01... Core To find detailed information about a module, enter the full name. For example, module-query raxml/7.3.0-fasrc01 Slide 14
15 Modules: Hierarchies modulefiles]$ module-query raxml/8.1.5-fasrc raxml : raxml/8.1.5-fasrc Description: RAxML (Randomized A(x)ccelerated Maximum Likelihood) is a program for sequential and parallel Maximum-Likelihood-based inference of large phylogenetic trees. It can also be used for postanalyses of sets of phylogenetic trees, analyses of alignments and, evolutionary placement of short reads. It has originally been derived from fastdnaml which in turn was derived from Joe Felsentein s dnaml which is part of the PHYLIP package. Build comments: Building with SSE3 instructions. Some, but not all compute nodes have AVX. Spec file can be used properly as either Comp or MPI. When building as Comp, the CC line is removed from the Makefile so that the environment value is picked up. This module an be loaded as follows: module load gcc/4.8.2-fasrc01 openmpi/1.8.1-fasrc04 raxml/8.1.5-fasrc02 module load gcc/4.8.2-fasrc01 openmpi/ fasrc01 raxml/8.1.5-fasrc02 module load gcc/4.8.2-fasrc01 openmpi/1.8.3-fasrc02 raxml/8.1.5-fasrc02 module load gcc/4.8.2-fasrc01 mvapich2/2.0-fasrc03 raxml/8.1.5-fasrc02 module load gcc/4.8.2-fasrc01 mvapich2/2.0-fasrc02 raxml/8.1.5-fasrc02 module load intel/ fasrc01 openmpi/1.8.1-fasrc04 raxml/8.1.5-fasrc02 module load intel/ fasrc01 openmpi/ fasrc01 raxml/8.1.5-fasrc02 module load intel/ fasrc01 openmpi/1.8.3-fasrc02 raxml/8.1.5-fasrc02 module load intel/ fasrc01 mvapich2/2.0-fasrc03 raxml/8.1.5-fasrc02 module load intel/ fasrc01 mvapich2/2.0-fasrc02 raxml/8.1.5-fasrc02 [bfreeman@sa01 modulefiles]$ module load gcc/4.8.2-fasrc01 openmpi/ fasrc01 raxml/8.1.5-fasrc02 [bfreeman@sa01 modulefiles]$ which raxml /n/sw/fasrcsw/apps/mpi/gcc/4.8.2-fasrc01/openmpi/ fasrc01/raxml/8.1.5-fasrc02/bin/raxml Slide 15
16 Modules: Make your own! Why? Install newer version of open source so:ware than is currently available in central loca1ons Easily change version of applica1ons or libraries while being developed BeVer documenta1on for what so:ware is available. It is easy to install a new version of some sodware and place it in your personal PATH and forget about it. At least when it is in a module, it be listed in the loaded modules. Can be done in 3 easy steps 1. Add loca1on of local module files to module path 2. Create the module file with lua code for the appropriate program/package 3. Do the new module load (Voila!) 4. Include the op1onal modules path in your.bashrc login script Slide 16 hvps:// development/tacc- projects/lmod/advanced- user- guide
17 Modules: Make Your Own! # assumes that program & packages are in $HOME/apps [bfreeman@sa01 ~]$ ll apps/thin_fastq-1.0.0/ total 232 drwxr-xr-x 4 bfreeman rc_admin 46 Jan 26 08:41. drwxr-xr-x 17 bfreeman rc_admin 932 Jan 27 11:18.. drwxr-xr-x 3 bfreeman rc_admin 905 Jan 29 10:03 bin drwxr-xr-x 3 bfreeman rc_admin 32 Jan 26 08:41 include drwxr-xr-x 3 bfreeman rc_admin 905 Jan 29 10:03 lib # prepends your path to MODULEPATH environment variable. Can locate these anywhere. $ mkdir $HOME/modulefiles $ module use $HOME/modulefiles # assuming a program thin_fastq at $HOME/apps/thin_fastq-1.0.0/bin $ cd ~/modulefiles $ mkdir thin_fastq $ cd!$ # module file gets home directory from the environment # then uses Lmod 'pathjoin' function to join strings together # to pre-pends this program's location to the user's PATH $ cat > lua local home = os.getenv("home") local pkg = pathjoin(home,"apps/thin_fastq-1.0.0/bin") prepend_path("path", pkg) ^D # now load in the new module $ module load thin_fastq # this line will add the local (user) module environment for future logins $ echo 'module use $HOME/modulefiles' >> ~/.bashrc Slide 17
18 Overview 1. Environment basics Login scripts & environment variables 2. Module system How they work, what they look like, and what they do for you 3. Crea1ng your own modules 4. Java, Python, & Perl applica1ons 5. Upda1ng local modules & packages 6. Version control 7. Using FASRC / precompiled libraries 8. Compiling So:ware Methods Flags, op1ons, and environment variables Troubleshoo1ng Slide 18
19 Java Programs FASRC will no longer install Java apps and Python scripts they will probably not work correctly in read- only sec1ons of the cluster Python scripts use pip-install or python setup.py & doesn t work with some setups For Java Download the *.jar files or the install files into a home or lab apps/ or bin/ directory Include the java CLASSPATH statement in your.bashrc, OR Set up a bash environment variable in your.bashrc Call the so:ware using the java command, poin1ng to the appropriate rou1ne cd ~ mkdir p apps; cd apps wget longurl /Trimmomatic-0.36.zip unzip Trimmomatic-0.36.zip ln s Trimmomatic-0.36 trimmomatic export TRIMMOMATIC=$HOME/apps/trimmomatic echo 'export TRIMMOMATIC=$HOME/apps/trimmomatic' >> ~/.bashrc # in SLURM script or on command line module load java/1.8.0_45-fasrc01 cd ~/myfastqdirectory; mkdir trimmed # minheap (-Xms) and maxheap (-Xmx) options are optional but useful in some cases!! java -Xms128m Xmx4g -jar $TRIMMOMATIC/trimmomatic-0.32.jar SE -threads 1 \ PSG177_TGACCA.fastq.gz trimmed/psg177_tgacca.fastq ILLUMINACLIP:TruSeq3-PE.fa:2:40:15 LEADING:3 TRAILING:3 \ SLIDINGWINDOW:4:20 MINLEN:25 Slide 19 hvp://fasrc.us/installsw
20 Python Programs For Python we recommend: Use the standard module load python/2.7.6-fasrc01 for pulling in default modules Use the Anaconda environment for customizing modules & versions Mul1ple custom environments can be set up for home or lab folders (e.g. development or produc1on code). Check conda op1ons for 'non- standard' loca1ons module load python/2.7.6-fasrc01 # for any python version conda create -n ENV_NAME --clone="$python_home # created at ~/env/env_name # load in environment source activate ENV_NAME # load in new package conda install MYPACKAGE # local environment is default # replace MYPACKAGE # update a specific package # 'remove' first if you hit update problems conda update MYPACKAGE # other methods after source activate pip install MYPACKAGE python setup.py install MYPACKAGE Slide 20 hvp://fasrc.us/installsw
21 R / Python / Perl Libraries We've already covered Python libraries. R and Perl are much simpler! R: Use the R_LIBS_USER environment variable # load R, default packages, & set local install dir (must already exist!) module load R_packages/3.2.0-fasrc01... # can put this in.bashrc, but best to do this after R module load export R_LIBS_USER=$HOME/apps/R:$R_LIBS_USER # R install.packages('deplyr') # inside R For Perl, we can do the same thing! # load Perl, default modules, & set local install dir (must already exist!) module load perl-modules/ fasrc11 # can put these in your.bashrc export LOCALPERL=$HOME/apps/perl export PERL5LIB=$LOCALPERL:$LOCALPERL/lib/perl5:$PERL5LIB export PERL_MM_OPT="INSTALL_BASE=$LOCALPERL" export PERL_MB_OPT="--install_base $LOCALPERL" export PATH="$LOCALPERL/bin:$PATH" # and now do easy, local installs with cpan cpan FASTAParse Slide 21 hvp://fasrc.us/installsw
22 Overview 1. Environment basics Login scripts & environment variables 2. Module system How they work, what they look like, and what they do for you 3. Crea1ng your own modules 4. Java, Python, & Perl applica1ons 5. Upda1ng local modules & packages 6. Version control 7. Using FASRC / precompiled libraries 8. Compiling So:ware Methods Flags, op1ons, and environment variables Troubleshoo1ng Slide 22
23 Version Control Version control is a system that records changes to a file or set of files over 1me so that you can recall specific versions later. Typically used for source code files In reality you can do this with nearly any type of file on a computer. Slide 23 from Git SCM book
24 Installing Your Own Software Users can compile so:ware in their /home or /groups directory, where they have permission Binaries just require unzipping (ie tar -zxvf *.tgz) Common compiling libraries are found as modules: gcc/4.8.2-fasrc01 intel/ fasrc01 boost/ fasrc01 gcc/4.8.2-fasrc01 llvm/3.5.1-fasrc02 gcc/4.8.2-fasrc01 openmpi/1.8.3-fasrc02 gcc/4.8.2-fasrc01 mvapich2/2.0-fasrc03 Common math libraries are also present but may be compiler- specific gcc/4.8.2-fasrc01 intel-mkl/ fasrc01 Slide 24 hvp://fasrc.us/installsw
25 Installing A Binary (Pre-compiled) Users can place so:ware in their /home or /groups directory, where they have permission Binaries just require unzipping (ie tar - zxvf.tgz) cd ~ mkdir p apps; cd apps wget longurl /mybinarysoftware-1.0.tgz tar -xvf mybinarysoftware-1.0.tgz ls mybinarysoftware-1.0 ln s mybinarysoftware-1.0 mybinarysoftware # add to.bashrc; assumes all binaries are in top level of this folder echo 'export PATH=$HOME/apps/myBinarySoftware:$PATH' >> ~/.bashrc export PATH=$HOME/apps/myBinarySoftware:$PATH # and now run it! mybinary Slide 25 hvp://fasrc.us/installsw
26 Installing From Source Code Users can compile so:ware in their /home or /groups directories, where they have permission Source requires untaring and (o:en) selec1ng a compiler cd ~ mkdir p apps; cd apps wget longurl /mybinarysoftware-1.0.tgz tar -xvf mybinarysoftware-1.0.tgz ls mybinarysoftware-1.0 ln s mybinarysoftware-1.0 mybinarysoftware cd mybinarysoftware less README module load gcc/4.8.2-fasrc01./configure --prefix=$home/apps/mybinarysoftware make make install # add to.bashrc; assumes all binaries are in top level of this folder echo 'export PATH=$HOME/apps/myBinarySoftware:$PATH' >> ~/.bashrc # NB! 's export PATH=$HOME/apps/myBinarySoftware:$PATH # and now run it! mybinary Slide 26
27 Understanding Compiling Code The GNU Build System, also known as the Autotools, is a suite of programming tools designed to assist in making source code packages portable to many Unix- like systems. The GNU Build System makes it possible to build many programs using a two- step process: configure followed by make: module load gcc/4.8.2-fasrc01./configure \ --prefix=$home/apps/mybinarysoftware make make install Slide 27
28 Understanding Compiling Code Slide 28
29 Source Compile Considerations Source Code (.f.f.f95.f03,.c ) main code or driver variables func1on calls subrou1ne Includes / headers (.i or.h) CPATH, C_INCLUDE Libraries files object files (.o) LIBRARY_PATH shared object files (.so) LD_LIBRARY_PATH archive files (.a) sekngs ldconfig LD_LIBRARY_PATH Compilers Intel: ifort, icc, icp, mkl, mpif90,... GNU: gfortran, gcc PGI: pgf90, Abaso: configure scripts make: Makefiles Slide 29
30 Compiling: Configure./configure sets up compile sekngs & target op1ons based on your system environment - - prefix= place to store architecture- independent binaries ($HOME/apps) - - exec- prefix= place to store architecture- dependent binaries - - bindir= where the executables go ( bin/) - - sbindir= system admin executables ( /sbin/) - - includedir= C header files ( include/) - - libdir= object code libraries ( lib/)./configure - - prefix=$home/apps/program./configure - - prefix=$home/apps will have program/bin program/lib etc will have bin/ lib/ include/ etc Some influen1al environment variables: PYTHON python program CXX C++ compiler command CXXFLAGS C++ compiler flags LDFLAGS linker flags if libraries in a nonstd directory <lib dir>: - L$HOME/apps/lib CPPFLAGS C/C++ preprocessor flags if headers in a nonstd directory <include dir>: - I$HOME/apps/include CC C compiler command: gcc CFLAGS C compiler flags: - O3 CXXCPP C++ preprocessor Slide 30
31 Compiling: Make make is a Unix tool to simplify building program executables from many modules. reads in rules (list of target entries) from a user created Makefile run the commands for that given rule will only re- build things that need to be re- built A very simple Makefile all: g++ main.cpp hello.cpp factorial.cpp -o hello install: cp hello $(PREFIX)/bin So make # runs the first (default) command all make install # runs the install rule Influen1al environment variables from previous page affect how the Makefile runs Slide 31
32 Install from Source Code source_tarballs]$ tar xvf tophat tgz source_tarballs]$ ls -al tophat total 1574 drwxr-xr-x 3 bfreeman rc_admin 596 Oct drwxr-xr-x 9 bfreeman rc_admin 1677 Apr 27 17:20.. -rw-r--r-- 1 bfreeman rc_admin Sep aclocal.m4 -rw-r--r-- 1 bfreeman rc_admin 1011 Sep AUTHORS -rw-r--r-- 1 bfreeman rc_admin 7621 Sep ax_boost_base.m4 -rw-r--r-- 1 bfreeman rc_admin 3404 Sep ax_boost_thread.m4 -rw-r--r-- 1 bfreeman rc_admin 0 May ChangeLog -rwxr-xr-x 1 bfreeman rc_admin Sep config.guess -rw-r--r-- 1 bfreeman rc_admin 3079 Sep config.h.in -rwxr-xr-x 1 bfreeman rc_admin Sep config.sub -rwxr-xr-x 1 bfreeman rc_admin Sep configure -rw-r--r-- 1 bfreeman rc_admin 3888 Sep configure.ac -rw-r--r-- 1 bfreeman rc_admin 5206 Sep COPYING -rwxr-xr-x 1 bfreeman rc_admin Sep depcomp -rw-r--r-- 1 bfreeman rc_admin 9478 May INSTALL -rwxr-xr-x 1 bfreeman rc_admin 9233 Sep install-sh -rw-r--r-- 1 bfreeman rc_admin 1469 May LICENSE -rw-r--r-- 1 bfreeman rc_admin 114 Sep Makefile.am -rw-r--r-- 1 bfreeman rc_admin Sep Makefile.in -rwxr-xr-x 1 bfreeman rc_admin Sep missing -rw-r--r-- 1 bfreeman rc_admin 6612 May NEWS -rw-r--r-- 1 bfreeman rc_admin 848 Sep README drwxr-xr-x 4 bfreeman rc_admin 2249 Oct src -rw-r--r-- 1 bfreeman rc_admin 2140 Sep THANKS [bfreeman@sa01 source_tarballs]$ cd!$ # cool trick! last option on prev line [bfreeman@sa01 source_tarballs]$./configure # OPTIONS HERE ARE IMPORTANT INSTALL and README present. Make sure you look thru these! No Makefile here, so have to run./configure Slide 32
33 Compiling: Using Libraries Libraries allow you to pull in pre- compiled func1ons and code to your programs Many are already installed on Odyssey: MKL, NetCDF, LAPACK, etc Should be able to pull in with a simple module load module load beagle/2.1.trunk-fasrc01 But also libraries that may be part of the OS /lib, /lib64 can be pull in via environment variables [bfreeman@sa01 tophat ]$ module display beagle/2.1.trunk-fasrc /n/sw/fasrcsw/modulefiles/core/beagle/2.1.trunk-fasrc03.lua: help([[beagle-2.1.trunk-fasrc03 A general purpose library for evaluating the likelihood of sequence evolution on trees ]], [[ ]]) whatis("name: beagle") whatis("version: 2.1.trunk-fasrc03") whatis("description: A general purpose library for evaluating the likelihood of sequence evolution on trees") prepend_path("cpath","/n/sw/fasrcsw/apps/core/beagle/2.1.trunk-fasrc03/include") prepend_path("fpath","/n/sw/fasrcsw/apps/core/beagle/2.1.trunk-fasrc03/include") prepend_path("ld_library_path","/n/sw/fasrcsw/apps/core/beagle/2.1.trunk-fasrc03/lib") prepend_path("library_path","/n/sw/fasrcsw/apps/core/beagle/2.1.trunk-fasrc03/lib") prepend_path("beagle_path","/n/sw/fasrcsw/apps/core/beagle/2.1.trunk-fasrc03") prepend_path("pkg_config_path","/n/sw/fasrcsw/apps/core/beagle/2.1.trunk-fasrc03/lib/pkgconfig") or -L and -l flags gcc -ltrhead gcc -L/lib64 Slide 33
34 Compiling: CMAKE Tools CMake is an open- source, cross- pla~orm family of tools designed to build, test and package so:ware. CMake is used to control the so:ware compila1on process using simple pla~orm and compiler independent configura1on files, and generate na1ve makefiles and workspaces that can be used in the compiler environment of your choice. Tips: it's best to build out of the so:ware tree (ie mkdir build; cd build; cmake..) -DCMAKE_INSTALL_PREFIX="$HOME/apps/thisapp" -DCMAKE_INCLUDE_PATH:STRING="$BOOST_INCLUDE;$ZLIB_INCLUDE" for include paths -DCMAKE_LIBRARY_PATH:STRING="$BOOST_LIB;$ZLIB_LIB" for lib paths There is a newer version of cmake available as a module seems to be necessary for most these days. module load cmake/ fasrc01 Slide 34
35 Compiling: BOOST Libraries Boost provides free peer- reviewed portable C++ source libraries that work well with the C++ Standard Library. intended to be widely useful usable across a broad spectrum of applica1ons license encourages both commercial and non- commercial use. We aim to establish "exis1ng prac1ce" and provide reference implementa1ons so that Boost libraries are suitable for eventual standardiza1on. Ten Boost libraries are included in the C++ Standards CommiVee's Library Technical Report (TR1) and in the new C++11 Standard. C++11 also includes several more Boost libraries in addi1on to those from TR1. More Boost libraries are proposed for standardiza1on in C++17. FASRC this as a module that is constantly updated: module load boost/ fasrc01 setenv("boost_home","/n/sw/fasrcsw/apps/core/boost/ fasrc01") setenv("boost_include","/n/sw/fasrcsw/apps/core/boost/ fasrc01/include") setenv("boost_lib","/n/sw/fasrcsw/apps/core/boost/ fasrc01/lib") prepend_path("cpath","/n/sw/fasrcsw/apps/core/boost/ fasrc01/include") prepend_path("ld_library_path","/n/sw/fasrcsw/apps/core/boost/ fasrc01/lib") prepend_path("library_path","/n/sw/fasrcsw/apps/core/boost/ fasrc01/lib") If an autotools build does not give you an opportunity to specify non- standard loca1ons for libraries (i.e. there is no - - boost- home or - - boost- lib op1on) can force it by overriding the CC or CXX variable e.g. export CC="gcc -I$BOOST_INCLUDE -L$BOOST_LIB" Slide 35
36 Compiling: Troubleshooting Slide 36
37 Compiling: Take-home Messages Use your.bashrc file to include aliases and environment variables Keep it clean and sparse Organize your home folder as best as possible Use environment variables where possible Pay aven1on to the order of paths in your environment variables Read the README and INSTALL files!! Compiles may go faster on /tmp or /scratch on compute nodes Don't use sudo Watch the compile logs very carefully Be sure to use the module/compiler hierarchy consistently Google is your best friend Next in line is Stack Overflow And then your friendly FASRC person Slide 37
38 Gekng Help RC Website & Documenta1on - - only authorita1ve source hvps://rc.fas.harvard.edu/ Submit a 1cket on the portal hvps://portal.rc.fas.harvard.edu/ Best way to help us to help you? Give us... Descrip1on of problem Addi1onal info (login/batch? par11on? JobIDs?) Steps to Reproduce (1., 2., 3...) Actual results Expected results OdyBot, for quick- fix problems hvp://odybot.org/ 3 8
39 Research Compu1ng Please talk to your peers, and We wish you success in your research! hvp://rc.fas.harvard.edu Harvard 3 9
Modules and Software. Daniel Caunt Harvard FAS Research Computing
Modules and Software Daniel Caunt Harvard FAS Research Computing 1 What is Research Computing? Faculty of Arts and Sciences (FAS) department that handles non-enterprise IT requests from researchers. (Contact
More informationWhat is Research Computing?
Spring 2017 3/19/17 Modules and Software Plamen Krastev, PhD Harvard - Research Computing 1 What is Research Computing? Faculty of Arts and Sciences (FAS) department that handles nonenterprise IT requests
More informationCONTAINERIZING JOBS ON THE ACCRE CLUSTER WITH SINGULARITY
CONTAINERIZING JOBS ON THE ACCRE CLUSTER WITH SINGULARITY VIRTUAL MACHINE (VM) Uses so&ware to emulate an en/re computer, including both hardware and so&ware. Host Computer Virtual Machine Host Resources:
More informationUsing the MaRC2 HPC Cluster
Using the MaRC2 HPC Cluster Manuel Haim, 06/2013 Using MaRC2??? 2 Using MaRC2 Overview Get access rights and permissions Starting a terminal session (Linux, Windows, Mac) Intro to the BASH Shell (and available
More informationLmod Documentation. Release 7.0. Robert McLay
Lmod Documentation Release 7.0 Robert McLay Jul 12, 2018 Contents 1 PURPOSE 1 2 OVERVIEW 3 3 Introduction to Lmod 5 4 Installing Lmod 23 5 Advanced Topics 45 6 Topics yet to be written 99 7 Indices and
More informationIntroduction to Linux
Introduction to Linux January 2011 Don Bahls User Consultant (Group Leader) bahls@arsc.edu (907) 450-8674 Overview The shell Common Commands File System Organization Permissions Environment Variables I/O
More informationCSC209. Software Tools and Systems Programming. https://mcs.utm.utoronto.ca/~209
CSC209 Software Tools and Systems Programming https://mcs.utm.utoronto.ca/~209 What is this Course About? Software Tools Using them Building them Systems Programming Quirks of C The file system System
More informationCISC 220 fall 2011, set 1: Linux basics
CISC 220: System-Level Programming instructor: Margaret Lamb e-mail: malamb@cs.queensu.ca office: Goodwin 554 office phone: 533-6059 (internal extension 36059) office hours: Tues/Wed/Thurs 2-3 (this week
More informationLinux Training. for New Users of Cluster. Georgia Advanced Computing Resource Center University of Georgia Suchitra Pakala
Linux Training for New Users of Cluster Georgia Advanced Computing Resource Center University of Georgia Suchitra Pakala pakala@uga.edu 1 Overview GACRC Linux Operating System Shell, Filesystem, and Common
More informationLinux Clusters Institute:
Linux Clusters Institute: 3 rd Party Software Management Instructor: Timothy Bouvet Title: System Engineer NCSA Email: tbouvet@illinois.edu 3 rd Party Software Management Topics: Best Practices Software
More informationLinux Command Line Interface. December 27, 2017
Linux Command Line Interface December 27, 2017 Foreword It is supposed to be a refresher (?!) If you are familiar with UNIX/Linux/MacOS X CLI, this is going to be boring... I will not talk about editors
More informationContents. Note: pay attention to where you are. Note: Plaintext version. Note: pay attention to where you are... 1 Note: Plaintext version...
Contents Note: pay attention to where you are........................................... 1 Note: Plaintext version................................................... 1 Hello World of the Bash shell 2 Accessing
More informationGNU/Linux 101. Casey McLaughlin. Research Computing Center Spring Workshop Series 2018
GNU/Linux 101 Casey McLaughlin Research Computing Center Spring Workshop Series 2018 rccworkshop IC;3df4mu bash-2.1~# man workshop Linux101 RCC Workshop L101 OBJECTIVES - Operating system concepts - Linux
More informationInstalling the Quantum ESPRESSO distribution
Joint ICTP-TWAS Caribbean School on Electronic Structure Fundamentals and Methodologies, Cartagena, Colombia (2012). Installing the Quantum ESPRESSO distribution Coordinator: A. D. Hernández-Nieves Installing
More informationSupercomputing environment TMA4280 Introduction to Supercomputing
Supercomputing environment TMA4280 Introduction to Supercomputing NTNU, IMF February 21. 2018 1 Supercomputing environment Supercomputers use UNIX-type operating systems. Predominantly Linux. Using a shell
More informationUsing the computational resources at the GACRC
An introduction to zcluster Georgia Advanced Computing Resource Center (GACRC) University of Georgia Dr. Landau s PHYS4601/6601 course - Spring 2017 What is GACRC? Georgia Advanced Computing Resource Center
More informationBuilding and Installing Software
Building and Installing Software On UD HPC Community Clusters William Totten Network & Systems Services Conventions when Installing Software Installation base directory /opt/shared /home/work/ lab/sw/name/version
More informationPlatform Migrator Technical Report TR
Platform Migrator Technical Report TR2018-990 Munir Contractor mmc691@nyu.edu Christophe Pradal christophe.pradal@inria.fr Dennis Shasha shasha@cs.nyu.edu May 12, 2018 CONTENTS: 1 Abstract 4 2 Platform
More informationCSC209. Software Tools and Systems Programming. https://mcs.utm.utoronto.ca/~209
CSC209 Software Tools and Systems Programming https://mcs.utm.utoronto.ca/~209 What is this Course About? Software Tools Using them Building them Systems Programming Quirks of C The file system System
More informationUnix Handouts. Shantanu N Kulkarni
Unix Handouts Shantanu N Kulkarni Abstract These handouts are meant to be used as a study aid during my class. They are neither complete nor sincerely accurate. The idea is that the participants should
More informationHPCC - Hrothgar. Getting Started User Guide User s Environment. High Performance Computing Center Texas Tech University
HPCC - Hrothgar Getting Started User Guide User s Environment High Performance Computing Center Texas Tech University HPCC - Hrothgar 2 Table of Contents 1. Login Shell... 3 2. Environment Variables...
More informationIntroduction to Unix: Fundamental Commands
Introduction to Unix: Fundamental Commands Ricky Patterson UVA Library Based on slides from Turgut Yilmaz Istanbul Teknik University 1 What We Will Learn The fundamental commands of the Unix operating
More informationDocumentation of the chemistry-transport model. [version 2017r4] July 25, How to install required libraries under GNU/Linux
Documentation of the chemistry-transport model [version 2017r4] July 25, 2018. How to install required libraries under GNU/Linux Contents 1 pnetcdf and NetCDF4 formats 2 1.1 Problems with NetCDF4 files..........................
More informationMcGill University School of Computer Science Sable Research Group. *J Installation. Bruno Dufour. July 5, w w w. s a b l e. m c g i l l.
McGill University School of Computer Science Sable Research Group *J Installation Bruno Dufour July 5, 2004 w w w. s a b l e. m c g i l l. c a *J is a toolkit which allows to dynamically create event traces
More informationCon$nuous Integra$on Development Environment. Kovács Gábor
Con$nuous Integra$on Development Environment Kovács Gábor kovacsg@tmit.bme.hu Before we start anything Select a language Set up conven$ons Select development tools Set up development environment Set up
More informationLmod. Robert McLay. Jan. 11, The Texas Advanced Computing Center
Robert McLay The Texas Advanced Computing Center Jan. 11, 2015 Why invent Y.A. Module System? Many thing right with the Original. But it was designed before multi-compilers/mpi Sites make it work but with
More informationMills HPC Tutorial Series. Mills HPC Basics
Mills HPC Tutorial Series Mills HPC Basics Objectives Overview: Mills Community Cluster Part I: Get your feet wet Part II: Jump in Overview: Mills HPC Basics Mills Community Cluster Background What is
More informationIf you had a freshly generated image from an LCI instructor, make sure to set the hostnames again:
Storage Node Setup A storage node (or system as your scale) is a very important unit for an HPC cluster. The computation is often about the data it produces and keeping that data safe is important. Safe
More informationCS631 - Advanced Programming in the UNIX Environment. UNIX development tools
CS631 - Advanced Programming in the UNIX Environment Slide 1 CS631 - Advanced Programming in the UNIX Environment UNIX development tools Department of Computer Science Stevens Institute of Technology Jan
More informationLezione 8. Shell command language Introduction. Sommario. Bioinformatica. Mauro Ceccanti e Alberto Paoluzzi
Lezione 8 Bioinformatica Mauro Ceccanti e Alberto Paoluzzi Dip. Informatica e Automazione Università Roma Tre Dip. Medicina Clinica Università La Sapienza Sommario Shell command language Introduction A
More informationCompilers & Optimized Librairies
Institut de calcul intensif et de stockage de masse Compilers & Optimized Librairies Modules Environment.bashrc env $PATH... Compilers : GNU, Intel, Portland Memory considerations : size, top, ulimit Hello
More informationLecture 4: Build Systems, Tar, Character Strings
CIS 330:! / / / / (_) / / / / _/_/ / / / / / \/ / /_/ / `/ \/ / / / _/_// / / / / /_ / /_/ / / / / /> < / /_/ / / / / /_/ / / / /_/ / / / / / \ /_/ /_/_/_/ _ \,_/_/ /_/\,_/ \ /_/ \ //_/ /_/ Lecture 4:
More informationcget Documentation Release Paul Fultz II
cget Documentation Release 0.1.0 Paul Fultz II Jun 27, 2018 Contents 1 Introduction 3 1.1 Installing cget.............................................. 3 1.2 Quickstart................................................
More informationEE516: Embedded Software Project 1. Setting Up Environment for Projects
EE516: Embedded Software Project 1. Setting Up Environment for Projects By Dong Jae Shin 2015. 09. 01. Contents Introduction to Projects of EE516 Tasks Setting Up Environment Virtual Machine Environment
More informationCS Unix Tools. Lecture 3 Making Bash Work For You Fall Hussam Abu-Libdeh based on slides by David Slater. September 13, 2010
Lecture 3 Making Bash Work For You Fall 2010 Hussam Abu-Libdeh based on slides by David Slater September 13, 2010 A little homework Homework 1 out now Due on Thursday at 11:59PM Moving around and GNU file
More informationIntroduction to Linux Workshop 1
Introduction to Linux Workshop 1 The George Washington University SEAS Computing Facility Created by Jason Hurlburt, Hadi Mohammadi, Marco Suarez hurlburj@gwu.edu Logging In The lab computers will authenticate
More informationIntroduction to Linux
Introduction to Operating Systems All computers that we interact with run an operating system There are several popular operating systems Operating Systems OS consists of a suite of basic software Operating
More informationUNIX COMMANDS AND SHELLS. UNIX Programming 2015 Fall by Euiseong Seo
UNIX COMMANDS AND SHELLS UNIX Programming 2015 Fall by Euiseong Seo What is a Shell? A system program that allows a user to execute Shell functions (internal commands) Other programs (external commands)
More informationLinux for Beginners. Windows users should download putty or bitvise:
Linux for Beginners Windows users should download putty or bitvise: https://putty.org/ Brief History UNIX (1969) written in PDP-7 assembly, not portable, and designed for programmers as a reaction by Bell
More informationLezione 8. Shell command language Introduction. Sommario. Bioinformatica. Esercitazione Introduzione al linguaggio di shell
Lezione 8 Bioinformatica Mauro Ceccanti e Alberto Paoluzzi Esercitazione Introduzione al linguaggio di shell Dip. Informatica e Automazione Università Roma Tre Dip. Medicina Clinica Università La Sapienza
More informationLING 408/508: Computational Techniques for Linguists. Lecture 5
LING 408/508: Computational Techniques for Linguists Lecture 5 Last Time Installing Ubuntu 18.04 LTS on top of VirtualBox Your Homework 2: did everyone succeed? Ubuntu VirtualBox Host OS: MacOS or Windows
More informationThe WAF build system
Sebastian Jeltsch Electronic Vision(s) Kirchhoff Institute for Physics Ruprecht-Karls-Universität Heidelberg 31. August 2010 Sebastian Jeltsch 31. August 2010 1 / 19 Introduction WorkBuildflow Sebastian
More informationInstalling CEA on an Intel Mac
Installing CEA on an Intel Mac Jonah Zimmerman April 20, 2012 1 Introduction Chemical Equilibrium with Applications (CEA) is a computer program published by NASA Glenn with the purpose of calculating chemical
More informationStudent Remote Login Procedure (see picture below): 1. Start SSH Secure Shell 2. Click the computer icon (4 th on the toolbar) 3.
Student Remote Login Procedure (see picture below): 1. Start SSH Secure Shell 2. Click the computer icon (4 th on the toolbar) 3. Enter stargate.ncc.edu in the text field labeled Host Name: 4. Enter the
More informationExercise 1: Basic Tools
Exercise 1: Basic Tools This exercise is created so everybody can learn the basic tools we will use during this course. It is really more like a tutorial than an exercise and, you are not required to submit
More informationFollow us on Twitter for important news and Compiling Programs
Follow us on Twitter for important news and updates: @ACCREVandy Compiling Programs Outline Compiling process Linking libraries Common compiling op2ons Automa2ng the process Program compilation Programmers
More informationUnix Filesystem. January 26 th, 2004 Class Meeting 2
Unix Filesystem January 26 th, 2004 Class Meeting 2 * Notes adapted by Christian Allgood from previous work by other members of the CS faculty at Virginia Tech Unix Filesystem! The filesystem is your interface
More informationApplied Informatics POCO PRO C++ Frameworks
Applied Informatics POCO PRO C++ Frameworks Getting Started Guide Version 1.10 Purpose of This Document This document guides developers interested in the POCO PRO C++ Frameworks by Applied Informatics
More informationLinux Operating System Environment Computadors Grau en Ciència i Enginyeria de Dades Q2
Linux Operating System Environment Computadors Grau en Ciència i Enginyeria de Dades 2017-2018 Q2 Facultat d Informàtica de Barcelona This first lab session is focused on getting experience in working
More informationIntroduction to the NCAR HPC Systems. 25 May 2018 Consulting Services Group Brian Vanderwende
Introduction to the NCAR HPC Systems 25 May 2018 Consulting Services Group Brian Vanderwende Topics to cover Overview of the NCAR cluster resources Basic tasks in the HPC environment Accessing pre-built
More informationCS197U: A Hands on Introduction to Unix
CS197U: A Hands on Introduction to Unix Lecture 11: WWW and Wrap up Tian Guo University of Massachusetts Amherst CICS 1 Reminders Assignment 4 was graded and scores on Moodle Assignment 5 was due and you
More informationIntroduction to Linux Scripting (Part 2) Brett Milash and Wim Cardoen CHPC User Services
Introduction to Linux Scripting (Part 2) Brett Milash and Wim Cardoen CHPC User Services Overview Advanced Scripting Compiling Code Getting the exercise files For today s exercises, open a session to one
More informationCrash Course in Unix. For more info check out the Unix man pages -orhttp://www.cs.rpi.edu/~hollingd/unix. -or- Unix in a Nutshell (an O Reilly book).
Crash Course in Unix For more info check out the Unix man pages -orhttp://www.cs.rpi.edu/~hollingd/unix -or- Unix in a Nutshell (an O Reilly book). 1 Unix Accounts To access a Unix system you need to have
More informationIntroduction Into Linux Lecture 1 Johannes Werner WS 2017
Introduction Into Linux Lecture 1 Johannes Werner WS 2017 Table of contents Introduction Operating systems Command line Programming Take home messages Introduction Lecturers Johannes Werner (j.werner@dkfz-heidelberg.de)
More informationUsing a Linux System 6
Canaan User Guide Connecting to the Cluster 1 SSH (Secure Shell) 1 Starting an ssh session from a Mac or Linux system 1 Starting an ssh session from a Windows PC 1 Once you're connected... 1 Ending an
More informationIntroduction to Linux Scripting (Part 2) Brett Milash and Wim Cardoen CHPC User Services
Introduction to Linux Scripting (Part 2) Brett Milash and Wim Cardoen CHPC User Services Overview Advanced Scripting Compiling Code Getting the exercise files For today s exercises, open a session to linuxclass.chpc.utah.edu
More informationMaemo Diablo GNU Make and makefiles Training Material
Maemo Diablo GNU Make and makefiles Training Material February 9, 2009 Contents 1 GNU Make and makefiles 2 1.1 What is GNU Make?......................... 2 1.2 How does make work?........................
More informationToday. Operating System Evolution. CSCI 4061 Introduction to Operating Systems. Gen 1: Mono-programming ( ) OS Evolution Unix Overview
Today CSCI 4061 Introduction to s Instructor: Abhishek Chandra OS Evolution Unix Overview Unix Structure Shells and Utilities Calls and APIs 2 Evolution How did the OS evolve? Dependent on hardware and
More informationHPC Introductory Course - Exercises
HPC Introductory Course - Exercises The exercises in the following sections will guide you understand and become more familiar with how to use the Balena HPC service. Lines which start with $ are commands
More informationToday. Operating System Evolution. CSCI 4061 Introduction to Operating Systems. Gen 1: Mono-programming ( ) OS Evolution Unix Overview
Today CSCI 4061 Introduction to s Instructor: Abhishek Chandra OS Evolution Unix Overview Unix Structure Shells and Utilities Calls and APIs 2 Evolution How did the OS evolve? Generation 1: Mono-programming
More informationIntroduction to Linux. Woo-Yeong Jeong Computer Systems Laboratory Sungkyunkwan University
Introduction to Linux Woo-Yeong Jeong (wooyeong@csl.skku.edu) Computer Systems Laboratory Sungkyunkwan University http://csl.skku.edu What is Linux? A Unix-like operating system of a computer What is an
More informationbwunicluster Tutorial Access, Data Transfer, Compiling, Modulefiles, Batch Jobs
bwunicluster Tutorial Access, Data Transfer, Compiling, Modulefiles, Batch Jobs Frauke Bösert, SCC, KIT 1 Material: Slides & Scripts https://indico.scc.kit.edu/indico/event/263/ @bwunicluster/forhlr I/ForHLR
More informationIntroduction to Linux
Introduction to Linux Prof. Jin-Soo Kim( jinsookim@skku.edu) TA - Kisik Jeong (kisik@csl.skku.edu) Computer Systems Laboratory Sungkyunkwan University http://csl.skku.edu What is Linux? A Unix-like operating
More informationIntroduction to Linux
Introduction to Linux Mukesh Pund Principal Scientist, NISCAIR, New Delhi, India History In 1969, a team of developers developed a new operating system called Unix which was written using C Linus Torvalds,
More informationNew User Seminar: Part 2 (best practices)
New User Seminar: Part 2 (best practices) General Interest Seminar January 2015 Hugh Merz merz@sharcnet.ca Session Outline Submitting Jobs Minimizing queue waits Investigating jobs Checkpointing Efficiency
More informationardpower Documentation
ardpower Documentation Release v1.2.0 Anirban Roy Das May 18, 2016 Contents 1 Introduction 1 2 Screenshot 3 3 Documentaion 5 3.1 Overview................................................. 5 3.2 Installation................................................
More informationModules. Help users managing their shell environment. Xavier Delaruelle UST4HPC May 15th 2018, Villa Clythia, Fréjus
Modules Help users managing their shell environment Xavier Delaruelle UST4HPC May 15th 2018, Villa Clythia, Fréjus whoami I am Xavier Delaruelle Joined CEA in 2007 as HPC system
More informationCSC BioWeek 2018: Using Taito cluster for high throughput data analysis
CSC BioWeek 2018: Using Taito cluster for high throughput data analysis 7. 2. 2018 Running Jobs in CSC Servers Exercise 1: Running a simple batch job in Taito We will run a small alignment using BWA: https://research.csc.fi/-/bwa
More informationVaango Installation Guide
Vaango Installation Guide Version Version 17.10 October 1, 2017 The Utah Vaango team and Biswajit Banerjee Copyright 2015-2017 Parresia Research Limited The contents of this manual can and will change
More informationUnix/Linux Basics. Cpt S 223, Fall 2007 Copyright: Washington State University
Unix/Linux Basics 1 Some basics to remember Everything is case sensitive Eg., you can have two different files of the same name but different case in the same folder Console-driven (same as terminal )
More informationEECS Software Tools. Lab 2 Tutorial: Introduction to UNIX/Linux. Tilemachos Pechlivanoglou
EECS 2031 - Software Tools Lab 2 Tutorial: Introduction to UNIX/Linux Tilemachos Pechlivanoglou (tipech@eecs.yorku.ca) Sep 22 & 25, 2017 Material marked with will be in your exams Sep 22 & 25, 2017 Introduction
More informationDAVE LIDDAMENT INTRODUCTION TO BASH
DAVE LIDDAMENT INTRODUCTION TO BASH @daveliddament FORMAT Short lectures Practical exercises (help each other) Write scripts LEARNING OBJECTIVES What is Bash When should you use Bash Basic concepts of
More informationEnvironment Variables
Environment Variables 1 A shell is simply a program that supplies certain services to users. As such, a shell may take parameters whose values modify or define certain behaviors. These parameters (or shell
More informationbwunicluster Tutorial Access, Data Transfer, Compiling, Modulefiles, Batch Jobs
bwunicluster Tutorial Access, Data Transfer, Compiling, Modulefiles, Batch Jobs Frauke Bösert, SCC, KIT 1 Material: Slides & Scripts https://indico.scc.kit.edu/indico/event/263/ @bwunicluster/forhlr I/ForHLR
More informationIntroduc)on to Linux HPC Workshop
Introduc)on to Linux HPC Workshop Erin Shaw and Cesar Sul Advanced Cyberinfrastructure Research and Educa)on Facilita)on USC Center for High-Performance Compu)ng Introduction Show New User slides Linux
More informationFiles
http://www.cs.fsu.edu/~langley/cop3353-2013-1/reveal.js-2013-02-11/02.html?print-pdf 02/11/2013 10:55 AM Files A normal "flat" file is a collection of information. It's usually stored somewhere reasonably
More informationVi & Shell Scripting
Vi & Shell Scripting Comp-206 : Introduction to Week 3 Joseph Vybihal Computer Science McGill University Announcements Sina Meraji's office hours Trottier 3rd floor open area Tuesday 1:30 2:30 PM Thursday
More informationIntroduction to Linux
Introduction to Linux Prof. Jin-Soo Kim( jinsookim@skku.edu) TA - Dong-Yun Lee (dylee@csl.skku.edu) Computer Systems Laboratory Sungkyunkwan University http://csl.skku.edu What is Linux? A Unix-like operating
More informationLecture 3. Unix. Question? b. The world s best restaurant. c. Being in the top three happiest countries in the world.
Lecture 3 Unix Question? Denmark is famous for? a. LEGO. b. The world s best restaurant. c. Being in the top three happiest countries in the world. d. Having the highest taxes in Europe (57%). e. All of
More informationDuke Compute Cluster Workshop. 11/10/2016 Tom Milledge h:ps://rc.duke.edu/
Duke Compute Cluster Workshop 11/10/2016 Tom Milledge h:ps://rc.duke.edu/ rescompu>ng@duke.edu Outline of talk Overview of Research Compu>ng resources Duke Compute Cluster overview Running interac>ve and
More informationAASPI Software Structure
AASPI Software Structure Introduction The AASPI software comprises a rich collection of seismic attribute generation, data conditioning, and multiattribute machine-learning analysis tools constructed by
More informationIntroduction to High-Performance Computing (HPC)
Introduction to High-Performance Computing (HPC) Computer components CPU : Central Processing Unit cores : individual processing units within a CPU Storage : Disk drives HDD : Hard Disk Drive SSD : Solid
More informationMAKING CONTAINERS EASIER WITH HPC CONTAINER MAKER. Scott McMillan September 2018
MAKING CONTAINERS EASIER WITH HPC CONTAINER MAKER Scott McMillan September 2018 NVIDIA GPU CLOUD (NGC) Simple Access to Ready to-run, GPU-Accelerated Software Discover 35 GPU-Accelerated Containers Deep
More informationMeeting of the Technical Steering Committee (TSC) Board
http://openhpc.community Meeting of the Technical Steering Committee (TSC) Board Tuesday, July 31 th 2018 11:00am ET Meeting Logistics https://zoom.us/j/556149142 United States : +1 (646) 558-8656 -Meeting
More informationBeyond Makefiles: Autotools and the GNU Build System
SEA : Autotools and the GNU Build System Patrick Nichols Software Engineer CISL Consulting Group UCAR SEA December 10, 2015 Why do we need more tools? Diversity! 1. Compilers, programming languages and
More informationPerl and R Scripting for Biologists
Perl and R Scripting for Biologists Lukas Mueller PLBR 4092 Course overview Linux basics (today) Linux advanced (Aure, next week) Why Linux? Free open source operating system based on UNIX specifications
More informationCombining CVMFS, Nix, Lmod, and EasyBuild at Compute Canada. Bart Oldeman, McGill HPC, Calcul Québec, Compute Canada
Combining CVMFS, Nix, Lmod, and EasyBuild at Compute Canada Bart Oldeman, McGill HPC, Calcul Québec, Compute Canada Motivation 1. New bigger national systems replacing many smaller local clusters, with
More informationIntroduction to HPC Programming 4. C and FORTRAN compilers; make, configure, cmake. Valentin Pavlov
Introduction to HPC Programming 4. C and FORTRAN compilers; make, configure, cmake Valentin Pavlov About these lectures This is the fourth of series of six introductory lectures discussing
More informationVirtual Machine. Linux flavor : Debian. Everything (except slides) preinstalled for you. https://www.virtualbox.org/
Virtual Machine Anyone have problems installing it? VM: Virtual Box - allows you to run a different operating system within the current operating system of your machine. https://www.virtualbox.org/ Linux
More informationIntroduction to Linux Basics
Introduction to Linux Basics Part-I Georgia Advanced Computing Resource Center University of Georgia Zhuofei Hou, HPC Trainer zhuofei@uga.edu Outline What is GACRC? What is Linux? Linux Command, Shell
More informationUNIX System Programming Lecture 3: BASH Programming
UNIX System Programming Outline Filesystems Redirection Shell Programming Reference BLP: Chapter 2 BFAQ: Bash FAQ BMAN: Bash man page BPRI: Bash Programming Introduction BABS: Advanced Bash Scripting Guide
More informationGNU make... Martin Ohlerich, Parallel Programming of High Performance Systems
... Martin Ohlerich, Martin.Ohlerich@lrz.de Parallel Programming of High Performance Systems Outline 1 2 3 Leibniz Rechenzentrum 2 / 42 Outline 1 2 3 Leibniz Rechenzentrum 3 / 42 Common Situation Larger
More informationShifter at CSCS Docker Containers for HPC
Shifter at CSCS Docker Containers for HPC HPC Advisory Council Swiss Conference Alberto Madonna, Lucas Benedicic, Felipe A. Cruz, Kean Mariotti - CSCS April 9 th, 2018 Table of Contents 1. Introduction
More informationInstalling CEA on an Intel Mac
Installing CEA on an Intel Mac Jonah Zimmerman April 20, 2012 1 Introduction Chemical Equilibrium with Applications (CEA) is a computer program published by NASA Glenn with the purpose of calculating chemical
More informationLmod: A Modern Environment Module System XALT: Understanding HPC Usage via Job Level Collection. June 14, 2017
Lmod: A Modern Environment Module System XALT: Understanding HPC Usage via Job Level Collection June 14, 2017 Outline What are Environment Modules? What is Lmod? Can Lmod help manage your site? Advanced
More information24 Writing Your First Script
In the preceding chapters, we have assembled an arsenal of command line tools. While these tools can solve many kinds of computing problems, we are still limited to manually using them one by one on the
More informationReview of Fundamentals
Review of Fundamentals 1 The shell vi General shell review 2 http://teaching.idallen.com/cst8207/14f/notes/120_shell_basics.html The shell is a program that is executed for us automatically when we log
More informationQuick Start Guide. by Burak Himmetoglu. Supercomputing Consultant. Enterprise Technology Services & Center for Scientific Computing
Quick Start Guide by Burak Himmetoglu Supercomputing Consultant Enterprise Technology Services & Center for Scientific Computing E-mail: bhimmetoglu@ucsb.edu Contents User access, logging in Linux/Unix
More informationLinux Systems Administration Getting Started with Linux
Linux Systems Administration Getting Started with Linux Network Startup Resource Center www.nsrc.org These materials are licensed under the Creative Commons Attribution-NonCommercial 4.0 International
More informationRunning LAMMPS on CC servers at IITM
Running LAMMPS on CC servers at IITM Srihari Sundar September 9, 2016 This tutorial assumes prior knowledge about LAMMPS [2, 1] and deals with running LAMMPS scripts on the compute servers at the computer
More information